Absolutely. JPG is 24 bit but that's only 8 bit per color. Where fits is 16bit and grayscale and also linear, that's a significant difference. Also let's not forget that the JPG data is also compressed. But the real issue is that the JPG data is NOT linear coming out of the camera. Things get automatically stretched and white balance gets applied. This causes issues mostly with plate solving but to a lesser amount with focusing.
I'm not being OCD here. We initially used JPG for plate solve and auto focus and had significant issues with it. So we went back to raw and those problems started going away. We don't just use RAWs for fun, we would much prefer JPG as they're smaller and easier to deal with than having to convert the RAW data. Some plate solvers deal with JPG better than others...
It might not be the download time. If you have "Image History" turned on SGP will also do some analytics on your image. These can take some time if you're using a slow machine. May want to turn that off and see if your time improves.
Raw download time on my 6D is about 7 seconds as well. If I have "Image History" turned on that can add a couple more seconds using my i7 with SSD.
Thanks,
Jared