You should check to make sure the Camera Raw Cache folder isn't near the size limit you have set (10GB). If so increase the size accordingly.
I have a similar system as yours and use 5D MKII raw files, but my display is 1920x1080. At 'Fit' view with only the R & L panels pulled in the delay to sharp image and where I can start making adjustments is ~2 sec. If I pull the R, L & Bottom panels out so the image is large as possible I see the same response of ~1 to 2 sec sharp, then blurry another 3 to 4 sec., typically 5 sec. total.
Technically there is nothing to stop you from working in the Develop module once the spinning Loading icon goes away (~2 sec.). Yes, the image may not be sharp, but start setting Exposure, Contrast, etc. in the "top-down" suggested flow for the Basic panel controls. By the time you reach 'Blacks' the image should be sharp and you can move onto things like Sharpening and NR. Zooming to 1:1 at this point takes about 2 sec. on my system.
I hope this helps.
