filters - Circular polarizer works great at shorter focal lengths, but significantly degrades the image at longer lengths. Why?

I bought a cheap CP recently to play around with, and noticed an interesting phenomenon:

Using it with my 24mm lens on full-frame, images turn out tack-sharp - there is hardly any difference in sharpness/clarity with and without the filter.

However zooming in to longer focal lengths, I notice that the image quality degrades significantly. At 50mm you can begin to notice that images are not quite as sharp. At ~100mm images are significantly blurrier, and out to ~200mm the filtered images are pretty much unusable.

What about a circular polarizer would cause varying optical performance dependent on focal length? Is this something that higher-end filters can solve, or is it a physical limit (like diffraction limiting?) at play?