I’d suggest just using a metal pole rather than PVC and the problem will go away. I found that conduit hangers on the 4x4 post for the fence were perfect for mounting the pole especially if you use stainless bolts+nuts. Then you don’t even need tools to take the pole down periodically, just pinch the hangers and it’s easy to spin the nut off. Works great.
Ancient links to the products I use are:
I used the same conduit hangers, though the bigger ones, for the Harbor Freight telescoping flagpole that I have my Davis VP2 on with the anemometer at 19’ elevation or so. One person job to pop that down rarely for cleaning. Rock solid.