That would happen when you won't lift the POE HAT straight up from the PI, it seems indeed the weakest link, but this header is structurally no different than the main 40-pin connector, its just that if you don't pull the board off straight, you can put a lever effect on it that rips the smaller connector right off the board.
I suggest that a drop of super glue dripped on the hole where the plastic pin sticks through the board will strengthen the mechanical connection if you plan to remove the POE HAT often, but more important is using the right way to lift off the POE HAT, I suggest putting a flathead screwdriver between the 4-pin pinheader and this connector to assist in separating both boards.
Another HAT can be put on the POE HAT, (specific details have been changed to make that possible) and how to do that is described on the BLOG.
the trick is to use a specific pin header extenders with longer pins that stick through the holes of the POE HAT 40-pin female "transparent" push-through connector. for this 4-pin connector a similar extender (type SSQ-102-01-G-D) is needed, its all described in the blog here;
https://www.raspberrypi.org/blog/introd ... t-poe-hat/
If I'm allowed to give the producers of the POE HAT a TIP, I would enlarge the solder pads for the 4-pin connector, at least to double the current size, perhaps with a few via's around them, and I would either add a drop of glue, or melt the plastic pin of the 4-pin connector, but yes it would only mitigate the problem, as it will always be possible to lever the connector off.