There's a difference between "supposed to be rare" and frustrating game design. Frustrating game design is saying we're ok with having 100's of people farming the same location for hours, days, months, years...etc...because...we want it to be rare. Not frustrating game design is when its rare but you're only competing with a few people because you chose a server to begin with that is not high pop - just medium.
You make a good point in your first couple of sentences, and that's something we're evaluating. That said, I don't agree with your conclusion that the "good game design" option is the one where the world is just under populated to the point that competition isn't much of a factor. Having low-pop realms wasn't a design decision -- it's a symptom of numerous variables. It's not acceptable for us to say that game design should be such that players can decide how rare something actually is by choosing the most accommodating realm to play on.
What we want to do is continue to ensure the world feels populated, but that seeing people around you doesn't always feel like a negative thing. You've seen some of this in the way we've been experimenting with mob tapping the last couple of patches. Seeing people from your own faction in the world, especially, should be more of a positive thing. It's part of the fiction that you're supposed to be allies, not nuisances to each other.
We don't have any specific plans to share for how we might change the way rare pet spawns work, though competition is likely to remain an inherent factor. But we'll continue to look for ways we can mitigate some of the annoyances of competing for some types of rare spawns with large numbers of players.
It's important to note, however, that tinkering with these types of design decisions requires a metaphorical scalpel, when most of the feedback we get is akin to demanding the use of a butcher knife. No, we're not going to hastily redesign entire systems in the game because people are interested in catching a rare spawn, and can't all seem to succeed within less than 24 hours out of the (approximately) 2160 hours it has a chance to spawn this year.