broad-specific

There are broad domains in which everyone has experience, like driving & weather, making them accessible topics of conversation. Everyone can relate to these things; small talk is so-called because the risk of miscommunication is small. In specific domains*, the odds that a random interlocutor will be able to entertain a conversation shrinks. It is somewhat unlikely that someone watches anime, improbable that they watched Evangelion, and vanishingly unlikely that they watched Serial Experiments Lain. It is somewhat unlikely that someone remembers high school math, improbable that they have undergraduate mathematics knowledge, and vanishingly unlikely that they would enjoy discussing solution techniques for second order partial differential equations. So these topics make less and less sense to bring up around strangers.

The problem is that everyone has their own niche areas of interest and feels unique in their isolation. So, it is not enough to say that one's interests are more niche than anyone else's; it's just that there's unlikely to be much overlap between any two people. Part of what I believe makes a great conversationalist is the ability to recognize the similarities between qualitatively dissimilar topics, making people feel connected in novel ways.

*a college English professor might say discourse communities.