The main conflict in the Harry Potter series lies in the belief that those who do not have magical abilities are inferior to those who do. This particular form of othering is entirely fictional, yet it characterises many prejudices and resonates with readers. We may like to envision ourselves as witches and wizards when we are immersed in the story, but the truth is that we all identify with the Muggles and Muggle-borns. Lord Voldemort's attack on them is an attack on us all.
In contrast, the witches and wizards who live in their bubbles may not necessarily have vindictive thoughts towards Muggles, but they still view them as "other." We, like them, are liable to focus on what affects us and not what impacts others. Yet, someone who doesn't oppress you can still oppress other people, and someone who supports one aspect of yourself can still oppose another. Does Voldemort appeal to his followers because he supports every facet of their identity, or does he so acutely tap into their fears and prejudices that they are willing to overlook all else? After all, one of Voldemort’s followers is Fenrir Greyback, a werewolf. Yet Voldemort speaks of werewolves with a similar disdain to Muggles.
Now, we know that Greyback is nasty piece of work, but we also know, because of Remus Lupin, that not all werewolves are like him. This was a lesson Remus' father, Lyall Lupin, had to learn the hard way, when he said that werewolves were "soulless, evil, deserving nothing but death." We repeatedly see prejudice and ignorance, often from characters we identify with or consider to be good people. Ron is horrified to discover that Hagrid is half-giant, describing them as "vicious killers," and Hagrid himself makes negative generalisations about centaurs. Hermione may be outraged at the treatment of House Elves, but the majority of her peers are dismissive or complacent on the issue.
Prejudice is not as clear-cut as Magic vs No-Maj. There is a lot of intersection. It's all too easy to be wrapped up in ourselves, to be unaffected by another group's struggles, or to dismiss our own prejudices because they aren't as blatant as others'. I know that we all wish sometimes that we were witches and wizards and could do things we only dream of and read about in fiction. Yet I think there is so much more strength in striving to be an open-minded and kindhearted Muggle than there is to being a witch or a wizard in a bubble.