Partly because they've created a society where they have most of it by their flawed definition.

Privilege comes from the Latin meaning "Private Law." So in a literal sense, unless there are laws to some effect, favoring or harming others, there is no privilege. For example, if there was a dedicated driving lane for those of a certain class, that would be privilege. I'd also be part of it and sneer at you inferior drivers, because honestly, that really should be a thing.

Actor, Action, Patient, Agent:

The ACTOR acts out.  The ACTION affects the PATIENT, for positive or negative. An AGENT is anyone capable of acting.  In a hypothetical extreme privilege society, the PATIENT has no AGENCY and cannot fight the ACTION.  Slavery in the antebellum South is a good example of this. Slaves had very little agency whatsoever, and when they did, it was usually through their owner by proxy with no personal say in the matter.

But that's not the world we live in now.  The law officially applies to everyone equally. Now, enforcement varies, and it usually works to the advantage of the wealthy and celebrities of ANY DEMOGRAPHIC.

But if, let's say, a cop and a judge decide to charge and sentence two people differently, based on race, religion, political affiliation.  The PATIENTS of these ACTIONS don't have much AGENCY to change the ACTORS' intentions. Possibly a better lawyer or publicity can change things, but there's no official strata, no, "Hey, you can't do that to me, I have this card that says ___" (Well, sometimes politicians and cops have such a card. Foreign diplomats do to an extent.  But it's exceptionally rare for anyone else.)

"They treated you better, YOU HAVE PRIVILEGE!"

No, I fucking don't. And by claiming so, YOU have just diverted the ACT from the ACTOR to me, one of the PATIENTS.  "You got treated better! You are evil!" Wrong. The ACTORS did the wrongdoing. You and I are both PATIENTS.  You are literally arguing that if you get punched in the face and I don't, that it's because I'm the bad guy, not the asshole who's punching people.

Stop with the bullshit claims of PRIVILEGE for the patients of the action, and go after the fucking ACTORS. Because every time you accuse someone else of "privilege" for actions totally beyond their control, you piss them off and make it even harder to go after the ACTORS who cause the problems.