To begin with, the very concept of "privilege", defined as a benefit ascribed to those who are claimed not to suffer from racial or ethnic discrimination, is completely redundant. We already have the much clearer concept of discrimination itself, and even the somewhat-more-vague concept of pervasive or systemic discrimination. Turning these around and ascribing "privilege" to those who are not victims of discrimination does nothing to make our understanding of the phenomenon clearer.
On the contrary, it clouds understanding by implicitly recasting a quantitative phenomenon as a qualitative one. Even its proponents admit that as defined, everyone has some degree of "privilege". (Black people, for instance, would appear to benefit from the "privilege" of not being the victims of anti-Hispanic discrimination, and vice versa.) But nobody ever asks a speaker to "check how much privilege you have compared to some other people". Both its definition and its common use encourage its misguided interpretation as an all-or-nothing property.
Even worse, this qualitative property is then attributed purely based on racial or gender category. Thus all white people are lumped together as benefiting from "white privilege", irrespective of their personal or family attributes and experiences. "White privilege", then, is best described as a property attributed uniformly to all members of a particular racial group. There's a word for such attributions: they're called racial stereotypes.
Defenders of the term argue that "white privilege" is nevertheless different from pernicious stereotypes such as "black criminality" or "Jewish greed" in that all white people really do derive some net benefit from systemic discrimination against non-white people--that is, from "privilege". There's a subtle sleight-of-hand in this argument: the absence of a particular disadvantage is used to imply the presence of a benefit. It's probably true, after all, that all white people are less negatively affected by discrimination against blacks than blacks themselves are. But that does not imply that all whites derive a net benefit from discrimination against blacks. On the contrary, since (as defenders of the concept of "privilege"--again--concede) discrimination is generally negative-sum, hurting non-victims as well as victims, it's likely that a great many--perhaps even most--whites suffer net harm from discrimination against blacks, rather than a net benefit, compared with a world without such discrimination. Widespread discrimination against blacks in hiring, for instance, would have broad negative economic effects that would for most whites dwarf any hiring advantage gained as a result--particularly, say, in the case of a long-term unemployed white person, for whom an unrealized hypothetical hiring advantage has yielded precious little benefit, but whose employment problems may well have been exacerbated by the economic side effects of discrimination.
This simple observation renders the entire concept of "privilege" completely nonsensical. It would be absurd, for instance, to claim that Americans "benefited" from "privilege" as a result of the 2011 Japanese tsunami, even though they obviously suffered less, on average, than the direct victims did. Likewise, given that racial or ethnic discrimination against minority groups is bad for society in general, it makes no sense to claim that all whites "benefit" from it simply because they are generally hurt less than minority group members. Hence "white privilege" is no more accurate than "black criminality" or "Jewish greed"--they're all false and unfair generalizations based on race or ethnicity.
There is, however, one real and significant difference: "white privilege" wasn't generated spontaneously by a bigoted culture, but instead was deliberately invented in the American academic community and then vigorously promoted to both students and the public at large. Why would anyone--let alone American academics, who typically declare themselves passionately devoted to racial equality--do something so perverse?
The answer is actually quite obvious: to justify their own brand of racial discrimination. "Affirmative action"--that is, legally sanctioned discrimination against certain racial and ethnic groups in academic and vocational contexts--is under fire in the political sphere, since it is hugely unpopular, violates important, deeply held American values and has largely failed to close the achievement gaps it was meant to address. Its supporters needed a rationalizing justification for its continuation, and turned to the tried-and-true method of racists throughout history: the negative racial stereotype. Thus do the parallels between "affirmative action" and the racist discrimination it was supposedly enacted to redress continue to grow ever closer and more compelling.