The most common person there is the white male, who generally does *not* feel besieged. After all, the civil rights movement was about half a century ago and there still aren't many (or often any) black people around in the meeting, except maybe as a guest speaker. The white women are comfortably white, and the Asians there are typically quite white-ish.

The class is typically being run by an outside contractor who is nominally teaching the people how to embrace these lofty goals but really teaching the people how not to get sued. This person is almost always a white woman whose business is dependent on getting invited to do this again. Her ticket to that invitation is to get through things as quickly as allowed without making the people present (who are mostly white and mostly male) too uncomfortable.

What makes (many of those) people uncomfortable is having to confront the details of being oppressed or their agency in that. The term "person of diversity" is a thoroughly generic term that includes none of those details. It can be said quickly and cover that ground theoretically without actually having to confront any of the unpleasant details.

The biggest threat to adoption of this term isn't backlash. It's the development of an alternative.