Is every story that includes characters who aren't straight, white and Western "pandering?"

Obviously not.

You're certainly wrong in believing that until recently, no one ever wrote about characters who were not straight, white, and Western.