I’m one of the lucky few authors who have had the chance to see the inside of the publishing process firsthand. I worked for a Big Five publishing house for about five years. I’ve seen the way previous sales and comparable titles factor into the decision to buy a book, along with many discussions about how much a publishing team does or doesn’t love a project. “The team just didn’t love it enough to make an offer” was a phrase I often heard.

For much of those five years, I was the only black person at the table in meetings and one of the few people of color. Our office was actually more diverse than similar imprints. I was glad that I had a chance to make my voice heard. Still, I struggled with the challenges of being the only one.

For a few years, I was the only black local student at a private school on St. Thomas in the U.S. Virgin Islands, and I was the only black student in some college classes on an overwhelmingly white campus. My experiences of being “the only one” are what helped shape my fantasy novel Queen of the Conquered as I looked at the complexities of privilege and oppression through a Caribbean-inspired fantasy lens.

Maybe it’s because I’ve spent so much of my life as “the only one” that I’m attuned to the subtleties of discrimination. Being the only black child in my elementary school classes, I was often the only one in trouble with my white teachers. My college offered spaces for people of color, but there were white students who complained that our protests about unarmed black people shot by police were “too loud.” Working in a corporate job as an editor, the greatest difficulty was convincing white coworkers and supervisors that books I wanted to acquire featuring characters with brown skin were worthy.

“I didn’t connect with the character”; “The plot didn’t have enough tension”; “I didn’t find the story relatable”: often times, these are phrases said by white readers who can’t connect with a character of color because they haven’t been asked to in the way that so many people of color are regularly asked to connect with white characters. Many speculative books feature stories of slavery and discrimination and oppression but no people of color. Our stories and settings are taken because they are deemed exciting—the stories of underdogs rising from the ashes—but our society teaches us that brown skin isn’t worth a featured role. People of color aren’t often afforded the opportunity to tell our stories with main characters that look like us.

Fantasy and science fiction novels are metaphors for our world. If a speculative book looks at our world through the eyes of a marginalized person, white readers might not connect with that character, because they haven’t faced racism. They’re not excited by a plot where black and brown characters must fight white oppressors, because they have to see themselves as the villain rather than the hero. They see a main character that doesn’t look like them, and without quite knowing why, they don’t find the story relatable.

After the first Hunger Games movie came out, racist fans were shocked to learn that Rue, a major character and friend of Katniss, is black. “Awkward moment when Rue is some black girl and not the little blonde innocent girl you picture,” one viewer wrote on Twitter.

This mentality prevails. If a book features a character whom white readers can’t conceive of as relatable, they decide the book simply isn’t good. They don’t realize they’re centering the book’s worth on their own identity and ignoring the possibility that the book wasn’t written for them, in the same way that most stories I read growing up featuring all white, straight, and cis protagonists weren’t written for me.

To create more diverse lists, publishers with overwhelmingly white teams need to see and understand this, first and foremost: sometimes, books just weren’t written for them. If a story isn’t relatable to their own life, it doesn’t mean the book isn’t good. It means that they and the writer view the world through different lenses. To publish more diverse lists across the board, including science fiction and fantasy, publishers need to hire diverse editors who might relate to and connect with more diverse stories, so that they can help to put those much-needed books into the world.

Kacen Callender is the author of the Lambda Literary Award–winning middle grade novel Hurricane Child and the YA novel This Is Kind of an Epic Love Story. Queen of the Conquered is their first adult novel, out from Orbit on November 12.