I feel like "It's just a preference" is such a lazy and uncritical response to the question. Writing someone off based on their race is like literally one of the definitions of racism. So while you may not be
actively racist, you are certainly operating in racist modes of thinking.
It's one thing to say you prefer light eyes and fair skin, and something quite else to say that you won't date a black/Asian/Latino man because most of them may not fit into that mold. Every racial group on the planet (and especially "black" people) have such tremendous phenotypic diversity that it's foolish to say any one group all looks the same. So I would say it's not racist to have a preference for certain physical characteristics, but it is very much so to discount or include someone solely based on race. I am most strongly attracted to big, muscular, hairy men but I would never say I don't date Asians because most of them (in my experience) don't look like that.
All that being said, the phenomenon of "black men only dating white men" is such a convoluted and layered discussion. There's often a degree of fetishization that is reciprocated in both partners, and you can't skim over the whitewashed, Eurocentric beauty standards that the Western World operates in. Social conditioning is very real and very rarely is there an aspect of our lives that isn't affected by it. I'm not saying that all or even most black-white gay relationships have such pathologies, but it's certainly something worth pondering.
And honestly I will NEVER understand the sentiment of only dating people of one race.
There are fine men of EVERY creed and color. I have been with men of almost every race in the past, and look quite forward to continuing that on into the future.