You're putting "colored" before "people" as if that's more important than his or her personhood. It's a pejorative in this context. It's not like saying "short person" or "tall person."
Besides that, in the US there were "colored bathrooms" and "colored water fountains" in the super racist 1900s-1960s. It's just...