Yes, you are absolutely right. I find that small errors in semantics or grammar that don't actually hinder communication at all will make the xenophobes jump on people, precisely because, as you say, language is the most visible marker.

Thank you for bringing that up, I think that's an important aspect.