Jewish Group

It is not patronizing or condescending, it is outright hostile. It has been used almost exclusively by hardcore antisemites to label their Jewish (not Zionist) targets. It has a history that goes back over a century, and still there are people who refuse to acknowledge it to be antisemitic.

The stealth antisemitism you are referring to comes from an unquestionable core presumption that Jews are always up to something nefarious. This presumption requires no evidence for justification. On the contrary, it is the unshakable cornerstone of antisemitism. In the absence of this presumption, antisemitism ceases to exist.

In your case, when this unshakable core foundation has been challenged by the fact of you being a nice and harmless person that the people who got to know you cannot deny, most of them will still find it impossible to abandon their core belief That's when they turn to condescending sympathy for the poor stupid Jew who doesn't realize how evil he is.

It takes a lot more honesty and self-confidence to admit that being antisemitic is the evil side of this relationship.