Honest answer...
The Jewish side of it I think plays a part. Historically, we've been subject to antisemitic chants right up until what, 5-10 years ago? Mostly from the usual suspects, but there are others that feel that way towards the club as well.
Then there's the fact that we've been a banter club for a while, after we crashed from a great height under Scholar.
There are a lot of people, in this country, whose go to emotions are anger and hate, we've been an easy target for a while.
But then we started getting our house in order, and built ourselves back to being a contender. This just solidifies that hatred for the also runs, as we've done what they couldn't. Rather than hate on their own club for their failures, the easy target is, once again, us.
There are Jewish connections in both Chelsea and Woolwich, but they don't have the "history" that we do as a "Jewish" club, so that gets overlooked. Christ, we had Baddiel accusing US of being racist for calling ourselves Yids, completely ignoring the FACT that the moniker was bestowed on us by them, he just hated the fact that we deweaponised it. Baddiel is older than me, so he'd have been present when the gas chamber chants were used. Didn't hear him playing the race card then.
People in this World don't really need to much urging to hate in Football, citing "tribalism" as their excuse. In my experience, it's a mask to darker leanings for far too many. The reactions to certain Black players in the last Euro's emphasised that. However, when there's no individuals to focus that hatred on during the domestic season, our club appears to be the fall back option.