The difficulty I see is that you would not use the term on the street or outside the context of the stadium, because it is antisemitic. We all know that the context in the ground, when it's chanted, is not meant to be in any way racist.
Now I don't think it's helpful to put this in the way that 'this is David Baddiel wanting to shut down spurs fans', because if no one else backed his view, he'd be an irrelevance by now.
Now, there are plenty of Jewish people who don't go to football, have no relationship with Spurs or North London, who may have the term 'Yid' used against them in a offensive way.
The nagging issue for me is that we're keeping the term alive by continuing it's use, even with it's positive context, there is a knock on effect outside of our sphere and that's where I have some discomfort, it's not about faux offence from other sets of supporters, because to a great deal of them, it's just a game.
I'm not going to stop anyone else from chanting it or lecture anyone else during a match and I'm pretty sure that it is not illegal the last time the issue was tested, but there is a moral argument that deserves debate.