Hi DF, I remember in the 60s there was no such vitriol as there is now. I went to West Ham with a few Spurs fans, a West Ham fan, a Man U fan (the dreaded 'cockney red') who was supporting West Ham. We stood together, and had some banter, and I wasn't remotely scared.
It was a game of football, a great game BTW, the 'classic in the mud', Greaves and Gilly for us after Hurst and Peters for them. We were right behind the goal where we scored. I can close my eyes and hear the swish of the net, like it was yesterday. England, yep Mike England, dancing down the wing to set up Greaves' goal, great memories. Now I wouldn't even go to that hellhole, let alone sit amongst a mixed crowd of Spurs and West Ham fans.
The hatred started building from them in the 70s I guess, as the hooliganism grew, plus other factors. I remember last time I went in 1980, it was vivid, intense, and fucking nasty. They looked to ambush fans on the tube trains amongst other places, I think an oppo fan (Millwall?) had died not long before under the wheels of a train. I was stood amidst commuters as the 'Ammers got on the train looking for Spurs. I remember the fear and disgust as they worked their way past me.
We lost that time, but I was never going back, win or lose.
It all then get worse re us and them as they decided that Woolwich were out of their lge, they were out of Millwall's lge, so we must be rivals. I began to notice the difference as I moved jobs and the easy banter of my 60s schooldays and workmates in the 70s started to turn confrontational.
The day of the norovirus sealed it for me. Their reaction that day and 100% support for Woolwich meant there was no going back, they were the most hated for me from then on.
Another reason BTW, is the move from London to the new towns and Home Counties. This is particularly marked in Essex, where West Ham dominate, but in some places Spurs dominate, and the clashes can get very severe as New Town cockneys can be a bit 'tasty', on both sides, to say the least

.