Casual fans are fans, but they are less important fans, so it's not retarded. Someone who shrugs and says ManU is their favorite team is not on equal footing with someone who watches broadcasts and streams of every match (and probably some other teams' matches), buys kits, scarves, refrigerator magnets, and even the occasional plane and match ticket. Of the latter more important category, there is a healthy ratio of Spurs supporters. Fans are not fans.
Bringing this again back to to the original point, the idea that Americans are generally unhappy about Yedlin because Spurs are disliked in America is baseless.
Americans are suspicious that he won't get playing time or he will get loaned to a championship team and then he won't develop as a better player. Not sure if Roma is all that much better, but there had been a plan to loan him to Genoa from there which many American fans see as a better long range plan.
Of interest, there is a pocket of Tottenham Hotspur fans in Texas related to the success of the NBA team San Antonio Spurs. Apparently you can buy Tottenham jerseys and other gear inside the Alamodome.