They'd probably end up having to play in a different town, and that would severely hamper the potential for success.
Aren't there any municipal owned stadiums (multipurpose or what not) that the club could rent? Even if they have to share it with say an athletics club it would still be playing within Bury proper?