Found something interesting, if I place new building of the type I have invisible ones on the map, select the build designations and go into statistics of it then choose "delete all" it actually deletes the invisible ones too.
And once all the bugged ones are gone your bots start moving again.
Seems like the normal delete/sell options aren't properly removing the building.
also this
the build option lets me choose all the building that the bot that usually builds the building can build