Villagers living next to Belton House say people have been knocking on doors asking for directions.