Why do town historians believe the Headless Horseman is always hanging out around the church?

The Headless Horseman is believed to be the ghost of a dead Hessian (German) soldier whose head was blown off by a cannon ball during a battle of the Revolutionary War. According to the town "historians," the soldier's body is buried at the village church. The spirit or ghost of the man wanders around at night haunting roads and valleys searching for his missing head near the scene of the battle. However, the Headless Horseman tends to stick close to the church because he knows he can only travel around in the darkness and must get back to the churchyard by morning. It is easier for him if he doesn't go too far. He is also said to rush like the wind to get back to the churchyard in time. We are told that:

the ghost rides forth to the scene of battle in nightly quest of his head; and that the rushing speed with which he sometimes passes along the Hollow, like a midnight blast, is owing to his being belated, and in a hurry to get back to the churchyard before daybreak.