Where was the wall?

In Robert Frost's poem, the wall lies between an apple orchard owned by one man and a pine grove owned by another. Every spring the two men cooperate to mend the wall, which has been damaged in a kind of cyclical process by both nature and men. The one man repeats his observation "Something there is that doesn't love a wall" while his neighbor responds with the aphorism that "good fences make good neighbors."
One might consider a less literal answer to your question about the location of the wall. I was tempted, in this context, to type the word as "wall," in quotation marks. A wall means separation, not merely between two plots of land but between people, cultures, ideas, religions, and many other things. Perhaps the key to the central metaphor lies in the lines:

I see him there, Bringing a stone grasped firmly from the topIn each hand, like an old-stone savage armed.He moves in darkness as it seems to me,Not of woods only and the shade of trees.

Here one might ask, what is implied by the image of the man, who thinks walls are good, "mov[ing] in darkness"—and not just a natural absence of light made by the "shade of trees"? Is the darkness one of ideas, symbolic of the divisions between people anywhere, in any time?