What is a summary of the poem "The Listeners"?

"The Listeners" (De la Mare) is a poem in which there is not a great deal of tangible action, but there is a great deal of mysterious atmosphere that allows us to speculate and imagine a story behind it.
Simply put, a man traveling by horse on a moonlit night approaches a house in a forest, knocks several times on the door, gets no answer, shouts at the house, and then departs. If I broke that up into a few lines and recited it to you, your response would most likely be "So what?" A summary takes all the joy out of a poem, but there are reasons this poem has held up for over a hundred years.
There are "listeners" in this old, seemingly deserted house with its turret and ivy growing up the sills.  These are ghosts of some sort, which we know because of their "strangeness" (De la Mare line 21) and their characterization as "phantom" (line 13).  They listen only and say nothing, unable to reply to the horseman.
Finally, after the horseman pounds on the door a final time,


‘Tell them I came, and no one answered,   
   That I kept my word,’ he said (lines 27-28).

 
The horseman turns then, gets on his horse, and gallops away, leaving the house to the listeners, the forest, and the silence. 
 
There is a story here, a story about the horseman's promise to someone or someones in this house.  He may have waited too long to keep a promise to a woman he loved.  Perhaps he had pledged to protect the people in the house, he took too long, and they all died.  Maybe he was a prodigal son who waited too long to go back home to his family. We can imagine what might have delayed him, a war, perhaps, or an imprisonment. He may have delayed himself with drink, gambling, or women.
 
The poem sets for us a mysterious little scene, with wonderful sensory details about the forest, the man, the house, and even the horse. We can see the scene. And then we can pour meaning into it.  You might want to try to write the "prequel" to this lovely little poem.