What does the phrase "a host of phantom listeners" imply in "The Listeners"?

That line is probably both my favorite and least favorite line from this poem  I love the line because it provides readers with so many possibilities of what exactly it means. I don't like the line because deep down I just want to be told an author's specific meaning.  

The dictionary lists a few different definitions for the word "phantom." One is that a phantom is a ghost. It's an actual spirit of a dead person. The other definition is that a phantom is a figment of the imagination.  

In relation to this poem, that means the phantom listeners are likely the ghostly spirits of the house's former inhabitants, but the phantoms could be merely figments of the visitor's imagination. With no explanation as to why the man is there or why the house is empty, readers are left to ponder the possibility of both options. Thinking that the phantom listeners are ghosts definitely makes the poem creepier, while thinking they are merely imaginary projections makes the poem more realistic.

My other frustration with the line comes from the word "host." It could refer to the fact that the listeners are actual hosts. Somebody that has a party at their house is called the host. "A host of phantom listeners" might mean ghosts are hosting the man as a visitor. On the other hand, "a host" might reference a number. "A host" of something means a large number. The line from the poem could mean there are a lot of ghosts in the house. Overall, the line is open to a lot of interpretation. That's why it's easy to get a lot of discussion from my students about this poem.