In Chapter 15, why do the men finally leave Atticus and Tom alone?

In Chapter 15, Tom Robinson has been moved to the town jail.  He is not being detained there; rather, he is placed there for his own protection.  Atticus decides to further shield Tom from potential mob violence by sitting on the door step of the jail, blocking the entrance.

This turns out to be a smart move, as a mob does, indeed, converge on the jail.  What Atticus doesn't count on, however, is that Jem, Scout, and Dill -- all of whom have followed him to the jail -- also appear on the scene.

The men in the mob have their hats pulled down and their distinguishing features covered.  Even so, Scout spots and recognizes Walter Cunningham, the father of her classmate and a former client of her father's.  When she strikes up a conversation about Walter's son, Walter feels ashamed that he has been confronting Atticus as part of the mob, and he convinces the mob to leave.