What sin do the townspeople commit in response to the minister's veil?

In response to Mr. Hooper's black veil, the townspeople begin to judge him almost immediately.  His fiancee, Elizabeth, tells him that most people do not believe that Mr. Hooper wears the veil to signify some "innocent sorrow," that they whisper that "[he] hides [his] face under the consciousness of secret sin."  Despite the religious injunction that we not judge one another, his congregation grows suspicious of Mr. Hooper and judges him rather harshly. In the book of Matthew, Christians are told, "Do not judge lest you be judged.  For in the way you judge, you will be judged; and by your standard of measure, it will be measured to you" (Matthew 7.1-2). Only God is supposed to be our judge, and yet Mr. Hooper “spent a long life, irreproachable in outward act, yet shrouded in dismal suspicions; kind and loving, though unloved, and dimly feared; a man apart from men, shunned in their health and joy […].”  He became an outcast as a result of his parishioners' suspicions and judgment.  Ultimately, they judged him for his secret sins, continuing to insist on their own sinlessness (exactly the lie that Mr. Hooper wished to draw attention to by wearing the veil in the first place).