In "To Kill a Mockingbird", why does the Idlers' Club outside the courthouse object to Atticus's defending Tom Robinson?

Page 174..."thinks he knows what he's doing,"one said.  Oh-h now, I wouldn't say that," another said.  "Atticus Finch's a deep reader, a mighty deep reader."  "He reads all right, that's all he does." The club snickered.  "Lemme tell you something not Billy," a third said, "you know the court appointed him to defend this nigger."  "Yeah, but Atticus aims to defend him.  That's what I don't like about it."

The Idler's club is a group of men who are caught up in the race hatred of the era.  They will give Atticus the benefit of the doubt because he has been appointed to defend Tom, but they don't want him to do his job.  They just want him to go in and throw the case.  They don't like the idea of a black man getting a defense.  In their minds he is guilty and should not even get a trial.