Melville's The Confidence-Man: who is the con man in chapter 3?

I've returned to Melville- I'm reading The Confidence-Man: His Masquerade. What can be a better read now, whilst one follows the elections in the US? 
People who know the book all know that it's about a con man or more in different disguises. What bothers me is, who is the con man in chapter 3? Everywhere I look, people say it's the crippled black man, because he wins people's pity and thus gets money, but is he? What I see in the chapter is that out of nowhere "a limping, gimlet-eyed, sour-faced person", a custom-house officer, appears and loudly says the black man's deformity is a sham, "got up for financial purposes". How does he know? What are those allegations based on? Does he have evidence or anything to back up those claims?After saying it's a sham, the custom-house officer just says "He can walk fast enough when he tries, a good deal faster than I; but he can lie yet faster. He's some white operator, betwisted and painted up for a decoy. He and his friends are all humbugs." but doesn't bother to prove any of his words. He is like a Donald Trump, loudly and confidently throwing out claims and accusations based on nothing, showing no regard for facts. Then he leaves, but before that, has successfully sowed a seed of doubt in everyone's minds. 
Doesn't that make him, rather than the black cripple, a con artist?