Their condemnation came as Mdladlana shrugged off suggestions that his statements were racist.
The association was reacting to the minister's claim that the Chinese might regret having gone to court to be declared black for the purposes of economic empowerment legislation.
He also said Chinese employers now had no excuse to mistreat their workers or pretend to labour inspectors that they could not speak English.
In a statement issued through its attorneys, the association said it was "surprised" by the minister's remarks.
His comments conflicted with the constitution and the legislation he was supposed to administer.
"His unfounded generalisations about the South African Chinese community are factually inaccurate," it said.
"The misconduct of any individual cannot rationally be attributed to a community as a whole."
Had the minister read the high court papers, which detailed the extent to which the South African Chinese community suffered for a period of over 100 years, he would not have made the "incorrect and irrational" statements attributed to him.
"We, in the South African Chinese community, ask the people of South Africa as a whole to respect our dignity and the judicial process to which we were compelled and entitled to appeal for clarity regarding our status."
Earlier on Wednesday, Mdladlana dismissed accusations that his comments were racist.
Responding to members' statements in the National Assembly, Mdladlana said he would not have joined the African National Congress if he had been a racist.
"The ANC has never been a racist organisation and I will never be a racist - that's why I joined the ANC," he said.
Democratic Alliance labour spokesperson Anchen Dreyer called on him to retract his statements.
"It is grossly inappropriate for a national minister to make the kinds of claims that entrench the idea that South Africa is a nation of separate and hostile identities," she said in a statement.
"Mdladlana should retract his divisive comments."
She said his comments suggested that the ANC, despite its claim to non-racialism, saw South Africans in fundamentally racist terms.
- Sapa