@Doubting Thomas: Among most of the 1850s slavers, anyway. Then you had people like George Fitzhugh, who believed that if there was a problem with slavery, it was that it was being improperly constrained TO one race--even for whites, most of them would be far better off as chattel slaves than as necessarily oxymoronic free workers. As he put it:
We promised to write no more in this chapter; but, like Parthos, when "we have an idea," we want to give others the benefit of it. We agree with Mr. Jefferson, that all men have natural and inalienable rights. To violate or disregard such rights, is to oppose the designs and plans of Providence, and cannot "come to good." The order and subordination observable in the physical, animal and human world, show that some are formed for higher, others for lower stations--the few to command, the many to obey. We conclude that about nineteen out of twenty individuals have "a natural and inalienable right" to be taken care of and protected; to have guardians, trustees, husbands, or masters; in other words, they have a natural and inalienable right to be slaves. The one in twenty are as clearly born or educated, or some way fitted for command and liberty. Not to make them rulers or masters, is as great a violation of natural right, as not to make slaves of the mass. A very little individuality is useful and necessary to society,--much of it begets discord, chaos and anarchy.
(Just to be clear, "husbands" there is probably as in "animal husbandry", considering that Fitzhugh actually saw it right to extend to women the right to vote...well, among the women of the 5% he deemed as proper rulers, I imagine.)
At least according to Wikipedia (and we all know how volatile its articles are, yes?), he disagreed with Jefferson on everything else. The article seems to deem Fitzhugh as a non-liberal, non-Marxist socialist.