Sorry, I see that I was not clear.
I did not mean to say that slaves, indentured servants, minorities, and immigrants were mostly men. What I should have said is that all of those categories include men. Work done by men who are slaves, indentured servants, minorities, and immigrants is still work done by men.
Men did most of the heavier, muscle work. Women contributed to the effort in their own way. Everyone contributed.