The brain is shaped by experience, and experience is delivered through a body. There are characteristics of male and female bodies that are distinct from each other. Male brains are shaped by experiences that are delivered through a male body and female brains are shaped by experiences that are delivered through a female body. In that respect there is indeed a distinct male brain and a distinct female brain.
A man perceives and experiences a woman differently than he perceives another man. He reacts and interacts with her differently than with a man. The same is true for a woman. Characteristically distinct interactions produce characteristically different experiences. Different experiences produce differently shaped brains. Different experiences with the opposite sex is another form of experience that shapes male and female brains differently.
Male brains are different from female brains. That fact is not invalidated because some may have used it to justify unfairness.
You are proposing what amounts to a tabula rasa or blank-slate theory.
That theory is fundamentally flawed. If all of a person's traits are learned from others, then from where did the others learn them? Tabula-rasa is infinitely regressive, it fails to account for the traits' origins.
