Consider the “signal-noise” aspect.

The human brain is structured to facilitate detection and recognition of patterns, and to distinguish patterns of difference from random differences.

If even a few behavioral tendencies are observed to be associated with one sex but not the other, then people will detect those behavioral tendencies, even if they are displayed by a minority of each sex. The distinct behavioral tendencies of each sex will be detected as a pattern, even if the “noise” of random individual differences is “louder” than the “signal” produced by the pattern.