Autism spectrum disorder (ASD) and schizophrenia have been conceived as partly opposing disorders in terms of systemizing vs. empathizing cognitive styles, with resemblances to male vs. female average sex differences. Left–right asymmetry of the brain is an important aspect of its organization that shows average differences between the sexes and can be altered in both ASD and schizophrenia. Here we mapped multivariate associations of polygenic risk scores for ASD and schizophrenia with asymmetries of regional cerebral cortical surface area, thickness, and subcortical volume measures in 32,256 participants from the UK Biobank. Polygenic risks for the two disorders were positively correlated ( r = 0.08, p = 7.13 × 10 −50) and both were higher in females compared to males, consistent with biased participation against higher-risk males. Each polygenic risk score was associated with multivariate brain asymmetry after adjusting for sex, ASD r = 0.03, p = 2.17 × 10 −9, and schizophrenia r = 0.04, p = 2.61 × 10 −11, but the multivariate patterns were mostly distinct for the two polygenic risks and neither resembled average sex differences. Annotation based on meta-analyzed functional imaging data showed that both polygenic risks were associated with asymmetries of regions important for language and executive functions, consistent with behavioral associations that arose in phenome-wide association analysis. Overall, the results indicate that distinct patterns of subtly altered brain asymmetry may be functionally relevant manifestations of polygenic risks for ASD and schizophrenia, but do not support brain masculinization or feminization in their etiologies.