Previous research has found a genetic component of human reproduction and childlessness. Others have argued that the heritability of reproduction is counterintuitive due to a frequent misinterpretation that additive genetic variance in reproductive fitness should be close to zero. Yet it is plausible that different genetic loci operate in male and female fertility in the form of sexual dimorphism and that these genes are passed on to the next generation. This study examines the extent to which genetic factors influence childlessness and provides an empirical test of genetic sexual dimorphism. Data from the Swedish Twin Register (N=9942) is used to estimate a classical twin model, a genomic-relatedness-matrix restricted maximum likelihood (GREML) model on twins and estimates polygenic scores of age at first birth on childlessness. Results show that the variation in individual differences in childlessness is explained by genetic differences for 47% in the twin model and 59% for women and 56% for men using the GREML model. Using a polygenic score (PGS) of age at first birth (AFB), the odds of remaining childless are around 1.25 higher for individuals with 1 SD higher score on the AFB PGS, but only for women. We find that different sets of genes influence childlessness in men and in women. These findings provide insight into why people remain childless and give evidence of genetic sexual dimorphism.