Phage-displayed immunoprecipitation sequencing (PhIP-seq) has enabled high-throughput profiling of human antibody repertoires. However, a comprehensive overview of environmental and genetic determinants shaping human adaptive immunity is lacking. In this study, we investigated the effects of genetic, environmental, and intrinsic factors on the variation in human antibody repertoires. We characterized serological antibody repertoires against 344,000 peptides using PhIP-seq libraries from a wide range of microbial and environmental antigens in 1,443 participants from a population cohort. We detected individual-specificity, temporal consistency, and co-housing similarities in antibody repertoires. Genetic analyses showed the involvement of the HLA, IGHV, and FUT2 gene regions in antibody-bound peptide reactivity. Furthermore, we uncovered associations between phenotypic factors (including age, cell counts, sex, smoking behavior, and allergies, among others) and particular antibody-bound peptides. Our results indicate that human antibody epitope repertoires are shaped by both genetics and environmental exposures and highlight specific signatures of distinct phenotypes and genotypes.