Loss of Y-chromosome (LOY) is associated with increased cancer mortality in males. The prevalence of LOY in male breast cancer (BC) is unknown. The aim of this study is to assess the presence and prognostic effect of LOY during male BC progression. We included male BC patients diagnosed between 1989 and 2009 (n = 796). A tissue microarray (TMA) was constructed to perform immunohistochemistry and fluorescent in situ hybridization (FISH), using an X and Y probe. We also performed this FISH on a selected number of patients using whole tissue slides to study LOY during progression from ductal carcinoma in situ (DCIS) to invasive BC. In total, LOY was present in 12.7% (n = 92) of cases, whereby LOY was associated with ER and PR negative tumors (p = 0.017 and p = 0.01). LOY was not associated with the outcome. Using whole slides including invasive BC and adjacent DCIS (n = 22), we detected a concordant LOY status between both components in 17 patients. In conclusion, LOY is an early event in male breast carcinogenesis, which generally starts at the DCIS stage and is associated with ER and PR negative tumors.