In breast cancer the transcription factor SOX4 has been shown to be associated with poor survival, increased tumor size and metastasis formation. This has mostly been attributed to the ability of SOX4 to regulate Epithelial-to-Mesenchymal-Transition (EMT). However, SOX4 regulates target gene transcription in a context-dependent manner that is determined by the cellular and epigenetic state. In this study we have investigated the loss of SOX4 in mammary tumor development utilizing organoids derived from a PyMT genetic mouse model of breast cancer. Using CRISPR/Cas9 to abrogate SOX4 expression, we found that SOX4 is required for inhibiting differentiation by regulating a subset of genes that are highly activated in fetal mammary stem cells (fMaSC). In this way, SOX4 re-activates an oncogenic transcriptional program that is regulated in many progenitor cell-types during embryonic development. SOX4-knockout organoids are characterized by the presence of more differentiated cells that exhibit luminal or basal gene expression patterns, but lower expression of cell cycle genes. In agreement, primary tumor growth and metastatic outgrowth in the lungs are impaired in SOX4KO tumors. Finally, SOX4KO tumors show a severe loss in competitive capacity to grow out compared to SOX4-proficient cells in primary tumors. Our study identifies a novel role for SOX4 in maintaining mammary tumors in an undifferentiated and proliferative state. Therapeutic manipulation of SOX4 function could provide a novel strategy for cancer differentiation therapy, which would promote differentiation and inhibit cycling of tumor cells.