Diet quality among short- and long-term gastrointestinal (GI) cancer survivors with different tumor sites was investigated compared to a reference population cohort. Diet quality of GI cancer survivors (n = 307) was compared to an age- and sex-matched reference population with no history of cancer (n = 3070). All were selected from Lifelines, a population-based cohort. GI cancers were defined as having a history of cancer of the bowel, esophagus, or stomach. Diet quality was assessed by a self-administrated food frequency questionnaire in terms of: (i) Lifelines Diet (LLD) scores, where higher scores indicate higher diet quality; (ii) the adherence to dietary guidelines, quantified by the percentage of meeting dietary recommendations, as given by Dutch dietary guidelines; and (iii) the mean daily intake of food components. All analyses were adjusted for lifestyle factors. Diet scores in GI cancer survivors were not different from the reference population (OR = 0.97, 95% CI: 0.73-1.23). Stratification for time since diagnosis and tumor site gave similar results. The intake of vegetables, unsweetened dairies, and nuts and legumes was almost 50% lower than the recommended amount, and the mean intake of unhealthy food components was at least one serving/day among GI cancer survivors, as well as in the reference population. In the long run, GI cancer survivors do not differ from the reference population in their diet quality. In conclusion, both groups can improve their diet quality.