Expanded polyglutamine (polyQ) stretches in at least nine unrelated proteins lead to inherited neuronal dysfunction and degeneration. The expansion size in all diseases correlates with age at onset (AO) of disease and with polyQ protein aggregation, indicating that the expanded polyQ stretch is the main driving force for the disease onset. Interestingly, there is marked interpatient variability in expansion thresholds for a given disease. Between different polyQ diseases the repeat length vs. AO also indicates the existence of modulatory effects on aggregation of the upstream and downstream amino acid sequences flanking the Q expansion. This can be either due to intrinsic modulation of aggregation by the flanking regions, or due to differential interaction with other proteins, such as the components of the cellular protein quality control network. Indeed, several lines of evidence suggest that molecular chaperones have impact on the handling of different polyQ proteins. Here, we review factors differentially influencing polyQ aggregation: the Q-stretch itself, modulatory flanking sequences, interaction partners, cleavage of polyQ-containing proteins, and post-translational modifications, with a special focus on the role of molecular chaperones. By discussing typical examples of how these factors influence aggregation, we provide more insight on the variability of AO between different diseases as well as within the same polyQ disorder, on the molecular level.