We study the leading-power gluon transverse-momentum-dependent distributions (TMDs) of relevance to the study of asymmetries in the scattering off transversely polarized hadrons. Next-to-leading-order perturbative calculations of these TMDs show that at large transverse momentum they have common dynamical origins but that in the limit of a small longitudinal momentum fraction x, only one origin remains. We find that in this limit, only the dipole-type gluon TMDs survive and become identical to each other. At small x, they are all given by the expectation value of a single Wilson loop inside the transversely polarized hadron, the so-called spin-dependent odderon. This universal origin of transverse spin asymmetries at small x is of importance to current and future experimental studies, paving the way to a better understanding of the role of gluons in the three-dimensional structure of spin-polarized protons.