Asymmetries in heavy quark pair and dijet production in electron-proton collisions allow studies of gluon TMDs in close analogy to studies of quark TMDs in semi inclusive DIS. Here we present expressions for azimuthal asymmetries for both unpolarized and transversely polarized proton cases and consider the maximal asymmetries allowed. The latter are found to be rather sizeable, except in certain kinematic limits which are pointed out. In addition, we consider the small-x limit and expectations from a McLerranVenugopalan model for unpolarized and linearly polarized gluons and from a perturbative, large transverse momentum calculation for the T-odd gluon TMDs. Comparison to related observables at RHIC and LHC is expected to provide valuable information about the process dependence of the gluon TMDs. In particular this will offer the possibility of a sign change test of the gluon Sivers TMD and two other T-odd gluon TMDs. This provides additional motivation for studies of azimuthal asymmetries in heavy quark pair and dijet production at a future Electron-Ion Collider.