It is shown that odd harmonic azimuthal correlations, including the directed flow v(1), in forward two-particle production in peripheral proton-nucleus (pA) collisions can arise simply from the radial nuclear profile of a large nucleus. This requires consideration of the C-odd part of the gluonic generalized transverse momentum dependent (GTMD) correlator of nucleons in the nucleus. The gluonic GTMD correlator is the Fourier transform of an off-forward hadronic matrix element containing gluonic field strength tensors that are connected by gauge links. It is parametrized in terms of various gluon GTMD distribution functions (GTMDs). We show (in a gauge invariant way) that for the relevant dipole-type gauge link structure in the small-x limit the GTMD correlator reduces to a generalized Wilson loop correlator. The Wilson loop correlator is parametrized in terms of a single function, implying that in the region of small x there is only one independent dipole-type GTMD, which can have a C-odd part. We show that the odderon Wigner distribution, which is related to this C-odd dipole GTMD by a Fourier transform, generates odd harmonics in the two-particle azimuthal correlations in peripheral pA collisions. We calculate the first odd harmonic v(1) for forward production within the color glass condensate framework in the limit of a large number of colors. We find that nonzero odd harmonics are present without breaking the rotational symmetry of the nucleus, arising just from its inhomogeneity in the radial direction. Using a CGC model with a cubic action, we illustrate that percent level v(1) can arise from this C-odd mechanism. In contrast, we show that only even harmonics arise in diffractive dijet production in ultra-peripheral pA collisions where this gluon dipole GTMD also appears.