Human observers tend to naturally track moving stimuli. This tendency may be exploited towards an intuitive means of screening visual function as an impairment induced reduction in stimulus visibility will decrease tracking performance. Yet, to be able to detect subtle impairments, stimulus contrast is critical. If too high, the decrease in performance may remain undetected. Therefore, for this approach to become reliable and sensitive, we need a detailed understanding of how age, stimulus contrast, and the type of stimulus movement affect continuous tracking performance. To do so, we evaluated how well twenty younger and twenty older participants tracked a semi-randomly moving stimulus (Goldmann size III, 0.43 degrees of visual angle), presented at five contrast levels (5%-10%-20%-40%-80%). The stimulus could move smoothly only (smooth pursuit mode) or in alternation with displacements (saccadic pursuit mode). Additionally, we assessed static foveal and peripheral contrast thresholds. For all participants, tracking performance improved with increasing contrast in both pursuit modes. To reach threshold performance levels, older participants required about twice as much contrast (20% vs. 10% and 40% vs. 20% in smooth and saccadic modes respectively). Saccadic pursuit detection thresholds correlated significantly with static peripheral contrast thresholds (rho = 0.64). Smooth pursuit detection thresholds were uncorrelated with static foveal contrast thresholds (rho = 0.29). We conclude that continuous visual stimulus tracking is strongly affected by stimulus contrast, pursuit mode, and age. This provides essential insights that can be applied towards new and intuitive approaches of screening visual function.