Time is an integral part of all adaptive behavior; we continuously adapt to the dynamic structure of an everchanging environment. Recent theoretical approaches have moved from the idea that time arises from specialized stopwatch-like mechanisms, instead proposing the view that time is inherently encoded in a host of neural dynamics. However, we argue that much of our theorizing is—even when an intrinsic view is proposed—still driven by the implicit assumption that clearly marked, isolated stopwatch-like intervals are the fundamental unit of time in our environment. This assumption ignores the challenges of interacting with an uncertain, ever-changing environment: (a) Relevant intervals need to be distilled from a continuous stream of actions and events, and (b) time is never estimated for its own sake but instead used to adaptively tune cognition. We discuss an “intrinsic-adaptive” view that, in contrast to studying isolated stopwatch intervals, considers how organisms learn and adapt behavior to temporal structures from experience in natural worlds.