The development of memristive device technologies has reached a level of maturity to enable the design and fabrication of complex and large-scale hybrid memristive-Complementary Metal-Oxide Semiconductor (CMOS) neural processing systems. These systems offer promising solutions for implementing novel in-memory computing architectures for machine learning and data analysis problems. We argue that they are also ideal building blocks for integration in neuromorphic electronic circuits suitable for ultra-low power brain-inspired sensory processing systems, therefore leading to innovative solutions for always-on edge-computing and Internet-of-Things applications. Here, we present a recipe for creating such systems based on design strategies and computing principles inspired by those used in mammalian brains. We enumerate the specifications and properties of memristive devices required to support always-on learning in neuromorphic computing systems and to minimize their power consumption. Finally, we discuss in what cases such neuromorphic systems can complement conventional processing ones and highlight the importance of exploiting the physics of both the memristive devices and the CMOS circuits interfaced to them.