The data set consists of 1237 pairs of thermal and RGB (640 x 320 pixels and 320 x 240 pixels) images with 136 classes (i.e. 136 different cows) with a mean of 9 images per class/cow. Each folder name is the collar id of the cattle and contains its respective thermal and RGB images. The data set was collected at the Dairy Campus in Leeuwarden, The Netherlands. In order to explore the temperature values of the thermal images, FLIR tools can be used by installing the software (https://www.flir.com/products/flir-tools/). Due to the large number of files the data set is provided in 5 zip files. The first 4 zip files contain the data of 30 cows each and the 5th zip file contains the data of 16 cows.