Why Are Deep Representations Good Perceptual Quality Features?

Taimoor Tariq*, Okan Tarhan Tursun, Munchurl Kim, Piotr Didyk

*Bijbehorende auteur voor dit werk

Onderzoeksoutput: Conference contributionAcademicpeer review

8 Citaten (Scopus)


Recently, intermediate feature maps of pre-trained convolutional neural networks have shown significant perceptual quality improvements, when they are used in the loss function for training new networks. It is believed that these features are better at encoding the perceptual quality and provide more efficient representations of input images compared to other perceptual metrics such as SSIM and PSNR. However, there have been no systematic studies to determine the underlying reason. Due to the lack of such an analysis, it is not possible to evaluate the performance of a particular set of features or to improve the perceptual quality even more by carefully selecting a subset of features from a pre-trained CNN. This work shows that the capabilities of pre-trained deep CNN features in optimizing the perceptual quality are correlated with their success in capturing basic human visual perception characteristics. In particular, we focus our analysis on fundamental aspects of human perception, such as the contrast sensitivity and orientation selectivity. We introduce two new formulations to measure the frequency and orientation selectivity of the features learned by convolutional layers for evaluating deep features learned by widely-used deep CNNs such as VGG-16. We demonstrate that the pre-trained CNN features which receive higher scores are better at predicting human quality judgment. Furthermore, we show the possibility of using our method to select deep features to form a new loss function, which improves the image reconstruction quality for the well-known single-image super-resolution problem.

Originele taal-2English
TitelComputer Vision – ECCV 2020 - 16th European Conference, 2020, Proceedings
RedacteurenAndrea Vedaldi, Horst Bischof, Thomas Brox, Jan-Michael Frahm
UitgeverijSpringer Science and Business Media Deutschland GmbH
Aantal pagina's17
ISBN van geprinte versie9783030585419
StatusPublished - 2020
Extern gepubliceerdJa
Evenement16th European Conference on Computer Vision, ECCV 2020 - Glasgow, United Kingdom
Duur: 23-aug.-202028-aug.-2020

Publicatie series

NaamLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12367 LNCS
ISSN van geprinte versie0302-9743
ISSN van elektronische versie1611-3349


Conference16th European Conference on Computer Vision, ECCV 2020
Land/RegioUnited Kingdom

Citeer dit