Instance-level recognition for artworks

Abstract

In this thesis, the creation of a new dataset and benchmark for large-scale instance-level recognition (ILR) in the domain of artworks is addressed. The proposed dataset, called the Met dataset, exhibits a number of different challenges such as large inter-class similarity, long-tail distribution, and many classes. It relies on the open access collection of The Met museum to form a large training set of about 224k classes, where each class corresponds to a museum exhibit with photos taken under studio conditions. The evaluation set is primarily composed of photos taken by museum guests depicting exhibits, which introduces a distribution shift between training and testing. It is additionally composed of a set of images not related to Met exhibits making the task resemble an out-of-distribution detection problem. The proposed benchmark follows the paradigm of other recent datasets for ILR on different domains to encourage research on domain independent approaches. In order to offer a testbed for future comparisons, a number of suitable approaches are evaluated. Self-supervised and supervised contrastive learning are effectively combined to train CNNs that produce the image representation used in combination with a non-parametric classifier, showing a promising direction for ILR. The dataset webpage (also contains reference code) is: http://cmp.felk.cvut.cz/met/.