Automatic detection and visualization of garment color in Western portrait paintings

Abstract

Paintings give us important clues about how males and females were perceived over centuries in the Western culture. In this article, we describe a system that allows scholars to automatically visualize how the clothing colors of male and female subjects changed over time. Our system analyzes a large database of paintings, locates portraits, automatically classifies each portrait's subject as either male or female, segments the clothing areas and finds their dominant color. An interactive, web-based visualization is proposed to allow further exploration of the results. To test the accuracy of our system, we manually annotate a portion of the Rijksmuseum collection, and use state-of-the-art image processing and computer vision algorithms to process the paintings. We use a deep neural network-based style transfer approach to improve gender recognition (or more correctly, sex recognition) of the sitters of portraits. The annotations and the code of the approach are made available.