Before the nineteenth century, most portraits were, almost by definition, depictions of people who were important in their own worlds. But, as a walk through almost any major art museum will show, a large number of these portraits from before the nineteenth century have lost the identities of their subjects through the fortunes of time. Traditionally, identification of many of these portraits has been limited to often quite variable personal opinion. FACES (Faces, Art, and Computerized Evaluation Systems) proposes to establish the initial parameters of the application of face recognition technology to works of portrait art–this highly subjective aspect of art history–while at the same time retaining the human eye as the final arbiter.
During this grant period, FACES began work establishing these parameters, asking such questions as: is face recognition technology, originally designed for actual (that is, photorealistic) human faces, applicable to works of portrait art, which are subject to a process of visual interpretation on the part of the artist? Which of the many different face recognition techniques should be used? Which functions of the many functions of a given technique would apply most effectively to our subjects? What culture and period would work best in this initial stage of testing? What types of portraits would best be used, sculpture (three-dimensional) or painting and drawing (two-dimensional)–or both? How will the identifying characteristics in a portrait of one sitter by an artist with a distinctive artistic style compare to a portrait of the same sitter but by a different artist who also has a distinctive style? If face recognition technology works with sculpture, will the identical process be able to be used for painting and drawing? If face recognition technology operates best with a straight-on view of the subject, how will the rate of successful tests be with three-quarter view portraits, the standard pose for portraits in early modern Western culture? For two-dimensional works, will the medium–oil painting, tempera, pencil, chalk, engraving, and so on–affect the test results differently? What about copies of portraits (for example, of famous sitters, like Isaac Newton) and copies of copies–how closely will they retain the identifying characteristics found in the original portrait? What about extreme or poor lighting in painting and drawing? What about aging as found in multiple portraits of the same sitter made over a long period of time? By the same artist? By different artists? What about the vary artistic ability of the individual artist?
In the course of initial investigation, it gradually became clear that of all the different methods of face recognition technology, two gave the most dependable results: the computation of anthropometric distances and of local features. These two methods were part of a larger, more complex process we call the FACES algorithm (detailed below).
While the FACES algorithm was constantly developed throughout the course of this two year project, we began by testing the death mask of a known individual against (Fig. 1) an identified sculptural portrait of the same individual (Fig. 2). That is, we tested an analogue–an unmediated image of the subject, not a work of art–against the image of a three-dimensional work of art that, in this case, physically approaches the subject in form and size but that nevertheless partakes of the subjectivity of artistic interpretation.
We then left the relative security of the analogue and work-of-art pairing, and tested paradigms of exclusively three-dimensional works of art–that is, we then tested two works both of which were now subject to the subjectivity of artistic interpretation. (We use the term paradigm here to mean a logically chosen body of related images directed toward a particular demonstrative end.) More specifically, we tested a sculptural portrait of a known individual with another sculptural portrait of the same individual, both around the same stage of the individual’s life and both depicted by the same artist–in other words, we proceeded with as much control over variables as possible (Figs. 3 and 4).
Incrementally, we broadened our tests–too involved to fully detail here–introducing a similarly controlled but wide-ranging series of systematically chosen variations extending from more controlled paradigms to less controlled (that is, more challenging) ones. These included the same stage of an individual’s life but by different artists, different stages of an individual’s life by the same artist, and different stages of an individual’s life by different artists–all in three-dimensional imagery.
Then we tested two-dimensional imagery, first simply comparing two two-dimensional images of the same subject by the same artist (for example, Figs. 5 and 6), and then mixing media by testing a number of sculpture vs painting (that is, three-dimensional vs two-dimensional) paradigms, employing a systematic series of distinctions similar to those already mentioned (different ages, different artists, and so on) (for example, Figs. 4 and 7). Finally, we tested a few identified portraits of individuals against unidentified ones.
In the first year of FACES, we established proof of concept. Practically speaking, this means that we identified the issues, established the basic methodology (even if not fully worked out yet), and applied this methodology to a particular set of paradigms.
In the second year, we developed the optimum feature set (the most effective body of identifying facial features, given the unique demands of portrait art), expanded the gallery of images with which establish non-match averages (that is, a standard with which to compare a given image under investigation), and continued to work on the problems of angle views and aging. But our work increasingly focused on the questions of the degree of influence on a portrait of the style of the individual artist. For example, a given artist generally tends to render the same detail in the same way, even in an individualized portrait. And so individual artistic style was investigated through a close and systematic study of a large number of portraits of different sitters by the same artist in order to model–that is, to teach the computer–the individual style of the artist.
We also rose to a new level of testing in the application of our newly worked out methods to a series of interesting and sometimes important “identifications.” By “identification” I sometimes mean the actual identity of the subject (for example, Mary Queen of Scots) and sometimes merely the ordering of the material into “identities”: group X, group Y; Lord X, Lady Y; and so on. This is not the place to go into any detail about these identification attempts, except to say that all of them were important and some of them exciting. Although all paradigms were not conclusive–sometimes for very complex reasons–some of the best known works include: what appears to be the earliest known likeness of Galileo Galilei, painted perhaps around 1590 (Fig. 8); Nicholas Hilliard’s Young Man Among Roses (Fig. 9; c. 1588), said to be perhaps the most famous miniature ever painted;” the tangled web of portraits that different proponents have claimed at one time or another portray William Shakespeare; a portrait said to represent Mary Queen of Scots (Fig. 10; National Portrait Gallery, London, NPG 96); and a portrait that is thought by some to be of James Scott, duke of Monmouth, first duke of Buccleuch, and illegitimate son of Charles II, lying in bed with the covers pulled up to his chin, apparently in order to conceal the fact that James’s head has been cut off and–at least in the painting–put back on again (Fig. 11; National Portrait Gallery, London, NPG 1566).