The Epistemic Reverse Side of Instrumental Images

Research topic

The object of investigation is instrumental images (images used by human actors) with reference to knowledge that arises in connection with their development and that is unavailable to the actors, such as open research questions and unproven concepts. Just as the respective form/technology of the image affects the relationships of understanding, representation and operative function of the instrumental images, so too – depending on image – does the type and manner of (non-) communication between research and application. The distance of actors from the epistemic and symbolic preconditions of the instrumental images increases with digitalisation. Images serving as examples are those used in medicine and psychology which reflect (1) the relation between facial morphology and signalling (expression) and (2) the relation between anatomy and function (the hand). The interdisciplinary team pursues the aims of (1) examining the role of instrumental images in activities in the transition from drawing/diagram to digital image in two different cases and (2) linking epistemic knowledge with instrumental images, working with (a) a data bank at the Zuse Institute Berlin (ZIB) of Facial Action Coding System (FACS) (archive of expressions) and (b) the development of a platform (an expression atlas) for providing users in research, teaching and practice with access to history of science and image questions/problems that are relevant to current research on the facial expression.

Subprojects

I. Instrumental Images in Hand Surgery: From Hand-Drawn to Digital Images

The hand, Aristotle writes, is »the instrument of all instruments « (instrumentum instrumentorum). More recently, this body part has become central to theoretical debates thanks to smartphones, tablets, and other new technology (Ruf). While case studies of the hand exist in the fields of medical, cultural, and art history (Leroi-Gourhan; Gadebusch Bondio), a study of the hand from an image studies perspective has not yet been undertaken. This project takes the hand as its object of study in order to fill this important gap in research.

The project examines the hand as a dialectical object between form and function and addresses questions concerning the hand as a figure in drawings and instrumental imaging in the context of hand surgery and its emergence as a unique discipline. The study will also look at the field’s epistemological prehistory together with its role in the history of images. Other important areas to be touched upon are technological developments in medicine and how they relate to advancements in arms technology as well as the geometrization and disciplining of the body with regard to both Early Modern theoretical debates on art and architecture and current issues concerning changes in media. In addition, the project will analyze the practice of drawing as an epistemic tool and its embeddedness in pictorial practices used in scientific research and other realms of collecting, contrasting, and organizing information. The study also looks at exchanges between different media and drawing, for example, photography (slides, x-rays), print media such as newspaper pictures, and reproductions of artworks. Informed by the theory of drawing (Kemp) and the history of hand-related knowledge (Sherman), the project utilizes the posthumous archive of the hand surgeon J. William Littler as the basis for investigations into the material culture and pictorial practices of hand surgery. This archive contains images of the hand used for research and in patient consultations. Key questions ask about the concept of »style« in surgery, the interrelationship between photography and hand-drawn pictures, as well as how changes in media technology play a role in the use of images in hand surgery.

II. Dissertation Project: Coding Feelings. The Scientific History of the FACS (working title)

The dissertation project deals with the scientific history of the Facial-Action-Coding-System (FACS) and its continued development as a digital resource in the field of affective computing. The FACS was developed in the 1970s as an objective tool to code emotions based on facial expressions. To this day, the code system is still the paradigm used to explain and understand the expression of affects in psychology, experimental psychology, behavioral psychology, neuroscience, etc. The project suggests an historical epistemological framework to examine the conditions that contributed to establishing the FACS as the fundamental tool in the study of emotions. These conditions relate to epistemological, discursive, and technological questions as well as to issues in the history of images, including experimental and photographic practices and video making. This study of FACS’s development looks at, on the one hand, how computer-based methods of diagnosing psychopathologies have used patterns of facial expressions over the years. On the other hand, the study concentrates on digital learning tools and more approachable computer technology that have been designed to help people learn to recognize and read affective patterns of expression in real time, as in the context of treating autism spectrum disorders. The overarching goal is to investigate the implicit and explicit development of FACS and the lingering effects of historical knowledge within the realm of affect studies. The project also covers an examination of the epistemological and discursive ruptures in the history of psychopathology, its methods of diagnosis, and its role in in the history of images. Historical practices of reading and understanding the ›soul/psyche‹ through the affective surface of the body also inform the study.

III. Dissertation Project: Building a Database of Facial Expressions in 3D

Models of human facial morphology and its dynamics provide the foundations for developing computer-based tools to plan and analyze facial surgery. They are also crucial in the neurocognitive study of how facial expressions are perceived and processed. Medical and neuroscientific applications have long used models of facial expression such as the Facial-Action-Coding-System. However, these models pose significant limitations: in addition to promoting a categorical coding system and a terminological and methodological conflation of emotion with induced facial expressions, they are also restricted by their primarily qualitative description of morphological patterns of expression (e.g., the FACS uses photographs of faces as examples). Furthermore they reduce facial dynamics to mere muscular movement. New applications and questions, however, call for a quantitative model of dynamic facial forms that is better able to take into account subtle differences.

The dissertation project therefore focuses on establishing a highly detailed 3D face and expression database (the ”Mimik Archiv”) intended to support the statistical analysis of the quantitative characteristics of facial morphology. The resulting morphological patterns make it possible to generate a combined model of facial expressions, called the »Mimik-Atlas«. It is intended to serve as a tool for medical and neuroscientific applications.

The first phase of the project aims to develop and improve existing stereo-photogrammetric methods for the acquisition of 3D facial morphology. The Camera Facialis project at the Konrad-Zuse-Center for Information Technology (ZIB) created a multi-camera setup that makes it possible to precisely measure the 3D geometry and colors of facial surfaces within a fraction of a second. The level of detail contained in the digital models, both in terms of geometric and photometric data, surpasses the capacities of currently available systems. Over the course of the project, the Camera Facialis will be used to collect a wide database of facial expressions.

Further research will be concerned with the development of new methods for the automatic processing of newly captured facial expression data. The goal is to enable the rapid integration of digital 3D models into the archive as well as to facilitate retrieval and analysis of the stored morphological data.