Multi-Instance Dynamic Ordinal Random Fields for Weakly-supervised Facial Behavior Analysis

TitleMulti-Instance Dynamic Ordinal Random Fields for Weakly-supervised Facial Behavior Analysis
Publication TypeJournal Article
Year of PublicationSubmitted
AuthorsRuiz, A
Secondary AuthorsOgnjen, R, binefa, X, Pantic, M
JournalIEEE Transactions on Image Processing
Mutliple-Instance-Learning (MIL) has become a popular modelling framework for addressing weakly-supervised problems in Computer Vsion. In MIL, a training set is formed by bags (sets of feature vectors or instances) and only labels at bag-level are provided. In this work, we address this problem when instance labels are naturally represented as ordinal variables and bags are structured as temporal sequences: Multi-Instance Dynamic-Ordinal-Regression (MI-DOR). In our approach, we assume two scenarios: (i)  bag label is equal to the maximum label of its constituent instances, (ii)  bag-label provides information about the evolution of the instance ordinal labels within the sequence. To address these modeling problems, we propose Multi-Instance Dynamic Ordinal Random Fields (MI-DORF). In this framework, we treat instance-labels as temporally-dependent latent variables in a graphical model. The different MIL assumptions are modelled by incorporating a high-order potential relating bag and instance-labels into the energy function. Moreover, we extend the proposed framework for Partially-Observed MI-DOR problems, where a subset of instance labels are also available during training. We show the benefits of the proposed framework for weakly-supervised facial behavior analysis. Specifically, we evaluate the model on the tasks of Facial Action Unit (DISFA dataset) and Pain Intensity  (UNBC dataset) estimation. We show that the proposed framework significantly outperforms alternative approaches on these tasks.