Varied sources of error contribute to the challenge of facial action unit detection. Previous approaches address specific and known sources. To address the ubiquity of error, we propose a Confident Preserving Machine (CPM) that follows an easy-tohard classification strategy. During training, CPM learns two confident classifiers. A confident positive classifier separates easily identified positive samples from all else; a confident negative classifier does same for negative samples. During testing, CPM then learns a person-specific classifier using “virtual labels” provided by confident classifiers. To evaluate CPM, we compared it with a baseline single-margin classifier and stateof-the-art semi-supervised learning, transfer learning, and boosting methods in three datasets of spontaneous facial behavior. With few exceptions, CPM outperformed baseline and state-of-the art methods.