Slot Attention-based Concept Discovery To Mitigate Spurious Correlations
Published in CVPR (Submitted), 2023
Models relying on spurious correlations present in their training data can yield brittle predictions and introduce undesired biases. In this paper, we introduce a mechanism to mitigate these spurious correlations by leveraging unsupervised concept discovery. In the forward pass, we decompose images using object-centric slots, each attending to distinct regions in the input. During training, we create clusters of concepts by matching extracted slot-wise features with a learned dictionary of vector quantized codes. This procedure provides a means of monitoring and influencing the features learned by the classifier. Specifically, by controlling the sampling in stochastic gradient descent, we over-represent images of desired concepts, thereby reducing the impact of spurious correlations. We assess our method on various benchmark datasets for subpopulation shifts, demonstrating consistent improvements in performance without human-annotated groups.