Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

Many deep learning systems benefit from multi-task learning that combines several regression and classification objectives, but the authors observe that performance depends strongly on the relative weighting between each task's loss. Tuning these weights by hand is difficult and expensive, which can make multi-task learning impractical. They propose a principled approach that weighs multiple loss functions by considering the homoscedastic uncertainty of each task, allowing the model to learn quantities with different units or scales simultaneously in both classification and regression settings.

The method is demonstrated on a model that learns per-pixel depth regression together with semantic and instance segmentation from a single monocular input image. Perhaps surprisingly, the model can learn the multi-task weightings automatically and outperform separate models trained individually on each task, showing that uncertainty-based loss weighting both removes manual tuning and improves accuracy.