Multi-Dataset, Multitask Learning of Egocentric Vision Tasks

Georgios Kapidis*, Ronald Poppe, Remco Veltkamp

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review


For egocentric vision tasks such as action recognition, there is a relative scarcity of labeled data. This increases the risk of overfitting during training. In this paper, we address this issue by introducing a multitask learning scheme that employs related tasks as well as related datasets in the training process. Related tasks are indicative of the performed action, such as the presence of objects and the position of the hands. By including related tasks as additional outputs to be optimized, action recognition performance typically increases because the network focuses on relevant aspects in the video. Still, the training data is limited to a single dataset because the set of action labels usually differs across datasets. To mitigate this issue, we extend the multitask paradigm to include datasets with different label sets. During training, we effectively mix batches with samples from multiple datasets. Our experiments on egocentric action recognition in the EPIC-Kitchens, EGTEA Gaze+, ADL and Charades-EGO datasets demonstrate the improvements of our approach over single-dataset baselines. On EGTEA we surpass the current state-of-the-art by 2.47 percent. We further illustrate the cross-dataset task correlations that emerge automatically with our novel training scheme.

Original languageEnglish
Pages (from-to)6618-6630
Number of pages13
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Issue number6
Publication statusPublished - 1 Jun 2023


  • Action Recognition
  • Activity recognition
  • Annotations
  • Egocentric Vision
  • Feature extraction
  • Multi-dataset Training
  • Multitask Learning
  • Object detection
  • Task analysis
  • Training
  • Training data


Dive into the research topics of 'Multi-Dataset, Multitask Learning of Egocentric Vision Tasks'. Together they form a unique fingerprint.

Cite this