Bridging distributional discrepancy with temporal dynamics for video understanding

Thumbnail Image
Chen, Min-Hung
AlRegib, Ghassan
Associated Organizations
Supplementary to
Video has become one of the major media in our society, bringing considerable interests in the development of video analysis techniques for various applications. Temporal Dynamics, which characterize how information changes along time, is the key component for videos. However, it is still not clear how temporal dynamics benefit video tasks, especially for the cross-domain case, which is close to real-world scenarios. Therefore, the objective of this thesis is to effectively exploit temporal dynamics from videos to tackle distributional discrepancy problems for video understanding. To achieve this objective, firstly I identified the benefits for exploiting temporal dynamics for videos, including proposing Temporal Segment LSTM (TS-LSTM) and Inception-style Temporal-ConvNet (Temporal-Inception) for general video understanding, and demonstrating that temporal dynamics can help reduce temporal variations for cross-domain video understanding. Since most previous work only evaluates the performance on small-scale datasets with little domain discrepancy, I collected two large-scale datasets for video domain adaptation: UCF HMDB_full and Kinetics-Gameplay to facilitate cross-domain video research, and proposed Temporal Attentive Adversarial Adaptation Network (TA3N) to simultaneously attend, align and learn temporal dynamics across domains. Finally, to utilize temporal dynamics from unlabeled videos for action segmentation, I proposed Self-Supervised Temporal Domain Adaptation (SSTDA) to jointly align cross-domain feature spaces embedded with local and global temporal dynamics by two self-supervised auxiliary tasks, binary and sequential domain prediction, and demonstrated the usefulness of adapting to unlabeled videos across variations.
Date Issued
Resource Type
Resource Subtype
Rights Statement
Rights URI