Homemade Light! UniFormer, a high-scoring spatiotemporal representation learning model

Producer : Towhee Technical Team

Jointly produced by the Chinese Academy of Sciences, National University of Science and Technology, Shanghai Artificial Intelligence Laboratory, SenseTime, and the Chinese University of Hong Kong, the SoTA model UniFormer (UNIFIED TRANSFORMER) has achieved excellent results in mainstream data sets: on Kinetics-400/Kinetics600 Achieved 82.9% / 84.8% top-1 accuracy; achieved 60.9% and 71.2% top-1 accuracy on Something-Something V1 & V2. Once published, his paper received high marks and was finally included in ICLR 2022 (up to 7.5 points in the preliminary review: 8 8 6 8).

UniFormer Architecture

UniFormer proposes a Transformer structure that integrates 3D convolution and spatiotemporal self-attention mechanism, which can strike a balance between computation and accuracy. Different from the traditional Transformer structure that uses self-attention mechanism in all layers, the relation aggregator proposed in this paper can deal with redundant information and dependent information of video respectively. At a shallow level, the aggregator utilizes a small learnable matrix to learn local relations, greatly reducing computation by aggregating token information from small 3D neighborhoods. In the deep layer, the aggregator learns global relations through similarity comparison, and can flexibly establish long-range dependencies between tokens of distant video frames.

References:
Model use case: action-classification/video-swin-transformer
Paper: [UNIFORMER: UNIFIED TRANSFORMER FOR EFFICIENT
SPATIOTEMPORAL REPRESENTATION LEARNING]( https://arxiv.org/pdf/2201.04676v3.pdf )
more info:
High-scoring essay! UniFormer: A Unified Transformer for Efficient Spatio-Temporal Representation Learning
ICLR2022 UniFormer: Seamless integration of Transformer, a more efficient framework for learning spatiotemporal representations

---

For more project updates and details, please pay attention to our project ( https://github.com/towhee-io/... ), your attention is the powerful driving force for us to generate electricity with love, welcome to star, fork, slack three consecutive :)