Multiscale Fusion of Transformer and Temporal Convolutional Networks for Action Recognition

Abstract

Action recognition constitutes an important research direction within computer vision, and it has been widely applied to intelligent monitoring, motion analysis, virtual reality and other practical fields. Most current approaches fail to effectively extract local temporal characteristics while modeling global context relationships, which restricts their capability when handling complicated long-duration action samples. Targeting this challenge, we design an innovative multi-module integrated network named WTCNet (Whale-Transformer-TCN Network). It leverages temporal convolutional networks to extract fine-grained temporal features, adopts Transformer to establish long-range global correlations, and applies the whale optimization algorithm to tune hyperparameters, so as to boost model performance and running efficiency. Comparative experiments are conducted on UCF-101 and Kinetics-400, two mainstream public datasets. The proposed method obtains 92.5% Top-1 accuracy and 98.3% Top-5 accuracy on UCF-101, and gains 82.1% Top-1 accuracy and 94.8% Top-5 accuracy on Kinetics-400, surpassing mainstream baseline algorithms. Meanwhile, our network shows obvious advantages in inference and training speed, verifying its high efficiency and application potential. This work delivers an effective scheme for challenging action recognition tasks, and also inspires follow-up research on multi-module combination and long-sequence learning. The proposed WTCNet can serve as a useful reference for both theoretical exploration and real-world deployment in action recognition.