Spatio-temporal network-based HAR analysis of RGB and depth data

Abstract

The problem of recognizing human actions has shifted from one of video processing to that of multi-model machine learning. This paper's goal is to use recurrence models to address the a forementioned issue. Here, a spatio-temporal model is built out of solely RGB video data by combining recurrent neural networks (RNN) with spatial convolutional neural networks (CNN). The model employs a variant of recurrent neural networks known as long short-term memory (LSTM) networks. To encapsulate the time series of the convolutional features extracted by CNNs, LSTM networks are used. In a series of videos, LSTMs study the evolution of action features over time. Results have demonstrated that temporal traits, when combined with spatial ones, are the most important.