Gradient Forward-Propagation for Large-Scale Temporal Video Modelling
release_jualyispircjzntv35bkpcx7jq
by
Mateusz Malinowski and Dimitrios Vytiniotis and Grzegorz Swirszcz and Viorica Patraucean and Joao Carreira
2021
Abstract
How can neural networks be trained on large-volume temporal data efficiently?
To compute the gradients required to update parameters, backpropagation blocks
computations until the forward and backward passes are completed. For temporal
signals, this introduces high latency and hinders real-time learning. It also
creates a coupling between consecutive layers, which limits model parallelism
and increases memory consumption. In this paper, we build upon Sideways, which
avoids blocking by propagating approximate gradients forward in time, and we
propose mechanisms for temporal integration of information based on different
variants of skip connections. We also show how to decouple computation and
delegate individual neural modules to different devices, allowing distributed
and parallel training. The proposed Skip-Sideways achieves low latency
training, model parallelism, and, importantly, is capable of extracting
temporal features, leading to more stable training and improved performance on
real-world action recognition video datasets such as HMDB51, UCF101, and the
large-scale Kinetics-600. Finally, we also show that models trained with
Skip-Sideways generate better future frames than Sideways models, and hence
they can better utilize motion cues.
In text/plain
format
Archived Files and Locations
application/pdf 12.0 MB
file_mvztank5ifc2thjbzagjzdnp5e
|
arxiv.org (repository) web.archive.org (webarchive) |
2106.08318v1
access all versions, variants, and formats of this works (eg, pre-prints)