Learning Long-Term Dependencies in Irregularly-Sampled Time Series
release_xhbs6b46ejdb5l3n3whavqubai
by
Mathias Lechner, Ramin Hasani
2020
Abstract
Recurrent neural networks (RNNs) with continuous-time hidden states are a
natural fit for modeling irregularly-sampled time series. These models,
however, face difficulties when the input data possess long-term dependencies.
We prove that similar to standard RNNs, the underlying reason for this issue is
the vanishing or exploding of the gradient during training. This phenomenon is
expressed by the ordinary differential equation (ODE) representation of the
hidden state, regardless of the ODE solver's choice. We provide a solution by
designing a new algorithm based on the long short-term memory (LSTM) that
separates its memory from its time-continuous state. This way, we encode a
continuous-time dynamical flow within the RNN, allowing it to respond to inputs
arriving at arbitrary time-lags while ensuring a constant error propagation
through the memory path. We call these RNN models ODE-LSTMs. We experimentally
show that ODE-LSTMs outperform advanced RNN-based counterparts on non-uniformly
sampled data with long-term dependencies. All code and data is available at
https://github.com/mlech26l/ode-lstms.
In text/plain
format
Archived Files and Locations
application/pdf 855.8 kB
file_du3lw536fffstptlf3i4eebrx4
|
arxiv.org (repository) web.archive.org (webarchive) |
2006.04418v3
access all versions, variants, and formats of this works (eg, pre-prints)