Learning Time Sequences

There are many tasks that require learning a temporal sequence of events. These problems can be broken into 3 distinct types of tasks:

Some of the methods that are used include

Tapped Delay Lines / Time Delay Neural Networks

One of the simplest ways of performing sequence recognition because conventional backpropagation algorithms can be used.

Downsides: Memory is limited by length of tapped delay line. If a large number of input units are needed then computation can be slow and many examples are needed.

A simple extension to this is to allow non-uniform sampling:

where wi is the integer delay assoicated with component i. Thus if there are n input units, the memory is not limited simply the previous n timesteps.

Another extension that deals is for each "input" to really be a convolution of the original input sequence.

In the case of the delay line memories:

Other variations for c are shown graphically below:

This figure is taken from "Neural Net Architectures for Temporal Sequence Processing", by Mike Moser.

[Top] [Next: Recurrent Networks I] [Back to the first page]