Before we learn about linear regression we first learn the equation of a line / hyperplane. Of course the same can be said with regards linear classification, the linear autoencoder, etc., as well as their nonlinear extensions: we need to have a basic understanding of fundamental mathematical models before we try to perform supervised / unsupervised learning (on static datasets) using them. In this Section and the one that follows we introduce dynamic systems, which are the basic modeling tools used to model dynamic datasets - that is ordered data (often ordered in time). These modeling tools are used throughout virtually all fields of science, engineering, and mathematics, just as the line / hyperplane, polynomials, the relu and tanh functions, etc., are. Once we have a firm grasp on how these models we can then use them to perform e.g., supervised learning - much in the same way we fit a hyperplane to data when performing linear regression by tuning its paramters via the minimization of a cost function.

In this Section we introduce the most fundamental dynamic systems model: the dynamic system with fixed order.

15.2.1 A prototypical example: the moving average

Suppose we have a time series like the one shown below. As discussed first in Chapter 14 in the context of convolutional networks, when analyzing such time series for trends it is quite common to first smooth them. One way to do this is via a moving average - wherein we take a small window and slide it along the time series from its start to finish and average the values inside. Taking the average inside of each little window tends to cancel out noisy values, resulting in a smoothed version of the original series that is easier to study. Below we animate the process of building a moving average, and as you move the slider from left to right you will see the window in which each average is computed, which strattled on both sides by vertical blue bars, move from left to right across the series with the resulting moving average shown as a pink series.

Below we animate the process above, only on the first $50$ elements of the input series.

If we denote by $x_t,\,x_2,\,...,x_T$ the original time series - that is a set of ordered points (where $x_1$ comes before $x_2$, $x_2$ comes before $x_3$, and so forth) then we can easily express the moving average shown above algebraically / programatically. There we used a window size of length $15$, if you move the slider almost all the way to the left where the first set of moving average values are displayed. These first $15$ values of the moving average are simply set to the values of the input series itself

\begin{equation} h_t = x_t \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, t = 1,...,15. \end{equation}

After these first initial values we create those that follow by averaging the preceeding $15$ elements of the input series. That is, to create the $h_{p+1}$ we compute the average

\begin{equation} h_{t} = \frac{1}{15}\sum_{i=1}^{15}x_{t - i} \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, t = 16,...,T. \end{equation}

This simple moving average process is a popular example of what are called dynamic systems with fixed order. The 'dynamic systems' part of this jargon-phrase simply means that the system - here defined by the set of moving average values $h_1,\,h_2,\,...,h_T$ - is defined in terms of recent values of the input sequence $x_1,\,x_2,\,...,x_T$. The 'fixed order' part refers to just how many preceeding elements of input each value $h_t$ is based on - here this value was $15$ - and this value is fixed for each value of $h_t$ created.

Notice: the moving average is an example of a convolution that we saw in the previous Chapter in Section 14.1.

15.2.2 The generic fixed order dynamic system

The generic form of a dynamic system with fixed order looks very much like the moving average process detailed above, only it employs a function $f\left(\cdot\right)$ - which may not be a mere averaging function - to combine a window of $\mathcal{O}$ prior elements of an input sequence as

\begin{array} \ h_{t} = \gamma_t \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, t=1,...,\mathcal{O} \\ h_{t} = f\left(x_{t-1},...,x_{t-\mathcal{O}}\right) \,\,\,\,\, \,\,\,\,\,\,\, t = \mathcal{O} + 1,...,T. \\ \end{array}

Here the first values $h_t = \gamma_t$ are called the initial conditions of the system, and can in general be set to any values. Note this is only a generic form - there are unending numbers of variations on this basic theme. This is one of the most common ways of modeling ordered data - often modeling relationships in data over time.

Fixed order dynamic systems are used in a variety of scientific and engineering disciplines - especially in the filtering and analysis of time series (like the moving average shown above) (see e.g., this phenomenal introduction to the subject). Convolutional operations in general are prime examples of a dynamic systems with fixed order, and are frequently used to filter, process, or adjust digital signals e.g., Section 14.1 we saw a number of general convolutional operations often employed on digital signals like images, including blurring, edge detection, etc.,

Example 1. The moving average - examining a range of orders

Below we show an input sequence (in black) and animate a set of moving average sequences based on it, from $\mathcal{O} = 1$ to $\mathcal{O} = 50$. Note here that the initial condition in each instance $h_t = x_t$ is set to the actual input sequence for the first $t = \mathcal{O}$ values (instead of e.g., mirroring or zero-padding the sequence as was done when detailing the one dimensional convolution in Section 14.1).

Note as you increase the order how the moving average gets smoother, but mirrors the structure of the true underlying input sequence less and less. Also notice how the delay of the moving average - how its values consistently trail those of the original series - increases with the order of the system and is an artifact of using a large history of equally weighted examples of the series as a predictor of its next values.

Example 2. General filtering via convolution

All of the filtering examples discussed in Section 14.1 are examples of this broader class of dynamic systems with fixed order.

Example 3. Memoryless systems

With fixed order dynamic systems with input/output data we can in fact use order $\mathcal{O} = 0$ relationships, where each element of the output sequence is dependent on only the current input point as

\begin{equation} h_{t} = f\left(x_{t}\right). \end{equation}

These kind of systems are called memoryless since the dynamic system - and here in particular the output sequence - is constructed without any knowledge of the past input values.

For example, below we illustrate the sequence $h_{t} = \text{sin}\left(x_{t}\right)$, which is memoryless.

<matplotlib.text.Text at 0x11fc04dd8>