code
share
Previously when dealing with linear supervised learning (see e.g., Sections 8.4, 9.4, 10.3, and 11.3) we saw how normalizing each input feature of a dataset significantly aids in parameter tuning by improving the shape of a cost function's contours (making them more 'circular'). Another way of saying this is that we normalized every distribution that touches a system parameter - which in the linear case consists of the distribution of each input feature. The intuition that normalizing parameter touching distributions carries over completely from the linear learning scenario to our current situation - where we are conducting nonlinear learning via multilayer perceptrons. The difference here is that now we have many more parameters (in comparison to the linear case) and many of these parameters are internal as opposed to weights in a linear combination. Nonetheless each parameter - as we detail here - touches a distribution that when normalized tends to improve optimization speed.
Specifically - as we will investigate here in the context of the multilayer perceptron - to completely carry over the idea of input normalization to our current scenario we will need to normalize the output of each and every network activation. Moreover since these activation distributions naturally change whenever during parameter tuning - e.g., whenever a gradient descent step is made - we must normalize these internal distributions every time we make a parameter update. This leads to the incorporation of a normalization step grafted directly onto the architecture of the multilayer perceptron itself - which is called every time weights are changed. This natural extension of input normalization is popularly referred to as batch normalization.
When discussing linear model based learning in Chapters 8 - 11 we employed the generic linear model
\begin{equation} \text{model}\left(\mathbf{x},\mathbf{w}\right) = w_0 + x_1w_1 + \cdots + x_Nw_N \end{equation}for both regression and classification. When tuning these weights via the minimization of any cost function over a dataset of $P$ points $\left\{\mathbf{x}_p,y_p\right\}_{p=1}^P$ we can see how the $n^{th}$ dimension of each input point $x_{p,n}$ touches the $n^{th}$ weight $w_n$. Because of this, as we discussed previously, the input distribution along the $n^{th}$ dimension $\left\{x_{p,n} \right\}_{p=1}^P$ touching $w_n$ strongly affects the contours of any cost function along the $w_n$ direction. If these input distributions differ, as is typically the case in practice, the contours of a cost function stretch out in long directions making optimization via gradient descent quite challenging irregardless of the choice of steplength. However by normalizing each input dimensions via the standard normalization scheme - i.e., by mean centering and re-scaling by its standard deviation (see Sections 8.4, 9.4, and 10.3) - we can subtantially temper the contours of a cost funtion making it much easier to optimize.
Performing standard normalization along the $n^{th}$ input feature means making the replacement
\begin{equation} x_{p,n} \longleftarrow \frac{x_{p,n} - \mu_{n}}{\sigma_{n}} \end{equation}where $\mu_n$ and $\sigma_n$ are the mean and standard deviation along the $n^{th}$ feature of the input, respectively. Performing standard normalization on the input of a linear model helps temper any cost function along each parameter direction $w_1,\,w_2,\,...,w_N$. Of course once normalized these input distributions $\left\{x_{p,n} \right\}_{p=1}^P$ for each $n$ never change again - they remain stable regardless of how we set the parameters of our model / during training.
Below we repeat a simple version of this standard normalizer for convenience, which we will use repeatedly below.
# standard normalization function - input data, output standaard normalized version
def standard_normalize(data):
# compute the mean and standard deviation of the input
data_means = np.mean(data,axis = 1)[:,np.newaxis]
data_stds = np.std(data,axis = 1)[:,np.newaxis]
# check to make sure thta x_stds > small threshold, for those not
# divide by 1 instead of original standard deviation
ind = np.argwhere(data_stds < 10**(-2))
if len(ind) > 0:
ind = [v[0] for v in ind]
adjust = np.zeros((data_stds.shape))
adjust[ind] = 1.0
data_stds += adjust
# return standard normalized data
return (data - data_means)/data_stds
Now let us think about our current context, where we are in general employing a linear combination of $B$ multilayer perceptron feature transformations in a nonlinear model that takes the general form
where note here the variable $\mathbf{w}$ on the left hand side denotes all parameters of the model (the weights of the linear combination $w_0$,...,$w_B$) as well as those internal to each nonlinear feature transformation $f_1$,...,$f_B$. As we can see by studying simple examples, here standard normalization of the input only tempers the contours of a cost function along the weights internal to the first layer of a multilayer perceptron only, as these are the weights touched by the distribution of each input dimension.
For example let us look at the simplest case (of a multilayer perceptron) and suppose each feature transformation of our model is a single hidden layer perceptron, the $b^{th}$ of which takes the general form (as introduced in the previous Section)
\begin{equation} f^{(1)}_b\left(\mathbf{x}\right)=a\left(w^{\left(1\right)}_{0,\,b}+\underset{n=1}{\overset{N}{\sum}}{w^{\left(1\right)}_{n,\,b}\,x_n}\right) \end{equation}where $a\left(\cdot\right)$ is an activation function. Hence this means that our model takes the form
In this instance we can see that the data input does not touch weights of the linear combination $w_1,\,w_2,...,w_B$ (as was the case with our linear model), but touches the internal weights of each perceptron i.e., the $n^{th}$ input feature dimension touches the internal weights $w_{b,n}$ for $b=1,...,B$. So in other words, by performing standard normalization on the input here we temper the contours of a cost function along the internal weights of a model employing single hidden layer elements.
But what about the contours of the cost function along the weights of the linear combination $w_1,\,w_2,...,w_B$? Which data distributions touch them? A simple glance at our model above and we can see that it is the distribution of each perceptron / activation output over the input data. In other words, the distribution $\left\{f^{(1)}_b\left(\mathbf{x}_p\right) \right\}_{p=1}^P$ touches the weight $w_b$ of the linear combiination. Now if we know that the distribution touching the weight of a model affects any cost function along this dimension then, in analogy to how we have normalized the input data, perhaps if we (standard) normalize each unit / activation output distribution we can similarly make it easier to optimize our model.
To do this we would standard normalize each unit itself by mean centering and re-scaling it according to the mean / standard deviation of the distribution over our input as
\begin{equation} f_b^{(1)} \left(\mathbf{x} \right) \longleftarrow \frac{f_b^{(1)} \left(\mathbf{x}\right) - \mu_{f_b^{(1)}}}{\sigma_{f_b^{(1)}}} \end{equation}where
\begin{array} \ \mu_{f_b^{(1)}} = \frac{1}{P}\sum_{p=1}^{P}f_b^{(1)}\left(\mathbf{x}_p \right) \\ \sigma_{f_b^{(1)}} = \sqrt{\frac{1}{P}\sum_{p=1}^{P}\left(f_b^{(1)}\left(\mathbf{x}_p \right) - \mu_{f_b^{(1)}} \right)^2}. \end{array}Would standard normalizing - also referred to as batch normalizing in the context of single / multilayer perceptrons - each unit of our single layer perceptron model help speed up optimization (much in the same way standard normalizing input helps in this manner)? It seems promising given our experience with normalizing our input - but the only way to know for sure is to experiment with the idea to find out. As we show via several experiments below, it does indeed help with optimization as we have intuited it would.
Note how for an arbitrary single layer unit $f^{(1)}\left(\mathbf{x}\right)$ how we can easily adjust the generic recipe for building single layer functions introduced in the previous Section to include this standard normalization step. To construct our batch normalized single layer perceptron function we simply follow the adjusted perceptron recipe below.
1:Β Β input: Activation function $a\left(\cdot\right)$ and input data $\left\{\mathbf{x}_p\right\}_{p=1}^P$
2:Β Β Compute linear combination: $\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,v = w_{0}^{(1)}+{\sum_{n=1}^{N}}{w_{n}^{(1)}\,x_n}$
3:Β Β Pass result through activation: $\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, f^{(1)}\left(\mathbf{x}\right) = a\left(v\right)$
4:Β Β Compute mean: $\mu_{f^{(1)}}$ / standard deviation $\sigma_{f^{(1)}}$ of: $\,\,\,\,\,\,\,\,\left\{f^{(1)}\left(\mathbf{x}_p\right) \right\}_{p=1}^P$
5:Β Β Standard normalize: $ \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,
f^{(1)} \left(\mathbf{x} \right) \longleftarrow \frac{f^{(1)} \left(\mathbf{x}\right) - \mu_{f^{(1)}}}{\sigma_{f^{(1)}}}
$
6:Β Β output: Batch normalized single layer unit $\,\, f^{(1)} \left(\mathbf{x} \right)$
However it is important to note that - unlike the distribution of the input data - the distribution of each of our single layer units changes every time the internal parameters of our system are changed. In other words, since each of the single layer units $f^{(1)}_b$ has internal parameters the distribution $\left\{f^{(1)}_b\left(\mathbf{x}_p\right) \right\}_{p=1}^P$ varies depending on the setting of these internal weights (as we will illustrate via examples below). In the jargon of deep learning this is often referred to as internal covariate shift or just covariate shift for short of a network model.
Since the weights of our model will change during optimization - e.g., from one step of gradient descent to the next - in order to keep the unit distributions normalized we have to normalize them at every step of parameter tuning (e.g., via gradient descent). To do this we can simply build in a standard normalization step directly into the perceptron architecture itself, as we show in the Python implementation next.
The distribution of single layer units / activation outputs $\left\{f^{(1)}_b\left(\mathbf{x}_p\right) \right\}_{p=1}^P$ changes depending on the setting of the units' internal parameters, an issue referred to as internal covariate shift in the jargon of deep learning. This means that - if we want to temper these distributions at each step of a run of e.g., gradient descent - we must normalize them every time weights are changed. This can be easily done by inserting a normalization step into the recursive algorithm for constructing perceptron units.
Python implementation of batch normalization for single layer unitsΒΆBelow we illustrate how one can implement batch normalization for single layer perceptron units. First we provide the same Python implementation for our original single layer perceptron feaeture_transforms described in the previous Section. We also define a relu activation function for testing.
# relu activation
def activation(t):
return np.maximum(0,t)
# a feature_transforms function for computing
# U_1 single layer perceptron units efficiently
def feature_transforms(x,W_1):
# pad with ones (to compactly take care of bias) for first layer computation
o = np.ones((1,np.shape(x)[1]))
x = np.vstack((o,x))
# compute linear combination of current layer units
v = np.dot(x.T, W_1).T
# pass through activation
a = activation(v)
return a
To standard normalize each unit we simply need to add a normalization step to the end of this function, as shown below. To do this we employ the standard normalization function standard_normalize given above. To differentiate this batch normalized version from the one above we call it feature_transforms_batch_normalized.
# a feature_transforms function for computing
# U_1 single layer batch normalized
# perceptron units efficiently
def feature_transforms_batch_normalized(x, W_1):
# pad with ones (to compactly take care of bias) for first layer computation
o = np.ones((1,np.shape(x)[1]))
x = np.vstack((o,x))
# compute linear combination of current layer units
v = np.dot(x.T, W_1).T
# pass through activation
a = activation(v)
# NEW - perform standard normalization on
# first layer unit distributions
a = standard_normalize(a)
return a
Now we can run a few experiments to verify that our batch normalzied version is indeed easier to optimize than the standard single layer perceptron model.
In this example we illustrate the covariate shift of a single layer perceptron with two relu units $f^{(1)}_1$ and $ f^{(1)}_2$
employing the two-class classification dataset shown below.
We now run $5,000$ steps of gradient descent to minimize the softmax cost using this single layer network, where we standard normalize the input data. We use a set of random weights for the network loaded in from memory.
Below we show an animation of this gradient descent run, plotting the single unit distribution $\left\{f^{(1)}_1\left(\mathbf{x}_p\right),\,f^{(1)}_2\left(\mathbf{x}_p\right) \right\}_{p=1}^P$ at a subset of the steps taken during the run. In the left panel we show this covariate shift or activation output distribution at the $k^{th}$ step of the optimization, while the right panel shows the complete cost function history curve where the current step of the animated optimization is marked on the curve with a red dot. Moving the slider from left to right progresses the run from start to finish.
As you can see by moving the slider around, the distribution of activation outputs - i.e., the distributions touching the weights of our model's linear combination $w_0$ and $w_1$ - change dramatically as the gradient descent algorithm progresses. We can intuit (from our previous discussions on input normalization) that this sort of shifting distribution negatively effects the speed at which gradient descent can properly minimize our cost function.
Now we repeat the above experiment using the batch normalized single layer perceptron - making a run of $10,000$ gradient descent steps using the same initialization used above. We then animate the covariate shift / distribution of activation outputs using the same animation tool used above as well. Moving the slider below from left to right - progressing the algorithm - we can see here that the distribution of activation outputs stays considerably more stable.