{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Neural Learning" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true } }, "source": [ "Last revision: Wed 6 Apr 2022 21:09:00 AEST" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Author: Omar Ghattas" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this lab we will expand on some of the concepts of neural learning, starting with an extension of the perceptron to the linear unit, which can be trained using gradient descent. \n", "\n", "The remainder of the lab goes into some \"hands-on\" aspects of supervised learning for neural networks, based on the multi-layer perceptron trained by error back-propagation. \n", "\n", "This code is for explanatory purposes only – for real neural networks you would use one of the many code libraries that exist. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Acknowledgements:\n", "Autograd Theory:\n", "\n", " 1. https://people.cs.umass.edu/~domke/courses/sml2011/08autodiff_nnets.pdf\n", " 2. Mathematics for Machine Learning by Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong.\n", "\n", "Pytorch Neural Network Examples:\n", "\n", " 1. https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html\n", " \n", "MLP Code from Scratch\n", "\n", " 1. https://github.com/rcassani/mlp-example" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Useful Functions\n", "We will rely on the following helper functions which you can treat as a black box." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "def generate_data(n=20, means=[[3,3],[-1,1]], seed=1):\n", " '''\n", " generate data from two gaussians\n", " '''\n", " np.random.seed(seed)\n", " m1 = np.array(means[0])\n", " m2 = np.array(means[1])\n", " S1 = np.random.rand(2,2)\n", " S2 = np.random.rand(2,2)\n", " dist_01 = np.random.multivariate_normal(m1, S1.T @ S1, n)\n", " dist_02 = np.random.multivariate_normal(m2, S2.T @ S2, n)\n", " X = np.concatenate((np.ones(2*n).reshape(-1,1), \n", " np.concatenate((dist_01, dist_02))), axis=1)\n", " y = np.concatenate((np.ones(n), np.zeros(n))).reshape(-1,1)\n", " shuffle_idx = np.random.choice(2*n, size=2*n, replace=False)\n", " X = X[shuffle_idx]\n", " y = y[shuffle_idx]\n", " return X, y\n", "\n", "def plot_perceptron(ax, X, y, w):\n", " pos_points = X[np.where(y==1)[0]]\n", " neg_points = X[np.where(y==0)[0]]\n", " ax.scatter(pos_points[:, 1], pos_points[:, 2], color='blue')\n", " ax.scatter(neg_points[:, 1], neg_points[:, 2], color='red')\n", " xx = np.linspace(-6,6)\n", " yy = -w[0]/w[2] - w[1]/w[2] * xx\n", " ax.plot(xx, yy, color='orange')\n", " \n", " ratio = (w[2]/w[1] + w[1]/w[2])\n", " xpt = (-1*w[0] / w[2]) * 1/ratio\n", " ypt = (-1*w[0] / w[1]) * 1/ratio\n", " \n", " ax.arrow(xpt, ypt, w[1], w[2], head_width=0.2, color='orange')\n", " ax.axis('equal')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Augmenting the Perceptron with Gradient Descent\n", "\n", "Recall from that the Perceptron is a mistake-driven algorithm - it continues to update so long as it makes mistakes on the data. In the linearly separable setting, where can separate the data exactly with a linear separator, the algorithm is always guaranteed to terminate. If however the data is not linearly separable, the algorithm will never terminate. In this case, we might still want to learn a linear classifier that achieves low error (classifies most of the points correctly), and instead of looking at a mistake-driven algorithm, we look instead to a loss-based algorithm. In this section, we introduce learning linear classifiers via gradient descent. You should view this section of the lab as a precursor to learning in multilayer perceptrons (Neural Networks) via Backpropagation. This extension is known as the linear unit, it is basically a one-node neural network." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The original perceptron uses the sgn activation function, which we saw earlier has a 'S' shape. We can use a smoother version of the sgn activation known as a sigmoid. The reason we prefer smoother activations is that we can take derivatives (any function with sharp angles (discontinuities) is not differentiable). We use here a type of sigmoid known as the logistic sigmoid: \n", "\n", "$$\n", "\\sigma(x) = \\frac{1}{1+e^{-x}}\n", "$$\n", "\n", "Now, note that\n", "\n", "$$\n", "\\frac{d \\sigma(x)}{d x} = \\sigma(x) (1-\\sigma(x))\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now implement the sigmoid and its derivative and observe its shape for different values of the steepness parameter:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def sigmoid(x):\n", " return 1/(1+np.exp(-1*x))\n", "\n", "def grad_sigmoid(x):\n", " return sigmoid(x) * (1-sigmoid(x))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# plot sigmoid functions \n", "xx = np.linspace(-6, 6, 1000)\n", "plt.plot(xx, sigmoid(xx), label='$\\sigma(x)$')\n", "plt.plot(xx, grad_sigmoid(xx), label='$\\sigma\\'(x)$')\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Instead of updating $w$ whenever we make a mistake, i.e. whenever $w^T x y < 0$, we will consider updating the weight vector by moving in a direction that minimizes the loss as much as possible, this is known as gradient descent/\n", "\n", "Recall that the prediction (output) of the standard perceptron was $\\hat{y}=\\text{sgn}(h_w(x))$. Since we are no longer using the sign activation, but the sigmoid activation, our output now is $\\hat{y} = \\sigma(h_w(x)) = \\sigma(w^T x)$. Next, consider the cross-entropy loss\n", "\n", "$$\n", "L(w) = - \\sum_{i=1}^N y_i \\ln p_i + (1-y_i) \\ln (1-p_i), \\qquad p_i = \\sigma(w^T x_i) = \\frac{1}{1+e^{-w^T x_i}}.\n", "$$\n", "\n", "We would like to find $\\hat{w}$ that minimizes this loss, i.e. we want to solve \n", "\n", "$$\n", "\\hat{w} = \\arg \\min_w L(w).\n", "$$\n", "\n", "Unfortunately in this case, we cannot solve for $\\hat{w}$ in closed form, and so we must resort to numerical techniques such as gradient descent." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " \n", "\n", "#### Exercise:\n", " \n", "1. Show that $\\frac{d p_i}{dw} = p_i (1-p_i) x_i$, where $p_i = \\sigma(w^T x_i)$.\n", "2. Show that $\\frac{d \\ln p_i}{dw} = (1-p_i) x_i$.\n", "3. Using the results of the previous parts, show that $\\frac{d L(w)}{dw} = - \\sum_{i=1}^n (y_i - p_i) x_i$. Write down the gradient descent update for $w$ with step size $\\eta$.\n", "4. Fill out the code below to implement gradient descent for the model on a randomly generated dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def loss_i(w, x_i, y_i):\n", " '''cross entropy loss for i-th data point'''\n", " # your code here\n", " return # your code here\n", " \n", "def grad_loss_i(w, x_i, y_i):\n", " '''grad loss for i-th data point'''\n", " # your code here\n", " return # your code here\n", "\n", "def gradient_descent(X, y, eta, T):\n", " N = X.shape[0]\n", " w = np.array([0,0,0]) # init w\n", " for t in range(T):\n", " loss = 0\n", " grad_loss = 0\n", " for i in range(N):\n", " loss += loss_i(w, X[i], y[i])\n", " grad_loss += grad_loss_i(w, X[i], y[i])\n", " print(f\"loss = {loss}\")\n", " w = w - eta * grad_loss\n", " fig, ax = plt.subplots()\n", " plot_perceptron(ax, X, y, w)\n", " plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "np.random.seed(12)\n", "X, y = generate_data(n=40, means=[[2,1],[1,0]])\n", "gradient_descent(X, y, 0.1, 20)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So far so good, we now have an algorithm that can give us something useful even when the data is not linearly separable. This is important since in practice we cannot hope to achieve zero error. One problem however is that in order to compute the gradient at each step, we need to run over the entire data set, which can be computationally infeasible. This brings us to the idea of Stochastic Gradient Descent (SGD). The idea behind SGD is very simple, instead of computing the true gradient by summing over the entire dataset, we instead estimate the gradient by looking at the gradient on a single observation. In other words, we update the weight vector after each observation." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " \n", "\n", "#### Exercise:\n", " \n", "5. Instead of SGD, which might have high variance, try to implement a batch gradient descent, in which we consider a subset of size $m \\ll n$ to estimate the gradient with. Plot your results after each update. How does this compare with the results from gradient descent?\n", "6. Consider the case of Linear Regression from the first tutorial. In this case, the optimisation can be done in closed form (least squares solution). Use gradient descent instead to solve linear regression on a simulated dataset, does the gradient descent solution achieve the least squares solution?\n", "7. Investigate other sigmoid functions and implement them: https://en.wikipedia.org/wiki/Sigmoid_function" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Neural Networks\n", "\n", "At this point, you should be comfortable with the two concepts covered so far: 1. Perceptrons and 2. Gradient Descent. These are the two basic building blocks for neural networks, also referred to as Multi-layer percetrpons. The idea is to chain together perceptrons to be able to learn very complex decision surfaces (as opposed to just linear decision surfaces in the case of the standard perceptron). Once we chain perceptrons together however, how do we learn the weights? We do so via Backpropagation, which at its core is an algorithm that implements the chain rule from calculus. Recall that the chain rule comes up any time we have a composition of functions. For example, consider two functions $g_1(x) = e^x$ and $g_2(x)=x^2-3$, then we can compose these functions:\n", "\n", "$$\n", "(g_1 \\circ g_2) (x) = g_1(g_2(x)) = e^{x^2 - 3}.\n", "$$\n", "\n", "We could also compose them in the reverse direction:\n", "\n", "$$\n", "(f_2 \\circ g_1) (x) = g_2(g_1(x)) = e^{2x} - 3.\n", "$$\n", "\n", "If we have a third function, say $g_3(x) = \\log (x)$, then we can create the following 3-fold composition:\n", "\n", "$$\n", "(g_1 \\circ g_2 \\circ g_3)(x) = g_1(g_2(g_3(x))) = g_1( (\\log x)^2 - 3 ) = e^{(\\log x)^2 - 3 }.\n", "$$\n", "\n", "A neural network is essentially one big composition of functions, let's see why. Assume we have a $D$ layer neural network, and assume that the $i$-th layer has $n_i$ many units (or perceptrons), which can be different from layer to layer. The $j$-th perceptron in layer $i$ can therefore be represented by its weight vector $w^{ij}$ and its bias term $b^{ij}$. Instead of keeping track of all these vectors, we combine all the weights in a single layer into a single weight matrix, $W_i$, where each row of the matrix corresponds to the weights of a perceptron. Similarly, we combine all bias terms into a single vector $b_i$ (See the diagram below). To see why this is a composition, we can write the forward step of a NN as follows:\n", "\n", "\\begin{align*}\n", "f_0 &:= x\\\\\n", "f_i &:= \\sigma(W_{i-1} f_{i-1} + b_{i-1}) \\quad i=1,\\dots,K.\n", "\\end{align*}\n", "\n", "In other words, we set the input $x$ to be output of the zero-th layer, and the output of the $i$-th layer is the element-wise sigmoid of the matrix product $W_{i-1} f_i + b_{i-1}$. What is this product? Recall from above that the product $W f$ returns a vector where each element is the dot product $\\langle w_i, f\\rangle$, where $w_i$ is the $i$-th row of $W$, i.e. the weights of the $i$-th perceptron/unit in the corresponding layer. Adding $b$ adds the bias, and the sigmoid is applied element-wise. This output then becomes the input of the next layer, and so on.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's unroll this process and show explicitly why it is a composition of functions. Let $f_K$ denote the output of a $K$ layer NN, then\n", "\n", "\\begin{align*}\n", "y \n", "&= f_K(x)\\\\\n", "&= \\sigma(W_{K-1} f_{K-1}(x) + b_{K-1})\\\\\n", "&= \\sigma(W_{K-1} \\sigma(W_{K-2} f_{K-2}(x) + b_{K-2}) + b_{K-1})\\\\\n", "& ~~~~ \\vdots\\\\\n", "&= \\sigma(W_{K-1} \\sigma(W_{K-2} (\\cdots \\sigma( W_{0} x + b_0) )+ \\cdots + b_{K-2}) + b_{K-1}),\n", "\\end{align*}\n", "\n", "which is equivalent to writing:\n", "\n", "$$\n", "y = (f_K \\circ f_{K-1} \\circ \\dots \\circ f_1)(x) = f_K(f_{K-1}(f_{K-2}(\\dots(f_1(x)) \\cdots)).\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have understood the forward pass of a neural network, we can turn our attention to learning the parameters of the network. In this case, we need to learn the weights and biases in every layer, and we can combine our parameters into a single set $\\theta := \\{W_0, b_0, W_1, b_1,\\dots, W_{K-1}, b_{K-1}\\}$. Let $\\theta_i = \\{W_i, b_i\\}$. Assume that we have some loss function $\\mathcal{L}$, say MSE. Then, to perform gradient descent, we need to compute gradients of the loss with respect to each of the parameters, and update as in the gradient descent discussion above. The gradeints can be computed using the chain rule:\n", "\n", "\\begin{align*}\n", "\\frac{\\partial \\mathcal{L}}{\\partial \\theta_{K-1}} &= \\frac{\\partial \\mathcal{L}}{\\partial f_{K}} \\color{red}{\\frac{\\partial f_K}{\\partial \\theta_{K-1}}}\\\\\n", "\\frac{\\partial \\mathcal{L}}{\\partial \\theta_{K-2}} &= \n", "\\frac{\\partial \\mathcal{L}}{\\partial f_{K}} \n", "\\color{blue}{\\frac{\\partial f_K}{\\partial f_{K-1}}}\n", "\\color{red}{\\frac{\\partial f_{K-1}}{\\partial \\theta_{K-2}}}\\\\\n", "\\frac{\\partial \\mathcal{L}}{\\partial \\theta_{K-3}} &= \n", "\\frac{\\partial \\mathcal{L}}{\\partial f_{K}} \n", "\\color{blue}{\\frac{\\partial f_K}{\\partial f_{K-1}} \\frac{\\partial f_{K-1}}{\\partial f_{K-2}} }\n", "\\color{red}{\\frac{\\partial f_{K-2}}{\\partial \\theta_{K-3}}}\\\\\n", "&\\vdots\\\\\n", "\\frac{\\partial \\mathcal{L}}{\\partial \\theta_{i}} &= \n", "\\frac{\\partial \\mathcal{L}}{\\partial f_{K}} \n", "\\color{blue}{\\frac{\\partial f_K}{\\partial f_{K-1}} \\cdots \\frac{\\partial f_{i+1}}{\\partial f_{i+1}} }\n", "\\color{red}{\\frac{\\partial f_{i+1}}{\\partial \\theta_{i}}}\\\\\n", "\\end{align*}\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The terms in blue are the partial derivatives of the output of a layer with respect to its inputs, and the terms in red are the partial derivatives of the output of a layer with respect to its parameters. We make the important observation that if we simply compute each of the gradients naively, then we are redoing the same computations over and over, since the same (blue) terms crop up in many of the calculations. The idea behind backpropagation is to do the chain rule but in a smarter way, so as to re-use the computations already done at a layer to solve for gradients in later layers. For a detiled look at backpropagation, please refer to the more detailed derivation here: https://www.youtube.com/watch?v=dVexyOg3HnQ&t=1s&ab_channel=OmarGhattas " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Automatic Differentiation and a short intro to PyTorch\n", "\n", "Before we move on to computation, let us explore the computational aspects of Automatic Differentiation. Backpropagation is one special case of automatic differentiation, and many deep learning libraries, such as PyTorch, are general use automatic differentiation libraries. We saw earlier that standard chain rule can be computationally naive, and in fact is infeasible for compositions of a large number of functions, which is important if we want to do anything with deep networks! Let us consider a simplified example to demonstrate the power of autodiff.\n", "\n", "Consider the following function:\n", "\n", "$$\n", "f(x) = \\exp (\\exp(x) + \\exp(2x)) + \\sin(\\exp(x) + \\exp(2x)).\n", "$$\n", "\n", "This looks quite complicated, but with a bit of work we can show that the derivative of $f$ is\n", "\n", "$$\n", "f'(x) = \\exp (\\exp(x) + \\exp(2x))(\\exp(x) + 2\\exp(2x)) + \\cos(\\exp(x) + \\exp(2x))(\\exp(x) + 2\\exp(2x))\n", "$$\n", "\n", "For those of you that attempted this by hand, you should see that the standard differentiation approach is somewhat wasteful, since we have the same factor $(\\exp(x) + \\exp(2x))$ in both expressions. Let's consider a more algorithmic approach. Define the following intermediate variables:\n", "\n", "\\begin{align*}\n", "a &= \\exp(x)\\\\\n", "b &= a^2\\\\\n", "c &= a+b\\\\\n", "d &= \\sin(c)\\\\\n", "e &= \\exp(c)\\\\\n", "f &= d+e\n", "\\end{align*}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This may look strange at first, but we are basically rewriting $f(x)$ as an iterative procedure. We can represent this in graph form, using a structure known as a computational graph:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The square nodes are computation nodes, they implement the functions they are named after, and the circular nodes are variable nodes, which store the intermediate variables. Viewing $f$ in this way allows us to compute the gradient in an iterative manner. We compute the derivatives of the intermediate variables with respect to their inputs:\n", "\\begin{align*}\n", "\\frac{\\partial a}{\\partial x} &= \\exp(x)\\\\\n", "\\frac{\\partial b}{\\partial a} &= 2 a\\\\\n", "\\frac{\\partial c}{\\partial a} &= 1 = \\frac{\\partial c}{\\partial b}\\\\\n", "\\frac{\\partial d}{\\partial c} &= \\cos(c)\\\\\n", "\\frac{\\partial e}{\\partial c} &= \\exp(c)\\\\\n", "\\frac{\\partial f}{\\partial d} &= 1 = \\frac{\\partial f}{\\partial e}\n", "\\end{align*}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The goal is to compute $\\frac{\\partial f}{\\partial x}$, we can do so by working our way backwards through the computation graph from the output node (node $f$) to the input node (node $x$). We first introduce the following notation $\\text{child}(y)$ to denote the children of a node $y$. For example $\\text{child}(c) = \\{d, e\\}$. Then, for any node $y$ in the graph, the chain rule tells us that\n", "\n", "$$\n", "\\frac{\\partial f}{\\partial y} = \\sum_{z \\in \\text{child}(y)} \\frac{\\partial f}{\\partial z} \\frac{\\partial z}{\\partial y}\n", "$$\n", "\n", "Let's apply this formula, we already have the partial derivatives of $f$ with respect to $d,e$ from above, so we start with $c$:\n", "\n", "\\begin{align*}\n", "\\frac{\\partial f}{\\partial c} &= \\frac{\\partial f}{\\partial d}\\frac{\\partial d}{\\partial c} + \\frac{\\partial f}{\\partial e}\\frac{\\partial e}{\\partial c} = 1 \\times \\exp(c) + 1 \\times \\cos (c)\\\\\n", "\\frac{\\partial f}{\\partial b} &= \\frac{\\partial f}{\\partial c}\\frac{\\partial c}{\\partial b} =\\frac{\\partial f}{\\partial c} \\times 1\\\\\n", "\\frac{\\partial f}{\\partial a} &= \\frac{\\partial f}{\\partial b}\\frac{\\partial b}{\\partial a} + \\frac{\\partial f}{\\partial c}\\frac{\\partial c}{\\partial a} = \n", "\\frac{\\partial f}{\\partial b} \\times 2 a + \\frac{\\partial f}{\\partial c} \\times 1\\\\\n", "\\frac{\\partial f}{\\partial x} &= \\frac{\\partial f}{\\partial a} \\frac{\\partial a}{\\partial x} = \\frac{\\partial f}{\\partial a} \\times \\exp(x)\n", "\\end{align*}\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Observe that the computation required for calculating the derivative is proportional to the computation required to calculate the function. You can visualise this process as if each node in the computational graph stores the derivatives of itself with respect to any child nodes, then to compute the gradient of the graph, we start from the output and work our way backwards, at each step re-using gradients computed earlier in the process. As you become more familiar with backpropagation, you should see that the two concepts are identical. Let's finalise our understanding by looking at a toy example using Pytorch. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since we have an excplicit form for the gradient, the goal will be to compare our explicit form to the solution computed numerically with PyTorch. Note that this is numerical differentiation, not symbolic, so the value of the gradient will change as we vary the input. We will compare the two results at the end." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import torch # run \"! pip3 install torch\" in jupyter if you do not have torch\n", "\n", "def func(x):\n", " t = np.exp(x) + np.exp(2*x)\n", " return np.exp(t) + np.sin(t)\n", "\n", "def grad_func(x):\n", " t1 = np.exp(x) + np.exp(2*x)\n", " t2 = np.exp(x) + 2*np.exp(2*x)\n", " return t2 * (np.exp(t1) + np.cos(t1))\n", "\n", "def sequential_func(x):\n", " inp = torch.tensor([[x]], dtype=torch.float64, requires_grad=True)\n", " a = torch.exp(inp)\n", " b = torch.pow(a, 2)\n", " c = a + b\n", " d = torch.exp(c)\n", " e = torch.sin(c)\n", " f = d + e\n", " f.backward()\n", " return inp.grad.item()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "x_input = 0.2\n", "print(\"explicit gradient: \", grad_func(x_input))\n", "print(\"autograd gradient: \", sequential_func(x_input))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " \n", "\n", "#### Exercise:\n", " \n", "8. Consider the function $f(x) = \\sqrt{x^2 + \\exp(x^2)} + \\cos(x^2 + \\exp(x^2))$. Compute its derivative explicitly, and then write down its computational graph, repeat the analysis and use PyTorch to check your solutions.\n", "9. We have just scratched the surface of numerical optimisation with PyTorch, try to re-implement the gradient descent perceptron, note that we no longer need to explicitly compute the gradient, we can just rely on automatic differentiation.\n", "10. Repeat 2. but for Linear Regression, or any other loss-based model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To finish of the lab, we will see how to use pytorch to train a neural network on the MNIST dataset, and then we will look at some code that implements a neural network from scratch. In practice, we will obviously make use of existing libraries like pytorch, but it is also important to be able to build things from scratch." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### MNIST Dataset\n", "First we load in the MNIST dataset using the pytorch utilities. We transform the data and create a loader object that makes training networks more straight forward. We can set the batch size directly in the loader object, and this is analogous to the discussion on batch gradient descent earlier." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import torchvision #! pip3 install torchvision\n", "import torchvision.transforms as transforms\n", "from torchvision.datasets import MNIST\n", "\n", "transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])\n", "trainset = MNIST(root = './', train=True, download=True, transform=transform)\n", "testset = MNIST(root = './', train=False, download=True, transform=transform)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "batch_size = 5\n", "trainloader = torch.utils.data.DataLoader(trainset, \n", " batch_size=batch_size, \n", " shuffle=True)\n", "testloader = torch.utils.data.DataLoader(testset, \n", " batch_size=batch_size, \n", " shuffle=False)\n", "classes = (0,1,2,3,4,5,6,7,8,9)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pytorch makes it very simple to define networks, you need to create a Net() class that inherits from the nn module. You then need to define your architecture in the __init__ method, and a forward pass of your network. In this case, we are constructing a very simple neural network with a single hidden layer. We are dealing with MNIST, which are $28 \\times 28$ arrays (images), and so we have $28 \\times 28 = 784$ features. Therefore, the first layer (fully connected layer fc1) needs to have input dimension equal to 784. The output dimension can be anything, and I chose 100 arbitrarily. The second fully connected layer needs to have the same input dimension as the previous output dimension. Finally, since we have 10 classes, the output dimension of the last fully connected layer, in this case fc2, must be 10.\n", "\n", "The forward function describes what we want the network to do, and we are simply implementing the function \n", "\n", "$$\n", "y = f_2(f_1(x)),\n", "$$\n", "\n", "where $f_i(x) = \\sigma(W_{i-1}x+b_{i-1})$, and where we have chosen the ReLU activation, $\\sigma(x) = \\max\\{0,x\\}$, one of the most popular choices in the current literature. \n", "\n", "Next, we define a loss function (criterion), in this case we use the CrossEntropy Loss, which is appropriate for multi-class classification, and finally we define an optimizer, in this case we just use stochastic gradient descent. The rest of the code explains how to call the loader object to get some data, compute the network output, compute the gradients, and then update the weights." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torch.optim as optim\n", "\n", "class Net(nn.Module):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", " self.fc1 = nn.Linear(784, 100)\n", " self.fc2 = nn.Linear(100, 10) \n", "\n", " def forward(self, x):\n", " x = x.view(-1, 784)\n", " x = F.relu(self.fc1(x))\n", " x = self.fc2(x)\n", " return x\n", "\n", "net = Net()\n", "criterion = nn.CrossEntropyLoss()\n", "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)\n", "\n", "for epoch in range(5): \n", " running_loss = 0.0\n", " for i, data in enumerate(trainloader, 0):\n", " # get the inputs; data is a list of [inputs, labels]\n", " inputs, labels = data\n", "\n", " # zero the parameter gradients\n", " optimizer.zero_grad()\n", "\n", " # forward + backward + optimize\n", " outputs = net(inputs)\n", " loss = criterion(outputs, labels)\n", " loss.backward()\n", " optimizer.step()\n", "\n", " # print statistics\n", " running_loss += loss.item()\n", " if i % 2000 == 1999: # print every 2000 mini-batches\n", " print('[%d, %5d] loss: %.3f' %\n", " (epoch + 1, i + 1, running_loss / 2000))\n", " running_loss = 0.0\n", "\n", "print('Finished Training')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have trained our network, let's check out its test accuracy:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "correct = 0\n", "total = 0\n", "with torch.no_grad():\n", " for data in testloader:\n", " images, labels = data\n", " outputs = net(images)\n", " _, predicted = torch.max(outputs.data, 1)\n", " total += labels.size(0)\n", " correct += (predicted == labels).sum().item()\n", "\n", "print('Accuracy of the network on the 10000 test images: %d %%' % (\n", " 100 * correct / total))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As many of you know, it is more appropriate to use convolutional neural networks for image data, and the following shows it is fairly straightforward to extend the above to the CNN setting. For those interested, you can read more here: https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "class ConvNet(nn.Module):\n", " def __init__(self):\n", " super(ConvNet, self).__init__()\n", " # 1 input image channel, 6 output channels, 3x3 square convolution\n", " self.conv1 = nn.Conv2d(1, 10, kernel_size=5)\n", " self.conv2 = nn.Conv2d(10, 20, kernel_size=5)\n", " self.conv2_drop = nn.Dropout2d()\n", " self.fc1 = nn.Linear(320, 60)\n", " self.fc2 = nn.Linear(60, 10)\n", "\n", " def forward(self, x):\n", " x = F.relu(F.max_pool2d(self.conv1(x), 2))\n", " x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))\n", " x = x.view(-1, 320)\n", " x = F.relu(self.fc1(x))\n", " x = F.dropout(x, training=self.training)\n", " x = self.fc2(x)\n", " return F.log_softmax(x, dim=1)\n", "\n", "net = ConvNet()\n", "criterion = nn.CrossEntropyLoss()\n", "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)\n", "\n", "for epoch in range(5):\n", "\n", " running_loss = 0.0\n", " for i, data in enumerate(trainloader, 0):\n", " inputs, labels = data\n", "\n", " # zero the parameter gradients\n", " optimizer.zero_grad()\n", "\n", " # forward + backward + optimize\n", " outputs = net(inputs)\n", " loss = criterion(outputs, labels)\n", " loss.backward()\n", " optimizer.step()\n", "\n", " # print statistics\n", " running_loss += loss.item()\n", " if i % 2000 == 1999: # print every 2000 mini-batches\n", " print('[%d, %5d] loss: %.3f' %\n", " (epoch + 1, i + 1, running_loss / 2000))\n", " running_loss = 0.0\n", "\n", "print('Finished Training')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "correct = 0\n", "total = 0\n", "with torch.no_grad():\n", " for data in testloader:\n", " images, labels = data\n", " outputs = net(images)\n", " _, predicted = torch.max(outputs.data, 1)\n", " total += labels.size(0)\n", " correct += (predicted == labels).sum().item()\n", "\n", "print('Accuracy of the network on the 10000 test images: %d %%' % (\n", " 100 * correct / total))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that the CNN achieves performance similar to the standard NN, and this is almost definitely due to requiring more training. The code in this section has been instructional, and you should try to tweak the parameters/network architectures to get better results." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Advanced Exercise\n", "Find a recent paper (NeuRIPS, ICML, ICLR, etc..) that introduces a novel deep learning architecture and implement it in pytorch" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### MLP from scratch\n", "Finally, we will now take a look at a MLP code example built from scratch. We will no longer rely on pytorch to build models (although we will use the same mnist data installed from torch)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.preprocessing import OneHotEncoder\n", "\n", "# we will work with a small random subset of 10000 images\n", "np.random.seed(123)\n", "idxs = np.random.choice(range(60000), size=10000)\n", "\n", "Xtrain = trainset.data.numpy()[idxs]\n", "Xtrain = Xtrain.reshape(Xtrain.shape[0], -1)\n", "ytrain = trainset.targets.numpy()[idxs]\n", "labels = np.unique(ytrain)\n", "\n", "# one hot encoded version of y, called Y\n", "Ytrain = OneHotEncoder(sparse=False).fit_transform(ytrain.reshape(-1,1))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# for example, we can look at an image of one of the MNIST figures by doing:\n", "ii = 23\n", "plt.imshow(Xtrain[ii].reshape(28,28), cmap=\"gray\")\n", "plt.title(f\"label: {ytrain[ii]}\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true } }, "source": [ "## Implementing a Multi-layer Perceptron from Scratch" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "class Mlp():\n", " # reference: Based on the Mlp example code by Raymundo Cassani\n", " # https://github.com/rcassani/mlp-example\n", " def __init__(self, size_layers):\n", " '''\n", " Arguments:\n", " size_layers : List with the number of Units for:\n", " [Input, Hidden1, Hidden2, ... HiddenN, Output] Layers.\n", " '''\n", " self.size_layers = size_layers\n", " self.n_layers = len(size_layers)\n", " \n", " # initialize weights (in a smart way)\n", " self.initialize_weights()\n", "\n", " def train(self, X, Y, iterations=100):\n", " '''\n", " Given X (feature matrix) and y (class vector)\n", " Updates the Weights by running Backpropagation N tines\n", " Arguments:\n", " X : Feature matrix [n_examples, n_features]\n", " Y : Sparse class matrix [n_examples, classes]\n", " iterations : Number of times Backpropagation is performed\n", " default = 100\n", " '''\n", " n_examples = Y.shape[0]\n", "\n", " for iteration in range(iterations):\n", " self.gradients = self.backpropagation(X, Y)\n", " self.gradients_vector = self.unroll_weights(self.gradients)\n", " self.weight_vector = self.unroll_weights(self.weights)\n", " self.weight_vector = self.weight_vector - self.gradients_vector\n", " self.weights = self.roll_weights(self.weight_vector)\n", "\n", " def predict(self, X):\n", " '''\n", " Given X (feature matrix), y_hat is computed\n", " Arguments:\n", " X : Feature matrix [n_examples, n_features]\n", " Output:\n", " y_hat : Computed Vector Class for X\n", " '''\n", " A , Z = self.feedforward(X)\n", " Y_hat = A[-1]\n", " return Y_hat\n", " \n", " def initialize_weights(self):\n", " '''\n", " Initialize weights - initialization method depends\n", " on the Number of Units in the current layer\n", " and the next layer. The weights for each layer as of the \n", " size [next_layer, current_layer + 1]\n", " '''\n", " self.weights = []\n", " size_next_layers = self.size_layers.copy()\n", " size_next_layers.pop(0)\n", " for size_layer, size_next_layer in zip(self.size_layers, size_next_layers):\n", " # Method presented \"Understanding the difficulty of training deep feedforward neural networks\"\n", " # Xavier Glorot and Youshua Bengio, 2010\n", " epsilon = 4.0 * np.sqrt(6) / np.sqrt(size_layer + size_next_layer)\n", " # Weigts from a uniform distribution [-epsilon, epsion] \n", " tmp = epsilon * ( (np.random.rand(size_next_layer, size_layer + 1) * 2.0 ) - 1)\n", " self.weights.append(tmp)\n", " return self.weights\n", "\n", " def backpropagation(self, X, Y):\n", "\n", " g_dz = lambda x: self.sigmoid_derivative(x)\n", " n_examples = X.shape[0]\n", " \n", " # Feedforward\n", " A, Z = self.feedforward(X)\n", "\n", " # Backpropagation\n", " deltas = [None] * self.n_layers\n", " deltas[-1] = A[-1] - Y\n", " # For the second last layer to the second one\n", " for ix_layer in np.arange(self.n_layers - 1 - 1 , 0 , -1):\n", " tmp = self.weights[ix_layer]\n", " tmp = np.delete(tmp, np.s_[0], 1)\n", " deltas[ix_layer] = (tmp.T @ deltas[ix_layer + 1].T ).T * g_dz(Z[ix_layer])\n", "\n", " # Compute gradients\n", " gradients = [None] * (self.n_layers - 1)\n", " for ix_layer in range(self.n_layers - 1):\n", " grads_tmp = deltas[ix_layer + 1].T @ A[ix_layer]\n", " gradients[ix_layer] = grads_tmp / n_examples \n", " return gradients\n", "\n", " def feedforward(self, X):\n", "\n", " g = lambda x: self.sigmoid(x)\n", " A = [None] * self.n_layers\n", " Z = [None] * self.n_layers\n", " input_layer = X\n", "\n", " for ix_layer in range(self.n_layers - 1):\n", " n_examples = input_layer.shape[0]\n", " # augment input layer with vector of ones to account for bias\n", " input_layer = np.concatenate((np.ones([n_examples, 1]), input_layer), axis=1)\n", " A[ix_layer] = input_layer\n", " # Multiplying input_layer by weights for this layer\n", " Z[ix_layer + 1] = input_layer @ self.weights[ix_layer].T \n", " output_layer = g(Z[ix_layer + 1])\n", " # Current output_layer will be next input_layer\n", " input_layer = output_layer\n", "\n", " A[self.n_layers - 1] = output_layer\n", " return A, Z\n", "\n", " def unroll_weights(self, rolled_data):\n", " '''\n", " Unroll a list of matrices to a single vector\n", " Each matrix represents the Weights (or Gradients) from one layer to the next\n", " '''\n", " unrolled_array = np.array([])\n", " for one_layer in rolled_data:\n", " unrolled_array = np.concatenate((unrolled_array, one_layer.flatten(\"F\")) )\n", " return unrolled_array\n", "\n", " def roll_weights(self, unrolled_data):\n", " '''\n", " rolls a single vector to a list of matrices\n", " Each matrix represents the Weights (or Gradients) from one layer to the next\n", " '''\n", " size_next_layers = self.size_layers.copy()\n", " size_next_layers.pop(0)\n", " rolled_list = []\n", " extra_item = 1\n", " for size_layer, size_next_layer in zip(self.size_layers, size_next_layers):\n", " n_weights = (size_next_layer * (size_layer + extra_item))\n", " data_tmp = unrolled_data[0 : n_weights]\n", " data_tmp = data_tmp.reshape(size_next_layer, (size_layer + extra_item), order = 'F')\n", " rolled_list.append(data_tmp)\n", " unrolled_data = np.delete(unrolled_data, np.s_[0:n_weights])\n", " return rolled_list\n", "\n", " def sigmoid(self, z):\n", " return 1.0 / (1.0 + np.exp(-z))\n", "\n", " def sigmoid_derivative(self, z):\n", " return self.sigmoid(z) * (1 - self.sigmoid(z))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we can train the MLP on the MNIST dataset, using the same architecture as in the PyTorch case" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "mlp_classifier = Mlp(size_layers = [784, 100, 10])\n", "\n", "# Training with epochs\n", "epochs = 400\n", "loss = np.zeros([epochs,1])\n", "\n", "# train the MLP\n", "for i in range(epochs):\n", " mlp_classifier.train(Xtrain, Ytrain, 1) # 1 iteration of backprop\n", " Y_hat = mlp_classifier.predict(Xtrain)\n", " loss[i] = (0.5) * np.mean((Y_hat - Ytrain)**2)\n", " \n", "# Plot loss vs epochs\n", "plt.figure()\n", "plt.plot(np.arange(epochs), loss)\n", "\n", "# Training Accuracy\n", "MLP_preds = mlp_classifier.predict(Xtrain) # predicted values\n", "class_idxs = np.argmax(MLP_preds, axis=1) # take argmax to find idx of predicted class\n", "y_hat = labels[class_idxs] # find predicted class\n", "\n", "print(\"\\nMLP accuracy: \", accuracy_score(ytrain, y_hat))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " \n", "\n", "#### Extra Exercise:\n", " \n", "Update the MLP class so that it has the following functionality:\n", " - user specified option to include/exclude a bias term\n", " - ability to handle different activation functions beyond the logistic sigmoid\n", " - advanced: explore different approaches to initialising the neural net weights, this can make a big difference to training\n", " - advanced: implement different optimisation schemes" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.9" } }, "nbformat": 4, "nbformat_minor": 4 }