Backpropagation for Fully-Connected Neural Networks

Forward Pass

In the following, superscripts represent the layer associated with each variable:

• : Input data having dimension n-by-f, where n is the number of samples and f the number of features. For a batch of 32 MNIST samples, would have dimension (32, 784).

• : Target variable. classifying a single digit from MINST, a vector populated with 0s and 1s indicating the ground truth label for the sample (8 or not 8). Has the same length as the first dimension of .

• : Trainable weights. Projects previous layer activations to lower dimensional representation. Again referring to the first set of weights for a batch of 32 MNIST samples, ’s first dimension will match the second dimension of the activations from the previous layer (784), and ’s second dimension will be some lower dimension, say 256. will therefore have dimension (784, 256).

• : Bias term, a one-dimensional vector associated with each hidden layer having length equal to the second dimension of the hidden layer. will have dimension (256,).

• : Output of layer , which is the matrix product of the previous layer activations and current layer weights (plus bias term).

• : Activations associated with layer . Passes through a non-linearity such as sigmoid or ReLU.

More concretely, assume a 2-layer fully-connected neural network with one hidden layer of size 256, through which a dataset of dimension 32-by-784 is passed to predict whether each of the 32 images is an 8 or not. The forward pass looks like:

• Randomly initialize (784×256), (256×1), (256×1) and (1×1)
• (32×784)
• (32×256)
• (32×256)
• (32×1)
• (32×1)

The final output, , represents the probability that each sample is the number 8 or not.

With the actual labels and our predicted probabilities , we can define our loss function, the cross-entropy loss for binary classification:

Backward Pass (Backpropagation)

The goal of backpropagation is to compute the partial derivatives of the loss function with respect to any weight or in the network. In order to update our weights, we need to take derivatives of w.r.t. and , then update and using the derivatives. Backpropagation starts by taking the derivative of the loss function. We first compute the derivatives of the loss function w.r.t. and . Here we make use of the chain rule:

Once we have and , and are updated as follows:

for some learning rate . This holds for all layers. For given layer , the update rule for and is:

Let’s start with unpacking . The first entry on the r.h.s., , represents the derivative of the loss function w.r.t. , which is

The second term on the r.h.s., , is the derivative of the sigmoid activation (). The derivative of the sigmoid function is given by

therefore is given by

For the third term on the r.h.s., , recall that . Therefore

Finally, we have

As a notational convenience, we define :

This way, can be expressed as

We proceed in a similar fashion for :

since .

For the first layer we re-use many of these calculations, but for new terms on the r.h.s., we employ the chain rule in the same way. For reference, restate the terms from the forward pass:

We next consider :

Considering each term on the r.h.s:

Resulting in:

As before, we define as

which allows us to write as

Similarly for :

Considering each term on the r.h.s:

Therefore

To complete the backpropagation algorithm, it is necessary to define :

Assume is a 32×784 batch of MNIST images, and our network has one hidden layer of size 256. Our task is to identify which digit 0-9 a sample most closely resembles. We first declare a number of functions, then implement the forward and backward passes along with weights update.

import numpy as np


def sigmoid(X):
    """
    Compute the sigmoid activation for the input.
    """
    return 1 / (1 + np.exp(-X))


def sigmoid_dev(X):
    """
    The analytical derivative of sigmoid function at X.
    """
    return sigmoid(X) * (1 - sigmoid(X))


def softmax(scores):
    """
    Compute softmax scores given the raw output from the model.
    Returns softmax probabilities (N, num_classes).
    """
    numer = np.exp(scores - scores.max(axis=1, keepdims=True))
    denom = numer.sum(axis=1, keepdims=True)
    return np.divide(numer, denom)


def cross_entropy_loss(ypred, yactual):
    """
    Compute Cross-Entropy Loss based on prediction of the network and labels
    """
    yactual = np.asarray(yactual)
    ypred = ypred[np.arange(len(yactual)), yactual]
    return -np.mean(np.log(ypred))


def compute_accuracy(ypred, yactual):
    """
    Compute the accuracy of current batch.
    """
    yactual = np.asarray(yactual)
    yhat = np.argmax(ypred, axis=1)
    return (y == yhat).sum() / y.shape[0]

# Stand in for batch of 32 MNIST images. 
X = np.random.randint(0, 256, size=(32, 784))
y = np.random.randint(0, 10, size=32)

# Reshape labels to 32 x 10. 
Y = np.zeros((32, 10))
Y[np.arange(X.shape[0]), y] = 1  # (32, 10)

# Learning rate.
alpha = .05 

# Initialize weights.
b1 = np.zeros(256)
b2 = np.zeros(10)
W1 = 0.001 * np.random.randn(784, 256)
W2 = 0.001 * np.random.randn(256, 10)

# Forward pass.
Z1 = X @ W1 + b1   # (32, 256)
A1 = sigmoid(Z1)   # (32, 256)
Z2 = A1 @ W2 + b2  # (32, 10)
A2 = softmax(Z2)   # (32, 10)

# Compute loss and accuracy.
loss = cross_entropy_loss(A2, y)
accuracy = compute_accuracy(A2, y)

# Backward pass.
dZ2 = A2 - Y                            # (32, 10)
dW2 = (A1.T @ dZ2) / 32                 # (256, 10)
db2 = np.sum(dZ2, axis=0) / 32          # (10,)
dA1 = dZ2 @ W2.T                        # (32, 256)
dZ1 = np.multiply(dA1, sigmoid_dev(Z1)) # (32, 256)
dW1 = (X.T @ dZ1) / 32                  # (784, 256)
db1 = np.sum(dZ1, axis=0) / 32          # (256,)

# Update weights.
W2 = W2 - alpha * dW2
b2 = b2 - alpha * db2
W1 = W1 - alpha * dW1
b1 = b1 - alpha * db1

The code starting with the forward pass would be iterated over a set of batches for a pre-determined number of epochs. The final weights would then be used for inference.