Introduction

Neural networks are one of the most popular machine learning algorithms. They have been successfully used in many areas like image classification and time series forecasting, making them valuable in both business and research. Understanding how neural networks work is crucial because it helps diagnose issues and lays the groundwork for learning more advanced deep learning algorithms.

In this article, we will break down how a neural network works. We will go through the algorithm step-by-step and show you how to set up a simple neural network in PyTorch.

Combination of Functions

Let's start with some basic concepts. Imagine two simple linear functions:

Here, the coefficients (-1.75, -0.1, 0.172, 0.15) are chosen just for illustration. Now, we define two new functions, a1​ and a2​, using a special function called the sigmoid function:

The sigmoid function, σ(x) creates an S-shaped curve and plays a key role in neural networks.

Next, we define another function that combines a1​ and a2 linearly:

   z3​=0.25a1​+0.5a2​+0.2

Again, the coefficients (0.25, 0.5, 0.2) are arbitrarily chosen. By combining these functions, we create more complex functions capable of capturing intricate patterns. For example, the combined function z3 looks more complex than a1 or a2​.

If we apply the sigmoid function to z3 and combine it with another similar function, we can create even more complex patterns. By adjusting the coefficients, we can fit our final function to complex data sets. This is the basic idea behind neural networks: combining simpler functions to represent complicated variations in data.
Now, let's delve into the framework of a neural network and see how it all comes together.

Forward Pass

Now, let's introduce the concept of a forward pass in a neural network. The forward pass is the process of moving input data through the network to get an output. This involves multiple layers, each performing specific calculations on the data and passing the results to the next layer.

Here is an example:

As shown in the diagram, we start with our training data X. This data is fed into the first layer of our neural network. Initially, this layer uses randomly assigned weights and biases. These weights and biases are the parameters that our model will learn and optimize during the training process.

The output from this layer is then passed through an activation function. The activation function introduces non-linearity into the model, allowing it to learn and represent complex patterns in the data. Common activation functions include the sigmoid, ReLU, and tanh functions.

After applying the activation function, we get a set of predictions from the neural network. These predictions are then compared to the true values (the actual labels or outcomes we want to predict) to calculate the loss. The loss is a measure of how far off our predictions are from the true values. A common loss function for classification tasks is cross-entropy loss, while mean squared error is often used for regression tasks.

The optimizer then comes into play. The optimizer uses the calculated loss to adjust the weights and biases in the neural network, with the goal of reducing the loss in future iterations. Popular optimization algorithms include gradient descent, Adam, and RMSprop. The optimizer updates the weights by computing the gradient of the loss function with respect to each weight and bias, and then making adjustments in the direction that minimizes the loss.

This process of passing data through the layers, applying activation functions, calculating loss, and updating weights is known as the forward pass. It is repeated for many iterations, or epochs, allowing the neural network to learn and improve its performance over time.

By continually optimizing the weights and biases, the neural network becomes better at making accurate predictions. This iterative process of training helps the model generalize well to new, unseen data

How to update weights?:

To understand how weights are updated in a neural network, let's revisit what we got from our forward pass. We spoke about the loss calculation or error, which measures the difference between our predicted values and true values. This difference is plotted along the X-axis, while the Y-axis corresponds to the actual loss. Because we're squaring the differences, the loss function curve takes on a U-shape.

The objective of our optimizer is to minimize this loss. Ideally, the goal is to bring this loss function value as close to zero as possible. In the real world, the loss function values won't be exactly zero, but we will aim for values close to it, like L1 or L2, as shown in the diagram. If we get a loss value of L1, we need to increase the difference to reach the minimum value. Conversely, if we get a loss value of L2, we need to decrease the difference.

To achieve this, we need to delve into a bit of mathematics. We can place a tangent (shown as a red line) on the curve of the loss function. This tangent represents the gradient, which is the derivative of the loss function. This process is called gradient descent. We use these gradients to update the weights. Regardless of whether we're at L1 or L2, the gradient will guide us in the right direction to minimize the loss.

If you are far from the optimal minimum point, your gradient will be larger, indicating a larger error. Larger errors result in larger absolute gradients. This is where the concept of "Learning Rate" into play.

Let's consider the scenario where our error is at L1. If we do not have a learning rate, it is possible that we might overshoot the minimum point and jump to the other side, reaching L2, and then oscillate between L1 and L2. This means we might never reach the minimum. To handle this problem, we introduce something called the learning rate.

   Wnew = W - (dL/dw . LR)

The learning rate controls how much we adjust the weights with respect to the gradient. Instead of subtracting the entire gradient from the current weight, we multiply the gradient by the learning rate and then subtract this product from the weights. Typical learning rates are values like 0.01, 0.001, etc

Choosing the right learning rate is crucial. A learning rate that is too high can cause the model to converge too quickly to a suboptimal solution, or even diverge, while a learning rate that is too low can make the training process very slow.

In summary, the process of updating weights involves:

1. Calculating the gradient of the loss function with respect to the weights.

2. Multiplying this gradient by the learning rate

3. Subtracting the result from the current weights to get the updated weights

This process ensures that we make small, controlled steps towards minimizing the loss, ultimately leading to a well-trained neural network.

Chain Rule

Implementing the Backward pass

Understanding the DOT PRODUCT

Conclusion