A Gentle Introduction to Deep Learning - AlexNet

published: 8/2/2025 and updated: 8/2/2025

Deep Learning Deep Learning , CNN , AlexNet

AlexNet#

AlexNet is a classic convolutional neural network proposed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton in the 2012 ImageNet image classification competition. The paper can be found here.

The architecture of this network is not complex and includes several fundamental layers in deep learning networks—input layer, convolutional layer, pooling layer, activation layer, and fully connected layer. Therefore, it is very suitable as an entry-level architecture for understanding some important concepts.

The AlexNet architecture is shown in the following diagram:

Readers might be intimidated by these strange concepts, numbers, and structures, but after reading this section, you will naturally understand them.

Input Layer#

The input layer is the starting point of image classification networks. Here, images are separated into RGB three channels and fed into the network. This layer also typically performs resize operations to ensure all inputs have the same size.

In the above diagram, we have a 227×227 image that enters AlexNet as a 227×227×3 tensor after input.

Convolutional Layer#

The convolutional layer is the most important part of deep learning networks, used to extract features from images. In a single convolutional layer, there are usually many kernels of the same size, called convolutional kernels. For example, AlexNet’s first Conv layer contains 96 kernels of size 11×11×3. Note that the width and height of kernels are usually the same, with depth matching the number of channels.

So how does a single convolutional kernel extract features? Let’s take a grayscale image as an example, where the kernel size is 3×3. It will slide across the image in a Z-pattern to extract all possible features. The convolutional kernel multiplies each pixel value on the kernel with the source pixel and sums them up, with the sum becoming the value of the kernel’s center pixel.

Readers can easily see that for an image, each convolution operation shrinks it by one circle, so zero-padding operations are needed around the edges to ensure consistent image size.

The process for three-channel color images is basically the same and won’t be elaborated here.

In actual networks, there will be multiple convolutional layers to better extract image features.

Important Parameters#

Stride#

Here we need to introduce the concept of stride, which takes effect in both height and width dimensions, controlling the length of each slide. When we say stride=2, it means the stride is 2 in both dimensions.

Padding#

The padding parameter controls zero-padding operations, generally having two values: valid and same:

• valid: No zero-padding operation is performed; each convolution reduces both dimensions by F-1, where F is the kernel size
• same: Zero-padding or data replication is performed around the input image pixels to ensure the same size before and after convolution

AlexNet has a total of 5 convolutional layers.

Activation Layer#

Let’s first introduce the concept of loss function.

The loss L value is usually calculated by the following formula:

Therefore, the training process is about finding a function that minimizes the loss as much as possible, so that predicted values will approximate actual values as closely as possible.

The activation layer performs activation operations through activation functions, adding nonlinear elements to the model to ensure continuous gradient descent.

Now ReLU Nonlinearity is commonly used for activation, but in the era when AlexNet was released, Sigmoid and tanh were commonly used. The latter two have the problem of vanishing gradient, which is not suitable to expand here. Interested readers can refer to this.

Pooling Layer#

The pooling layer is used in CNN networks to reduce the spatial dimensions of feature maps while retaining the most important features, aiming to reduce the overall network parameters and computational load while maintaining accuracy.

The pooling process is essentially a downsampling process in two dimensions and does not change the depth of the image. It’s equivalent to using a manually defined convolutional kernel sliding over the image, but this kernel contains no parameters.

There are usually two methods: max pooling and average pooling:

• Max Pooling: Takes the maximum value within the pooling window
• Average Pooling: Takes the average value within the pooling window

Fully Connected Layer#

The fully connected layer flattens multi-layer features into a one-dimensional vector, takes weighted sums of various categories from each layer, and outputs final scores for each category.

In AlexNet, the role of fully connected layers is:

1. Feature Integration: Integrating local features extracted by convolution and pooling into global features
2. Classification Mapping: Mapping features to specific classification results
3. Dimensionality Reduction: Implementing feature dimensionality reduction and filtering through weight matrices

Summary#

We have currently discussed some important concepts involved in AlexNet, which are also fundamental concepts in deep learning. In the next section, we will implement AlexNet from a code perspective and train it to demonstrate its practical application process, while also evaluating the network’s performance.

By understanding AlexNet’s architecture, readers should have mastered:

• Basic components of convolutional neural networks
• The role and parameter settings of each layer
• Basic flow of forward propagation

This knowledge will lay a solid foundation for subsequent learning of more complex network architectures.