Introduction to Neural Networks

A neural network is a machine learning model inspired by the human brain. It learns by adjusting the strength of connections between artificial neurons. Start with this: a neural network is layers of math. Input goes in, gets transformed by weights and biases, passes through an activation function, moves to the next layer, and eventually produces an output. That process, repeated with adjustments, lets it learn patterns.

Understanding how they work at a theoretical level before you start building them makes everything clearer. The difference between a network that learns and one that flails around is often not code—it’s understanding the layers, the activation functions, and why they matter.

Introduction to Neural Networks: What is Deep Learning?

Deep learning is a subset of machine learning that uses neural networks to solve complex problems. Instead of you selecting features by hand, the network learns which features matter. Deep learning excels when data is unstructured: images, text, audio. It discovers patterns humans would never think to code.

The deep part means multiple layers. A single layer is too simple for most real problems. Multiple layers let the network learn increasingly abstract features. Early layers might detect edges in an image. Middle layers combine those edges into shapes. Deep layers recognize objects from shapes. That layering is where the power comes from.

Machine learning the traditional way requires you to engineer features: “Here, model, look at height, weight, and age to predict if someone will like a product.” Deep learning is lazier: “Here’s the data, find the patterns yourself.” That freedom costs more compute, but it handles messy data better.

The Biological Neuron (Briefly)

Neural networks get their name from neurons in the brain, but don’t take the analogy too far. A biological neuron receives signals on dendrites, processes them in the nucleus, and fires an output down an axon. An artificial neuron does something similar: it receives weighted inputs, sums them, applies an activation function, and outputs a value to the next layer.

That’s the entire concept. Everything else is just stacking these simple units and tuning their connections.

The Artificial Neuron

An artificial neuron takes inputs, multiplies each by a weight, adds a bias, and applies a function:

output = activation(sum(weights * inputs) + bias)

The weights and bias are learnable parameters. The activation function introduces nonlinearity. Without it, stacking layers would just be repeated matrix multiplication—you’d end up with a single, giant matrix. Activation functions let the network learn curves and boundaries, not just straight lines.

Forward Pass: How Data Flows

Let’s trace data through a tiny network with two layers:

import numpy as np

# Input: a single data point with 2 features
X = np.array([0.5, 0.3])

# Layer 1: 2 inputs -> 2 hidden neurons
W1 = np.array([[0.1, 0.4], [0.2, 0.5]])  # Weights
b1 = np.array([0.1, 0.2])                # Biases

# Linear transformation
z1 = X @ W1 + b1
print(f"After linear transform: {z1}")
# Output: [0.21 0.55]

# Apply ReLU activation
a1 = np.maximum(0, z1)
print(f"After ReLU: {a1}")
# Output: [0.21 0.55]

# Layer 2: 2 hidden neurons -> 2 output neurons
W2 = np.array([[0.3, 0.6], [0.4, 0.7]])
b2 = np.array([0.05, 0.1])

z2 = a1 @ W2 + b2
print(f"After second linear transform: {z2}")
# Output: [0.333 0.611]

# Apply sigmoid activation (for classification)
output = 1 / (1 + np.exp(-z2))
print(f"After sigmoid: {np.round(output, 3)}")
# Output: [0.582 0.648]

That’s a forward pass. Data flows left to right through transformations. The weights and biases change during training to minimize error. The activation function keeps things nonlinear so the network can learn complex relationships.

Activation Functions

Activation functions are what give neural networks their power. Without them, a deep network would collapse into a single linear function—layers would just be matrix multiplications, which combine into one giant matrix.

An activation function is applied to each neuron’s output before passing to the next layer. It introduces nonlinearity so the network can model curves, not just lines. Scikit-learn’s multi-layer perceptron docs cover the same set: hyperbolic tangent by default in hidden layers, logistic for binary output, softmax for multi-class output.

Common Activation Functions

ReLU (Rectified Linear Unit)

f(x) = max(0, x)

Simplest and most common in hidden layers. It’s zero for negative inputs, linear for positive. The name is literal: “rectify” means make straight, and this makes all negative values zero.

x_vals = np.array([-2, -1, 0, 1, 2])
relu = np.maximum(0, x_vals)
print(f"ReLU({x_vals}): {relu}")
# Output: [0 0 0 1 2]

ReLU is fast to compute and keeps training stable. It dominates modern deep learning. Drawback: “dying ReLU” problem—if many neurons output zero during training, they might never recover.

Sigmoid

f(x) = 1 / (1 + e^(-x))

Maps any input to a value between 0 and 1. Historically the standard. Useful for output layers in binary classification (predicting true/false).

sigmoid_vals = 1 / (1 + np.exp(-x_vals))
print(f"Sigmoid({x_vals}): {np.round(sigmoid_vals, 3)}")
# Output: [0.119 0.269 0.5 0.731 0.881]

Downside: gradients near the extremes are tiny, so training slows when a neuron’s output is 0 or 1.

Tanh (Hyperbolic Tangent)

f(x) = (e^x - e^(-x)) / (e^x + e^(-x))

Similar to sigmoid but outputs between -1 and 1. Centered at zero, which sometimes helps training. Used in hidden layers and output layers for regression.

tanh_vals = np.tanh(x_vals)
print(f"Tanh({x_vals}): {np.round(tanh_vals, 3)}")
# Output: [-0.964 -0.762 0. 0.762 0.964]

Stronger gradients than sigmoid (steeper slope), so training is faster. But still suffers from saturation.

Choosing an Activation Function

Hidden layers: Use ReLU. It’s fast, works, and is the industry standard. If your model trains poorly, try Leaky ReLU (allows small negative values) or ELU before changing the architecture.

Output layer: It depends on your task.

• Binary classification (true/false): sigmoid.
• Multi-class classification (pick one of N): softmax.
• Regression (predict a number): linear (no activation) or ReLU if the output must be positive.

The activation function is your lever to tune how the network transforms data. Change it and you change what the network can learn.

Layers: Stacking the Model

A neural network is a chain of transformations. Each layer has a linear part (weights and biases) and a nonlinear part (activation). Stacking multiple layers lets the network learn hierarchical features.

Early layers learn simple patterns. Later layers combine those into complex patterns. This hierarchy is why it’s called “deep” learning.

Common architectures:

• Shallow network: 1-2 hidden layers. Works for simple problems.
• Deep network: 3+ hidden layers. Handles complex data. Takes longer to train but can learn finer details.

More layers add power but also risk overfitting. The network memorizes the training data instead of learning generalizable patterns. This is why regularization and dropout exist—they’re constraints to keep the network from memorizing.

Backpropagation: How Networks Learn

The forward pass shows how data flows through the network. But how does the network change its weights to get better? That’s backpropagation.

Training a neural network means adjusting weights to minimize error. You calculate the loss (how wrong the prediction was), then propagate that error backward through the network, computing gradients for each weight. Gradients tell you which direction to nudge each weight to reduce error. This happens millions of times on billions of data points, and the network converges to a solution.

The math is dense, but the idea is simple: measure how wrong you are, figure out which weights caused that wrongness, nudge those weights in the right direction, and repeat. Activation functions matter here too—their derivatives affect how quickly errors flow backward.

Vanishing and Exploding Gradients

When networks get deep, a problem emerges. Gradients computed during backpropagation get multiplied together as they flow backward through layers. With sigmoid or tanh activations, gradients near the extremes are tiny (close to zero). Multiply many tiny gradients and they vanish—the weights deep in the network barely change during training. That’s the vanishing gradient problem.

The opposite can happen too. If gradients are large and you multiply them together, they explode. Weights blow up, training destabilizes.

This is one reason ReLU dominates modern deep learning. Its gradient is either 0 or 1 (the slope of the line), so it’s stable. It doesn’t compress gradients like sigmoid. Techniques like batch normalization and gradient clipping also help.

Why Architecture Matters

The choice of how many layers and how many neurons per layer shapes what the network can learn. Too few layers and it’s too simple—it underfits, missing patterns. Too many layers and it overfits, memorizing noise in the data. See overfitting vs underfitting for deeper insight into this tradeoff.

Width matters too. A wide layer has more neurons to learn features. A narrow layer acts as a bottleneck, forcing the network to compress information. Both strategies are used:

• Wide networks: can learn complex functions with fewer layers.
• Deep networks: learn hierarchical features by stacking many thin layers.

Modern practice leans deep. Depth adds expressiveness more efficiently than width.

Why Neural Networks Matter

Neural networks broke open image recognition, natural language processing, and game-playing AI. They work because:

1. Flexibility. Change the architecture, activation functions, or loss function and they adapt to new problems. A network trained for image classification can be retrained for image segmentation with minimal changes.
2. Scale. Feed them more data and more compute, and they keep improving. This scaling property is why LLMs are possible.
3. Automation. They automate feature engineering. You don’t hand-pick features. The network finds them. That’s powerful when you don’t know what features matter.

The tradeoff: they’re a black box. You train them, they work, but you can’t always explain why. That’s fine for many applications but risky for high-stakes decisions like healthcare or criminal justice.

From Theory to Practice

Understanding the theory prevents common mistakes. You know why ReLU works. You know why deep networks can learn hierarchical features. You know why activation functions are nonlinear—without them, stacking layers does nothing.

Next steps to build real intuition:

• Implement forward and backward passes by hand with numpy. It’s slow but it clicks.
• Build a tiny network from scratch (2 layers, 3 neurons each). Run it on toy data. Watch weights change during training.
• Use TensorFlow to create and train tensors. See how frameworks abstract the math while letting you control the details. After that, train a real network on house prices with Keras in Python.
• Then worry about optimization algorithms, regularization, and advanced architectures.

The concepts here—layers, neurons, weights, biases, activation functions, backpropagation—are the atoms of all deep learning. Transformers, CNNs, RNNs, everything is built from these fundamentals.

What part of neural networks feels most mysterious to you: how weights are initialized, why training sometimes fails, or how the network actually learns?