The forward pass

A neural network is just a chain of matrix multiplies with a squishing function between them. Ours has one hidden layer: two inputs → sixteen hidden units → one output. That hidden layer is what lets it bend a curve instead of drawing a line.

The weights

Each layer is a weight matrix and a bias vector. We start them at small random values — W1 maps 2 inputs to 16 hidden units, W2 maps those 16 down to 1 output. The sqrt(2 / fan_in) scaling (He initialization) keeps the signal from blowing up or vanishing through the ReLU. Add to net.py:

def he(shape, fan_in):
  return rng.standard_normal(shape) * np.sqrt(2.0 / fan_in)

H = 16                          # hidden units
W1 = he((2, H), 2);  b1 = np.zeros((1, H))
W2 = he((H, 1), H);  b2 = np.zeros((1, 1))

Two activation functions

Between the matrix multiplies we need non-linearity — without it, two stacked linear layers collapse into one, and we're back to a straight line. We use two:

• ReLU on the hidden layer: max(0, z). Simple and fast; it's what the PyTorch course used too.
• Sigmoid on the output: squashes any number into (0, 1) so we can read it as the probability of class 1.

def sigmoid(z):
  return 1.0 / (1.0 + np.exp(-z))

def forward(X):
  z1 = X @ W1 + b1        # (N,2) @ (2,16) -> (N,16)
  a1 = np.maximum(0, z1)  # ReLU hidden activation
  z2 = a1 @ W2 + b2       # (N,16) @ (16,1) -> (N,1)
  a2 = sigmoid(z2)        # probability of class 1
  return z1, a1, a2

We return the intermediate values z1 and a1, not just the final a2 — backpropagation in Chapter 4 will need them. Every step is a shape you can check: two inputs fan out to sixteen, sixteen collapse to one.

Predict — badly

Run a forward pass on the whole dataset. The weights are random, so the network has learned nothing yet:

_, _, a2 = forward(X)
print("mean predicted probability:", round(float(a2.mean()), 3))

mean predicted probability: 0.458

Right around 0.5 — the network is essentially flipping a coin, which is exactly what an untrained network should do. To improve, it first needs to know how wrong it is. That's the next chapter.