Measuring how wrong

Before the network can improve, we need a single number that says how bad its guesses are — a loss. Training is nothing but making that number smaller. For a yes/no problem with probability outputs, the right choice is binary cross-entropy.

The idea

For each point, the network outputs a probability p that the label is 1. Cross-entropy rewards confident, correct probabilities and punishes confident, wrong ones — harshly:

• True label 1, network says 0.99 → tiny loss.
• True label 1, network says 0.01 → huge loss.

In code

A one-liner. The tiny eps keeps log(0) from blowing up when a prediction is exactly 0 or

1. Add to net.py:

def bce(a2, y):
  eps = 1e-8
  return -np.mean(y * np.log(a2 + eps) + (1 - y) * np.log(1 - a2 + eps))

Check the loss of the still-untrained network:

_, _, a2 = forward(X)
print("starting loss:", round(float(bce(a2, y)), 4))

starting loss: 0.6947

That 0.69 is the telltale value of a coin-flipping binary classifier — it's −log(0.5). Our whole job now is to drive it toward zero. To do that, we need to know which way to nudge each weight — and that means computing the gradient of this loss. Time for the main event: backpropagation.