Module 1 Lab: Forward pass and activations

Contents

Module 1 Lab: Forward pass and activations#

Build intuition for how nonlinear activations change an MLP’s decision behavior.

Run the setup cell, inspect the printed diagnostics, and then complete the exercises at the end. The lab is intentionally small enough to run in GitHub Codespaces without a GPU.

import torch
from torch import nn
import matplotlib.pyplot as plt

torch.manual_seed(11)
X = torch.linspace(-3, 3, 120).unsqueeze(1)
y = torch.sin(X) + 0.15 * torch.randn_like(X)

def train_activation(activation):
    model = nn.Sequential(nn.Linear(1, 16), activation, nn.Linear(16, 1))
    opt = torch.optim.Adam(model.parameters(), lr=0.03)
    loss_fn = nn.MSELoss()
    losses = []
    for _ in range(160):
        opt.zero_grad()
        loss = loss_fn(model(X), y)
        loss.backward()
        opt.step()
        losses.append(loss.item())
    return model, losses

relu_model, relu_losses = train_activation(nn.ReLU())
sigmoid_model, sigmoid_losses = train_activation(nn.Sigmoid())
print(f"ReLU final loss: {relu_losses[-1]:.4f}")
print(f"Sigmoid final loss: {sigmoid_losses[-1]:.4f}")

plt.figure(figsize=(5, 3))
plt.plot(relu_losses, label="ReLU")
plt.plot(sigmoid_losses, label="Sigmoid")
plt.xlabel("epoch")
plt.ylabel("MSE")
plt.legend()
plt.close()

ReLU final loss: 0.0225
Sigmoid final loss: 0.0420

Lab exercises#

  1. Change one model or data parameter and rerun the lab.

  2. Record whether the metric improved, worsened, or stayed roughly the same.

  3. Add one sentence connecting the result to From neurons to multilayer networks.

  4. Identify one limitation of this toy setup before applying the idea to a real dataset.

# Reflection workspace
observation = ""
next_experiment = ""
print({"observation": observation, "next_experiment": next_experiment})

{'observation': '', 'next_experiment': ''}
Contents