Module 3 Assignment: Optimizer and regularization comparison

Module 3 Assignment: Optimizer and regularization comparison#

Theme#

Optimization, loss, and regularization

Scenario#

A modeling team has a prototype that trains inconsistently and needs a defensible training configuration before scaling experiments.

Exercises#

  1. Train the same small model with SGD and Adam on a synthetic dataset.

  2. Compare at least two regularization settings such as weight decay or dropout.

  3. Summarize loss curves, final accuracy, and run-to-run sensitivity.

  4. Recommend the next training gate and justify the tradeoff.

Evidence Requirements#

  • A short explanation of the data, tensors, objective, and evaluation signal used in the starter experiment.

  • At least one meaningful modification to the starter code, with the changed variable named explicitly.

  • A comparison against the unmodified starter result or another defensible baseline.

  • A limitation statement that separates what the toy experiment demonstrates from what a production model would require.

Submission#

Submit a 600-900 word technical memo plus code, plots, tables, or shape traces needed to support your claims. The memo should read like a review artifact for another AI practitioner: concise, reproducible, and honest about uncertainty.

Rubric Focus#

  • Technical correctness and appropriate neural-network vocabulary.

  • Evidence from the starter experiment or a documented extension.

  • Connection between design choices and data/problem structure.

  • Clear treatment of limitations, failure modes, or next experimental gates.

import torch
from torch import nn

torch.manual_seed(3)
X = torch.randn(160, 6)
y = ((X[:, :3].sum(dim=1) + 0.35 * torch.randn(160)) > 0).long()

def train(optimizer_name="Adam", weight_decay=0.0):
    model = nn.Sequential(nn.Linear(6, 16), nn.ReLU(), nn.Linear(16, 2))
    opt_cls = torch.optim.Adam if optimizer_name == "Adam" else torch.optim.SGD
    opt = opt_cls(model.parameters(), lr=0.04, weight_decay=weight_decay)
    loss_fn = nn.CrossEntropyLoss()
    losses = []
    for _ in range(90):
        opt.zero_grad()
        loss = loss_fn(model(X), y)
        loss.backward()
        opt.step()
        losses.append(loss.item())
    return losses[-1]

for name in ["SGD", "Adam"]:
    print(name, train(name, weight_decay=1e-3))

SGD 0.38726890087127686
Adam 0.010066337883472443

Reflection Prompts#

  • What changed when you modified the starter experiment, and why should that change matter?

  • Which result surprised you, and what diagnostic would you run next?

  • What assumption would you document before handing this model to another practitioner?

  • Which failure mode from the module reading is most relevant to your result?