Semester course: Principles of AI Alignment and Safety

Week 1: PyTorch Fundamentals & Practical Deep Learning Skills Core concepts review (neural networks, linear algebra, probability, information theory, KL divergence); einops/einsum tensor manipulation; batched tensor operations via a ray tracing exercise; GPU-style parallelism; coding best practices and type annotations

Week 2: CNNs, ResNets & Optimization Building nn.Module subclasses; training loops with dataloaders and validation; implementing convolutions from scratch; batch normalization; assembling ResNet34 and loading pretrained ImageNet weights; SGD, RMSprop, and Adam implemented from scratch; loss landscape visualization; Weights & Biases for experiment tracking and hyperparameter sweeps; introduction to distributed training

Week 3: Backpropagation & Generative Models | Quiz 1 Computational graphs and automatic differentiation; implementing a complete autograd framework from scratch (forward pass, topological sort, backward pass); variational autoencoders (VAEs) — encoder/decoder, ELBO loss, reparameterization trick; generative adversarial networks (GANs) — adversarial training dynamics, discriminator and generator objectives; transposed convolutions

Week 4: Transformer from Scratch GPT-2 style decoder transformer: multi-head self-attention, positional encodings, layer normalization, MLP layers, residual stream; implementing the full forward pass in PyTorch following the TransformerLens architecture; tokenization; cross-entropy language modeling loss; autoregressive text generation with greedy, top-k, and nucleus sampling

Week 5: Mechanistic Interpretability I: TransformerLens & Induction Heads TransformerLens library: loading HookedTransformer models, caching activations, tokenizer usage; circuitsvis for attention pattern visualization; induction heads — identifying in-context learning circuits; hook-based ablation; factored QK and OV matrix analysis; reverse-engineering induction circuits from weights using composition scores

Week 6: Mechanistic Interpretability II: Probing, Function Vectors & Model Steering | Quiz 2 Linear probes on residual stream activations to detect internal representations; probing for deceptive reasoning; nnsight library for causal interventions on large models (GPT-J-6B); task-encoding hidden states (h-vectors); reproducing function vector results; steering vectors in GPT2-XL — replicating behavioral changes from Turner et al.

Week 7: Mechanistic Interpretability III: Sparse Autoencoders & IOI Circuits Sparse autoencoders (SAEs): theory, training objective (L1
reconstruction loss), monosemantic feature extraction, feature visualization; indirect object identification (IOI) — full circuit replication in GPT-2 Small including name mover heads, duplicate token heads, and induction heads; path patching; logit lens and logit attribution

Week 8: Mechanistic Interpretability IV: Superposition & Toy Models The superposition hypothesis: why neural networks represent more features than dimensions; polysemanticity; feature geometry and the role of sparsity; training SAEs on toy superposition models; implications for scalable interpretability; connection to monosemanticity research at Anthropic

Week 9: RL Fundamentals, DQN & Policy Gradients Markov decision processes; policies, value functions, and Bellman equations; analytic solutions (policy iteration, value iteration); sampling-based methods (multi-armed bandits, ε-greedy, UCB); Deep Q-Networks (DQN) with experience replay and target networks; OpenAI Gym environments; Vanilla Policy Gradient (VPG) with baseline and reward-to-go

Week 10: PPO & Reinforcement Learning from Human Feedback | Quiz 3 Proximal Policy Optimization (PPO): clipped surrogate objective, entropy bonus, generalized advantage estimation (GAE); implementing and training a PPO agent; reward model training from human preference comparisons; complete RLHF pipeline — supervised fine-tuning, reward modeling, and PPO-based alignment applied to a transformer language model

Week 11: LLM Evaluation: Threat Modeling, Dataset Generation & Inspect Introduction to LLM safety evaluations and why they matter; LLM API usage basics; alignment faking as a worked case study; threat modeling and safety cases; evaluation specification and design; synthetic dataset construction and quality control; running evaluations at scale using the UK AISI Inspect framework; interpreting and communicating eval results

Week 12: LLM Agents & Alignment Science LLM agent architectures: tool use, agent loops, multi-step reasoning; agent evaluation methodologies; emergent misalignment in language models — power-seeking, sycophancy, deceptive alignment, sandbagging, corrigibility, and self-preservation; theoretical frameworks for the science of misalignment; mechanistic analysis of chain-of-thought and reasoning model internals; LLM psychology and persona vectors