Interpretability explains why a model behaves the way it does. Training data attribution goes a step further, connecting that behavior back to the data that caused it - something that's only recently become possible at LLM scale. The challenge I care about is bringing this to post-training, so that when a model misbehaves we can trace it to a cause: was it the data, the base model, or something else?

This project aims to use data attribution as a diagnostic step on post-training runs. Take a freshly post-trained model that shows a specific misbehavior, use gradient-based attribution to find which training examples were most responsible, and compare signals across the base model and the fine-tuning data to work out where the behavior came from. The goal is to automate this process, linking a misbehavior back to its cause.

A combination of mathematical and empirical methods are likely needed for safe transformative AI. Gradient-based training data attribution is grounded in calculus and is one of the most theory-dense and computationally expensive fields in AGI Safety. In recent years it has shown promise in model forensics explaining why a model learned a particular behavior during training.

J is a DPhil student at the University of Oxford, supervised by Jakob Foerster. As well as a Research Associate in the AGI Safety and Alignment team at Google DeepMind, working on interpretability and data attribution. Prior work includes an internship in the pretraining team at Cohere and work with UK AISI on agentic scaffolds for Inspect.