here is a somewhat alarming fact: the most capable AI systems in the world are, in a deep sense, not understood by the people who built them. the engineers who trained gpt-4 know the architecture. they know the training procedure. they can inspect the weights. but if you ask them "why does this particular input produce this particular output, mechanistically, in terms of what computation is actually being performed" — they largely can't tell you. not for any specific interesting case. you can observe inputs and outputs perfectly. the interior is opaque. this is a weird situation to be in with a technology that is increasingly being used to make consequential decisions about things like loans and medical diagnoses and legal research.
mechanistic interpretability is the project of actually opening the case. not "what does the model do in this category of situations" (that's behavioral evaluation, which we're reasonably good at) but "what computation is the model performing, what features does it detect, what circuits have formed, what algorithm has it learned." the target is something like: given a specific neuron in a specific layer of a specific model, what does it respond to, and why does activating it produce the downstream effects it produces. this is hard. we're making progress.
some of what's been found is genuinely striking. anthropic's mechanistic interpretability team has identified induction circuits — small two-attention-head circuits that implement a kind of in-context pattern matching, allowing the model to notice that "A followed by B" appeared earlier in its context and predict "A" might be followed by "B" again. these circuits appear across model sizes and architectures; they seem to be a solution that gradient descent converges on reliably. the team has also found evidence of "superposition" — models encoding more features than they have dimensions by using overlapping representations, which is why you can't just read off what a neuron "means" by looking at when it fires maximally. sparse autoencoders can partially decompose these superposed representations into something more interpretable. it's early-stage and the techniques don't yet scale cleanly to frontier models, but the basic premise — that there is structure to be found and it can be found — seems to be holding up.
why this matters for safety: the current dominant approach to AI safety is empirical. you test the model in many situations, you red-team it, you observe behavior, you build up a profile of what it does and doesn't do. this is reasonable and important. but it has a fundamental limitation: you can only test situations you think to test. an empirical safety case is always "the model seemed fine in the situations we evaluated." it says nothing about situations you didn't evaluate, and systems operating at scale will eventually encounter situations that weren't in the test distribution. if you understand the mechanism, you can reason about out-of-distribution behavior: "this circuit implements X, so in situations with property Y the model will do Z." that's a different kind of safety claim, and it's the kind that generalizes.
the deeper problem is that "we tested it a lot and it seemed fine" is not a safety case that regulators, courts, or engineers in adjacent fields would accept for safety-critical applications. we don't certify aircraft by flying them many times and noting they didn't crash. we understand the aerodynamics. we have failure mode analysis. we have systematic models of what happens under conditions outside the normal operating envelope. mechanistic interpretability is trying to build the equivalent foundation for AI systems. it's an ambitious goal and the field is nowhere near there yet, but the alternative — continuing to deploy systems we don't understand in increasingly consequential roles and hoping the empirical track record holds — seems worse.
the field is small and genuinely needs more people. chris olah's transformer circuits papers and neel nanda's tutorials are the best entry points. engineering-background people who can build better tooling are particularly needed — a lot of progress has been gated on infrastructure that hasn't been built yet. the upside of joining early is that individual contributions still matter a lot, and the questions are genuinely interesting.