# Neural Networks Are Already Clean · 神经网络本来就很干净

A plain-language, bilingual guide to Nick Cammarata's essay "Neural Networks Are Already Clean" (Goodfire, 15 October 2026), for people who don't work in AI. It follows the six paragraphs of AI/TLDR's brief. Summary and figures by David Jiang · https://dj.theory-a.com/neural-networks-are-already-clean

If you work in AI, read the original instead: https://already-clean.goodfire.com/ (full technical argument, data and interactive figures).

- AI/TLDR brief: https://ai-tldr.dev/releases/goodfire-neural-networks-already-clean/
- Hacker News discussion: https://news.ycombinator.com/item?id=50105647

The essay calls itself "a working hypothesis". Its language-model numbers are preliminary, and a technical paper is still to come. The claims below are Cammarata's, not settled science.

Nobody writes the rules inside an AI model. It learns them from examples, and the result is billions of numbers no one designed. For about a decade, the expert view was that the inside is a tangle you can't read until you untangle all of it. Cammarata helped build that view: he co-wrote *Zoom In* (2020), one of the papers that made a "cat and car" neuron the textbook example of the tangle. Now he argues the tangle was mostly a misreading.

AI 模型里的规则不是人写的，是它从例子里学来的。十年来，专家都说里面是一团乱麻，不整团解开就读不懂。Cammarata 正是建立这个看法的人之一，现在他说那团乱麻多半是看错了。

## Three ideas first · 先讲三个概念

1. **A neuron** is one of millions of tiny dials inside a model. Each turns up for some things and down for others. Nobody sets them; they settle during training. · 神经元是模型里几百万个小旋钮之一，看到某些东西调高，看到另一些调低，是在训练中自己定下来的。
2. **Interpretability** is the project of saying what each neuron does, so we can check what a model is really doing, not just what it says. · 可解释性研究就是说清每个神经元在做什么，好检查模型实际在做什么，而不只是听它说了什么。
3. **The worry:** many neurons seemed to mix unrelated things (cats *and* cars). The leading theory, **packing** (researchers say *superposition*), said models cram more ideas than they have neurons into shared neurons, so you first need **untangling tools** (usually *sparse autoencoders*, SAEs) before you can read anything. · 担忧：很多神经元好像混着不相干的东西。主流理论“打包”（术语叫叠加）认为模型把概念塞进共用的神经元，所以得先用分解工具（通常是稀疏自编码器）拆开。

Other words: a **mixed-up neuron** is what researchers call *polysemantic*; an **offshoot** is a related side-job a neuron grows while learning (the essay says *recursive generalization*); **parameters** are the adjustable numbers a model is made of.

Example: a neuron in a small image model turns up for cars, trains and carts, and turns down for boats, planes and an ox-cart on water. Both sides are one idea: "moves on land" or not.

## ¶1 · Nine in ten neurons have one job · 十个神经元里，九个只干一件事

**AI/TLDR:** AI models are already mostly readable. In a first check of a large open-source language model (Qwen3-32B), about 90 in 100 neurons do one main job, and only 2 to 4 in 100 are still a mystery.

**简报：** AI 模型本来就大体读得懂。在对大型开源语言模型 Qwen3-32B 的第一次普查里，每 100 个神经元约 90 个只做一项主要工作，还没搞懂的只有 2 到 4 个。

**In plain words:**

- In 2023, a major Anthropic study found the typical neuron of a tiny model made no sense to its raters (it scored 0 out of 14). An automated OpenAI project the same year, which Cammarata started, could explain about 2 in 100 neurons at best.
- Three years later, with better tools and a modern model: about 90 in 100 have one main job, often with a small related side-job; about 7 have two related jobs (like the same idiom in two languages); 2 to 4 aren't understood yet. So far the pattern holds in models with trillions of parameters. Strictly, the essay counts each side of a neuron (what turns it up, what turns it down) separately.
- The exact number depends on where you draw the lines. In the examples he examined most closely, counting strong signals only, it was closer to 97 in 100.

Real neurons from the essay: a "mixed-up comma neuron" that actually tracks Colorado (turn it up and the model wants to go hiking); a neuron for "a sense of righteousness when a bargain purchase turns out to work better than the expensive alternative"; a "pr" neuron whose odd-looking web-code and Bengali firings both turn out to be "pr"; and Santa, who lives on the "turn-down" side of eight neurons.

## ¶2 · The mess was mostly misreading · 乱，多半是看错了

**AI/TLDR:** Most neurons that looked mixed-up were misread, and models are tidier than the packing theory said.

**简报：** 大多数看起来混杂的神经元只是被看错了，模型比“打包”理论说的要整齐。

**In plain words:**

- Packing says a model short on neurons crams unrelated ideas together because they rarely come up at the same moment. If so, what shares a neuron means nothing, and you must train a second AI to pull the ideas apart.
- Ordinary learning predicts the opposite: things share a neuron because they have something real in common. A floppy-eared-dog neuron makes sense; floppy-eared dogs plus Latin poetry is what packing would allow. Watching a neuron grow tells you which is happening.
- When a neuron looks mixed-up there are five suspects: weak tools, a wrong explanation, bad learning, a related offshoot, and, last, real packing. Calling a neuron mixed-up "is often more a claim about the observer than the neuron." Cammarata now puts packing last because it's the hardest to test, and its only symptom ("this doesn't make sense") is also what weak tools look like.
- Cases from the essay, each solved by another suspect: the cat-and-car neuron (a V shape); a 2019 packing "discovery" (cars and dogs sharing a pose, plus a misleading picture-making tool); the "pr" neuron; the Colorado comma neuron; a "coffee and universities" neuron (researchers only showed it two paragraphs at a time); a neuron that made sense once it was shown health spam websites; an old model's last layer (a design that learned badly); and the "iy" neuron (an offshoot). The essay knows of no confirmed packing in a real model since 2019, and that case was a mistake.
- The opposite risk, insisting every neuron has a tidy story, is limited because the other suspects can each be tested, and real explanations make sharp predictions.

## ¶3 · The cat-and-car neuron was a V · 猫与汽车神经元，其实是个 V

**AI/TLDR:** An informal essay arguing that the textbook examples of mixed-up neurons, like the famous "cat and car" neuron, were misread. The mixing came from neurons growing related offshoots, plus weak tools and too little data.

**简报：** 一篇非正式长文，认为混杂神经元的经典例子（比如“猫与汽车”神经元）都被看错了。混杂来自神经元长出的相关分支，再加上工具太弱、数据太少。

**In plain words:**

- Neuron 4e:55 in a 2014 image model (InceptionV1) responded to cat faces and car fronts, and was the field's favorite exhibit of a mixed-up neuron. When Cammarata finally asked what *one* thing it might be, the answer was instant: a V. Its strongest inputs are V-shaped flames, fireboat spray and fox faces; round O shapes turn it down.
- Neurons are often better read as verbs than nouns: they draw a distinction (pointed vs. round) that later parts of the model use. Describe a neuron without listing nouns, then test the pattern on new things (bullet trains, pool-table racks).
- Neurons that really do two or three things mostly grew that way, like a branch sprouting twigs. An "iy" neuron (as in "Miyazaki") grew to "diy", then to "do it yourself", which has no "iy". A neuron in a retrained image model started on yellow flowers in photos that sometimes had dogs, then grew toward other scenery, green things like balls, and dogs with no flowers.

## ¶4 · Less room makes neurons broader, not jumbled · 空间少了，神经元变粗，不会打结

**AI/TLDR:** He re-examines classic examples, retrains image models across a 300-fold range of sizes, checks a large language model's neurons, and squeezes one image-model layer from 480 neurons to 10. An untangling tool on the squeezed layer finds nothing hidden inside, which he reads as evidence against packing.

**简报：** 他重新审视经典例子，在 300 倍的尺寸范围内重新训练图像模型，普查大型语言模型的神经元，并把一层从 480 个神经元压到 10 个。分解工具在压缩层里找不到藏着的东西；他认为这是反对打包的证据。

**In plain words:**

- The old model's last layer, once the main evidence for packing: about 5% readable with 2014 tools (another estimate: about none); about 50% once the tools were fixed; about 70% after retraining with modern habits; about 99% in a modern design (ConvNeXt) of the same size on the same photos.
- Three modern models from 92 thousand to 28.6 million parameters (~300×). The small one isn't jumbled, just broader: one neuron for cars, trains and carts, turning down for boats, planes and an ox-cart on water. The big one separates taxis from ambulances, broccoli from cauliflower. Less room means a model stops sorting sooner; it doesn't mix unrelated things.
- An early layer famous for ten curve detectors, cut from 480 neurons to 10 (48×) and retrained: one "curves and loops" neuron instead of ten curve detectors, two circle neurons instead of several. It still names the right one of 1,000 kinds of object 58.35% of the time (81.14% within five guesses). An untangling tool on the squeezed layer finds broad, bunched shapes, "not ten curves."
- In general, untangling tools found nothing the neurons didn't already have: in the last layer, 88% of what they found was near-copies of single neurons, and deleting the rest didn't hurt accuracy. They were useful for asking "what's in this picture?", most of all in the smallest model.

## ¶5 · Just look at the neurons · 直接去看神经元

**AI/TLDR:** Much research assumes neurons are jumbled and must be untangled first, usually with sparse autoencoders. If Cammarata is right, researchers can study neurons directly. He calls it a working hypothesis; the technical paper is still to come.

**简报：** 很多研究默认神经元是乱的，必须先用稀疏自编码器拆开。如果他是对的，研究者就可以直接研究神经元。他称之为工作假设，技术论文还没发表。

**In plain words:**

- Two ways to study a model: ordinary science on the model itself (look at neurons and connections, run experiments, now with AI helpers), or start from a guessed list of concepts and build elaborate tools to extract them. The field retreated to the second because the model seemed to be a knot.
- Untangling tools can mislead: on the V neuron's layer, one split the V into a "car" piece and a "cat" piece, "like reconstructing Brooklyn from seventy disconnected pieces of Tokyo." A medical-diagnosis process Anthropic traced in its Claude model with untangled pieces shows up in another model's plain neurons, with a hub of just four neurons.
- Understanding snowballs, like a jigsaw or crossword: each neuron you understand makes its neighbors easier, and what won't fit points at a broken tool.
- The four steps: (1) models learn by one rule, finding real patterns; (2) with modern models and tools the science goes far better than expected; (3) understanding snowballs; (4) highly speculative: this may be most of what we need. "Maybe there's a bitter lesson for interpretability too: scaling ordinary investigation beats engineering a clever new decomposition."
- The mistakes don't cost the same. Assume tidy and be wrong: a mixed-up neuron is a lead. Assume packed and be wrong: it's a dead end. "I spent most of a decade figuring out how to look at a model rather than looking at it."

## ¶6 · Become another reader · 做网络的另一个读者

**AI/TLDR:** Interpretability researchers and machine-learning engineers.

**简报：** 可解释性研究者和机器学习工程师。

**In plain words:**

- Inside a model, each part's output is read and built on by later parts, so the parts are already made to be read, by the model itself: "almost a fractal of explanations." The researcher's job is to become another reader.
- Bigger models are easier to navigate, like finding the financial district in New York rather than a small town. "Bigger models learn narrower neurons that are easier to recognize."
- Safety: deception has no reason to hide from the model's own parts, because they need to read it. "My fear is that deception will be right there, in plain sight, and we won't know how to read it."
- Advice to his younger self ends: whenever a neuron looks mixed-up, look harder.
- Today's most advanced language models may be where image models were eight years ago: readable, waiting on many small fixes, and now with AI helpers to do the work.

**Hold loosely:** a working hypothesis in an informal essay, not a peer-reviewed paper; early, preliminary checks; the 90% figure depends on where you draw the lines; full checks of the largest models haven't begun; the technical paper is still to come; the essay comes from Goodfire, a company that works on reading AI models, and most experiments are the author's team's own.

## The essay in six lines · 六句话读完

1. Nine in ten neurons have one main job. · 十个神经元里，九个只有一项主要工作。
2. "Mixed-up" is often about the viewer, not the neuron. · “混杂”往往是在说看的人，不是在说神经元。
3. The cat-and-car neuron was a V the whole time. · 猫与汽车神经元，一直是个 V。
4. Less room makes neurons broader, never jumbled. · 空间少了，神经元只会变粗，不会打结。
5. Understanding snowballs: the map fills in faster as it goes. · 理解会滚雪球：地图越填越快。
6. Don't untangle the network. Read it, the way its own parts do. · 别急着解开网络，像它自己的部件那样去读它。