What a Vision Transformer Knows About Space

A Vision Transformer trained on ImageNet only ever sees one signal during training: the class label of each image. Cat. Dog. Coffee mug. Boeing 747. Twenty-one thousand of them, repeated across tens of millions of pictures. There is no annotation telling the model where the boundary of an object lies, no supervision about depth, no hint that the world being photographed is three-dimensional and that surfaces recede into the distance. And yet, when you take a frozen ViT off the shelf and bolt a small head onto it, it transfers cleanly to detection, segmentation, and depth estimation. That should be a little unsettling. The model was never told space exists. So why does it know about it?

This is the question I set out to answer in a recent project. The frame I chose was old-fashioned and useful: probe the model. Not interpret what it does in some abstract sense, but ask, layer by layer, where exactly inside the residual stream you can read off geometry with nothing more than a linear classifier. Probing is simple, sometimes embarrassingly so, but it has the rare property that the answers it produces can be falsified. If a linear probe at layer L can predict the depth of a patch, the depth signal lives in a separable subspace at that layer. If not, it doesn't. That's the floor.

A Tale of Two Probes

I picked two complementary tasks. The first was patch boundary detection on BSDS500: for each 16×16 patch, does it overlap a contour that human annotators agreed on? This is a deeply local property. You don't need to know what's in the image to detect a boundary; you need contrast, edges, the kind of signal that a few self-attention layers can lift out of raw pixels.

The second was per-patch depth regression on NYU Depth V2: given the patch's hidden state, predict its mean depth in the scene. Depth is the opposite kind of property. There is no purely local cue for it. To know that a chair is far and a wall is close, the model has to integrate perspective convergence, occlusion ordering, and relative object size from across the entire image. You can no more infer depth from one patch than you can guess how far a leaf is from the camera by staring at one square of pixels.

So one probe asks for local structure, the other for global spatial layout. If both peak at the same layer, locality doesn't matter and the model is doing something more uniform. If they peak at different layers, the offset directly measures how many transformer blocks of cross-patch communication the network needs to go from edges to layout.

I also probed a randomly-initialized ViT-B/16 as a control. This matters more than it sounds. Positional encodings alone can carry surprising amounts of information. In indoor scenes, "the bottom of the image is usually closer than the top" is a strong prior that a random network can read off without any training at all. Without that baseline, you can't tell which part of the signal is learned and which is just architectural artefact.

A Spatial Hierarchy

The numbers told a clean story. Boundary detection peaks at layer 5, with a linear probe hitting an average precision of 0.833, a +0.23 gap over the random baseline that stays flat around 0.60 across all layers. Depth peaks two to three layers later, at layer 8, with a mean absolute error of 0.0875 in normalised units, roughly 0.875 metres of error and a 23% reduction over the random control. Then, at layer 12, both signals collapse. Boundary AP drops to 0.617. Depth MAE rises to 0.0992. The final transformer block specialises for class discrimination, and when it does, geometry stops being readable.

That offset between L5 and L8 is the central finding. Local structure becomes linearly accessible early, after only a few layers of self-attention have aggregated contrast across neighbouring patches. Global structure takes more work. To estimate depth, the model has to ferry perspective and occlusion cues across the full 14×14 patch grid, and that propagation costs two or three additional transformer blocks. The network is building space the same way a textbook visual system might: edges first, layout next, identity last.

There is a striking parallel to the primate visual cortex here, which I won't push too far because analogies between transformers and brains are easy to overrun. But the early-to-late ordering, V1/V2 detecting edges and V4/IT representing depth, shape, and object identity, was identified decades ago. A classification-trained ViT, with no architectural inductive bias for vision and no spatial supervision, finds an analogous organisation through gradient descent on category labels alone. That's not a coincidence so much as a hint: the structure of natural images forces certain representations to emerge in a particular order, regardless of substrate.

Linear Is Enough

One of the cleanest findings in the experiment came from a comparison I almost didn't run. For each task, I trained both a linear probe (a single 768-dimensional weight vector, 769 parameters in total) and an MLP probe (a 768→256→1 network with about 197,000 parameters, 257× larger). The MLP should win, or at the very worst tie. It has the capacity to recover any nonlinear structure the linear probe misses.

It didn't. The linear probe matches or beats the MLP on both tasks, and beats it dramatically at layer 12. On depth at the peak layer, linear hits MAE 0.0875 while the MLP hits 0.0920. On boundary, the linear AP edges out the MLP at L5. The MLP wins slightly on raw accuracy at the 0.5 threshold for boundaries, but loses the threshold-free comparison and degrades sharply at the final layer where its extra capacity has nothing useful to fit.

What that tells you is geometric. The information lives in a clean linear subspace. The MLP's extra capacity has no useful nonlinearity to fit, so it fits noise. The depth signal isn't smeared across hundreds of interacting dimensions; it's concentrated, and you can read it with a single weighted sum. That observation shaped what I did next.

From "Information Is There" to "the Model Uses It"

Probing is correlational. It tells you that information is present at a given layer, not that the model uses it for anything downstream. The standard caveat is well known, but it can be addressed. If depth really sits in a single direction in the 768-dimensional residual stream, then removing that direction should be catastrophic for the probe, while removing any other direction should not.

So I projected out the depth probe's weight vector at every layer and re-ran the probe on the now-mutilated activations. The MAE shot up by 49% to 165% depending on the layer. At layer 5, it tripled. Then I repeated the experiment with ten random unit vectors, projecting them out one at a time. The MAE moved by less than 1% in every case. The depth signal is not diffusely spread; it's pinned to one direction out of 768. A direction that is, mechanistically, a thin slice of the model's representation, but the load-bearing slice for spatial layout.

That's the strongest single result in the project. It rules out the naive worry that the linear probe might be picking up an arbitrary projection only weakly aligned with the model's own encoding. If that were true, partially erasing the direction would produce erratic, non-monotonic damage. Instead, a dose-response sweep, scaling the ablation from 0 to 1, produces smooth graded degradation. Remove half the depth direction, lose roughly half the depth signal. The encoding is real and graded.

Active Maintenance, Not Passive Carry

The last experiment was designed to answer a more subtle question. If the depth signal becomes linearly readable at layer 5 and stays readable through layer 8, is the model passively carrying it through the residual stream, copying it forward untouched, or is it actively re-deriving it at each layer from the full representation?

I tested this with a targeted activation patching variant. Standard activation patching swaps an entire layer's activations between a source and a destination input. Mine was finer: at layer L, I replaced only the depth-direction component of the source's hidden state with the destination's depth-direction component, leaving the other 767 dimensions untouched. Then I tracked, layer by layer, how much of that swap survived downstream.

If depth were passive freight, the swap would persist near 1.0 through to layer 12. It didn't. The effect decayed. But the rate of decay told its own story. A swap at layer 1 lost 88% of its effect by layer 2 and was effectively gone by layer 12. A swap at layer 8 retained 76% of its effect at layer 9 and 18% even four layers downstream. Mid-layer interventions persist; early ones get erased.

That's the signature of active maintenance. Each transformer block partially overwrites the depth representation and re-derives it from the full hidden state. Early-layer depth directions are fragile statistical correlations, easily overwritten by subsequent attention. Mid-layer depth directions are stable geometric features, established and continuously refreshed by the surrounding layers. The peak isn't just where the probe accuracy is highest; it's where the encoding has the longest causal reach.

What This Changes

The most concrete practical takeaway is unsentimental: if you're building a downstream system on top of a frozen classification ViT and you care about geometry, don't read from the last layer. The L12 features are the worst place to put a depth or segmentation head, because the final block actively destroys spatial information in service of class discrimination. Mid-layer features, around L5 to L8, are where geometry lives. A small linear head on the right layer will outperform a deeper head on the wrong one, and at a fraction of the compute.

The more interesting takeaway is methodological. The progression in this project, from probing ("information is present") to probe-direction ablation ("the information is concentrated and load-bearing") to targeted patching ("the information is actively re-derived"), is a template that generalises. You can run it on any property, in any model, where you can train a linear probe. Most ViT interpretability work stops at the first step. The second is where correlation becomes causation. The third is where you learn whether the model is carrying or constructing.

The broader lesson is about objectives. ImageNet classification is, on paper, a narrow signal. It contains no annotation about depth, occlusion, or geometry. And yet a model trained on it alone develops a layered spatial hierarchy whose shape echoes what neuroscientists have spent forty years documenting in the primate visual stream. The objective doesn't tell the whole story of what the model learns. The data does, and natural images are saturated with geometric structure. Train a sufficiently flexible model on enough of them, and the geometry gets in even when no one is looking for it. The model has to learn space to do its job, even if no one ever tells it what space is.

That, more than any one number in the paper, is what I find worth dwelling on. The interpretability question, where and how, is just the surface. The deeper question is why, and the answer seems to be that the world we photograph is three-dimensional, that classification implicitly demands a representation rich enough to handle it, and that gradient descent on enough images quietly reinvents the structure of vision from the ground up.