13 minute read

In our last paper we observed that local tangent/normal projections were a plausible tool for separating curved and linear components of the activation manifold. We were primarily focused on how the J-lens deals with manifold structure, but I think this idea is quite a bit more versatile! I argue here that if we can obtain good approximations of tangent spaces (and we can!), we can leverage them to construct a training-free path to unsupervised latent decomposition in language models.

This essay discusses some early results we’ve had in this direction. My primary purpose is to collect and share ideas with collaborators—but the results are also promising, and hopefully interesting if you’ve ended up here! While we look forward to putting these and similar discussions in a paper, this note is more of a casual discussion of an ongoing project than a formal write-up. If you want to skip the surprisingly large amount of math, check out four two dimensional spaces we isolated in Llama’s residual stream in Figure 1—we both recover known multidimensional structures, and discover what we believe to be new ones—or view the full gallery of detected latents here!

Top activating tokens on four multidimensional spaces in Llama-3-3.2B's activations Figure 1: Top activating tokens on four multidimensional spaces in Llama-3-3.2B’s activations, isolated by analyzing tangent geometry on components with high normal signature, following the method described below. See the full gallery here.

Decomposition of conic manifolds

Suppose we’re given a point cloud X, generated by an unknown sparse mixture of conic components

\[x = \hat{c} + \sum_i a_i c_i, \quad c_i \in \mathbb{R}^d,\; a_i \ge 0,\]

along with a tangent hyperplane TxiX for each point (calculated e.g. by the pushforward Jacobian). By using a least squares regression on tangent planes, we can approximate the cone apex, and subtract it from X to obtain X̃ = X − c*, so we can assume the cone-point to be 0.

Geometrically, there’s a very intuitive approach to finding the generators: the tangent space at x = ∑i ∈ I aici is given by

\[T_xX = \text{Span}\{c_i \ | \ a_i > 0 \}.\]

If we have two data points x1, x2 and the coefficients of the sparse decomposition a1,i, a2,i > 0, then ci ∈ Tx1X ∩ Tx2X—in fact, this intersection precisely calculates the span of the set of ci that are active in both x1 and x2. If we repeat this procedure for a large collection of data points, we will either isolate each ci separately, or find the minimal subspace A that contains always coactivating latents.

In the remainder of this section, I lay out a procedure to calculate these intersections efficiently. In Lifting we begin by lifting the representations into Sym2(ℝd), to make the frame (overcomplete decomposition) {ci} linearly independent, under certain hypotheses. In Efficiently finding intersections, I lay out two approaches to algorithmically identifying intersections. Finally, in Generalizing to include curved components, I describe a simple trick to generalize from sparse combinations of conics to arbitrary sparse combinations.

Lifting

We typically think of concept latents as being vectors, but it’s useful to think of them as vector subspaces: adding any contribution in a certain direction changes the extent to which the concept corresponding to that direction is expressed.

With this in mind, there’s a natural inclusion of concepts as points in the Grassmannian.1 This neatly removes the separation between 1-dimensional and multi-dimensional features for free—they’re all points in different strata of the Grassmannian! To deal with these practically, we use the canonical embedding of the Grassmannian into the space of symmetric matrices Sym2(ℝd), given by V ↦ PV, and perform our search for latent directions in this lifted space Sym2(ℝd). Intuitively, instead of tracking the vectors in ℝd, we’re tracking kernels that extract those vectors.

Remark. One very nice feature of this translation is that we remove (some of) the linear dependence of the overcomplete concepts, and can use the more traditional tools of linear algebra to study operators in it. While there’s still a dimension cap—Sym2(ℝd) is d(d + 1)/2-dimensional—note that this cap is precisely the Welch bound/Gerzon’s packing limit on how many near-orthogonal vectors can be embedded in ℝd! Of course, this is no coincidence—those bounds come from the same embedding into Sym2(ℝd) we consider.

If PV are our points now, we also need to lift subspaces to intersect: recall, our motivating idea is that the intersections Tx1M ∩ Tx2M are a lens to the sparse generating objects. To this end, consider the sym functor sym: V ↦ Sym2(V) on the poset categories associated to ℝd and Sym2(ℝd). Since Sym2(V) ∩ Sym2(W) = Sym2(V ∩ W) for V, W ⊂ ℝd, we can evaluate the intersections in the lifted space, and recover the intersections in ℝd.2

The reason this connection is so nice to work with is that sym has a left adjoint: consider F: poset(Sym2(ℝd)) → poset(ℝd), which assigns a subspace M ⊂ Sym2(ℝd) to the span of the column-space of all elements of M. Preservation of intersections follows immediately, since right adjoints preserve limits.

Efficiently finding intersections

Combinatorially, directly solving equations in all measured tangent spaces is a bit of a nightmare, and small amounts of noise will cause two high-codimension vector spaces to fail to intersect. Instead, we study matrices in Sym2(ℝd) ⊗ Sym2(ℝd), where we combine the projection matrices PV, V = Sym2(TxX) for our datapoints x, into an algebraic criterion that expresses intersection. For any matrix A and subspace V ⊂ Sym2(ℝd), we know the commutator [A, PV] = APV − PVA is 0 if and only if A|V has range V and A|V⊥ has range V⊥. If we then find matrices A in the kernel of both [A, PV1] and [A, PV2], they must separately preserve V1 ∩ V2, V1, V2, and the associated perpendiculars. In particular, if the intersection V1 ∩ V2 is 1-dimensional, then it must be an eigenvector of A. Broadening to our full dataset, we can express the desired restrictions for A using the Schur commutant

\[\mathcal{Q}(A) = \sum_i \|[A, P_{V_i}]\|_F^2. \tag{1}\]

(Note that since ‖Z‖F2 = Tr(Z⊤Z) we can express (1) in terms of the first and second moments of PVi, and use this to find elements of the kernel.) For any matrix A ∈ ker 𝒬, the non-degenerate eigenvectors must be contained in either the tangent space, or its orthogonal complement, for every contribution Vi!

Everything everywhere all at once

If we can find a generic A as in (1), and all eigenvalues are distinct (excepting for multidimensional primitive components), then directly analyzing the spectrum of A can tell us the generators of our sparse mixture.

There’s a small hitch here—as we noted above, the commutator relationship forces A to preserve both the subspaces Vi and their orthogonal complements. As a result, if the underlying generators are not orthogonal (which they can’t be!), we’ll end up with degenerate eigenspaces, instead of having A cleanly resolve the generators along its spectrum. We can solve this, though, by whitening: if the features are uncorrelated, we can orthogonalize them by calculating the sample covariance

\[\Sigma_S = \sum_m \sigma_m^2 P_m \otimes P_m\]

and whitening within Sym2(ℝd) by multiplying by ΣS−1/2. (Note the connection with Park et al.’s well known discussion of intrinsic metrics for language models!)

In principle, this tells the full story: after whitening the lifted representations and solving for A in the kernel of the commutant, we can analyze its eigenspaces. For general A, the eigenvalues will be distinct except in multi-dimensional subspaces—by tracking eigenvalue differences, we can therefore automatically identify both the generators of the sparse mixture and their intrinsic dimensions.

In practice, exploiting kernel tricks allows this to be done efficiently on subsets of the data—for example, after identifying data points with high normal contribution, we can attempt to isolate multidimensional components of the sparse mixture, of which there should be relatively few for language models3—but it simply doesn’t scale to large models evaluated over large datasets in an approachable manner. (Or at least, we have not yet been able to identify a way to do so.) Below, I describe a streaming strategy, designed to extract all latents by treating the commutant as a loss function, and optimizing it on a wide array of low dimensional matrices. This approach has more favorable scaling, similar to that of sparse autoencoders.

Solution in pieces

Sym2(ℝd) is d(d + 1)/2-dimensional, so an element of Sym2(ℝd) ⊗ Sym2(ℝd) consists of O(d4) entries. Even for fairly small language models, this makes accessing the kernel of the Schur commutant (1) prohibitive. Instead, we’ll need to use some kernel tricks to operationalize the approach.4

Given a basis B of a subspace V ⊂ ℝd, the projection operator PSym2(V) onto Sym2(V) ⊂ Sym2(ℝd) is given by conjugation by BB⊤. For any low rank w = FF⊤ ∈ Sym2(ℝd), ‖w‖F2 = 1, we can compute the commutator loss of PSym2(V) on A = w ⊗ w as s(w) − s(w)2, where

\[s(w) = \|P_{\operatorname{Sym}^2(V)}(w)\|_F^2 = \|BB^\top w BB^\top\|_F^2 = \|B^\top w B\|_F^2. \tag{2}\]

To see this, dropping subscripts for convenience, note that

\[\|[A, P_{\operatorname{Sym}^2(V)}]\|_F^2 = \operatorname{Tr}\!\left((AP - PA)^\top (AP - PA)\right) = 2\operatorname{Tr}(PA^2P) - 2\operatorname{Tr}(APAP).\]

Then the first term is precisely s(w), and APA = ⟨w ⊗ w, P⟩ w ⊗ w = s(w) w ⊗ w, and the result follows.

Geometrically, since w has unit norm, the projection s(w)1/2 = ‖PV(w)‖F is the cosine of the angle between w and V. Thus s(w)(1 − s(w)) is minimized as either w becomes a vector within V (s(w) = 1), or a vector orthogonal to V (s(w) = 0).

In order to operate in a streaming fashion, without materializing any matrix A in memory, we initialize a collection of low rank latents, and perform updates iteratively, including a decorrelation penalty to prevent the latents from collapsing.

Whitening also plays an important part in this version of the story: direct optimization on an array of low dimensional latents Di on 𝒬(Di) using 𝔼[s(Di) − s(Di)2] results in collapse for sparse, non-orthogonal latents, with most features dying. By instead optimizing

\[1 - \frac{\mathbb{E}[s(D_i)^2]}{\mathbb{E}[s(D_i)]}\]

—that is, by normalizing by the average activation associated to the direction Di, or equivalently whitening—we can cleanly optimize without creating dead latents.

Generalizing to include curved components

Recent literature has found an array of nonlinear manifold components of language model activation space—in arithmetic, in a variety of cyclical time data, etc. As a result, the tangent space of an activation that includes some point on these curved moduli spaces doesn’t actually generate the near-orthogonal subspace that contains the curved component. There’s a surprisingly easy fix, though—if we replace TxX with Span(TxX, x) then we can treat irreducible curved components in the same way we treat the conic components!

(If the category theory above wasn’t quite enough for you, note that this construction is the universal conic manifold containing the activation manifold—in other words, it’s the smallest extension we can make to the true manifold that has the desirable properties of a conic manifold, which is another cute adjunction on poset categories. By operating on this space, we can use our nice construction, and then reduce back to observations about the original activation space.)

Results

Distinguished multidimensional subspaces in Llama-3.2-3B

We scaled this approach to up to Llama-3.2-3B Instruct on 1.5 million tokens drawn from WikiText-2 and GSM8k. We tried both methods described in Everything everywhere all at once and Solution in pieces, and show full galleries of recovered subspaces here. For the commutator, we subsampled data with high normal contribution, so that we could work with sample covariances, rather than with operators on Sym2(ℝd). For the iterative approach, we initialized 1024 latents, of which 545 remained distinct. In each chart, we graph the top activating tokens, where we’ve used SVD to determine the projection dimension. Top tokens are filtered to only include tokens that appear at least 15 times in the dataset, and we use the context deflated (Appendix: In context deflation) representations averaged among all token appearances in our dataset. Labeling prioritizes highest activation—in particular, no human or LLM-based filtering occurred.

Not all of the subspaces look particularly interpretable, but many of them do! Within the commutator eigenvectors approach, for example, we recover representations of cities (M63), countries (M24), US states (M10), all reflecting this classic paper. We recover months (M55), another classic result, though note that M55 also captures some seemingly unrelated antipodal data. I particularly like M37, which on one half represents a variety of different artists, and opposite them their corresponding art form. In M60 we see the pronouns represented on a 1D manifold; M46 shows familial relationships; M33 shows units of time; M22 parameterizes languages; M14 shows mid-century American names (maybe?); M04 consists on one side of ordinal numbers, and on the other side cardinals; M03 shows scientific/professional suffixes; M02 shows various synonyms for ‘frequently’; M29 shows units of measure; M61 is the number helix, etc.

A similar story holds for the iterative approach: M801 captures months, but also days of the week, seasons, and times of day, each as isolated clusters. Units of measure are captured in M892; cities and US states in M932; team sports are parameterized in M986; countable objects in word problems in M1009; C840 captures synonyms for ‘perfect’; C621 combat; C659 beliefs; C196 size, etc.

In the iterative study, we additionally tested how deflating multidimensional components affects SAE training at matched sparsity, and show a strong correlation between normal energy and the effective dimension of isolated directions (Figure 2). We also perform a preliminary directional ablation study, showing that isolated subspaces, particularly isolated multidimensional subspaces, are much more impactful than random ablations on math, calendar, and prose predictions.

Rayleigh commutant and ablation results Figure 2: (a) Rayleigh normalization/kernel whitening prevent collapse and correctly decompose overcomplete synthetic sparse mixtures. (b) Average normal energy of latents in detected components versus effective rank of the discovered latent. (c) Deflating multidimensional components when training Top-k SAEs improves reconstruction at matched sparsity budgets. (d) Causal ablation of detected multidimensional, linear, and random subspaces (all matched to be 32 dimensional) on math, calendar, and prose datasets.

Appendix: In context deflation

During implementation, the coding agent noticed a variety of factors that it chose to modify from the basic picture to improve results—I’m specifically highlighting in context deflation, which is very reminiscent of temporal SAEs.

As a sequence of tokens is processed a common, passage-level component becomes an important shared summand of the residual stream. This is especially true in mid-to-late layers, where many attention maps have aggregated data. This results in a high variance accumulated context term. Since this ‘context direction’ is active throughout the sequence, it’s easy to identify: we isolate the highest variance direction(s) in the preceding sequence, and subtract off a projection onto these directions. I’m not sure this is optimal, and didn’t experiment substantially with it, but might play with this in the future.

One place that this is particularly important is if you want to use k-NN estimators for your tangent space. While I’m generally an exponent of using the Jacobian pushforward, one thing I spent some time on was cleaning up the k-NN estimator, which is very messy in high dimensional spaces. By removing the contextual correlation between tokens in the same sequence, the dataset for obtaining tangent structures becomes much cleaner. Expect a follow-up post with an expansion on this—the other key improvement is flag deflation: once a direction is identified as part of the tangent space, deflating that direction from the data removes much of the noise in assessing the next direction to add to the approximated tangent space.

  1. Note there is a key limitation of this approach, which is that it identifies antipodal vectors. Antipodal vectors don’t always represent dual concepts, as nonlinearities can extract the two opposite rays separately. ↩

  2. To see this directly/without appealing to category theory, note that Sym2(V) consists of symmetric matrices whose eigenvectors with nonzero eigenvalues all lie in V. Then if A ∈ Sym2(V) ∩ Sym2(W), all of A’s eigenvectors with nonzero eigenvalues lie in V ∩ W, so A ∈ Sym2(V ∩ W). ↩

  3. Which is how we obtained the teaser images above! ↩

  4. I should note that this isn’t something I specialize in—almost all the work done in this section was proposed by LLMs, with limited human corrections and improvements. While I’ve attempted to distill the main ideas, I apologize for any errors, omissions, or sub-optimizations in the implementation. ↩

Updated: