Tangent structures in language models
How can we measure curved and linear components in language model activations? And how does that structure affect our ability to interpret what’s going on? O...
Recent Posts
TTT is not linear attention—but it might be something weirder
An interesting paper by Liu et al. came out last week, arguing that we should reconceptualize the variety of test time training (TTT) with KV-binding archite...
Self-distillation through the ages
A couple of really phenomenal recent papers have brought up new approaches to self distillation in language models and reinforcement learning, and motivated ...
Residual expansion: hyper-connections, virtual width, and attention in the depth direction
There have been quite a few interesting takes on what I’m going to call “residual expansion” over the last year, particularly notably Deepseek’s Manifold-con...
Moduli Regularization
This post is a brief, intuitive summary of my paper, “Geometric sparsification in recurrent neural networks.” Academic publications emphasize formal descript...