TTT is not linear attention—but it might be something weirder
An interesting paper by Liu et al. came out last week, arguing that we should reconceptualize the variety of test time training (TTT) with KV-binding archite...
Recent Posts
Self-distillation through the ages
A couple of really phenomenal recent papers have brought up new approaches to self distillation in language models and reinforcement learning, and motivated ...
Residual expansion: hyper-connections, virtual width, and attention in the depth direction
There have been quite a few interesting takes on what I’m going to call “residual expansion” over the last year, particularly notably Deepseek’s Manifold-con...
Moduli Regularization
This post is a brief, intuitive summary of my paper, “Geometric sparsification in recurrent neural networks.” Academic publications emphasize formal descript...