Residual expansion: hyper-connections, virtual width, and attention in the depth direction
There have been quite a few interesting takes on what I’m going to call “residual expansion” over the last year, particularly notably Deepseek’s Manifold-con...