What exciting architectural moves are we looking to see next in AI models?

We know from all the major chinese labs the current cutting edge architectures such as: conditional engram memory, hyper connections, attention residuals, sparse attention, etc... In the near horizon, we can likely expect looped transformers, Gated Delta Net 2 variants, maybe revival of diffusion transformers somehow but without the intelligence loss, and likely many others that I myself don't know of yet. What are they cooking? submitted by /u/Sadge404 [link] [comments]