SplitMOE: experts with shared dimensions [ suprisingly worked near or better than standard MOE architecture ]
Reddit r/deeplearning4d4 min read
The intuition is similar to the DeepSeek MOE, but architecture is different. Please do check it out, and let me know your views. [ https://github.com/Priyanshu-5257/SplitMoE ] submitted by /u/Melodic_Story609 [link] [comments]