Released model

• Sarashina2-8x70B

Sarashina2-8x70B is a larger model based on a mixture of experts (MoE) (Note 1) architecture leveraging insights gleaned through the development of Sarashina2-70B. The model achieved the highest performance in multiple benchmark Japanese language performance tests run by SB Intuitions. Amid the release of relatively small MoE models, it is uncommon to achieve training success with an MoE model architecture of this size. We have released the model to promote research and development of LLMs through various analyses in academia and industry (Note 2). We expect to release details at a later date.

Note 1: Mixture of Experts is a model architecture that combines multiple expert models and as a result achieves even higher inference accuracy. The model was built using the upcycling technique applied to the base Sarashina2-70B model and trained using eight expert models in the field-forward network in the transformer. This combination realizes high inference accuracy.

Note 2: This released model is a pre-trained model with no instruction tuning.
Because it has not been tuned to follow human instructions, we cannot provide it as a practical service as is.
This model is licensed for non-commercial use.

URL for released model

https://huggingface.co/sbintuitions/sarashina2-8x70B

At SB Intuitions, we will continue to advance even larger LLMs and promote research and development for the social implementation of LLMs.