A post on the Hugging Face Blog outlines an effort to scale pretraining for Indic languages, reporting a dataset scale of 2.8 trillion tokens. The work is accompanied by an open data generation pipeline.
Why it matters
Large-scale, openly described data pipelines for Indic languages address the volume of training data available for languages that are often underrepresented in model development.
Who should care
Researchers and practitioners working on multilingual and Indic-language models, as well as teams interested in open data generation methods, may find the described scale and pipeline relevant.