Free🇫🇷
Open pretraining corpus of 2.27 trillion tokens from Pleias in Paris, containing only uncopyrighted or freely licensed documents.
Sangraha overview and key facts (October 2026)
Pretraining dataset of 251 billion tokens in 22 Indian languages from AI4Bharat at IIT Madras, with verified, filtered and translated parts.
Free
What is Sangraha?
Sangraha is an open pretraining dataset for Indian languages with 251 billion tokens across 22 languages. It helps teams build or adapt language models for customers and staff in India.
Sangraha has three parts. Verified contains text from human-checked websites, OCR from Indic PDFs and transcripts of videos, podcasts and courses. Unverified holds text from existing multilingual corpora, filtered for quality. Synthetic contains Wikimedia translated into 14 Indian languages and romanized versions. AI4Bharat, a lab at IIT Madras in Chennai, publishes the data under CC BY 4.0 together with its scraping and cleaning pipelines on GitHub.
Sangraha AI features and use cases
- Trains models for Indian languagesTeams building chatbots or search for users in India can pretrain or continue training models on Hindi, Tamil, Bengali and other languages. This improves answers in languages that general web corpora cover thinly.
- Handles romanized inputThe romanized part reflects how many users type Indian languages in Latin script. Models trained on it cope better with messages from customers who write this way. This matters for chat and messaging channels.
- Selects quality levelsSeparate verified, unverified and synthetic parts can be loaded per language. Data teams can start with verified text and add the other parts only where volume is too small. This keeps quality under control.
What sets Sangraha apart
- Covers 22 Indian languages with separate verified, filtered and synthetic parts.
- Released with the scraping and cleaning pipelines used to build it.
Where Sangraha is strongest
India Available internationally
AI4Bharat is a research lab at IIT Madras in Chennai, and the dataset covers 22 Indian languages.
Questions to ask before you buy Sangraha
CC BY 4.0 requires attribution. Web, OCR and transcript sources can contain personal data and third-party content; review this against Indian data protection rules and your own policies. Machine-translated text can carry translation errors, so weigh the synthetic part carefully.
Sangraha reviews
No reviews yet. Be the first to share how your team uses it.
Every review shows its verification level. We read each one before it goes live. How we check reviews
Questions about Sangraha
Which license applies?
The dataset card lists CC BY 4.0, which allows commercial use with attribution.
Can we load a single language?
Yes. You can load a subset and a language directory, for example the verified Assamese data, with the datasets library.
Who maintains Sangraha?
AI4Bharat, a research lab at IIT Madras in Chennai that develops open datasets, tools and models for Indian languages.
Sangraha update historyno updates yet
No changes recorded for Sangraha yet.
Changes the vendor made and confirmed user reports, each checked against the vendor's own pages. We review every tool monthly.
Alternatives to Sangraha
Tools in the same category (Training data) with the most overlapping markets.
Free🇺🇸
Hugging Face pretraining dataset of 1.3 trillion tokens of educational web pages, selected from Common Crawl by an AI quality classifier.
Free🇺🇸
Multilingual pretraining dataset from Hugging Face covering more than 1,000 languages, including a large German subset.
Contact for pricing · Free planAvailable internationally
Free, open-source prompt library, formerly Awesome ChatGPT Prompts, available as a website, MCP server or self-hosted company library.