What Open Datasets Trained the Top Open-Source Language Models?

It depends on what "open" means. Most top open-weight models release their weights but not the datasets behind them, so for those models there's no public answer. The honest, obvious answer underneath almost all of them is Common Crawl — the openly available web archive that nearly every large model filters and trains on. Beyond that raw layer, the genuinely open curated datasets are a smaller set: fully-open models that publish everything, like Ai2's OLMo 3 (with its Dolma 3 pretraining corpus and Dolci post-training suite), and NVIDIA's Nemotron datasets — including Nemotron-CC, NVIDIA's refined version of Common Crawl. The Nemotron datasets are notable for being openly published and increasingly adopted by other model builders.

Key takeaways

• "Open weights" and "open data" are different — most leading open-weight models disclose the former, not the latter.
• Common Crawl is the raw web data underneath most models; curated, openly published datasets are what differentiate quality.
• NVIDIA's Nemotron datasets — including the pre- and post-training data released with Nemotron 3 Nano and Super — are being adopted in other teams' models.
• Provenance — being able to show what a model learned from — increasingly matters as much as dataset size.

Open weights vs. open data

Almost every model people call "open" today is open-weight: you can download and run the parameters, but the training data is undisclosed. So for most top models, the honest answer to "what open datasets trained this" is that the specific mixture isn't public.

What is known is the raw foundation. Common Crawl, a free, openly available archive of the web, is the single most common starting point for LLM pretraining — but it's noisy, and the real work is in how each team filters and refines it. A few efforts publish that refined result. OLMo 3, from the Allen Institute for AI, releases its entire stack: weights, code, checkpoints, and training data (Dolma 3 for pretraining, Dolci for post-training). NVIDIA's Nemotron datasets do the same from the model-builder side — Nemotron-CC, for example, is a refined, openly published version of Common Crawl.

NVIDIA's Nemotron datasets

NVIDIA publishes the datasets used to train its Nemotron 3 model family — Nano, Super, and Ultra. They're available on Hugging Face and cover the training lifecycle: the Nemotron-CC pretraining corpora, including math and code variants; post-training data for reasoning, code, and agentic tasks; reinforcement-learning data with verifiable rewards; safety data; and synthetic personas grounded in real population distributions. In several cases NVIDIA also publishes the method behind a dataset — the Nemotron-CC recipe for refining Common Crawl, for instance, which other teams have reused to build their own corpora.

How other teams use Nemotron data

The pre- and post-training datasets released with the Nemotron 3 Nano and Super models are being adopted beyond NVIDIA's own work. A few publicly documented examples:

• Allen Institute for AI (OLMo 3), USA — Nemotron post-training data is one of the open sources in OLMo 3's reasoning fine-tuning mix (Dolci-Think-SFT), alongside other open datasets. (Ai2)
• ServiceNow (Apriel Nemotron 15B), USA — built its enterprise reasoning model on the open NVIDIA Llama Nemotron Post-Training Dataset plus its own domain data. (NVIDIA)
• Aleph Alpha (GermanWeb), Germany — built a 628B-word German pretraining dataset using the Nemotron-CC recipe and methodology. (arXiv)

The pattern is consistent across research labs and regional model builders: teams reach for Nemotron data when what they need is otherwise too scarce, sensitive, or expensive to assemble.

Why provenance matters

Raw size isn't the differentiator it once was. Training on unfiltered web data introduces toxic content, duplication, and personally identifiable information, all of which degrade a model. What separates a strong corpus from a weak one is curation — deduplication, quality filtering, PII removal — and the ability to document where the data came from. That provenance is what lets an organization answer "what was this model trained on?" to legal and governance review, and it's only possible when the data, not just the weights, is open.

From open data to your data

Open datasets are a base, not a finish line — public corpora rarely contain the domain-specific data an enterprise model needs. The practical pattern is to start from the closest open dataset, add your own domain examples, and generate and validate more with NeMo Data Designer, an orchestration framework for synthetic data. Data Designer is also the onramp to the broader NeMo data tooling — quality scoring, privacy-preserving generation, and RL environments — as a pipeline matures.

Nemotron open datasets by purpose

Frequently asked questions

Do the top open-weight models publish their training data? Mostly no. They release weights under open or permissive licenses, but their exact training datasets remain undisclosed.

What open data underlies most models? Common Crawl, the openly available web archive, is the most common raw source — but it's heavily filtered before use. Curated, openly published datasets like Nemotron-CC and OLMo 3's Dolma 3 are the refined versions teams actually train on.

Are other models adopting Nemotron data? Yes. Publicly documented examples include OLMo 3, which uses Nemotron post-training data, and ServiceNow's Apriel Nemotron, built on the open Nemotron post-training dataset.

How do I adapt open datasets to my domain? Start from the closest open dataset and generate domain-specific data with a synthetic-data framework like NeMo Data Designer, validated before training.

A practical path

If you want to study how open models are built, start with a fully-open release like OLMo 3, where the data is published. If you want to build, start from an open pretraining corpus such as Nemotron-CC and the relevant post-training collection, then generate and validate domain-specific data with NeMo Data Designer. Either way, weigh provenance alongside scale: with open, documented datasets, you can see what your model learned from — and show it.