Delivering sovereignty and trustworthiness meant building capabilities at every layer, with NVIDIA as a key partner in helping us get there.
For our model layer development, NVIDIA’s early bet on open source provided a foundational framework that eliminated the need to develop core infrastructure from scratch. They'd already open-sourced a large part of the training stack, and we built both our general-purpose and specialized models on that foundation. What follows is how that collaboration shaped our process, and why organizations seeking sovereignty don't have to start from scratch either.
The early beginnings
At Domyn, we started developing our own models in 2023. The first one was Italia 9B, a model of the Llama 2 era. For training, we used LitGPT, a popular open-source framework for training and fine-tuning LLMs.
The data mix used was deliberately heavily biased towards Italian sources, to match the quality of language-specific models of the time. More importantly, it allowed us to set up a first version of our curation, pretraining, instruction tuning, and evaluation pipelines.
At the same time, since the model lacked more general capabilities, it became clear that we needed broader data and more scale. We found both when we embraced the NVIDIA NeMo open-source libraries stack, which paved the way for our next round of models, Italia 10B and Colosseum 355B.
Scaling on top of the NVIDIA NeMo open-source libraries
The collaboration with NVIDIA proved to be instrumental throughout the scaling phase. The state-of-the-art NVIDIA Megatron-Core library had been built for throughput and stability at a scale very few organizations operated at back then. We adopted a shared architecture, tokenizer, and data strategy for both models.
Colosseum 355B targeted multilingual capability at frontier scale, with a 256k token vocabulary, more than 50 languages supported, and a deliberate upweighting of European languages.
Colosseum 355B allowed us to develop large-scale, distributed training expertise head-on. At this scale, distributed checkpointing became a challenge to tackle actively, as parallel filesystems failed in invisible ways until several hundred nodes hit them at once, and a single degraded node could drag an entire synchronous job down.
In this setting, NVIDIA's battle-tested stack helped take care of the problems we were facing. It allowed us to get MFU past 40% through parallelism restructuring, computation-communication overlap, and FP8 training on the first Hopper clusters.
The multilingual effort allowed us to develop data processing as an industrial operation, sifting through hundreds of terabytes of raw text across more than 50 languages. This is also where we laid down the basis for compliance with the EU Copyright directive and EU AI Act obligations, through ML classifiers and rule systems running over whole corpora, provenance tracking, and strict honoring of opt-out requests.
We also experienced fast growth across the evaluation phase of the process. We built Domyn Swarm to run inference at that scale on HPC, which was released as open source. For the first time, we developed our own evaluation suite, Domyn Eval, which extended already-existing frameworks, such as NVIDIA NeMo Gym (formerly NeMo Evaluator), and reused benchmarks where possible. In other cases, we built our own: Evalita-LLM was created with FBK and the University of Turin. We also relied on custom evals on enterprise tasks such as text-to-SQL, text-to-Cypher, knowledge graph triplet extraction and compliance classification.
The advent of thinking
By 2025, "thinking" had made its way into frontier models. Colosseum 355B was a capable model but it was trained to answer directly, without leveraging the inference-time compute that reasoning brought forward. Thinking brought the need for more extended context than what Colosseum originally offered. And since generation ran longer, the costs had to come down accordingly.
Colosseum 355B was a solid model, so we used it as a basis to build a stronger model that checked all the boxes. That's how Domyn Large emerged.
Domyn Large
The first step to build Domyn Large was to prune Colosseum's 355B parameters to 263B. The second was to distill from the pruned model to recover the capabilities that were lost.
On the first attempt, we cut a fifth of the layers and distilled for 1,000 steps, which led to a performance drop of roughly 15 percentage points with no recovery. We fixed it by distributing the reduction evenly: 10% along depth and 10% along width, which cost only 2.3 points on the final run. Splitting this into two sequential lighter cycles gave no measurable gain over a single cut. Also, we found a 0.2 distillation / 0.8 language-modeling loss weighting clearly best, and recovery saturated fast. Still, the damage on our benchmark suite was uneven: multilingual and coding tasks suffered most while knowledge-heavy benchmarks like MMLU were barely affected. In practice this meant not only choosing how much to compress but which capabilities to buy back afterwards.
Once the model was pruned, we pushed its context further through long-context continued pretraining, with the sequence length raised through a curriculum. This phase allowed us to push the performance of Domyn Large past that of Colosseum.
On the enterprise suite from the previous section, Domyn Large recorded the highest execution accuracy on both text-to-SQL and text-to-Cypher, outperforming its peers Qwen3 253B and GPT-OSS 120B.
The whole process ran on roughly half a billion distillation tokens and 50 billion continued pretraining tokens, against the 11 trillion tokens that produced the base model. NVIDIA helped with research frameworks, including Minitron and Puzzle, both of which are techniques within the NVIDIA Model Optimizer, which we contributed to extend and adapt.
Beyond obtaining a capable model to use in our vertical applications, Domyn Large proved that we could effectively steer models and let them acquire capabilities they didn't originally have. As usual, the key was a strong data strategy and an even stronger evaluation strategy.
Domyn Small
Following Domyn Large’s release, the next goal was to build a strong, smaller, multilingual model for the agentic world that was materializing. Ideally an heir to Italia 10B that we could use for vertical specializations in our customer use cases.
Since the model needed strong agentic capabilities, we turned heavily to post-training. Domyn Small was built on top of Italia 10B through a multi-stage pipeline: continued pretraining to extend the context window, supervised fine-tuning to teach the tasks, then reinforcement learning to make the model reliable at them.
Following continued pre-training (CPT), supervised fine-tuning (SFT) yielded the most significant findings. A consistent pattern emerged across dozens of full training runs with varying data mixtures: broad domain mixtures represented a fundamental trade-off that could not be resolved through hyperparameter tuning. Beyond a critical sample threshold, mathematics and code domains exhibited mutual interference, with every blend sacrificing performance in at least one area. This motivated the subsequent approach of training specialized per-domain teachers, subsequently combined via multi-teacher on-policy distillation.
The next phase employed reinforcement learning (RL) to execute multi-environment Group Relative Policy Optimization (GRPO) with verifiable rewards, complemented by preference optimization.
Two key observations emerged. First, learning rate selection must precede all other hyperparameter decisions. GRPO exhibited sensitivity to learning rate mis-specification, with divergence potentially manifesting late in training; extended warmups masked this instability, instead of mitigating it. System learning rate search requires a short warmup. This contrasted sharply with SFT, where learning rate variations produced negligible result differences.
Second, the functional roles of SFT and RL should be clearly delineated. RL improved pass@1 reliability but did not expand the capability ceiling: at pass@1024, RL-optimized models solved no additional problems beyond the base model's reach, regardless of algorithm (see "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?", arXiv:2504.13837). The practical implication was a division of labor: maximized pass@k during SFT, where new capabilities are acquired, then applied RL to convert that coverage into pass@1 reliability. Unlike RL, the distinction shifted the pass@k curve, providing the complementary rationale for specialized teacher training.
What NVIDIA brought to the project
The NVIDIA stack helped us twice in this process. First, NVIDIA NeMo RL gave us an open-source reinforcement learning library built for distributed training with online generation, which we later evolved into our Domyn RL stack.
Second, NVIDIA's open post-training dataset — released alongside Nemotron models — substantially lowered our synthetic data generation bill. This included high-quality bulk coverage across mathematics, code, reasoning traces and instruction following, and allowed us to focus synthetic data generation (SDG) on the specific gaps our evaluations identified.
All of it ran on CINECA’s Leonardo Supercomputer — public European infrastructure. The result is a model that reaches quality comparable to Qwen 3.5 9B on several reasoning tasks while producing roughly 33% fewer tokens. Accuracy per token is a metric that was later adopted by other teams to report model efficiency.
We released the model openly under an MIT license in early 2026, including weights, training recipe, and details about data, as captured in detail in the Domyn Small Technical Report.
In this process, we ensured that all our models were EU AI Act-compliant. Specifically, we submitted a full Safety and Security Model Report for Domyn Large to the European Commission AI Office in March 2026.
What comes next
Looking ahead, two exciting developments are already in the pipeline for the near future.
The first will be the release of Domyn Edge, a strongly agentic 4B-parameter model with 1M context length, trained from scratch on over 13T tokens. It features a carefully chosen architecture and state-of-the-art methodologies. Domyn Edge will be released in October 2026, and we are very excited about the early results.
Domyn Edge will add a highly customizable model that can be deployed flexibly behind an endpoint or on device, capable of taking on highly specialized, long-horizon tasks.
The second development is EUROPA, an open frontier model, trained on European infrastructure and developed together with the Fraunhofer institute as part of the European AI Grand Challenge. Release is planned for the second half of 2027, and it will be an unprecedented effort on EU infrastructure.
Both projects lean on the same foundation that got us here. NVIDIA's open NeMo libraries accelerated how we built our first models, and it's shaping how we build the next ones too — sovereignty, built with partners whose significant contribution left a mark well past the symbolic.
Sovereign AI is a muscle
Training LLMs is a craft: beyond demanding computational resources, it requires building expertise through extended experimentation and accumulated experience.
- Domyn Small: model card and technical report: https://arxiv.org/html/2607.20448v1
- Domyn Large: joint technical blog with NVIDIA: https://www.domyn.com/blog/domyn-large-the-journey-of-a-european-sovereign-ai-model-for-regulated-industries
- Colosseum 355B: joint technical blog with NVIDIA
- Evalita-LLM: arXiv 2502.02289
- Enterprise task evaluation: methodology, code and datasets on Hugging Face: https://huggingface.co/datasets/domyn/Enterprise-Task-Evaluation
- Domyn Swarm: open source LLM serving on HPC https://github.com/igeniusai/domyn-swarm
Frequently Asked Questions
