At Domyn, we care about helping organizations own their AI. To that end, we provide them with models that are capable straight out of the box, cost-effective to use, and able to specialize on agentic tasks.
Over the last few months we have been training a new 4B-parameter language model from scratch, Domyn Edge. Today, we are launching the first phase of a rolling release that includes not only the weights, but the datasets, intermediate checkpoints, and training code. All will be made available under open licenses so the community can audit, reproduce, and build on our work.
Domyn Edge was trained from scratch on 13 trillion tokens, end-to-end on European public infrastructure. It is built for agentic work at long context, trained up to 1M tokens and recommended at a 256k working context.
The model was trained on Leonardo, the EuroHPC supercomputer hosted by CINECA. By releasing the intermediate checkpoints, training recipes, tooling and data we used, we push to foster the European AI ecosystem and show how public infrastructure can be utilized to build AI capabilities for the agentic era.
A phased release
Domyn Edge is being released in phases. This first phase covers the pre-trained base model, its intermediate checkpoints, the student checkpoint and the training codebase. Over the course of the next few weeks, the final post-trained model, data, and remaining research artifacts will follow, as shown in Figure 1.
Open by design, not default
Most small models today are open weights: you can run them, but you cannot see how they were made, audit them or reproduce them, which hampers the ability to take them in new directions to serve the needs of vertical industries. We built Domyn Edge to enable all this and be the strongest fully open model in its class.
Full transparency across weights, code, and datasets
By releasing our training datasets alongside intermediate checkpoints and code, we make it possible to inspect, audit, reproduce, and continue training the model end-to-end — a necessary condition for regulated organizations to own their models and for research groups to build on them.
Built on public compute
Domyn Edge was trained on Leonardo with compute allocated through the EuroHPC AI Factories programme: about 2.3 million GPU-hours for the pre-training run, and around 5 million across the whole project, from ablations and data generation to post-training and evaluation.
Building Domyn Edge also meant building tools to run LLM workloads on shared HPC clusters, which are released as part of the Domyn Edge release bundles. Domyn Swarm serves LLM inference at scale on SLURM, for synthetic data generation and evaluation. SDAG defines and runs pipelines as DAGs on SLURM, with task caching. Domyn Forge, the training code of Domyn Edge, joins them today. Our data processing and evaluation tooling will follow.
Small enough to own
At 4B parameters, Domyn Edge runs on a single GPU, on premise or air-gapped, and is cheap to fine-tune or otherwise post-train on your own data.
Architecture and pre-training
Domyn Edge is a 36-layer dense transformer with grouped query attention. The attention is interleaved, three sliding-window layers for every full-attention layer, with only the sliding-window layers carrying rotary embeddings. The full-attention layers are also given a larger head size, so capacity sits where the model attends widest. Full hyperparameters are in the model card.
The tokenizer extends the GPT-4 BPE vocabulary with custom tokens for the five main target languages: English, Italian, Spanish, French, and German. This reflects our goal of streamlining tokenization across key European languages while maintaining strong efficiency on English and code.
As shown above, training ran in six stages over 13 trillion tokens:
- Pre-training to 8.5T tokens, on a mixture of web, code, STEM, multilingual and synthetic data at 8k sequence length.
- Specialized pre-training to 10.1T, raising the weight of high-quality reasoning, maths and code data.
- Context extension to 10.3T, the first step toward long context.
- Mid-training to 12.3T, on 32k sequences, adding agentic tool-calling and coding trajectories.
- Context extension to 12.7T, the second step, reaching a 1M-token training context.
- Annealing to 13T, the final decay.
We are releasing a checkpoint every trillion tokens through pre-training, plus the endpoint of every subsequent stage, so that the development of each capability during training can be studied.
Training ran on 128 nodes of 4 NVIDIA A100 64GB GPUs, 512 GPUs in total, under TorchTitan with a hybrid FSDP intra-node and DDP inter-node strategy.
The training code is released as Domyn Forge, which uses TorchTitan as a dependency. In the first release bundle, we ship the pre-training scripts and recipes, so continued pre-training of Domyn Edge on your own data is reproducible from the same code we used.
A comprehensive technical report will be published with the final release bundle, covering data lineage, dataset mixtures, sampling weights for every stage, architectural and tokenizer ablations, as well as key lessons learned. For the time being, all additional information is available in our Hugging Face model cards, which we will keep on updating as new releases land.
The SFT model and the post-training protocol
The first release bundle also comes with the SFT checkpoint: the base model after a broad supervised fine-tuning pass over maths, code, tool calling, and instruction following.
We call it the student model because of the Multi-Teacher On-Policy Distillation (MOPD)[1] phase that will follow, according to our multi-teacher post-training protocol. Concretely, we train specialized teachers with SFT and GRPO on individual capability domains, then distill them into a single student. The checkpoint released today is the starting point of that process, not its final output. As such, the checkpoint released today has not undergone any reinforcement learning-based post-training.
Already a capable model in its current form, we are releasing it as part of the first bundle for organizations to try it and start building on it. We will later release all teachers specialized on their respective domains as part of the second bundle, as well as the final student in the third. We are confident that this will provide an invaluable resource for researchers and builders in the ecosystem.
Benchmarks
Base model
At the end of pre-training, Domyn Edge is the strongest fully open model in its class. In the sub-4B space (Domyn Edge is just under 4B parameters) we compare against SmolLM3-3B, the only other model of this size released with its datasets and training recipes: the base model is ahead of it on ten of the fourteen benchmarks below, level on one, and behind on Winogrande, basic skills and Spanish Global-MMLU. It holds the same line against the open-weights field, ahead of Gemma 3 4B on eight benchmarks out of ten and trading evenly with Ministral 3 3B, six benchmarks to four. Qwen3.5 4B is the reference for the class and leads on several benchmarks, with Domyn Edge base narrowing the gap on Winogrande and MBPP
Note: Bold indicates the overall best score; underline indicates the best fully open-source model. All models shown are base models before post-training.
For long-context evaluation, we report RULER benchmark results at the final checkpoint across various context lengths, comparing our performance against Qwen3.5 4B.
Initial usable context length was bounded at 8k tokens. A two-phase context extension over 0.5T tokens elevated long-context capabilities to perform strongly against Qwen3.5 4B, followed by targeted annealing that achieved peak scores across context lengths up to 256k tokens.
SFT (pre-RL student) model
Below, we report benchmark evaluation results for the student model prior to reinforcement learning. Because this pre-RL checkpoint lacks post-training alignment, code and mathematics capabilities are evaluated using pass@4 and pass@16 metrics.
Note: Bold indicates the overall best score; underline indicates the best fully open-source model. Benchmarks compare the unaligned student model against fully post-trained baselines, as our final post-trained model will be released in bundle 3.
Compared to SmolLM3-3B, the only other fully open model in this size class, the student model demonstrates substantial gains, leading on 27 of 28 shared benchmarks.
Against the fully post-trained Qwen3.5 4B, this initial checkpoint already matches or exceeds performance on core quantitative and coding tasks (e.g., AIME 2025/2026 pass@16 and MBPP pass@4). While expected gaps remain on knowledge-intensive (MMLU-Pro, GPQA Diamond) and single-attempt agentic benchmarks (IFBench, τ²-bench/tau3), these results demonstrate a highly promising foundation that we expect to close with the final model release.
Test it Yourself
Domyn Edge is a very strong model in its category, one you can fully adapt, control, and deploy on your own terms. It provides a highly capable, small-footprint solution for custom fine-tuning, private on-premise execution, and deployment in regulated environments requiring EU AI Act compliance. Thanks to its fully open design, it also serves as an invaluable resource for researchers studying capability formation during training.
With Phase 1 available today, you don't need to wait for the full rollout to start: take the base model, fork an intermediate checkpoint, or begin fine-tuning the SFT checkpoint now - or just plain use it as is. As Phases 2 and 3 land, you'll have the entire pipeline in hand — datasets, teacher models, and the final post-trained release — to audit, extend, or rebuild from the ground up. We're looking forward to seeing what the community builds with it, and even more to hearing what you find along the way.
Acknowledgments
Domyn Edge was trained on Leonardo, hosted by CINECA, with compute allocated through the EuroHPC AI Factories program. We thank CINECA and the Leonardo team for their support throughout training.
Resources
- Base model: domyn/Domyn-Edge-Base
- Pre-training datasets: domyn/Domyn-Edge-Pretraining-Mix-8.4T
- Pre-training checkpoints: domyn/Domyn-Edge-Base-checkpoints
- Student model: domyn/Domyn-Edge-SFT
- Training code: Domyn Forge on GitHub
- HPC tooling: Domyn Swarm and SDAG on GitHub
- Collection: Domyn Edge on Hugging Face
Reference
[1] Ma, W., et al. (2026). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration. arXiv:2606.30406.
Frequently Asked Questions
