Knowledge graphs provide this structure and have become a key component for grounding AI agents for financial reasoning. This allows users to navigate complex financial scenarios assisted by AI agents where every answer is traceable to verified source data, with reliability and regulatory compliance implications.
Building knowledge graphs agentically involves several steps, as we have previously described in FinReflectKG: spotting candidate facts, checking them, formatting the extracted facts in the form of a triple, that is, two entities and the relationship between them. Agentic pipelines performing these tasks rely on the ability of the underlying models to perform these steps, either in isolation or in combination, when presented with passages from 10-K filings.
For a knowledge graph to connect information across passages and filings, recognising the facts is only part of the task. The model also needs to express them using consistent entity names and relationship labels, which constitute the ontology against which the knowledge is organized.
In order to improve precision and recall of models to extract structured information against a specific ontology, we fine-tuned NVIDIA Nemotron 3.5 Super VL to read a passage of a 10-K and write down its facts as structured knowledge graph triples, such as (NKE, Discloses, Revenue). We trained the model with the NVIDIA Megatron-Bridge and its fine-tuning recipes, and ran it with vLLM for evaluation.
The workflow we put in place is an example on how one would specialize a model in specific subdomains and for specific tasks, for instance on filings from seven companies the model never saw in training set, scored against the public FinReflectKG ontology. Our results show that the fine-tuned model finds more reference facts and expresses them consistently with the required ontology.
Specifically the fine-tuned model:
- Matches twice as many of the benchmark's reference facts in one pass, from 16% to 33%, surpassing both its base as well as closed frontier models at the same task.
- Gets a larger share of its triples right on the average passage, as graded by a large language model judge: from 64% to 72%.
- Gains about as much in industries not included in the training set, it matches 17 points more of the reference facts in a test company in energy and healthcare, respectively.
- Makes less than half as many rule violations, on 3.8% of its triples instead of 8.7%.
These results came from training on a relatively small set of 1,978 examples, showing that the process can take place on a case-by-case basis with limited overhead.
The following sections describe the methodology and the results in detail and provide a concrete example from a financial filing.
Constructing knowledge graphs from 10-K filings
Building a knowledge graph on 10-K filings means identifying and extracting facts like which company makes which product, which segment reports which revenue, which risk touches which business. The public FinReflectKG pipeline splits each filing into passages of a few paragraphs or one table, with a median of 330 words in length. It extracts the facts of each passage with several LLM calls, such as an extraction, then rounds of critique and correction. Every fact uses a fixed list of allowed relations and entity types, with facts from different filings connecting into a global knowledge graph. Our goal was to fine-tune Nemotron 3.5 Super VL so as to improve zero-shot extraction for specific ontologies.
Assessment metrics
The performance of the base and fine-tuned models has been assessed with the following methodology.
Recall is the share of the benchmark's reference facts that the model finds. The reference facts are the triples the benchmark counts as correct. These were obtained in FinReflectKG using a LLM-based pipeline. Recall comes in three levels of strictness: pair recall needs the two entities of a reference fact; triple recall also takes the relation into account; typed recall also needs the entity types, the category of each entity. Each recall is the average recall over all passages.
An LLM judge is a second language model that grades the output with the grading guide of the benchmark. We use Qwen3-32B because the FinReflectKG construction paper used it as its judge. We give it the rubric prompts of FinReflectKG-EvalBench, in the single-judge setup. For each triple it gives three grades: precision (the fact is correct), faithfulness (the text supports it) and relevance (the fact matters for the filing). For the whole passage the judge returns a score from 1 to 3 for how much of the content the triples cover (comprehensiveness). Each judge's score is the average over passages. Final results were sanity-checked using human evaluation.
Rule violations are triples that break at least one rule represented in the prompt: a rule of the ontology, such as using only allowed relations and naming the filing company by its stock ticker (NKE for Nike), or an output rule, such as the length of an entity name. The rule violation rate is the share of triples with at least one violation.
How we did it
Starting from Nemotron 3.5 Super VL
Nemotron 3.5 Super VL is the successor to Nemotron 3 Super in NVIDIA's Nemotron family. Before any training, it already does better than Nemotron 3 Super in the same 353 test passages:
The new model finds more of the right entity pairs, covers more of each passage, and has a rule violation rate less than half as high.
The base model does not use the ontology labels. Triple recall stays at 16%. When Nemotron 3.5 Super VL misses a reference fact, it usually names the company correctly but writes the other entity differently from the benchmark, or leaves it out.
The provided Nemotron 3.5 Super VL model has sufficient common knowledge for this task. However, through fine-tuning, we were able to teach the model how to extract the ontology ruleset.
The training data
We started from FinReflectKG-EvalBench, the evaluation release of FinReflectKG, and extracted a subset of the samples as the training set, keeping it separate from the actual evaluation set. Unlike the main FinReflectKG dataset, FinReflectKG-EvalBench ships each triple with the verdicts of LLM grader as judge on precision, faithfulness and relevance. We kept the 38 companies where at least 30% of the triples passed all three verdicts, because a lower rate points to a failed grading run. Within those, we kept the 2,435 passages with the top comprehensiveness grade, at least 70% of the triples accepted by all three verdicts, and every relation one of the 51 in the ontology. That is 32% of those companies' passages with reference facts.
Each passage and its accepted triples make one training example. We split the examples by company, so the test measures the model on companies it has never seen. Our split differs from the one in the FinReflectKG papers, so our numbers do not compare directly with the published results. The validation set is a small held-out set that we check during training:
Two test sectors, energy and healthcare, have no company in training or validation.
The training stack, Megatron-Bridge and vLLM
We trained with low-rank adaptation (LoRA). The original weights of the model stay frozen, while training adjusts a small set of added weights. NVIDIA Megatron-Bridge has a LoRA recipe for Nemotron 3 Super, the same recipe worked with Nemotron 3.5 Super VL weights without change. We ran the base model and the fine-tuned model with the same vLLM version and parallel layout on a GPU cluster, through domyn-swarm — our public tool for running vLLM on clusters that use Slurm, a common job scheduler for supercomputers.
Results
Example result on a real filing
This passage from Bristol Myers Squibb's 2024 10-K is a table of what the company paid for an acquisition. Both base and fine-tuned models find the same six-line items, with different labeling:
The base model links every line item with "Has_Value," a generic relation. The LLM judge rejects all six on precision, with the reason "generic relation and missing value," because the triple says the company has a value but the value doesn’t appear. The tuned model uses "Discloses," the relation the reference facts used for a disclosed amount. Both relations are in the ontology, so this is a convention that the ontology leaves open, and the fine-tune model correctly captures it.
Results on the test set
The same shift shows in all 353 test passages. Triple recall doubles and judge ratings similarly increase. The last column gives the 95% confidence range for each change:
The gain in recall did not cost precision or faithfulness. Triple recall gains more points than pair recall. When the tuned model finds the right pair, it also picks the benchmark's relation more often, as in the Bristol Myers Squibb example.
Where the base model did not match any reference fact, the tuned model often does. In 124 passages, the base model matched no reference fact. In 89 of those passages, the tuned model matched at least one, and in 36 of them it matched half or more. Across all passages, on triple recall score, the tuned model has a higher triple recall than the base model in 224 passages and a lower one in 35.
The results above show that fine-tuning significantly outperforms the base Nemotron 3.5 Super VL model. At the same time, absolute extraction performance shows that the end-to-end task requires a harness and cannot be performed end-to-end in a single shot. This led us to ask: do state-of-the-art closed models perform better in these regards on this test set, or do they face similar ceiling effects?
Closed models on the same test set
To contextualize these results against proprietary industry baselines, we evaluated top-performing commercial closed models under identical conditions. Specifically, we ran GPT-6.1 Sol and Gemini 3.5 Flash on the same 353 test chunks, with the same prompt and the same scorer. Neither model can turn its reasoning off, so each ran at its lowest documented setting: "low" for Sol and "minimal" for Flash. We also ran both at their default, medium. The calls went through an OpenAI-compatible endpoint that does not pass a temperature, so the closed models ran at the provider default.
The fine-tuned model has twice the triple recall of the best closed row. GPT-6.1 Sol at low trails Nemotron 3.5 Super VL on every recall level, with triple recall 4.20 points lower (1.75 to 6.69). Both closed models have fewer rule violations. More reasoning does not raise recall: both closed models score lower at medium than at their lowest setting.
Results on sectors not in the training set
The fine-tune also lifts triple recall in sectors that are not included in the training set. On two test companies, each on a new sector (healthcare and energy), triple recall rose from 17% for Nemotron 3.5 Super VL to 34%, against 16% to 32% in the four sectors that training covered.
Left bar of each pair: Nemotron 3.5 Super VL before the fine-tune. Right bar: fine-tuned. The shaded sectors have no company in training or validation. Each sector holds one or two test companies.
Conclusion: The value of task-specific fine-tuning
With just 1,978 training examples, fine-tuning NVIDIA Nemotron 3.5 Super VL doubled reference-triple recall, improved precision and faithfulness, and more than halved rule violations. Improvements also extended to companies and sectors absent from training. The findings demonstrate the value of specialisation for financial knowledge extraction: teaching a model not only to identify facts, but to express them consistently within the structure required by a knowledge graph based on a specific ontology. They also show that even a narrow, well-defined task can yield substantial gains from targeted fine-tuning, while only requiring a limited amount of high-quality data.
Acknowledgments
We'd like to thank NVIDIA for Nemotron 3.5 Super VL and Megatron-Bridge, as well as the authors of the vLLM project and the FinReflectKG for the public data and the benchmark.
Resources
- NVIDIA on Hugging Face, for the Nemotron 3.5 Super VL model card.
- Megatron-Bridge, with the LoRA recipe.
- lm-evaluation-harness, the framework that runs our scorecard task.
- FinReflectKG paper: the knowledge graph construction
- FinReflectKG EvalBench data on Hugging Face.
- domyn-swarm, for running vLLM on Slurm clusters.
Frequently Asked Questions
