FINCH: Financial Intelligence using Natural language for Contextualized SQL Handling
Artificial Intelligence for financial services has faced unique challenges when it comes to adopting Natural Language Processing (NLP) technologies. Financial data is often complex, tightly regulated, and deeply nuanced, making it difficult for general-purpose AI models to deliver accurate and reliable results.
Domyn’s research team addressed this gap by introducing a dataset and evaluation metric designed to enhance the application of natural language processing (NLP) to financial Text-to-SQL tasks – the conversion of natural language questions into database queries. Developed by Bhaskarjit Sarmah, Stefano Pasquali and Avinash Kumar Singh, FINCH, the large-scale, curated financial dataset, consists of 292 tables and over 75,000 natural language-to-SQL query pairs, which enables both fine-tuning and rigorous benchmarking of financial AI models. Accompanying the dataset is a novel evaluation metric, the FINCH Score, which captures nuances often missed by traditional metrics, to provide a more reliable assessment of model performance in financial contexts.
Surprisingly, this study revealed that smaller, domain-specific models could outperform much larger general-purpose LLMs. For example, Arctic-Text2SQL-R1-7B, a relatively compact model emerged as one of the top performers, demonstrating that adapting to a specific domain can be more effective than increasing model size alone.
If adopted widely, FINCH could bring a significant shift in how financial institutions build and evaluate AI systems. Instead of relying on generic models trained on broad datasets, companies could prioritize purpose-built models tested and certified with finance-aware metrics.
Co-authored by Bhaskarjit Sarmah, Stefano Pasquali, and Avinash Kumar Singh, the research examines critical weaknesses in current benchmarks and evaluation methods, and proposes a more rigorous, forward-looking framework to power the next generation of reliable AI solutions for financial services.
