Teaching language models to replicate expert investor judgment
Can a language model be taught to triage financial documents the way an experienced investor would? Thinking Machines AI says yes, with high-quality human annotations, its proprietary model beat every frontier model tested on information filtering tasks drawn from investors’ daily workflows, while costing a fraction as much to run.
The company published a write-up focused on a narrow case: filtering and processing financial documents to surface information relevant to investment decisions.
Why investor judgment is hard to automate
Outperforming the market is hard.
Even on simple investor tasks, off-the-shelf large language models often struggle.
What is the real work in investing?
Investors face a daily stream of news articles, research reports, company filings, emails, and internal write-ups. The harder work is making the small, repeated judgments: filtering, interpreting, segmenting, and finding the useful signal.
The team’s goal was to automate that triage step, flagging what is relevant and interesting to read, so investors can spend their attention on higher-level synthesis and decision-making.
Can LLMs learn financial judgment?
Given that off-the-shelf LLMs perform poorly on simple financial tasks, the team asked whether it may be possible to teach those models financial judgment directly. Their finding was that high-quality human annotations can be used to train language models to interpret financial text more accurately.
The company reports that its proprietary model outperformed all frontier models tested on information accuracy and recall, at a fraction of the cost, on a subset of data cleared for public release.
How were frontier models evaluated?
Models were evaluated on six information filtering tasks drawn from investors’ daily workflows, with internal tasks showing similar patterns. The team measured accuracy, defined as the percentage of documents correctly labeled according to their investors, and, for classification tasks, the F1 score.
One example task: given a financial article, classify whether it is relevant to a C-suite investment professional, scored on F1 and accuracy.
Toward differentiated intelligence
The team describes a direction it calls differentiated intelligence, in which models are tuned for specific organizational needs rather than deployed as general-purpose assistants. The financial judgment work is described as an early example of that direction. The team suggests domain expertise encoded in training data can sometimes outperform general frontier models on narrow, high-value tasks.