Teaching language models to replicate expert investor judgment
Can a language model be taught to triage financial documents the way an experienced investor would? Thinking Machines AI says yes, with high-quality human annotations, its proprietary model beat every frontier model tested on information filtering tasks drawn from investors’ daily workflows, while costing a fraction as much to run.
The company published a write-up focused on a narrow case: filtering and processing financial documents to surface information relevant to investment decisions.
Why investor judgment is hard to automate
Outperforming the market is difficult when every investor has access to the same public sources. Alpha has to come from unique insight built on taste and judgment. That judgment is difficult to articulate and teach, whether the audience is another person or a language model, because it comes from experience.
Even when an investor’s job is broken into its simplest tasks, those tasks turn out to be surprisingly hard for off-the-shelf large language models.
What is the real work in investing?
Investors are bombarded each day with news articles, research reports, company filings, emails, and internal write-ups. Reading is the easy part. The harder work consists of the small, repeated judgments layered on top: filtering, interpreting, segmenting, and identifying where the useful signal lies.
The team’s goal was to automate that triage step, flagging what is relevant and interesting to read, so investors can spend their attention on higher-level synthesis and decision-making.
Can LLMs learn financial judgment?
Given that off-the-shelf LLMs perform poorly on simple financial tasks, the team asked whether it is possible to teach those models financial judgment directly. Their finding: with high-quality human annotations, language models can be trained to interpret text with expert-level taste and judgment.
The company reports that its proprietary model outperformed all frontier models tested on information accuracy and recall, at a fraction of the cost, on a subset of data cleared for public release.
How were frontier models evaluated?
Models were evaluated on six information filtering tasks drawn from investors’ daily workflows, with internal tasks showing similar patterns. The team measured accuracy, defined as the percentage of documents correctly labeled according to their investors, and, for classification tasks, the F1 score.
One example task: given a financial article, classify whether it is relevant to a C-suite investment professional, scored on F1 and accuracy.
Toward differentiated intelligence
Based on the results, the team sketches a broader vision of differentiated intelligence: models tuned for specific organizational needs rather than deployed as general-purpose assistants. The financial judgment work is positioned as an early example of that direction, where domain expertise encoded in training data can outperform general frontier models on narrow, high-value tasks.