📄 Paper | 📊 Information | 💻 Code
Right now we’re releasing Previous Concepts, Novel Issues, a brand new research of how reliably massive language fashions can consider the novelty of scientific analysis concepts. Alongside the paper, we’re publishing a benchmark, information, and code for testing novelty judges, plus an automatic pipeline researchers can use to construct new analysis units.
At Ai2, we’ve been growing AI programs that help completely different elements of scientific discovery. DataVoyager helps researchers discover and analyze structured information, AutoDiscovery generates and checks hypotheses, and Theorizer synthesizes proof throughout massive our bodies of literature into candidate scientific theories. Along with Asta, these tasks level towards programs that may assist researchers transfer from understanding prior work to proposing, testing, and refining new concepts.
However producing analysis concepts is just a part of the problem of automating scientific discovery. These concepts additionally must contribute one thing genuinely new, which suggests these programs want a strategy to distinguish a novel course from one which principally repackages present work.
That judgment is usually handed to a different LLM—what we name a novelty decide. Novelty judges assess whether or not a analysis thought is new relative to prior work, both by itself or compared with one other thought. There’s no customary strategy to construct one; analysis groups select a mannequin, immediate, analysis format, reasoning funds, and generally a literature-retrieval system. These decisions can materially have an effect on the decision. But these judges are sometimes validated solely on human-written papers fairly than the AI-generated concepts they’re really meant to judge.
Our new benchmark outcomes counsel novelty analysis itself is way from dependable. Throughout six LLM judges – GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 – seemingly small modifications to the analysis setup produce dramatically completely different judgments of the identical concepts. In a single experiment, altering just some phrases in a immediate shifts a mannequin’s accuracy in figuring out which of two concepts is extra novel by greater than 50 share factors. Retrieval and extra reasoning don’t constantly enhance judgments, and two purpose-built novelty analysis programs carry out worse than a easy prompted Claude Opus 4.6 baseline.
If AI goes to play a bigger position in scientific discovery, we’d like extra reliable methods to inform whether or not it’s really proposing one thing new. Our work offers instruments for finding out that downside, whereas exhibiting how a lot stays unresolved.
Constructing a benchmark for novelty judges
To check how reliably novelty judges can distinguish extra novel analysis concepts from much less novel ones, we first wanted examples the place there was robust proof concerning the originality of the underlying scientific contribution.
So we constructed an automatic pipeline round ICLR 2026 peer critiques. Somewhat than treating paper acceptance as a proxy for novelty, we mined the critiques themselves for passages the place reviewers explicitly mentioned a contribution’s originality. That distinction is essential; a paper could be accepted with out being significantly novel so long as it’s robust sufficient in different facets. Likewise, a rejected paper with an unique contribution can produce other weaknesses.
We created a high-novelty pool of 154 human-authored analysis concepts by choosing accepted papers within the top-rated 10% of their ICLR analysis space whose reviewers positively recognized novelty and the place no reviewer disputed it. We utilized the reverse standards to construct a lower-novelty pool: 145 concepts from rejected papers within the backside 10% whose reviewers raised originality considerations with none reviewer positively figuring out novelty. In a separate test of the review-mining course of, an annotator agreed with the extracted novelty-signal polarity in 47 of fifty circumstances; not one of the disagreements would have modified whether or not a reviewer was counted as praising or disputing the paper’s novelty.
Collectively, these two swimming pools type our first analysis setting, Human-Solely, which checks whether or not novelty judges can inform aside human-authored concepts with robust proof of upper or decrease novelty. However in observe, novelty judges are more and more getting used to judge AI-generated concepts. So we created a second setting, Human+Generated.
For this pool, we paired the identical 154 validated human concepts with 154 concepts produced by a easy LLM generator. We assumed the generated concepts have been much less novel than their human-authored counterparts; these are weak labels, not verified judgments for each thought. The generator obtained solely a analysis space and an instruction to suggest a novel thought, with no literature search, instruments, iteration, or suggestions. This gave us a conservative comparability between concepts from a fundamental generator and contributions from top-10% ICLR papers whose reviewers had explicitly endorsed their originality.
As extra checks, an professional independently reviewed 30 Human+Generated pairs, closely weighted towards circumstances our strongest decide acquired unsuitable, and we rewrote the human- and AI-generated concepts into a standard construction to check whether or not judges have been merely reacting to variations in writing type. The professional most well-liked the human thought in all 30 pairs and judged not one of the generated concepts novel.
Importantly, a lot of our core findings don’t rely on the labels in any respect. If two affordable analysis setups give reverse solutions for a similar pair of concepts, that disagreement itself reveals instability.
Small modifications can produce very completely different verdicts
Utilizing our collections of human-authored and AI-generated analysis concepts, we examined how nicely six LLM judges – GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 – might consider novelty by way of each pointwise (i.e., absolute novelty scores) and pairwise (i.e., evaluating two concepts to seek out essentially the most novel) comparisons. We various one issue at a time: the immediate, reasoning funds, whether or not the decide might see associated papers, how the concepts have been formatted, what number of judgments have been mixed right into a ultimate verdict, and – for AI-generated concepts – which mannequin generated them.
The most important shifts in efficiency got here from how we prompted the mannequin to guage novelty.
For instance, we confirmed GPT-5.4 the identical Human+Generated pairs – every containing one validated human-authored thought and one AI-generated thought – with two prompts. One immediate informed the mannequin that reviewers had discovered precisely one thought in every pair to be novel and requested it to determine which one; the opposite merely requested which thought was novel. After we switched to the less complicated immediate, accuracy dropped from 93.5% to 40.9% – beneath probability – and the mannequin modified its verdict on greater than half of the identical pairs.
In contrast, the identical immediate change shifted accuracy by solely about seven share factors on the Human-Solely dataset, which pairs human-authored concepts with stronger and weaker proof of novelty. That implies a novelty decide can seem comparatively secure on human-authored analysis however behave very otherwise on AI-generated concepts.
That mentioned, human-authored concepts weren’t immune. We noticed substantial shifts on the Human-Solely dataset as nicely; the identical change moved Claude Opus 4.6 accuracy by 21 factors. The shifts have been simply much less extreme than on the AI-generated concepts.
There was additionally no universally higher immediate. For instance, when judging concepts individually, dropping an instruction that warned judges to not deal with trivial or incremental contributions as novel improved accuracy for some judges whereas decreasing it for others. A immediate that works nicely for one mannequin could not switch to a different, in different phrases.
We additionally examined a standard use of novelty judges: evaluating concepts from two AI mills. We paired concepts from Claude Opus 4.5 with Claude Sonnet 4.5, and GPT-5.4 with GPT-5.1. Throughout the examined immediate modifications, roughly 1 / 4 to a 3rd of verdicts modified on common throughout six judges. This disagreement could be measured with out assuming which generator produces extra novel concepts, exhibiting how the selection of decide immediate can have an effect on comparisons between ideation programs.
Extra search and reasoning didn’t essentially produce higher novelty judgments. We gave GPT-5.6 Sol an extra-high reasoning funds and stay, iterative search restricted to arXiv, then in contrast it with standard-retrieval judges that every noticed 5 pre-retrieved abstracts. (All judges on this comparability ran as soon as per thought on Human+Generated.) The agentic decide scored 0.78 macro-F1 – 11 factors behind Claude Opus 4.6 at 0.89, and in roughly the identical vary as less complicated Claude Opus 4.5 and GPT-5.4 setups – whereas costing roughly 10× extra per resolution at present API costs. (On this context, macro-F1 measures how nicely a decide identifies novel and non-novel concepts; greater is healthier.)
Objective-built evaluators weren’t extra correct both, regardless of costing considerably extra. On the Human+Generated set, a easy Claude Opus 4.6 immediate utilizing low reasoning and majority voting throughout three calls achieved 0.854 macro-F1 at a complete analysis price of $2.59. By comparability, Thought Novelty Checker, a literature-grounded retrieval and reranking pipeline, scored 0.642 at $30.66, whereas Sakana AI’s AI Scientist, whose novelty evaluator iteratively searches Semantic Scholar, scored 0.772 at $82.05. (Each devoted evaluators used Claude Opus 4.6, with one name per resolution.)
In different phrases, the easy prompted baseline was each extra correct and significantly cheaper than both purpose-built system.
Taken collectively, these outcomes counsel that novelty analysis is especially brittle on the AI-generated concepts these programs are more and more being requested to guage.
Towards extra dependable analysis of novel analysis
Our outcomes counsel that LLM-based novelty analysis stays a serious open downside. Judges that seem dependable on human-authored analysis could behave otherwise on AI-generated concepts; prompts that assist one mannequin can harm one other; and extra search or compute doesn’t essentially produce extra constant judgments.
Future work ought to take a look at judges on the sorts of concepts they may really consider, measure sensitivity throughout prompts and fashions, enhance how prior literature is integrated, and construct benchmarks that may evolve because the scientific literature grows and AI programs enhance. Our benchmark and reusable information pipeline are a place to begin—researchers can apply them to new peer critiques, different scientific communities, and future generations of novelty judges.
Automated ideation depends upon greater than producing believable analysis instructions; we additionally want methods to inform when these concepts are genuinely new. By making the weaknesses of present novelty judges simpler to measure, we hope this launch helps researchers construct extra strong methods to judge AI-generated analysis concepts.
