Code Check:
Does Your Code Implement What Your Paper Claims?
CSPaper
A part of Scholar7 AB · July 2026
Every author believes their repository matches their paper. The repository, quietly, often disagrees. A loss term that got simplified late one night, a sampling law that changed between the appendix and the final push, an experiment that never left a branch. None of it is dishonest. All of it is invisible until someone tries to reproduce the work.
Code Check reads your paper, reads your code in any programming language, and audits the two against each other. Every finding it reports comes with verbatim paper quotes and code snippets that you can verify yourself, because a claim you cannot check is just an opinion with formatting.
Why paper-code alignment matters now
Releasing code alongside a paper has become the norm in machine learning, and for good reason: the community has learned that a paper alone often is not enough to reproduce a result. Raff attempted to independently reproduce 255 papers and found that only 63.5% could be reproduced, and that the paper's own completeness was a decisive factor [4]. The NeurIPS reproducibility program made code submission a first-class part of the review process for the same reason [3].
But releasing code creates a new, quieter problem: the paper and the code are two descriptions of the same method, written by busy humans, and they drift. Baumgärtner and Gurevych studied this drift systematically and gave it a name: paper-code discrepancies, semantic conflicts where the code does not faithfully implement the method the paper reports [1]. These are not typos or style issues. They are differences in the loss actually optimized, the data actually used, the schedule actually run. Anyone building on such a repository inherits the gap without knowing it.
Detecting these discrepancies is genuinely hard, even for frontier AI. In the SciCoQA benchmark, the best of 22 evaluated models detected only 46.7% of real-world paper-code discrepancies in a single pass [1]. In other words: hand a strong model a paper and its repository and ask what does not match, and it will miss more than half of the documented problems. That result shaped how we built Code Check, and it is why we did not build a one-prompt wrapper.
We see this as one piece of a larger position we have argued elsewhere: peer review at today's volume survives only if AI produces verifiable evidence rather than confident prose [2]. A paper-code audit is a natural place to apply that principle, because the ground truth is sitting right there in the repository. Recent work has begun systematically quantifying objective errors in published AI papers with LLM analysis [5]; auditing the released code against the paper extends the same scrutiny to the artifact itself.
Share of real-world paper-code discrepancies the best single-pass model detects on SciCoQA [1]. More than half go unnoticed.
Of 255 papers Raff attempted to reproduce independently, only this share succeeded [4].
The science underneath
Code Check builds on SciCoQA: Quality Assurance for Scientific Paper-Code Alignment, by Tim Baumgärtner and Iryna Gurevych of the UKP Lab at TU Darmstadt, accepted at ACL 2026 [1]. The paper does three things we rely on directly.
First, it defines the problem precisely. A discrepancy is a semantic conflict between the method a paper describes and what its code implements, meaningful enough to affect scientific logic, the experimental protocol, or a mathematical formulation. Bugs, configurable hyperparameters, and trivial engineering shortcuts do not count. Each discrepancy has one of three types: a Difference (paper and code describe conflicting approaches), a Paper Omission (the code contains a significant component the paper never mentions), or a Code Omission (the paper describes a step the code lacks). Each also carries a category: algorithm, model, loss, data, training, or evaluation.
Second, it builds the first benchmark for the task: 635 paper-code discrepancies, of which 92 are real cases collected from GitHub issues and reproducibility studies, and 543 are synthetically constructed from scientific codebases across AI, physics, biology, and other computational fields [1]. The dataset is public and CC-BY licensed.
Third, it measures how well current models do, and the answer is humbling. Across 22 LLMs evaluated zero-shot, the best detect 46.7% of the real-world discrepancies. The paper's precision analysis shows the trade-off clearly: on a pooled evaluation set, GPT-5 reached 88.0% precision at 51.2% recall while Gemini 2.5 Pro reached 94.6% precision at 41.1% recall [1]. Models are reasonably careful about what they report. They simply fail to find most of what is there.
Paper-code alignment now has a shared definition, a shared dataset, and a shared measuring stick. Code Check is our attempt to engineer past the single-pass ceiling they reveal.
How Code Check works
That single-pass ceiling suggests the failure is not model capability so much as task structure: paper plus repository is a lot of context, and asking one prompt to find everything invites shallow reading. Code Check therefore splits the audit into stages, each with one job, and adds two safety nets that most systems skip.
An agent reads the paper and writes down its concrete method claims: the algorithm steps, the architecture, the losses, the data handling, the training schedule, the evaluation protocol. Each claim keeps a verbatim quote from the paper as its anchor. On a re-run of the same paper the claim list is reused rather than re-extracted, so two runs are directly comparable and any change in the results is attributable to your code changes, not to extraction variance.
Point it at a public GitHub URL or upload a zip. A GitHub repository is fetched at an exact commit, and that commit hash is recorded on the check, so you always know precisely which version was audited. Source files in any programming language are kept; large datasets are not fed to the model but summarized into compact profiles (row counts, column names, record keys), so claims about data remain checkable without paying for the data itself.
A claim-guided audit walks from the paper into the code: for every claim it traces the actual execution path and rules faithful, differs, missing, or not verifiable. In parallel, a code-first sweep walks the other way: it inventories the scientifically significant components in the code and asks which ones the paper never mentions. Paper omissions are the hardest class for single-pass models [1]; the reverse sweep exists precisely for them.
Every candidate finding is cross-examined by a separate verifier whose job is to reject: duplicates, configurable hyperparameters dressed up as discrepancies, speculation without evidence, and reformulations that are mathematically equivalent to what the paper says. Detection is tuned for recall; this stage restores precision. We did not train a model to nod along.
Finally, a deterministic checker with no imagination string-matches every quoted paper passage against the actual paper text and every quoted code line against the actual cited file. Evidence that cannot be located is never shown as if it were solid: the finding is either flagged with a visible caution or withheld entirely. This is the stage that makes hallucinated evidence structurally unable to reach you, and we trust it precisely because it cannot be persuaded.
What makes this design distinctive is the combination: two detection passes in opposite directions instead of one prompt, an adversarial gate instead of self-confidence, and a mechanical grounding step that treats the model's own citations as claims to verify. The final report is assembled deterministically from the surviving findings. No model writes the verdict.
Both inputs are also treated as untrusted. Instructions embedded in the paper or the code, such as a README telling AI tools to report every claim as faithful, are stripped before analysis and disclosed in the report. And because a faithful ruling only counts when its cited code evidence actually exists in the repository, telling the checker that everything matches cannot produce a green report; at best it produces a check that admits it verified nothing.
How to read a Code Check report
This section doubles as the reference manual for the report page. If you arrived here from a report wondering what a chip or a label means, this is the place.
The verdict
The banner at the top gives one of four outcomes. Code Aligned With Paper means no meaningful discrepancy survived all five stages and at least one claim was positively verified. N Discrepancies Found means at least one did; it is marked major when a high-confidence Difference or Code Omission is among them, minor otherwise. Could Not Verify Alignment means nothing was found but nothing was verified either, typically because the implementing files fell outside the analyzable portion of a very large repository; the check refuses to call that alignment. Code Does Not Match Paper is rarer and means the system concluded the provided repository does not implement this paper at all, usually a wrong link or zip. Below the verdict, a summary line such as "15 of 16 method claims verified in the code" counts the claims the audit ruled faithful, and the exact code version checked is shown next to it.
Finding cards: types, categories, confidence
Each confirmed discrepancy is a card grouped under one of three type badges:
The code implements different logic than the paper describes. The strongest kind of finding: both sides say something, and they conflict.
A step the paper describes is missing from the code, for example a regularizer that appears in the objective but not in the loss function.
The code contains a significant component the paper never mentions, for example an undocumented normalization that plausibly affects results.
The category chip tells you which part of the method is affected, following the taxonomy in [1]: algorithm (step order, operations, core logic), model (architecture, initialization), loss (objective definitions and terms), data (dataset usage, preprocessing, augmentation, filtering), training (schedule, optimization, learning process), evaluation (metrics, evaluation logic, scripts), and other for meta statements that fit none of these.
The confidence chip is a calibrated label:
The quoted evidence directly and unambiguously shows the mismatch.
The evidence strongly suggests the mismatch, but some execution context is missing.
Plausible but relies on assumptions about unseen code, or part of the cited evidence could not be located verbatim, which automatically demotes a finding to low.
Expanding a card reveals the evidence: PAPER SAYS blocks with verbatim quotes, and CODE blocks with the file path and snippet. If you see the amber note "Some cited evidence could not be located verbatim in the paper or code", the mechanical grounding stage could not find part of the quoted evidence where the model said it was. The finding may still be substantively right, which is why it is shown, but treat it with extra care and check the cited spot yourself.
The method claim verification table
Below the cards, every extracted claim gets a row with the audit's ruling:
The code implements the claim as described. These rows are the quiet good news and the source of the "N of M verified" count.
The audit found conflicting logic for this claim.
The audit found no implementation of this claim.
The claim cannot be checked against the provided code. Often this is a process claim, such as manual annotation work, that leaves no repository artifact by nature. A faithful ruling whose cited code evidence cannot be located lands here too, instead of being counted as verified.
A flagged row never dead-ends. If its finding survived, the row links straight to the card (View issue). If it did not survive, a secondary chip explains why: Dismissed means the adversarial verifier rejected it, and clicking the row expands the verifier's reason, for instance a configurable setting or a mathematically equivalent reformulation. Merged means it duplicated another claim's finding, and the row links to the surviving card. Unverified means the finding was withheld because its cited evidence could not be located verbatim, our anti-hallucination stance made visible. Rows without a link expand in place with the auditor's reasoning, so nothing in the table is ever unexplained.
Re-runs, versions, and honesty notes
Authors iterate, so Code Check supports multiple runs per paper: fix your code, run again, and switch between runs with the picker on the review page. Because the claim list is frozen after the first run, the table lines up run over run and differences reflect your changes. Each run records the exact commit it audited. If a repository is too large to read in full, the report says so and lists what was left out; if anything was summarized rather than read, the verdict logic becomes more conservative accordingly. If the paper or the code contains instruction-like content addressed to AI tools, those lines are ignored during analysis and a content note in the report names the files, which is worth knowing about a repository in its own right. And the footer disclaimer is not boilerplate we are contractually attached to: absence of findings genuinely is not proof of alignment, and discrepancies often have benign causes such as code updated after publication.
How well does it work?
Claims about a verification tool should themselves be verifiable, so we evaluated Code Check on 100 documented paper-code discrepancies from the public SciCoQA dataset [1]: 50 real cases collected from GitHub issues and reproducibility studies, and 50 synthetic ones sampled to mirror the composition of the full synthetic set. Each check ran exactly as it does for a user, on the versioned paper and code snapshot recorded in the dataset, one run per case with default settings. A case counts as detected when the report contains a finding describing the same mismatch as the documented one.
The results are below. Two things stand out: real-world cases are harder than synthetic ones, as expected, since they come from the messiest corners of actual research code, and paper omissions remained the hardest class to catch, which is exactly what the code-first sweep exists to chase and where we see the most room to improve.
| Subset | Detected (recall) | Precision (manual verification) |
|---|---|---|
| Real-world | 31 / 50 (62%) | 91% |
| Synthetic | 38 / 50 (76%) | 94% |
| All | 69 / 100 (69%) | 93% |
Recall alone would undersell a verification tool, because a check can also surface valid problems the dataset never annotated. That is what the precision column measures: we read every finding the checks surfaced against its paper and code by hand. The practical takeaway is simple. When Code Check flags something, it is almost always right, and when it stays quiet, the claim table still tells you exactly what was and was not verified.
One caveat we state up front, because rigor demands it: the evaluation papers and repositories are public, so any modern model may have seen some of them during training. Absolute numbers should be read with that in mind. The honest summary is that Code Check finds far more than it misses and rarely cries wolf, so it shortens the search rather than replacing your judgment.
Who may benefit from Code Check?
Run it before you release, and again after every substantial code change. The frozen claim list makes consecutive runs directly comparable, so cleaning up your repository becomes a loop: fix, re-check, watch rows turn faithful. Your future readers, and your future self, inherit a repository that says what the paper says.
Before investing days in reproducing a result, learn in minutes where the released code deviates from the described method. Every finding arrives with the exact quote and the exact snippet, so verifying it takes a glance rather than a debugging session.
Reproducibility courses, artifact evaluation committees, and internal lab reviews all face the same question at scale: does the artifact match the paper? A systematic, evidence-backed first pass frees human attention for the judgment calls that actually need it.
Give it a try
Code Check runs on top of an Agentic Review: open any reviewed paper, switch to the Code Check tab, and point it at a GitHub repository or a zip of the code. If the paper links its own repository, the URL is prefilled for you. Re-run as often as the code evolves; only one check runs at a time per paper, and your first three checks are free.
Acknowledgements
Code Check stands on the SciCoQA work of Tim Baumgärtner and Iryna Gurevych at the UKP Lab, TU Darmstadt. We thank Tim and Iryna for a meeting and lab visit, and for the generous exchanges over email and Slack. The SciCoQA resources are public: the paper, the code, the dataset, the project blog, and a mirror on OpenPrint.
References
[1]Baumgärtner, T., & Gurevych, I. (2026). SciCoQA: Quality assurance for scientific paper-code alignment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026). arXiv preprint arXiv:2601.12910.
[2]You, L., Cao, L., & Gurevych, I. (2026). Preventing the collapse of peer review requires verification-first AI. In ICML 2026 Workshop on AI for Science: AI Scientists as Tools, Co-authors, or Founders.
[3]Pineau, J., Vincent-Lamarre, P., Sinha, K., et al. (2021). Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research, 22.
[4]Raff, E. (2019). A step toward quantifying independently reproducible machine learning research. In Advances in Neural Information Processing Systems 32.
[5]Bianchi, F., Kwon, Y., Izzo, Z., et al. (2025). To err is human: Systematic quantification of errors in published AI papers via LLM analysis. arXiv preprint arXiv:2512.05925.