How de novo sequencing is scored

The metrics deep-learning papers report, and how their code computes them

Peptide recall, amino-acid precision, precision against coverage and the rest: what deep-learning de novo sequencing papers say they measured, and what the evaluation code of each tool and each of its versions actually computes. Every number links to the code that computes it.

Nearly every deep-learning de novo sequencing paper reports the same few numbers: peptide recall, amino-acid precision and, more and more often, a curve of precision against coverage. They read like one standard. This page checks whether they are, twice over: against what 115 papers say they computed, and against the evaluation code of 45 implementations, version by version.

How it was done. The papers are the catalog’s 124 deep-learning method papers; 9 of them have no full text in the catalog’s PDF library and are left out. Every metric a paper reports, with its definition quoted and the section it came from, is in papers.jsonl. The code was read at the commit each row of the implementation table links to, on 2026-11-22. Every number is a link to an entry in Reproducing every number; the worked examples run transcriptions of each tool’s matching loop, and the entry names the file and commit each was copied from.

Four things to know before putting two numbers side by side.

  1. “Peptide precision” is usually a recall. Casanovo’s code calls its peptide figure pep_precision and divides it by every spectrum, and NovoBench, denovo_benchmarks and PowerNovo2 do the same. 12 papers define a “peptide precision” over all spectra; 3 define a “peptide recall” over predicted peptides only.
  2. Amino-acid precision and recall are swapped in a whole family of models. Casanovo v3.0 to v3.5 passed the prediction where the truth belonged, and v4.0 fixed it. The models that copied the v3 code did not: 9 entries in the table below still swap the two, among them ContraNovo, PrimeNovo, RefineNovo, RankNovo and π-HelixNovo. denovo_benchmarks swapped them from April 2024 to September 2026.
  3. “Correct” means the right mass, not the right sequence. The standard rule accepts K for Q (0.036 Da apart), F for oxidised M (0.033 Da) and D for deamidated N (0.000 Da), at peptide level too. ProteoBench’s ppm tolerances and Casanovo-TMT’s exact-peptidoform rule do not.
  4. “Peptide AUC” names at least three different areas. NovoBench integrates precision against recall, which can never exceed the peptide recall; others integrate against coverage. For the same 1,000 predictions, half of them right and ranked first, the two give 0.50 and 0.85.

What the papers report

97 of the 115 papers report a peptide-level figure, under one name or another, 78 report amino-acid precision or recall, and 54 a precision-coverage curve or the area under it. Fewer report an FDR (25), PTM precision or recall (21) or calibration (9), and 23 introduce a metric of their own.

The rule most of them share

The rule comes from DeepNovo (2017). A predicted residue is correct when its mass is within 0.1 Da of the true residue’s and the prefix mass up to and including it is within 0.5 Da of the true prefix’s. Casanovo added a second pass from the C-terminus, so a residue can also be matched on its suffix mass. A peptide is correct when it has as many residues as the truth and every one is correct.

From those matches come the usual figures. Amino-acid precision divides the correct residues by the predicted residues, amino-acid recall by the true residues. Peptide recall divides the correct peptides by all labelled spectra, peptide precision by the spectra that got a prediction. Sorting the predictions by the model’s confidence and computing precision at each depth gives precision against coverage.

Two things follow from matching on mass. I and L weigh the same, so they always match, with no I/L rule needed. And so do pairs that differ by less than 0.1 Da: under the rule, PEPTIDEQ is a correct call for PEPTIDEK (1 means yes). PepPr points out that this makes deamidated N equal to D and scores its deamidation calls on identity instead; Casanovo-TMT rejects the rule for the same reason and requires the exact peptidoform. ProteoBench keeps the rule’s shape but in ppm: 20 ppm per residue and 50 ppm on the prefix. K and Q are 284 ppm apart, so there they count as different.

What the papers say about it

43 papers state the 0.1 / 0.5 Da rule. 17 count a residue or a peptide correct only on an exact sequence match, and 4 use a stricter mass rule or compare position by position: SMSNet wants the residue mass exact to 0.0001 Da and the prefix within 0.03 Da, and PepNet and DPST compare position by position, so one inserted residue makes everything after it wrong. 51 papers state no rule at all. On I and L, 24 papers say they count as equal and 88 do not say; that matters only for the papers that compare strings.

The wording varies more than the rule. Most papers write that the prefix or the suffix mass must agree; RankNovo’s text says both must. Several measure a peptide as a string while measuring residues by mass, so their peptide and amino-acid numbers rest on different definitions. And the names do not follow the denominators: the table marks the 12 papers that call a recall “precision” and the 3 that call a precision “recall”.

Where the code disagrees

Nearly every implementation descends from one of two loops: DeepNovo’s, which walks forward only, and Casanovo’s evaluate.py, which adds the pass from the C-terminus. What separates the numbers is mostly not the rule but what happens around it: which argument is the truth, which spectra are counted, and which version of the code ran.

Forward only, or forward and back

DeepNovo’s loop walks both peptides from the N-terminus. When the prefix masses drift apart by more than 0.5 Da, as they do after an inserted or wrong-mass residue, nothing after the drift can match again. Casanovo’s second pass from the C-terminus recovers the residues after the error. Truth PEPTIDEK against the prediction PEPGTIDEK matches 3 residues forward only and 8 with Casanovo’s rule. 12 implementations in the table are forward only: the DeepNovo family, GraphNovo, DIANovo, SMSNet, DiNovo and InstaNovo, whose matcher carries an “adapted from Casanovo” comment but is DeepNovo’s loop. Their amino-acid numbers come out lower on the same predictions; their peptide numbers do not change, because a correct peptide never drifts.

Precision and recall, swapped

Casanovo’s aa_match_batch returns the residue counts in the order its two peptide arguments came in, and aa_match_metrics expects the true count first. From v3.0.0 to v3.5.0 the validation step passed the prediction first, so the number printed as amino-acid precision was the recall and the other way round; commit 3a18fed fixed it for v4.0.0. For truth PEPTIDEK and prediction PEPTIDE, precision is 1.000 and recall 0.875; the swapped code prints 0.875 as precision. Peptide figures are unaffected, because the match itself is symmetric.

Most of the models built on Casanovo v3 kept the swap, and their latest code still has it: ContraNovo, PrimeNovo and every copy of it (π-PrimeNovo, ScopeNovo), RefineNovo, RankNovo, DiffNovo, Transformer-DIA and DiffNovo-DIA, and every tag of π-HelixNovo. TSARseqNovo copied the fixed v4.0 file and kept the old argument order; pUniFind’s own copy swaps them too. XuanjiNovo swapped them at its v1.1 tag and has since fixed it, and π-MNovo fixed it in v0.2.0. The multi-tool benchmark denovo_benchmarks reversed the arguments in April 2024 and fixed it on 26 September 2026 (commit 3a92eba), so its “AA precision” from that period is amino-acid recall.

Which spectra are counted

Up to v3.5.0, Casanovo dropped every spectrum whose prediction was empty or malformed from both sides of the comparison, so its peptide figure was computed over the spectra that got a usable prediction, and a model that abstained more scored higher. ContraNovo, the PrimeNovo family, RefineNovo, DiffNovo and π-HelixNovo inherited that. From v4.0 Casanovo keeps them; NovoBench, AdaNovo and dIon keep them too.

InstaNovo excludes empty predictions from precision only, which is what precision means. But what counts as empty changed: before 1.2.0, beam search returned nothing when no beam fitted the precursor mass, so precision was computed over precursor-matched predictions only. From 1.2.0 those beams are kept, ranked after the matching ones, so InstaNovo precision before and after 1.2.0 is not like-for-like; the recall is. denovo_benchmarks made the opposite move on 8 February 2025: until then its “Pep precision” was correct over predicted spectra, a precision, and from then on over all labelled spectra, a recall.

One tool, several versions

Casanovo.

  • v1.1 to v2.1 matched on the mass before each residue in a single pass.
  • v3.0 rewrote that as the prefix-then-suffix rule used since.
  • v3.0 to v3.5 swapped precision and recall and dropped empty predictions. v4.0 fixed both, but in validation a spectrum whose beams were all discarded silently shifts the rest of its batch onto the wrong truth.
  • v5.0 to v5.1.2: casanovo sequence --evaluate counted the padding of the true sequence as residues, so it printed a peptide precision of 0. v5.2.0 fixed it.
  • v5: the validation code still splits a peptide before every capital letter, so M[Oxidation] becomes two pieces of mass 0, and any modified residue matches any other.
  • The confidence that orders a precision-coverage curve changed three times: the mean of the residue scores minus 1 off-precursor (v3 to v4.3), the product minus 1 (v5.0 to v5.1), and the product with no penalty (v5.2 on). The same model gives three different curves.

InstaNovo.

  • 0.1.0 to 0.1.7 had the resynchronisation step inverted: after the first mismatch it advanced the peptide that was already ahead, so no later residue could match. GGPEPTIDE against NPEPTIDE matches 0 residues in 0.1.x and 7 from 1.0.0. Peptide figures were unaffected. InstaNovo-P still ships the 0.1 step, including in its PTM matcher.
  • 1.0.0 made its amino-acid error rate count I and L as equal; 0.1 counted every I/L call as an error.
  • 1.2.0 changed which predictions are empty, as above.

denovo_benchmarks has had four evaluators: correct but over predicted spectra only (March 2024), swapped (from April 2024), swapped and over all labelled spectra (from February 2025), and correct over all labelled spectra (from September 2026). ProteoBench stored no AUC until 4 August 2026, and has since integrated precision against coverage at every distinct score.

Smaller differences that still change a number

  • DeepNovo’s peptide rule ignores the prediction’s length: a peptide counts as correct when every true residue is matched, so PEPTIDEK is a correct call for PEPTIDE. Its DIA and PointNovo descendants, given more than one candidate per spectrum, keep the candidate closest to the truth; with the default of one it makes no difference.
  • DiffNovo logs, per batch, the better of its two decoders, which no deployed system could choose, and its peptide slot holds an amino-acid value.
  • RankNovo averages per-batch peptide recalls without weighting them by batch size, and weights the amino-acid figure by the first candidate’s length.
  • MemNovo strips every modification before matching and its mass table has no glycine, so a deleted G costs nothing; under its PrimeNovo model name it falls back to counting letters regardless of order.
  • PowerNovo2’s precision-recall plot counts each correct residue as a negative as well, which caps the plotted precision at 0.5, and a results file holding any empty prediction is silently skipped.
  • PTM precision and recall in NovoBench and AdaNovo inherit the mass rule, so F in place of an oxidised M counts toward PTM recall.
  • Unknown tokens weigh 0 in Casanovo and its copies, so a label with an unparsed modification is scored with every later prefix mass off; InstaNovo and the DeepNovo family refuse it.

Three areas called AUC

  • NovoBench (and LIPNovo and SemiNovo, which copy it) integrates precision against recall: the area can never exceed the peptide recall, and wrong predictions add no width.
  • Precision against coverage, coverage being the share of all spectra predicted, is what denovo_benchmarks (at 200 downsampled points), ProteoBench (at every distinct score), dIon and Casanovo’s unreleased dev branch integrate. The Pairwise scripts compute coverage over predicted spectra only.
  • InstaNovo integrates peptide precision against recall at 21 confidence quantiles.

On the same predictions the first two give 0.50 and 0.85; neither is wrong, but they are not the same number.

Metrics beyond recall

RankNovo’s PMD and RMD

RankNovo reranks the candidates of several sequencers, and to train the reranker it introduced two measures of how far a candidate is from the truth. Both are training targets, not evaluation metrics: the paper still reports peptide recall and amino-acid precision, and the released code computes those with the swapped Casanovo v3 call.

PMD, Peptide Mass Deviation, is an edit distance in which inserting or deleting a residue costs one unit and substituting one costs the mass difference; the released code fixes the unit at 51.4238 Da. It is not the difference between the two peptides’ masses: GG against N, which weigh the same, costs 2.11 units, a missing residue costs 1 unit whatever it weighs, and K for Q costs 0.0007.

RMD, Residual Mass Deviation, gives each residue of the candidate the signed offset between its prefix mass and the nearest prefix mass of the truth. For PEPGGK against PEPNK it is -57.02 Da at the first G and 0.00 Da at the end, where the masses agree again. In the released inference code only the PMD head chooses the candidate; the RMD head is computed and discarded. No other repository implements either.

Error rates without a search engine

A recall needs labels, which only database search provides, so the field has started to estimate error rates on unlabelled spectra: Winnow turns calibrated confidence into an FDR (the mean of one minus the confidence above a threshold); DiNovo maps predictions to a target-decoy protein database; π-NovoQC compares counterpart spectra; Casanovo’s 2023 preprint and Casanovo-TMT count matches to a shuffled or decoy digest, the latter saying plainly that this is not a search-engine FDR; and Cascadia counts a DIA prediction correct when it matches the reference proteome exactly.

Other measures

Casanovo 5 reports a calibration curve. DiffuNovo reports the error of the predicted peptide’s mass and the share within 0.001 Da. BiATNovo adds a position-weighted BLEU and a mass-aware alignment score, the QuasiNovo thesis a longest common subsequence anchored on terminal masses, OmniNovo backbone and modification-site accuracy, Spectralis precision and recall of its fragment-bin corrections, and NovoCRF recall split by how much fragment-ion evidence each spectrum holds.

Reading a number

Before comparing two published figures, find out:

  1. Which code, at which version, computed it. A Casanovo v3 descendant and a Casanovo v4 one print the same labels for different quantities.
  2. The denominator. All labelled spectra, spectra with a prediction, or spectra with a well-formed prediction.
  3. The residue rule. The 0.1 / 0.5 Da mass rule, ppm, or an exact sequence.
  4. For a curve, which confidence orders it, and what the area is taken against.

Reproducing every number

Every number above, in the order it first appears: what it counts, the public files it comes from, the scripts that produce its input (in run_all.sh’s order), and the code in page_numbers.py that computes it. page_numbers.py --only <id> recomputes one.

115 (Of those, papers whose full text was read). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl. Recompute: page_numbers.py --only n_read.

Code
@number("n_read", "Of those, papers whose full text was read", ["papers"], ["papers.jsonl"])
def _():
    return len(papers())

45 (Implementations compared (a tool, or one version range of it)). Reads implementations.json: one row per tool and version range, with the commit read. Input from implementations.json. Recompute: page_numbers.py --only n_impl.

Code
@number("n_impl", "Implementations compared (a tool, or one version range of it)", ["code"], ["implementations.json"])
def _():
    return len(code())

124 (Deep-learning method papers in the catalog considered here). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl. Recompute: page_numbers.py --only n_papers.

Code
@number("n_papers", "Deep-learning method papers in the catalog considered here", ["papers"], ["papers.jsonl"])
def _():
    return len(all_papers())

9 (Papers with no full text in the library (paywalled or not yet fetched)). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl. Recompute: page_numbers.py --only n_unread.

Code
@number("n_unread", "Papers with no full text in the library (paywalled or not yet fetched)", ["papers"], ["papers.jsonl"])
def _():
    return len(all_papers()) - len(papers())

12 (Papers that call a peptide figure ‘precision’ and define it over all spectra). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl, build_site_data.py. Recompute: page_numbers.py --only n_prec_named_recall.

Code
@number("n_prec_named_recall", "Papers that call a peptide figure 'precision' and define it over all spectra", ["papers"], ["papers.jsonl", "build_site_data.py"])
def _():
    return sum(p["naming"] == "precision over all spectra" for p in site()["papers"])

3 (Papers that call a peptide figure ‘recall’ and define it over predicted peptides). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl, build_site_data.py. Recompute: page_numbers.py --only n_recall_named_prec.

Code
@number("n_recall_named_prec", "Papers that call a peptide figure 'recall' and define it over predicted peptides", ["papers"], ["papers.jsonl", "build_site_data.py"])
def _():
    return sum(p["naming"] == "recall over predictions" for p in site()["papers"])

9 (Entries in the table whose latest code still swaps amino-acid precision and recall). Reads implementations.json: one row per tool and version range, with the commit read. Input from implementations.json. Recompute: page_numbers.py --only n_swapped_now.

Code
@number("n_swapped_now", "Entries in the table whose latest code still swaps amino-acid precision and recall", ["code"], ["implementations.json"])
def _():
    latest = {}
    for c in code():
        latest[c["tool"]] = c  # rows are in version order, so the last one per tool is the latest
    return sum(c["order"] == "swapped" for c in latest.values())

0.036 (Mass difference between K and Q, in Da). Reads Casanovo v4.0.0 residue masses, casanovo/config.yaml. Recompute: page_numbers.py --only d_kq.

Code
@number("d_kq", "Mass difference between K and Q, in Da", ["casanovo_masses"], fmt="{:.3f}")
def _():
    return _da("K", "Q")

0.033 (Mass difference between F and oxidised M, in Da). Reads Casanovo v4.0.0 residue masses, casanovo/config.yaml. Recompute: page_numbers.py --only d_fm.

Code
@number("d_fm", "Mass difference between F and oxidised M, in Da", ["casanovo_masses"], fmt="{:.3f}")
def _():
    return _da("F", "m")

0.000 (Mass difference between deamidated N and D, in Da). Reads Casanovo v4.0.0 residue masses, casanovo/config.yaml. Recompute: page_numbers.py --only d_nd.

Code
@number("d_nd", "Mass difference between deamidated N and D, in Da", ["casanovo_masses"], fmt="{:.3f}")
def _():
    return _da("n", "D")

0.50 (NovoBench’s AUC for 1,000 spectra, half right and all of those ranked first). Reads NovoBench, novobench/metrics/evaluate.py (curve_auc). Recompute: page_numbers.py --only ex_auc_half_nb.

Code
@number("ex_auc_half_nb", "NovoBench's AUC for 1,000 spectra, half right and all of those ranked first", ["novobench"], fmt="{:.2f}")
def _():
    return novobench_auc(HALF)

0.85 (Area under precision against coverage for the same 1,000 spectra (trapezoid, coverage = rank / N)). Reads NovoBench, novobench/metrics/evaluate.py (curve_auc). Recompute: page_numbers.py --only ex_auc_half_cov.

Code
@number("ex_auc_half_cov", "Area under precision against coverage for the same 1,000 spectra (trapezoid, coverage = rank / N)", ["novobench"], fmt="{:.2f}")
def _():
    n = len(HALF)
    cs = list(accumulate(int(c) for c in HALF))
    prec = [c / (k + 1) for k, c in enumerate(cs)]
    cov = [(k + 1) / n for k in range(n)]
    return sum((cov[k] - cov[k - 1]) * (prec[k] + prec[k - 1]) / 2 for k in range(1, n))

97 (Papers reporting a peptide-level figure (recall or precision, under any name)). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl. Recompute: page_numbers.py --only n_pep.

Code
@number("n_pep", "Papers reporting a peptide-level figure (recall or precision, under any name)", ["papers"], ["papers.jsonl"])
def _():
    return sum(reports(r, "peptide_recall", "peptide_precision") for r in papers())

78 (Papers reporting amino-acid precision or recall). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl. Recompute: page_numbers.py --only n_aa.

Code
@number("n_aa", "Papers reporting amino-acid precision or recall", ["papers"], ["papers.jsonl"])
def _():
    return sum(reports(r, "aa_precision", "aa_recall") for r in papers())

54 (Papers reporting a precision-coverage curve or its area). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl. Recompute: page_numbers.py --only n_curve.

Code
@number("n_curve", "Papers reporting a precision-coverage curve or its area", ["papers"], ["papers.jsonl"])
def _():
    return sum(reports(r, "precision_coverage_curve", "AUC_of_precision_coverage") for r in papers())

25 (Papers reporting an FDR-based figure). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl. Recompute: page_numbers.py --only n_fdr.

Code
@number("n_fdr", "Papers reporting an FDR-based figure", ["papers"], ["papers.jsonl"])
def _():
    return sum(reports(r, "FDR-based") for r in papers())

21 (Papers reporting PTM precision or recall). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl. Recompute: page_numbers.py --only n_ptm.

Code
@number("n_ptm", "Papers reporting PTM precision or recall", ["papers"], ["papers.jsonl"])
def _():
    return sum(reports(r, "PTM_precision/recall", "PTM_precision") for r in papers())

9 (Papers reporting a calibration metric). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl. Recompute: page_numbers.py --only n_calib.

Code
@number("n_calib", "Papers reporting a calibration metric", ["papers"], ["papers.jsonl"])
def _():
    return sum(reports(r, "calibration metric") for r in papers())

23 (Papers that introduce a metric of their own). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl. Recompute: page_numbers.py --only n_new_metric.

Code
@number("n_new_metric", "Papers that introduce a metric of their own", ["papers"], ["papers.jsonl"])
def _():
    return sum(bool(r.get("new_metrics")) for r in papers())

1 (Truth PEPTIDEK, prediction PEPTIDEQ: does the 0.1 / 0.5 Da rule count the peptide correct? (1 = yes)). Reads Casanovo v5.2.1, casanovo/denovo/evaluate.py (aa_match_prefix, aa_match_prefix_suffix). Recompute: page_numbers.py --only ex_kq_pep.

Code
@number("ex_kq_pep", "Truth PEPTIDEK, prediction PEPTIDEQ: does the 0.1 / 0.5 Da rule count the peptide correct? (1 = yes)", ["casanovo_eval"])
def _():
    return int(all(casanovo_best("PEPTIDEK", "PEPTIDEQ")))

284 (K against Q in ppm of K’s mass (ProteoBench allows 20 ppm per residue)). Reads Casanovo v4.0.0 residue masses, casanovo/config.yaml, ProteoBench, proteobench/score/denovoscores.py (20 and 50 ppm). Recompute: page_numbers.py --only ppm_kq.

Code
@number("ppm_kq", "K against Q in ppm of K's mass (ProteoBench allows 20 ppm per residue)", ["casanovo_masses", "proteobench"], fmt="{:,.0f}")
def _():
    return _da("K", "Q") / MASS["K"] * 1e6

43 (Papers that state the 0.1 Da residue / 0.5 Da prefix rule). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl, build_site_data.py. Recompute: page_numbers.py --only n_rule_mass.

Code
@number("n_rule_mass", "Papers that state the 0.1 Da residue / 0.5 Da prefix rule", ["papers"], ["papers.jsonl", "build_site_data.py"])
def _():
    return _rule_count("0.1 / 0.5 Da mass rule")

17 (Papers that count a residue or peptide correct only on an exact sequence match). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl, build_site_data.py. Recompute: page_numbers.py --only n_rule_exact.

Code
@number("n_rule_exact", "Papers that count a residue or peptide correct only on an exact sequence match", ["papers"], ["papers.jsonl", "build_site_data.py"])
def _():
    return _rule_count("Exact sequence")

4 (Papers with a stricter mass rule or position-by-position matching). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl, build_site_data.py. Recompute: page_numbers.py --only n_rule_other.

Code
@number("n_rule_other", "Papers with a stricter mass rule or position-by-position matching", ["papers"], ["papers.jsonl", "build_site_data.py"])
def _():
    return _rule_count("Stricter mass rule") + _rule_count("Position by position")

51 (Papers that state no matching rule at all). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl, build_site_data.py. Recompute: page_numbers.py --only n_rule_none.

Code
@number("n_rule_none", "Papers that state no matching rule at all", ["papers"], ["papers.jsonl", "build_site_data.py"])
def _():
    return _rule_count("Not stated")

24 (Papers that say I and L are counted as equal). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl, build_site_data.py. Recompute: page_numbers.py --only n_il_equal.

Code
@number("n_il_equal", "Papers that say I and L are counted as equal", ["papers"], ["papers.jsonl", "build_site_data.py"])
def _():
    return sum(p["il"] == "Equal" for p in site()["papers"] if p["read"])

88 (Papers that do not say how I and L are counted). Reads papers.jsonl: every metric each paper reports, with its definition quoted. Input from papers.jsonl, build_site_data.py. Recompute: page_numbers.py --only n_il_unstated.

Code
@number("n_il_unstated", "Papers that do not say how I and L are counted", ["papers"], ["papers.jsonl", "build_site_data.py"])
def _():
    return sum(p["il"] == "Not stated" for p in site()["papers"] if p["read"])

3 (PEPTIDEK against PEPGTIDEK: residues matched forward only (DeepNovo, InstaNovo 1.x)). Reads DeepNovo, deepnovo_worker_test.py (_match_AA_novor), InstaNovo 1.0.0, instanovo/utils/metrics.py (_novor_match). Recompute: page_numbers.py --only ex_fwd.

Code
@number("ex_fwd", "PEPTIDEK against PEPGTIDEK: residues matched forward only (DeepNovo, InstaNovo 1.x)", ["deepnovo", "instanovo_100"])
def _():
    return forward_only("PEPTIDEK", "PEPGTIDEK")

8 (PEPTIDEK against PEPGTIDEK: residues matched with Casanovo’s prefix-then-suffix rule). Reads Casanovo v5.2.1, casanovo/denovo/evaluate.py (aa_match_prefix, aa_match_prefix_suffix). Recompute: page_numbers.py --only ex_best.

Code
@number("ex_best", "PEPTIDEK against PEPGTIDEK: residues matched with Casanovo's prefix-then-suffix rule", ["casanovo_eval"])
def _():
    return sum(casanovo_best("PEPTIDEK", "PEPGTIDEK"))

12 (Implementations that match residues in a forward pass only, with no suffix pass). Reads implementations.json: one row per tool and version range, with the commit read. Input from implementations.json. Recompute: page_numbers.py --only n_forward_only.

Code
@number("n_forward_only", "Implementations that match residues in a forward pass only, with no suffix pass", ["code"], ["implementations.json"])
def _():
    return sum("forward only" in c["rule"] for c in code())

1.000 (Truth PEPTIDEK, prediction PEPTIDE: amino-acid precision, correctly computed). Reads Casanovo v5.2.1, casanovo/denovo/evaluate.py (aa_match_prefix, aa_match_prefix_suffix). Recompute: page_numbers.py --only ex_swap_prec.

Code
@number("ex_swap_prec", "Truth PEPTIDEK, prediction PEPTIDE: amino-acid precision, correctly computed", ["casanovo_eval"], fmt="{:.3f}")
def _():
    return sum(casanovo_best("PEPTIDEK", "PEPTIDE")) / len("PEPTIDE")

0.875 (Truth PEPTIDEK, prediction PEPTIDE: amino-acid recall, which swapped code prints as precision). Reads Casanovo v5.2.1, casanovo/denovo/evaluate.py (aa_match_prefix, aa_match_prefix_suffix), Casanovo v3.5.0, casanovo/denovo/model.py (validation_step). Recompute: page_numbers.py --only ex_swap_rec.

Code
@number("ex_swap_rec", "Truth PEPTIDEK, prediction PEPTIDE: amino-acid recall, which swapped code prints as precision", ["casanovo_eval", "casanovo_v35"], fmt="{:.3f}")
def _():
    return sum(casanovo_best("PEPTIDEK", "PEPTIDE")) / len("PEPTIDEK")

0 (GGPEPTIDE against NPEPTIDE: residues matched by InstaNovo 0.1.x). Reads InstaNovo 0.1.0, instanovo/utils/metrics.py (_novor_match). Recompute: page_numbers.py --only ex_inv_01.

Code
@number("ex_inv_01", "GGPEPTIDE against NPEPTIDE: residues matched by InstaNovo 0.1.x", ["instanovo_010"])
def _():
    return forward_only("GGPEPTIDE", "NPEPTIDE", inverted=True)

7 (GGPEPTIDE against NPEPTIDE: residues matched by InstaNovo 1.0 and later). Reads InstaNovo 1.0.0, instanovo/utils/metrics.py (_novor_match). Recompute: page_numbers.py --only ex_inv_10.

Code
@number("ex_inv_10", "GGPEPTIDE against NPEPTIDE: residues matched by InstaNovo 1.0 and later", ["instanovo_100"])
def _():
    return forward_only("GGPEPTIDE", "NPEPTIDE")

2.11 (RankNovo PMD between GG and N, which have the same total mass (units of 51.4238 Da)). Reads RankNovo, RankNovo/RankNovo/msa/tokenizer.py (TMmass, Greedymass). Recompute: page_numbers.py --only pmd_gg_n.

Code
@number("pmd_gg_n", "RankNovo PMD between GG and N, which have the same total mass (units of 51.4238 Da)", ["ranknovo"], fmt="{:.2f}")
def _():
    return ranknovo_pmd("PEPGGK", "PEPNK")

1 (RankNovo PMD for a missing residue, whatever its mass (units of 51.4238 Da)). Reads RankNovo, RankNovo/RankNovo/msa/tokenizer.py (TMmass, Greedymass). Recompute: page_numbers.py --only pmd_del.

Code
@number("pmd_del", "RankNovo PMD for a missing residue, whatever its mass (units of 51.4238 Da)", ["ranknovo"], fmt="{:.0f}")
def _():
    return ranknovo_pmd("PEPTDE", "PEPTIDE")

0.0007 (RankNovo PMD between PEPTIDEK and PEPTIDEQ (units of 51.4238 Da)). Reads RankNovo, RankNovo/RankNovo/msa/tokenizer.py (TMmass, Greedymass). Recompute: page_numbers.py --only pmd_kq.

Code
@number("pmd_kq", "RankNovo PMD between PEPTIDEK and PEPTIDEQ (units of 51.4238 Da)", ["ranknovo"], fmt="{:.4f}")
def _():
    return ranknovo_pmd("PEPTIDEK", "PEPTIDEQ")

-57.02 (RankNovo RMD at the first G of PEPGGK against PEPNK, in Da). Reads RankNovo, RankNovo/RankNovo/msa/tokenizer.py (TMmass, Greedymass). Recompute: page_numbers.py --only rmd_gg_mid.

Code
@number("rmd_gg_mid", "RankNovo RMD at the first G of PEPGGK against PEPNK, in Da", ["ranknovo"], fmt="{:.2f}")
def _():
    return ranknovo_rmd("PEPGGK", "PEPNK")[3] * GAP

0.00 (RankNovo RMD at the last residue of PEPGGK against PEPNK, in Da (0: the masses have realigned)). Reads RankNovo, RankNovo/RankNovo/msa/tokenizer.py (TMmass, Greedymass). Recompute: page_numbers.py --only rmd_gg_last.

Code
@number("rmd_gg_last", "RankNovo RMD at the last residue of PEPGGK against PEPNK, in Da (0: the masses have realigned)", ["ranknovo"], fmt="{:.2f}")
def _():
    return abs(ranknovo_rmd("PEPGGK", "PEPNK")[-1] * GAP)
Back to top