Nine-species benchmark

benchmark · DDA · 11 versions · 44 papers

The field’s most-cited evaluation set: tryptic DDA runs from nine taxonomically distant organisms, used leave-one-species-out so a model is tested on a proteome it never trained on. Assembled by DeepNovo from nine unrelated public submissions.

The nine-species benchmark, version by version compares every version below spectrum by spectrum: which spectra each holds, how their labels and precursors differ, and how much of one version’s test set another trained on.

Kind benchmark
Acquisition DDA
Organisms Apis mellifera, Bacillus subtilis, Candidatus Thiodiazotropha endoloripes, Homo sapiens, Methanosarcina mazei, Mus musculus, Saccharomyces cerevisiae, Solanum lycopersicum, Vigna mungo
Home https://massive.ucsd.edu/ProteoSAFe/dataset.jsp?accession=MSV000081382

The field’s most-cited evaluation set: tryptic DDA runs from nine taxonomically distant organisms, used leave-one-species-out so a model is tested on a proteome it never trained on. Assembled by DeepNovo from nine unrelated public submissions.

Versions

original (DeepNovo, 2017)

The MGF files DeepNovo curated and deposited. What a paper means by “the nine-species dataset” unless it says otherwise.

released 2017-07-18 · introduced by De novo peptide sequencing by deep learning

Where it lives:

Assembled from 9 third-party submissions:

revised (main)

Re-curated to remove peptide redundancy between species, which leaked test peptides into training in the original, and re-released with a fix for a bug that wrongly detected shared peptides between species. Introduced by “A multi-species benchmark for training and validating mass spectrometry proteomics machine learning models”. 2,838,117 spectra in the Zenodo main archive. The MassIVE record holds three earlier builds in dated folders (2022, 2023, 2024), none identical to it: the 2024 one differs by 47,640 spectra, because each build redrew which species keeps a peptide found in several.

spectra 2,838,117 · released 2024-11-08 · introduced by A multi-species benchmark for training and validating mass spectrometry proteomics machine learning models

Where it lives:

InstaNovo split

A fixed train/validation/test split published as parquet with its own DOI, which makes it the only version of this benchmark that is reproducible by citation alone. Its 499,402 training spectra are the original 2017 deposit’s yeast-held-out fold, the same set NovoBench retrains every architecture on, and its test set is the original yeast file. A sixth of ProteoBench’s spectra are scans from its training split.

spectra 639,286 · train 499,402 · validation 28,572 · test 111,312 · released 2024-12-01

Where it lives:

ProteoBench selection

The selection ProteoBench’s de novo DDA-HCD module scores submissions against: the revised (balanced) variant record for record, with the species dropped and the scans renumbered.

spectra 779,879

Where it lives:

revised (balanced)

The same re-curation, cut to 150 whole raw files of main, taken in random order per species until it reached about 100,000 spectra (human, mouse and Candidatus endoloripes are kept whole). Shipped in the same Zenodo record as the main variant, so “the revised benchmark” is still two different things.

spectra 779,879 · released 2024-11-08 · introduced by A multi-species benchmark for training and validating mass spectrometry proteomics machine learning models

Where it lives:

representation benchmark (dIon)

Spectrum-retrieval and pair-discrimination benchmark derived from the revised benchmark (MSV000090982), with dIon’s development backbones excluded from the held-out set. Measures spectrum representations, not de novo sequencing.

validation 6,119 · test 182,393 · released 2026-10-09 · introduced by dIon: Fragmentation-Based Invariance for Self-Supervised Learning of Tandem Mass Spectra

Where it lives:

revised (MassIVE 2022)

The first revised build, the top-level peak/ folder of MSV000090982: Tide and Percolator over ThermoRawFileParser MGFs, searched without N-terminal modifications. 2,812,194 spectra. The 2023 re-search replaced about 200,000 of them and relabelled 41,000 more.

spectra 2,812,194 · released 2022-12-01

Where it lives:

revised (MassIVE 2023)

A re-search that added four N-terminal modifications, in the updates/2023-03-08_woutb_2c43c18c folder of MSV000090982. 2,854,651 spectra, with 1,458 peptides still in more than one species. Identical, record for record, to the ninespecies_updated files in InstaNovo’s Hugging Face repository.

spectra 2,854,651 · released 2023-03-08

Where it lives:

revised (MassIVE 2024)

The 2023 search with the peptides shared between species redrawn, in the updates/2024-05-14_woutb_71950b89 folder of MSV000090982. 2,844,905 spectra, and every difference from 2023 is a redrawn peptide. 353 peptides are still in more than one species, and Zenodo main is 47,640 spectra away.

spectra 2,844,905 · released 2024-05-14

Where it lives:

revised (build not stated)

A paper that names the revised benchmark, or its MassIVE accession MSV000090982, without saying which build. The accession holds three, from 2022, 2023 and 2024, and the published benchmark is a fourth, Zenodo main. They share their peak lists but differ by up to 440,000 spectra and in their labels.

Where it lives:

denovo_benchmarks selection

The nine 9_species datasets of the denovo_benchmarks living benchmark, deposited in its MassIVE record MSV000096182: 101 raw files, among them honeybee BM-2, which the revised search never had, relabelled by the benchmark’s own search. 946,155 labelled spectra, about half of them absent from Zenodo main; 4% most likely carry a spurious deamidation where main reads an isotope peak.

spectra 946,155

Where it lives:

Deposited by (3)

More than one paper here means one piece of work published twice, a preprint and its version of record; a deposit itself happens once.

Used by (42)

Checkpoints trained on this (2)

Method Checkpoint Version of this dataset Size Get it
Casanovo nine-species not stated 5.1 GB Zenodo
PhysNovo nine-species not stated 1.7 GB Zenodo

Each link rests on stated evidence rather than a text match:

  • Casanovo nine-species: the Zenodo record is titled ‘Casanovo model weights on nine-species benchmark’; it does not say WHICH version of the benchmark, so none is asserted
  • PhysNovo nine-species: the Zenodo record is titled ‘PhysNovo Model Weights Trained on the Nine-Species Benchmark Dataset’; it does not say which version

Methods on these papers (36)

Taken from the describing links only, so a paper that merely ran a tool on this data does not make that tool a method of it.

Seen in the charts

Back to the full map

Back to top