Advancing De Novo Glycopeptide Sequencing with InstaNovo in Glycoproteomics
thesis · 2025
| Date | 2025-06-12 |
| Type | thesis |
| Publisher | MSc thesis |
| Contribution | downstream-application |
| Supervisor | Jeroen Van Goey |
| Link | https://jeroen.vangoey.be/files/theses/glyco-finetuning-instanovo.pdf |
Abstract
This study investigates the adaptation of InstaNovo, a transformer-based model originally designed for de novo peptide sequencing, to the more complex domain of glycoproteomics. Through a series of fine-tuning experiments on glycopeptide datasets, we observe that while the model shows some capacity to learn from glyco spectra, the overall improvements remain limited. Notably, fine-tuning on unfiltered spectra, which partially overlap with a dataset the model was trained on, results in stronger learning signals. However, across all fine-tuning settings, the model suffers from catastrophic forgetting, losing accuracy on peptide sequences it previously handled well. Even when using unfiltered spectra, which provide stronger learning signals for glycopeptides, this forgetting effect persists. PCA (Principal Component Analysis) projections of spectrum embeddings further reveal a significant domain shift, which helps explain why InstaNovo struggles on glycopeptides sequencing adaptation. To address this, we propose treating overlapping, glyco spectra not as outliers to discard, but as informative examples that help the model distinguish between glycosylated and non-glycosylated peptides. While fine-tuning strategies prove insufficient for reliable glycopeptide sequencing, our results highlight promising directions for improvement. Future work should explore richer datasets, multi-task learning, attention-guided learning mechanism, contrastive learning, and transformations that preserve the embedding of the original InstaNovo spectra while shifting glyco spectra closer to them in embedding space, helping align their distributions.
Methods and tools
- InstaNovo: Knapsack beam search
- InstaNovo glycopeptide fine-tuning: Adaptation of InstaNovo, a transformer de novo peptide sequencer, to glycoproteomics by fine-tuning on glycopeptide spectra. Learning from glyco spectra is measurable but limited, and every fine-tuning setting suffers catastrophic forgetting, losing accuracy on peptides the base model handled well; fine-tuning on unfiltered spectra that partly overlap the original training set gives the strongest learning signal but does not remove the effect. PCA of spectrum embeddings shows a large domain shift that explains the difficulty, and the study argues for treating overlapping glyco spectra as informative examples rather than outliers to discard, alongside multi-task and contrastive objectives and embedding transformations that pull glyco spectra toward the original distribution.