Fine-tuning a protein model to find new antibiotics

A public research snapshot is on github. This write-up focuses on the question, the evidence, and the failure that changed how I evaluate models.

Antimicrobial resistance is projected to cause 10 million deaths a year by 2050. The pipeline of new antibiotics is thin because discovering them the normal way is slow and expensive. Antimicrobial peptides are one of the more interesting alternatives. They attack bacterial membranes directly, they evolve fast, and bacteria rarely build resistance to them. The catch is the search space. There are vastly more possible peptide sequences than anyone can test in a lab, so most candidates never get looked at.

This is exactly the kind of problem a protein language model should help with. Screen computationally, rank by probability, and hand the lab a short list instead of a haystack. I built a bounded experiment around a pretrained protein model and tested whether it could separate likely antimicrobial peptides from ordinary sequences.

The experiment

The central bet was that a pretrained protein model already carries enough general biological structure to make a focused classifier useful with limited data. I kept the experiment deliberately narrow: compare against a clean held-out benchmark, examine performance by class, and test transfer on data from a different source.

pretrained protein representation │ ▼ focused antimicrobial classifier │ ▼ leakage-aware held-out evaluation │ ▼ cross-source transfer check

What worked

On the primary leakage-aware benchmark, the binary classifier reached 88.3% F1. The important part is not only the number: the split was designed to prevent close sequence relatives from appearing on both sides, reducing the chance that recognition of near-duplicates inflated the result.

The detector then produced a ranked shortlist from previously unlabeled bacterial sequences. That is a useful computational output, but not a biological claim. The candidates still require laboratory validation before anyone should call them antibiotics.

The experiment that failed

An earlier multi-class experiment produced an impressive headline accuracy, but the per-class results told a much weaker story.

class	result
antibacterial	strong
antifungal	mixed
antiviral	weak
antiparasitic	failed

Antibacterial is strong. The others fall off fast, and antiparasitic fails completely because ESCAPE has almost no positive examples for it. A model cannot learn a class it barely sees. That is class imbalance doing exactly what class imbalance does, and the aggregate accuracy was hiding it.

The more important failure came next. I took the ESCAPE-trained model and tested it on GenPept, the leakage-free binary set. It performed essentially at random. The model had not learned "what makes a peptide antimicrobial." It had learned "what makes a peptide antibacterial in ESCAPE specifically," and that knowledge did not transfer one inch to a different dataset asking a slightly different question.

This is the result I care about most, and it is a negative one. Cross-benchmark transfer failed. If I had only ever evaluated on ESCAPE, I would have walked away thinking I had a great general AMP model. I had a great ESCAPE model. The only reason I know the difference is that I tested it somewhere it had never seen. That is the whole argument for leakage-aware, multi-dataset evaluation in one experiment.

What I took from it

Pretrained biological representations can be sample-efficient. A focused adaptation reached a useful result without an enormous training run.
Headline accuracy lies when classes are imbalanced. The 97% was meaningless on its own. Per-class F1 is where the truth was.
One benchmark is not evaluation. The transfer test is the part that told me what the model actually learned, and it is the part most people skip.

Binary AMP detection at 88.3% F1 on a clean benchmark, a ranked candidate shortlist, and a documented failure mode that sharpens how I evaluate everything else. That is the shape of result I am after: a tool that produces something useful, with the uncertainty and limits kept visible.

Public research snapshot on github. If you work in computational biology and want to talk, reach out.