Challenge closed · final report
What we built, what held up, and what did not.
The 2nd DREAM×CACHE Target-2035 DEL-ML challenge for PGK2 is over. This page is the short version of the whole campaign. The Overview, Now, History and Methods pages hold the detail, and every number below traces to an entry in the worklog.
Outcome as recorded in the repo
Blind test phase: 2 of 2 submissions used, both N_hits = 0, ROC-AUC 0.649. Validation: eight zeros, then one hit (p = 0.052). Active-learning phase: 50 of 100 validation-oracle credits used; the last repo-logged state (2026-10-12) shows 0 of 2 active-learning test slots used, with two staged candidate CSVs (results/al_oracle/test_candidates/). Any active-learning submission made after that date is not recorded here.
The campaign in five phases
1 · Build the pipeline (Jul)
DEL-ML on the PGK2 AIRCHECK data: LightGBM on ECFP4 with a matrix- and inhibitor-corrected enrichment label (count_PGK2 ≥ 4, above NTC and inhibitor); structure arm (Boltz-2 co-fold, smina docking gate); WDR91 as a published-recipe yardstick (Wellnitz 2025, ~15%).
2 · The wall (Aug–Sep)
Eight validation submissions from DEL-primary models scored 0 confirmed hits. A hardened 10-target referee with property-matched decoys showed DEL-to-drug-like transfer is at chance (top-50 ≈ 0, ROC at or below 0.5), reproducing the blind 0-hit result.
3 · The inversion (Sep)
Stop letting the DEL teach; mine what PGK2 binds in nature and in known PGK chemistry. A DEL-independent, pocket-gated canonical-binder prior scored the first validation hit (1 hit / 1 series, p = 0.052). The blind test phase closed 2/2 submissions with N_hits = 0 and ROC-AUC 0.649.
4 · Active learning (Oct)
The organizers released 6,580 labeled molecules (39 hits). A labels-trained model (DEL plus released labels, hits weighted 3) put the held-out series hits at a median 1% percentile under series-held-out CV, against 8.7% for DEL-only. A 50-credit validation-oracle campaign localized a benzothiazol-2-yl amide/urea chemotype (10 hits in 691 tested, 1.4%).
5 · Priors and synthetic labels (Oct 10-12)
Blind medchem agents (AUC 0.795 on 239 labeled compounds) independently favored the benzothiazole chemotype and were complementary to the labels model. A coverage map showed 98.6% of the 184,632-compound catalog has no DEL-positive or labeled-hit neighbor at Tc 0.4. Every attempt to fill the gaps with synthetic positives failed the held-out test (below).
Scoreboard of the final phase
Held-out percentile of the 39 released hits among the 184,632-compound catalog, with hit series held out (Butina Tc 0.4 on ECFP4, six folds). Thirty-nine hits is a small sample, so differences of a few points are not resolved.
| Method | Result | Verdict |
|---|---|---|
| DEL-only supervised model | median held-out hit percentile 8.7% | baseline |
| DEL + released labels (weight 3) | median 0.81%, 69% of hits in top 5% | adopted |
| + label-free physchem prior (z-blend 0.3-0.7) | median unchanged (0.83-0.94%), top-0.1% 13% to 15-21% | small tail gain, within noise |
| DEL label denoising (5 variants) | median 6.9-10.9% (DEL-only), no reliable gain | dropped |
| Model-derived synthetic positives for gap clusters (2,000) | median 0.81% to 6-8% | harmful |
| Agent-judged synthetic positives, labels-selected pool | median 0.44-0.53%, but random pool members gave 0.63-0.69% | gain was pool leakage |
| Agent-judged synthetic positives, label-free pool (600) | median 1.7% vs 2.0% for random picks | no benefit |
| AQAffinity PGK2 (239 compounds) | AUC 0.667, about the same as ring count (0.65) | generic, not used |
| Blind medchem-agent score | AUC 0.795 on 239 labeled; Spearman about -0.06 vs labels model | complementary signal |
Lessons
- DEL enrichment does not encode which drug-like compound binds, so DEL-only models transfer at chance to the Enamine catalog. Real labeled hits are what moved the ranking.
- Labels from the same signal the model already has (model-, prior- or agent-derived pseudo-positives) do not help, and a pool picked with the held-out labels leaks. Always run a random-pool control.
- Judge by held-out top ranks and series, not AUC alone. Small positive sets (39 hits) cannot resolve small effects.
- The validation oracle is the cheapest direct yield test; use it before spending a test slot.
Reproduce and navigate
docs/12_worklog.mdis the chronological record (D-real-1 to D-real-179).METHODS_BRIEF.mdis the readable tour of the methods.scripts/active_learning/: labels model, oracle blocks, submission builders.scripts/synth/: priors, denoising and synthetic-label tests.scripts/coverage/: coverage map.results/: outputs per experiment.- Regenerate this site with
scripts/gen_site.py.
Housekeeping at close
Synapse tokens used during the campaign should be rotated and the Hugging Face token revoked. Challenge data is under the AIRCHECK/Synapse data-use terms and is not committed.