Phase 1 / pre-benchmark protocol

Apple Silicon viability benchmark protocol

A frozen-methods draft for turning the portable lab into a publishable first paper without overclaiming beyond the evidence.

Phase 0 evidence

3 / 3
Real SimpleFold-100M public-tunnel predictions completed on engineering targets.

Phase 1 scope

8
Phase 1A is now frozen as a SimpleFold-3B first paper unit.

Reader bridge

manual
No-fixture SimpleFold recipe curation is complete; autonomous Reader remains a companion-methods target.

Target candidates

24 / 24
RCSB-selected candidate chains passed PDB+FASTA fetch checks; pending biological review.

Model cards

5
Install-card templates exist for SimpleFold, ESMFold, chai-mlx, Protenix, and Boltz-2.

Runner update

PDB:CHAIN
The runner now supports explicit chain labels while preserving Phase 0 PDB-only fixtures.

Rehearsal finding

280 s
The long target timed out publicly but completed through direct local prediction.

SimpleFold-3B

exec
Mid CLI repeat captured route/cache/progress metadata in the card.

Repeat/cache

3x
Same-target CLI smokes now define the timing, memory, and confidence reporting rule.

Benchmark slices

3
Three provisional SimpleFold-3B targets have real prediction and scoring cards.

Phase 1A Amendment

The first paper is now scoped to a smaller, publishable unit: an 8-target SimpleFold-3B benchmark frozen on 2026-05-17. The public protocol lives at Phase 1A protocol, with manifests at CSV and JSON. The original 24-target/multi-model plan remains Phase 1B.

Scientific Question

Which open protein-structure prediction systems can a single researcher actually install, run, score, and audit on a 128 GB M3 Max workstation?

Minimum Publishable Scope

DimensionMinimumExpansion
Targets8 frozen Phase 1A chains24-100 chains
Core models1 for Phase 1A3-6
Repeats1 deterministic seed3 seeds / ensembles where supported
MetricslDDT, TM-score, pLDDT, wall-clock, peak memory, failure tagCalibration and per-bin analyses
OutputWorkshop paper + bioRxiv + public atlas packetJournal extension

Target Stratification

BinCountRationale
Short proteins, 30-80 aa6Fast sanity and fold coverage without reusing Phase 0 targets.
Mid proteins, 150-300 aa9Main quality and throughput anchor.
Long proteins, 500-800 aa6Memory and runtime stress.
Hard / disorder-rich cases3Failure-mode coverage, not a standalone model ranking.

Candidate Target Set

The first reproducible candidate set was generated from RCSB Search/Data APIs with release dates from 2025-10-01 through 2026-05-13. It excludes the Phase 0 engineering targets and marks every row as needs_biological_review. Technical fetch checks passed for all 24 rows: downloadable .pdb, selected chain present in ATOM/HETATM records, and matching FASTA chain/header.

BinCountPublic artifact
short6CSV
mid9JSON
long6CSV
hard3JSON

The enriched target review sheet flags antibody-heavy contexts, complex-derived chains, NMR references, membrane hints, and low-resolution references before promotion to the final manifest. The target triage layer converts those flags into provisional keep, human-review, and replacement-recommended categories.

Candidate Model Tiers

TierModelPhase 1 roleRequired outcome
CoreSimpleFold-3BNative MLX anchor.Run full Phase 1A frozen set; document any target-level failure.
CoreESMFoldMature non-AF3 baseline.Run where install path is clean; report as baseline if needed.
CoreChai-1 via chai-mlxCommunity MLX port.Run documented install and inference attempt; tag port failures precisely.
AttemptProtenix-v1Open AF3-class model with uncertain Apple Silicon path.Either run or produce reproducible port_unavailable / dependency_missing finding.
OptionalBoltz-2Stress case because affinity scope and CUDA orientation may confound the core question.Include only if MPS path is stable enough to avoid attribution confusion.

Run Order

  1. Produce quote-grounded SimpleFold recipe without fixture. Manual bridge complete; autonomous Reader credentialing remains a companion-methods target.
  2. Compare the no-fixture manual recipe against the benchmark protocol and paper claim boundaries.
  3. Freeze pilot/data/test_set_v1_phase1a.csv. Complete.
  4. Create the SimpleFold-3B install card for Phase 1A. Complete as a model/runtime card; defer multi-model install cards to Phase 1B.
  5. Run a 3-target Phase 1 rehearsal on targets not used in Phase 0. Technical rehearsal complete; direct long-target recovery complete.
  6. Apply direct-local run policy for final benchmark inference.
  7. Use the SimpleFold-3B short smoke and mid CLI smoke as install/orchestration evidence, not as final performance evidence.
  8. Convert the difficulty log into Methods and Limitations text as evidence accumulates.
  9. Apply the repeat/cache protocol before interpreting timing or confidence values.
  10. Use the first benchmark slices to validate sequence-aware scoring before final quality claims.
  11. Run the 8-target Phase 1A benchmark. Complete for SimpleFold-3B.
  12. Generate figures and tables from result cards, not hand-entered values. Initial Phase 1A figures complete; venue polish remains.
  13. Write Discussion only after results exist. Updated from Phase 1A outcomes; tighten after biological target review.

Claim Boundaries

Allowed: viability under this exact frozen protocol, model-specific run/fail status, quality and throughput on held-out targets, and public auditability. Disallowed: Apple Silicon beats CUDA, model quality rankings beyond the target set, or SimpleFold paper reproduction without comparable inference and published-metric comparison.

Source Documents