OPTIMIZER AUTOPSY — Research Plan

00 · The paper in one page

Three contributions in priority order. The methodology leads.

Identity: causal localization + repair — not predict-and-repair. The abstract's one-liner: "We fork training at the moment of failure and intervene directly, showing where in the optimizer the damage lives, when it can be surgically repaired to recover the pre-spike trajectory, and when it provably cannot." Everything else supports that sentence. (SNR-SURGEON survives as the localizer inside C2, demoted from oracle to instrument.)

C1 · METHODOLOGY — the originality engine · ✅ BUILT, CUDA-VERIFIED

Training counterfactuals

A fork-and-intervene protocol: snapshot (w, m, v) at a detected pre-spike step, run matched branches under different surgical interventions plus controls — same data order, same seed — and measure divergence in final converged loss. Nobody does controlled interventional training-run science at scale; the closest work stitches checkpoints observationally (LLM360 K2). A reusable scientific instrument. Status: the harness, snapshot, and fork Δ==0 gate are done and verified on a Kaggle T4 (evidence in results/); the guarantee is being re-earned on AMD next.

C2 · SCIENCE

Where does the poison live?

Use C1 to causally attribute divergence across w, m, v, and test whether the damage is low-rank / subspace-structured or delocalized. Directly resolves the puzzle SPAM's moment-reset ablation raised. Directional SNR from microbatch disagreement is repurposed as the localizer/trigger — "which subspace is poisoned" — not a future-predicting oracle.

C3 · METHOD

Repair beats reset — and when it doesn't

Curvature/SNR-guided repair that rescales the identified poisoned subspace of v toward its pre-spike EMA while preserving descent information in the complement. Benchmarked against SPAM, ZClip, AdaGC, global reset, and the honest cheap baseline (spike-skip + clip) on recovery of final loss — not merely "survived divergence."

THEORY

A recovery theorem, not another onset theorem

Spike onset is taken (arXiv 2506.04805). We prove a localizability criterion for recovery: when rank-k repair provably matches global reset, when bulk poison mass makes reset provably necessary, and the computable crossover between them. Theory that predicts its own failure regime.

HONESTY MOVES

Built-in credibility

"Interventional attribution" in the sandbox; "causal" reserved for where it's earned. Natural spikes included (data-corruption class + LLM360 K2's released spike checkpoints), not just induced ones. Commutator correction measured, not assumed away. 3 seeds at 410M stated plainly.

⚠ Kill criteria (decided now, not in rebuttal)

If selective repair never beats global reset → pivot to "the poison is delocalized: a causal explanation of why global reset wins" — still a real result.
If final-loss recovery ≈ cheap spike-skip → the method is dead; ship C1+C2 as a science/benchmark paper to NeurIPS D&B.
Knowing the fallback is what makes this fundable.

01 · Background to master

Background to master

Concepts checklist

Adam's update rule, bias correction, first/second moments m and v, the preconditioner D = diag(√v + ε), and why v "lags" the instantaneous squared gradients (the decoupling mechanism of arXiv 2506.04805)
Hessian of the loss, eigenvalues/eigenvectors; GD stability threshold η·λ < 2; Edge of Stability (λ_max hovers at 2/η); adaptive EoS (preconditioned sharpness ≈ 38/η)
Stochastic linear recursions: AR(2) companion matrix, spectral radius, contraction rate, stationary variance — the entire theory workstream lives here
Gradient noise scale & SNR (McCandlish et al.); microbatch-disagreement variance estimation
Hessian-vector products via Pearlmutter's trick / double backprop; power iteration & Lanczos for top-k eigenpairs
Loss spikes: causes (LR schedule, low precision, Adam ε, data quality, hardware faults / silent data corruption) and current fixes (skip-and-reinject, checkpoint restart, clipping, moment reset)
Deterministic training replay: seeded data loaders, RNG state capture, bitwise-reproducible forks — the harness is only as good as its determinism
Small-scale-proxies methodology (Wortsman et al.) — how to induce instability deliberately and defend its validity

Videos — watch in this order

#	Resource	Why
1	3Blue1Brown — Essence of Linear Algebra (eigenvector chapters, esp. Ch. 14)	Eigen-intuition everything else builds on
2	3Blue1Brown — Neural Networks series (backprop chapters)	Gradient mechanics
3	Karpathy — Zero to Hero: "Let's build GPT" + "Let's reproduce GPT-2 (124M)"	The 124M video is your Week-2 infra tutorial. Watch twice.
4	Jeremy Cohen — Edge of Stability talks (YouTube: "Cohen Edge of Stability" — Simons Institute / ML theory seminars)	The stability regime the theory borrows
5	Any good "Adam optimizer explained / bias correction derivation" lecture + YouTube: "loss spikes LLM training"	You will be operating on m and v with a scalpel — know them cold

Courses, notes & tools

Roger Grosse — CSC2541 Neural Net Training Dynamics notes (Toronto) — the single best resource for this project's theory side
Stanford CS231n optimization notes (cs231n.github.io) · Boyd & Vandenberghe, Convex Optimization, Ch. 9 · any linear-systems treatment of AR(2) recursions
Tools: PyTorch autograd.grad(create_graph=True) for HVPs · PyHessian · nanoGPT (fork base) · ZClip official repo (baseline) · SPAM official implementation (linked from arXiv 2501.06842) · Weights & Biases — every run logged, non-negotiable
Natural-spike goldmine: LLM360 released paired spike/normal checkpoints from K2's real 65B training spikes (K2-Spike-1, K2-Spike-2 on HuggingFace) — study them in Week 1; they are free evidence of what real poison looks like

02 · Annotated reading list

Papers, in reading order

One shared Zotero library. Every paper gets a 5-line note: claim / method / what we borrow / how we differ / cite-where. Week 1: fresh arXiv sweep for 2025–26 spike papers. The onset-theory paper (2506.04805) must be cited prominently and the recovery claim staked sharply against it — same regime, different question.

Tier 1 — read fully, Week 1 · these define the battlefield

Paper	arXiv	Why it matters
Adaptive Preconditioners Trigger Loss Spikes in Adam (2025)	2506.04805	The onset theory — v decouples from squared gradients, preconditioned sharpness crosses threshold, five-stage spike anatomy. Our theory starts where this stops: recovery, not onset. Cite prominently.
Huang et al. — SPAM: Spike-Aware Adam with Momentum Reset	2501.06842	The named puzzle C2 resolves: their ablation shows moment reset matters but not where or why. Primary baseline + primary foil.
Kumar et al. — ZClip: Adaptive Spike Mitigation for LLM Pre-Training	2504.02507	Baseline — z-score EMA anomaly detection on gradient norms; reimplement from their repo
AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping	2502.11034	Baseline — per-parameter adaptive clipping; also the catalog of natural spike causes (data, hardware, precision, hyperparameters) our spike-sourcing section must answer
Molybog et al. — A Theory on Adam Instability in Large-Scale ML	2304.09871	The original "poisoned moment buffer" story — foundation of the repair framing
Wortsman et al. — Small-scale Proxies for Large-scale Transformer Training Instabilities	2309.14322	Spike-induction methodology; we inherit its legitimacy but must additionally validate mechanism transfer across our own two scales

Tier 2 — read fully, Weeks 1–2 · evidence, controls, neighbors

Paper	Where	Why
LLM360 — K2: Building a 65B 360-Open LLM (spike sections + released spike checkpoints)	2501.07124	The observational version of what we do interventionally; their public spike checkpoints are our natural-spike test set
Ma et al. — Understanding Silent Data Corruption in LLM Training	2502.12340	The SDC case: a spike near the end of fine-tuning → zero test accuracy. Proof that "reaction is enough" fails somewhere — the regime C1 must find
Cohen et al. — Adaptive Gradient Methods at the Edge of Stability	2207.14484	Preconditioned sharpness ≈ 38/η — the working eigenbasis for the localizer
Cohen et al. — GD Typically Occurs at the Edge of Stability	2103.00065	EoS foundations
McCandlish et al. — An Empirical Model of Large-Batch Training	1812.06162	Gradient noise scale / SNR formalism the localizer generalizes directionally
Damian, Nichani, Lee — Self-Stabilization: The Implicit Bias of GD at EoS	2209.15594	Implicit projected-GD machinery reused in the repair analysis
Arora, Li, Panigrahi — Understanding GD on the Edge of Stability	2205.09745	Quadratic-regime proof techniques
Chowdhery et al. — PaLM (loss-spike sections)	2204.02311	The restart-and-skip folklore recipe we turn into a measured operation

Tier 3 — skim for related work, Weeks 2–4

Zeng et al. — GLM-130B (arXiv 2210.02414) — real-world spike tricks (embedding-gradient shrink) · Takase et al. — Spike No More (2312.16903) — prevention-side neighbor
Ghorbani, Krishnan, Xiao — Hessian Eigenvalue Density via Lanczos (1901.10159) · Yao et al. — PyHessian (1912.07145) — spectral tooling
SDC follow-ups: Exploring SDC as a Reliability Challenge in LLM Training (2604.00726), LLM-PRISM (2604.10390) — natural-spike taxonomy
SAM (2010.01412) — contrast: prevention vs. autopsy · Zhang et al. — Why Gradient Clipping Accelerates Training (1905.11881)
Anything 2025–26 matching: "loss spike", "training instability", "optimizer state corruption", "Adam second moment", "counterfactual training"

03 · Formula sheet

All the math

Setup

L(θ), Hessian H = ∇²L(θ), eigenpairs (λᵢ, uᵢ), λ₁ ≥ λ₂ ≥ … Adam state: m_t (first moment), v_t (second moment), D_t = diag(√v_t + ε) preconditioned Hessian H̃ = D^(−1/2) H D^(−1/2) — top-k via Lanczos with preconditioned HVPs minibatch gradient g = ∇L(θ) + ξ, Cov(ξ) = Σ/|B|

C1 · The fork-and-intervene protocol — the instrument

# at localizer trigger step t₀ (pre-spike): snapshot S = (w_{t₀}, m_{t₀}, v_{t₀}, RNG state, data cursor) branches, ALL with identical seed + data order: B₀ no-op # let the spike happen — the damage measurement B_w repair w only B_m repair m only B_v repair v only B_g global reset of (m, v) # SPAM-style — the folklore arm B_r random-subspace repair, matched rank & strength # the causal control B_s spike-skip + clip # the honest ≈0-cost baseline B* clean counterfactual # spike batch replaced, run continues measurement: Δᵢ = L_final(Bᵢ) − L_final(B*) attribution: effect(component c) = Δ_noop − Δ_repair-c # minus B_r control # cost control (mandatory): establish ONCE, on a small set, the lead time between # immediate-recovery divergence and final-loss divergence; then use short forks # (500–2000 post-fork steps) as the proxy for the big sweep, full-length forks # only on the decision set. Without this the compute plan is fantasy.

C2 · The localizer — directional SNR, demoted from oracle to instrument

SNR_u = |ḡᵀu| / √(var̂(gᵀu)/m) # microbatch-disagreement estimator, m microbatch grads # grads g₁…g_m already exist under gradient accumulation — m dot-products per direction, free poison score along uᵢ: pᵢ = |uᵢᵀ(v_t − v̄_pre)| / uᵢᵀv̄_pre # v̄_pre = pre-spike EMA of v poisoned set: P = { i : pᵢ > τ_p and SNR_uᵢ < τ_s } spectral-mass concentration: ψ_k = ‖Π_k δ_v‖ / ‖δ_v‖, δ_v = v_t − v̄_pre, Π_k = top-k projector # VALIDATED CLAIM, not a footnote: measure noise–curvature alignment directly at both # scales, in bf16, and report where it degrades. Fallback localizer: a few power-iteration # steps on top-k curvature directions — cheap, robust rather than faith-based.

Theory · Recovery, not onset — the localizability criterion

# A spike injects perturbation δ = (δ_w, δ_m, δ_v) into the state. Model recovery as the # Adam-quadratic stochastic linear recursion acting on δ (frozen-preconditioner leading order): x_{t+1} = x_t − η D^(−1/2)(H x_t + ε_t)D^(−1/2)-terms + β-momentum coupling # per-direction: AR(2) with contraction rate ρᵢ(λ̃ᵢ, η, β) = spectral radius of companion matrix THEOREM 1 (selective repair suffices). If ψ_k ≥ ψ*(gap): rank-k repair on exactly the top-k curvature subspace restores ρ < 1 in every direction and the trajectory re-converges to the clean path up to O(η) ⇒ selective repair is provably as good as global reset, while preserving the complement's descent information. THEOREM 2 (reset necessity). If bulk mass 1 − ψ_k exceeds threshold: ANY rank-k repair leaves a residual amplification factor > 1 ⇒ global reset is provably necessary. # This explains SPAM's empirical result: magnitude-based selection picks the wrong # subspace, or the poison is genuinely delocalized in their regime. CROSSOVER: ψ* is a computable threshold in the poison's spectral-mass concentration relative to the top-k / bulk curvature gap (λ̃_k − λ̃_bulk). # HONESTY CLAUSE — the non-commutativity hole is load-bearing: decoupling per-direction # assumes D^(−1/2) commutes with H's eigenbasis; it doesn't. Frame everything as the # leading-order characterization and MEASURE the commutator's effect on ψ* at 124M. # Claim a leading-order theory with a measured correction — never a guarantee. # FALSIFIABLE PER-INSTANCE PREDICTION (top-venue signature): measured ψ_k predicts, # per-spike, whether selective repair beats global reset. Then test it.

C3 · The repair operator

# rescale the identified poisoned subspace of v toward its pre-spike EMA, # preserve descent information in the complement: v_t ← v_t − Σ_{i∈P} c_i · uᵢuᵢᵀ (v_t − v̄_pre) applied in the diagonal approximation m_t ← m_t − Σ_{i∈P} (m_tᵀuᵢ) uᵢ # remove poisoned first-moment components c_i ∈ [0,1] scheduled → full repair over R recovery steps # required ablations: (a) repair v vs m vs w vs all (b) subspace vs global # (c) magnitude-selected (SPAM-style) vs curvature/SNR-selected subspace (d) no-op

Metric hierarchy — in this order, always

1 · Final converged loss vs clean counterfactual — the headline. Reactive methods report "prevented divergence / lower perplexity"; almost none prove they recover the counterfactual clean trajectory. We do, because C1 gives us the counterfactual.
2 · Compute to recover — steps × FLOPs to re-enter δ-band around the clean path, vs checkpoint-restart-and-skip accounting
3 · Spike survival rate — the field's usual metric, reported last on purpose
Attribution map — effect(w), effect(m), effect(v) with CIs across ≥5 seeds; random-subspace control subtracted
Localizability — ψ_k distribution per spike class (induced-LR, induced-ε, induced-precision, data-corruption, K2-natural); theory-predicts-practice scatter: ψ_k vs (selective − global) outcome
Overhead — wall-clock % of localizer (eigenvector refresh + SNR tracking) and snapshot I/O

04 · Infrastructure & compute

Harness, data, GPUs — and the fork-cost arithmetic

Repo layout

optimizer-autopsy/
  harness/  # fork/snapshot/replay — the product
  localizer/ · repair/ · theory/
  experiments/llm/ (nanoGPT fork) · experiments/proxy/
  baselines/ (SPAM · ZClip · AdaGC · clip · skip)
  analysis/ · spikes/ (recipes + K2 checkpoints)

Deterministic replay is a first-class requirement: seeded loaders, RNG state in every snapshot, bitwise-identical trunk verification in CI. A 10-minute smoke test forks a tiny run and asserts branch divergence is zero under no-op-with-no-spike. Reproducibility isn't a virtue here — it's the instrument's calibration.

Datasets & spike sourcing

LLM track: OpenWebText or FineWeb 10BT sample (HuggingFaceFW/fineweb) on GPT-2 124M and Pythia-style 410M. You need spike windows, not convergence: 2–5B tokens per 124M run.

Induced spikes: high LR, tiny Adam ε (1e-12), bf16→fp16 on sensitive ops, weakened QK-layernorm.
Natural spikes (mandatory): data-injected corrupted-batch spikes — the cheapest natural class (AdaGC's catalog: data quality, hardware faults, precision, hyperparameters) — plus LLM360's released K2 spike/normal checkpoint pairs as a real-world validation set. A repair that only handles ε-perturbation spikes at 124M is a toy.

The audited AMD compute ask — roughly 15–40 GPU-hours, not 500

The earlier "500–600 GPU-hours" figure conflated two different resources: build-effort (person-hours) and GPU-compute. A bottom-up recount — grounded in a measured 0.0619 s/step at proxy scale (11M params, batch 16, Tesla P100; committed under research/kaggle/step_timer_results.md) and a full code audit of which components are actually GPU-bound versus engineering-time — puts the real AMD GPU ask at ~15–40 GPU-hours. That is ~5–10h to re-earn bit-exact determinism on AMD/ROCm (the one line nothing has tested — every measurement so far ran on NVIDIA/CUDA free-tier Kaggle) plus ~3–30h of proxy-scale GPU-bound science (calibration + attribution battery). The ~500 remaining hours are build-effort, which drives the ~24-week wall-clock, not the GPU grant. Path B: port every hardware-independent piece (data, model, spike recipes, fork/branch design, and the already-verified determinism/snapshot/fork spine) unchanged; re-earn only the ROCm determinism guarantee. A cheap go/no-go smoke test runs first — one forked pair, one step, compare m/v/w bit-for-bit over the exact ops this pipeline uses (HVP double-backward + Adam moment update) — because ROCm has no exact CUBLAS_WORKSPACE_CONFIG analog and bitwise Δ==0 on MI300X is not assumed.

Bottom-up recount. Build-effort = person-hours (drives wall-clock, not the GPU grant). GPU-h (audited) = real proxy-scale compute at the measured 0.0619 s/step, low–high; the assumption behind each range is stated. Rows marked ENG are engineering-bound (their GPU cost is only dev-validation runs); GPU rows are genuinely compute-bound.

Line item	Build-effort (h)	Bound by	GPU-h (audited)	Basis
AMD determinism re-verification	30	ENG + AMD	5–10	The one unverified line. Bit-exact Δ==0 has only ever run on NVIDIA/CUDA; re-earning it on ROCm is the go/no-go. Small compute, high risk.
Spike induction + detection	30	ENG	0.1–1.4	induce + detector built & run (1/4 recipes pass DoD); remainder is coding + threshold sweeps over cached runs, not retraining.
Cheap-fix kill-test	15	ENG (built)	0.1–0.5	Battery + Gate-B verdict built, not yet run: 4 branches × (2–4 recipes) × 3–5 seeds × 200 steps.
Localizer (C2)	70	ENG	0.1–2	Pure TODO stubs today; HVP/eigensolve on the 11M proxy is cheap — the 70h is coding, not GPU.
Repair operator (C3)	60	ENG	0.2–1.7	9-line stub; repair is a cheap tensor edit validated by 50–200 short forks vs the clean counterfactual.
Baselines	45	ENG	0.2–1.3	skip/clip/reset real; SPAM/ZClip/AdaGC are 3-line stubs — reproducing published methods is human time.
Short-fork calibration	55	GPU	2–14	Genuinely GPU-bound: ~20 full-length proxy forks (5000 steps) to fix the shortest fork length that preserves branch-ordering.
Full attribution battery (proxy)	120	GPU	0.4–4.8	recipes × 7 branches × seeds × fork-len × 0.0619s ÷ 3600: (2×7×3×500)→0.4h … (4×7×5×2000)→4.8h. A ceiling.
Commutator-error measurement	15	Analysis	0.1–1	Measured once at 1–4 confirmed spike sites (a few HVPs each).
Contingency	60	Buffer	—	Effort buffer, not GPU compute.
Total	~500	—	~15–40	The ~500 is build-effort (PLAN V6 §5 flagged the GPU-vs-dev-hours conflation). The audited AMD GPU ask is ~15–40 GPU-h.

Off the AMD ask — the 124M robustness check

The larger-scale (124M) robustness battery measures ~71–302 GPU-hours per battery (measured 124M step-time 1.08 s/step at batch 8 on P100, grad_accum=40; see step_timer_results.md §2). It is designed to run on free-tier Kaggle over multiple weeks per BUILD_PLAN Task 22's own weekly-cap survival design — not on the AMD grant, and it is not part of what is requested from Exea Labs.

Written cut order (decided now, not mid-crunch): (1) shrink to the recipes that work; (2) fewer seeds, flagged as a limitation; (3) drop the two adaptive-clipping baselines; (4) measure the commutator error once, not per site. Never cut: the exact-zero replay proof or the held-out validation methodology. Snapshots: proxy (w,m,v) fp32 ≈ 12–36 MB, 124M bf16 ≈ 0.75 GB → rotating latest.safetensors on HuggingFace. Underlying data + full audit: research/kaggle/step_timer_results.md; plan detail: PLAN_V6.md.

05 · Team & resources

Who's building it, and what's still open

Scheduling uses a conservative single-contributor-plus-mentor assumption. The team-size figure is a PLAN V6 open item: different numbers have circulated (five with a mentor; eight per outreach framing) against a commit history showing one author. It needs to resolve to one accurate figure before it appears in any funding or compute request — cheap to fix now, costly to credibility if a partner checks it later. If a verified team with assigned roles is confirmed, the localizer and repair builds can run in parallel and the wall-clock timeline compresses; the GPU-hour budget itself does not change.

Work area	State	Scope
Instrument (C1) — harness, snapshot, fork Δ==0 gate	✅ done, CUDA-verified	Deterministic replay, name-keyed (w,m,v,RNG) snapshot/restore, fork-and-compare. Re-earn on AMD via the go/no-go gate.
Spikes + kill-test	🟡 partial	1 of 4 induced-spike recipes solid; cheap-fix battery built, not yet run to a verdict.
Localizer (C2) + repair (C3) + baselines	⬜ not started	Directional SNR + preconditioned-Hessian curvature → poison score; rank-limited projection repair; SPAM/ZClip/AdaGC/skip+clip/reset baselines. The genuinely novel, highest-risk work.
Theory	⬜ not started	AR(2) recovery criterion + ψ_k crossover; commutator-error measurement; the per-spike falsifiable prediction. Mentor: proof checking, venue strategy, endorsement.

Compute (open items, from PLAN V6 §5): the audited AMD GPU ask is ~15–40 GPU-hours (see §04) — the earlier 500–600h figure was build-effort, not GPU-compute. Still to confirm with Exea Labs: the realistic single-project AMD allocation, plus GPU parallelism + any grant expiry, since those decide whether the (small) GPU-hours or personal dev-hours is the binding constraint. Nothing is locked until the Exea inquiry resolves.

06 · Phase by phase

The V6 timeline (~24 weeks of build-effort)

The per-phase hour tags below are build-effort (person-hours), not GPU-hours — the audited GPU-compute ask is the separate ~15–40 GPU-h in §04. Wall-clock is the single-contributor-plus-mentor pacing (~24 weeks); it compresses if a verified team or parallel GPUs materialize. The week numbers are relative to the AMD grant landing, not fixed calendar dates. Venue: TMLR (no deadline, judges claim-support) and/or a NeurIPS 2027 cycle — NeurIPS 2026 has passed, and 2027's deadline is a historically-grounded prediction (2024 May 22 → 2025 May 16 → 2026 May 6), not a locked date.

PHASE 1 · WKS 1–3

Re-earn determinism on AMD, finish the spike-maker

Week 1Go/no-go

The go/no-go determinism smoke test (~6h): one forked pair, one step, compare m/v/w bit-for-bit over the exact ops this pipeline uses (HVP double-backward + Adam moment update). This decides which tier is buildable.
Decision point: exact zero on MI300X → continue on a single-run bitwise proof. Can't hit zero → pivot Phase 1 to a tolerance-based statistical framework (paired seeds, many-run averaging).

Weeks 1–3Re-verify + spikes

Remaining AMD determinism re-verification (24h build; ~5–10 GPU-h — the one line nothing has tested, all prior gates ran on NVIDIA/CUDA): derive the ROCm strict-mode setup + pin set; re-run the noop-vs-noop gate at proxy and at the larger scale.
Finish spike induction to the relaxed V6 DoD of ≥2 solid recipes (30h). The old Kaggle P100-vs-T4 blocker is moot on AMD, but recipe numerics may shift and need re-checking.

PHASE 2 · WKS 4–5

The cheap-fix kill-test, early and pre-registered

Weeks 4–5Kill-test

Run the cheap-fix battery (15h) against a threshold written down first: does skip+clip already recover held-out loss everywhere? If yes → the repair method is unnecessary and C1+C2 ship as a science/benchmark paper. A legitimate, valuable outcome, decided before the expensive machinery is built.

PHASE 3 · WKS 6–20

Build the science: localizer, repair, attribution

Weeks 6–12C2 + C3

Build and locally validate the localizer (70h) — directional SNR + preconditioned-Hessian curvature → poison score + ψ_k — and the repair operator (60h), the rank-limited projection.
Reimplement baselines (45h): skip+clip, SPAM, ZClip, AdaGC, naive full-reset, each as a fork intervention.

Weeks 13–15Calibration

Short-fork calibration (55h build; ~2–14 GPU-h): find the shortest fork length that preserves the branch-ordering of Δ, back-solved against the measured 0.0619 s/step from the committed CUDA logs so the battery fits the budget.

Weeks 16–20Attribution

The full attribution battery (120h build ceiling; ~0.4–4.8 GPU-h at proxy scale): multiple spike sites × branches × seeds at the calibrated length, sized to the surviving recipe count, trimmed by the written cut order if hours run short.
Measure the commutator-error correction (15h) across confirmed spike sites.

PHASE 4 · WKS 21–24

Write up and submit

Weeks 21–24Paper

Write in contribution order: C1 methodology → C2 attribution → C3 repair → theory. Every claim maps to a figure; "causal" appears only where the random-subspace control earned it.
Internal + mentor review; reproducibility package (the fork harness is the artifact).
Submit to TMLR (no deadline) and/or freeze for a NeurIPS 2027 cycle per the mentor's read. The pre-committed stop-and-write-up triggers (PLAN V6 §8) decide when to stop, regardless of hours remaining.

07 · Publication strategy

Venues & honest odds

No reallocation of a fixed compute budget engineers a high acceptance rate — review is noisy even for strong papers, and the top tier needs a different category of contribution than a well-executed application of existing tools to a scoped question. The honest ceiling here is a genuinely strong, technically sound single contribution: exactly what TMLR is built to reward, and a reasonable NeurIPS D&B or workshop candidate. Not a near-certain top-tier acceptance, and this plan does not claim to be.

Artifact	When	Venue	Fit
Preprint (C1 + C2 + theory sketch)	After the attribution battery	arXiv	Direct
Full paper	~Weeks 21–24	TMLR (rolling; judges claim-support, no deadline)	Strong structural fit
Full paper (alt)	NeurIPS 2027 cycle (predicted ~May 2027)	NeurIPS 2027 main / D&B	Reasonable
If the method dies (kill-test)	Decided in Phase 2	NeurIPS D&B — the harness + causal benchmark	Reasonable
NeurIPS 2026 main	Out of reach — the May 6, 2026 deadline has passed.		—

BUDGET DISCIPLINE

Claim sized to compute

Three tiers (Floor / Core / Stretch) with a written cut order, so a smaller-or-larger grant needs no renegotiation. The kill-test and go/no-go run first, so months aren't spent on a dead or unprovable contribution.

ORIGINALITY

Concentrated in C1 + C2

Training counterfactuals as an instrument: no direct prior work found. C2 resolves a named, published puzzle (SPAM's ablation). C3 is "beat reset" — incremental, never the lead. Write the paper in exactly this order.

STRATEGY

Outcome-robust, honestly

Repair works → C1+C2+C3; cheap fix wins → C1+C2 benchmark; poison delocalized → "why reset wins." Every branch is a real paper. "Causal" only where the random-subspace control earned it; pre-registered thresholds remove the pull toward the exciting read.

Open logistics (PLAN V6 §5, §11): resolve the team-size figure, confirm the realistic Exea allocation, and confirm GPU parallelism + expiry before locking a tier. Author order, mentor affiliation, and endorsement follow from the team resolution.

08 · The brutal review, kept in the plan on purpose

What can kill this — pre-attacked

Every item below was raised adversarially against this plan. They stay on this page so nobody discovers them in a rebuttal.

Risk #1 — AMD/ROCm bitwise determinism may not be achievable (the load-bearing one)

The entire evidentiary standard is exact-zero replay, and the CUDA trick that earned it (CUBLAS_WORKSPACE_CONFIG set before torch imports) has no exact ROCm analog; rocBLAS/hipBLASLt determinism is handled differently and deterministic-kernel coverage is less complete than CUDA. If Δ==0 is unachievable on MI300X, the proof standard collapses on that hardware. Mitigation: the ~6h go/no-go smoke test runs first, on the exact ops this pipeline uses; if it fails, Phase 1 pivots to a tolerance-based statistical framework (paired seeds, many-run averaging) rather than discovering the wall weeks in. A useful narrowing: the eigensolve appears to run its iterative solve on the host and farm only matvecs to the GPU — so the burden may be one bit-reproducible HVP, not a whole multi-step solve. Confirm that.

Risk #2 — the compute grant, and its terms, aren't confirmed

The AMD restart is contingent on an Exea Labs grant that is requested, not approved. The audited ask is modest — ~15–40 GPU-hours (the old "500–600 GPU-hours" was build-effort mislabeled as compute) — so the binding constraint is more likely personal dev-time than GPU-hours, but "24 weeks" and any hour figure are only the same constraint if there's one GPU with no expiry. Mitigation: three scope tiers (Floor/Core/Stretch) so a smaller-or-larger grant needs no rewrite; PLAN V6 §5 lists the exact questions to resolve (realistic allocation, parallelism, expiry) before locking a tier. Abandoning a working CUDA stack for an unconfirmed one is the real cost if this falls through.

Risk #3 — C1 might show that reaction is enough

If the kill-test reveals that one-step-late spike-skip recovers held-out loss as well as any surgery, the repair contribution collapses — a live possibility, since the field skips-and-reinjects routinely. Mitigation is baked in: the cheap-fix battery runs early (Phase 2), against a pre-registered threshold, so the answer is known before the expensive machinery is built. If reaction wins → C1+C2 ship as a science/benchmark paper; the methodology de-risks everything either way.

Risk #4 — the theory's non-commutativity hole is load-bearing

Per-direction decoupling assumes the preconditioner commutes with the Hessian eigenbasis; it doesn't, and the onset paper (2506.04805) already lives in this regime. Mitigation: frame the decoupled result as the leading-order characterization, measure the commutator's effect on ψ* empirically, and claim a leading-order theory with a measured correction. A measured correction is more credible than a clean-but-false bound. PLAN V6 §8 pre-registers what a convincing ψ_k signal is before it decides anything.

Risk #5 — "causal" is still slightly overclaimed

Fork-and-intervene with matched seeds and a random-subspace control is genuinely interventional — far stronger than observational checkpoint stitching — but it's causal within the induced-spike sandbox; transfer to natural spikes is inference. Mitigation: say "interventional attribution," reserve "causal" for the sandbox, and let the K2 + corrupted-batch results carry the transfer argument.

Novelty audit — honest verdict

C1 (training counterfactuals as an instrument): freshest thing here, no direct prior work found, and it's already built and CUDA-verified. C2: resolves a named, published puzzle — reviewers value that above generic gaps. C3: "beat reset," incremental, never the lead. Theory: differentiated (recovery vs onset) but shares a regime with 2506.04805 — cite it prominently, stake the recovery claim sharply. Net: a defensible novelty profile whose strength is the methodology and the puzzle-resolution, not the optimizer tweak.

Immediate actions (PLAN V6 open items)

Confirm the Exea Labs AMD grant — get a yes/no and an actual hour figure; the audited need is only ~15–40 GPU-hours, so ask what a realistic single-project allocation looks like at that scale.
Confirm GPU parallelism + any grant expiry — this decides whether GPU-hours or personal dev-hours is the binding constraint.
Resolve the team-size figure to one accurate number before it appears in any funding or compute request.
Build the go/no-go determinism smoke test — the first thing to run on AMD, before the full determinism budget is spent.
Have the mentor (or another qualified reader) read the fork/branch design fresh for confounds before fully committing to Path B.