Training fixes before the next ePSF campaign: plan (2026-10-05)

Status: PLAN, awaiting user approval. Owner: Claude Code session 019KatSF. Experiment record: /astro/armin/koji/syndiff/dev_runs/training_fixes_20261005/README.md. Evidence behind every motivation below: /astro/armin/koji/syndiff/dev_runs/training_audit_20261005/README.md (figures fig1–fig6, epsilon/*.png).

0. Why

The 10-04 model-closure fits (25 K5 A3 folds over F1/F2/C1/C4/S22, plus the F1 confirmation arms) were audited on 2026-10-05:

1. Not converged. Stage 3 (colour terms train) hit its 2000-step cap in 24/25 folds while the loss still fell $1$–$4\times10^{-5}$ per step and the F1 fold-0 held-out core χ² still fell 3–4% per 500 steps.
2. The stop rule triggers falsely. It compares two single loss values 200 steps apart; on a loss with a period-2 Adam sawtooth (0.1–0.3%) it stopped C4 fold 3 and all three confirmation arms while the smoothed loss was still falling.
3. Bright stars are memorised. F1 held-out / in-training core χ² = 6.0 (T8–9), 2.3 (9–10), 1.39 (10–11), 1.0 (11–13). Only T8–11 stars train the shape; T8–9 carry 41% of the shape weight.
4. The weights are not calibrated. Held-out residuals scatter like a fractional model error of $\epsilon\approx2\%$ in bright pixels (measured on trainees: ~1%, i.e. memorisation), plus an additive excess (robust χ scatter 1.26 at sky level). Two of the three ePSF penalties are inert. The remaining penalty's strength changes with the number of pixels in the scene (×1.43 between C1 and C4, ×1.25 between a K5 fold and an all-star fit).

So the order is: fix the training code (Phase 0), measure how far the current endpoints still move when run to a true plateau (Phase 1), then decide the noise model (Phase 2) and the penalty weight (Phase 3) on converged fits. Phase 1 comes before the weight decisions because (a) it tells us whether the 10-04 verdicts survive, (b) it sets the step budget for the scans, and (c) ε must be re-measured on converged held-out residuals.

Phase 0: code fixes (no science runs)

Where. New worktree /home/kshukawa/syndiff_pipeline/.claude/worktrees/training-fixes, branch training-fixes-20261005, from 3609d46 (paper-pin-chain, the exact code of the 10-04 runs; scene_fit.py is identical on paper1-integration). Nothing changes in the live checkout or in any running pipeline. Every new behaviour is opt-in through a flag, so with the flags off the fitter is bit-identical to 3609d46.

0a. Smoothed stop rule with learning-rate decay

Running mean of the loss over the last $w$ steps and its relative decrease over a lag $K$:

$$\bar L_t=\frac1w\sum_{j=0}^{w-1}L_{t-j},\qquad s_t=\frac{\bar L_{t-K}-\bar L_t}{|\bar L_t|}.$$

• $L_t$: total loss at step $t$ (dimensionless).
• $w$: smoothing window, proposed 50 steps (longer than the observed period-2 sawtooth by a wide margin).
• $K$: comparison lag, proposed 500 steps.
• $s_t$: dimensionless relative decrease of the smoothed loss over the last $K$ steps.

A stage reaches a plateau when $s_t<\tau_L$ (proposed $\tau_L=10^{-4}$, i.e. 0.01% per 500 steps) and the physical quantities have stopped moving over the same lag (checked every 500 steps with one diagnose call):

$$\Delta_{\rm wcs}=\max_i\lVert \mathbf x_i(t)-\mathbf x_i(t-K)\rVert<\tau_x,\qquad \Delta_f=\operatorname{median}_i\left|\frac{f_i(t)-f_i(t-K)}{f_i(t)}\right|<\tau_f.$$

• $\mathbf x_i(t)$: modelled detector position (TESS px) of training star $i$ at step $t$, from the WCS leaf.
• $f_i(t)$: solved flux of star $i$ (e⁻/s).
• Proposed thresholds $\tau_x=0.5$ mpx and $\tau_f=10^{-4}$ (0.1‰): ten times below the effects we compare (route differences 0.59‰, model contrasts ≥0.37%).

On a plateau the learning rate of every bucket is multiplied by $\gamma=0.3$ and the window restarts; after $R=2$ reductions the next plateau ends the stage. This removes the sawtooth instead of stopping on it.

Also: the step caps become safety limits (20,000 per stage); the stop reason (plateau/cap) is written to history.jsonl and fit_meta.json, and a capped stage is labelled not converged; the gradient norm is logged per optimizer bucket (WCS, ePSF, colour), so we can see which parameters are still moving (needed to explain the confirmation fits' |g| jump from 5 to 200).

The thresholds $\tau_L,\tau_x,\tau_f$ are provisional. Phase 1 validates them by running every fit 2,000 steps past its stop and checking that the post-stop drift is below the thresholds.

0b. Penalty normalisation

The data term is an average over $N$ pixels, the penalty is not, so the same λ means a different strength in each scene. New form:

$$\mathcal L=\frac1N\sum_p\ell_p+\lambda_{\rm lp}\,\frac{N_{\rm ref}}{N}\,R_{\rm lp}[P].$$

• $\ell_p$: per-pixel data term, $\tfrac12\chi_p^2+\tfrac12\ln\sigma_{{\rm eff},p}^2$.
• $N$: number of active pixels in this fit (n_pix in the history).
• $N_{\rm ref}=1{,}581{,}636$: F1 fold-0 n_pix, so that F1 fold 0 is numerically unchanged.
• $R_{\rm lp}[P]$: the existing local-polynomial roughness of the ePSF $P$ (mean squared difference between $P$ and its 7×7 local-polynomial smoothing).

With this, λ chosen on K5 folds carries over unchanged to the all-star fit and to every field.

0c. Model-error floor and noise scale

$$\sigma_{{\rm eff},p}^2=s^2\sigma_p^2+\left(\epsilon\,\hat m_p\right)^2.$$

• $\sigma_p$: stored noise of pixel $p$ (e⁻/s).
• $s$: dimensionless noise scale (default 1), absorbs the additive excess seen at sky level.
• $\epsilon$: dimensionless fractional model error (default 0 = today's behaviour).
• $\hat m_p$: the scene model in pixel $p$ (e⁻/s), frozen at the last refresh (no gradient flows through it), recomputed at each stage start and every 500 steps. A frozen $\hat m$ keeps the loss comparable between refreshes (so the stop rule works) and stops the model from lowering its loss by inflating its own error.

$\sigma_{\rm eff}$ replaces $\sigma$ in both the loss and the island flux solve, so fluxes and shape are fitted with the same weights. The stop window restarts after each refresh. The fit has converged when, in addition, one refresh changes the median $\sigma_{\rm eff}$ by less than 1%.

0d. Recipe clean-up

lambda-lap and lambda-pixel-lap are set to 0 in the new recipe (they contribute ~10⁻⁹ of the loss today). Code kept.

0e. Tests (gate for Phase 1)

1. Flags off: 20 steps on F1 fold 0 reproduce the 10-04 history and parameters bit-identically.
2. Stop rule on synthetic histories: a sawtooth on a falling trend never stops; a flat sawtooth stops after $R$ decays.
3. Floor: with $\epsilon=0,s=1$ identical to (1); with $\epsilon>0$ the gradient through $\hat m$ is exactly zero.
4. Normalisation: F1 fold 0 loss unchanged; a fold with half the pixels gets twice the penalty factor.

Gate 0: tests pass; diff reviewed by me line by line; you approve before Phase 1 runs.

Phase 1: run current-recipe fits to a true plateau

Question. How far do the 10-04 endpoints move when run to a true plateau, and do two starting points then agree?

Recipe. Exactly the 10-04 A3 recipe (paper1_dataset: A3 + XP u, λ_lp 3×10⁸ (normalised = unchanged on F1 fold 0), no floor, no width, no background, no rejection) plus the Phase-0 stop rule. Same strict K5 scenes and training-only initialisations as 10-04, so each run starts exactly where its 10-04 counterpart started.

Fits (4, from scratch, not resumed — resuming would reset Adam and refresh the nuisance flux prior):

Each fit continues 2,000 steps past its stop at the final learning rate to validate the stop thresholds.

Measured, 10-04 cap endpoint vs plateau endpoint (same held-out cohort, same scorer): loss; held-out and in-training median core χ² per T bin; per-star flux change (‰, by T); WCS shift at star positions (mpx); colour coefficients; ePSF difference images; route agreement (two F1 starts: flux ‰, χ² ratio, WCS).

Figures (all inspected before reporting): learning curves to plateau with the 10-04 cap marked; matched held-out residual stacks cap vs plateau per T bin on a shared scale, plus their difference; ePSF plateau − cap images per tile; flux-change and WCS-shift maps; standard residual grid for each plateau fit (added to canvas UMr38JDhzXGZtTjv9kfCKC).

Resources. 8 CPUs, 16–24 GB per fit (10-04 measured 7.3–9.8 GB, CCD19 needed 24 GB). Expected 5k–20k steps at 0.5–0.7 s/step → 1–4 h each, run in parallel.

Gate 1 (report to you): if the endpoint moves more than the 10-04 contrasts (CCD19 +0.37%, width +1.85%) or the two starts still disagree by more than 0.5‰ in flux, the 10-04 verdicts are re-opened and must be re-run under the new rule. Phase 1 also gives the step budget for Phases 2–3.

Phase 2: decide the noise model (ε, s)

1. Measure on the Phase-1 plateau fits (F1 fold 0, F2 fold 0), held-out stars only: robust residual variance (1.4826·MAD)² per bin of model value $m$; fit $\operatorname{var}(r)=s^2\sigma^2+\epsilon^2m^2$ to get $\epsilon_0$, $s_0$. Check the form: implied ε flat in bright pixels and across T8–11 cores (it was on 10-04).
2. Self-consistency. Refit F1 fold 0 to a plateau with $(\epsilon_0,s_0)$, re-measure on held-out stars, repeat until $|\epsilon_{\rm meas}-\epsilon_{\rm used}|<0.2\%$ and $|s_{\rm meas}-s_{\rm used}|<0.05$ (at most 3 rounds).
3. Check by cross-validation, F1 K5 at plateau, ε ∈ {0, 1%, 2%, 3%} around the self-consistent value (20 fits), λ fixed at its normalised current value. Scored only with measures the floor cannot game (§ Scoring).

Gate 2: you approve (ε, s).

Phase 3: decide the penalty weight λ_lp

With (ε, s) fixed: F1 K5 at plateau, $\lambda_{\rm lp}=3\times10^8\times\{10^{-3},10^{-2},10^{-1},1,3\}$ (25 fits). The range is wide because the floor cuts the data's information on the ePSF core by $\times10^{-3}$–$10^{-2}$ and leaves the wings unchanged (audit epsilon/lambda_rescale.png). Then F2 K5 at the chosen value and its two neighbours (15 fits). Only if the best single λ leaves the core over-smoothed or the wings memorised: a per-zone λ (the penalty already has three radius zones) as a follow-up, with its own approval.

Gate 3: you approve λ_lp.

Scoring (the same for Phases 2–3; pass rules frozen in the experiment README before any scan runs)

1. Held-out median core χ² per T bin computed with the plain stored σ (not σ_eff), fixed cohort.
2. Held-out / in-training paired ratio $G$ per T bin (as in the 10-04 K5 figures), target ≤ 1.1 in every bin.
3. Width bias on the known-truth null test of localpoly_20260930.
4. Bullseye (m = 0) amplitude in held-out residual stacks.
5. Held-out flux vs Gaia-XP synthetic T per bin and near/far from the optical axis.

Every number comes with the residual stacks it was measured on (data − model per T bin and CCD cell, shared scale).

Out of scope (listed so they are not forgotten)

• Model capacity and roles (6×6 tile grid vs coarser/smooth; whether T11–13 stars train the shape): next plan, after Gate 3; probably the biggest memorisation lever.
• The native quasi-gradient (nuisance templates and anchors' ePSF derivatives are stop-gradient): a plateau is a fixed point of these estimating equations, not a minimum of the printed loss. Kept as today.
• Brightness width, background term, star rejection: unchanged (off).
• Final acceptance needs a fresh reserved exposure; the 10-04 F1 confirmation exposure has already been used.

Cost summary