Colour × magnitude residuals across the CCD

Final run (run5): pooled over the 5 cross-fit folds, 2026-10-14

The paper pipeline's last run (run5: run4 + PS1 template calibration), folds_final, all five fields. Each fold's ePSF is trained on 4/5 of the stars; the grids pool all 5 folds. Held-out: every star once, rendered by the model that did not train on it. In-fold: training stars, each 4×. All: every star under all 5 fold models (1 held-out + 4 in-fold). Same layout and colour scale rule as the tabs below.

Pooling: /astro/armin/koji/syndiff/dev_runs/group_meeting_slides_20261013/run5_pooled/pool.py (concatenates e2e_final_recipe_20261007/run5/<F>/folds_final/eval/fold<k>[/insample/{infold,all}]/raster_res_model.npz), grids by residual_grid.py. Held-out stars: F1 10536, F2 8766, S22 8491, C1 8224, C4 13528.

G1 held-out harness: out-of-fold grids, 2026-09-29

Paper 1 gate G1. The CCD is split into 2 folds of 128-px tiles. Each fold fit starts from a leak-free init (another sector's A3 fit for the same CCD; for F1 that is S25 C2K2) and uses the decided recipe: A3 + live moment-blind prior, 300/3000/2000 steps. Every star is then rendered by the fit that never trained on it. Tabs ho_F1_*: full_xsec = all-star fit, in-sample (the provisional Paper 1 calibration); oof_xsec / oof_smooth = out of fold, from two different inits. Record: dev_runs/heldout_harness_20260929/README.md.

• Bright rows out of fold show a sample-scale checkerboard at ±30e-4 that the in-sample grid hides. That is ePSF fine structure fitted to the training stars (R_mem = 40 at T8–9, 9 at T9–10, 2.7 at T10–11, 1.04 at T11–13).
• Held-out χ² rises with training. The optimum is at ≤300 ePSF steps. At T<11, no training fraction beats the other-sector init on unseen stars. The live moment-blind prior drives most of the faint-star drift: +36–45% with it, +6–9% with the plain prior (figure).
• The red-column dipole/ring is the same in and out of fold, so it is a model term, not memorisation.

Colour model v2, stage 2b and decision, 2026-09-29

Five hp_d frames, approved by the user: S24 C1K2 (cam1), S23 C2K1 (cam2), S26 C3K1 (cam3), S23 C4K2 (cam4, still running), and S25 C2K2, which is F1's CCD one sector later. Each is B14 → A3. Tabs s0023*/s0024*/s0025*/s0026*. Proposed model: A3, in dev/forward_epsf_wcs/docs/COLOUR_MODEL_V2_DECISION_20260929.md. It awaits approval.

• F1's width band is a fixed feature of camera 2 / CCD 2. S25 C2K2 reproduces F1's radial−tangential map almost cell for cell, a sector later: −11.9 vs −12.2 and −8.8 vs −11.0e-3 px². A3 leaves r−t χ²/n at 53 there, vs 56 on F1. Camera 2's other CCD shows only a weaker, repeatable remnant. On every other camera/CCD, A3 r−t χ²/n is 2.8–15.8.
• Width planes repeat per CCD across sectors. F1 vs S25 give q1_x +0.0039/+0.0048 and q2_0 −0.0019/−0.0020.
• Across all 10 fields, A3 takes the red-end shift from +12..+18 to −1..+5.5 mpx and red−mid to 0.6–0.8× of #14. Photometry doesn't regress. A constant red-broader round-width offset (≈ +2e-3 px²) remains everywhere.

S23 C4K2 (cam4), added after the write-up. The width is fixed: r−t χ²/n 41 → 9.9, 45° 8.3 → 2.1, and the q2 planes repeat S20 C4K2. But the red end overshoots, from +5.6 to −4.6 mpx over all stars, while beyond 1800 px it is still +3.8. This field also has the only photometric colour-slope regression, −0.23 %/mag. It is limit 6 in the decision doc.

Colour model v2, stage 2 (four more CCDs), 2026-09-29

Repeatability test of the two stage-1 models on raw FFIs S20 C3K3 (cam3; same frame as F2), S20 C4K2 (cam4) and S20 C2K1 (cam2), plus the hp_d lane S22 C3K3 (cam3). For each lane: B14 (#14 trained from the lane's stage-2 fit), then A3 (u + quad + planes) and A6 (u + quad + axis astigmatism), each warm-started from B14 for 1500 steps. Tabs s00*_B14/A3/A6. Record: dev_runs/chroma_v2_20260928/README.md.

• Shift: A3 brings the reddest bin to +1.6, +1.0, −1.0 and +3.6 mpx, from +12 to +15. Beyond 1800 px it reaches +1.8, +4.0, +3.8 and +7.5.
• Width: the CCD-frame planes work on all four lanes, with none failing the way F1 does. Radial−tangential χ²/n goes 39→9, 20→3, 22→9 and 42→9; 45° goes 16→5, 6→3, 15→8 and 10→6.
• Round-width offset: the round width is now the largest leftover (χ²/n 10–34). It is mostly a constant red-broader offset of about +2e-3 px² (χ²/dof about its mean is only 2–3).
• S20 C2K1 (cam2): it keeps a weaker version of F1's mid-distance radial−tangential band, −4e-3 px² vs F1's −12e-3. F1 is also a cam2 CCD, which points to camera 2. Stage 2b tests this on S23 C2K1 and S25 C2K2.
• A6 vs A3: A6 is never better than A3, and its 45° term is worse on three of four lanes. At r = 1500 px the axis astigmatism is about 0.003–0.0045 on cams 2 and 3 but −0.001 on cam4.
• Red−mid rms: it falls from 7.7–8.6 to 5.2–7.1. Photometry doesn't regress, and core χ² drops 7–14%.

Colour model v2, stage 1, 2026-09-28

Eight arms warm-started from #14 (sffi_rawgauge_20260924 F*_D1_raw_dilR), each run for 1500 steps with the new colour terms starting at 0. The code is dev branch chroma-v2 f5e3605. Scoring uses S4's scorecard on the fixed C5 star sets. Tabs F*_A0…A7. Record: dev_runs/chroma_v2_20260928/README.md.

• The quadratic-in-colour shift halves the red-end shift on both fields. F1 goes from +17 to +7 mpx and F2 from +15 to +6.5 mpx. The coefficients agree between fields (sq1 −0.0055 / −0.0057).
• Gaia XP u as the colour input does most of the rest. F2 lands within ±2 mpx at every colour, including beyond 1800 px from the axis (A3, A5). F1 reaches +4 to +5 mpx.
• The CCD-frame colour elongation planes fix F2's width but only part of F1's. Radial−tangential χ²/n: F2 42 → 14, F1 73 → 53. F1's leftover is a band in distance from the optical axis: about 0 near the axis corner, −12e-3 px² at mid-distance and −5e-3 px² in the far corner. A plane cannot fit that shape, and neither can an axis astigmatism linear in r. Axis astigmatism alone (A6) gives the same astig0 on both CCDs (+0.0030 / +0.0028) but fits worse than the planes.
• The red−mid residual rms falls from 9.9 to 7.2–7.4 on F1 and from 7.9 to 5.5 on F2. That is still short of the ≤5.5 / ≤5.2 targets. Core χ² and the Gaia flux colour slope improve slightly or stay the same.
• Stage-1 criteria are not met for F1's shift, for red−mid on either field, or for any width χ²/n. Stage 2 is waiting for a decision.

Held-out memorisation control, 2026-09-28

C5 recipe retrained on split30 (30% of T8–13 stars held out, C5 warm start). Tabs cl_F*_C5s_warm. Record: dev_runs/epsf_width_decision_20260928/README.md.

• ePSFs memorise their training stars. Held-out T8–9 core χ² is ×5.4 (F1) / ×5.6 (F2) higher than for production C5, which trained on those same stars. T11–13 anchors, trained by neither, agree to 1–3%.
• Earlier in-sample comparisons are affected. This covers "faint ePSF 30–40% better" (it is 2–10% on unseen stars), the bright blend a(q), and the lr 1e-5 win above (T10–11 held stars leak through the C5 warm start).
• The brightness-dependent width is real on held-out stars. It runs from +1.0 at T8.5 to −1.3 at T12.5 (1e-3 px², F1), a steeper trend than in-sample.

Prior fix + optimizer schedule: held-out sweep, 2026-09-25 evening

Faint training (isolated T10–13 trainees, 30% held out; scenes faint10split), 46 arms × F1/F2, all eval on the fixed C5 scene. Scored on the isolated held-out faint stars and on the crowded never-trained T10–13 stars. Code: worktree finenbr-orth. Record: dev_runs/finenbr_axis_20260925/README.md. Tabs sw_* below.

• No variant beats the plain prior at λ = 1e8 on held-out χ². Plain λ sweep: 3e7 ×1.055/×1.026, 1e8 ×1.00, 3e8 ×1.012/×1.101 (F1/F2). The earlier "1e8 is 2× too tight in scene mode" worry is not borne out for faint training.
• The width bias is a trade-off along λ. Axis-cell width: plain 3e7 −1.4/−1.05, 1e8 −2.2/−1.8, 3e8 −3.8/−3.6 (1e-3 px²). Moment-blind flattens it to about −1 CCD-wide (−1.3/−0.9 at 3e8), but costs ×1.08–1.12 held-out core χ².
• The separate width/shift smoothness prior never helps. Loose settings (k ≥ 0.1) are inert. k = 0.03 is worse (×1.18–1.19). Tight settings (k ≤ 0.01) break the fit (×2–9): real node-to-node width variation (~0.03 px²) is data-constrained, not what the plain prior was buying.
• The "Adam floor" was a misdiagnosis. Neither LR decay nor warmup fixes the known-truth drift. It sits on the scene's stop_gradient nuisance stars: with no prior, lr 1e-4 + decay recovers the truth on the trained stars (χ² 0.990 vs 0.994), while nuisance-star χ² is 1.64. In faint training the same effect makes a lower lr look better on T<10 stars (nuisance by design) by staying near C5.
• Lower lr (1e-5) does beat the converged fit on held-out faint stars (×0.95–0.97). The faint-only scene (~1.7k trainees) under-determines the fine structure, and the C5 warm start (bright-trained) carries information. That points at a prior toward the bright-trained ePSF's fine structure, not at neighbour smoothing.
• Photometric size: held-out flux differences between arms ≤ 0.5‰ by magnitude.

Quadratic-in-colour radial shift, 2026-09-25

New colour-model extras sq0,sq1 (worktree colour-quad, commit 599e12c): shift = [Δc·s(r) + (Δc² − ⟨Δc²⟩)·(sq0 + sq1·r)]·n̂, toward the optical axis. Both arms continue C5 for 1500 steps with C5's colour coefficients kept. Tabs F*_ctrl and F*_quad below. Record: dev_runs/colourquad_20260925/README.md.

• It halves the red-end shift on both fields. Leftover radial shift at BP-RP ≈ 2.1 falls from +18.8 to +9.1 mpx (F1) and from +17.2 to +8.5 mpx (F2), and from about 23 to 10 mpx beyond 1800 px from the axis. The blue end and the mid-colour dip flatten too. Fitted sq1 = −0.0062 (F1) and −0.0064 (F2).
• More training alone does nothing. The 8-term control reproduces C5 exactly.
• The red dipole in the grids barely changes. The red-minus-mid residual rms falls only 11–13%. What is left: about 8 mpx of shift (the red end rises faster than Δc²), a colour-dependent anisotropic width (quadrupole, including a 45° part, strong on F1), and a small round width. Colour coma explains nothing.
• Next: Δc² shift plus a colour anisotropic width term together, optionally on Gaia XP u, which orders the shift better than BP-RP (xp_dimension_20260925).
• Warm-start gotcha: without --chroma-g8-init, --chroma-model global8 resets the colour coefficients to 0, even with --init-params-file.

Option A tested: moment-blind fine prior, 2026-09-25

--fine-nbr-mode moment_blind (dev worktree finenbr-orth): before squaring each adjacent-node difference of the fine parts, project out the pair-mean flux and both nodes' own shift and width generators. Same recipe as faint10/faint11 (1500 steps from C5), λ = 1e8 and 3e8, eval on the fixed C5 scene. Tabs eval_F*_faint1*mb* below.

• The corner bias is real and not in-sample. Held-out stars show the same axis-cell sharpening as trained ones (plain prior −2.0 to −3.7).
• Known-truth check. Retraining from the truth with the plain prior broadens the core at every node (+0.1 to +4.4, all positive, worst at corners). With the moment-blind prior the node biases scatter ±0.5 around 0 (noise floor ±0.2). A pure width change leaks 57% through the σ = 2-sample split, not 32%.
• Real data: moment-blind removes the axis-corner pattern to the prior-off level: held-out axis cell F1 faint10 −2.35 → −1.24, F1 faint11 −3.53 → −0.91, F2 faint10 −1.99 → −0.89, F2 faint11 −3.70 → −1.39. F2 faint11 keeps a mild −1 CCD-wide.
• Held-out core χ² (star-matched median ratio vs plain prior): prior off +10 to +18%; moment-blind +1.4 to +8% overall. In the axis cell it is better on faint11 (F1 −13.5%, F2 −3%) and F2 faint10 at 3e8 (−2%), worse on F1 faint10 (+3 to +4%). λ = 3e8 is not uniformly better than 1e8.
• (Superseded by the held-out sweep above: the proposed width/shift prior does not recover the loss.) Conclusion: A fixes the width drag but loses some of the plain prior's noise suppression, because 63% of what the plain prior penalised was low-order shape that also carried real smoothing value. Next: A plus a weak node-smoothness term on the width/shift coefficients alone, and arm B (σ = 1) for comparison.

Fixing the corner: diagnosis and options, 2026-09-25

Two effects, measured on the faint10 fits (no retraining):

• Prior drag, both fields. Switching the prior off shifts the near-axis width residual by about +1 (at 150 px from the axis: F1 −4.2 → −2.1, F2 −2.4 → 0.0). 63% of what the current prior penalises is real shift/width variation between nodes. Meanwhile 97% of the junk it removes is not low-order shape, and that junk is broadband (0.1–2 cycles/px).
• Interpolation error, small. Near the axis, stars midway between nodes are sharper than stars at nodes. Excluding stars within 150–250 px of a CCD edge, where the edge nodes sit and stamps are truncated, leaves about 0.7 ± 0.3 on F1 and 0.3–0.6 ± 0.3 on F2 with the prior off. The uncorrected 1.4 ± 0.2 was partly that edge confound. In the 09-16 unit this is about 2e-4 of flux, below that study's 6×6-edge corner prediction of 4.6e-4: bilinear is doing as well as or better than predicted. (Corrected 09-25 evening.)

Validation. Synthetic first: inject a known sharpening near the axis into the retrain-from-truth scene, which gives a known answer. Then on real data, judge by the width map, radial profile, node-phase scallop, held-out χ² and flux bias vs magnitude and colour.

Axis-corner bullseye = fine-neighbour prior, 2026-09-25

In the cell nearest the optical axis, blue and mid-colour stars of every magnitude are sharper than the ePSF. That is why the pattern vanishes after subtracting the cell's all-star stack: it is a position effect shared by everyone there.

• With the fine-neighbour prior switched off (λ = 1e8 → 0), the width map flattens: axis cell −3.5 → −0.9 (F1 faint11) and −3.6 → −0.7 (F2 faint11). faint11, with the fewest training stars and so the most prior-dominated, is the worst when the prior is on.
• Why: the prior penalises node-to-node differences in the ePSF's "fine part" (the ePSF minus a σ = 0.5 px blur). 32% of a pure core-width change passes that filter, so the prior drags the one-sided corner node toward its broader neighbours.
• Removing the prior is not the fix. Held-out T10–13 median core χ² gets worse without it (F1 faint10 21.4 → 29.6, axis cell 27.2 → 40.0; F2 12.1 → 14.9), because the ePSF then picks up pixel-scale noise.
• Next test: keep the prior but make it blind to the low-order core shape: project the flux, shift and width derivatives out of the fine-part difference, or use a narrower split (σ = 1 sample).

Faint-only training, 2026-09-25

• Red stars dominate the residual. In every cell of both fields, the BP‑RP > 1.25 column carries a ±20e‑4 dipole. Mid-colour stars are close to clean, and blue stars show a weak residual with the opposite sense.
• The dipole is radial. Its excess flux points away from the optical axis in each cell: F1's axis is off the top-left corner (−56, 2099) and F2's is at the top-right (2103, 2107). The model places red stars too close to the axis, so the lateral-colour shift is under-corrected at the red end.
• Faint-only training does not change this. faint10 (trained on T10–13) looks almost the same as C5. faint11 (trained on T11–13) makes the near-axis T10–13 stars sharper than the model (cxx/cyy −3 to −5 in the top-left cell of F1).
• Confidence: the red-column pattern is coherent across 9 cells × 2 fields. The T8–10 bins have only 15–50 stars per cell and are noise-dominated, so read them as indicative only.
• Next step: test a colour term that is nonlinear in BP‑RP (or a separate red-end radial shift). The mid-colour column is where the colour gauge sits.

Residual grids

One tab per training run, newest last. Each new training run is added here.

Where it lives