# Matched ePSF training with faint neighbours included

2026-10-02. Owner: thread 01a0fd67-a4ba-7f21-ae2b-2e1914917556. Status: COMPLETE — bounded training pilot; magnitude-shape closure inconclusive.

Question: does including faint neighbours from the first iteration change the learned ePSF toward an independently trained RAW-faint reference, and improve held-out magnitude-dependent residuals?

## Important correction

The initial A/B runs, A_polish/B_polish and A_converged/B_converged, are **superseded and not science results**. Image inspection revealed that original `scene.tess_flux` values use a different scale from added sources. The inherited prior field is not a calibrated e-/s field. The initial diagnostic denominator also used it. The corrected runs use the same explicit nominal conversion for every source: 15,000 e-/s at T=10, scaled by 10 raised to −0.4 times (T−10). This is a nominal TESS magnitude calibration, not newly measured PS1 photometry. Old files remain for provenance. Do not quote their scores as the final result.

## Controlled experiment

C4 (S23 C4 K2), FFI tess2020087042920-s0023-4-2, crop-local x=[235,440), y=[1710,1940). Training y<1850; validation y>=1870; inactive 20-pixel guard. Original role assignments preserved. A has 252 sources, B adds 75 T13–18 nuisance sources. There are 17 training contributors and four validation contributor footprints before requiring core coverage. The actual useful bright validation core count is much smaller; population diagnostics are mandatory.

Both arms use identical observed pixels/noise, valid masks, initial RAW-faint ePSF, WCS, shape regularization and stopping rules. Train 18,903 unique pixels. Validation 6,666 unique pixels after excluding all possible RAW-faint calibration footprints. All source supports are disjoint between training and validation. No adaptive clipping. One common local ePSF is learned over this region, not separate magnitude/spatial models. WCS is frozen to the existing C4 solution (thus spatial validation is conditional on that existing WCS).

Faint/anchor templates are frozen to the RAW-faint reference; contributor templates change with the learned ePSF. All source fluxes are solved jointly every iteration. Gaussian nuisance prior strength is prior_kappa=0.1, relative to each template's inverse-variance information; it is not a 0.1-mag photometric uncertainty. Signed fluxes are allowed. Laplacian/pixel penalties 0.01, fine-neighbour penalty zero, local-polynomial penalty 3e8. No new background freedom.

The 75 added sources have finite, zero-valued 51×51 PS1 support at every recorded contributing occurrence in the published arrays. This does not certify infinite PSF wings, catalogue completeness, or every possible unrecorded cell. Existing Gaia/Tmag catalogues are used; PS1 photometry is pending another agent's work.

## Code and commands

Frozen producer package: `code/pin/syndiff_pipeline`, copied from `/home/kshukawa/syndiff_pipeline/.claude/worktrees/paper-pin-chain`. Copied peer scene helpers remain in `code/`. Owned scripts are backed up in dev branch `codex/ps1-stage-trace`. No production artifacts or other agents' files were changed.

Activate syndiff, set OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1. Run `code/prepare_training.py` for input construction (existing scene B is deliberately preserved). Corrected training uses TRAIN_TAG=_fluxfix with `python code/train_pair.py --arm A` and B. Each runs 1200 Adam steps, then `code/polish_pair.py --arm A` / B with POLISH_TAG=_fluxfix_final, POLISH_INIT_SUFFIX=_fluxfix, POLISH_STEPS=2200. The same optimizer tolerance, not equal convergence iteration counts, controls the final stop. Validation never selects the checkpoint.

Input checks and hashes: `inputs/preparation.json`, `inputs/removal_support_checks.csv`. Prior correction is in the training adapter, not a silent rewrite of the saved source scene. `code/evaluate_pair.py` uses PAIR_TAG to evaluate both checkpoints on exactly the same augmented B scene. EVAL_ALL_TARGETS=1 applies the candidate ePSF to original anchors as a separate transfer diagnostic. EVAL_SPLIT=train produces training-scene diagnostics. `code/shape_diagnostics.py` provides reference comparison and core-covered magnitude stacks.

## Conclusion

Actual corrected A/B training and common-scene validation completed. Shape RMS distance to RAW-faint reference B/A=1.0369. Training-family held-out score change=-0.3278%; transfer score change=-1.1754%. Only one bright core has ≥80% radius-2 coverage. This pilot does not establish removal of magnitude-dependent ePSF differences. See [complete evidence canvas](index.html), including underlying images, optimization, intervals and limitations. The next decisive experiment needs a larger intact-core sample and calibrated neighbour photometry.


## Final measurements and evidence

| Measurement | Result | Supporting images |
|---|---:|---|
| Learned pointwise shape difference | 6.05% of A peak maximum | figs/learned_epsfs_fluxfix_final.png |
| RMS difference from RAW-faint reference, B/A | 1.0369 (not closer) | figs/faint_reference_comparison.png |
| Primary held-out residual-score change | −0.328%; 95% block interval [−1.624%, +0.299%] | figs/validation_stamps_fluxfix_final.png |
| All-target transfer score change | −1.175%; conditional 95% block interval [−4.800%, −0.105%] | figs/magnitude_stacks.png; validation/*_alltargets/resgrid* |
| Usable radius-2 cores (≥80%), T bins 8–10 / 10–12 / 12–13 | 1 / 9 / 20 | figs/magnitude_stacks.png |

Absolute mean squared normalized residuals remain very large (primary A=537.90, B=536.14), so neither model is an adequate absolute fit. These are comparative diagnostics. Eight added validation source amplitudes are negative in the primary evaluation; signed nuisance flux freedom must not be read as clean physical photometry.

Both optimizers stopped on relative objective tolerance, not the gradient threshold. A gradient-infinity norm=0.0743, B=0.00802. Maximum pointwise shape changes after final-stage iteration 1000 were 0.441% and 0.306% of peak; see validation/stability.json. Initial parameters match bitwise and WCS remains fixed; inputs/initialization_audit.json. Fifteen of the seventeen training contributor footprints have ≥80% radius-2 core coverage.

Standard colour × magnitude × spatial grids were generated, visually inspected, and added to http://localhost:18888/residual-grids/resgrid.html for both shapes on common training and validation scenes and for the all-target transfer check. The external Claude Artifact mirror was not republished because its tool is unavailable. All final report images were inspected.

Report: http://localhost:18888/neighbour-epsf-pilot/ . Prior subtraction study preserved under previous-study/. Code, scene checks, checkpoints, logs and full raw diagnostics remain in this directory. No GPU billing or production changes.