# Neighbour-aware sky normalization: improvement with a remaining failure — 2026-10-04

**Conclusion: the correction fixes the original injection failure, but is
not accepted as a replacement.** Its worst amplitude error is 0.43669% for
the original source-consistent injection family, below the unchanged 0.5%
criterion. A separately fixed broader profile still fails at **0.64280%**.
The persistent real group histories remain, and some reference apertures
pay a substantial noise penalty. No star is removed to obtain these results.
The current nominal candidate and production data remain unchanged.

Run: `/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/`.
Code: `/home/kshukawa/syndiff_pipeline/dev/.worktrees/temporal-raw-epsf/`,
branch `codex/temporal-raw-epsf`, local only. Scripts:
`temporal_raw_epsf/neighbour_sky_*.py` and the extended
`source_consistent_prepare.py`. This follows
[SOURCE_CONSISTENT_SIGNALS_20261004.md](SOURCE_CONSISTENT_SIGNALS.md).
The [existing canvas](http://localhost:18888/temporal-raw-epsf/canvas/temporal_raw_epsf.html)
contains this milestone with the preceding evidence.

## Recipe fixed before viewing new light curves

Starting with the original sky ring, exclude pixels within 5 detector pixels
of catalogue neighbours whose estimated flux exceeds 0.1 times the target
flux. Keep every original target ID, primary r3/r5 aperture pixel and stellar
fit mask. The catalogue/WCS convention is the same as the previous masks;
the exact input checksum and reference FFI are recorded. No observed temporal
flatness chooses the radius or flux threshold.

The fixed hard exclusion supports 428 of 431 targets. Three retain only
10, 6 and 14 clean sky pixels, below the original minimum of 15. This hard
arm remains a failed support test; those stars are not dropped.

![Support and archived-noise audit](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/inputs/neighbour_sky/support.png)
![Actual RAW images, sky masks and aperture operators](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/inputs/neighbour_sky/images_and_weights.png)

A separate explicit weighted-support arm assigns

$$q_p=C_p+\lambda(S_p-C_p),\qquad
N_{\rm eff}=\frac{(\sum_p q_p)^2}{\sum_p q_p^2}.$$

Here $p$ indexes a pixel, $S_p$ is the original dimensionless binary sky-ring
mask, $C_p$ the new clean binary mask with $C_p\le S_p$, $q_p$ the dimensionless
sky-regression weight, and $\lambda$ the smallest excluded-pixel weight floor
that gives effective count $N_{\rm eff}\ge15$. The floor is computed solely
from geometry. The sky matrix must also retain condition number below $10^8$.
For the 428 supported hard masks, the floor is zero and their operators replay
exactly. All 216 unchanged operators also replay exactly.

The three nonzero floors are 0.08155, 0.25396 and 0.01375; their design-matrix
condition numbers are 20.54, 4.96 and 26.67. This intentionally retains some
potentially contaminated sky information in those targets. It is neither a
claim of clean sky nor a hidden restoration of the original operator.

![The three explicit weighted-support extensions](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/inputs/neighbour_sky_weighted/weighted_support.png)

## Remeasure the complete estimator

For each exposure, the sky-plane coefficients are

$$\widehat{\boldsymbol\beta}_t=
(\Phi_t^\mathsf T\Gamma\Phi_t)^{-1}
\Phi_t^\mathsf T\Gamma\mathbf d_t.$$

Here $t$ indexes an exposure; $\mathbf d_t$ is its calibrated pixel-rate vector
in electrons/second; $\Phi_t$ is the existing dimensionless response-aware
constant/x/y plane basis, evaluated on the original pixels; $\Gamma$ is the
diagonal matrix of the geometry-only weights $q_p$; and the three coefficients
$\widehat{\boldsymbol\beta}_t$ have units electrons/second. The fitted sky is
$\Phi_t\widehat{\boldsymbol\beta}_t$.

Both r3/r5 aperture/sky operators are recomputed, and the same stored physical
pixel count-loss images are projected through each matching operator. Source
profiles are relearned with the new sky and aperture normalization; source
flux/sky, science aperture-ratio transport and reference precision are refitted.
The ratio fits keep their original source-fold/two-day exclusions. Reference
precision still uses full histories and is not a time-held-out precision fit.

The full comparison retains all 431 targets, 172 science stars, 235 references,
1,224 FFIs, 165 eligible science stars and 354 audit groups. The ten-target
pilot includes the close pair, prior worst signal source, all three support
extensions and the worst predicted aperture-noise target. Its outputs replay
bitwise in the full run. Original nominal and original coupled-injection
fluxes replay exactly; all 110 unchanged science operators produce identical
old/new flux arrays.

## Two independently fixed injection profiles

The original analytic injection family is retained. Before examining the new
light curves, a second stress profile was fixed: core fraction 0.65 rather than
0.72, core width factors 1.08 in x and 0.95 in y, wing fraction 0.35, wing-width
factor 1.25 on both axes, and wing offset (−0.18,+0.14) pixels. It is not fitted
instrument truth. Source trajectories, periods/phases and the original 1-ppt
own-aperture injection convention remain fixed.

Each source is rendered once on the absolute detector grid and consistently
inserted into every relevant cutout. All 14 overlapping cutout pairs have
bitwise identical additions in both families. Conservative analytic tail
bounds are recomputed; five focused profile/geometry tests pass. The broader
family contains 20 cross-source links, with four science cutouts and sixteen
references receiving additional source light. No catalogue distance cut is
treated as complete physical support.

Expected signals are computed from the SAME injected pixels through each
new normalization and nominal physical flux. The experiment does not change
the photons to make a new operator pass. Global WCS, response, profile
predictors, photon means and physical count-loss expectations remain fixed;
new detector photons/readout noise are not simulated.

| Fixed diagnostic | Original sky | Neighbour-aware sky |
|---|---:|---:|
| Original-family worst amplitude error, % | 1.55152 | **0.43669, passes** |
| Broader-family worst amplitude error, % | 2.57761 | **0.64280, fails** |
| Original-family recovery RMS median/p95, ppm | 1.729/3.559 | 1.675/3.191 |
| Broader-family recovery RMS median/p95, ppm | 2.219/4.630 | 2.222/4.462 |
| Ten-star group six-hour RMS, ppt | 0.12332 | 0.12510 |
| Sixteen-star group six-hour RMS, ppt | 0.18360 | 0.18626 |
| Science six-hour RMS median/p95, ppt | 0.4880/2.1032 | 0.4851/2.2979 |

All amplitude criteria use every original science source. The remaining
broader-profile failure is still Gaia 1626038957153297152, not a newly excluded
star. Its original-family error falls 1.55152%→0.23121%, and its broader-family
error falls 2.57761%→0.64280%. The original-family worst error moves to another
source at 0.43669%; it is not the same source as the original worst case.

![Signal gates and actual persistent group histories](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/neighbour_sky_refit_full/calibration/gates_and_groups.png)
![Actual paired errors for both stars in the close pair](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/neighbour_sky_refit_full/evidence/paired_histories.png)
![Actual individual light curves underlying the group summaries](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/neighbour_sky_refit_full/calibration/individual_histories.png)
![Nominal pixel residuals on matched scales](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/neighbour_sky_refit_full/evidence/nominal_residual_pixels.png)

The real group histories persist. This tested correction is therefore not a
solution for the remaining shared trends; it does not rule out every possible
neighbour/background error. Nearly unchanged pixel residuals coexist with
substantially changed signal recovery, demonstrating again why image-fit
quality alone cannot validate the flux normalization.

## Precision and reference limits

![Observed precision and reference sensitivities](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/neighbour_sky_refit_full/calibration/precision_and_references.png)

Median adjacent-difference scatter changes 0.22428→0.22422 ppt. The historical
aperture-level reference-variable injection gives median/p95 imported RMS
86.67/119.05→85.98/120.06 ppm. The matched reference-half difference is
102.22/121.25→102.13/120.42 ppm. These reference-only injections are explicitly
aperture-level comparisons and do not close spatially consistent reference
source or detector-photon response.

Removing sky support increases conditional aperture noise, especially for some
faint references. The propagated archived pixel-error calculation is not a
pure Poisson budget and excludes full learned-profile/reference/WCS covariance.
Neither small adjacent scatter changes nor the passing first injection family
certifies noise dominance or absence of systematics.

## Execution, grids and next step

Frozen commands/exits: `condor/neighbour_sky_prepare`,
`neighbour_sky_regularize`, `source_consistent_broad_prepare`,
`neighbour_sky_refit_pilot`, `neighbour_sky_refit_full`,
`neighbour_sky_calibration`, `neighbour_sky_grids`, and
`neighbour_sky_evidence`. All jobs completed with exit 0. The hard-mask
scientific support gate remains failed despite its successful diagnostic
execution. Full refit job 2436.0 on schedd plscience3/execute plscience2
completed normally; all owned jobs are terminal.

All 24 standard absolute/relative grids and eight diagnostic figures were
inspected. Pilot grids have five eligible inputs, four in canonical range,
and no bin reaches MIN_N=5. Full grids have 165 eligible inputs, 159 in
canonical range. Fixed original stellar pixels and old-r5 display denominators
are used in all arms. Automatic per-image colour scales and cell-relative
subtraction remain explicit. The [existing shared gallery](http://localhost:18888/residual-grids/resgrid.html)
preserves all 133 prior entries and adds twelve labels (145 total). The
unavailable external Artifact mirror was not republished.

**Conclusion / next:** neighbour-aware normalization is a supported direction,
but this radius-based sky policy is incomplete and is not adopted. Next test
sky support that avoids the direction of bright neighbours without relying
on one assumed wing width, or explicitly model neighbour nuisance light in
the sky estimate. Audit support/conditioning/variance before fitting, retain
the weighted-support exceptions visibly, and use new independent-profile
controls. Do not increment a radius until this particular injection passes,
drop the failed star, or treat the unchanged group trends as solved. More sky
pixels or independent CCD-sky information may be needed for poorly supported
references; any enlarged stamp requires its own measured pixels and matching
physical correction, not zero extension of the existing prediction. The
calibration/WCS/full-population/photon-noise objective remains active and unmet.