# Detector-column exclusions and the background offset — 2026-10-03

> **2026-10-03 follow-up:** [HPD_SHADOW_CALIBRATION.md](HPD_SHADOW_CALIBRATION.md) now traces the defective column into uncalibrated virtual-row counts and verifies selected actual Hotpants regenerations. The full temporal/light-curve goal remains open.

**Conclusion:** the historical background fit uses pixels in detector columns
already excluded by the stellar extraction. At the previously selected hp_d
offset outlier, excluding these columns largely removes the modeled background
curvature contribution in four shadow tests. This identifies a concrete
background-generation defect. It does not yet constitute a rebuilt Hotpants
product, complete hp_d repair or photon-limited light curves. Production masks
and images are unchanged; the full goal remains active.

Run: `/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/`.
Worktree: `/home/kshukawa/syndiff_pipeline/dev/.worktrees/temporal-raw-epsf/`,
local branch `codex/temporal-raw-epsf`. No push or merge.
This extends `HPD_GENERATION_BRIDGE.md`, `BACKGROUND_CANDIDATES.md` and the
separate `RESPONSE_PROFILE_RELEARN.md` photometry development.

## What entered the background fit

The existing stellar bad-column set, in crop-local coordinates, is
222, 424, 1023, 1024, 1032, 1033, 1376, 1386, 1505, 1967, 1968.
Its provenance is the preexisting `pilot.py` extraction, independently
recorded by `strap_sky_validation/support_audit/summary.json`. All those
pixels have zero support in the saved science masks and reference weights.
This list was not selected by the new background outcomes.

The background estimator instead accepts mask values zero or FAINT_CAT=32,
plus its configured exclusions. Of the known bad columns, eight still supply
9,170 fitted background pixels per tested frame. At FFI 698, crop column 1023
alone supplies 1,103 pixels, with median RAW-minus-template value -101.99
electrons/second on that support. This is full-FFI column 1067, next to the
source whose global-background intervention was largest.

Direct calibrated-RAW reads distinguish this from merely a template mismatch:
the whole-column median at crop 1023 changes from +10.29 electrons/second in
FFI 1 to -99.86 in FFI 698, while neighboring crop 1022 changes from +71.75
to +286.71. The convolved-template medians are about 2.6–2.7 electrons/second
there. The shadow test does not determine the electronics/calibration origin
of the column defect; it tests the inappropriate use of those data as sky.

## Controlled shadow change

Start with the previously tested component-anomaly/harmonic/positive-RMS
background candidate. Add the existing bad-column set to a process-local
`extra_exclude` array. No original mask, residual input, template, source
catalogue or production file is modified. The downstream strap-ratio routine
is otherwise unchanged; this is specifically a background-fit exclusion test.

Use the same target, Gaia 1629810213318424704, chosen earlier from the hp_d
generation comparison. Repeat at FFIs 1, 601, 698 and 1225. Keep every science
source and its measured-pixel support fixed. For a background map $B$, project
its effect through the original RAW profile-plus-plane source estimator:

$$\delta_{s,t}=-10^6\,\frac{\sum_p o_{s,t,p}B_{t,p}}{F_{s,t}}.$$

Here $s$ is source, $t$ exposure and $p$ its measured stamp pixel; $o$ is the
dimensionless fixed source-rate operator, including the archived numerical
ridge; $B$ is the tested background in electrons/second/pixel; $F$ is the same
original physically corrected source rate in electrons/second; and $\delta$
is the direct projected effect in ppm. These are **effects of subtracting a
modeled background**, not measured errors relative to known background truth,
not reference-calibrated light-curve scatter, and not a full hp_d kernel refit.

| FFI | Historical biharmonic [ppm] | Combined harmonic [ppm] | Same candidate + bad-column exclusion [ppm] |
|---|---:|---:|---:|
| 1 | -17887.39 | -1499.73 | +9.30 |
| 601 | -16167.02 | -1245.55 | +159.12 |
| 698 | -94331.28 | -12417.80 | +160.25 |
| 1225 | -15765.91 | -1272.64 | +108.77 |

![Same-source background projections under the controlled mask change](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_badcolumn_assessment/source_effect.png)

The images show the affected RAW column, its previous inclusion in the
background fit, its explicit exclusion, and the corresponding disappearance
of the large background-curvature feature beneath the source. The native
source pixels and estimator are unchanged. A plane is removed only to display
curvature; the flux calculation uses each original full background map.

![Actual RAW images, fitting masks and background curvature](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_badcolumn_assessment/mask_and_background_images.png)

At the high-background epoch, the harmonic candidate's direct effect changes
from -1.2418% to +0.0160% solely through the extra exclusions. The remaining
small projection is not certified zero or unbiased. Stellar wings, neighboring
light and other background/model errors remain possible.

## Validation and scope

Every shadow run preserves the original frozen-input hash checks and required
historical replay qualification. Paired +1-ppt extended-source probes are
recomputed within the same process. Their fixed-r3 target errors remain small:
-0.07248/-0.08894/-0.07174/-0.07056 ppm in frame order. These are deterministic
conditional responses using the established RAW-core/empirical-wing family,
not universal signal-preservation or noise-floor bounds. The full fit mask
change and all 172 direct source projections are saved, not only the target.

Jobs on plscience3: 2151 (FFI 698), 2156 (1), 2157 (601), 2158 (1225), and
assessment 2167. Exact commands/code/hashes are in the matching run `condor/`
directories. The new flag is `background_transfer_probe.py
--exclude-known-bad-columns`; each test also specifies the combined candidate
and strict harmonic tolerance. All complete with exit 0 and converged solves.
Both figures were inspected. Earlier candidate backgrounds reconstructed from
stored float32 differences retain their previously reported rounding limits;
that precision is negligible compared with this intervention but is not
physical background uncertainty.

## Remaining repair work

1. Resolve the detector-column policy in the producing pipeline and make
   exclusions consistent across background and downstream stages. Do not
   apply this S24 C2K2 list universally to other CCDs without provenance.
2. Rebuild selected background/Hotpants outputs in a separate run, with
   fixed-kernel and refitted-kernel controls. Adding a map difference to hp_d
   alone does not validate the complete producing chain or its astrometry.
3. Revisit the response/static-light model with corrected background surfaces:
   its existing response is conditional on the old surfaces. Keep source,
   reference, pixel and time support matched and retain signal/noise controls.
4. Preserve the full goal: remaining RAW neighbourhood trends, updated noise
   and calibration covariance, broader/fainter/crowded sources and independent
   fields remain unresolved.

**Conclusion:** unexcluded detector-column data causally generate much of
this modeled background-offset feature. The shadow exclusion is a concrete
repair candidate; the complete hp_d/photometry repair is not yet verified.