# Repaired backgrounds through temporal response and full photometry — 2026-10-03

> **Illumination/centroid follow-up, 2026-10-03:** [ILLUMINATION_CENTROID.md](ILLUMINATION_CENTROID.md) adds nonlinear-response tests, fresh matched centroid transfer, latest-generation covariance and an independent two-minute coverage diagnostic. Absolute WCS and the full photon-limit goal remain open.

**Conclusion:** repaired background surfaces improve held-out sky prediction
and further reduce the excursions of two diagnostic stars. They do not remove
the persistent group trends. The full 1,224-exposure extraction and reference
calibration have been repeated on the same sources and pixels; no photon-limit
or instrumental-absence claim is justified. The overall goal remains active.

Run: `/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/`.
Worktree: `/home/kshukawa/syndiff_pipeline/dev/.worktrees/temporal-raw-epsf/`,
local branch `codex/temporal-raw-epsf`. Production and exported candidate v1
remain unchanged. This continues `HPD_SHADOW_CALIBRATION.md` and
`RESPONSE_PROFILE_RELEARN.md`; `RESPONSE_COVARIANCE.md` describes a new
conditional noise calculation for the preceding response-aware generation.

## Matched background campaign

All original 36 orbit-edge/middle epochs now have the component-anomaly,
positive-RMS, harmonic-fill background with the existing S24 C2K2 bad-column
exclusions. The four prior epochs are reused with explicit configuration
checks; the other 32 run in fresh processes. Every original full-frame
background replay is qualified before the new calculation. All harmonic
solves converge at relative tolerance 1e-10.

Each epoch repeats the same selected +1-ppt extended-source perturbation.
Its fixed-r3 recovery error spans −0.08921 to −0.06971 ppm of the original
RAW aperture rate. This remains one empirical core/wing source family with
fixed template/catalogue, not universal signal preservation. The preceding
report shows its actual injected pixels and background response. All source
responses, component images and solver records are retained per epoch under
`results/background_transfer_probe/positive_rms_badcols_fNNNN/`.

![All 36 paired recovery measurements](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_badcols_sample_validation/paired_response.png)

![All corresponding injected target stamps](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_badcols_sample_validation/injected_sources.png)

![All measured background-response target stamps](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_badcols_sample_validation/background_responses.png)

`inputs/temporal_strap_badcols/` uses exactly the original residual pixels,
312 usable columns, 365,062 native training pixels and 110,806 pad-6
training/evaluation pixels. No old usable pixel becomes invalid; no support
is removed or enlarged. Input hashes and surface-generation paths are saved.

## Repeat the original temporal model and holdouts

The model is unchanged:

$$R_{t,y,x}=B_{t,y,x}\,q_x(y)+c_{y,x}+\eta_{t,y,x}.$$

Here $t$ is exposure and $(y,x)$ are crop-local detector coordinates;
$R$ is actual RAW minus the frozen convolved template in electrons/second/pixel;
$B$ is the repaired pre-strap background surface in the same units;
$q_x(y)$ is a dimensionless, quadratic-in-row column response fitted from
training epochs; $c_{y,x}$ is a static additive residual in electrons/second/pixel
estimated only on observed sky pixels; and $\eta$ is the remaining residual.
The static term is never inferred beneath stellar pixels. Background/model
errors, template changes and astrophysical contamination can enter these
conditional components; $q$ is not automatically a stellar flat field.

The old/new comparisons use identical two-day holds with 0.25-day guards,
opposite-orbit training sets, support, model family and masks. Perturbing
held-out counts leaves training coefficients exactly unchanged. The minimum
illumination leverage is 0.10677, maximum scaled condition number 38.33, and
no fitted response row is nonpositive.

| Pad-6 model | Two-day held-out MSE, old → repaired | Opposite-orbit MSE, old → repaired |
|---|---:|---:|
| Surface only | 24.2157 → 26.1719 | 24.2157 → 26.1719 |
| Response only | 4.1847 → 4.6273 | 4.0723 → 4.4936 |
| Response + static light | 0.56458 → 0.38823 | 0.54821 → 0.36404 |

Units are (electrons/second/pixel)², averaged equally over epochs. The joint
model improves by about 31–34%; the simpler controls worsen. A repaired
interpolated surface alone is not the complete sky prediction. Pixel/epoch
correlations preclude treating 110,806 pixels as independent significance
trials. These are prediction errors, not a photon-noise floor.

![Matched held-out prediction errors](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/temporal_response_comparison/matched_prediction_scores.png)

![Actual withheld pixel errors and prediction changes](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/temporal_response_comparison/matched_prediction_images.png)

The residual images retain broad structure at the high-background epoch.
Prediction improves, but it is not exact. The response maps change smoothly
along detector rows and by column; both orbit-trained maps show similar changes.

![Actual old/new fitted response maps](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/temporal_response_comparison/response_maps.png)

## Full-cadence remeasurement

The opposite-orbit, pad-6 joint response is propagated into local sky bases,
r3/r5 aperture weights, temporal-profile learning and the physical count-loss
projection. The global background surface defines response training; the
stellar extraction still fits its own response-weighted local sky. All 172
science sources and 235 references are retained across 1,224 quality-zero
exposures. Original flux extraction replays to 3.80e-13 fraction. No source's
slow temporal flux is fitted as an instrumental template.

New development light curves:
`results/response_profile_badcols_full/reference_calibration/photometry.npz`.
The comparison below is against the preceding `response_profile_full`
generation, not the original uncorrected extraction. The fixed-precision
control retains the older legacy reference weights; full refitting uses the
new reference histories and transport.

| Six-hour RMS [ppt] | Previous response-aware result | Repaired-background response, refitted precision |
|---|---:|---:|
| Median source | 0.49476 | 0.48799 |
| Full-cohort median history | 0.07282 | 0.07426 |
| Preselected 10-star spatial group | 0.12264 | 0.12332 |
| Preselected 16-star neighbourhood | 0.18328 | 0.18360 |
| Gaia 1629810213318424704 | 0.45800 | 0.45849 |
| Gaia 1625600183294357504 | 0.40577 | 0.26605 |
| Gaia 1627184373392387072 | 0.89582 | 0.87430 |
| Gaia 1603178323705979648 | 0.69446 | 0.53982 |

![Actual same-source full-cadence histories](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_badcols_full/assessment/diagnostic_stars.png)

Two diagnostic stars improve further, while one still has conspicuous
repeated orbit-edge excursions. The group histories are essentially unchanged;
better sky prediction does not resolve their cause. Two conditional sign
associations remain: knn16_93 (family-adjusted p=0.004638) and knn16_104
(p=0.024652), on the original 354 groups. Shared calibration and asymmetric
astrophysics prevent interpreting these as pure instrumental significances.

![All-group and source comparison](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_badcols_full/assessment/lightcurve_assessment.png)

![Actual constituent histories underlying the persistent groups](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_badcols_full/assessment/constituent_histories.png)

## Pixel and signal validation

The following pixel figures compare the original plain-sky RAW profile with
the newly relearned profile, using the same old-r5 display normalization.
They are distinct from the preceding-generation light-curve comparison above.
Changes are predominantly column-shaped plus source-core normalization.
Small residual-image changes do not independently certify source fluxes.

![Measured normalized profile changes](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_badcols_full/evidence/profiles.png)

![Actual matched data-minus-model pixels](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_badcols_full/evidence/residuals.png)

Analytic +1-ppt primary signals with 5/10/20/30-day periods repeat profile
learning and reference transport/precision fitting. Worst amplitude error is
0.1748% before reference calibration and 0.4812% afterward. Median/p95
paired signal error is 1.692/3.375 ppm. Global response, WCS and photon
expectations remain fixed, so these are conditional tests of one independent
analytic profile family, not full physical signal closure.

![Known inputs, recovered signals and magnitude/colour checks](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_badcols_full/evidence/signal_validation.png)

Reference-variable injection still imports median/p95 86.67/119.05 ppm.
Independent reference halves differ by 92.87/110.21 ppm RMS. Their effects
include astrophysical variability and response/precision differences, and
must not be called independent detector noise.

![Actual reference histories, independent halves and injected leakage](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_badcols_full/reference_evidence/reference_sensitivity.png)

The standard colour×magnitude×3×3-CCD absolute and relative grids were
generated, inspected and added to the existing private gallery as
`raw_repaired_response_20261003_relearned`. All 118 prior entries are preserved.
Of 165 eligible sources, 159 lie in the canonical magnitude range; many bins
are sparse/empty. Relative grids subtract the cell-wide stack and do not
establish zero absolute residual. The external Artifact mirror was not republished.

![Standard absolute residual grid](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_badcols_full/residual_grids_checked/relearned/resgrid_raw_repaired_response_20261003_relearned.png)

![Standard relative residual grid](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_badcols_full/residual_grids_checked/relearned/resgrid_raw_repaired_response_20261003_relearned_rel.png)

## Reproducibility and next tests

Frozen commands/code are in `condor/background_badcols_sample_00`–`_07`,
`prepare_temporal_strap_badcols`, `temporal_strap_badcols_validation`,
`temporal_response_comparison`, `prepare_response_profile_badcols`, and
`response_profile_badcols_*`. Scheduler jobs 2220–2227 covered the background
extension; 2223 recovered with 16 GB after a memory hold, and 2226 was removed
before its frozen command completed locally during the collector outage.
Remaining computations ran locally on measured spare capacity, with process
IDs and terminal exit 0 in `execution.json`. The first grid renderer failed
on an argument-name collision after saving the raster; the corrected raster
arrays are exactly identical. The failed directory remains preserved.

Do not repeat the unchanged background-repair hypothesis as an explanation
for the surviving group trends. Next discriminate illumination-dependent
column response from static template light using the existing fixed sky
support and whole-orbit holdouts. The repeated large orbit-edge excursions
motivate this test, but do not identify its cause in advance. Also validate
fresh hp_d/RAW astrometry using the actual shadow products, improve
reference-variable transfer, and update conditional covariance for this
latest generation. Genuine photon/readout and shared calibration uncertainty,
broader/fainter/crowded populations, missing-source accounting and independent
fields remain required. Preserve true stellar variability throughout.

**Conclusion:** this is another measured partial correction and a useful
negative result for the persistent-group hypothesis. Better held-out sky
prediction and selected light-curve improvement do not discharge the full
instrument-free/photon-noise objective.