# Illumination response and RAW/hp_d centroid transfer — 2026-10-03

> Latest 2026-10-03: [CALIBRATION_PIXEL_BRIDGE.md](CALIBRATION_PIXEL_BRIDGE.md) adds current-generation channel/magnitude/colour histories and a completed same-pixel TP/FFI background attribution. Earlier generation statistics remain historical; no calibration-truth claim is established.

**Conclusion:** a quadratic illumination response modestly improves held-out
sky prediction but does not remove the major excursions in the conditional
photometry test. Separately, the combined background repair strongly reduces
the time-dependent hp_d centroid artifact at the previously identified
outlier. Neither result establishes absolute WCS accuracy or photon-limited
light curves. The original full goal remains active.

Run: `/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/`.
Code: `/home/kshukawa/syndiff_pipeline/dev/.worktrees/temporal-raw-epsf/`,
local branch `codex/temporal-raw-epsf`. Latest complete development photometry
remains `results/response_profile_badcols_full/reference_calibration/photometry.npz`.
No production image, WCS header or accepted light-curve product is replaced.

## Illumination-dependent response

Using the same repaired surfaces and fixed sky support, fit

$$R_{t,y,x}=B_{t,y,x}\left[q_{0,x}(y)+
 \frac{B_{t,y,x}-\bar B_x}{s_{B,x}}q_{1,x}(y)\right]+c_{y,x}+\eta_{t,y,x}.$$

Here $t$ is exposure; $(y,x)$ are crop-local detector pixels; $R$ is RAW minus
the frozen convolved template, in electrons/second/pixel; $B$ is the repaired
pre-strap surface in the same units; $\bar B_x$ and $s_{B,x}$ are the mean and
standard deviation of $B$ using only training epochs and training rows in
column $x$; $q_{0,x}$ and $q_{1,x}$ are dimensionless quadratic polynomials
in normalized detector row; $c$ is static additive residual light on observed
sky pixels only, in electrons/second/pixel; and $\eta$ is the remaining error.
No static light is filled beneath stars. Setting $q_1=0$ recovers the prior
model. Background/model errors can enter these terms; they are not uniquely
identified detector calibration coefficients.

`test_illumination_response_model.py` verifies known nonlinear response and
arbitrary static-light recovery, count-unit rescaling, degeneracy rejection,
linear-baseline identity, and invariance to perturbing held-out counts or
held-out background values. In actual validation, the linear control replays
the saved predictions and held-out count perturbations leave fits unchanged.

All 36 epochs, 312 columns, 110,806 evaluation pixels, pad-6 masks and the
original temporal folds are preserved. The maximum normalized-design
condition number is 84.24; evaluated responses remain positive. Some held-out
illumination lies outside its fold's training range (maximum column/fold
fraction 0.4321), so transfer is explicitly tested rather than assumed.

| Held-out scheme | Linear illumination MSE | Quadratic illumination MSE | Ratio |
|---|---:|---:|---:|
| Two-day block | 0.388226 | 0.367759 | 0.94728 |
| Opposite orbit | 0.364036 | 0.346882 | 0.95288 |

MSE units are (electrons/second/pixel)², averaged equally over epochs.
Correlated pixels are not independent significance trials.

![Held-out illumination-model scores](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/illumination_response_validation/heldout_scores.png)

![Actual held-out pixel residuals and prediction changes](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/illumination_response_validation/heldout_images.png)

The improvement is modest and does not remove the broad high-background
residual structure. This is evidence for a useful predictive term, not proof
of detector nonlinearity or the cause of the stellar excursions.

## Conditional photometry on the same 36 epochs

Opposite-orbit coefficients and each saved repaired surface evaluate the
response on all 431 auxiliary stamps. Local sky and source rates are refitted
on unchanged pixels. The current temporal-profile shapes remain fixed, but
each is renormalized to the new r5 aperture weights; physical pixel count loss
is reprojected through the new r3/r5 weights. Baseline source and reference
rates are replayed before comparison. Reference ratio/precision remain fixed.

These are sample-normalized single-exposure RMS values, **not** the six-hour
full-cadence values in the preceding report:

| Diagnostic | Current [ppt] | Conditional quadratic response [ppt] |
|---|---:|---:|
| Gaia 1629810213318424704 | 0.56883 | 0.56771 |
| Gaia 1625600183294357504 | 0.39906 | 0.39929 |
| Gaia 1627184373392387072 | 1.04832 | 1.03820 |
| Gaia 1603178323705979648 | 0.64658 | 0.66097 |
| Preselected 10-star group | 0.22461 | 0.22311 |
| Preselected 16-star group | 0.24430 | 0.24511 |

![Actual source histories on the predeclared sample](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/illumination_photometry_sample/source_histories.png)

![Actual persistent-group histories](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/illumination_photometry_sample/group_histories.png)

![Underlying source residuals and response changes](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/illumination_photometry_sample/pixel_evidence.png)

This particular conditional change does not explain the remaining large
excursions. It is not adopted. A full epoch-dependent-response implementation
would require relearning profiles with explicit per-epoch normalization:
mixing profiles normalized with different aperture weights does not preserve
unit response automatically. WCS, global response, photon means and full
reference calibration also need signal/covariance validation before promotion.
Do not repeat this unchanged pilot or interpret the small sky gain as a cure.

## Separate stellar-column sensitivity counterfactual

Correcting the sky response does not automatically correct stellar throughput
as a moving/broadening PSF illuminates the same column. A separate,
parameter-free pilot therefore tests $Q_\star=Q_{\rm sky}$ against the current
$Q_\star=1$, keeping the same raw-space sky fit. Its corrected aperture is

$$A_\star=\sum_{p\in\mathcal A}
 \frac{D_p-Q_{{\rm sky},p}B_p}{Q_{\star,p}}.$$

Here $\mathcal A$ is the existing fixed measured aperture; $p$ is detector
pixel; $D_p$ is actual RAW rate and $B_p$ the fitted local sky plane, both in
electrons/second/pixel; $Q_{\rm sky}$ is the independently opposite-orbit-trained
dimensionless response; $Q_\star$ is the explicitly hypothesized stellar
response; and $A_\star$ is the corrected source aperture rate in electrons/second.
The outer-ring sky fit remains in raw pixel space. Profile normalization and
physical count-loss projection change consistently with this aperture.
No response coefficient is fitted to a stellar light curve. Stellar and
scattered-light spectra/illumination geometry can differ, so equality of the
responses is a hypothesis, not a calibration fact.

![Actual profiles behind the stellar-throughput hypothesis](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/stellar_column_response_pilot/profile_evidence.png)

![Actual source histories under the parameter-free hypothesis](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/stellar_column_response_pilot/source_histories.png)

![Same persistent groups under the stellar-response hypothesis](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/stellar_column_response_pilot/group_histories.png)

The large excursions remain and the two group sample RMS values change only
0.22461→0.22548 and 0.24430→0.24316 ppt. This simple hypothesis is not a
solution and is not adopted. It does not exclude colour-dependent or finer
pixel response errors. Frozen profiles/reference transport limit the pilot;
no complete-chain signal or physical photon-mean validation is claimed.

## Fresh matched centroid measurements

`hpd_centroid_transfer.py` measures 172 sources at FFIs 1, 601, 698 and 1225
in actual RAW, archived hp_d and regenerated hp_d pixels. It retains the same
RAW empirical profile and measured mask in every reduction. Translation is
represented by exact pixel-integrated increments from the independent
RAW-trained latent reference:

$$P_{s,t}(\boldsymbol\delta)=P^0_{s,t}
 +\frac{L_s(\boldsymbol x_{s,t}+\boldsymbol\delta)}
 {\sum_p a_{s,t,p}L_s(\boldsymbol x_{s,t}+\boldsymbol\delta)_p}
 -\frac{L_s(\boldsymbol x_{s,t})}
 {\sum_p a_{s,t,p}L_s(\boldsymbol x_{s,t})_p}.$$

Here $s$ is source, $t$ exposure, $p$ stamp pixel; $P^0$ is the current
dimensionless normalized RAW profile; $L_s$ is the fixed pixel-integrated
latent reference for the source's spatial/colour/brightness covariates;
$\boldsymbol x$ is the inherited RAW position in stamp pixels;
$\boldsymbol\delta$ is the fitted two-dimensional displacement in pixels;
and $a$ is the same dimensionless r5 aperture operator used for normalization.
Source amplitude and the same response-weighted sky plane are profiled out
for each displacement. This does not train a new PSF or alter a WCS solution.

The measured transfer is the fitted hp_d displacement minus the fitted RAW
displacement for the same source/exposure. Its four-epoch mean is removed
only when reporting temporal scatter. All 2,064 fits converge without hitting
the ±0.2-pixel bounds. Exactly 164 of the 165 originally eligible stars have
complete nonfatal support across every arm/epoch; all sources remain in the
tables. No source is removed because the correction helps or hurts.

Noiseless +0.003/−0.002-pixel controls on eight sources/four epochs recover
their specified shifts to numerical precision within this same translation
family. This validates sign/scale, not the true optical derivative or absolute
Gaia accuracy. Opposite-source-fold robust quadratic spatial fits assess
whether the reduction difference is predictable across the CCD.

| Measurement | Archived hp_d − RAW | Repaired hp_d − RAW |
|---|---:|---:|
| FFI 698 median/p95 displacement [millipixel] | 1.004 / 4.126 | 0.998 / 3.630 |
| Four-snapshot temporal RMS, source median/p95 [millipixel] | 0.0664 / 0.4951 | 0.0440 / 0.1791 |
| Previously selected outlier, FFI 698 vector [millipixel] | (−31.474, −13.102) | (−1.979, −1.232) |
| Same outlier, temporal RMS [millipixel] | 11.4966 | 0.0944 |

A millipixel is 0.001 native TESS pixel. At the outlier the displacement
magnitude falls from 34.09 to 2.33 millipixels. Most other offsets are largely
static over these four snapshots. The tiny temporal differences are not
independent per-exposure astrometric error bars: the reductions share photons
and the same profile model.

![Measured CCD transfer and its temporal component](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/hpd_centroid_transfer/assessment/centroid_transfer.png)

![Actual centroid-fit residual images](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/hpd_centroid_transfer/assessment/centroid_residual_pixels.png)

The outlier's broad residual is greatly reduced. Another diagnostic star
retains substantial source/neighbor-shaped hp_d residuals after repair;
template-light accounting and PSF differences remain. The held-out spatial
field residuals remain about one millipixel at the median, so this is not a
validated replacement spatial WCS correction. Four epochs do not establish
full-sector or independent-field astrometric accuracy.

## Record and next actions

All jobs ran locally on measured spare capacity with frozen source snapshots
and recorded exit 0: `illumination_response_validation`,
`illumination_response_images` (display-only), `illumination_photometry_sample`,
`illumination_photometry_groups` (display-only), `hpd_centroid_transfer`,
`hpd_centroid_assessment`, and `stellar_column_response_pilot`. The residual renderer was updated to show
nearest-neighbour pixels and wrapped titles without changing fitted arrays.
Every figure in this record was inspected.

## Latest-generation covariance and independent-product inventory

The repaired-profile covariance has now been recomputed using its own 64
shared-pixel realizations, profile derivatives and reference refits. All three
local stages complete with exit 0 and replay their respective baselines.
The source/profile Jacobian's maximum checked central-difference discrepancy
is 1.87e-5 of noise RMS. All 172 science sources and 235 references are retained.

Median exposure-history RMS is 0.58734 ppt versus propagated conditional
0.19318 ppt. For six-hour histories it is 0.48271 versus 0.09361 ppt. The
preselected groups have measured/conditional RMS 0.12532/0.04590 and
0.18751/0.05200 ppt. These use orbit-aligned log-flux bins; their values are
not the earlier globally aligned fractional-flux statistic. Astrophysics is
retained. Archived uncertainty is not a pure Poisson budget and unknown
WCS/response/calibration errors are not estimated by this simulation.

![Latest estimator's actual pixels and uncertainty inputs](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_badcols_noise_assessment/pixel_inputs.png)

![Latest estimator's propagated covariance by timescale](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_badcols_noise_assessment/noise_timescales.png)

![Latest constituent histories and group covariance](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_badcols_noise_assessment/group_covariance.png)

The [official MIT two-minute target list](https://tess.mit.edu/public/target_lists/2m/all_targets_S024_v1.csv)
was downloaded and cross-matched locally. It contains 1,918 S24 C2K2 targets.
Using a unique nearest match within 2 arcsec, a separated second neighbour
and a 0.5-mag consistency gate gives 58/172 science and 56/235 reference
matches. These counts concern catalogue coverage, not usable product coverage.
Of the four existing diagnostic stars, only Gaia 1625600183294357504 matches:
TIC 198208038, separation 0.108 arcsec. Product headers still need identity,
coordinate, time/exposure and calibration checks before scientific comparison.

![Actual science-source overlap and matching diagnostics](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/short_cadence_inventory/coverage.png)

This creates an independent calibration/count-processing test path. The
[MAST bulk-product inventory](https://archive.stsci.edu/tess/bulk_downloads/bulk_downloads_ffi-tp-lc-dv.html)
provides archive URLs. Download scripts are parsed as data, never executed.
A detrended product is not assumed astrophysical truth: compare aperture,
background, crowding, quality, timing and onboard-count conventions first.
No private source list is uploaded for the catalogue match.

### First actual two-minute light-curve pilot

The official archive supplied the matched TIC 198208038 LC, DR35,
`spoc-4.0.36-20200520`. Its header coordinate is 0.363 arcsec from the Gaia
source. `NUM_FRM=60`, `NREADOUT=48`, `DEADC=0.792`: it must not be assumed to
be an untrimmed photon reference. Header crowding and aperture fractions are
model metadata, not independent calibration truth.

Exposure matching subtracts each LC sample's `TIMECORR` and uses the already
verified spacecraft-time FFI start/stop intervals. Geometric time coverage
and finite SAP cover all 1,224 FFIs. But only 758 have >99.5% finite,
quality-zero SAP/PDC coverage, so the planned 70%-of-FFIs coverage gate fails.
That failure is retained; the diagnostic rerun does not lower either gate.
It shows the available subset with missing six-hour bins blank, plus a
separately labelled all-flag SAP diagnostic. The main FFI cohort is unchanged.

The LC has 4,739 cadences flagged 4096. This is the target-specific scattered
light exclusion bit, distinct from the predicted FFI straylight flag, as
documented in [the TESS product description, quality-bit table](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/active-missions/tess/_documents/EXP-TESS-ARC-ICD-TM-0014-Rev-F.pdf)
and local `inputs/calibration_audit/tess_sector_24_drn35_v02.pdf` §2. Missing
PDC coverage includes the very orbit-edge/scattered-light intervals we need
to understand; it cannot validate them by omission.

![Measured time and quality coverage](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/short_cadence_pilot_compare/timing_coverage.png)

On the common subset's 78 populated six-hour bins, our reference-calibrated
curve has RMS 0.2762 ppt and PDC 0.2574 ppt, but their correlation is only
0.185 and their log-ratio RMS is 0.3288 ppt. Similar RMS therefore does not
mean matching astrophysical histories. SAP differs more strongly; aperture,
sky, common instrumental effects and PDC signal suppression are unresolved.

![Actual common-exposure light curves and differences](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/short_cadence_pilot_compare/matched_histories.png)

![All-flag SAP measurements, explicitly not accepted clean photometry](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/short_cadence_pilot_compare/all_quality_sap_diagnostic.png)

The all-flag SAP curve shows large repeated orbit-linked excursions compared
with our recorded-profile extraction. This identifies a reduction difference,
not which curve is true. The next discriminating comparison uses the actual
target pixels and identical detector apertures to separate calibration from
sky/aperture treatment. The single matching TP product is now downloaded in
`inputs/short_cadence_comparison/pilot_tp/`, with the same archive-manifest
provenance, matching identity, DR35 and readout metadata. Its pixel geometry
and count/calibration consistency are not yet tested. No pipeline curve is
adopted as truth.

**Conclusion:** the local background-induced centroid problem is now
measured and reduced, while the simple quadratic illumination hypothesis
does not resolve the large stellar histories in its controlled pilot.
Absolute WCS, reference variability, template/neighbour light, physical
noise and broader-population validation remain part of the full objective.