# Covariance after response-aware profile learning — 2026-10-03

> **Illumination/centroid follow-up, 2026-10-03:** [ILLUMINATION_CENTROID.md](ILLUMINATION_CENTROID.md) adds nonlinear-response tests, fresh matched centroid transfer, latest-generation covariance and an independent two-minute coverage diagnostic. Absolute WCS and the full photon-limit goal remain open.

**Conclusion:** propagating recorded pixel uncertainty through the changed
profiles, apertures and reference calibration does not explain the remaining
slow group histories. This is an updated conditional covariance calculation,
not a pure photon-noise budget or proof that the excess is instrumental.
Real stellar variability and unmodeled instrumental effects both remain.
No light curve is detrended or changed by this test; the full goal stays active.

Run: `/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/`.
Code: local branch `codex/temporal-raw-epsf` in
`/home/kshukawa/syndiff_pipeline/dev/.worktrees/temporal-raw-epsf/`.
Inputs are the completed `inputs/response_profile/` and
`results/response_profile_full/` generation. This calculation precedes the
repaired-background response refit and does not certify that later estimator.

## What is propagated

For source $s$, exposure $t$ and measured detector pixel $p$, draw

$$\epsilon_{r,t,p}=\sigma_{t,p}z_{r,t,p}.$$

Here $r$ indexes the 64 Monte Carlo realizations; $\sigma_{t,p}$ is the
archived calibrated-FFI uncertainty in electrons/second/pixel; and
$z_{r,t,p}$ is a unit normal variate generated by a Philox stream seeded by
the absolute detector coordinates. The same physical pixel has the same
variate in every overlapping stellar stamp. Distinct pixels and exposures
are assumed independent at this input stage. This assumption excludes
upstream shared calibration covariance. The science masks contain 55 shared
pixels; all 431 auxiliary masks contain 280 shared pixels, maximum multiplicity 2.

The learned profile uses the response-weighted outer sky operator and r5
normalization. Its dependence on training exposures is propagated as

$$\delta P_t=\sum_u H_{tu}\left[
 \frac{(I-B_uK^{\rm sky}_u)\epsilon_u}{A_u}
 -\frac{Y_u\,(a_u^\mathsf{T}\epsilon_u)}{A_u}\right],
\qquad
\delta F_t=k_t^\mathsf{T}\epsilon_t+
       \left(\frac{\partial F_t}{\partial P_t}\right)^\mathsf{T}\delta P_t.$$

Here $P_t$ is the dimensionless normalized pixel profile; $u$ indexes training
exposures; $H_{tu}$ is the original fixed time/state profile predictor with
zero diagonal and no cross-orbit weights; $I$ is the pixel identity operator;
$B_u$ contains the three response-weighted sky-plane basis images;
$K^{\rm sky}_u$ maps outer-ring counts to their sky coefficients;
$A_u=a_u^\mathsf{T}D_u$ is the measured r5 aperture rate in electrons/second;
$a_u$ is its dimensionless pixel-weight vector; $D_u$ is the observed pixel
rate vector; $Y_u=(I-B_uK^{\rm sky}_u)D_u/A_u$ is the normalized source image;
$F_t$ is the fitted source rate; and $k_t$ is the direct least-squares
source-rate operator. All pixel/rate units cancel appropriately in $\delta P$;
$\delta F$ is in electrons/second. The profile derivative follows the
weighted source-plus-sky normal equations, including the residual term.

An eight-source pilot compares this Jacobian with central nonlinear refits
at 1 and 0.1 times the recorded noise. Maximum mismatch is 0.000104 of noise
RMS. The full 172-science/431-auxiliary calculation repeats selected checks;
flux replay differs by at most 1.54e-10 fraction. The first 32 draws for every
pilot source are exactly identical in the full run, including aperture noise.

Reference calibration is then refitted for each realization. The r5/r3
transport retains its source-fold/two-day exclusions, and reference precision
is recomputed from the perturbed reference histories. The unperturbed
prediction and final source flux replay the saved generation. A four-draw
pilot and the first four draws of the 64-draw run are exactly identical.
All saved samples are finite and have the expected shape 64×4×1224×172.

The controls successively include science pixel/profile noise, reference
aperture noise, refitted ratio transport, and refitted reference precision.
Physical count-loss images/means, WCS, global response, explanatory features
and source eligibility remain fixed. Source perturbations use the validated
first-order profile Jacobian; the reference stage is refitted nonlinearly.
Genuine reference-star variability is not resampled as detector noise.

## Measured results

![Underlying actual pixels, recorded errors and learned profiles](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_noise_assessment/pixel_inputs.png)

The 165 eligible science histories are not detrended. Six-hour and one-day
bins start separately within each orbit. Medians are evaluated on the
observed histories plus each perturbation; taking a median of noise alone
would propagate a different statistic. First/second differences never span
the orbit gap. Log-relative flux is used here, so the exact bin values differ
slightly from the earlier globally aligned fractional-flux audit.

| Statistic | Median measured RMS [ppt] | Median propagated conditional RMS [ppt] |
|---|---:|---:|
| Exposure histories | 0.58984 | 0.19317 |
| Exposure first differences | 0.31800 | 0.27431 |
| Exposure second differences | 0.53491 | 0.47892 |
| Six-hour histories | 0.48900 | 0.09346 |
| One-day histories | 0.40333 | 0.06255 |

The median of per-star observed/noise ratios is 1.103 for exposure first
differences and 1.065 for second differences; these differ from dividing the
two population medians above. Fast differences can contain astrophysics and
do not prove whiteness or a photon limit. Monte Carlo standard errors of
individual expected RMS values are saved in `source_scores.csv`.

![Observed histories and propagated conditional covariance by timescale](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_noise_assessment/noise_timescales.png)

All 354 original overlapping groups are retained. For the preselected
10-star spatial group, measured six-hour median-history RMS is 0.12592 ppt;
propagated RMS is 0.04626±0.00045 ppt (Monte Carlo SE). For the 16-star
neighbourhood it is 0.18717 ppt versus 0.05208±0.00046 ppt. Reference-count
and refitting contributions change these estimates only modestly compared
with science-only values 0.04526 and 0.05133 ppt.

![Actual constituent histories and the same group statistic's uncertainty](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_noise_assessment/group_covariance.png)

This conditional pixel-noise calculation cannot explain the slow histories
on its own. That statement does not assign the excess to the instrument:
astrophysical target/reference variability and uncertain detector response
are separate terms. No variance term was tuned to make the curves pass.

## Execution record and limitations

Scripts: `response_profile_noise.py`, `response_reference_noise.py`,
`response_noise_assessment.py`. Pixel/profile pilot and full jobs 2232 and
2234 completed on plscience3's scheduler. Reference pilot 2237 had not started
when the collector became unreachable; it was explicitly removed, its abort
verified, and its frozen command executed locally with exit 0. The full
reference propagation and assessment also ran locally on measured spare
capacity, preserving snapshots under `condor/`.

The initial local-launch metadata writer failed after spawning those latter
two scientific children because `host` was supplied twice. The children
continued, completed their internal checks and wrote final products and DONE;
their exit codes were not captured by that failed wrapper. The bug is fixed
for subsequent launches. Completed numerical arrays were independently
validated, including exact pilot/full overlap, rather than blindly rerunning
the same jobs. This is a provenance limitation, not an ignored science failure.

Products: `results/response_profile_noise_full/`,
`results/response_reference_noise_full/`, and `results/response_noise_assessment/`.
All three figures were inspected. Remaining budget components include true
detected-photon/readout statistics, shared upstream calibration errors,
WCS/response estimation, physical count-loss uncertainty and reference
astrophysical leakage. They must be identified physically, not bundled into
an inflated independent noise floor.

**Conclusion:** the changed estimator now has a validated conditional
pixel/profile/reference covariance calculation. It neither eliminates the
remaining slow structure nor proves photon-noise dominance. The corrected
background response, independent sky/astrometry tests and broader source
population remain necessary.