# Effective applied calibration reconstruction — 2026-10-03

**Conclusion:** matched raw/calibrated pixels identify an effective row/column
transformation that reproduces held-out stellar aperture histories to
median/p95 0.221/0.501 ppm temporal disagreement on 36 FFIs. This is agreement
with the archive, not physical calibration accuracy or photon-limited light
curves. The inferred row shape is nearly fixed within each orbit. Freezing
that shape across the orbit boundary changes relative photometry by only
median/p95 0.724/5.370 ppm RMS in a conditional control, too small to explain
the larger persistent group variations. No replacement calibration is adopted.

Run: `/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/`.
Code: local branch `codex/temporal-raw-epsf` in
`/home/kshukawa/syndiff_pipeline/dev/.worktrees/temporal-raw-epsf`.
Scripts are under that worktree's `temporal_raw_epsf/`; outputs remain on
`/astro`. Nothing is pushed.

## Identify processing before interpreting its error

The previous collateral experiment compared alternatives to our AIC128
reconstruction, which was not known to be the exact archived calibration.
This step instead uses the same photons before and after archive processing.
It fits no temporal stellar signal and cannot establish detector truth from
archive reproduction alone.

For one output and exposure, solve

$$Z_{y,x}=\mathcal L(U_{y,x}-b_y)-s_x,
\qquad Z_{y,x}=C_{y,x}Q_{y,x}\Delta t/g.$$

Here $y,x$ are full-FFI science-row/within-output column coordinates; $U$ is
the raw coadd divided by `NREADOUT`, minus the candidate public static black
map, in ADU/readout; $\mathcal L$ is the declared nonlinear ADU response;
$b_y$ is an inferred row term in ADU/readout; $s_x$ is an inferred column term
in linearized ADU/readout; $C$ is the archived pixel rate in electrons/second;
$Q$ is the dimensionless candidate flat; $g$ is the output's header gain in
electrons/ADU; $\Delta t$ is integration seconds per retained readout; and
$Z$ is the archive expressed in linearized ADU/readout.

These are **effective terms conditional on the chosen black/flat references**.
Row/column components of an alternative static black map could trade with
them. The public 2018323 and 2018324 black arrays were checked and are identical.
The row mean is anchored to the overclock AIC estimate. In the linear limit
there is a row/column constant gauge; nonlinear response makes its physical
interpretation weakly identifiable. Explicit ±1-ADU anchor controls therefore
accompany the pilot. No absolute physical pedestal is inferred from the
arbitrary gauge convention.

The four-FFI pilot uses nominal `INT_TIME`. The 36-FFI extension uses

$$\Delta t=86400\,E/N.$$

Here $E$ is the calibrated header `EXPOSURE` in days and $N$ is `NREADOUT`
(720). This differs from nominal 1.98 seconds by about 0.445 ppm, almost
constant across this sector. The nonlinear response's declared 0.99
single-readout normalization remains fixed. This normalization refinement
is not an explanation of hundred-ppm temporal trends.

## Fixed support and solver checks

The inherited catalogue/background mask excludes all science stamps and
their guards. Fixed random 8×8 detector tiles, seed 914023, define two
complementary fitting/evaluation halves. Every row must be supported and the
bipartite row/column graph must be connected. Unsupported columns are flagged;
they are not silently inferred. All 165 eligible stellar apertures retain
complete support in every accepted fit. Bright stellar stamps never train
the row/column reconstruction.

Synthetic tests exposed a plain-checkerboard failure: it creates two
disconnected row/column gauges even when every row and column has pixels.
Connectivity checking and the connected tile split address that failure.
The joint linearized update projects out the row-mean gauge. Tests cover
known linear/nonlinear recovery, analytic derivatives, held-pixel invariance,
unsupported support, pedestal convention and a noisy/outlier case.

Adaptive-scale robust fits hit their iteration caps. Fixed-scale Huber and
an exact-column-minimization probe also failed to meet the unchanged
parameter tolerance in a literal-XML control. Those complete-pilot attempts
are not accepted results. Ordinary least squares is reported as a separate,
well-defined pixel-mapping objective, with the same physical parameters,
support, independent validation and 1e-8-ADU convergence tolerance. All
128 pilot and 288 sample fits converge within three iterations. Nothing is
made acceptable by dropping difficult sources or relaxing the tolerance.

The one fully completed robust FFI96 case supplies a limited loss sensitivity:
ordinary/Huber stellar aperture differences have median/p95 absolute values
0.00084/0.0280 ppm. This is not a statement that all robust cases converged.

## Response conventions: qualification of the older result

The earlier `legacy_xml` arm used the archived polynomial coefficients
directly. The pinned older TICA `SpocLinearity` implementation divides two
terms by two and three, whereas `SpocLinearity_Updated` uses a revised table
directly. The pilot therefore includes a separate divided-term convention
under the explicitly shared coadd/deadtime normalization. It is not a full
replay of every older pipeline convention.

Both tested older-table conventions leave substantial stellar-core deficits
and larger background residuals; the updated convention performs much better.
This supports the tested response function but does **not** uniquely identify
a reference-file version or rule out every possible interpretation of an
older table. The earlier broader wording in the calibration audit/report is
qualified accordingly. The existing physical simulations use the updated
response and are unchanged by this interpretive correction. Source:
[pinned TICA implementation](https://github.com/mmfausnaugh/tica/blob/1910591aead158c26ee1c696294a7676f6af9d9c/tica/LinearityModels.py).

![Matched-pixel residual maps under explicit response conventions](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/applied_calibration_ordinary/reconstruction_maps.png)

![Stellar pixels excluded from fitting distinguish the conventions](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/applied_calibration_ordinary/stellar_residual_stacks.png)

## Archive agreement and effective row structure

The 36-frame sample retains both independent pixel halves and all 165 eligible
science apertures. Absolute aperture disagreement is typically about 6 ppm;
there is a small static stellar-core mismatch on a zoomed scale. Removing
each source's constant offset gives these **processing-reproduction** metrics:

| Check | Median / 95th percentile [ppm] |
|---|---:|
| Temporal aperture disagreement, all 36 FFIs | 0.221 / 0.501 |
| Same statistic, 32 epochs beyond the pilot | 0.212 / 0.473 |
| Absolute difference between pixel-half predictions | 0.000932 / 0.0184 |

Per-epoch fits use that epoch's own background pixels. Thus “32 added epochs”
is a replication on additional observations with held-out pixels/stars, not
prediction of an unseen epoch's calibration. Matching two versions of the
same photons permits far closer agreement than independent photon noise.
The real astrophysical and instrumental variations are present in both.

![Actual individual disagreements and magnitude/colour/position distributions](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/applied_calibration_sample/assessment/archive_agreement.png)

![Held-out stellar residual images on a zoomed common scale](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/applied_calibration_sample/assessment/nominal_stellar_residuals.png)

The effective row shape is almost fixed within each orbit. Between-orbit
template differences have RMS 0.00438/0.00784/0.00246/0.00362 ADU/readout for
A/B/C/D. Typical within-orbit departures are orders of magnitude smaller;
bright-background epochs show larger small departures. The orbit boundary
coincides with the processing-version boundary, so neither a software bug
nor a physical detector change is established by that association.

![Inferred row-time structure, with separate between/within-orbit scales](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/applied_calibration_sample/assessment/row_structure_time.png)

![Inferred rows versus the earlier AIC reconstruction and between pixel halves](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/applied_calibration_ordinary/inferred_rows.png)

## Isolated row-shape counterfactual

Replace the inferred row shape with the first-orbit median shape, preserving
each exposure's mean pedestal. Separately replace it with its own-orbit
median to measure the small within-orbit component. Both independent pixel
halves supply the inferred shapes. The column term, flat, response, WCS and
profiles, physical count restoration and reference weights remain fixed.
Exact nonlinear pixel differences are propagated through the archived
science profile operator and the 235 disjoint r3 references. No unknown
epochs receive invented corrections.

| Counterfactual, pixel half 0 | Median/p95 source RMS [ppm] | Maximum [ppm] | Ensemble RMS [ppm] |
|---|---:|---:|---:|
| First-orbit row shape at every sampled epoch | 0.724 / 5.370 | 20.27 | 0.0937 |
| Each orbit's median shape within that orbit | 0.00088 / 0.00651 | 0.0256 | 0.00045 |

Half 1 gives 0.724/5.370 ppm for the first control. All 354 preexisting
groups are evaluated descriptively. The previously selected 10/16-star
groups change by only 0.197/0.0057 ppm RMS in that control. No new
instrumental-attribution significance or full-cadence acceptance claim is
made from these 36 samples.

![Conditional photometric effect of changing only the inferred row shape](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/applied_calibration_sample/row_counterfactual/row_impact.png)

![Underlying pixel differences behind that counterfactual](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/applied_calibration_sample/row_counterfactual/row_difference_stamps.png)

All published figures were inspected. This counterfactual can remove a needed
calibration term, so it is not a recommended repair. Its limited impact rules
out this isolated row-shape change as the dominant explanation of the much
larger group variations within the stated fixed-model calculation. It does
not clear gain drift, flat response, column terms, PSF/scene effects or
calibration covariance. The original photon-limited objective is not achieved.

## Reproducibility and remaining work

Frozen jobs on plscience3 include failed robust pilots 1415, 1428 and 1445,
failed same-objective probe 1467, accepted ordinary pilot 1490 (exit 0, 41 s),
sample configuration failure 1529, accepted sample 1541 (exit 0, 61 s),
assessment 1559 (exit 0, 16 s), and counterfactual 1581 (exit 0, 24 s).
Exact commands/source hashes are under `condor/`; failed histories remain
separate. A configuration-only duplicate-key error was fixed without changing
the sample's scientific settings. Main outputs are
`results/applied_calibration_ordinary/` and `results/applied_calibration_sample/`.

Next use this verified effective transformation as the reference for any
new calibration-component test, rather than the approximate AIC reconstruction.
Keep the detector-truth and processing-provenance questions separate. The
explicit hp_d missing-light/offset repair, correct PS1 removal footprints,
broader stellar/reference coverage, shared-pixel injections and the complete
noise budget remain required. No production pixels or science candidate are
replaced in this milestone.