# Independent collateral row-bias test — 2026-10-03

**Conclusion:** a shared row template improves prediction of unused trailing
calibration pixels, but the leading/trailing mismatch—especially output B—
limits transfer to illuminated pixels. The tested smoothed-bias alternative
changes final relative fluxes by median/p95 3.08/18.85 ppm on 36 FFIs. It
does not remove the larger preselected group variations. These are conditional
differences from a controlled reconstruction, not measured errors in the
archived calibration. No replacement CAL or science product is promoted.

Run: `/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/`.
Worktree: `/home/kshukawa/syndiff_pipeline/dev/.worktrees/temporal-raw-epsf`,
branch `codex/temporal-raw-epsf`; code remains local.

## Independent pixel tests

The same 36 preselected raw FFIs provide eleven leading and eleven trailing
serial-register pixels for each output. Two fixed selections are used:
central five columns, and eight readout-order trailing columns after omitting
the first three. B/D ordering is reversed as required by the instrument
readout convention. The leading strip is physically disjoint from the
trailing strip; shared electronic noise can still correlate them.

The data convention matches the earlier calibration reconstruction:

$$u_{t,a,y,k}=I_{t,a,y,k}/N_t-B_{a,y,k}.$$

Here $t$ denotes an exposure, $a$ output A/B/C/D, $y$ a full-FFI row and $k$
a collateral column; $I$ is the uncalibrated coadd in ADU, $N_t$ is header
`NREADOUT` (720 retained readouts), $B$ is the public static two-dimensional
black map in ADU/readout, and $u$ is the residual in ADU/readout. No stellar
flux trains or selects a bias model.

Alternating column halves train and evaluate one another. Static strip/column
differences and row bases are estimated only from epochs outside the evaluated
two-day block plus a 0.25-day guard. The evaluated exposure's training columns
fit its amplitudes, as normal calibration uses same-exposure collateral data.
Its held columns and leading-strip values do not train the prediction.

The learned-row family is

$$\widehat b_{t,a}(y)=\mu_a(y)+c_{t,a}
+\sum_{j=1}^{K}q_{t,a,j}v_{a,j}(y).$$

Here $\widehat b$ is the row-bias prediction in ADU/readout, $\mu$ the
training-epoch mean row template in the same units, $c$ the current exposure's
mean residual pedestal in ADU/readout, $v_j$ dimensionless orthonormal row
vectors learned by singular-value decomposition after removing per-exposure
constants, and $q_j$ their current-exposure amplitudes in ADU/readout. Ranks
$K=0,1,2,4,8$ are predeclared; no rank is selected using science flatness.

Other controls are the noisy per-row mean, training-template plus Gaussian
smoothing of the current residual at 8/32-row sigma, and a per-exposure
Legendre polynomial with AIC-selected degree 0–128. The cached polynomial
calculation reproduces the earlier implementation on actual frames. Known
row-mode recovery, constant-pedestal invariance and guarded time exclusions
pass independent tests. The AIC cap is reached in 188/864 partial/full-column
fits, so polynomial convergence is not established.

Validation reports full and row-centred error separately for rows 0–749,
750–2047 and smear rows 2060–2067. Plot labels `top_rows`/`lower_rows` mean
these numerical row ranges, not an independent optical-axis classification.
All models remain in the record. Paired bootstrap intervals resample whole
two-day blocks with equal block weight; they are not family-adjusted model-
selection confidence intervals. The plotted MSE ratios weight sampled
exposures equally, so their weighting differs from the interval's estimand.

![Held-out pixel prediction scores](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/collateral_bias/validation_comparison.png)

For central columns and rows 750–2047, template-plus-eight-row smoothing has
held-trailing MSE 0.837 times the AIC128 control. Its independent-leading MSE
ratio is 0.986. Simpler row models perform better on the leading strip than
the higher-rank alternatives. A noisy per-row estimate worsens prediction.

![Underlying measured row histories and prediction residuals](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/collateral_bias/collateral_images.png)

![Leading-strip offset versus row-shape discrepancy](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/collateral_bias/assessment/leading_transfer.png)

For the displayed central-column half-0, rank-2 example over rows 750–2047,
the temporal RMS of the leading-minus-predicted row mean is approximately
0.017, 0.295, 0.008 and 0.036 ADU/readout for A/B/C/D. This is a discrepancy
between collateral measurements, not a direct measurement of science-pixel
bias or detector gain. The images show that B's large offset extends across
many rows. Removing a row mean for a separate shape statistic does not remove
that discrepancy from the data or make it a validated science correction.

## Propagation through actual calibration and photometry

Every predeclared alternative is compared with the same selector's full-
column AIC128 reconstruction. It passes through the updated response table,
gain, mean-eight-row smear estimate and candidate public flat:

$$E_{t,p}=\frac{g_{a(p)}}{\Delta t}\,
\mathcal L_{a(p)}[U_{t,p}-\widehat b_{t,a(p)}(y_p)],
\qquad C_{t,p}=\frac{E_{t,p}-S_{t,x_p}}{Q_p}.$$

Here $p$ is a science or virtual pixel with row $y_p$ and column $x_p$;
$U$ is its raw-per-readout value after static black subtraction, in ADU/readout;
$g_a$ is the header gain in electrons/ADU; $\Delta t=1.98$ seconds is the
header integration time; $\mathcal L_a$ is the fixed updated nonlinear ADU
response used in the earlier reconstruction; $E$ is the resulting electron
rate; $S$ is its mean over virtual rows 2060–2067 for that column, in
electrons/second; $Q_p$ is the dimensionless candidate flat; and $C$ is the
calibrated pixel rate in electrons/second. The unknown original SPOC bias
curve is not asserted to equal this reconstruction.

A channel-constant bias cancels through the linear-response/smear limit,
including with a nonuniform flat. That identity and preservation of an added
science-pixel signal in the linear limit are tested. Nonlinearity and spatial
row structure prevent assuming cancellation in the actual calculation.

Pixel differences are projected through both fixed-r5 aperture/sky weights
and the actual archived temporal-profile linear flux operator. The latter
reproduces the saved RAW extraction within 0.0368 ppm. All 172 science sources
and the fixed 165 eligibility mask are retained. Independent auxiliary r3
counts reproduce exactly, and their overlapping science-r5 increments agree
exactly with the first propagation.

![Science magnitude, colour and measurement-operator sensitivity](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/collateral_bias/photometry/stellar_impact.png)

![Actual science-pixel calibration differences](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/collateral_bias/photometry/calibration_difference_stamps.png)

The reference calculation applies the same alternatives to all 431 sources,
including the fixed 235 disjoint references. Existing reference weights,
aperture transport, WCS/shape and physical count restoration remain fixed.
Corrections are calculated only at the 36 observed raw epochs; unsampled
epochs are not assigned zero or interpolated changes to refit full-history
weights. Source-centred RMS removes only a static relative-flux zero point.

| Central-column alternative | Profile-only change RMS median/p95 [ppm] | After fixed reference ensemble, median/p95 [ppm] |
|---|---:|---:|
| Template + eight-row smoothing | 2.70 / 18.64 | 3.08 / 18.85 |
| Template + pedestal only (rank 0) | 2.81 / 24.13 | 3.41 / 23.14 |
| Template + two row modes | 7.52 / 39.17 | 8.83 / 39.09 |
| Template + eight row modes | 9.34 / 45.87 | 10.36 / 43.07 |

These changes are not residual-error estimates. Some alternative models can
add calibration noise. In particular, refitting templates/bases by excluded
time block changes their finite-sample noise; a final shared template would
need its own fit and validation. The maximum post-reference change for the
smoothed control is 68.25 ppm; its ensemble change is only 0.67 ppm RMS.

![Consistent science/reference sensitivity and preselected groups](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/collateral_bias/references/reference_impact.png)

![Reference-pixel differences supporting that response](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/collateral_bias/references/reference_difference_stamps.png)

![Detector-position dependence on a common scale](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/collateral_bias/assessment/detector_impact.png)

All 354 existing magnitude, colour, spatial, neighbour and readout groups
receive a conditional response calculation. No new instrumental-attribution
test or full-cadence acceptance claim is made from 36 epochs. For the smoothed
control, the two preselected 10/16-star groups change by only 1.24/0.69 ppm
RMS, while their sampled group RMS remains about 0.224/0.237 ppt. These are
sampled-epoch scatters, not the previously reported six-hour full-cadence RMS.

![Actual individual and group histories remain variable](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/collateral_bias/assessment/actual_group_histories.png)

All figures were inspected. The calibration differences visibly have row
structure, rather than a new stellar PSF component. Masked pixels remain
missing. The group histories do not become instrument-free through this
conditional correction.

## Provenance and next action

Frozen jobs on plscience3: collateral validation 1258 (four CPUs/4,000 MB,
exit 0, 48 s), science propagation 1274 (two CPUs/4,000 MB, exit 0, 38 s),
reference propagation 1298 (same request, exit 0, 37 s). Assessment and display
jobs are 1314, 1316 and 1322; they use saved measurements and do not repeat
fits. Code/commands are frozen under the corresponding `condor/` directories.
Numerical outputs live under `results/collateral_bias/` on `/astro`.

The leading/trailing transfer issue needs a physical explanation before a
CAL replacement. This test neither establishes that row bias is the dominant
remaining cause nor clears other calibration components. Next infer the
row/column corrections actually applied to matched raw/calibrated pixels,
using the fixed response and flat model and holding science pixels out of
that reconstruction check. Relative row structure is identifiable separately
from the constant row/column gauge; absolute pedestal/nonlinearity conventions
need explicit constraints. Agreement with the archive establishes provenance,
not detector truth, but would remove the arbitrary AIC128 baseline from a
subsequent collateral-validated comparison.

The hp_d missing-light/offset repair remains a separate explicit requirement:
trace actual PS1 removal footprints and retained-light products before using
the earlier population-average correction. Full-cadence and wider stellar
coverage, complete injections and independent noise accounting also remain
required for the original objective.