# Strict K=5 conditional cross-validation

Status: bounded campaign complete, 2026-10-04. Owner: crossfit, model closure campaign. See final outcome below and the browser index. Earlier sections preserve chronological provenance.

Question: does full A3 generalize to unseen physical pixels on the archived five-field bootstrap generation? This is conditional on fixed upstream image, catalogue, exclusion, scene geometry and noise; it cannot accept the final template generation.

Runtime is an isolated git archive of parent 3609d467a7f17c9ed56a434d71f131ca5cf15528 under pin/. Recipe paper1_dataset: full A3, XP u, lp7s3e8, no brightness width/background/rejection. Original schedule300/3000/2000 is cap only, not convergence. D19 requires five folds; D16 requires converged answer and alternate initialization agreement, with no checkpoint selection by heldout loss.

Audit found original build_folds demotes held sources but retains their pixels in differentiable joint flux solve. Core-disjointness alone does not rule out transitive island leakage. This campaign masks complete heldout128px tiles on every repeated physical pixel in loss AND solve; hidden data are zeroed, noise set1, valid/finite set0. Photutils initialization sees the same sanitized and masked image, trains ePSF and WCS on training stars only. Evaluation uses untouched base scene. Boundary cores scored separately. Training-only init and original K5 folds use seed20260929 diagonal.

First bounded preflight: F1 fold0 initialization, then steps3/5/3 with8CPU affinity24–31, BLAS1, timed RSS. This is compatibility only. Full fits will use measured resources and fixed schedule, followed by convergence extensions if needed; both inits must agree.

Historical preflight conclusion (superseded by completed campaign below): no science OOF result yet. Existing five-field heldout directories contain startup logs, not completed science; original demotion-only harness cannot establish strict heldout pixel independence.

## Reviewed preflight and launch contract

Independent review: strict masking covers valid and finite paths; pool extraction positions and catalogue/geometry remain fixed from prepared scene. Allstar ePSF and WCS parameter seeds are replaced by training-only photutils outputs before training. This is conditional on that prepared geometry, not a blind end-to-end image-pipeline test.

Native nuisance templates vary in forward values but their direct gradient is stopped. This is the original quasi-gradient estimator; it is not scalar-objective gradient descent. Root explicitly retains it as reference while model_campaign owns a matched frozen-nuisance alternative. No exact stationarity claim will be made from its printed scalar loss.

F1 fold0 preflight: init140.64s, RSS1.31GiB; 3/5/3 fit101.57s, RSS3.86GiB,0.55–0.60s per compiled step. Full300/3000/2000 schedule estimated55min/fold,8CPU16GiB request. Physical holdout381951 original valid unique pixels. Held trainees2242; preflight raster2241 positive-flux stars. Figures below were inspected: large initialization residuals, sparse bright bins, red-star dipoles; these are11 optimization steps and have no model acceptance meaning. Grid includes boundary stars and uses BP-RP colour and catalogue Tmag.

![Preflight absolute grid](F1/preflight/analysis/resgrid_strict_k5_F1_preflight_20261004.png)
![Preflight relative grid](F1/preflight/analysis/resgrid_strict_k5_F1_preflight_20261004_rel.png)

Launch: `condor_submit crossfit/condor/fold0.sub`, submit host plscience3.stsci.edu. Executable code/full_fold.sh F1 0; pin3609d467a7f17c9ed56a434d71f131ca5cf15528; input scene SHA256cadf073f988c13fd037779cdd0a2515729531958e262245648c99a0b95c81cc9; strict scene2d3661fdfb785155c86df9ccd67b07ca13958f6f06b07a5034fba672d8dc5b06. Checkpoints every100 steps, fixed schedule, heldout values do not select checkpoints.

## Physical support audit and active job

Reviewer independently reproduced saved masked arrays; PASS, `/astro/armin/koji/syndiff/dev_runs/epsf_model_closure_20261004/review/strict_mask_review.json`. Held-data poisoning (NaN image, extreme noise) followed by the actual initializer sanitization gives exactly identical image/noise planes. This proves input-plane invariance under fixed preparation, not a second full optimizer run.

![Strict training masks and evaluation populations](F1/support.png)

The scorer measures radius3 about fractional x0/y0; therefore its exact complete-core population is2080 (162boundary cores) among2242 held sources. The earlier2045 count used the scene's nominal core flag and is superseded for scoring. `evaluation_selection.npz` is authoritative for the learning curve. All checkpoint scores use fixed original valid pixels and catalogue Tmag bins; nonpositive fitted flux is counted rather than silently deleting stars. Shape and WCS remain frozen during evaluation, with fluxes conditionally refitted on the evaluation image.

Condor2544 exited before Python because getenv=false omitted MAMBA_EXE; corrected environment and pinned official wrapper, replacement2545.0, plscience3 schedd, executing plscience2. Stage2 healthy, ~0.53s/step and Condor MemoryUsage7325MB. Original quasi-gradient estimator, archived image only. Folds1–4 strict scenes are prepared but full fits are intentionally not submitted: parent prioritizes latest-generation hp_d/scene, which input_generation is preparing. `full_fold_generation.sh FIELD FOLD CONFIG LABEL` supports explicit `inputs.training_scene` and `inputs.bootstrap_hp_d`; changing image requires a new LABEL and new initializer.

Code backup: dev branch `codex/strict-crossfit-20261004`, first checkpoint05af6c5, pushed. No merge to shared source. Shared residual-grid portal updated; external Claude Artifact mirror unavailable.

Conclusion (2026-10-04): strict masking and training-only initialization are executable and independently checked for this conditional prepared-scene test. A real full-schedule native reference is running; no K=5 completion, route agreement, convergence, final-image acceptance, or residual improvement is established yet.

## Completed archived F1 fold0 reference, 2026-10-04

Condor2545 returned0 after2630s (43.8min), measured MemoryUsage7325MB. Stage lengths241/1991/2000: the recipe's scalar-loss plateau stopped stages1–2; stage3 exhausted its cap. Final native quasi-gradient norm22.80. Snapshots are every500 steps: `--checkpoint-every 100` controls the rolling latest-state checkpoint, not the historical snapshot interval.

The scoring population is fixed at2062 sources: original trainees in held tiles, complete fractional-centre radius3-pixel cores, at least15 original valid core pixels. Fixed catalogue-Tmag bin counts are30/76/196/551/1209 for8–9/9–10/10–11/11–12/12–13. Endpoint fitted flux is positive for all2062. Initial/final rasters use the exact same2062 IDs and original support. Flux is re-solved on the evaluation image with ePSF and WCS fixed, so this tests conditional shape/WCS prediction rather than a catalogue-flux prediction.

| Catalogue Tmag | Sources | Median held core chi-square, after WCS-only stage | At final cap | Final / minimum saved checkpoint |
|---|---:|---:|---:|---:|
|8–9|30|10089.64|945.95|1.0000|
|9–10|76|3287.63|457.39|1.0000|
|10–11|196|1071.91|133.93|1.0166|
|11–12|551|303.40|39.88|1.0000|
|12–13|1209|86.41|14.00|1.0000|

![Fixed-support learning curves and underlying matched residual stacks](F1/fold0/diagnostics/learning_and_stacks.png)

Every figure was inspected. The broad core/ring initialization residual is much smaller at the cap, while structured residuals remain. The figure places both absolute stacks and final-minus-stage1 differences on a common scale in each magnitude column. Both states and their difference use the same per-star final fitted-flux denominator, following independent reviewer feedback. The prior own-denominator version is preserved only as learning_and_stacks_legacy_own_flux.png. The endpoint is within1.7% of the best *saved* heldout median in all five bins; this is a descriptive single-fold result with no uncertainty or route agreement established. No checkpoint was selected by heldout score.

![Final legacy comparison grid](F1/fold0/diagnostics/final/resgrid_strict_old_F1_fold0_final_20261004.png)
![Final legacy relative comparison grid](F1/fold0/diagnostics/final/resgrid_strict_old_F1_fold0_final_20261004_rel.png)

**Correction to earlier grid labels:** the standard `residual_grid.py` ignores supplied catalogue tess_mag and derives its Tmag from fitted flux. Therefore all standard grids here use GaiaBP-RP and *fitted-flux-derived* Tmag, as legacy compatibility views; they are not the fixed catalogue-Tmag population used in the primary figure/table. The primary figure above uses catalogueTmag. Grid absolute colour scales differ between stage1 and final; use the primary matched-stack figure for visual amplitude comparisons.

Native quasi-gradients also exclude anchor ePSF-base derivatives while their rendered values vary. The separate17D frozen-nuisance chromatic derivative check is not proof of full-joint scalar-objective gradient correctness.

Conclusion: the strict conditional heldout machinery works and a real native reference fit is now measured. This one archived-generation fold improves substantially from its training-only initializer but retains high core chi-square and structured residuals; its cap endpoint does not establish convergence. K=5 replication, alternate starts, latest-generation inputs and final noise/model acceptance remain required. No additional old-generation folds have been launched.

Reviewer scope clarification: scoring also recalibrates the global nuisance-flux prior scale from contributors on the evaluation scene, including held sources. This is explicitly conditional nuisance re-estimation with frozen learned shape/WCS. It is not independent flux/zero-point validation or fully untouched-pixel prediction. A frozen-training-prior sensitivity check would isolate that scoring convention; it has not been run. The primary image comparison was corrected to a common final per-star flux denominator and re-inspected; chi-square curves are unchanged.

## Current-best generation: matched A3 versus CCD19 development

The full generation rebuilt latest native PS1 templates, methodA background, and latest F4 Hotpants; producer support checks preserve every scene array except data, including noise, for C4 and F1. We regenerated strict scenes and training-only photutils initialization on each full-generation image. C4 init2550 and F1 init2553 completed. Template-only scenes with old background are compatibility controls only.

CCD19 adds two CCD-axis colour translations to A3's existing17 chromatic coefficients, without replacing its radial shifts. Both fit arms use the identical renderer graph and zero added leaves initially; A3 freezes these atzero, CCD19 trains them only in the native chromatic stage. AdapterSHA25649ad6a444f2470f203ec3526921c4f6cc861d11caabdb95214b95f16d0a267cb, produced by model_campaign/ccd_shift_preflight and copied immutably into adapter/. It preserves native anchor/nuisance gradient policies. Construction checks: exact same-graph zero parity, rank19 in the conditional colour Jacobian, finite differences below0.1%; full joint WCS identifiability is a separate reviewer check.

C4 full pair2552.0(A3)/2552.1(CCD19),8CPU16GiB each, still running at writing; measured ~4s/step and9.8GB due5003-source overlap island. F1 pair2555.0(A3)/2555.1(CCD19), same recipe/resources; CCD19 hit cgroup16GB hold onscience4 atstep33. Entire failed fit was preserved under F1_full_generation/fold0/CCD19/failed_attempt_2555_1; onlyheld2555.1 removed. Retry2556.0 started cold with identical inputs/recipe,24GB onscience2, completedstage3cap2000. A3 completed2555.0 in2728s (45.5min),7.3GB. Pair endpoint evaluation follows with fixed-score support and no heldout checkpoint selection.

No stage sharing/restart modification was made: native warmstarts unconditionally refresh the nuisance-flux prior even when a saved state is supplied, so an ordinary stage3 restart is not identical to uninterrupted continuation. The native optimizer also resets at each stage. Both actual paired fits retain the full original recipe.

Per-fit launch_contract.json records input scene/init/config/runner/adapter hashes. Development endpoint is median paired log(core chi-square CCD19/A3). The conditional10,000-draw bootstrap resamples128px-tile components joined through shared UIDs or overlap-solve islands, seed20261004; it does not propagate common evaluation-calibration/noise uncertainty. No interval is claimed for one dependence component. Acceptance on untouched confirmation is separately reserved by the reviewer; these development runs cannot retroactively become preregistered confirmation.

## Completed K=5 campaign and numerical limits

All25 current-generation A3 folds completed. F1/F2/S22/C1 each have5 stage3 cap endpoints; C4 has4 caps and one scalar-plateau stop at1919 steps. A DONE marker is not convergence. S22fold0 first failed before training because photutils produced a nonfinite tile PSF; the isolated recovery treated tile(2,5),35 training cutouts, as the native `fit_failed` case and used the existing mean of35 successful training tiles. No held data or scored-source cut changed. Original attempt2577 and successful recovery2595 remain separate. The main DAG retains its historical failed node; the explicit recovery supplies the actual completed scientific fold.

The [full K5 figure report and source browser](http://localhost:18888/epsf-model-closure/crossfit/figures/) contains all60 inspected standard grids, matched F1 measurements/models/residuals, and the C4 negative-amplitude exception. Independent audit verified49,541 unique scene rows, one OOF row each, exact masks/core geometry and all metric arithmetic. These are scene rows across five images, not49,541 independent astrophysical sources.

For each source the in-training comparator is the median score from the OTHER FOUR current-generation fold models evaluated on the SAME source and original pixels. It is not an archived all-star fit. Changing the training footprint changes spatial coverage, initialization and learned prediction uncertainty; the gap is descriptive and is not a clean one-variable causal overfit test. Core chi-square is the mean squared supplied-noise-standardized residual, without nuisance/model DOF correction. Evaluation amplitudes and the global nuisance-prior scale are refitted conditionally on the evaluation image.

One C4fold4 source (row11902, Gaia1436295372260016000,T12.461) has negative fitted amplitude. It remains in every primary score. Its flux-normalized compatibility grid omits that normalization only; raw measured/model/residual stamps are displayed explicitly in the figure report. No source was silently removed from the primary cohort.

The [independent spatial-start report](http://localhost:18888/f1-spatial-start/) preserves the same initial prior and support but starts with a spatially constant mean of the training-only empirical PSFs and header-only WCS. It retains empirical fine/wing structure. Exact2062 sources yield median paired core-chi-square ratio1.003527 (conditional95%0.998360–1.009578), median absolute flux difference0.591751per mille (0.521242–0.702321), and median WCS displacement0.004356px. Both final stages hit caps. Sub-0.5per-mille route equivalence and convergence remain unestablished.

## Model and score definitions

For CCD19 the added displacement is

$$\Delta x_i=(u_i-u_{\rm ref})t_x,\qquad \Delta y_i=(u_i-u_{\rm ref})t_y.$$

Here $i$ is a source row; $u_i$ is its recorded dimensionless Gaia-XP colour (with the fitter's documented BP-RP fallback); $u_{\rm ref}$ is the training-population reference saved by the fitter. The added displacements $\Delta x_i,\Delta y_i$ are in detector pixels. The fitted coefficients $t_x,t_y$ have units pixels per colour unit. They remain exactlyzero in A3 and A3w. This is a colour-dependent translation, not evidence of an intrinsic PSF mechanism.

A3w uses the native fixed linear width term with covariance scale

$$\Delta\sigma_i^2=b\,(q_i-q_{\rm ref})/Q_0.$$

Here $\Delta\sigma_i^2$ is the added per-axis width scale in pixel squared, $b=0.8\times10^{-3}$pixel squared is fixed by the historical recipe, $Q_0=10^4$electrons is the charge unit, and $q_i$ is predicted peak charge in electrons per2-second read, capped at200,000electrons. It comes from external catalogue amplitudes scaled on training pixels and the training-only initial PSF/geometry. The gauge $q_{\rm ref}$ is the training-contributor weighted reference, also in electrons per2-second read. No charge is inferred from held photons, and $b$ is never fitted.

The paired endpoint is

$$d_i=\log(c_{i,\rm candidate}/c_{i,\rm A3}),\qquad D=\operatorname{median}_{i\in\mathcal H}d_i.$$

Here $c_{i,m}$ is the dimensionless mean squared standardized residual over the fixed valid3-pixel core for source$i$ and model$m$; $\mathcal H$ is the pre-frozen physical held-core cohort, and $D$ is the dimensionless equal-source paired statistic. The supplied noise is the stored error plane in electrons persecond. This statistic is not DOF-corrected reduced chi-square. Bootstrap components join128-pixel tiles sharing input UIDs or an overlap-solve island, conditional on learned predictions and realized evaluation calibration.

## Prospective confirmation status

The fresh reserved exposure uses header/Gaia geometry and analytic static seeds, with a new root-frozen catalogue eligibility rule and pre-frozen v1min IDs; historical residual-derived masked_extra/in_contrib_group choices were not copied. The initializer additionally rejects every fresh scene-invalid physical pixel, including the consumed time-dependent mask, plus held tiles. Producer limitation: TNS download403 left bit64 absent; asteroid timing was corrected JD→BTJD. Common full-image upstream preparation remains an explicit conditional input, so this is not end-to-end raw-photon blindness.

Initializer2602 and A32603.0, cold CCD19 placement recovery2604.0, and A3w2608.0 completed. The first CCD19 attempt2603.1 used an older16GB submit and matched the intentionally excluded slow science4; its entire output was preserved separately before a cold24GB restart from the identical initializer. No optimizer state or histories were mixed and no held outcome was inspected. A3w used fresh14933-row charges (SHA1de22fd2f130744d460e402a618c4853869d5f7188bda975689e5624aff11485) and the same starting parameters, with width active from iteration1; no trained A3 prefix was reused.

Independent review approved all fresh input/q/launch/checkpoint guards before a single locked endpoint evaluation2611. The exact2249-source cohort SHA is f1c185d9a451e30fc947902989aac71e5a53147ebac2f1ba38fc2bed9b3e8cc1. Both candidate/A3 contrasts use the SAME10,000 component draws and97.5% intervals (1.25/98.75percentiles), giving the declared95% conditional two-contrast family coverage. Endpoint choice is fixed; no held-out checkpoint selection.

## Final bounded outcome — 2026-10-04

All 25 current-generation A3 folds and the three locked prospective confirmation arms completed. Independent audit reproduced both 10,000-draw confirmation bootstrap columns bit-exactly: 2,249 fixed cores, 50 components, largest component 3.20%. No additional family or checkpoint was selected.

| Confirmation contrast | Median paired log ratio | Conditional 97.5% interval | Conclusion |
|---|---:|---:|---|
| CCD19 / A3 | +0.00374233 | −0.00225252 to +0.00901412 | No demonstrated improvement |
| Fixed-width A3w / A3 | +0.01848306 | +0.00709430 to +0.03024686 | Worse under the conditional endpoint |

The simultaneous two-contrast family coverage is 95%, conditional on the learned models and realized evaluation calibration. These intervals do not propagate common upstream/noise/calibration uncertainty and cannot establish converged-family equivalence or reject a physical broadening mechanism. The width result applies to the single frozen historical value, charge construction and capped recipe.

![Common-denominator confirmation residuals](F1_confirmation/fold0/confirmation/confirmation_residuals.png)

All three rows use each source's SAME A3 evaluation flux denominator, fixed catalogue-T bins and the frozen fitted-colour reference. Absolute rows share a scale; difference rows share a separate finer scale. The central and dipolar structures remain. Sparse bright bins have 52 total stars. Supplemental standard grids use their legacy fitted-flux-derived T and BP-RP definitions and each arm's own flux; they are not the primary matched comparison.

![Absolute confirmation score levels](F1_confirmation/fold0/confirmation/confirmation_score_levels.png)

Median A3 core scores in T=8–9 through 12–13 are 1786.35, 575.33, 183.63, 48.15 and 15.17. These are mean squared supplied-error-standardized residuals, not DOF-corrected reduced chi-square; all three models remain far from a noise-level description. The fresh exposure uses a new epoch/header geometry and a separately frozen catalogue role/population policy, so these absolute levels must not be compared directly with the development cohort as a controlled change.

Confirmation A3 and CCD19 hit the stage2 cap of 3,000, then stopped stage3 on scalar plateaus at 1,322 and 1,399; A3w stopped stage2 at 2,890 and stage3 at 1,671. Final scalar losses are 318.9, 318.6 and 320.3, with quasi-gradient norms 239, 201 and 242. These are not stationary scalar-objective solutions: native stopped derivative policies are retained. Completion, scalar plateau and cap labels do not establish convergence.

Conclusion: keep A3 as the simpler mandatory reference; neither locked extension earns promotion. This bounded investigation is complete, while noise closure, convergence and sub-per-mille route stability remain unaccepted. No further models or reruns are part of this decision.

Records: [browser](http://localhost:18888/epsf-model-closure/crossfit/), confirmation/summary.json and locked_paired_scores.npz under F1_confirmation/fold0; independent audit ../review/confirmation_outcome_review.json; inspected image hashes in confirmation/inspected_figures.json. Source backup: codex/strict-crossfit-20261004 (owned script/report branch); child K5 figures codex/k5-residual-figures-20261004 commit32c30d8; child alternate-start report codex/start-control-report-20261004 commit8744f83. No science data are in Git. The existing shared residual-grid portal was updated; no external Artifact mirror tool was available.