# PS1 removal ledger implementation

2026-10-05. Status: **C4 numerically validated, pending visual review** (DAG 345 finished 10-04, status OK). Steps 2/3 (C4 figures, match-quality report) and step 5 (C1/F1/S22/F2 campaigns) started 10-05, see the last section. 2026-10-03 status was: incomplete; validation memory hold requires intervention. User approved stages A–E of the ledger plan in this chat ("alright, please get going"). Production merge, template replacement and removal-physics changes remain outside approval.

Question: can every pixel deletion and affected Gaia/PS1/image source be accounted for through the actual template ownership and seams?

Code: isolated branch `codex/ps1-removal-ledger`, based on vrow-onewriter `d128192`, at `/home/kshukawa/syndiff_pipeline/.claude/worktrees/codex-ps1-removal-ledger`. Historical paper v2 products stay pinned to `648c0bc`; v3 and v2 provenance must remain distinct.
Inputs: immutable paper dataset, prior C4 audit and cached SEP/footprint products. Outputs only here. No production science files changed.
Plan: `/home/kshukawa/syndiff_pipeline/doc/ps1_ledger_fix_plan_20261002.md`.

## Headline / Conclusion
**2026-10-05:** DAG 345 exited status 0 at 2026-10-04 04:21 EDT with no failed nodes. All 975 C4 cell ledgers have `replay_exact: true`; all 975 recipient validations plus the field reduction completed (976 `validation_complete` states). `field_validation/C4/summary.json`: status `numerically_validated_pending_visual_review`, rebuilt template bit-identical to the published one (sha256 `20e92828…`), max closure residual 1.66e-3, integrated deleted signal 1.24e9. Jobs 2377.0, 2401.0, 2419.0 were held at 16 GiB (hold code 34) and completed after release at 32 GiB with unchanged code/inputs (Condor-reported usage ~8 GB, so the overflow is a short peak). Still NOT certified: catalogue completeness, blended per-star flux, whole-field visual review, F2 cross-projection case, other four fields. Not merged. Superseded status below this line is kept for history.

Previous (10-03): validation job 2377.0 exceeded 16 GiB; this CLI session cannot access Condor networking. See the recovery record below. Implementation underway; no completeness or seam-validation claim yet. First inventory cached products and build exact cell ledger plus PS1 catalogue acquisition and association. Then validate transport before expanding across all five paper fields.

## Implementation checkpoint 2026-10-02

Initial checkpoint `7445326` committed and pushed to `origin/codex/ps1-removal-ledger`. Source package: `syndiff_pipeline/template_creation/processing/removal_ledger/`; tests `tests/test_removal_ledger*.py`. Unmerged, isolated from the live editable checkout.

- Optional recording hooks capture actual capped supports before zeroing; ordinary image output is unchanged.
- Lossless identifiers, distinct component/source records, atomic ledger publication and payload validation.
- Gaia query has no photometric selection; PS1 DR2 stack queries preserve non-primary and duplicate-object measurements, require count/page agreement, retain metadata and raw-page hashes.
- Gaia/PS1 candidates use proper-motion propagation to the PS1 mean epoch; ambiguous/missing-motion cases are not forced into identities.
- Cached C4 footprint replay exactly reproduces the published image; all 21 previously omitted Gaia IDs have explicit deletion associations.
- One-cell historical seam transport reproduces the published padding-aware image exactly. The transported deletion closes against the paired render within the declared float32 arithmetic allowance. This is a scoped operator test, not full-CCD or ideal cross-projection certification.
- Initial tests: 45 passed, one pre-existing strict expected failure for the known F2 cross-projection image disagreement; added matching tests pass. Further tests continue as modules are completed.

Additional reusable caches were found at the centre cell of each of the five fields: their raw inputs contain no exactly zero finite pixels, so the saved background-suppressed image identifies the SEP-positive/bright-core union unambiguously. Code rejects this recovery when original zero pixels make it ambiguous. Five-field pilot catalogue/ledger validation is now running, reusing these caches without SEP.

Commands use `mamba activate syndiff`, `PYTHONDONTWRITEBYTECODE=1 OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1`, with cwd the isolated worktree:

```
python -m syndiff_pipeline.template_creation.processing.removal_ledger.pilot --out /astro/armin/koji/syndiff/dev_runs/ps1_ledger_fix_20261002 --inventory
python -m syndiff_pipeline.template_creation.processing.removal_ledger.pilot --out /astro/armin/koji/syndiff/dev_runs/ps1_ledger_fix_20261002 --field C4 --cell skycell.2528.005
python -m syndiff_pipeline.template_creation.processing.removal_ledger.validate_transport --out /astro/armin/koji/syndiff/dev_runs/ps1_ledger_fix_20261002
```

## Current limits

Full-field backfill and all-seam validation are not complete. Candidate support circles do not certify complete stellar PSF removal. Catalogue-image astrometry is being checked separately; published pilot tables retain ambiguities. No new removal algorithm, production merge or template replacement has occurred.


## Full C4 campaign submitted

{
  "cluster": 345,
  "submit_host": "plscience3.stsci.edu",
  "time": "2026-10-02T21:29:53.823629-04:00",
  "code_sha": "d3549ee83c0d58b89d94ceb7545d46d4140877b5",
  "code_pin": "/astro/armin/koji/syndiff/dev_runs/ps1_ledger_fix_20261002/code/pin_d3549ee",
  "dag": "/astro/armin/koji/syndiff/dev_runs/ps1_ledger_fix_20261002/campaign/backfill.dag",
  "dag_sha256": "eb4837fd6745caa1e0e6b306975fce6021ab00be470f79dd6390edd1958fe1ca",
  "scope": "975 C4 cell ledgers, 975 recipient validations, then full-template reduction",
  "owner": "PS1 ledger repair chat; independent HTCondor DAG, not SynDiff supervisor",
  "max_source_jobs": 8,
  "max_validation_jobs": 4
}

Monitor with `condor_q -name plscience3.stsci.edu 345 -dag` and the campaign logs. No other task jobs were changed. Other fields have completed centre-cell pilots; full-field expansion follows the C4 validation gate.


### Campaign launch verified

First C4 cell task completed on the pool and eight source tasks are running; no failures at the initial check. The pinned job snapshot is d3549ee, separate from the development worktree. Source catalogue/image calibration, original coordinates, exact operation support and deleted signal are saved. The two previously unmatched nonzero halos now have explicit candidate associations to their off-cell Gaia stars; all 21 known omitted Gaia sources remain recovered after image calibration.

The real L5 check now uses all five frozen inter-cell rim patches for 2528.005: every one of its 79,000 published contribution pixels is reproduced exactly. The paired deletion closure passes the fixed arithmetic allowance. See `transport/skycell.2528.005/report.json`. These are pilot gates, not a declaration that the full C4 campaign has finished.

Latest code tests: 57 targeted tests passed (including existing removal and seam primitives), plus the previous canonical-cell suite with its documented expected failure for the physical F2 cross-projection disagreement. All newly added ledger tests pass.

### 2026-10-03: user-authorized detached follow-up watcher

Started `ps1-ledger-watch-345` tmux session on plscience3, polling only DAG 345 every 30 minutes. Authenticated Codex CLI preflight and watcher state tests passed. On completion/failure, launches one separate CLI continuation using `watcher/handoff.txt`; immutable campaign pin remains unchanged. This does not resume the desktop chat. See [watcher record](watcher/README.md), `watcher/status.json`, and eventual `watcher/followup_result.md`. Stops after the continuation exits. No production merge or replacement authorized.


## 2026-10-03 CLI follow-up: intervention required

DAG 345 / job 2377.0 / skycell.2487.034 hit hold code 34 (16384 MB cgroup memory limit). No job controls were executed. At 2026-10-04 01:14:11 UTC, the bounded metadata audit found 975/975 exact-replay result records and 815/975 passing recipient reports; 160 reports and full-template reduction remain outstanding. This is not independent payload-checksum or full-field scientific certification. Other DAG tasks may continue.

[Recovery record and exact commands](watcher/recovery_20261003/RECOVERY.md) · [Audit with saved evidence SHA256s](watcher/recovery_20261003/metadata_audit.json). The existing implementation report and progress page were updated; progress.html is regenerated by running campaign tasks and may overwrite its intervention banner, so this README and index.html are the durable notices. Existing pilot figures were inspected; full-field visual and Gaia/PS1 completeness review remain pending.

Conclusion: scheduler-confirmed memory hold, unresolved here because queue/history/SSH networking is blocked by the CLI sandbox. Resume from the existing owned job with a measured single-task resource trial; do not submit a duplicate campaign. No watcher replacement, production change, merge or adoption.

### 2026-10-04 dependent experiment recovery handoff

The atlas/neighbor-training parent session, following the user's instruction to proceed through readiness and training, re-queried held DAG-345 job 2377.0 and started the prescribed single-task 32768 MB trial on that existing job. It was observed running with unchanged code/inputs. Pre/post evidence and the bounded recovery watcher are in /astro/armin/koji/syndiff/dev_runs/neighbour_readiness_20261004/recovery/. That watcher may release only existing jobs 2401.0 and 2419.0 at the same memory after 2377.0 exits zero and its recipient report passes. Check that status before any overlapping recovery; no duplicate campaign or production change was made. Other held jobs, if any, need separate diagnosis.

## 2026-10-05: C4 review and four-field expansion (Claude Code session)

User asked for steps 1–3 and 5 of the follow-up list: (1) records updated (this file, INDEX.md); (2) whole-C4 figures and visual review; (3) match-quality report; (5) run the same pinned code on the other four paper fields. Step 4 (diagnosing the F2 cross-projection disagreement) was NOT requested; F2 is run, but any F2 seam failure is recorded, not diagnosed.

New subdirectories (the existing C4 campaign, DAG 345 files and `campaign/` are untouched):

- `review_20261005/figures/`: whole-C4 figures (step 2)
- `review_20261005/matching/`: match-quality report (step 3)
- `fields_20261005/main/`: campaign root for C1+F1+S22; `fields_20261005/f2/`: separate root for F2, so the known F2 disagreement cannot trip the three-failure circuit breaker for the other fields. Same immutable code pin `code/pin_d3549ee` (d3549ee). Validation requests 16 GiB with automatic escalation to 32 GiB on hold code 34.

Results are appended below as they land.

### 2026-10-05 step 2 result: C4 whole-field visual review PASSES (closure)

Record: `review_20261005/figures/README.md` (figures fig1–fig6 there; reviewed by the orchestrating session, every PNG viewed). Closure allowance is the code's float32 rounding bound (cell: `4·eps32·(|b|+|a|+|d|)+1e-10`, transport.py L72-73; TESS-bin: `8·eps32·(...)+1e-8`, field_validation.py), not a fitted tolerance. Over all 68,161,536 template pixels max |res|/allowance = 0.082, zero pixels exceed; residual is symmetric speckle at bright-star cores, no seam/projection/skycell structure; medians identical across interior (532), same-projection edge (286), cross-projection edge (116) and corner (41) recipients; worst cells are interior. Both off-cell halos are now explicitly `possible_partial_support` (centre outside, Gaia identity, no trigger id); all 21 omitted faint Gaia sources in 2528.005 are `centre_removed`/`centre_membership` on saved deleted flux. Observation (orchestrator): faint stars inside both off-cell halos are deleted with the halo — the known chained-footprint removal behaviour (removal_hole_20261001), recorded by the ledger, not a ledger error. Open small items: 110 TESS bins with negative deleted flux not examined; 39.6% of pre-removal flux is deleted (not independently checked). Closure is linear by construction: it certifies bookkeeping/ownership, not catalogue completeness, blended per-star flux or cross-projection physics.

### 2026-10-05 step 3 result: C4 matching — catalogue integrity passes, source-quality gate NOT cleanly met

Record: `review_20261005/matching/README.md` (tables/, figs/; key figures viewed by the orchestrating session). All 975 cells: complete Gaia and PS1 queries (row counts = service counts, no truncation), matching/image calibration `calibrated`, no duplicate keys. 26,671 nonzero regions carry 1.99e9 PS1-cell flux: 99.44% has a Gaia centre inside support, 0.55% only a PS1 centre, 3e-5 neither (58 regions, mostly 1–6-unit specks plus 4 off-cell-halo edge strips). Audit cases fixed: 21/21 faint Gaia sources centre-removed and linked; both off-cell halos explicitly candidate-level; all 702 old unmatched rows map 1:1 to new regions (700 zero-flux).
Gate failures / open items:
1. Saturated triggers: centre sits on a NaN core outside exact support (1,536 regions, 19.5% of flux; 75–90% of T<9 triggers), so identity rests on a 160–480 px candidate circle. Ledger-rule fix candidate: treat enclosed NaN holes as support for identity.
2. PS1-only removed centres (74% of removed PS1 centres) are mostly spurious stack detections inside halo footprints (sample 4 plausible / 17 doubtful / 3 wrong); not evidence of removed stars.
3. 5.3% of accepted pairs reference sources absent from the ±600 px-trimmed `sources.parquet` (all cells): referential inconsistency.
4. 91 in-cell T<13 Gaia rows are not region triggers (90 ambiguous, 80 on NaN cores), cause not investigated.
5. Match tolerance is the fixed 0.3" floor everywhere; 109 outlier cells and a Dec 55–57° band with larger pair separation (p95 ≤0.147", below tolerance).
6. Region-sum deleted flux 1.99e9 (PS1 cells) vs field summary 1.24e9 (template) not reconciled; hypothesis = off-CCD cell area + overlaps, untested.
By-eye checks were one reader, n=24 per gallery. No completeness claim beyond the Gaia DR3 and PS1 DR2 stack queries.

## 2026-10-06: step 3 (match quality) for C1, F1, S22, F2
Record: `review_20261006_fields/matching/README.md` (+ `cross_field_table.md`). Parameterised C4 scripts reproduce the C4 review (full C4 rerun; one tied nearest-neighbour z value differs, outlier counts unchanged). Conclusion: catalogue integrity passes in all four fields; matching fails the gate for the same reasons as C4, with no new failure class (PS1-only deletions largely unverifiable, 4-8/24 sampled junk; saturated-trigger identity on candidate circles, 17.5-25.8% of flux; 0.3" floor tolerance; region-level flux only). Field-specific: the Dec 54.5-57.5 pair-separation band recurs in F1/S22 (sky-position, not field); F2 has a vertical column of flagged cells along projections 2611/2628 (high calibrator scatter, p95 to 0.149"), not diagnosed; sources-margin bug 2 grows with Dec (F2 12.4%). Bug item 7 (region-sum vs field deleted flux) waits for the validation REDUCEs.

## 2026-10-06: v3 transport proposal (not run)
Facts: `review_20261006_fields/v3_transport_facts.md` (pin d3549ee already contains the v3 renderer `canonical_cell.canonical_cell_image`; v3 operator is linear; only renderer injection, convolved-store key and published-product paths change). Proposal: `/home/kshukawa/syndiff_pipeline/doc/ps1_ledger_v3_transport_proposal_20261006.md` — Target A (v3 combined field templates via FrozenFieldOperator, new inventory_v3, bit-exact gates, ~1 Condor-day for 5 fields) and Target B1 (paper per-band v3 templates via a per-band-operator adapter, gated on perband contribs); per-band deletion split (B2) deferred to the inline ledger. Awaiting user decision.

## 2026-10-06: C4 flux budget and negative bins (items 6 and the 110 bins) — neither is a bug
Record: `review_20261006_fields/flux_budget/README.md` (figs viewed by the orchestrating session: budget_waterfall, neg_mechanism_zoom). Region sum 1.9877e9 → template 1.2409e9: overlap strips owned by a same-projection neighbour −2.69e8, projection-edge frame with no bin −6.3e7, cell area with no TESS bin −2.00e8 (= 1.4557e9 on assigned pixels, exact over 975 cells); convolved-mask bit 12 (deleted bright-star cores) −2.15e8 (taken as exact C−A minus the sampled convolution term; direct 82-cell estimate −2.03e8 ± 0.28e8 agrees); convolution + padding +6e5 ± 2.1e6. The 110 negative bins are float32 kernel-fringe residue (|deleted| ≤ 1.6e-35, 15–23 bins from real deleted flux, single recipient each) from negative-valued rim pixels in the signed deletion. Correction to the 10-05 step-2 record: "39.6% of pre-removal flux deleted" refers to unmasked bins only; ~10.8% of region deleted flux (bit-12 cores) never enters template or validation maps.

## 2026-10-06: association revision r2 (items 1, 3 and the PS1-only neighbour exclusion)
Record: `assoc_r2_20261006/README.md`. Code `claude/ledger-assoc-r2-20261006` @ 5c29e8f (new `removal_ledger/revision.py` + tests; derived layer over published ledgers, not pushed/merged). All 4,880 cells of C4/C1/F1/S22/F2. Saturated-trigger flux tied only by candidate circle: 17.5–25.8% → 0.11–0.21% (enclosed zeroed core, ≥30 px component, identity only). Pair references to absent sources: 2.7–12.3% → 0. Neighbour lists exclude PS1-only centres (kept C4 49,879 / excluded 108,850). Open: 259–469 non-trigger Gaia stars per field in saturated cores not yet in neighbour lists (user decision).

## 2026-10-07: results canvas
https://claude.ai/artifact/GnMskXQMERYAQtYin4tFCV (source `/home/kshukawa/syndiff_pipeline/doc/ps1_ledger_canvas/`, figure copies `canvas_20261007/`). Status, findings and open decisions (v3 validation targets, Gaia stars in saturated cores, merging assoc r2).

## 2026-10-07: user decision — no full v3 validation
Reason: the closure is linear by construction and the v3 templates are built from the identical combined cells with the renderer already in the ledger pin; v3 changes only which cell writes the vertical overlap strip. A full v3 run (~4,900 cells, ~1 Condor-day) would mostly re-prove that. Consequences: no whole-field closure figures for C1/F1/S22/F2 (C4 v2 figures stand as the worked example); the v3 transport proposal (`/home/kshukawa/syndiff_pipeline/doc/ps1_ledger_v3_transport_proposal_20261006.md`) is not executed. Optional, not requested: a ~3-cells-per-field bit-exact spot-check against the published v3 and per-band products.