# C4 DAG 345: memory hold and restricted continuation

Recorded 2026-10-03 EDT by the separate, user-authorized Codex CLI continuation. No desktop chat was resumed or notified.

## Diagnosis and evidence

`VAL_C4_00786`, recipient `skycell.2487.034`, job **2377.0** on schedd **plscience3.stsci.edu**, was held at **2026-10-03 21:10:44 EDT** on `slot1_3@plscience8.stsci.edu`. Both the task event log and DAG node log report **hold code 34, subcode 0: exceeded the 16384 MB cgroup memory limit**. The last sampled usage was 8291 MB; that sample does not measure or disprove the transient cgroup peak. Stdout/stderr are empty: no Python traceback identifies the allocating instruction. Runtime before eviction was 468 seconds. Resource request was 2 CPUs, 16384 MB.

The immutable source loads all donor images and deletion arrays, then materializes three fixed-domain arrays for each donor plus rendering temporaries (`field_validation.validate_recipient`, `transport.transport_cell`). This is a plausible memory pressure mechanism, **not a profiled allocation diagnosis**. No science mismatch has been established for this recipient.

Direct `condor_q` and `condor_history` attempts failed because this restricted session cannot initialize the configured network interface. SSH first encountered a system SSH configuration permission problem; `ssh -F /dev/null` then failed with socket `Operation not permitted`. These failures do not prove the scheduler is down. See saved `queue_attempt.txt` and `history_attempt.txt`. Live queue status cannot be independently certified here.

## Metadata audit at 2026-10-04 01:14:11 UTC

- Inventory: **975 C4 cells**. All **975/975 saved result JSONs report exact replay**. This continuation did not rehash all ledger payloads or rerun replay.
- **815/975 recipient reports** exist and report exact published contributions, operator closure and L5 closure, with contribution files present. **160/975** recipient reports are absent. The snapshot state has 815 validation-complete and 3 validation-running records; the held job's stale running record is not proof of execution. Other DAG work can still progress.
- Available recipient reports record **4179 applied rim patches and zero missing patches**, restricted to those 815 recipients. This is not full-field seam certification.
- Full-template reduction summary is absent. No completed full-template result is claimed.
- `metadata_audit.json` saves the missing-cell list, audit scope and SHA256s of copied evidence. Source and event-log snapshots are under this directory. Full image/payload checksum validation remains pending.

## Recovery action required

Use a session with working Condor access on plscience3. Re-query **only DAG 345** and inspect current job 2377.0 before any change; it may have changed since this snapshot:

```sh
condor_q -name plscience3.stsci.edu -constraint 'DAGManJobId == 345 || ClusterId == 345' -af ClusterId ProcId JobStatus HoldReason RequestMemory Args
condor_q -name plscience3.stsci.edu 2377.0 -long
condor_history -name plscience3.stsci.edu 2377 -limit 1 -long
condor_status -af Machine Cpus Memory MemAvailableMB LoadAvg State Activity
```

If 2377.0 is still the same owned held validation with hold code 34, an appropriate **single-task resource trial** is to increase that existing job to **32768 MB** and release only it, retaining the same immutable code pin. This is provisional headroom above the failed 16 GiB allocation, not a measured successful requirement. Record the pre/post ClassAd and commands; measure the rerun peak and output validation before changing any broader request:

```sh
condor_qedit -name plscience3.stsci.edu 2377.0 RequestMemory 32768
condor_release -name plscience3.stsci.edu 2377.0
```

These commands **were not executed**. Check available capacity first. Do not submit a duplicate DAG or validation alongside the held job. Do not edit the pin or running wrappers. If a code change/new run is necessary instead, use the isolated editable checkout, tests, a fresh immutable pin and explicit non-overlapping ownership; reconcile the original DAG before submission. Remain within C4.

After recovery, finish all 975 recipient validations and the existing reduction node; independently check saved payload checksums and frozen-template replay. Complete Gaia AND PS1 count/page/coverage and deduplication checks, unknown components, off-cell halos, no-op versus overlap attribution, and recovery of the 21 known omitted faint Gaia IDs with explicit denominators. Associations/support windows are not individual stellar flux or verified stars. Complete approved independent null/closure tests and field seam/edge/unmatched-source image review through measured-resource Condor jobs. Do not infer that the existing arithmetic bounds alone fulfill every independent-oracle acceptance gate.

## Images inspected and scientific limits

The existing `figs/ps1_only_examples.png` and `figs/seam_closure.png` were visually inspected. The former shows deletion around bright halos, including ambiguous catalogue positions; it does not prove all plotted PS1 detections are separate stars. The latter is a one-cell intervention with fixed neighbours and a visually near-zero closure panel at its plotted scale. It cannot establish full-field closure or ideal seam physics. Existing report captions remain applicable.

![Existing PS1 unmatched-identity pilot examples](../../figs/ps1_only_examples.png)
![Existing one-cell seam pilot](../../figs/seam_closure.png)

Prior pilot evidence (three-cell replay, 21 recovered Gaia IDs, two off-cell saturated halo candidates, 79000 exact L5 pixels and five rim patches) remains pilot evidence. Known F2 cross-projection physical mismatch remains expected failure; no repair is claimed.

**Conclusion:** high confidence in a scheduler-reported memory hold, with exact allocation peak/instruction unmeasured. The C4 ledger repair is incomplete and needs scheduler access plus a controlled resource recovery. No jobs were released, restarted, duplicated, cancelled or otherwise changed; no immutable/live-main code, production products, removal physics or paper adoption was changed. No additional watcher/agent was scheduled. Only shared-portal records were updated; no external Artifact mirror or messages were sent.