Gaia XP for the chromatic ePSF · S24 C2 K2 + S20 C3 K3 · updated 2026-09-28 (superseded note)

Gaia XP synthetic PS1 vs real PS1

The one page for Gaia-spectra work. Part A checks whether Gaia XP spectra can stand in for PS1 photometry of the ePSF training stars (T < 13, saturated in PS1). Part B asks what the forward model should use as its colour input: one number or several, the XP band-mix colour u or BP−RP, and linear or quadratic in it.

Superseded 2026-09-28: Part B's "no quadratic" conclusion is wrong

dev_runs/colour_evidence_20260928 re-ran §9 without the outlier clip on the total shift. That clip removed 83–89% of the reddest stars (BP−RP > 2.2), which carry the signal. With them kept, a quadratic-in-colour shift is needed: an 8–10σ gain on all four fits, with cubic adding nothing. It agrees with the independent quadratic-colour retrain, which halved the red-end shift. u is a small extra (+0.2–0.3% over quadratic BP−RP), and most of the 10σ "u beats BP−RP" signal in §6–§8 was the missing colour curvature. Part A (XP = PS1) and "one colour number is enough" (§5) still stand. Colour-model v2 is now handled by the chroma v2 orchestrator session (plan dev/forward_epsf_wcs/docs/CHROMA_V2_IMPL_PLAN_20260928.md).

Current status

doneXP = PS1: XP synthetic photometry in the PanSTARRS1_Std system matches real PS1 r, i, z to ≤0.001 mag with no colour trend (Part A). Use Std, not raw.
doneHow many numbers: One. The training stars lie on a one-dimensional track in band-share space, and a second number adds at most a small shift signal on one field (§5).
doneModel form: The current form (shift linear in colour × quadratic in distance from the axis) fits the lateral-colour shift. Its trained coefficients match a direct refit to within 0–10%, and quadratic or cubic colour terms, a cubic in distance or a brightness term add ≤0.7 points (§9). Corrected 19:20: the earlier "quadratic for the shift" (§7) came from the tails of the residual distribution and does not hold up.
doneWhich colour: Neither BP−RP nor u alone. BP−RP under-corrects the shift and u over-corrects it by about the same amount. The best single number is about 45% BP−RP + 55% u, on all four fits. Per-band templates can reach it if each band's shift label is learned rather than fixed at the band's wavelength (§9). The effect is ~1–1.5×10⁻³ px rms, about 1.3% of the per-star residual variance.
openYour decision: Whether to retrain with a better single colour for the shift (the BP−RP/u mix, or learned per-band shift labels), keeping the current form. The expected gain is small (~1×10⁻³ px). The forward model has not been changed.

Part A · Does XP reproduce PS1?

1. Per band, as a function of colour

Eight panels of XP synthetic minus real PS1 magnitude against BP−RP for r, i, z and y, raw system on top and standardised system below; running medians lie on zero except raw z and y — XP synthetic minus PS1 mean-PSF magnitude. Unsaturated (14.5 < m < 18.5), error < 0.02, one PS1 object within 1.5″. Top: GaiaXPy raw PanSTARRS1 bandpasses. Bottom: PanSTARRS1_Std. Red line: running median; band: 16–84%. In the standardised system the r, i and z medians sit on zero from BP−RP 0.3 to 3.2. The raw z band bends to about −0.02 mag at the red end, and Std removes it. y has the widest scatter, with a negative tail at BP−RP ≈ 0.8–1.0.

band	n	raw: median / robust sd	Std: median / robust sd
r	101,926	+0.002 / 0.010	−0.001 / 0.009
i	97,408	+0.003 / 0.008	+0.001 / 0.007
z	95,105	−0.008 / 0.009	+0.000 / 0.008
y	93,162	−0.001 / 0.019	−0.008 / 0.015

2. As a function of magnitude

XP minus PS1 against PS1 magnitude per band: flat inside the green verification window, dropping to negative values brighter than about 14 mag — Brighter than about 14 mag the difference drops to −0.1…−0.3 mag, because real PS1 saturates there. So the training stars cannot use real PS1, and XP is the only source of their band content. The green window is the verification range.

3. The band-mix colour u

u = Σ_b s_b λ_b is the template-weighted mean band wavelength. s_b is a star's share of light in PS1 band b within the combined template, using w = (0.238, 0.344, 0.283, 0.135); λ = 617, 752, 866, 962 nm. The training stars span u = 775–829 nm (1st–99th percentile).

Left: u from XP minus u from real PS1 against BP−RP, flat near +0.18 nm. Middle and right: residuals of u about a BP−RP trend from XP against those from PS1, correlated along the diagonal — Left: u from XP minus u from real PS1. The median is +0.18 nm and flat with colour (0.10–0.21); robust sd 0.47 nm, including PS1's own noise. Middle and right: u minus a smooth BP−RP trend, XP against real PS1. They correlate at 0.70, the same in every colour range. The part both agree on, and so the real part, has an rms of 0.81 nm, about 0.03 mag in BP−RP-equivalent units. The XP-only noise part is 0.63 nm.

Conclusion (Part A)

XP synthetic photometry in the standardised system reproduces real PS1 in r, i and z to within 0.001 mag median and 0.007–0.009 mag scatter, with no colour trend. The y band is usable but noisier (−0.008 median, 0.015 scatter). Band shares for the saturated training stars can come from XP. Use the Std system, because raw z bends at the red end.

u from XP matches u from real PS1 to 0.47 nm rms. That is about 1% of the 54 nm colour range. The +0.18 nm offset is a constant that the colour gauge absorbs.

Beyond a smooth function of BP−RP, u carries only about 0.8 nm (≈0.03 mag) of real per-star information. For u, a calibrated nonlinear map of BP−RP already captures nearly everything. XP adds that map, plus exact band shares for stars that PS1 saturates. Part B tests whether that 0.8 nm matters to the PSF, and whether the four-band share vector needs more than one number.

Confidence is high for r, i and z: each band is based on 93–102k stars, and each median bin holds thousands. The one assumption is that PS1 DR1 mean-PSF photometry is the truth. The forward model was not changed.

Part B · What colour input should the forward model use?

4. How the test works

Each per-band PS1 template convolved with its own PSF P_b gives every star the PSF Σ_b s_b P_b, which is linear in its four band shares. The current colour model (#14) drives all its chromatic terms with one number, BP−RP. I asked what the existing fits leave behind. For each star, the residual of the fit is decomposed into an x-width (c_xx), a y-width (c_yy) and a radial shift toward the optical axis. If a better colour number or a second number carried PSF information, the residuals would line up with it.

Four fits: F1 = S24 C2 K2 and F2 = S20 C3 K3, each with the C5 fit (all T ≤ 13 stars) and the faint-trained ePSF. 7,265–7,994 isolated stars with a valid core per fit.
Band shares from XP PanSTARRS1_Std for 14,602 (F1) and 15,426 (F2) scene stars.
Every regression includes 3×3 CCD-region offsets and a quadratic in peak charge (the brighter-fatter width), so brightness cannot pose as colour. Scores are 5-fold out-of-fold, with paired errors per star.

5. One colour number is enough

Six panels. Top row: band shares against u form thin curves; the spread across the track is 0.07 of the spread along it; residuals against a second colour number are flat. Bottom row: radial-shift residual falls steadily with u beyond BP−RP while width stays flat; u beyond BP−RP scatters ±1 nm, larger for giants; u predicts the full-spectrum TESS wavelength eight times better than BP−RP. — F1, C5 fit. **Top row, one number:** (a) all four band shares are functions of u alone, so the stars form thin curves. (b) The spread across the track (PC2 at fixed u) is 0.07 times the spread along it. (c) The fit residuals do not depend on that second number (x-width 1.1σ, shift 0.9σ). **Bottom row, u vs BP−RP:** (d) at the same BP−RP, stars with a redder u than their BP−RP predicts have a radial-shift residual the BP−RP model missed (11σ), while the width shows nothing. (e) The ±1.1 nm that BP−RP misses comes mostly from giants and metallicity. (f) The four-band u predicts each star's full-spectrum TESS-weighted wavelength to 0.14 nm, compared with 1.27 nm for BP−RP, so per-band templates lose almost nothing to colour variation within a band.

Across the four fits, the band-share PCA gives PC1 97.1–97.3%, PC2 2.3–2.4% and PC3 0.5%. PC2 is the curvature of the same track, not a separate population. Residuals against a second number at fixed u are under 2σ everywhere except the F2 radial shift (3.6σ and 4.3σ). That F2 effect is about 1.5×10⁻³ px end to end, a third of the u effect, and does not appear on F1. The option of free per-band colour terms (27 parameters) is therefore unconstrained by these stars, and a single coordinate is the right form.

6. u against BP−RP, in four fits

u against BP−RP for 63,149 unsaturated stars from real PS1, a tight nearly straight relation; residuals coloured by extinction and surface gravity; residual histogram 0.84 nm wide — The two colours are nearly the same number. u from real PS1 photometry against BP−RP for 63,149 unsaturated T 13.5–16 stars: Pearson 0.996, 99.5% of the variance of u explained. The residual (0.84 nm robust, ≈0.03 mag of BP−RP) is well above PS1 noise (0.18 nm) and follows metallicity (Spearman −0.59) and surface gravity.

Four bar charts over the four fits: radial-shift residual against u beyond BP−RP is 10 to 12 sigma in every fit, widths mostly under 4 sigma; against a second colour number only F2's shift passes 3 sigma; the shift residual needs a quadratic colour term, the width residual does not — (a) Significance of the residual trend with u beyond BP−RP. The radial shift passes 10σ in all four fits. The x-width reaches 3.5–4σ on F2 only. (b) The same against a second colour number at fixed u: only F2's shift passes 3σ. (c, d) Out-of-fold share of the residual explained by each colour form, discussed in §7.

fit	stars	shift vs u beyond BP−RP	slope [10⁻³ px/nm]	x-width vs u beyond BP−RP	shift vs 2nd number
F1 C5	7,994	10.9σ	−1.51	2.2σ	0.9σ
F2 C5	7,858	10.0σ	−1.25	4.0σ	3.6σ
F1 faint ePSF	7,514	11.9σ	−1.38	0.5σ	1.4σ
F2 faint ePSF	7,265	11.9σ	−1.30	3.5σ	4.3σ

The shift slope is the same in every fit (−1.25 to −1.51 ×10⁻³ px per nm). With u beyond BP−RP scattering by about ±1.1 nm, a BP−RP model misplaces the lateral-colour shift by about 1.3×10⁻³ px rms and up to ±4×10⁻³ px for the most off-track stars. That is a tenth of the ~0.01 px effects under study, so the gain is real but modest. The width gain is at most 0.2×10⁻³ px², on F2 only.

7. Linear or quadratic in colour (superseded by §9)

Superseded 19:20. With outliers clipped on the total shift rather than the residual, the quadratic gain shrinks from ~5% to 0.3–1.9%. It is not significant on F1. §9 fits the total shift directly and finds no need for a quadratic.

Nine panels for F2 faint-trained ePSF: width residuals fall in straight lines against BP−RP, the radial-shift residual is a hump that is negative at both ends; middle column trends against u beyond BP−RP; right column against a second colour number — F2, faint-trained ePSF, per-star residuals (binned means ± s.e.). Left column against BP−RP: both widths fall in straight lines, so their colour dependence is linear and only the slope in the fit is off. The radial shift is a hump, negative at both the blue and the red end. That shape is a missing quadratic term. Middle column: against u beyond BP−RP. Right column: against a second colour number at fixed u.

fit	shift: BP−RP lin / quad / cubic	shift: u lin / quad / cubic	shift: BP−RP + u	x-width: u lin / quad / cubic
F1 C5	0.9 / 4.7 / 4.6	1.5 / 4.8 / 4.8	6.9	5.4 / 5.6 / 5.6
F2 C5	−0.1 / 4.9 / 4.9	0.0 / 4.6 / 4.6	6.7	5.7 / 5.7 / 5.6
F1 faint ePSF	1.1 / 5.7 / 5.5	1.9 / 5.8 / 5.8	8.4	9.0 / 9.1 / 9.3
F2 faint ePSF	0.2 / 5.4 / 5.3	0.6 / 4.9 / 5.1	8.1	9.3 / 9.2 / 9.2

Out-of-fold reduction of the residual mean-square, in %, relative to region + brightness covariates only. Paired errors are 0.3–1.0 points.

For the shift, a linear colour term recovers 0–2% of the residual and a quadratic one 4.6–5.8%; a cubic adds nothing. For the widths, linear, quadratic and cubic all tie. In the residual test, quadratic in u and quadratic in BP−RP score about the same on their own. The combination BP−RP + u scores best (6.7–8.4%) because the residual is the true shift minus the BP−RP model's shift: removing it needs both numbers. That is what we expect if the true shift follows u. Only a retrain on u can show the gain directly.

8. Stacked residuals: the trained model with u in place of BP−RP

This is the direct check that u does better, without retraining. Each fit keeps its trained colour shift S(r) = c₀ + c₁r + c₂r², and only the colour it is fed changes: BP−RP is replaced by u, mapped linearly onto BP−RP units so the colour gauge is unchanged. Each star's model moves by the resulting shift change (2.4–3.5×10⁻³ px rms), and its residual is updated to first order. Nothing is fitted to the residuals. Stamps are stacked in a frame rotated so +x points to the optical axis, in three equal groups by u beyond BP−RP. The standard colour × Tmag grid cannot show this: it bins by BP−RP, and the effect averages out within each BP−RP bin.

Stacked residual stamps for F1 in three groups by u beyond BP−RP, before and after feeding u to the trained colour shift; the redder-minus-bluer difference is a dipole along the axis direction that shrinks after the swap; a cut and per-star means confirm it — F1 (S24 C2 K2), C5 fit. Top row: as trained. Middle row: the same fit fed u. Right column: redder-u third minus bluer-u third. It is a dipole along the axis direction, meaning the two groups sit at different positions. Bottom left: a cut through that difference, before (red) and after (blue). Bottom right: mean per-star radial-shift residual in each third.

The same stacked residual comparison for F2; the dipole shrinks after the swap — F2 (S20 C3 K3), C5 fit, same layout. The faint-trained ePSF fits look the same (figures in the run directory).

fit	offset between thirds, stack [10⁻³ px]	per-star slope vs u beyond BP−RP [10⁻³ px/nm]	difference stamp rms: shift / rest / noise
F1 C5	5.44 → 4.36 (−20%)	−1.64 ± 0.14 → −0.78 ± 0.14	2.29 / 0.68 / 0.65
F2 C5	3.52 → 2.65 (−25%)	−1.41 ± 0.13 → −0.59 ± 0.13	1.67 / 0.75 / 0.61
F1 faint ePSF	5.01 → 4.02 (−20%)	−1.45 ± 0.12 → −0.61 ± 0.12	2.10 / 0.66 / 0.61
F2 faint ePSF	3.54 → 2.64 (−25%)	−1.43 ± 0.11 → −0.66 ± 0.11	1.68 / 0.68 / 0.58

Offset between thirds: the shift fitted to the redder − bluer difference stamp with the rotated mean model's gradient. Stamp rms in 10⁻⁴ of flux per px², r < 3.5 px, before the swap.

The swap helps on every fit, and in the same direction. The difference stamp is almost entirely a shift: once the shift is removed, what is left matches the stack noise, so there is no second shape effect hiding here. The swap does not remove everything, though. After it, a third to half of the offset remains, and the per-star slope is still 5σ from zero. Corrected 19:20: I first read this as the data wanting a larger coefficient on u. A direct refit shows otherwise. Refitted on u alone, the trend reverses sign, so u overshoots as much as BP−RP undershoots, and the best colour is a mix of the two (§9).

9. What form and which colour the shift wants (superseded 09-28, see top)

Here the fit targets each star's total colour-dependent radial displacement: the shift the model applied, plus the residual left over. Each candidate form is refitted to it out-of-fold, with the colour-independent part (distance-only terms, which the WCS absorbs), CCD region and brightness as nuisance terms.

Total chromatic shift against colour in three distance bins for F1; the data lie on the trained linear line from +35 to −50 thousandths of a pixel, for both BP−RP and u — F1, C5 fit. The total colour shift spans about +35 to −50 ×10⁻³ px across the colour range. In every distance bin the data (blue: against BP−RP; orange: against u) sit on the trained shift (dashed), which is linear in colour. At this scale BP−RP and u are indistinguishable.

shift per unit colour [10⁻³ px] at distance	500 px	1000	1500	2000	2500
F1 trained	−38.6	−71.4	−88.4	−89.8	−75.4
F1 refitted	−34.2 ± 2.0	−71.2 ± 1.0	−89.4 ± 1.1	−89.0 ± 1.0	−70.0 ± 3.3
F2 trained	−34.1	−56.2	−68.9	−72.3	−66.3
F2 refitted	−28.1 ± 1.8	−53.1 ± 0.8	−66.5 ± 0.8	−68.3 ± 0.8	−58.4 ± 2.5

The form is fine. Out-of-fold, the current form explains 61–66% of the total-shift variance. Changing it (quadratic or cubic colour, cubic distance, a brightness interaction) moves that by ≤0.7 points, within about 2σ. The trained profile matches the refit on F1 and is 4–10% too large on F2.

Three panels: out-of-fold gain peaks for a colour that is about 45 percent BP−RP; the leftover trend against u beyond BP−RP crosses zero at the same mix; free per-band share weights do about as well as the best mix — (a) Scanning single colours c = a·BP−RP + (1−a)·u: the gain over BP−RP alone peaks at a ≈ 0.25–0.55 on all four fits. (b) The leftover trend against u beyond BP−RP changes sign between the two colours and is zero near a ≈ 0.45. BP−RP under-corrects and u over-corrects by similar amounts. (c) A colour made from the band shares with free weights per band, which is what per-band templates with learned band shift labels deliver, performs about as well as the best mix.

This is the reason feeding u to the trained model removed only about half the effect (§8). The shift is not a function of u alone. Physically, lateral colour comes from optical dispersion, which does not have to weight wavelength the same way u does. The effect is small: 1.2–1.4% of the per-star residual variance, about 1–1.5×10⁻³ px rms, against a chromatic shift that spans ~80×10⁻³ px and that the current model already captures.

Conclusion (Part B) (superseded 09-28, see top)

One colour number is enough, and the current shift form is right. Linear in colour × quadratic in distance from the axis fits the ~80×10⁻³ px lateral-colour shift. Its trained coefficients agree with a direct refit to 0–10%, and no richer form earns its place (§9).

The colour it is fed is slightly wrong. BP−RP and u both miss the shift by about 1–1.5×10⁻³ px rms, in opposite directions. The best single colour is about 45% BP−RP + 55% u on all four fits. Per-band templates can reach it if each band's shift label is learned rather than fixed at its wavelength (§9). In the stacks, feeding u to the trained model already shrinks the colour dipole by 20–25% (§8), and the mix should do better.

Widths: no colour input does better than BP−RP (§6). The width residuals are linear in colour, so the linear width terms are the right form.

Next step (needs your go-ahead): this is a ~10⁻³ px refinement. If it is worth doing, retrain #14 with learned per-band shift labels, or the mixed colour, on F1 and F2, on a fixed evaluation star set. Then compare §8's stacks, the residual grid and held-out χ² against C5. It is not the cause of the larger residuals under study.

Limits: residual tests on fits trained with BP−RP, plus first-order swaps, not retrains. Two fields, one FFI each. The mix weight is chosen in-sample, so its gain is slightly optimistic. XP sampling stops at 1020 nm.

Part A data: /astro/armin/koji/syndiff/dev_runs/xp_colour_20260925/ (README.md). Code: dev/forward_epsf_wcs/template_kernel/xp/. XP comes from 6 Gaia DR3 bulk files (135,023 of 135,904 stars) and PS1 DR1 from VizieR II/349 via CDS XMatch. Part B data and code: /astro/armin/koji/syndiff/dev_runs/xp_dimension_20260925/ (README.md; analyse.py, perstar_terms.py; results_*.json per fit).