Add salvage and merge utilities for recovering from partial batch failures

[Copilot]

When dvmdostem batch runs partially fail (e.g., 2 of 10 cells), users must either restart the entire batch or manually merge outputs. This adds pyddt-salvage and pyddt-merge commands to automate recovery.

Implementation

Core modules

• salvage_run.py: Identifies failed cells from run_status.nc, backs up outputs, generates run-mask for failed cells only
• merge_runs.py: Merges salvaged and rerun outputs using xarray.where() with proper dimension broadcasting

CLI integration

• Added pyddt-salvage and pyddt-merge entry points to pyproject.toml
• Both support --verbose flag for detailed progress

Merging strategy

Uses run_status to select data per-pixel across all dimensions:

# For each output variable with dims (time, Y, X) or (time, layer, Y, X):
condition = rerun_status > 0  # Shape: (Y, X)
condition_broadcasted = condition.broadcast_like(variable)  # Matches variable dims
merged = xr.where(condition_broadcasted, rerun_data, salvaged_data)

Usage

# After partial failure in output/
pyddt-salvage output/ backup/ --original-mask run-mask.nc --new-mask failed-cells.nc

# Rerun with failed-cells.nc, then merge
pyddt-merge backup/ output/ merged/

Testing

• 5 unit tests covering identification, backup, mask creation, file merging
• Integration test simulating full workflow with multi-dimensional outputs
• CodeQL: 0 alerts

Documentation

Added pyddt/docs/SALVAGE_AND_MERGE.md with workflow examples and troubleshooting.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Continuing a failed run in a batch</issue_title>
<issue_description>From Slack discussion:

Elchin Jafarov
I need a script that can run failed batches without restarting them over.
Suppose I have a run-mask with 10 enabled grid cells, 8 out of 10 finished, I need to re-run the other 2, so that I can continue writing into the existing result files.
How should I modify the command line so that it continues writing into existing restart and other results files?

tcarman
ooh interesting question. I have not tried this, so you’d need to experiment with it, but it might work to make a new run mask with your 2 cells enabled and the 8 cells disabled. Then run dvmdostem with the —no-output-cleanup flag?

Elchin Jafarov
I will try and let you know. Thank you!

Elchin Jafarov
Unfortunately, it does not do it. it cleans the results files and runs only enabled cells. we need to copy previous results files and then merge them together.

tcarman
hmm, shoot. so --no-output-cleanup is not working?

Elchin Jafarov
it is designed for keeping the restart files, not other results

tcarman
ah, ok, that sounds kinda familiar.
I wonder if we could easily modify it so that it leaves the other outputs untouched…trying to think if that would create other problems…
Here is another workflow idea to test:

• copy failed run to “backup” directory
• create new run mask with only failed cells enabled
• start second run
• merge outputs with backup copy. (I haven’t written this logic, but I think with xarray it might not be too terrible some …would take some experimenting. might have to transpose & concat and then re-transpose)

</issue_description>

<agent_instructions>Can you use the information described in this issue to start solving the issue? </agent_instructions>

Comments on the Issue (you are @copilot in this section)

@tobeycarman Sample code to setup trivial experiment for testing(note that this uses the command line interface that will exist when PR #809 is merged):

pyddt-swd --input-data-path /work/testing-data/standard/inputs/cru-ts40_ar5_rcp85_ncar-ccsm4_IMNAVIAT_CREEK_10x10/ test-merge-failed-runs
ls test-merge-failed-runs/
cd test-merge-failed-runs/

pyddt-runmask --reset --yx 0 0 run-mask.nc 
pyddt-runmask --yx 1 1 run-mask.nc 
ncdump run-mask.nc 

pyddt-outspec -s config/output_spec.csv 
pyddt-outspec config/output_spec.csv --help
pyddt-outspec config/output_spec.csv --on CMTNUM monthly
pyddt-outspec -s config/output_spec.csv 

ncdump /work/testing-data/standard/inputs/cru-ts40_ar5_rcp85_ncar-ccsm4_IMNAVIAT_CREEK_10x10/vegetation.nc 

dvmdostem -p 10 -e 10 -s 10 -t 10 -n 5 -l monitor
ncdump output/run_status.nc 
mv output/ output_A

cp run-mask.nc run-mask_A.nc 
pyddt-runmask --reset --yx 9 9 run-mask.nc 
pyddt-runmask --yx 8 8 run-mask.nc 
ncdump run-mask.nc 
dvmdostem -p 10 -e 10 -s 10 -t 10 -n 5 -l monitor

ncdump output/run_status.nc 
mv output output_B
cp run-mask.nc run-mask_B.nc

@tobeycarman Sample to try merging outputs, given above setup

    import xarray as xr
    ds_A = xr.load_dataset('output_A/CMTNUM_yearly_tr.nc')
    ds_B = xr.load_dataset('output_B/CMTNUM_yearly_tr.nc')
    msk_A = xr.load_dataset('run-mask_A.nc')
    msk_B = xr.load_dataset('run-mask_B.nc')
    stat_A = xr.load_dataset('output_A/run_status.nc')
    stat_B = xr.load_dataset('output_B/run_status.nc')
    xr.where(stat_B.run_status, ds_B.CMTNUM, ds_A.CMTNUM)
```</body></comment_new>
<comment_new><author>@tobeycarman</author><body>
```python
In [27]: xr.where(stat_B.run_status, stat_B.run_status, stat_A.run_status)
Out[27]: 
<xarray.DataArray 'run_status' (Y: 10, X: 10)>
array([[100.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],
       [  0., 100.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],
       [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],
       [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],
       [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],
       [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],
       [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],
       [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],
       [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0., 100.,   0.],
       [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0., 100.]])
Dimensions without coordinates: Y, X

@tobeycarman Would be a little bit of work to write a generalized script that could do this for any set of dvmdostem outputs...Maybe 2 scripts even 1) **`salvage_run.sh`**: moves outputs to "backup" location, resets run mask based on original mask and mask of successfully run pixels (run-status.nc value? any other...

• Fixes Continuing a failed run in a batch #810

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.