Distributed may deadlock in `one_rank_first` when an exception occurs on the target rank

[Prathamesh8989]

Bug description

In ignite/distributed/utils.py, the one_rank_first context manager is susceptible to a distributed deadlock if an exception occurs within the context block on the target rank.

The current implementation uses two barriers to synchronize ranks. If the process with the designated rank encounters an error (e.g., a network failure during data download or a disk I/O error) within the yield block, it will never reach the second barrier. Consequently, all other processes that are either waiting at the first barrier or expecting to synchronize at the second will hang indefinitely.

Steps to reproduce

Run the following logic in a distributed environment (2+ ranks):

import ignite.distributed as idist

# Simulate a crash only on rank 0
with idist.one_rank_first(rank=0):
    if idist.get_rank() == 0:
        raise RuntimeError("Rank 0 crashed!")
    # Other ranks are now stuck at the first barrier or will hit the second

Expected behavior

The exception should propagate, and the entire distributed job should terminate gracefully. Instead, the non-crashed ranks hang, requiring a manual kill of the processes.

Code Snippet

Current implementation in ignite/distributed/utils.py:

if current_rank != rank:
    barrier()

yield  # <--- If an exception happens here on 'rank'

if current_rank == rank:
    barrier() # <--- Other ranks never reach or pass synchronization

[Prathamesh8989]

cc @vfdev-5 — just extending my thoughts on this.

If the diagnosis above looks correct, I’d be happy to work on a fix and submit a PR. Would appreciate your perspective on whether this should be addressed in one_rank_first.

[vfdev-5]

@Prathamesh8989 I'm curious how did you get to this issue? Have you seen the distributed execution stuck for real, if yes which backend you have used, what collective ops backend ?

[Prathamesh8989]

Hi @vfdev-5, thanks for the quick response!

I came across this while doing a code audit of the synchronization utilities in idist. I haven't encountered the issue during a real distributed training run yet, but the control flow in one_rank_first caught my attention because the barriers are not guarded by a try...finally.

My concern is that if the designated rank raises an exception inside the context block (for example during an I/O operation such as download_dataset() — which is also referenced in the docstring example), that rank would exit the context before reaching the second barrier. In that scenario, the other ranks could potentially remain blocked waiting for synchronization.

I agree that it would be better to confirm this with an actual distributed reproduction rather than relying purely on static analysis.

I can put together a minimal script using the gloo backend to simulate this scenario and check whether the job indeed hangs. If it does reproduce, I’d be happy to propose a fix (likely by guarding the synchronization with a try...finally block).

Would you prefer that I first open a small reproduction example, or should I go ahead and prototype a PR demonstrating the issue and a possible fix?

[vfdev-5]

Try to reproduce the issue with an example first and report it here

[Prathamesh8989]

Hi @vfdev-5,

I've investigated the reproduction. While my local Windows environment has limitations with the gloo backend (encountering makeDeviceForHostname errors), I was able to verify the deadlock logic by auditing the execution flow with a mock-based trace.

The issue is that if the target rank raises an exception within the context, it exits before reaching the second barrier. This leaves all other ranks (which are waiting at their first barrier) deadlocked.

Logic Reproduction (Mock Trace)

This script demonstrates that the exit synchronization point is bypassed when an exception occurs:

import ignite.distributed as idist
from unittest.mock import MagicMock, patch

def test_one_rank_first_deadlock():
    # Mock Rank 0 and the barrier
    idist.get_rank = MagicMock(return_value=0)
    mock_barrier = MagicMock()
    
    with patch('ignite.distributed.utils.barrier', mock_barrier):
        try:
            with idist.one_rank_first(rank=0):
                raise RuntimeError("Rank 0 crashed!")
        except RuntimeError:
            pass

    print(f"Number of exit barriers reached: {mock_barrier.call_count}")
    # Current result is 0, proving the deadlock.

if __name__ == "__main__":
    test_one_rank_first_deadlock()

[vfdev-5]

@Prathamesh8989 this is a bit like speculations.

I tested on with gloo and nccl backends on two devices and it does deadlock only with nccl and terminates with an exception for gloo:

import time
import ignite.distributed as idist

def pprint(*args, **kwargs):
    rank = idist.get_rank()
    time.sleep(rank * 0.1)
    print(f"Rank {rank}:", end=" ")
    print(*args, **kwargs)


def run(local_rank):
    pprint("START")
    with idist.one_rank_first(rank=0):
        raise RuntimeError("Rank 0 crashed!")
    pprint("END")


with idist.Parallel("gloo") as parallel:
    parallel.run(run)

torchrun --nproc_per_node=2 test_deadlock.py

You can also try to run distributed setting with gloo on https://colab.research.google.com/github/pytorch-ignite/playground/blob/main/index.ipynb

import time
import ignite.distributed as idist


def pprint(*args, **kwargs):
    rank = idist.get_rank()
    time.sleep(rank * 0.1)
    print(f"Rank {rank}:", end=" ")
    print(*args, **kwargs)


def run(local_rank):

    pprint("START")

    with idist.one_rank_first(rank=0):
        raise RuntimeError("Rank 0 crashed!")    

    pprint("END")    
    
idist.spawn("gloo", run, (), nproc_per_node=2)    

and it does terminate with error:

Rank 0: START
W0317 21:54:15.245000 293 torch/multiprocessing/spawn.py:165] Terminating process 1448 via signal SIGTERM
---------------------------------------------------------------------------
ProcessRaisedException                    Traceback (most recent call last)
[/tmp/ipykernel_293/677840733.py](https://localhost:8080/#) in <cell line: 0>()
     22     pprint("END")
     23 
---> 24 idist.spawn("gloo", run, (), nproc_per_node=2)

3 frames
[/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py](https://localhost:8080/#) in join(self, timeout, grace_period)
    209         msg = f"\n\n-- Process {error_index:d} terminated with the following error:\n"
    210         msg += original_trace
--> 211         raise ProcessRaisedException(msg, error_index, failed_process.pid)
    212 
    213 

ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 87, in _wrap
    fn(i, *args)
  File "/usr/local/lib/python3.12/dist-packages/ignite/distributed/comp_models/native.py", line 341, in _dist_worker_task_fn
    fn(local_rank, *args, **kw_dict)
  File "/tmp/ipykernel_293/677840733.py", line 20, in run
    raise RuntimeError("Rank 0 crashed!")
RuntimeError: Rank 0 crashed!

[Prathamesh8989]

Thanks for testing this with both backends — that clarification helps a lot.

From your results it seems the behavior differs by backend: with gloo the job terminates with the raised exception, while with nccl it appears to deadlock. So the issue may not be purely a control-flow problem in one_rank_first, but rather how the synchronization interacts with certain backends when one rank raises an exception.

My initial concern came from noticing that the exit barrier is not guarded by a try...finally, which means the designated rank may exit the context before reaching the second synchronization point if an exception occurs. I wondered whether enforcing symmetric barriers could help, although I realize that with NCCL a collective can still hang if a rank terminates unexpectedly.

Would you recommend:

• trying to make one_rank_first more exception-safe (e.g., enforcing symmetric barriers), or
• documenting that exceptions inside the context may lead to hangs with certain backends like NCCL?

Happy to experiment and submit a PR if there’s a preferred direction.

[vfdev-5]

We can experiment more on that. I do not know what can be done here from ignite side.
I would just keep this issue open if someone comes here with another report in the real use-case.

[pinjarlakeshavkumaryadav5-lab]

Can we use "else statement" and terminate the error by which it will not crash

[Prathamesh8989]

Thanks for the suggestion!

Using an else statement alone would not fully solve the problem in this case. The main issue arises from how distributed synchronization primitives (like barrier()) behave.

In one_rank_first, all ranks must reach the same synchronization points. If the designated rank raises an exception inside the context block, it exits before reaching the second barrier(). Meanwhile, the other ranks are still waiting for synchronization, which can lead to a hang depending on the backend (as observed with NCCL).

Because of this, simply adding an else block would not guarantee that all ranks reach the same collective operation. The underlying problem is ensuring symmetric synchronization even when an exception occurs, which is difficult since distributed collectives require participation from all processes.

At the moment, this behavior seems to depend on the backend implementation (Gloo vs NCCL), so documenting the limitation or handling failures at the application level might be the safer approach unless a backend-safe workaround is identified.