Deadlocks in auth to repositories

[glightfoot]

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

buildkitd can become effectively wedged when a client disconnects unexpectedly during registry auth callback flow (notably session/auth.VerifyTokenAuthority during 401 Unauthorized handling).

Observed behavior

• A build triggers registry auth callback.
• The client disconnects ungracefully (silent network drop, half-open TCP, abrupt termination).
• The callback path can block indefinitely waiting for session/auth response.
• That blocked request holds resolver/auth synchronization long enough to cause:
  • lock contention in authorizer/resolver paths
  • request pileup in flightcontrol waiters for image resolution
• The daemon often still appears healthy (CPU/memory), but builds requiring registry resolution stop progressing until buildkitd restart.

Expected behavior

• Dead/disconnected client sessions should be detected and fail in bounded time.
• Auth callback failures should not allow indefinite lock hold.
• Unrelated builds/auth flows should continue to make progress even when one auth path is slow or dead.

Related fixes in progress

• PR #6630: timeout/keepalive safeguards to prevent indefinite deadlock behavior.
• PR #6631: authorizer/session-manager concurrency refactors to reduce lock contention and blast radius.

Goroutine dump evidence

Snippet 1: Root cause (1 waiter in auth callback/session wait path)

1 @ ...
#	0x482fd8	sync.runtime_notifyListWait+0x138								/usr/local/go/src/runtime/sema.go:606
#	0x493e52	sync.(*Cond).Wait+0x72										/usr/local/go/src/sync/cond.go:71
#	0xabee2b	github.com/moby/buildkit/session.(*Manager).Get+0x1cb						/src/session/manager.go:179
#	0xabbfc4	github.com/moby/buildkit/session.(*Manager).Any+0x244						/src/session/group.go:78
#	0x12bd7f8	github.com/moby/buildkit/session/auth.VerifyTokenAuthority+0xb8					/src/session/auth/auth.go:73
#	0x12c9f64	github.com/moby/buildkit/util/resolver.(*authHandlerNS).get+0x1a4				/src/util/resolver/authorizer.go:74
#	0x12ca74f	github.com/moby/buildkit/util/resolver.(*dockerAuthorizer).Authorize+0xef			/src/util/resolver/authorizer.go:129
#	0xc7e70e	github.com/containerd/containerd/v2/core/remotes/docker.(*request).authorize+0x2e		/src/vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go:546
#	0xc7f2de	github.com/containerd/containerd/v2/core/remotes/docker.(*request).do+0x2de			/src/vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go:621
#	0xc7fd2e	github.com/containerd/containerd/v2/core/remotes/docker.(*request).doWithRetriesInner+0x4e	/src/vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go:717
#	0xc7fae8	github.com/containerd/containerd/v2/core/remotes/docker.(*request).doWithRetries+0x68		/src/vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go:698
#	0xc7c464	github.com/containerd/containerd/v2/core/remotes/docker.(*dockerResolver).Resolve+0xb24		/src/vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go:308
#	0x12cfdc8	github.com/moby/buildkit/util/resolver.(*Resolver).Resolve+0x208				/src/util/resolver/pool.go:250
#	0xcc5028	github.com/moby/buildkit/util/imageutil.Config+0x508						/src/util/imageutil/config.go:107
#	0x1aeee70	github.com/moby/buildkit/source/containerimage.(*Source).ResolveImageMetadata.func2+0x90	/src/source/containerimage/source.go:185
#	0xdf142c	github.com/moby/buildkit/util/flightcontrol.(*call[...]).run+0x14c				/src/util/flightcontrol/flightcontrol.go:122

Snippet 2: Lock contention (at least 14 waiters total across resolver/authorizer locks)

12 @ ...
#	0x482ca4	internal/sync.runtime_SemacquireMutex+0x24								/usr/local/go/src/runtime/sema.go:95
#	0x493b7c	internal/sync.(*Mutex).lockSlow+0x15c									/usr/local/go/src/internal/sync/mutex.go:149
#	0x12ce70a	github.com/moby/buildkit/util/resolver.(*Pool).GetResolver+0x2ea					/src/util/resolver/pool.go:103
#	0x1aed574	github.com/moby/buildkit/source/containerimage.(*Source).ResolveImageMetadata+0x2b4			/src/source/containerimage/source.go:179
1 @ ...
#	0x482ca4	internal/sync.runtime_SemacquireMutex+0x24							/usr/local/go/src/runtime/sema.go:95
#	0x493b7c	internal/sync.(*Mutex).lockSlow+0x15c								/usr/local/go/src/internal/sync/mutex.go:149
#	0x12ca6cd	sync.(*Mutex).Lock+0x6d										/usr/local/go/src/sync/mutex.go:46
#	0x12ca6af	github.com/moby/buildkit/util/resolver.(*dockerAuthorizer).Authorize+0x4f			/src/util/resolver/authorizer.go:125
#	0xc7e70e	github.com/containerd/containerd/v2/core/remotes/docker.(*request).authorize+0x2e		/src/vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go:546
1 @ ...
#	0x482ca4	internal/sync.runtime_SemacquireMutex+0x24							/usr/local/go/src/runtime/sema.go:95
#	0x493b7c	internal/sync.(*Mutex).lockSlow+0x15c								/usr/local/go/src/internal/sync/mutex.go:149
#	0x12cadaf	sync.(*Mutex).Lock+0x8f										/usr/local/go/src/sync/mutex.go:46
#	0x12cad94	github.com/moby/buildkit/util/resolver.(*dockerAuthorizer).AddResponses+0x74			/src/util/resolver/authorizer.go:150

Snippet 3: Flightcontrol pileup (19 waiters observed)

10 @ ...
#	0xdf0f97	github.com/moby/buildkit/util/flightcontrol.(*call[...]).wait+0x4b7	/src/util/flightcontrol/flightcontrol.go:168
#	0xdf0033	github.com/moby/buildkit/util/flightcontrol.(*Group[...]).do+0x233	/src/util/flightcontrol/flightcontrol.go:79
#	0xdf04d2	github.com/moby/buildkit/util/flightcontrol.(*Group[...]).Do+0x92	/src/util/flightcontrol/flightcontrol.go:37
#	0x179670d	github.com/moby/buildkit/util/flightcontrol.(*CachedGroup[...]).Do+0xcd	/src/util/flightcontrol/cached.go:30
#	0x19853c7	github.com/moby/buildkit/frontend/dockerfile/builder.(*withResolveCache).ResolveImageConfig+0x1e7	/src/frontend/dockerfile/builder/resolvecache.go:36
5 @ ...
#	0xdf105a	github.com/moby/buildkit/util/flightcontrol.(*call[...]).wait+0x57a	/src/util/flightcontrol/flightcontrol.go:173
#	0xdf0033	github.com/moby/buildkit/util/flightcontrol.(*Group[...]).do+0x233	/src/util/flightcontrol/flightcontrol.go:79
#	0xdf04d2	github.com/moby/buildkit/util/flightcontrol.(*Group[...]).Do+0x92	/src/util/flightcontrol/flightcontrol.go:37
#	0x179670d	github.com/moby/buildkit/util/flightcontrol.(*CachedGroup[...]).Do+0xcd	/src/util/flightcontrol/cached.go:30
#	0x19853c7	github.com/moby/buildkit/frontend/dockerfile/builder.(*withResolveCache).ResolveImageConfig+0x1e7	/src/frontend/dockerfile/builder/resolvecache.go:36
2 @ ...
#	0xdf0f97	github.com/moby/buildkit/util/flightcontrol.(*call[...]).wait+0x4b7	/src/util/flightcontrol/flightcontrol.go:168
#	0xdf0033	github.com/moby/buildkit/util/flightcontrol.(*Group[...]).do+0x233	/src/util/flightcontrol/flightcontrol.go:79
#	0xdf04d2	github.com/moby/buildkit/util/flightcontrol.(*Group[...]).Do+0x92	/src/util/flightcontrol/flightcontrol.go:37
#	0x1aed7ad	github.com/moby/buildkit/source/containerimage.(*Source).ResolveImageMetadata+0x4ed	/src/source/containerimage/source.go:184
1 @ ...
#	0xdf0f97	github.com/moby/buildkit/util/flightcontrol.(*call[...]).wait+0x4b7	/src/util/flightcontrol/flightcontrol.go:168
#	0xdeff19	github.com/moby/buildkit/util/flightcontrol.(*Group[...]).do+0x119	/src/util/flightcontrol/flightcontrol.go:65
#	0xdf04d2	github.com/moby/buildkit/util/flightcontrol.(*Group[...]).Do+0x92	/src/util/flightcontrol/flightcontrol.go:37
1 @ ...
#	0xdf0f97	github.com/moby/buildkit/util/flightcontrol.(*call[...]).wait+0x4b7	/src/util/flightcontrol/flightcontrol.go:168
#	0xdeff19	github.com/moby/buildkit/util/flightcontrol.(*Group[...]).do+0x119	/src/util/flightcontrol/flightcontrol.go:65
#	0xdf04d2	github.com/moby/buildkit/util/flightcontrol.(*Group[...]).Do+0x92	/src/util/flightcont

Reproduction

1. Start a buildkitd instance with remote client/session-based auth callback behavior.
2. Trigger a build that requires pulling a private image (or any registry flow that returns 401 and requires callback credential resolution).
3. During the auth callback window, drop the client connection ungracefully (for example, kill client process, blackhole network traffic, or simulate half-open TCP).
4. Start additional builds that resolve the same image (and/or other registry-backed builds).
5. Observe builds blocking indefinitely and goroutine accumulation in auth/resolver/flightcontrol paths.

While auth is in-flight, terminate or disconnect the client ungracefully.
Then retry additional builds and observe wedged behavior.

Version information

• buildkitd --version
  0.28.1-rootless
• buildctl --version
  0.20.2
• docker buildx version (if applicable)
  v0.31.1-desktop.1

[crazy-max]

I took a closer look at the current state of your issue and the failure seems plausible.

Looking at your PRs:

• resolver/auth: prevent deadlocks on disconnected clients and narrow authorizer lock scope [1/3] #6630
• Refactor session manager to use simpler concurrency control [2/3] #6631
• Handle instances where runc cannot be killed because it is not running [3/3] #6632

I don't think this is the right upstream shape.

There are a few concrete places in the current code where a slow or disconnected client can block unrelated work. In

buildkit/session/manager.go

Lines 45 to 90 in 8543ce4

	func (sm *Manager) HandleHTTPRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) error {
	hijacker, ok := w.(http.Hijacker)
	if !ok {
	return errors.New("handler does not support hijack")
	}
	id := r.Header.Get(headerSessionID)
	proto := r.Header.Get("Upgrade")
	sm.mu.Lock()
	if _, ok := sm.sessions[id]; ok {
	sm.mu.Unlock()
	return errors.Errorf("session %s already exists", id)
	}
	if proto == "" {
	sm.mu.Unlock()
	return errors.New("no upgrade proto in request")
	}
	if proto != "h2c" {
	sm.mu.Unlock()
	return errors.Errorf("protocol %s not supported", proto)
	}
	conn, _, err := hijacker.Hijack()
	if err != nil {
	sm.mu.Unlock()
	return errors.Wrap(err, "failed to hijack connection")
	}
	resp := &http.Response{
	StatusCode: http.StatusSwitchingProtocols,
	ProtoMajor: 1,
	ProtoMinor: 1,
	Header: http.Header{},
	}
	resp.Header.Set("Connection", "Upgrade")
	resp.Header.Set("Upgrade", proto)
	// set raw mode
	conn.Write([]byte{})
	resp.Write(conn)
	return sm.handleConn(ctx, conn, r.Header)

HandleHTTPRequest holds sm.mu across hijack and upgrade response writes.

And in

buildkit/util/resolver/authorizer.go

Lines 123 to 223 in 8543ce4

	// Authorize handles auth request.
	func (a *dockerAuthorizer) Authorize(ctx context.Context, req *http.Request) error {
	a.handlerNS.muHandlers.Lock()
	defer a.handlerNS.muHandlers.Unlock()
	// skip if there is no auth handler
	ah := a.handlerNS.get(ctx, req.URL.Host, a.sm, a.session)
	if ah == nil {
	return nil
	}
	authHeader, err := ah.authorize(ctx, a.sm, a.session)
	if err != nil {
	return err
	}
	req.Header.Set("Authorization", authHeader)
	return nil
	}
	func (a *dockerAuthorizer) getCredentials(host string) (sessionID, username, secret string, err error) {
	return sessionauth.CredentialsFunc(a.sm, a.session)(host)
	}
	func (a *dockerAuthorizer) AddResponses(ctx context.Context, responses []*http.Response) error {
	handlerNS := a.handlerNS
	handlerNS.muHandlers.Lock()
	defer handlerNS.muHandlers.Unlock()
	last := responses[len(responses)-1]
	host := last.Request.URL.Host
	handler := handlerNS.get(ctx, host, a.sm, a.session)
	for _, c := range auth.ParseAuthHeader(last.Header) {
	switch c.Scheme {
	case auth.BearerAuth:
	var oldScopes []string
	if err := invalidAuthorization(c, responses); err != nil {
	handlerNS.delete(handler)
	if handler != nil {
	oldScopes = handler.common.Scopes
	}
	handler = nil
	// this hacky way seems to be the best method to detect that error is fatal and should not be retried with a new token
	if c.Parameters["error"] == "insufficient_scope" && parseScopes(oldScopes).contains(parseScopes(strings.Split(c.Parameters["scope"], " "))) {
	return err
	}
	}
	// reuse existing handler
	//
	// assume that one registry will return the common
	// challenge information, including realm and service.
	// and the resource scope is only different part
	// which can be provided by each request.
	if handler != nil {
	return nil
	}
	var username, secret string
	sessionID, pubKey, err := sessionauth.GetTokenAuthority(ctx, host, a.sm, a.session)
	if err != nil {
	return err
	}
	if pubKey == nil {
	sessionID, username, secret, err = a.getCredentials(host)
	if err != nil {
	return err
	}
	}
	common, err := auth.GenerateTokenOptions(ctx, host, username, secret, c)
	if err != nil {
	return err
	}
	common.Scopes = parseScopes(append(common.Scopes, oldScopes...)).normalize()
	handlerNS.set(host, sessionID, newAuthFetcher(host, a.client, c.Scheme, pubKey, common))
	return nil
	case auth.BasicAuth:
	sessionID, username, secret, err := a.getCredentials(host)
	if err != nil {
	return err
	}
	if username != "" && secret != "" {
	handlerNS.set(host, sessionID, newAuthFetcher(host, a.client, c.Scheme, nil, auth.TokenOptions{
	Username: username,
	Secret: secret,
	}))
	return nil
	}
	}
	}
	return errors.Wrap(cerrdefs.ErrNotImplemented, "failed to find supported auth scheme")
	}

Authorize and AddResponses hold the global authorizer mutex while doing session callbacks and potentially slow auth work.

Then in

buildkit/session/auth/auth.go

Lines 28 to 52 in 8543ce4

	func CredentialsFunc(sm *session.Manager, g session.Group) func(string) (session, username, secret string, err error) {
	return func(host string) (string, string, string, error) {
	var sessionID, user, secret string
	err := sm.Any(context.TODO(), g, func(ctx context.Context, id string, c session.Caller) error {
	client := NewAuthClient(c.Conn())
	resp, err := client.Credentials(ctx, &CredentialsRequest{
	Host: host,
	})
	if err != nil {
	if grpcerrors.Code(err) == codes.Unimplemented {
	return nil
	}
	return err
	}
	sessionID = id
	user = resp.Username
	secret = resp.Secret
	return nil
	})
	if err != nil {
	return "", "", "", err
	}
	return sessionID, user, secret, nil
	}

CredentialsFunc uses context.TODO(), which makes that callback path effectively uncancellable. Those are credible issues for exactly the kind of wedge described in your issue.

But I don't think the answer is to take the current PR series as-is. It mixes the reported bug with broad concurrency refactors in a very sensitive area, which makes review much harder than it needs to be. I agree with @tonistiigi comment here that the most useful next step is a minimized reproducer and failing coverage. From there, the upstream fix should be a narrow patch against the current architecture, likely around making the credential path cancellable and removing lock-plus-I/O behavior from the session and authorizer paths.

Also looked at #6638 and I don't think it fully satisfies the request for a runnable repro.

It adds useful targeted tests around suspected lock and callback behavior, so it's not without value, but these are still unit-style concurrency tests built around blocking test doubles. They demonstrate that some of the current code paths can block in undesirable ways. They don't yet reproduce the reported issue end to end on upstream BuildKit, namely a client disconnect during the registry auth callback flow followed by subsequent registry resolution or builds becoming wedged.

What I think would be more useful here is a minimized integration-style repro on current master. For example, something that brings up buildkitd, exercises a real auth callback path against a test registry or auth stub, forces the client to disappear at a controlled point during that exchange, and then shows that later work against the same registry gets stuck or otherwise fails in the reported way. That would make it much easier to validate both the bug and any narrow upstream fix.

So my view is that #6638 is useful supporting coverage, but it's not the same thing as the runnable reproducer that was being asked for.

[glightfoot]

Thanks for taking a look @crazy-max. I am not sure exactly which issue we were encountering, which is why we didn't start with the failing tests (we only had the goroutine dump showing where it was locked up). I had cursor write some integration tests that are failing and pushed them to #6638

[tonistiigi]

The client disconnects ungracefully (silent network drop, half-open TCP, abrupt termination).

@glightfoot Are you seeing this issue because the low-level network connection drops for session or because the credential request specifically does not receive answer. For example the latter could be caused by credential-helper being stuck. There is no timeout at for the credential call and we could add one. If it is issue with network connection not propagating then I don't understand why you would be seeing this specifically with the credential call and not with all the other session services.

[glightfoot]

@tonistiigi unfortunately we don't have any evidence of exactly what's happening, only that we get wedged and the goroutine dumps when the builder is stuck. It's likely either an issue with a slow/dropped connection from the builder to the image repository, or from the client to the builder. We always see it manifest the same way though, where a build on a stuck builder node hangs forever on the first FROM image ... line

[tonistiigi]

@glightfoot If you get a deadlock can you capture a stacktrace from the client as well (just send SIGQUIT), to see if it may be stuck because the client is stuck on responding to your auth request. And you don't use any non-standard credential-helper setup?