# Coordinator leader-election shutdown race (druid-stg)

## Symptom

Every time the Coordinator StatefulSet is rolling-updated on `druid-stg`:

1. Indexing task logs temporarily disappear from the Druid console (they remain intact in deep storage / S3 the whole time).
2. All Kafka supervisors restart their consumer sessions.

## Triggering log line

```
2026-09-30T20:55:31,627 ERROR [LeaderElectorAsyncWrapper] org.apache.druid.k8s.discovery.K8sDruidLeaderSelector - listener.stopBeingLeader() failed. Unable to stopBeingLeader: {exception
Type=java.util.concurrent.RejectedExecutionException, exceptionMessage=Task org.apache.druid.server.coordinator.loading.HttpLoadQueuePeon$$Lambda$6516/0x00007f448529a970@5641f1f0 rejected
from java.util.concurrent.ThreadPoolExecutor@3a4a007[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 10817], class=org.apache.druid.k8s.discovery.K8sDruidLeaderSelector}
```

## Root cause

`K8sDruidLeaderSelector` (from `druid-kubernetes-extensions`) dispatches its "stop being leader" callbacks through a small internal `ExecutorService`. On SIGTERM, Druid's `Lifecycle` framework stops registered components in a fixed order, and that executor is shut down *before* the leadership-relinquish callback chain finishes submitting work to it. The result is a `RejectedExecutionException` instead of a clean handoff.

This only manifests on the pod that currently holds Coordinator leadership — non-leader replicas terminate without issue, since they never registered the leadership callbacks in the first place. That also explains why it's specific to Coordinator restarts.

It has real effect here because this cluster runs Coordinator and Overlord combined in one process (`druid.coordinator.asOverlord.enabled=true`). The callback that fails (`HttpLoadQueuePeon`'s lambda) is part of the Overlord-role shutdown path, which owns `SupervisorManager` and the in-memory view of running peon tasks/logs. Because that shutdown gets cut off mid-way:

- The new leader has to cold-start `SupervisorManager` and rebuild supervisor/task state from the metadata store rather than receiving a clean handoff — hence every Kafka supervisor restarting its consumer session.
- The task-log UI briefly loses track of in-flight tasks while that state rebuilds from metadata + deep storage. Nothing is actually lost; the log files were never touched.

## Evidence trail

- Confirmed via `helm diff` + staging cluster inspection (see prior investigation) that the Coordinator is rolling-updated whenever common config changes — including config changes (e.g. request-logging) that have nothing to do with indexing/supervisors.
- `druid.coordinator.asOverlord.enabled=true` and `druid.coordinator.asOverlord.overlordService=druid/overlord` are set in the Coordinator's `runtime.properties` (`druid/helm/druid-cluster/templates/druid.yaml`), confirming Coordinator and Overlord share one process/leader-election context — there is no separate `overlord` nodeSpec.
- The failure is a JVM-internal lifecycle-ordering race (executor torn down before its own pending callback), not a Kubernetes scheduling or timing issue — it fires almost immediately on SIGTERM, well inside the existing 300s `terminationGracePeriodSeconds`, so grace-period tuning does not address it.

## Why this is a recurring risk, not a one-off

Any future Coordinator rollout — image bump, resource change, unrelated common-config edit — will trigger this same leadership handoff and the same supervisor/task-log disruption, as long as Overlord duties are bundled into the Coordinator process.

## Candidate fixes (not yet applied)

1. **Split Overlord into its own dedicated node type** (recommended durable fix): set `druid.coordinator.asOverlord.enabled=false` on the Coordinator nodeSpec and add a separate `overlord` nodeSpec with its own StatefulSet/leader election. After this, Coordinator-only rollouts never touch `SupervisorManager` or task-log state; only a deliberate Overlord restart would.
2. Check upstream Apache Druid release notes for a fix to `K8sDruidLeaderSelector`'s executor-shutdown ordering — this looks like a lifecycle-ordering bug rather than intended behavior, so a version upgrade may resolve it outright.
3. Interim mitigation only: impact is disruptive but not destructive — Kafka supervisors resume from committed offsets and no task logs are actually lost — so no immediate data-loss risk while a permanent fix is scheduled.

No changes have been made to the cluster or Helm values for this issue yet.