# Rancher Error: `failed to wait for roles to be populated`

**Version:** 2.14.2  
**Trigger:** Helm chart installation via Rancher UI  
**Log entry:** `[ERROR] Unknown error: failed to wait for roles to be populated`

---

## Root Cause

**File:** `pkg/catalogv2/helmop/operation.go` → `createNamespace()`

When installing a chart into a namespace with a `projectID`, Rancher starts a **30-second Kubernetes Watch** waiting for the `InitialRolesPopulated: True` condition in the `cattle.io/status` annotation on the namespace.

The watch is started **without `ResourceVersion: "0"`**, meaning it only receives **future** change events. If the namespace already exists and the condition was **already set** in the past, the watch **misses it entirely** and times out → error.

```go
// pkg/catalogv2/helmop/operation.go
w, err := adminClient.CoreV1().Namespaces().Watch(ctx, metav1.ListOptions{
    FieldSelector:  "metadata.name=" + namespace,
    TimeoutSeconds: &thirty,   // hardcoded 30 seconds
})
// ...
// If condition already set before watch started → never received → timeout:
return nil, fmt.Errorf("failed to wait for roles to be populated")
```

---

## Conditions Being Watched

The annotation that must be present on the namespace:

```yaml
metadata:
  annotations:
    cattle.io/status: |
      {"Conditions":[
        {"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"..."},
        {"Type":"ResourceQuotaValidated","Status":"True","Message":"","LastUpdateTime":"..."},
        {"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"..."}
      ]}
```

---

## Diagnosis

```bash
# Check if the condition is already set on the target namespace:
kubectl get namespace <your-namespace> \
  -o jsonpath='{.metadata.annotations.cattle\.io/status}'
```

```bash
# Find ALL namespaces missing InitialRolesPopulated:
kubectl get namespaces -o json | python3 -c "import json,sys; ns=json.load(sys.stdin); missing=[n['metadata']['name'] for n in ns['items'] if 'InitialRolesPopulated' not in n['metadata'].get('annotations',{}).get('cattle.io/status','')]; print('Missing:', missing) if missing else print('All namespaces OK')"
```

---

## Fix / Workarounds

### 1. Patch a single namespace

If the condition is missing or the namespace is old:

```bash
kubectl annotate namespace <your-namespace> \
  'cattle.io/status={"Conditions":[
    {"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2024-01-01T00:00:00Z"},
    {"Type":"ResourceQuotaValidated","Status":"True","Message":"","LastUpdateTime":"2024-01-01T00:00:00Z"},
    {"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2024-01-01T00:00:00Z"}
  ]}' --overwrite
```

Then retry the chart install in Rancher UI.

### 2. Patch all affected namespaces at once

If many namespaces are missing the condition (systemic authz controller issue):

```bash
for ns in $(kubectl get namespaces -o json | python3 -c "import json,sys; ns=json.load(sys.stdin); [print(n['metadata']['name']) for n in ns['items'] if 'InitialRolesPopulated' not in n['metadata'].get('annotations',{}).get('cattle.io/status','')]"); do
  echo "Patching $ns..."
  kubectl annotate namespace $ns \
    "cattle.io/status={\"Conditions\":[{\"Type\":\"ResourceQuotaInit\",\"Status\":\"True\",\"Message\":\"\",\"LastUpdateTime\":\"2026-06-30T13:00:00Z\"},{\"Type\":\"ResourceQuotaValidated\",\"Status\":\"True\",\"Message\":\"\",\"LastUpdateTime\":\"2026-06-30T13:00:00Z\"},{\"Type\":\"InitialRolesPopulated\",\"Status\":\"True\",\"Message\":\"\",\"LastUpdateTime\":\"2026-06-30T13:00:00Z\"}]}" \
    --overwrite
done
```

Verify afterwards:
```bash
kubectl get namespaces -o json | python3 -c "import json,sys; ns=json.load(sys.stdin); missing=[n['metadata']['name'] for n in ns['items'] if 'InitialRolesPopulated' not in n['metadata'].get('annotations',{}).get('cattle.io/status','')]; print('Missing:', missing) if missing else print('All namespaces OK')"
```

> ⚠️ If many system namespaces are affected, also investigate why the authz controller never ran:
> ```bash
> kubectl logs -n cattle-system deploy/rancher --tail=100 | grep -i "InitialRoles\|authz\|ERROR" | head -30
> ```

### 2. Other scenarios

| Cause | Fix |
|---|---|
| Authz controller slow (busy/large cluster) | **Retry the install** — 2nd attempt usually works after the controller catches up |
| Old namespace missing condition | Patch annotation (above) **or** delete & recreate the namespace |
| `rancher-webhook` unreachable (Private GKE / Windows RKE2) | Check `rancher-webhook` pod health in `cattle-system`; verify firewall allows kube-apiserver → port 9443 |
| No project required | Install chart with **no project assigned** — skips the watch entirely |

---

## Related GitHub Issues

| Issue | Version | Status | Notes |
|---|---|---|---|
| [#33554](https://github.com/rancher/rancher/issues/33554) | 2.5.8 | Closed (stale) | Original report (2021) |
| [#37951](https://github.com/rancher/rancher/issues/37951) | 2.6.5 | Closed | Monitoring app install |
| [#38674](https://github.com/rancher/rancher/issues/38674) | 2.6-head | **Open** | Windows RKE2 specific |
| [#40920](https://github.com/rancher/rancher/issues/40920) | 2.7.2-RC6 | Closed (v2.7.2) | Gatekeeper webhook cause |
| [#41605](https://github.com/rancher/rancher/issues/41605) | 2.7.6 | Closed (v2.9.2) | Most relevant; JIRA SURE-6365 |
| [#41142](https://github.com/rancher/rancher/issues/41142) | 2.7.2 | Closed (v2.7.7) | Private GKE + webhook firewall |

---

## Status in v2.14.2

The underlying **watch race condition is still present** in the codebase as of v2.14.2. No targeted fix has been merged for the v2.14.x branch. The workarounds above remain the recommended approach.