Troubleshooting
Start with the client's conditions and events:
kubectl -n <ns> get iamc <name> \
-o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
kubectl -n <ns> describe iamc <name> # Events mirror the audit records
kubectl -n charter-system logs deploy/charter-iam | grep '"reconcile failed"'
Conditions
| Condition |
Reason |
Meaning |
What to do |
Ready=True |
Reconciled |
Entra matches the approved spec |
Nothing |
Ready=False |
PolicyViolation |
A policy rule failed; the message lists all of them |
Fix the spec |
Ready=False |
PendingApproval |
The approval gate failed |
See below |
Ready=False |
PushSecretFailed |
Entra is fine, but the push to OpenBao failed |
See OpenBao |
SecretPushed=False |
Pending |
Waiting for External Secrets to push |
Wait up to PUSH_SECRET_REFRESH_INTERVAL |
PendingApproval messages
| Message |
Cause |
missing requested-by / missing approved-by / missing change-ref |
An approval annotation is absent or empty |
separation of duties: approver is the requester |
Same person in both (case-insensitive) |
admin-only fields changed without a new change-ref |
An admin-only field changed but change-ref equals status.changeRef |
PolicyViolation messages
| Message |
Cause |
redirect URI must use https: ... |
http:// or another scheme |
redirect host X is not in the allowed domains |
Host not under ALLOWED_REDIRECT_DOMAINS |
wildcard redirect URIs are not allowed: ... |
A * in the URI |
loopback redirect URIs are disabled by policy: ... |
localhost / 127.0.0.1 without ALLOW_LOCALHOST_REDIRECTS=true |
not an absolute URI: ... |
Missing scheme |
at least one human owner is required for accountability |
spec.owners is empty |
secretRotation must look like 90d or 720h, got ... |
Bad format, below 1 hour or above 730 days |
secretRotation Nd exceeds the policy maximum of Md |
Above MAX_SECRET_LIFETIME_DAYS |
dex issuer: ... |
dex.issuerURL breaks a redirect rule |
pushSecret: ClusterSecretStore X is not in the allowed push stores |
Store not in ALLOWED_PUSH_SECRET_STORES (or that list is empty) |
pushSecret: pushSecret has no clusterSecretStore and the controller has no default |
pushSecret: {} with no DEFAULT_PUSH_SECRET_STORE |
Replication lag
Microsoft Graph is eventually consistent: for a few seconds after a write, a
read can return the old state, and a write can conflict with the previous one
still being applied. These errors are usually transient; the controller retries
after 30 seconds and converges. Each attempt still leaves a failed audit
record.
| Error |
Typical moment |
403 Authorization_RequestDenied: ... the backing application of the service principal being created must in the local tenant |
Creating the service principal just after creating the app |
config: application missing right after upsert |
Reading the app back just after creating it |
409 Request_MultipleObjectsWithSameKeyValue: The service principal cannot be created ... already in use |
A retry creating a service principal that was in fact created |
400 Request_BadRequest: One or more added object references already exist for the following modified properties: 'owners' |
Re-adding an owner that a stale read did not show |
409 Directory_ConcurrencyViolation |
Two writes to the same app within about a second, for example removing two passwords |
Log line delivered key not visible in Graph yet; waiting for replication |
A reconcile right after a rotation; expected, not an error |
If one of these persists for more than a few minutes, it is not lag.
Throttling
429 Too Many Requests means Graph is throttling the tenant or the
controller's app. The controller retries after 30 seconds. Small or new tenants
throttle sooner; other automation and portal use count against the same limits.
uniqueName already exists
graph 400 Bad Request: Another object with the same value for property uniqueName already exists.
The client's app was deleted with deletionPolicy: Delete within the last 30
days and is still soft-deleted, holding its uniqueName. Either restore it
from App registrations > Deleted applications (Charter adopts it on the
next reconcile) or permanently delete it there.
Controller does not start
| Log |
Cause |
config: AZURE_TENANT_ID not set (or AZURE_CLIENT_ID) |
Workload identity webhook did not inject, or variables missing locally |
config: workload identity required: AZURE_FEDERATED_TOKEN_FILE is not set |
No federated token and client-secret auth not enabled. In-cluster: check the pod label azure.workload.identity/use: "true" and the service account annotation |
graph 401 / AADSTS700016 / AADSTS70021 |
Wrong client ID, or the federated credential's issuer or subject does not match the service account |
Token works but every call is 403 |
Admin consent for Application.ReadWrite.OwnedBy missing |
OpenBao delivery
| Symptom |
Cause |
SecretPushed=False Failed: external-secrets.io/v1alpha1 PushSecret is not installed in this cluster |
External Secrets Operator (or its CRDs) missing |
SecretPushed=False Failed: <message from External Secrets> |
The store rejected the write: check kubectl get clustersecretstore and the store's policy |
SecretPushed=False Pending for a long time |
External Secrets is not running, or PUSH_SECRET_REFRESH_INTERVAL is long. While pending, the previous Entra secret is not revoked |
403 on pushsecrets in the controller log |
The controller's ClusterRole lacks the external-secrets.io rule |
Harmless log lines
tried to reconcile object ... that was not found in local store: a queued
retry for an object that was just deleted.
client secret auth enabled; this is for local development only: expected
with ALLOW_CLIENT_SECRET_AUTH=true, never in production.