Kubernetes admission webhooks: how they work and how they fail

Where webhooks sit in the API request

A request to the Kubernetes API server passes through authentication and authorization first. It then enters admission: mutating webhooks run and can change the object, the API server validates the object against its schema, and validating webhooks run and can only accept or reject it. The object is written to etcd only after all of that succeeds.

Three properties follow from that order, and they explain most webhook behaviour you will see in production:

• Mutating webhooks see each other's output. The order is not something you should rely on, which is why reinvocationPolicy: IfNeeded exists: it lets a webhook run again if a later webhook changed the object.
• Validating webhooks see the final object, so they are the right place for policy that must hold regardless of what mutation happened.
• Admission happens once, at write time. A webhook that injects an init container into pods has no effect on pods that already exist, because pod specs are largely immutable after creation. New behaviour arrives when the pod is next created.

That last point matters for instrumentation tools. A webhook that adds a tracing agent to Java pods does nothing to running workloads. The agent appears the next time the workload restarts for its own reasons, or when you choose to run a rollout.

A complete configuration, field by field

Here is a mutating webhook for pods. The names are examples. The fields are from the admissionregistration.k8s.io/v1 API.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: example-pod-injector
webhooks:
  - name: inject.example.internal
    admissionReviewVersions: ["v1"]
    sideEffects: None
    failurePolicy: Ignore
    timeoutSeconds: 3
    reinvocationPolicy: IfNeeded
    matchPolicy: Equivalent
    clientConfig:
      service:
        namespace: example-system
        name: pod-injector
        path: /mutate
        port: 443
      caBundle: <base64 PEM of the CA that signed the serving certificate>
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
        scope: Namespaced
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values: ["kube-system", "example-system"]
    objectSelector:
      matchExpressions:
        - key: example.internal/skip-injection
          operator: DoesNotExist

The fields that deserve a deliberate decision:

What the webhook actually returns

The API server sends an AdmissionReview containing a request with a uid and the object. The webhook returns an AdmissionReview with a response. The uid must be echoed back. A mutation is a base64-encoded JSON Patch:

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "<uid from the request>",
    "allowed": true,
    "patchType": "JSONPatch",
    "patch": "<base64 of the JSON Patch array>"
  }
}

Decoded, the patch might be [{"op":"add","path":"/spec/initContainers/-","value":{...}}]. A frequent bug hides there: JSON Patch add with a /- path fails if the initContainers array does not exist, and a pod with no init containers has no such array. Handle both cases, or the webhook will work on your test pod and fail on half of production.

Keep the handler deterministic and idempotent. Because of reinvocation and retries, it can see an object it has already mutated. Check for your init container before adding it, so you do not inject two.

failurePolicy: choose by what the webhook protects

The fail-open or fail-closed decision is really a question about which failure you would rather have.

**Fail closed (Fail)** is right when the webhook enforces something that must never be bypassed: a security policy, an image signature requirement, a tenancy rule. The cost is that a webhook outage turns into a pod creation outage for everything it matches.

**Fail open (Ignore)** is right when the webhook adds value but the workload is fine without it: tracing injection, sidecars that are not needed for correctness, default labels. The cost is that pods created during an outage silently lack the addition, and you need a way to find them.

A good test is to ask whether you would accept a payment service failing to scale during a traffic peak because this webhook was down. For an observability agent the answer is plainly no. For a control that auditors depend on, it might be yes, and then you should staff and monitor the webhook like the critical dependency it has become.

The failure modes worth rehearsing

The webhook blocks its own pods

If a fail-closed webhook matches pods in its own namespace and its replicas all die, new replicas cannot be admitted because the webhook that would admit them is down. The cluster has a deadlock that lasts until someone deletes or edits the configuration. Exclude the webhook's namespace with namespaceSelector, as in the example above. Kubernetes sets the kubernetes.io/metadata.name label on every namespace automatically, so you can match on it without labelling anything yourself.

System components get caught

A webhook matching all pods also matches CNI agents, DNS and control plane add-ons. During a node failure or a cluster upgrade, these components are the ones being recreated, and they are the worst possible things to block. Exclude system namespaces. On OpenShift this includes the openshift-* namespaces, so an allow list of labelled namespaces is often easier to maintain than a deny list.

Latency adds up invisibly

A rolling update of a 200-replica deployment means 200 pod creates. With a webhook taking 3 seconds, that is a noticeable slowdown. The real problem is usually subtler: the ReplicaSet controller creates pods, so any delay or rejection appears on the ReplicaSet, not on the Deployment you are watching.

kubectl describe replicaset -n payments payments-api-7c9d8f6b5
kubectl get events -n payments --field-selector reason=FailedCreate

If a deployment seems stuck with no new pods and no errors on it, look for FailedCreate events mentioning failed calling webhook. Those words identify the webhook by name.

Certificate expiry

The API server calls webhooks over TLS and verifies the certificate against caBundle. When that certificate expires, every call fails and failurePolicy decides the outcome. Automate this:

• On OpenShift, the service CA does the job: annotate the Service with service.beta.openshift.io/serving-cert-secret-name to get a serving certificate, and annotate the webhook configuration with service.beta.openshift.io/inject-cabundle: "true".
• On other clusters, a certificate controller can issue the serving certificate and keep caBundle in sync, or the webhook's own installer can generate a long-lived CA and rotate the serving certificate under it.

Whichever you use, track expiry as you would any other certificate. Our guide on TLS certificate expiry monitoring covers internal certificates that public scanners never see.

Network policy changes

The call comes from the API server, not from a pod. A default-deny NetworkPolicy in the webhook's namespace can break it, and the source address varies by distribution and network plugin. Test this after any network policy rollout, since the symptom looks like a generic webhook timeout.

Monitoring a webhook

The API server exposes metrics for every webhook, labelled by name. The ones to chart:

• apiserver_admission_webhook_admission_duration_seconds is a histogram of call latency. Alert on the p99 approaching your timeoutSeconds.
• apiserver_admission_webhook_rejection_count counts rejections, including those caused by errors and timeouts, with labels that separate the reasons.
• apiserver_admission_webhook_fail_open_count counts requests admitted because a fail-open webhook failed. Anything other than zero means pods were created without your mutation.

The fail-open counter is the one people overlook. With Ignore, an outage is silent by design. This metric is often the only record of it, so give it an alert that opens a ticket.

Also monitor from the webhook's side: request rate, handler latency, replica count. Run at least two replicas across different nodes with a PodDisruptionBudget, so a node drain cannot remove them all at once.

Do you still need a webhook?

Since Kubernetes 1.30, ValidatingAdmissionPolicy is generally available. It runs CEL expressions inside the API server, with no network call and no service to operate. If your webhook only checks fields, such as requiring a label or forbidding hostNetwork, a policy is simpler and cannot suffer from a network timeout.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-team-label
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments"]
  validations:
    - expression: "has(object.metadata.labels) && 'team' in object.metadata.labels"
      message: "Deployments must carry a team label"

A policy takes effect only with a matching ValidatingAdmissionPolicyBinding. Mutating admission policies are also arriving upstream, so check the feature state for your Kubernetes version before planning around them. Webhooks remain the right tool when the decision needs external data or complex logic, or when the mutation depends on the contents of the image or the language runtime. The OpenTelemetry operator is a well-known example of that second kind, using a pod webhook to add language agents based on annotations.

How Wrentrix uses a webhook

Wrentrix adds zero-code method-level tracing for Java and .NET through an admission webhook. The design follows the reasoning above: the webhook is fail-open, so if the agent is unavailable pods start without instrumentation. Its overhead is under 25 ms at p95 per pod start. Wrentrix never restarts workloads, so tracing begins on a workload's next restart, on your schedule. The platform runs under the default restricted-v2 SCC on OpenShift, with no root and no extra SCCs.

Frequently asked questions

What is the difference between a mutating and a validating admission webhook?

A mutating webhook can change the object by returning a JSON Patch, and it runs before schema validation. A validating webhook can only allow or deny the request, and it runs after mutation and schema validation, so it sees the final object. Use validating webhooks for policy that must hold after all mutations are complete.

Should failurePolicy be Fail or Ignore?

Use Ignore when workloads can run correctly without the webhook, such as observability injection. Use Fail when the webhook enforces a control that must never be skipped, and treat it as critical infrastructure with multiple replicas and alerts. Note that Fail is the default in the v1 API, so an omitted field means fail closed.

Why is my deployment stuck with no new pods and no error?

Pods are created by the ReplicaSet controller, so admission errors appear as FailedCreate events on the ReplicaSet rather than on the Deployment. Run kubectl describe replicaset on the newest ReplicaSet and look for failed calling webhook. The message names the webhook, which you can then inspect or temporarily relax.

Do admission webhooks affect pods that are already running?

No. Admission runs when an object is written, so a new webhook only affects pods created afterwards. To apply it to an existing workload, the pods have to be recreated, for example by a rollout. This is why injection-based tooling takes effect on the next restart.