Runbook automation with Ansible AWX

Which runbooks are worth automating

Not every runbook should run by itself. Sort them by two questions: how often does this happen, and how bad is it if the automation acts at the wrong moment?

Start with diagnostics. Capturing a thread dump automatically when latency rises is safe and valuable, because that evidence is often gone by the time a person logs in. The Java and .NET guides list what to collect. It also builds trust in the pipeline before anything can change state.

Before automating a remediation, ask whether the problem should exist at all. A nightly restart job for a memory leak is acceptable as a stopgap. It turns into a problem once nobody remembers it is there, and the leak keeps growing.

Write the playbook with guardrails

Take a common runbook step: restart a Deployment and confirm it recovers. A naive version just patches the annotation. The version below refuses out-of-scope targets and checks the result. It uses modules from the kubernetes.core collection, which your execution environment has to include.

---
- name: Rolling restart with verification
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    allowed_namespaces: [payments, orders]
    rollout_retries: 30
    rollout_delay_seconds: 10
  tasks:
    - name: Refuse targets outside the allow list
      ansible.builtin.assert:
        that:
          - target_namespace in allowed_namespaces
          - target_deployment is match('^[a-z0-9-]+$')
        fail_msg: "Target is not allowed for this template"

    - name: Trigger a rolling restart
      kubernetes.core.k8s:
        state: patched
        api_version: apps/v1
        kind: Deployment
        namespace: "{{ target_namespace }}"
        name: "{{ target_deployment }}"
        definition:
          spec:
            template:
              metadata:
                annotations:
                  kubectl.kubernetes.io/restartedAt: "{{ now(utc=true).isoformat() }}"

    - name: Wait for the rollout to finish
      kubernetes.core.k8s_info:
        api_version: apps/v1
        kind: Deployment
        namespace: "{{ target_namespace }}"
        name: "{{ target_deployment }}"
      register: deployment
      until: >-
        deployment.resources[0].status.observedGeneration | default(0) == deployment.resources[0].metadata.generation
        and deployment.resources[0].status.updatedReplicas | default(0) == deployment.resources[0].spec.replicas
        and deployment.resources[0].status.availableReplicas | default(0) == deployment.resources[0].spec.replicas
      retries: "{{ rollout_retries }}"
      delay: "{{ rollout_delay_seconds }}"

Some details are worth noting. The annotation is the same one kubectl rollout restart sets, so a restart done by automation looks the same as one done by hand, and a rollout respects your maxUnavailable and PodDisruptionBudgets. The observedGeneration check closes a race: right after the patch, the status can still describe the old generation and look complete.

If the wait fails, the job fails, and that failure should reach a human. A remediation whose failure goes unnoticed leaves everyone assuming the problem was handled.

For Linux hosts, the same structure uses ansible.builtin.systemd_service for the action and ansible.builtin.uri to confirm the health endpoint afterwards. Use --check when writing a playbook, though remember that check mode cannot predict every module's result.

Properties to require of every remediation playbook

• Idempotent or safely repeatable. The trigger may fire twice. A second run should not do additional harm.
• Validated input. Variables arriving from an alert are untrusted text. Check them against an allow list, as above.
• Pre-checks. Stop if the state is not what the runbook assumes, such as fewer healthy replicas than the rollout needs.
• Verification. Finish by testing the outcome, not just the command's exit code.
• Small blast radius. Act on one target per run. Handle many targets with a separate, reviewed process.

Set up the AWX job template

In AWX, a job template joins a playbook from a project (a Git repository), an inventory, credentials and an execution environment. For the example above:

1. Create a project pointing to the Git repository that holds your playbooks. Require reviews on that repository, since anything merged there can run in production.
2. Create an inventory. For a local-connection playbook it can be just localhost.
3. Create a credential of a suitable type, such as an OpenShift or Kubernetes API bearer token. Its service account should have a narrow Role in only the permitted namespaces.
4. Create the job template, and enable "Prompt on launch" for variables so the caller can pass the target. Without this setting, AWX ignores extra_vars sent at launch.
5. Add a survey if people will run it by hand too, with fixed choices for namespace.
6. Turn off concurrent jobs for this template, so a burst of triggers cannot overlap.

Give the Kubernetes service account only what the playbook uses:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: remediation-restart
  namespace: payments
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "patch"]

Launch it from the API

Launching a job is a POST to the template's launch endpoint, authenticated with an OAuth2 token:

curl -sS -X POST \
  -H "Authorization: Bearer $AWX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"extra_vars": {"target_namespace": "payments", "target_deployment": "checkout"}}' \
  https://awx.internal/api/v2/job_templates/42/launch/

The response includes the job's id. Poll /api/v2/jobs/<id>/ until status reaches successful, failed, error or canceled. Whatever triggers the launch should keep this result and attach it to the incident, so responders can see what ran.

Create a dedicated AWX user for the integration, with only the Execute role on specific templates. Then a stolen token can launch your restart job with allow-listed targets, but it can't edit playbooks or run arbitrary commands.

AWX's job template webhooks are aimed at source control events, so most alert-driven integrations call the launch API directly or go through a small relay that maps alerts to templates. A relay is also a good place to hold rate limits and validation.

Approval gates

For medium-risk actions, use a workflow job template with an approval node. The workflow pauses until a person with the Approve role accepts or denies it, and you can set a timeout on the node. A typical sequence:

1. Collect diagnostics automatically
2. Pause for approval, showing what will run and on what
3. Perform the action
4. Verify, and branch on failure to a job that notifies the team

Choose a timeout that fits the urgency. If an approval expires, the action must not proceed, and the incident must stay visibly unhandled.

Stopping automation from causing an incident

The classic automation failure is a loop. A pod is unhealthy because its database is down, so automation restarts it, which helps nothing, so it fires again, and every restart adds load to a database that's trying to recover. Protect against this at several levels:

Count your remediations and review them. A rising trend means an underlying fault is getting worse behind the automation. Each automated run is a piece of incident data, so it belongs on the same timeline as your change correlation view.

Audit and access

Regulated environments will want to know who or what changed production, on what grounds, and who approved it. AWX keeps job history, with the launching user, variables, and output. Three things to get right:

• Mark sensitive variables. Output and extra_vars are stored in job history. Use no_log: true on tasks that handle secrets, and keep credentials in the AWX credential store, not in variables.
• Link the trigger. Pass an incident or alert ID in extra_vars so each job can be traced back to its cause.
• Retain history as your policy requires. AWX has cleanup jobs that delete old job records, so check their schedule against your retention rules.

AWX is the upstream open source project, and its release cadence and support differ from a commercially supported automation platform. Check the project's current status before building critical processes on it.

Connecting alerts to AWX in Wrentrix

Wrentrix alerting and remediation can run actions through webhooks or Ansible AWX job templates, either automatically or with approval. Rate limits and allow lists in your playbooks and templates should remain in place whichever system triggers them. Those controls belong next to the action.

Frequently asked questions

What is the difference between Ansible and AWX?

Ansible is the automation engine that runs playbooks, usually from a command line. AWX is a web application and REST API on top of it, adding job templates, credentials, role-based access, scheduling, workflows and job history. For alert-driven runbooks, the API and access controls are what make AWX useful.

How do I trigger an AWX job template from an alert?

Send a POST to /api/v2/job_templates/<id>/launch/ with a bearer token and optional extra_vars. The template must have "Prompt on launch" enabled for variables, or AWX will ignore them. Treat all values from the alert as untrusted, and validate them inside the playbook.

Is automated remediation safe in production?

It can be, provided that the action is low risk, tightly scoped, rate limited and verified afterwards. Start with diagnostics, then move on to narrow actions such as restarting a single stateless replica. Keep approvals in place for rollbacks and anything touching data, and always have a global off switch.

How do you add an approval step to an AWX job?

Create a workflow job template and put an approval node before the node that makes the change. The workflow pauses until a user with approval rights accepts or denies it, or the node's timeout passes. Make sure that a timeout leaves the incident visibly open rather than silently closed.

What should stay manual in runbook automation?

Decisions with a high cost of error should stay with people: database failover, data deletion, and anything with an uncertain blast radius. You can still automate the mechanical steps once somebody has made the call. The aim is fewer manual keystrokes, not removing judgement from the riskiest moments.