An alert is a request for someone's attention
Before any tooling, test each alert with these questions. If you can't answer them, the alert is not ready, whichever system sends it.
- Does this need a person at all?
- Does it need that person now, or during working hours?
- What would they do first? Is that written down?
- How will they know when it's over?
The Wrentrix alert wizard walks you through creating an alert, with channels for delivery and runbooks attached. Tooling helps, but the most common cause of noise is alerting on causes and not on symptoms. "CPU above 85%" tells you about a machine. "Checkout error budget is burning 14.4 times too fast" tells you about a customer. The alert fatigue guide goes deeper on pruning, and burn-rate alerts explains the alert type we would start with for paging.
Severity by the action it demands
A simple table often does more for an on-call rotation than any automation:
| Level | Meaning | Delivery | Example |
|---|---|---|---|
| Page | Harm now, human action needed | Wakes someone | Fast burn-rate pair, front door probe failing |
| Ticket | Will become harm if ignored | Work queue, next working day | Slow burn, TLS certificate 30 days from expiry |
| Context | Useful during an investigation | No notification, visible on the problem | New log pattern, method outside baseline |
| Remove | Nobody acts on it | Nothing | The alert everybody mutes |
The bottom row is part of the design. If an alert was ignored ten times running, then it is either wrong or useless, and leaving it in place trains people to ignore the next one.
Writing a runbook someone can use at 03:00
Attach a runbook to any alert that can page. A good one is short, and written for a tired engineer who hasn't seen this service in months:
# Orders API: fast error budget burn
What this means: over 1% of order requests have been failing.
Check first:
1. Is there a grouped problem for this service? Read its top candidate.
2. Was anything deployed or changed in the last hour?
3. Is the database reachable from the orders namespace?
If a recent deployment is the suspect:
oc rollout undo deployment/orders-api -n prod
Watch the alert. It should resolve within about 5 minutes.
If the connection pool is full:
See the connection pool section below.
Escalate to: orders on-call, then database on-call.Keep the commands exact, and keep each runbook where the alert links to it, since a wiki page three clicks away isn't read during an incident. Each time a runbook is used, fix what was wrong in it.
AI-generated alert summaries help in a related way. They describe what fired and what surrounds it in plain language for the person just getting the notification. They are generated by your local model and can be turned off along with the rest of AI. Treat them as a quick orientation. The runbook is the authority on what to do.
From runbook to automation
Once a runbook step is performed the same way each time, it's a candidate for automation. Wrentrix can trigger remediation through webhooks or Ansible AWX job templates, either automatically or with approval.
Ansible AWX job templates
This route suits organisations that already have automation controlled through AWX, because credentials, inventory, permissions and job history stay there. Wrentrix doesn't need your SSH keys. It requests a job, and AWX enforces who and what is allowed.
The underlying AWX API call is a POST to the job template's launch endpoint. This is how you'd test a template by hand before connecting anything to it:
curl -s -X POST \
-H "Authorization: Bearer $AWX_TOKEN" \
-H "Content-Type: application/json" \
-d '{"extra_vars": {"target_deployment": "orders-api", "target_namespace": "prod"}}' \
https://awx.example.internal/api/v2/job_templates/42/launch/
The template needs "prompt on launch" enabled for variables
(ask_variables_on_launch), or AWX ignores what you pass in
extra_vars. Create a dedicated token for this integration, with execute
permission only on the templates that are intended for remediation, and not a token with
broad rights. The AWX runbook automation
guide walks through template design, surveys and limits.
Validate variables inside the playbook itself. Whatever arrives from an alert should be treated as input, and a playbook should refuse targets that it wasn't written for:
- name: Restart one allowed deployment
hosts: localhost
gather_facts: false
vars:
allowed_deployments: [orders-api, payments-api]
tasks:
- name: Refuse targets outside the allow list
ansible.builtin.assert:
that: target_deployment in allowed_deployments
fail_msg: "Target not allowed for automated restart"Webhooks
Webhooks fit everything else: an internal ticketing system, a chat bridge, your own automation service. For your own receiver, verify the sender, make the handler idempotent, because retries can happen, and log every request. Details of Wrentrix's request format are best looked at against your actual installation, so ask us if you need them while planning.
Automatic, or with approval?
This is the real design decision. A way to reason about it:
| Question | Leans automatic | Leans approval |
|---|---|---|
| Can the action be undone? | Fully and quickly | Partly, or not at all |
| How large is the blast radius? | One pod, one instance | A whole service or shared system |
| Could it hide a real problem? | Rarely | Often |
| Have you run it by hand many times? | Yes, and it always works | Not much |
| Is data affected? | No | Yes |
| Is this a change under your change process? | Pre-approved class | Needs a record |
Examples of actions that often suit automatic mode: clearing a known temp directory when a disk fills up, or scaling a stateless service within a fixed range. Actions that nearly always deserve approval: database failover, anything which modifies data, and anything spanning multiple services.
Restarting a pod is a common example that looks safe and may not be. It can fix a leak temporarily, but if the restart loop continues, it hides a defect, and it may also add load to systems that are struggling. In the connection-pool scenario, restarting pods makes each new instance open connections all at once. So build the guardrails first:
- A rate limit. Do not repeat the same action on the same target more than a set number of times in a window. After that, stop and page a human.
- A verification step. After the action, check that the alert is cleared. The self-resolving SLO alerts make that check natural.
- A global stop. A way to disable all automation during a major incident or a freeze window.
- Change windows. Check whether automated changes in your environment need to respect freeze periods.
- An audit trail. Wrentrix's audit log records every change, and AWX keeps its own job history. Regulated teams will want to see that those two line up. See security and access control.
AI-suggested remediation needs extra care
If a language model takes part in choosing an action, remember that its inputs include logs and other text from outside. That text can include instructions someone planted. So give the automation a fixed menu of pre-written, allow-listed actions with validated parameters. Never let generated text become a command. With approval mode, the person reviewing sees what's proposed and on what evidence, and the root-cause evidence chain is what they should check before saying yes.
A sensible order of rollout
- Clean up the alerts. Automating around noise only automates noise.
- Write runbooks for what pages.
- Attach automation in approval mode, even for actions you expect to end up automatic. Approving a few dozen correct suggestions is how trust is earned.
- Move individual actions to automatic mode once the record supports it, starting with the most reversible.
- Review monthly. Which actions fired? Did they work? Should the underlying fault just be fixed? An automated action that fires every night is a bug report that nobody has read.
If your organisation requires human approval for every production change, remediation still has value: approval mode prepares the action and leaves the decision to a person.
Frequently asked questions
Can Wrentrix fix incidents automatically?
It can trigger remediation through webhooks or Ansible AWX job templates, either automatically or after approval. You decide which actions run in each mode. We would start in approval mode and only move reversible, well-understood actions to automatic.
How does Wrentrix integrate with Ansible AWX?
Remediation can launch AWX job templates, so inventory, credentials and job history stay in AWX. Use a dedicated token limited to remediation templates, and validate variables inside the playbooks. AWX keeps its own record of each job alongside the Wrentrix audit log.
How do I reduce alert noise?
Start by paging only on symptoms users feel, such as fast SLO burn, and move cause-based alerts to tickets or to context. Remove any alert that nobody has acted on recently. Multi-window burn-rate alerts and grouping by topology into a single problem also cut duplicate notifications.
Is automated remediation safe in production?
It can be, if actions are reversible, narrowly scoped, rate limited and verified afterwards. Risk grows when a tool is able to restart things repeatedly or hide a real defect. Keep anything touching data or several services in approval mode.
What are AI-generated alert summaries?
They are short plain-language descriptions of what fired and what surrounds it, written by the model you host locally. They're intended to orient the person who receives the notification, and they don't replace a runbook. You can turn them off by disabling AI.