Human-in-the-loop for DevOps and SRE agents
The on-call engineer is not at the terminal.
Your agent stops before the risky change, and Pushary gets the on-call engineer's answer back fast.
How does an AI DevOps agent get approval from an on-call engineer?
Pushary lets an AI DevOps agent get approval from the on-call engineer on their phone. Put one Pushary call in front of the migration, the restart or the scale-down. The engineer taps Approve or Deny on the lock screen. Your agent waits for the answer. If nobody answers in time, the change does not run. Every answer is saved with who decided and when, ready for the incident review.
What needs a yes
The calls that should not be automatic
- runMigration(name)
- A migration you cannot cleanly reverse is the classic action worth waking someone for. It is also the classic action an agent will happily run at 02:00.
- scaleDown(service, replicas)
- Cutting capacity during an incident you have misdiagnosed turns a slowdown into an outage, and everyone feels it.
- rotateSecret(name)
- Rotating a credential fixes one problem and can break every service that cached the old value. That takes judgment, and a threshold cannot supply it.
- deleteResource(arn)
- Cleanup agents are useful right up to the moment one deletes something that looked idle and was load bearing.
- promoteToProduction(sha)
- Shipping is the step where your users pay for a mistake, instead of a test suite catching it.
The code
One call inside the step you already have
Wrap the action, not the whole agent. Everything the agent does that is safe keeps running without asking anyone.
pip install pusharyimport os
from pushary import PusharyServer
px = PusharyServer(api_key=os.environ["PUSHARY_API_KEY"])
def remediate(incident, plan):
# Read-only diagnosis and reversible fixes run unattended.
if plan.reversible and plan.blast_radius == "single-pod":
return apply(plan)
decision = px.decisions.ask(
external_id=incident.on_call_id,
type="confirm",
question=f"{plan.summary}\nBlast radius: {plan.blast_radius}. Reversible: {plan.reversible}.",
)
if not decision["approved"]:
return page_secondary(incident, reason="human declined automated remediation")
return apply(plan)Who answers
The approver is not at a desk
The approver is whoever is on call, and being on call means you are not at a terminal. Most remediation agents post the request in a chat channel. That works if the engineer is in the channel with notifications on. The rest of the time it fails quietly: the change is correctly blocked and the incident keeps burning. Pushary reaches the on-call engineer's phone, by the id your rota uses.
We already post approvals to Slack. What is different?
Slack is a good place to see a request and a bad place for a run to depend on. The message arrives whether or not the right person is looking. Your code gets no answer it can wait on. Nothing stops the change if nobody replies. So you end up building the missing half yourself: a poller, a timeout, state that survives a deploy in the middle of an incident, and a record of who clicked what for the review. Pushary is that half, as one call that returns an answer. If your team lives in Slack, the request can still go there. Either way, every answer is saved in one place instead of scattered across channel history.
Worth reading: Google's SRE book on release engineering and change management.
Add it to your product. Your users never make a Pushary account or pay us.
FAQ
Questions, answered
Does the wait survive a deploy in the middle of an incident?
Can approval route to a rota rather than one person?
How do we keep from paging someone for every trivial change?
What do we hand an auditor afterwards?
Put a person on the step that needs one.
Connect a user once, ask with one call, and run the action only on a yes. Every answer is saved.
Start free