The Runbook You Didn't Write Is Already Running in Production
By Tatiana Mikhaleva · Developer Advocate · Docker Captain · IBM Champion
Your vendors now ship opinions about how their products should be operated. The opinions are executable, and almost nobody is reading them.
Somewhere in your stack, a procedure now runs against production that nobody on your team wrote, reviewed, or can see. It came from a vendor, packaged as a skill for an AI agent, and it encodes their view of how their product should be operated: which signals to check, what counts as broken, what to do about it. It arrived with an install command. It did not arrive with a pull request.
You can watch this happening in the open. Open Anthropic’s plugin directory and scroll. Not the community catalog. The official one, the list Anthropic curates itself. More than 250 entries at the time of writing.
AWS is there, with roughly a dozen plugins, most of them authored by Amazon Web Services directly. Azure is there. GitLab, covering repositories, merge requests and CI/CD pipelines. HashiCorp’s Terraform. Oracle. Vercel, managing deployments, builds, logs and domains. Netlify, covering functions, edge functions, blobs, database and image CDN. Render ships skills, an agent, slash commands and a validation hook for render.yaml. Railway covers project setup, deploys, networking, monitoring — and troubleshooting. PagerDuty scores your pre-commit diffs against historical incident data. SonarQube enforces itself inside the agent coding loop.
Pulumi is not in that catalog, and the absence turns out to be more interesting than the presence would have been. Pulumi ships its skills anyway: through its own plugin marketplace, and through a cross-platform CLI that follows the open Agent Skills specification, reaching Claude Code, Cursor, Copilot, VS Code, Codex and Gemini CLI. We will come back to Pulumi, because Pulumi is where the shape of this gets clearest.
Let me be precise about what is new here, because “AI does ops now” is not a new headline, and until recently it mostly meant autocomplete for YAML: a faster way to write the config a human still reviewed. That is not this.
This piece is an analysis of what a dozen vendors shipped in the space of a few months, and of what none of them addressed. Every technical claim below comes from a vendor’s own documentation, their own published writing, or the public record. The argument does not need me to be right about anything. It needs them to have been quiet about the same six questions. They were.
What changed, in plain terms
Here is how it used to work.
A company bought a tool: a database, a cloud service, a CI system. The engineers read the vendor’s documentation. Then they wrote their own runbook, the procedure for what to check, in what order, when that tool misbehaved. That runbook lived in the repository. A colleague reviewed it. It was version-controlled, argued over, and owned by the team.
Here is how it works now.
The vendors (AWS, GitLab, Vercel, Pulumi and the rest) ship ready-made plugins and “skills” for AI agents like Claude or Copilot. You describe the problem in a chat. The agent reads it, decides how to diagnose it, and applies the fix to the running system, following logic the vendor wrote, not you.
The runbook used to be something you wrote. Now it arrives from the vendor, pre-written, and it executes. That is the whole shift, and everything below follows from it.
Access is not the same as a procedure
This distinction matters more than anything else in the piece, so let me borrow the clearest statement of it I have found. It comes, tellingly, from a vendor’s own blog rather than from a critic.
A post on Pulumi’s blog draws the line: MCP servers give Claude access to things; skills teach Claude how to think about things. Different jobs.
An MCP server is a door. It says: here are my endpoints, here is what you may call. Your team still decides what to do with them.
A skill is a procedure. It says: here is how you diagnose me, in this order, checking these signals, escalating at this threshold.
The same Pulumi post is explicit about why they ship them: generic AI assistance produces code that looks reasonable but breaks the conventions the community learned the hard way, so the skills teach Claude those conventions.
Read that twice, with an operator’s ear. The vendor is not shipping you tools. They are shipping you their opinion, in a form that runs. They said so themselves, in their own marketing, as a feature. And they are right that it is a feature.
What this replaces
Think about what the old runbook actually was.
You worked out what mattered in your environment, which is never quite what mattered in theirs. You wrote it down. It went through code review. When it was wrong in the middle of the night, you fixed it, and the fix was yours.
That runbook was an artifact of your team’s understanding. It was, in a real sense, the thing your platform team was.
Now consider a vendor-shipped skill called something like /troubleshoot.
It encodes a definition of what troubleshooting that product means. Which signals to check. In which order. What counts as normal. What counts as a problem worth escalating. Someone decided all of that.
Not you.
And vendors deciding is not the new part. Vendors have always had opinions, and they published them as docs, where a senior engineer read the advice and discarded the two-thirds that did not apply to their environment. That adaptation step has been removed from the loop. The opinion is no longer a document you read. It is a procedure that runs.
The honest case for it
I am not going to write the skeptic’s column. The skeptic’s column is easy and mostly wrong, and if I skip this section the rest of the piece is worthless.
The vendor’s runbook is better than yours. At least on day one. They know their own failure modes. They know which metric actually predicts trouble and which three are decorative. Your first runbook is a guess assembled from their docs and a Stack Overflow thread. Theirs is the accumulated support load of every customer who ever paged them.
Onboarding collapses. The gap between “we bought this” and “we can operate this in the middle of the night” has historically been measured in weeks of a senior engineer’s time.
It travels. The failure mode this fixes is the one the ecosystem calls tribal knowledge. One engineer works out the smart diagnostic sequence, and it lives in their head and a Slack thread of partial screenshots. New hires inherit an empty config directory. A packaged skill set is that knowledge, versioned and installable.
It is simply a better interface. Natural language is not magic; remembering which of forty subcommands and which flag combination was never a good use of human memory.
All of that is real. Which is exactly why the next part matters.
Where it breaks
Six places. None has a clean answer yet, and I have not seen a vendor address any of them in a launch post.
1. The agent fixes production, and GitOps deletes the fix
Start with the one that is mechanical rather than philosophical, because it is the one that bites first.
If your platform runs on GitOps (and for a large share of teams now, it does), the repository is the source of truth. Not the cluster. The cluster is a projection of the repository, reconciled continuously.
Now put an agent in that picture. At night, a vendor’s /troubleshoot skill works out what is wrong and edits live state to fix it: the replica count, the image tag, the config of a resource Git declares. Not a restart the deployment would undo on its own; a change to something the repository has an opinion about.
Self-heal is not on by default; changes made to the live cluster do not trigger a sync unless you opt in. But with selfHeal: true, drift between Git and the cluster is simply corrected. The controller does not know or care that a model made the change with good intentions. It sees exactly what it sees when a panicking engineer runs kubectl against prod: live state that disagrees with declared state.
So the agent fixes production, and the platform reverts the fix, because the fix was never in Git. How quickly depends on your configuration. That it happens does not. This is a failure mode you assemble rather than a default: it needs selfHeal: true, an agent with write access, and the freedom to act unsupervised. Three switches you turn on yourself. The point is that nothing in the tooling pushes back when you line them up, and plenty of teams will.
There is a sharper edge too, straight from Argo CD’s own documentation: rollback cannot be performed against an application with automated sync enabled. The escape hatch is not sealed (you disable auto-sync, roll back, re-enable), but it now has an extra step, and you are taking it under time pressure, in the middle of the night, on a system you did not just watch change.
But the revert is not the worst part. The agent’s fix is not just undone. It is unrecorded. It never became a commit. No diff, no review, no history. The one artifact the whole organization trusts has no idea any of it happened. Your source of truth is now, precisely, not the truth.
2. The catalog disagrees with itself
The same tension is visible one level up, in who ships what.
This is the contradiction Pulumi embodies most cleanly. Its whole thesis is that infrastructure is code: versioned, reviewed, living in a repository. And it now ships agent skills that operate that same infrastructure conversationally, distributed to six platforms as something it plainly considers core, not a side integration.
But the problem is not Pulumi’s alone. Everyone owns it, and the cleanest evidence sits in the catalog I opened at the top of this piece.
Two entries, a few lines apart in the same file. aws-core, author field Amazon Web Services, offers skills to author infrastructure as code. aws-agents-for-devsecops, author field Amazon Web Services, investigates incidents and runs penetration tests.
One of those exists because the repository should decide. The other exists because, in the middle of an incident, something faster than a human should. I have not found a page on either that mentions the other.
To be fair to AWS: their DevOps Agent investigates and recommends, and a human still applies the remediation. That is the responsible design, and most vendor agents today are read-heavy for exactly this reason. The mutation risk does not live in the vendor’s own incident agent. It lives one layer over, in a coding agent that has your kubectl credentials and a vendor’s skill telling it what “fix this” means. And it lives in the trajectory, because every one of these products is moving in the same direction, and none of the vendors shipping the procedures is moving toward asking permission more often.
And one vendor stayed out. Terraform, now an IBM product, taught a generation that the repository is the source of truth, and its catalog entry is an MCP server: a door, not a procedure. The one whose identity was built on “the repository decides” is the one that shipped no runbook. Read it as caution or as lag; either way it is the exception worth watching.
The pattern is uniform. The same vendors selling “the repository decides” are selling “the model decides, live.” The controller is not broken; Argo CD reverting an out-of-band change is doing its job. That is the point. Both product lines work exactly as designed, and the designs disagree about who may change production. Nobody has reconciled them, and most teams will find out during an incident rather than a design review.
3. The audit log records the action, not the intent
Consider what the log looks like when a deploy was initiated by a model responding to the sentence “the staging cluster looks unhappy, sort it out.”
The line reads: deployment updated, timestamp, actor. It is accurate. It is also nearly useless in a postmortem, because the thing you need to reconstruct is not what happened but why the system believed that was the right thing to do. The reasoning is not in the log. In most setups it is not anywhere. It evaporated with the session.
The better incident agents do capture this. AWS’s DevOps Agent, for one, keeps an investigation journal with its reasoning. But that discipline lives in the specialized agents, not in the coding agent holding your kubectl credentials, and it is the coding agent that mutates state. We spent fifteen years building the discipline that every production change should be traceable to a reviewed, intentional decision. The audit trail was the enforcement mechanism. It still records changes perfectly. In the path that matters most, it has quietly stopped recording decisions.
4. You are running someone else’s runbook without reviewing the diff
This is the one that should worry you most, and it is structural rather than hypothetical.
Your runbook went through code review. That was the entire point: a second engineer read it before it could page anyone.
The vendor’s skills ship with the plugin. When the plugin updates, the procedures update. Ask yourself honestly: are you reviewing that diff? Is anyone on your team reading the changelog for the diagnostic logic that will run against your production system?
The mechanics are not subtle. If your plugin reference tracks a branch or a moving commit, upstream changes arrive in your engineers’ sessions unannounced. That is not a criticism of anyone’s design; it is how git works. But it means the vendor’s operational logic is an unversioned dependency sitting in your production path, and the update lands without a pull request.
There is a second problem underneath, and it is worse. A skill is a markdown file that the agent reads as instruction, not as data. Whatever is in it enters the model’s context with the authority of a procedure. If that file is ever modified (upstream, in a dependency, by anyone with commit access to a repository you do not watch), the modification is not a config change you can diff in review. It is an instruction the agent will follow. The security literature has been shouting about prompt injection through tool descriptions and skill content for years. The industry response has been to ship more skills.
The same literature has a name for the passive version of this risk: a rugpull, where a component that behaved one way when you vetted it behaves differently later. And it points at the mental-model problem underneath. Users experience this as connecting their agent to a service, when what they are actually doing is running someone else’s code with their own credentials.
Anthropic clearly sees this. Their plugin system will not let a plugin-shipped agent bring its own hooks, MCP servers or permission mode, documented as a deliberate security decision. Plugins can ship disabled by default, specifically for cases that add scope a user should consciously opt into. Their marketplace pins approved plugins to specific commits rather than tracking a branch.
Those are good primitives. But they are primitives on the client side. They say nothing about whether your organization reviews your vendor’s operational logic before it runs. Right now almost nobody does, because the review step never existed. There was never a vendor runbook to review.
5. Responsibility has no owner
An agent deployed the wrong thing in the middle of the night.
You typed a sentence. A model interpreted it. A vendor wrote the skill the model invoked. A second vendor’s platform executed it.
Where does the postmortem land? That question is not philosophy. The incident review will ask it first, and there is currently no convention for answering it. “The agent did it” is not an answer any organization accepts twice.
The uncomfortable version: the engineer who typed the sentence will absorb it, because they are the only human in the chain. That is not a stable arrangement. It is how you lose senior people.
6. If the runbook self-executes, what is the on-call rotation for?
If a vendor’s skill investigates on its own, do you still need someone awake?
Most teams will answer yes, of course, and then let the rotation atrophy, because the pager fires less often and the incentive to staff it decays.
The failure case is subtler than the agent missing something. The agent handles the routine 90% competently, the humans lose their calibration on the system because they have not touched it in four months, and then the 10% arrives, and the person who picks up the page has never operated the thing unassisted.
We have seen this film, and it is worth being precise about the ending, because the ending is not reassuring.
This is an old problem wearing new clothes, and it is worth saying so plainly rather than presenting it as a discovery. In 1983, the cognitive psychologist Lisanne Bainbridge named it the irony of automation: automate most of a process but not all of it, and you have not removed the operator. You have handed them a job they can no longer do, because the skills to take over are exactly the ones the automation stopped them practicing.
Aviation has spent the decades since failing to solve it, with far more pressure than we have. An FAA human factors report picked it up in 1996. In 2009, Air France 447 fell into the Atlantic after the autopilot handed control back to a crew short on hand-flying practice. Forty years, a regulator, mandatory training regimes, an industry where this failure mode kills people in visible numbers. Still unresolved. It is common enough by now to have its own review literature under titles like Ironies of Automation: Still Unresolved After All These Years.
I raise this as calibration, not warning. If you are assuming platform engineering will work this out in a couple of release cycles, look at what it took aviation to not work it out. We have no regulator, no mandatory training, no shared incident database, and our failure mode costs a quarter’s revenue rather than 228 lives, which means less pressure to solve it, not more.
What to ask before you turn it on
The advice circulating right now is uniform and correct as far as it goes: start read-only, scope write access carefully, use dedicated service accounts rather than your own credentials, never hand an agent write access to a production database. Take that advice. I have written about the enforcement half of it before: the ceiling that makes an agent physically unable to cross a line, and the six-phase framework for taking agents to production at all.
But that advice addresses the access problem, and access is not the interesting problem here. Here is what I would ask a vendor before enabling their skills against anything that matters.
-
What is your update policy for these skills? Do they follow the plugin version, or can I pin them independently? If I cannot pin them, your operational logic is an unversioned dependency in my production path.
-
Do procedure changes appear in your release notes? The files are readable; they are markdown in a repository. The question is whether you treat a change to diagnostic logic as a documented release or as a silent commit. If your changelog does not mention skills, that is my answer.
-
What does the audit record capture when the actor is a model? Specifically: does it capture the prompt, the tool sequence and the reasoning, or just the resulting state change?
-
Which of these commands mutate? Not “which are dangerous”. Which write. Give me the list, and give me a way to disable the mutating half without disabling the diagnostics.
-
Can the agent roll back its own mutations? And the harder half: does it know when it should? A rollback is a judgment, not a capability; something has to decide the intervention made things worse. If the answer is that a human steps in, be clear about what you are asking of that human. They are arriving at an undocumented state that a model produced and did not explain, under time pressure, at whatever hour this happens.
-
What is your position when your skill causes an outage? Not for the lawyers. I want to know whether you have thought about it at all.
If a vendor has good answers to all six, they have thought about this harder than most. I have not yet found one who has published answers to any. Not in a launch post, not in the docs. The conversation is happening elsewhere: lawyers are writing about the liability gap, insurers are excluding agent-related losses, regulators are naming autonomy and auditability as risks. The people shipping the procedures are not part of it yet.
The runbook used to be the artifact of a team’s understanding. It was slow to write, and it was ours.
The new one arrives working, on day one, better than the one we would have written. That is a real gift, and I am not being sarcastic.
But we should be clear about the trade. We are accepting a vendor’s opinion about how their product should be operated, and we are accepting it in a form we do not read.
That is not automatically wrong. Every dependency is a decision to trust someone else’s judgment, and we make that decision a hundred times a day without agonizing. The difference is that code dependencies come with twenty years of convention: lockfiles, review, CVE feeds, SBOMs.
For operational dependencies, we have a marketplace listing and an install command.
Go read the skills your vendors shipped you this quarter. They are all markdown. It is a short read, and it is your production.