When the CAB Becomes the Safety System

TL;DR: Change Advisory Boards were designed to coordinate business risk, not to validate engineering decisions. Yet in many organizations they have gradually become the primary mechanism for protecting production. The result is a dangerous illusion: we replace engineering controls with administrative controls and mistake approval for correctness. The irony becomes obvious during an outage, when the same process that approved the faulty change prevents engineers from restoring service quickly.

For most of my career, I was incredibly lucky. I managed to bypass the Change Advisory Board almost entirely because I worked in environments where trust was the absolute foundation of our engineering culture.

That being said, I still recognize why these boards exist in traditional enterprises. When functioning correctly, their goals are entirely valid:

• They avoid conflicting maintenance windows.
• They ensure business stakeholders understand the impact of upcoming work.
• They force engineers to think through rollback procedures before touching production.

None of those objectives are unreasonable, and every large organization needs some form of governance.

What has always fascinated me, however, is what happens when the approved change is the one that breaks production.

Every experienced infrastructure engineer knows the sequence. A maintenance window is carefully prepared. The implementation plan is reviewed, the rollback procedure is documented, and every stakeholder has signed off. The CAB unanimously approves the change because from a governance perspective everything appears under control.

Then Monday morning arrives.

A payment platform is unreachable. An API cannot reach its database. A firewall policy blocks legitimate traffic. Whatever the exact failure, the engineers usually understand the problem surprisingly quickly. They identify the offending configuration, verify the traffic flow, and know exactly what must be changed to restore service.

Ironically, fixing the outage often proves harder than diagnosing it.

Before anyone is allowed to modify production, an emergency change must be opened. Documentation has to be written while the incident is still unfolding. Managers must be contacted. Approvals must be collected. Sometimes the final authorization comes from people who neither diagnosed the issue nor possess the technical expertise to evaluate the proposed solution.

That contradiction has always struck me as one of the strangest aspects of enterprise IT.

The organization trusted the approval process enough to introduce the outage, but it does not trust its engineers enough to remove it.

At that moment, the CAB is no longer protecting production. It has become part of the recovery path.

Approval Is Not an Engineering Control

The underlying problem is surprisingly subtle because it comes from confusing two completely different forms of risk management.

Administrative controls answer questions such as: Have the stakeholders been informed? Is the maintenance window acceptable? Does a rollback procedure exist? Have the operational teams been notified? Those are important questions, and governance exists for good reasons.

Engineering controls answer completely different questions. Will this firewall policy accidentally shadow another rule? Can this routing policy leak prefixes under an unexpected topology? What happens if this service disappears halfway through the deployment? How quickly can we detect the failure? Can we recover automatically?

One category manages process.

The other manages systems.

Unfortunately, many organizations unconsciously assume that strengthening the first somehow compensates for weaknesses in the second. It does not. A committee cannot validate a BGP policy any more than it can prove a software algorithm is correct. Technical correctness comes from testing, simulation, automation, observability and architecture, not from consensus.

Approval creates accountability.

It does not create correctness.

Why Bureaucracy Always Grows

One pattern appears repeatedly across large enterprises. Whenever the engineering system cannot provide enough confidence, process grows to compensate.

If configuration cannot be validated automatically → another reviewer is added.

If the blast radius cannot be estimated confidently → another approval is required.

If rollback remains unreliable → maintenance windows become increasingly restrictive.

If production lacks visibility → documentation requirements continue to expand.

Every individual decision appears reasonable because each one addresses a legitimate concern. Over time, however, they accumulate into something entirely different. Engineers spend more effort navigating process than improving the platform itself. Bureaucracy becomes a substitute for engineering maturity.

This is why I don't believe CABs are the real problem.

They are symptoms.

A large CAB usually indicates that an organization does not fully trust its infrastructure. Instead of investing in systems that are easier to validate, easier to observe and easier to recover, it invests in mechanisms that attempt to control the people operating them.

That strategy eventually reaches its limits because complexity continues to grow while human review scales linearly.

Build Infrastructure That Deserves Trust

The goal is not to eliminate governance, but to shift operational safety out of meetings and directly into the platform.

Infrastructure must behave as an internal product where safety is built in:

• Generate configurations from structured intent instead of manual entry.
• Analyze firewall and routing policies automatically before deployment.
• Use real time observability to detect anomalies instantly.
• Ensure deployments become safer through continuous platform validation.

Networking technology has made massive leaps through dynamic routing, intent based automation, and deep telemetry. Yet many organizations still restrict these modern systems with outdated governance models designed for handcrafted legacy networks. The technology has simply outpaced the bureaucracy.

Engineering for Failure

Perhaps the biggest misconception in enterprise IT is the belief that enough governance can prevent failure.

It cannot.

Distributed systems fail. Hardware fails. Software contains defects. Humans make mistakes. None of those realities disappear because another approval was added to the workflow.

What distinguishes mature engineering organizations is not the absence of incidents but the way they prepare for them. They invest in limiting blast radius, validating assumptions before deployment, detecting anomalies quickly and restoring service without unnecessary friction. Every incident becomes an opportunity to improve the platform itself rather than to introduce another procedural checkpoint.

After a production outage, I find one question far more interesting than "Who approved this?"

I prefer asking "What engineering capability was missing that made this approval necessary in the first place?"

The answer is often far more valuable than the incident report.

The Role of the CAB

Ironically, I think Change Advisory Boards become more valuable when they try to do less.

Their responsibility should be to coordinate business impact, identify scheduling conflicts and ensure that engineering teams have produced sufficient evidence to justify a risky deployment. They should review the quality of the engineering process, not replace it.

A CAB should never become a compiler for infrastructure changes.

It should never become a substitute for testing.

And it should certainly never become an obstacle between qualified engineers and the restoration of a production service.

The strongest infrastructures I have worked with were not the ones protected by the largest number of approvals. They were the ones that required the fewest approvals because confidence had already been engineered into the platform.

I don't think stable systems emerge because enough people voted in favor of a change.

I think they emerge because architecture gradually removes the need for humans to make the same operational decisions over and over again.

Perhaps that is the real objective.

Not to build better CABs.

But to build infrastructures that no longer need them to feel safe.