Building Enterprise Internet Connectivity: A Practical Guide from the Trenches

A Note on Context: Network engineering is rarely black and white. The following blueprint represents a methodology honed over 15+ years of working in the trenches of Internet Edge architecture. Treat this as a field report on what has worked for me rather than a universal standard.

The Sovereign Edge: Why Your Default Route is a Liability

Most enterprise connectivity architecture is an afterthought - until the moment it breaks.

If you are relying on provider-assigned address space and a default route, you are essentially a "digital tenant." You rent your IP addresses, you rely on their routing logic, and if you leave, you lose your identity.

True network resilience requires Sovereignty: running your own Autonomous System (AS), owning your IP space, and treating the Internet edge not as a utility but as a strategic asset.

Here is my blueprint for architecting an enterprise edge that owns its fate.

1. Monitoring: The Inside-Out Blind Spot

Tools like Akvorado are fantastic for flow analysis, but they suffer from the "Inside-Out" blind spot: they only see traffic after it hits your interface. They cannot tell you that your packets are being dropped in Frankfurt three hops before they ever reach your border in Paris.

• Internal Metrics: Use open source (Akvorado/ElastiFlow) to track utilization and internal anomalies.
• Global Synthetics: You need active, external probing. This is why I rely on commercial platforms like Kentik.
• The Reality: Spinning up a flow collector is easy; maintaining 500+ external agents worldwide to test your global reachability is an operational nightmare. Don't build it - buy the leverage.

2. The "Two-Provider" Trap & The Diversity Audit

Too many enterprises rely on a "Minimum Viable Edge" - a pair of Tier-1s or a regional provider plus one transit. This looks fine on a spreadsheet but it is a single point of failure disguised as redundancy.

The hidden risks:

• The "Dual-Homed" Illusion: Connecting to the same ISP via two different routers gives you hardware redundancy, but zero logical redundancy. If that ISP has a routing meltdown, you go dark on both links.
• Fate Sharing & The First Mile Blind Spot: Unlike purchasing wavelengths where you can demand KMZ path files, IP transit often treats the physical layer as a black box. Even if the ISP core is redundant, you rarely have visibility into the "first mile." Frequently, two "competitors" lease the exact same last-mile tail from the local incumbent. A single backhoe exposes that your diversity was only on paper.
• The Peering Wars: Tier-1 providers sometimes get into disputes (e.g., the Cogent vs. Google history). When giants fight, your enterprise traffic can become collateral damage if you're pigeonholed behind one upstream provider.

If you have limited path diversity, you have limited options to scrub or divert traffic.

3. Capacity is a Security Feature (100G is the New 10G)

Stop sizing ports based on your average utilization. If your hardware supports it, make 100G your new baseline.

• The 10G Bottleneck: During a volumetric attack, a 10G port saturates instantly. Whether you have upstream protection or not, you will suffer from this saturation - your connectivity degrades and you go dark before you can even analyze the attack vector.
• The 100G Buffer: High-capacity ports provide the "headroom buffer" necessary to absorb the initial hit, analyze the flow, and apply mitigation filters or Flowspec rules.

Bandwidth is relatively cheap. Downtime is very expensive. Buy the headroom.

4. The Transit/Peering Hierarchy

There is no "Transit vs. Peering" debate. They are different tools for different situations:

• Transit: The universal key. It gives you global reach by paying a provider to carry your traffic across the public Internet. Essential for reaching walled gardens.
• IXP (Internet Exchange Point): The cheat code. You physically connect to an exchange and peer directly with other networks. Traffic stays local, latency drops, costs plummet. But you can't IXP your way to everyone - major residential ISPs ("Eyeball Networks") operate walled gardens and demand unattainable traffic ratios.
• The Mix: Use IXPs for content, cloud, and open networks. Use Transit to brute-force your way into the walled gardens.

5. The Cloud Connectivity Pivot: Kill the VPN

The era of the IPsec VPN as a primary pipe is over. It is a "best effort" gamble running over the public Internet, subject to jitter, packet loss, and congestion.

If you join an IXP, you unlock the Cloud Exchange. Most major IXPs offer direct Layer 2 connectivity to AWS (Direct Connect), Azure (ExpressRoute), and GCP via 802.1Q VLANs on your existing port.

• Performance: Deterministic latency and higher throughput (10G+).
• Cost & Agility: Significantly cheaper than dark fiber, provisions in minutes, not months.
• Action: Demote your VPN to backup status. Use the IX fabric for your heavy lifting.

6. The Case for Regional Providers

Everyone chases Tier-1 bandwidth. Bigger is better, right? Not always. Good luck getting a Tier-1 NOC engineer on the phone during a routing crisis.

Regional ISPs often outperform Tier-1s for local traffic because they peer directly with regional eyeballs, keeping traffic local rather than hair-pinning it through a major hub. And when regional paths aren't available, they're connected to Tier-1 providers for global reachability.

More importantly, you can actually talk to their engineers. I have resolved complex routing loops via direct phone call with regional engineers in hours - whereas a Tier-1 ticket would still be in the "triage queue."

7. The Control Plane Cheat Code: BGP Communities

If you are running your own AS, you must master BGP Communities. They are the difference between a passive network and an active defense.

Most quality providers offer community tags that allow you to control how they propagate your routes:

• The Blunt Tool: AS Path Prepending (making your path look longer). This is often ineffective - upstream providers may ignore your prepend if they have a shorter-path policy.
• The Surgical Tool: BGP Communities. You tag your routes to say "Do not announce this prefix to your upstream providers." Traffic from local peers flows through a specific high-performance port, while the rest of the global Internet reaches you through your Tier-1s. This is granular control.

The Blueprint: My Ideal Enterprise Edge

Stop looking for the "perfect" provider. Build a resilient blend:

• 2x Tier-1 Providers: For global reach and physical path diversity.
• 2x Regional Providers: For latency superiority and "human factor" support.
• IXP Presence: For peering and direct Cloud Connectivity (kill the VPN).
• Full Visibility: Commercial synthetics (Kentik) to see the outside world.
• Oversize: 100G ports as your insurance policy against volumetric attacks.

Next: We turn this foundation into an automated fortress - BGP Flowspec, surgical RTBH, and the art of killing bad traffic at the edge before it ever touches your compute.