DDoS Defense

The Enterprise-as-a-CDN: Mastering Anycast Ingress from the Trenches

Hervé Hildenbrand

— 3 min read

The following blueprint is not a theoretical lab exercise or a "cloud-native" wish list. It is a methodology honed over years of managing a sovereign enterprise backbone where we own our Autonomous System (AS) our IPs, and our compute rather than relying on public cloud abstraction.

In previous articles, the author explored using BGP Flowspec as a migration tool and why stateful firewalls for ingress create performance issues. This piece examines transforming your backbone into a private Anycast CDN.

By having Load Balancers inject /32 service prefixes directly into your backbone, you create a distributed destination offering transparency rather than interception. This requires deep understanding of Traffic Gravity.

1. The Challenge: Predicting Traffic Gravity

In an Anycast model using latency as the IGP metric, traffic flows to the "closest" node. However, BGP is not load-aware; it follows the "path of least resistance," creating massive dependency on inbound traffic patterns.

Key challenges include:

  • The Concentration Trap: Plugging high-capacity local peering and major transit into the same PoP attracts disproportionate load, potentially crushing nodes under their own success.
  • The Inbound Influence: Transit and peering choices directly determine which PoP processes traffic.
  • The Balancing Act: Physical entry points and transit capacity must match user source geography to avoid overwhelming single PoPs.

2. Strategic Survival: From "Gravity Trap" to "Honeypot"

The Gravity Trap can be weaponized as a defensive strategy. By intentionally orchestrating traffic gravity, you can architecturally isolate massive volumetric attacks.

Defensive approaches include:

  • Early Mitigation: Use Flowspec or ACLs at the network edge to drop unnecessary protocols before they reach compute layers.
  • Capacity as Security: Size ports at 100G baseline to provide headroom absorbing initial attack hits, allowing time for flow analysis and filter application.

3. The Intentional Honeypot: Weaponizing Gravity

Tier-1 providers often deliver massive attacks directly. By manipulating gravity, you convert this characteristic into a defense mechanism.

The approach involves:

  • Setting up a Domestic PoP with inbound connectivity strictly via domestic ISPs or local peering, avoiding global transit.
  • Serving core audience from this DC while a second DC connected to international carriers serves the rest of the world.
  • Allowing garbage traffic to gravitate toward the international Tier-1 DC during attacks while domestic customers remain on the "clean" datacenter.
  • Automating this mode using BGP Communities to instruct local providers not to announce your prefix to their Tier-1 upstreams.

4. Protecting the Edge: The Multiplexing Shield

Moving security logic to the Load Balancer layer solves the "Ingress Paradox." While /32 precision manages internal gravity, the /24 remains the global "Sovereign" minimum for internet reachability.

Benefits include:

  • Horizontal Scaling: A fleet of small instances advertising the same IP via BGP spreads load naturally across the fleet.
  • The Multiplexing Advantage: LBs accept thousands of edge connections, routing them over a few persistent connections to backends, saving massive CPU cycles.

5. Traffic Engineering: Internal vs. External Precision

To prevent single PoP bottlenecks, apply BGP tools at the correct layer.

  • The /24 Stability: Avoid prepending or manipulating external /24 announcements. External prepending often proves ineffective as upstream networks prioritize their own policies.
  • The /32 Scalpel: Apply AS Path Prepending or route withdrawals strictly to /32 service prefixes injected by Load Balancers into the internal backbone.
  • Surgical Redirection: During maintenance, "drain" a PoP by withdrawing the /32 locally. The external /24 stays stable keeping latency low, but internally the router redirects packets to the next-closest healthy Load Balancer.

6. The Convergence Engine: Why PIC Matters

BGP Prefix Independent Convergence (PIC) is non-negotiable for sub-50ms redirections. Without PIC, routers scan millions of prefixes during failures. With PIC, backup paths are pre-installed in a hierarchical FIB structure.

Upon detecting failure—ideally via hardware Loss of Signal reacting in 2-5ms—the router performs a single pointer flip, redirecting traffic before users notice.

The Verdict: Architecture is a Full-Stack Game

Building your own CDN using Anycast requires moving from Managing Boxes to Managing Flow. It demands sovereign edge thinking: owning your AS, IPs, and routing logic.

Common fears about "Split Brain" routing typically stem from Stateful Firewalls/Load Balancers. In reality, this setup fits perfectly on short-lived TCP sessions. For long-lived sessions, the architecture must be consistent at every layer—backend and especially database stacks must be designed for a distributed world.

What's Next?

Future articles will cover:

  • "Global Spraying": Alleviating hot potato routing by forcing equal traffic distribution regardless of IGP metric.
  • "Consistent Hashing at Scale": Ensuring the same user hits the same cache node even during dynamic fleet scaling.

Read more