FlowSpec – Using BGP for Rapid DOS Response

Using BGP FlowSpec to Push an ACL to the Edge of the Network, to Stop a DOS Attacks, and Build a DOS Response Architecture

Version (0.7)

FlowSpec provides large networks with an ability to push a layer 4 ACL rapidly to the edge of the network using the Network Layer Reachability Information (NRLI) expansion of the Border Gateway Protocol (BGP).

Think of a situation where you are under attack. Your network has +20 routers spread all over a large geographic area. You are not sure where the attack is ingressing into your network, but you need to do something quick. You see the source port. It is the same everywhere. You quickly go to your “BGP Trigger Router” that you use for Remote Trigger Black Hole (RTBH) and add a FlowSpec rule to push an ACL to the edge of the network that would drop all traffic with that specific source port. BGP pushed the FlowSpec rule to the edge of the network to all +20 routers. The router accepts the route and apply it to the ingress ports (a configuration rule that is prepared). Packets start dropping on the edge. You now pull up your network monitoring and find there are three routers doing all the packet drops. You now know the ingress points. Now you need more information. These routers are set up with attached sinkhole. Sinkholes in a network are attachments (switch with gear attached) which allow you to shunt BGP traffic to forensic tools before dropping the packet. You update your BGP FlowSpec rule for these three routers with a BGP Extended Community to shunt the packets to the Sinkhole vs dropping the packets (set next-hop). You now have the packets going into your forensic gear, allowing you to see the packet characterics of the attack.

This situation described here is just one of many types of use cases for BGP FlowSpec. Dissemination of Flow Specification Rules – RFCs 5575 was an Internet community evolution of the initial Source/Destination Remote Triggered Black Hole (RTBH) and Cisco FIB Entry Feature (QoS Policy Propagation via BGP – QPPB). FlowSpec allowed Operators to push out policy directly into a preconfigured ACL on a router.

The original plan was to use BGP FlowSpec as a capabilities bridge until the completion of the Threat Information Distribution Protocol (TIDP) and Threat Mitigation Services (TMS) where completed. TIDP/TMS was never completed nor pushed forward into the IETF. Given the concepts and principles in TIDP/TMS were industry driven, current work for a more detailed capability is now in the IETF as the DDoS Open Threat Signaling (DOTS). Other related work is being craft in the P4 Consortium that would push threat policy into network forwarding plane. While there is always development work, the fact that FlowSpec exist.

FlowSpec is Operationally Deployed and Multi-Vendor Supported!

While FlowSpec’s vendor implementation took a while, the Operator community continued to push their vendors to implement FlowSpec. Today we have large and small operators deploying & using FlowSpec. We have cloud companies like Cloudflare who depend on FlowSpec as one of their many situation response tools. FlowSpec has worked with networks under severe DOS attacks, leveraging on the resiliency of BGP to push out the policy throughout their network to the network elements which will drop the attack.

There is a full list of FlowSpec supporting vendors, their white papers, documentation, and other materials listed in the reference section below.

Operator Collaboration for Collective Defense with FlowSpec

Operational Confidence in FlowSpec has open the door for a collective of Operators to collaborate together. AT&T and Charter Networks are inviting any and all Operators to join them in a DOS Peering Collective. Details can be seen during the NANOG 71 presentation Operationalizing ISP cooperation during DDoS attacks:

Inter-operator collaboration using FlowSpec has always been envisioned. The AT&T/CTL DDOS Peering effort is focused on protecting critical infrastructure (backbones, Operator’s services, IXPs, Private Peering, and other critical infrastructure). It is not scoped to be a replacement for DOS Mitigation services nor to replace the DOS Mitigation services the individual Operators provide their customers.

BGP FlowSpec Tools, Code, and Scripts

Several operators are coding scripts, tools, and capabilities to allow them to effectively deploy FlowSpec. The following is a list.

Auto-Flowspec Docker Container for DDoS Mitigation

Created by Charter Networks as a tool for their FlowSpec deployment. Auto-Flowspec docker is a self-contained flowspec controller in a container. The auto-flowspec.py script will listen for syslog messages from an Arbor SP PI device. If the syslog message contains “Residential” and “importance 2” (high level alert) and doesn’t contain “is now done” (attack is over) then a API call will be made with Flowspec rules to a flask server which controls ExaBGP. ExaBGP will then send the rules to one or two route reflectors. The alert details are also written to a MySQL database. Once a syslog message saying that the attack “is now done”, another API call is made to ExaBGP to remove the rules. There is also a clean-up script that will remove any rules that have not been withdrawn after a specified amount of time.

https://github.com/racompton/docker-auto-flowspec/blob/master/README.md

bgpflowspectool

This tool has been developed to help network engineers to deal with undesirable traffic that is passing through their Internet network. This tool has been design to propose a graphical user interface to manage network features like:

  • Filter traffic with BGP flowspec,
  • Drop malicious traffic with BGP blackhole,
  • Design and configuration of RTBH (Remote Triggered Blackhole),
  • Interface analytics system,
  • …

This first version is currently Alpha and needs to go through a set of test to make it an usable version. For now, it only supports BGP flowspec (RFC5575). This tool is not suppose to be installed in production network but rather be used for lab / test purposes.

https://github.com/Pragma-Innovation/bgpflowspectool

flowspy

flowspy is a function of GRNET Firewall on Demand platform. Firewall on Demand applies via NETCONF, FlowSpec rules to a network device. These rules are then propagated via e-bgp to peering routers. Each user is authenticated against shibboleth.

BGP FlowSpec Reference Materials, Videos, and Training

FlowSpec Videos to Watch

NANOG 63 – DDoS Mitigation using BGP Flowspec by Justin Ryburn. Justin’s presentation is the foundation presentation for FlowSpec. It is the place to start your exploration of this tool.

A good supplement to this NANOG session would be Justin’s Packet Pusher’s Priority Queue podcast – PQ Show 78: BGP Flowspec For DoS Mitigation.

NANOG 63 – Selective Blackholing – How to Use & Deploy by Job Snijders. A comprehensive BGP community design used to managed peering, setup black holes, manage traffic into sinkholes, AND integrate FlowSpec would be part of any deployment plan. This session by Job is a good place for all Operators to craft their plan. NTT has been very open with their BGP Community plan/policies both to set the example but to also be used as a learning template.

NANOG 38 – Deployment Experience With BGP Flow Specification panel session with Derek Gassen, Craig Labovitz, Raul Lozano, and Danny McPherson.

NANOG 58 – Traffic Diversion Techniques for DDoS Mitigation using BGP FlowSpec by Leonardo Serodio

FlowSpec Blogs, Post, & Articles

Vendor’s FlowSpec Implementation Guides, White Papers, and Documentation

The following is as a complete list as possible of vendor implementations. We’re listing the vendors in alphabetical order, but credit goes to Juniper Networks for keeping the faith in the FlowSpec concept, allowing networks to validate the model work, and laying the foundation for the industry to move forward.

Cisco Systems

Cisco is a marketing/communications machine. There is no surprise that once Cisco committed to FlowSpec that it would charge forward with a library of materials. It is well worth looking at all the vendor oriented materials on FlowSpec to get ideas for the how to architect and deploy your FlowSpec in your network with your vendors.

FlowMon

FlowMon’s DDoS Defender 3.0 supports BGP Flowspec.

Fortinet’s FlowSpec

IXIA FlowSpec Testing

Ixia has BGP FlowSpec Test that can be used in the lab to validate the functionality in IPv4 & Ipv6.

Juniper Network’s FlowSpec