RTBH Fundamentals

You have three choices when you stand in front of an on rushing force. You can push back directly against that force. You can step aside and let the force push past you. Or, you can redirect the force to a location that you choose. Now think of that “force” in the form of a Denial of Service (Attack). When DOS attacks are forces on the Internet which can be addressed with either of the three options. Of the three, redirection has been one of the mostly commonly used tools to “redirect” the DOS attack to a location which the Operator can better manage the attack. The foundation tool for that redirection is the used of the Border Gateway Protocol (BGP) and the Remote Triggered Black Hole (RTBH).

For over 15 years, the core anti-DOS security tool is the same tool we use to glue the Internet. The Border Gateway Protocol (BGP) is the glue which connect the Internet together. We use BGP as a tool to move and route GOOD traffic from one part of the Internet to the other. This also means we can use BGP to move BAD traffic to places that minimize the hard to the Internet – including places that “just drop” BAD traffic. We call this BGP security technique a Remote Triggered Black Hole (RTBH). (This work was originally published 2002-08 as a supplement to the ISP Essentials book by Barry Raveendran Greene, and Philip Smith.)

Remote Triggered Black Hole is a flexible ISP, Carrier, and Large Enterprise Security tool that will route packets to Null0 (i.e. black holed). The Cisco ISP Essentials book covers the fundamentals of the single router based black hole routing technique. It did not cover the remote triggered black hole routing technique. Remote triggering via iBGP allows ISPs to activate a network wide destination based black hole throughout their network. This technique is especially useful in some of the new ISP security classification, traceback, and reaction techniques. What we’ll cover are the essential RTBH principles along with deployment experience from the 15 years of operational deployment. RTBH is a foundation Operator’s Security tool for which many other security tools are built upon.

Yes, “Null Route” is a Packet Filter

Forwarding packets to Null 0 (aka Null Route) is a common way to filter packets bound to specific destination. These are often done by creating specific static host routes and point them to the pseudo interface Null0. This technique commonly referred as black hole routing, black hole filtering, or Remote Triggered Black Hole (RTBH). Null0 is a on a router/switch pseudo-interface. The Null 0 pseudo-interface functions similarly to the null devices available on most operating systems. This interface is always up and can never forward or receive traffic. While Null0 is a pseudo interface, within modern forwarding architectures (like Cisco’s CEF), it is not a valid interface. Hence, whenever a route is pointed to Null0, it will be dropped via router’s forwarding logic.

The null interface provides an alternative method of filtering traffic. You can avoid the overhead involved with using access lists by directing undesired network traffic to the null interface. The following Cisco example configures a null interface for IP route 127.0.0.0/16 and the specific host 171.68.10.1 (subnet mask 255.255.255.255):

interface Null0
no icmp unreachables
ip route 127.0.0.0 255.0.0.0 null 0
ip route 171.68.10.1 255.255.255.255 null 0

The no icmp unreachables command is used to prevent unnecessary ICMP Unreachable replies whenever traffic is passed to the Null0 interface. This minimizes the risk of router getting overloaded with hundreds of pending ICMP Unreachable replies. It is common practice to use the no ip unreachables command on the Null0 interface. There may be cases where you want the router to respond to with ICMP Unreachables. For these cases, work with your vendor to deploy a ip icmp unreachable rate-limit command to minimize the risk of a router getting overloaded with ICMP Unreachable processing. For Cisco IOS, this specific rate-limiting command adjusts the default of on ICMP Unreachable every 500ms to a value between 1ms to 4294967295 ms.

ip icmp rate-limit unreachable 2000
ip icmp rate-limit unreachable DF 2000

RTBH uses the strength of the router’s forwarding performance to drop blacklisted packets. A router’s #1 job is for forward packets – not filtering packets. The black hole routing technique uses the packet forwarding power to drop all packets bound for sites on the blacklist. In the ASIC, FPGA, Network Processing forwarding world, this blackholing has zero impact in the performance of the router (packets black holed to Null0 are cleared through a register clock). Software forwarding devices have some extra cycles needed to clear out and black holed packet. If a software-forwarding device is expected to do a lot of black hole work, consider a black hole shunt interface (see the section on black hole shunts).

There are two main limitations to with the black hole filtering technique. First, black hole filtering is L3 only – not L4. Access to all L4 services at a given site will be blocked. If selective L4 filtering is necessary, use prepared extended ACLs or FlowSpec rule (see notes section). For example, if you wish to drop all packets to a specific destination, the black hole filtering is applicable. But, if you wish to drop all telnet packets to a destination, then black hole filtering is not applicable and an extended ACL is the optimum mitigation tool. Extended ACLs offer the fine L4 granularity needed to filter at the application level. Second, it is hard to bypass or provide exceptions with the black hole filtering technique. Any organization that wishes to bypass the blacklist must actually find a way to bypass the filtering router’s forward table. Compensation for either limitation are not trivial tasks. Yet. With due consideration and planning, options are available for both.

Figure 1 – Using static host routes to null 0 for black list filtering

Why is RTBH so Useful?

Black Hole Filtering on a single router has been around the industry since the last 1980s. It is a useful tool on a single router. But, how do you use this tool when you have a network of hundreds of routers? How do you log into hundreds of routers on the edge of a network and configure a black hole filter? The answer is in you don’t. ISPs engineers who respond to a security incident needs to think of their key strength – routing. ISPs engineers know how to route traffic – putting the traffic where they want it to flow through their network. Remote Triggered Black Hole Filtering uses that routing strength to trigger all the routers in the network with a routing update. The routing update – sent via iBGP by a trigger router – actives a pre-configured static route to activate a black hole for the destination address.

Lets use an example to illustrate the concept and strength of this technique. Figure 2 illustrates an ISP’s customer under attack by a DDOS. The DDOS is coming in from all the entry points of the ISP’s network. These entry points can number from a few to thousands – depending on the size of the ISP. DDOS traffic far exceeds the customer’s link, so the circuit saturates, causing either DOS Flapping or collateral damage inside the POP. This collateral threats other section of the ISP’s network. An immediate reaction is necessary to shift the packet drops from the customer’s circuit and collateral routers to the edge of the network.

What is DOS Flapping? DOS Flapping is a form of collateral damage that comes when the circuit under attack goes into congestion collapse. The DOS Flap happens when the IGP route locking the iBGP advertisement for the customer drops with the saturated circuit. The more specific for that customer is removed, flapping the DOS flow to the next best path. That could be a router in the POP or somewhere else in the network. Note: Sinkholes might be a way to shunt DOS Flaps so they do not cause collateral damage in the POP.

Figure 2 – Customer Under Attack by a DDOS

Remote Triggered Black Hole filtering is used to push the packet drops off the customer/POP routers and shift them to the edge of the network. Figure 3 shows how an ISP uses a trigger router in the NOC to send an iBGP advertisement. This iBGP advertisement has the prefix of the customer under attack with metric attached to insure it becomes the preferred path. This iBGP “trigger” advertisement goes to all the iBGP speaking routers in the ISP’s network. These routers all have an unused prefix that points to Null 0. The iBGP “trigger” advertisement has its next-hop equal to this “Null0ed prefix.” When the “iBGP trigger advertisement reaches the router, it gets glued to the static, activating the Null0 black hole, and having all traffic to the customer’s prefix get dropped on the edge of the ISP’s network.

The key benefit in this situation is that dropping on the edge of the network mitigates the DDOS’s aggregated traffic load. This now gives the ISP and the customer time to work the attack without the worries of collateral damage to other customers.

Figure 3 – Trigger Router Activates the Black Hole Throughout the Network

Who proved the production viability of RTBH?

RTBH was an interesting tool, but it was not well known. It was not until 2002 that very visible usage of RTBH use at WorldCom/UUNET provided the best example through practice. The practice was illustrated in the NANOG 23 Tutorial – ISP Security – Real World Techniques (see http://www.nanog.org/mtg-0110/greene.html). Christopher L. Morrow and Brian W. Gemberling (with Barry Greene assisting) demonstrated how they use remote triggered blackhole filtering together with sinkholes and backscatter to create a innovative tool to rapidly traceback attacks to the edge routers on their network (usually takes between 5 to 15 minutes across AS numbers 701, 702, and 703). So the response for those who doubt the technique can be assured that it does work. If UUNET can integrate this technique as one of their core DOS/DDOS response tools, then there should be little problem for others to implement the technique.

RTBH Safety Measures

Remote Triggering via iBGP requires the ISP to take some safety measures to insure the iBGP trigger advertisement does not leak out and affect other networks. There are several ways this can be done. Applying the principle of Murphy’s Law of Networking, it is recommended that an ISP implement several – if not all – of these safety measures.

No-export BGP community. The no-export community in BGP is a well- known value that most routers recognize by default. It should – when working properly – keep the prefix within the ISP (i.e. no advertisements to peers).

Extra Community that filters. The ISP can add a community that does the same as the no-export community. A BGP community filter will be used on with the ISP’s peers to mark which communities are exported. This step helps prevent a leak by someone who is cleaning up the excess communities in the prefix – inadvertently filtering the no-export community.

Lower Boundary on the Egress Prefix Filter. ISPs can place a lower boundary on the prefixes sent to their peers. For example, ISPs can block all prefixes less than /24. This would filter any iBGP trigger advertisement between /25 and /32 – which is a normal range of addresses blocks allocated to customers.

Remember the consequences of “Murphy’s Law!” Pakistan found out in 2008 the consequences of using RTBH without the safeguards and assuming that something will go wrong. A local order to block access from within Pakistan to Youtube ended up impacting large part of the Internet causing a massive flow of Youtube traffic in the direction of Pakistan – knocking out networks along the way.

Preparing the Network for RTBH

It is imperative that ISPs prepare for remote triggered black hole filtering, practice the technique, and have everything ready long before using it to mitigate an attack. Fortunately, all the preparation steps involve non-intrusive configurations that have no impact on the operation of the network.

Step 1- Configure the Static Route to Null0 on All the Routers

The first of these preparation steps is the configuration of a static route on each of the routers that will be triggered. This is a prefix that will never be used in the network. It can be a block of addresses allocated from the RIR allocations. It can be a RFC 1918 prefix. The author’s favorite is to use the Test-Net: 192.0.2.0/24. Test- Net was a IANA allocation made for people to do documentation. The idea was for documentation to use a block of addresses that would never get used. That way customers who copy the documentation will not mess up someone else’s network. Hence, Test-Net is one of the IANA Designated Special Use Addresses (DUSA) that should never appear on the Internet …… making it a great choice for the static route for remote-triggered Black Hole Filtering.

ip route 192.0.2.0 255.255.255.0 Null0

Step 2 – Prepare the Trigger Router

The trigger router does not have to be a big router. A Cisco 26XX or 36XX router configured as an iBGP route reflector client and accepting no routes works very well as a trigger router. In fact, the trigger router does not have to be a dedicated router. A production router can be used. For this example, we will be using a dedicated trigger router.

On the router, the iBGP is configured to redistribute static routes. That way the “trigger” is an engineer or tool adding and removing static routes. A route-map is used to match the static tag and set all the metrics for the iBGP advertisement. That way all triggering is consistent and done the same way each time.

router bgp 109
.
redistribute static route-map static-to-bgp
.
!
route-map static-to-bgp permit 10
match tag 66
set ip next-hop 192.0.2.1
set local-preference 50
set community no-export 600:000
set origin igp
!
Route-map static-to-bgp permit 20

In the above example, we match a static tag of 66. If matched, we set the iBGP next- hop to the Test-Net (pre-configured on the routers to Null0), set the local preference to 50 (to override the original customer advertisement), set the BGP community to no-export with a safety community of 600:000 (which blocks advertisement, and finally set the origin to igp. This sets up the trigger router to be ready for the time when the ISP needs for rapid reaction.

Step 3 – Activation

The ISP adds a static route with a tag of 66 to activate the remote-triggered black hole. In this example, we’ll use 171.68.1.1 as a the address under attack. So we add this static with the tag of 66:

ip route 171.68.1.1 255.255.255.255 Null0 Tag 66

The trigger router will then send a advertisement to all the iBGP speaking routers in the network (see Figure 3). When the iBGP advertisement is received, the BGP RIB sees the local preference of 50 and selects this new path as the best path. The recursive look-up passes since there is a static route to this new path’s next-hop (i.e. the Test-Net). This iBGP best path is passed from the BGP RIB to the router’s FIB. The FIB sees the prefix, the next-hop of Test-Net, and Test-Net’s next-hop of Null0. It then glues them together (depending on the FIB technology used) resulting 171.68.1.1 now having a next-hop of Null0. This is visually illustrated in Figure 4. PAGE_BREAK: PageBreak

Figure 4 – How the Router takes the iBGP Trigger and Activates the Black Hole

One of the key advantages of remote triggered black hole is the number of prefixes that can be filtered. The limit is the size of the FIB routers in the network can carry. This would mean thousands of black holed prefixes being added. It is just a factor of adding more static routes to the trigger router. Principles of aggregation can be used, but mindfulness needs to be applied to make sure the iBGP trigger advertisement is equal to or more specific to the original customer advertisement.

Step 4 – Removing Trigger Advertisement

The trigger advertisement will need to be removed when the attack is over or the ISP wishes to move to a different mitigation technique. Removing the static route does this. The trigger router will then send out a iBGP withdrawal to all its BGP peers, which in then will withdraw the route from the BGP RIB, which then pulls the route from the route’s FIB. This clears the path for the router’s BGP RIB to select the original customer advertisement, placing that prefix as the best path, and allowing the FIB to resume normal forwarding to the customer’s network.

RTBH Limitations

Each security tool has its strengths and limitations. The key limitation of remote- triggered black hole filtering is the effect of an unintended consequence. An example of an unintended consequence is having a DDOS attack specifically designed to force an ISP to black hole a customer. The attack would be setup to cause collateral damage to adjacent customers. The collateral damage forces the ISP to react with a remote-triggered black hole – taking out the path to the customer. The problem is that the black hole is exactly what the attacked wanted – removing the customer’s ability to receive traffic.

This limitation is one of the reasons ISPs need a suite of mitigation tools. Focus on any single DOS/DDOS mitigation tool limit an ISP’s options and make them vulnerable to a manipulating attack profile – where the attack manipulates the ISP to react to the benefit of the attacker.

Limitations with Blacklisting and Censorship

Blacklist filtering refers to organizations attempting to place barriers to information on the Internet. The Internet – by its nature, culture, and heritage – resist any barrier to communication. In fact, the Internet is the ultimate of internetworking where everything and everyone is interconnected through transparent end-to-end connectivity. Yet, because of the social and political forces the Internet brings with it, there are organizations and government who wish to block and/or protect customers/citizens from access certain sites on the Internet. Pornographic, Political, and OnLine Gambling are the top three topics that organizations and governments wish to include in a black list filter.

Remote-Triggered Black Hole filtering is one way to accomplish Blacklist filtering. If an ISP has a legal requirement to block access to specific sites, destination black holes will accomplish that objective. The core issue is the operational overhead of maintaining that list. Remote-triggered black hole filtering helps minimize the operational overhead by having one router in one location holding the master list (i.e. the trigger router). This allows for frequent and rapid change to the black list. This change feature is critical, since as mentioned earlier, the Internet resist attempts to block the flow of information. Once a site is blacklisted, that site will physically or logically move. A simple change of IP addresses could be all that it would take.

Evolution of RTBH – All the Different Flavors

The flavors of RTBH as evolved once the industry gained confident with the fundamentals. Once the RTBH was dusted off, Barry Greene and Steve Johnson saw how Unicast Reverse Path Forwarding (uRPF) could be combined with RTBH to drop packets based on the source. This evolved into Source RTBH. Unicast RPF was then modified with the “loose check.” This allows rRPF to applied on the Operator’s peering/transit edge combined with RTBH to drop attacks based on the source addresses of the attack. When used with Netflow/IPFIX, the Operators are able to have a powerful tool to mitigate attacks. To give a context, we now refer to RTBH as “destination” RTBH.

Chris Morrow, then extended the usefulness of RTBH with BGP communities that UUNET/Verizon Business’s customers can use to activate rules. This “customer triggered” RTBH work for source or destination (depending on the BGP community) to give customers a tool to trigger blocks one ASN upstream.

The use of BGP Communities with RTBH then expanded are more operators deployed RTBH. We then say how you can use RTBH with BGP Communities. The best example is Job Snijders campaign at many of the Operator Groups on the flexibility of a BGP community plan. Two videos of Job’s talks are NLNOG 2015 – Job Snijders – Ultra fast DDoS detection & mitigation at ColoClue and APRICOT 2015 – Routing Session. Both sessions provide insight as to how modern Operator’s have deployed RTBH with BGP communities.

If you are using BGP Communities, then it is logical to use some of the other tools that can be triggered with RTBH using BGP Communities. Cisco has configured rate limited with QOS Policy Propagation for BGP (QPPB) which combined with RTBH to trigger rate limits as a tool to rapidly respond to DOS attacks. This then evolved to RFC 5575 with Dissemination of Flow Specification Rules (BGP FlowSpec). The work continues with the IETF’s DDoS Open Threat Signaling (dots).

All of these tools will be expanded on in other sections of the Operator’s Security Toolkit. What is important today is a sound fundamental understanding of RTBH, an initial deployment of RTBH, and a operational confidence to use RTBH to protect your business.

Acknowledgements

While I do not remember who taught me this technique, the first time I personally used it was back in 1991 on a couple of US Military networks I was helping to operate. Since that time, this technique seemed to be forgotten – explaining why many people did not understand how uRPF Loose Check and Source Based Remote-Triggered Black Hole Filtering will work (see the separate paper on that topic). It was not until Christopher L. Morrow chris@UU.NET, Brian W. Gemberling brian@UU.NET, and UUNET’s NetSec Team demonstrated to the world that Remote-Triggered Black Hole filtering as part of their Backscatter Traceback technique that people woke up to the power of this mitigation technique. So a lot of credit goes to UUNET for showing that this technique works. As some say, if it can work at UUNET, then 99% of the deployment issues have been covered. ☺