f you have not already heard this Netflix announcement last week, CEO Reed Hastings, and CFO David Wells said ….
“We already offer Netflix in about 50 countries and have learned a great deal about the content people prefer, the marketing they respond to and how to best organize ourselves for steady improvement. Acceleration to 200 countries is largely made possible by the tremendous growth of the Internet in general, including on phones, tablets and smart TVs.”
This is a surprising rapid acceleration of Netflix’s international expansion. Netflix’s plans will trigger a response from companies like Hulu, Amazon, and other “Over the Top” Video solutions. For telecoms operators around the world, this would be a huge change that will impact their network. The impact will happen with or without their participation. Netflix’s announcement is a forewarning for operators to prepare now for the coming video explosion. Telecom operators have a choice embrace the video explosion now and capitalize on the OTT’s investment or be blindsided by their customer’s hunger for video.
Where to start? Understand that the OTT players are watching your Customer’s Experience!
The OTTs players are watching your network, are you watching their network? Quality of the customer’s experience is critical to any service on the Net. In the context of Netflix, their whole business model is dependent on the quality of their customer’s experience. Once of Netflix’s limitations is their coupled dependency with the operator’s quality of experience. An operator’s network to poor peering, mismanagement bandwidth, and architectures that add latency to the network will get reflected Netflix’s quality of experience. It impacts Netflix’s the ability to grow new customers and keep them coming back. Hence, companies like Netflix put significant investment into their peer relationship to improve everyone’s quality of experience. Netflix, like other OTTs operators, publicly publish tools that rate the quality of operators in their service markets. The Netflix ISP Speed Index (http://ispspeedindex.netflix.com) uses data from the take the peak three hours of Netflix streaming on a daily basis per ISP (details). Reports like Netflix’s ISP Speed Index provides data per Service Provider (SP) and per Country (see figures). In one way, it is a tool for Netflix. In another way, it is a tool that can be used to externally metric that provider provides the better services based on the applications consumers desire.
Understanding the implications – the CDN – OTT players in the market know more about the quality of experience in your network than you do. They will know more about the operator’s customers then the operator. The OTT players are they are watching your network. Are you watching them?
Courtesy of ispspeedindex.netflix.com
The problem for the operator is that their current Netflow, SNMP, DPI based tools do not provide the means measure the quality of experience based on the OTT’s application. The vast majority of tools inside an operator’s network work well for the “pre-application” universe. They are circuit level with KPIs based on load, bandwidth, and uptime. They are system level measuring capacity, transactions, and storage. Lots of vendors are now trying to retrofit these tools to provide a user experience index. The problem is that these tools are not tracking what the users are doing on their smartphones, tablets, and PCs. These are now application based. These traditional tools do not track the user’s application performance, the OTT’s performance or the quality of experience of what people are doing with their applications. How would an operator identify and fix problems when they appear poorly in an OTT’s “speed index?”
The answer is a new generation of tools built for today’s application driven – user-driven market. The “quality of the customer’s experience” is directly dependent on the quality of the application experience. Application experience and KPIs based on the application’s performance is true for customers who connection to the network constantly migrates from 3G/4G to WIFI to work to home. The one constant in today’s network is the application. Each of these applications can measure their quality of experience with every network where their customers reside. The operator cannot ignore this factor. Operators need to re-think their KPIs to “application-oriented” KPIs. KPI tools that provide an application-oriented view gives the operator specific insight into how an application is being used inside their network. It would let them know in Netflix’s case, which customer, which location, which devices, and at one KPIs throughout their network. What the operator needs to know is that there is a cluster of Netflix users in a specific are that has overloaded resources (backhaul, network gear, capacity, etc.). One problem cluster can skew the data for the entire network.
Courtesy of www.niometrics.com
How can an operator embrace the video explosion?
Most of the OTT video operators use a system that has regional Content Distributed Networks (CDNs) that work with co-located Cache/CDNs in the operator’s network. Continuing with the Netflix example, there would be regional CDNs that are closely interconnected with Internet Exchange Point (IXPs). Operators can then work under Netflix’s Open Connect Initiative, applying for an Netflix Open Connect Appliance to be placed in their network (see illustration from the Netflix Open Connect Deployment Guide). This model is a proven win-win model. The OTT operator (Netflix in this case) is able to offer better quality of service to their customers. The operator wins by also providing a better quality of service to their customers. Both the operator and the OTT saves on bandwidth cost. The OTT even provides tools for the operator to example the performance of the system. What is missing?
Most of the OTT video operators use a system that has regional Content Distributed Networks (CDNs) that work with co-located Cache/CDNs in the operator’s network. Continuing with the Netflix example, Netflix has regional CDNs. These are closely interconnected with the nearest Internet Exchange Point (IXPs). Operators can then use Netflix’s Open Connect Initiative to improve their connectivity to Netflix using two paths. The first is effective peering on the IXP. The second is to apply for a Netflix Open Connect Appliance to be placed in their network (see illustration from the Netflix Open Connect Deployment Guide). This model is a proven win-win model. The OTT operator (Netflix, in this case) can offer better quality of service to their customers. The operator wins by also providing a better quality of service to their customers. Both the operator and the OTT saves on bandwidth cost. The OTT even provides tools for the operator to example the performance of the system. What is missing?
Courtesy of openconnect.itp.netflix.com
Given then, all operators should do the following to prepare for the OTT video explosion:
Each Operator must have a CDN strategy. If you know, they are coming, then prepare for them know. Prepare the data center by reserving space, power, and cooling. Prepare the network by having network ports reserved for “CDN Partners.” Prepare the Internet Gateway Module of the network to allow for these CDN Partners to be in special CDN security “DMZ.” The special architectural zone allows a level of compartmentalization for security and, more importantly, for measuring the effectiveness of the CDN partnership. Prepare the DNS Resolvers to be able to work with the CDN Partners (vs. having everyone go to 8.8.8.8). Look for extras that would help understand the real cost, for example, power monitoring per cabinet and cooling monitoring per cabinet. Granular measurements of power, capacity, cooling, and other factors would help understand the total cost/benefit of the CDN Partnership.
Aggressively seek out CDN partnerships. Akamai started the model with Cisco and Netaid in 1999. Since then, Google, Limelight, Microsoft, Facebook, and others are replicating this model. Netflix is continuing the expansion of their Open Connect Initiative and their steady international expansion. Expect others to follow (Hulu, Amazon, and others). As noted earlier, there is not one operator who has reported “regrets”. The only question from upper management is how effective – how much is saved? How is it impacting the customer experience? When do we need to upgrade?
New angles of monitoring the effectiveness. Operators should start looking at the new startups who are building application oriented tools monitor the KPIs for each application in the network. One of the key functions they need to look for is the ability to monitor the end to end performance of each application. That way when an application reports like Netflix ISP Speed Index illustrate poor performance, the operator has the tools to troubleshoot and take action. In addition, these application oriented tools should monitor the effectiveness of the CDNs the OTTs place inside the network. This would provide a 3rd party view of the effectiveness of the OTT – CDN Partnership.
Invest in the multiple Internet Exchange Points (IXPs). IXPs are the places where Operator interconnect with each other. CDN Partners like Netflix put a lot of work to co-locate content at these IXPs. But, operators cannot take their IXPs for granted. They need to invest in this critical infrastructure. Start with tools that measure the latency and jitter to all their peers connected on the via the IXP. DO NOT USE PING! Use RFC 6349 – Framework for TCP Throughput Testing – that provides more granularity on the latency and jitter between network elements. RFC 6369 more accurately matches the applications experience.
Monitor the End-to-End Application Latency to the CDN Partners. Do you know the latency and bandwidth capabilities to your nearest Amazon Cloud? Is there jitter on the path that might be impacting the application? How about to the nearest Microsoft Cloud? The explosion in applications is a direct consequence of the explosion of cloud oriented networking. This is not going to stop. Many think it will accelerated. Netflix is an excellent explain of a company using these cloud services to expand to “200 countries by the end of 2016.” Fast, low latency interconnections with these cloud services is one way of engineering a network for optimal quality.
Who to ask for help?
All the potential CDN Partners will send their engineers to the regional Network Operations meetings. If you are in Asia, come to APRICOT (www.apricot.net). If you are in Europe, go to RIPE (www.ripe.net). If you are in the Middle East or North Africa, go to MENOG (www.menog.org) If you are in North America, go to NANOG (www.nanog.org). If you are in Latin America, go to LANOG (www.lacnog.org). In African, go to AFNOG (www.afnog.org). Then look for local and national Network Operations meetings. These meetings are places you can meet face to face with the engineers deploying these CDNs and peers who have already deployed CDNs in their network.
Reach out to your peers in other operators within your country. While this is hard for people to make the first step, well worth the effort. One characteristics of great performers in the telecom industry is their ability to reach out to their peers – even if they are the competition.
Explore the new tools for performance monitoring in a network. We live in a world of application oriented networking. Look for big data monitoring solutions that provide KPIs per application, including the OTT applications.
Ask me. E-mail or Linkedin both work. I can provide pointers to the right people at the potential OTT Partners.
Here are some links to CDN Partnerships:
- Akamai – http://www.akamai.com/html/solutions/akamai-accelerated-network-partner.html
- Google Global Cache (GGC) – https://peering.google.com/about/ggc.html
- Microsoft’s Cache – http://www.microsoft.com/peering
- Netflix – https://openconnect.itp.netflix.com/openconnect/index.html
Barry Greene is an Internet Technologist, System Architect, long time CyberSecurity Expert, and mentor of new talent. Connect to Barry via Linkedin (www.linkedin.com/in/barryrgreene/), follow on Twitter (@BarryRGreene), catch his blogs on Senki (www.senki.org).
