I’m grateful to my peers who spent the time to have coffee over this past week. We all see big changes in the industry and are really worried about the skills needed to capitalize on these changes. What follows are some thoughts that might be useful as we all explore how we can bridge the Internet skills gap.
The severe lack of IP oriented skills is the dominating telecom CXO level “coffee shop” conversation in Asia. The whole industry is now “IP.” CXOs are seeing the opportunities, but do not see their teams in a position to capitalize on these opportunities. What is missing is “Internet skills.” What do we mean by “Internet Skills?” Internet skills = an understanding of how to design, build, tune, and troubleshoot an “end-to-end” Internet service. “Internet skills” are all the knowledge of the systems use to glue the Internet together. “Internet skills” are the technology, protocols, and systems that have enabled the massive empowerment of human communication.
This lack of Internet skill is impacting all parts of the telecommunications industry. Broadband providers cannot find people who know SDNs, OTT streaming video, or new routing protocols. 3G/4G operators cannot find people who understand an application driven “Internet model” that has taken over the mobile world. Traditional data center companies are finding that “SDNs” is not just about “racking servers,” but involve a re-think of ‘compute’ in all parts of the organization. In fact, the LTE deployments add to the skills complexities – requiring knowledge of TCP/IP behavior over multiple mediums (wireless, wired, and high latency services). These Internet skills gaps are exacerbated by the addition of cloud services, virtualization, and ‘user-driven’ environment driven by the “app.” Key architectural principles like low latency networking, render time optimization, and big data analytics are competitive advantages for “Internet-oriented” businesses. Software Defined Networking (SDN), Network Function Virtualization (NFV), and OpenFlow mean that all “telecommunications companies” need to add UNIX to the core skill set for the majority of that stuff. The cost and operational savings of network service convergence, where optical networking is directly connected to IP devices (routers and switches), means all the traditional transmission teams need to synergies and merge with the IP Backbone teams.
Lessons from the past – two paths to success
Unfortunately, telecommunications skills gap during a massive technology shift is not new. Today’s skills gaps are very similar to the emergence of the Internet in 1994. Many companies scrambled looking for the right talent to fulfill their ranks. There was a high dependence on the vendors to help. What worked? Only two paths where successful:
Invest in the Team – Hire a Coach. The first path pulls in a selected expert who now has the capability to coach, mentor, train, lead, and build a team. Technology might change, but the corporate culture, practices, and procedures of the company are organizational wisdom that is crucial for execution. These coaches are “experts” on the Internet, but they have the additional character of investing in their teammates. Organizations who use this approach were able to ride the waves of Internet change in 1994-1999.
Build a brand new Team (Startup Mode). Many of the companies that dominate today’s Internet – Google, Facebook, Twitter, Juniper Networks, etc. built teams from scratch. These teams were crafted – with the leadership seeking a combination of skills that would meet their company’s expectations. Some of these companies were startups – scaling like any normal startup. Others selectively hired the best for a focused team to deliver a focused product. Other companies build “startups” inside companies – building an independent unit that had the freedom hire the best team for the job. This “inside startup” would then be the foundation for transformation.
Which of these are best for 2014? Either would work. What would not work is a rapid hiring of perceived talent without a plan. Both of these modules from the mid-’90’s had a core factor in success – emphasis on team skills. It is not just the knowledge; it is the ability to work with peers in the organization – be it a specialized team or a team where the “expert coach” is investing in his/her peers.
Suggestion: Rethink the Organization Now to Prepare for the Future Changes
Don’t wait for until the last minute to realize there is a skills crisis. Investing in the existing team and finding the right talent to fill in the skills gaps takes time. Time is not on the operator’s side. It would be worth the time to example the full organization and re-think how the team structure and skills inventory needed to succeed over the long term (the next five years). The following are observations and suggestions to help CXOs understand some of the requirements.
Converge the Transmission and the IP Backbones Teams
The Transmission engineers need to learn IP, TCP, MPLS, and all the routing protocols. The IP Backbone engineers need to learn about all the microwave, optical, and transmission technology. Both teams need to learn about the new ways of doing networking. Orchestration, Netconf/Yang, open flow, and full network visibility are changing the way we do networking. Those companies that transition will save on CAPEX/OPEX – impacting the overall margin of the network. Those who keep the traditional telecoms model will bear the cost of their slow adaptation to the new paradigm.
Does this mean the telecommunications company needs a new team? No. It needs to merge the teams, invest in the cross training in the current technology, and aggressively drive people to learn the new models of IP – TX networking. Finding a seasoned IP network engineer who has done both transmission and IP to be a coach would be optimal.
Transform the Peering Experts to Multi-Service Interconnections Team
It has always been hard to find experienced Internet Peering specialists. These are IP engineers who are experts with Border Gateway Protocol (BGP), Internet Exchange Points (IXPs), Netflow, and the full knowledge of how the global Internet is glued together. Today’s “Peering Specialist” will be managing multiple services – voice (VoLTE), streaming services, cloud interconnects, and the traditional Internet peering. Each of these peering systems will have inter-company service level agreements (SLAs), Quality of Service (QoS), and contractual Key Performance Indicators (KPIs) that govern each of these services. The days of the “best effort Internet” are fading. Today’s Internet consumer seeks a quality of experience that can only be delivered if the Peering engineer is crafting quality on all the interconnected services.
Where to find these peering engineering? NANOG, APRICOT, RIPE, SANOG, and the private IXP forums are the best place to find these engineers. Expecting an engineering with some knowledge of IP networking to be a peering engineer does not work. This is a unique community of engineers who understand how to interconnect services on the Internet world. New talent is “mentored” by the community. Will the traditional peering engineers be able to cope with the new service peering? Yes, most of the seasoned peering engineers already understand how to interconnect voice, multi-provider VPNs, and video services. It is a step evolution for VoLTE, Cloud, and application peering. The one thing the telecommunications organization should consider is breaking the Peering and Interconnection Team into a separate planning – operations team. Allow them to focus and serve the needs of the entire organization.
Invest in an Internet Services Team to Create and Operate the “Private Cloud.”
For most telecommunications companies today, “Internet Services” are the over the top (OTT) companies that their customers are using. While OTT “Internet Services” are OK, they are driving the telecom company to be a “dumb pipe.” Paraphrasing Nicholas Negroponte, “the Internet is about changing how we live.” If a telecoms company want to be in the forefront of a long-term meaningful relationship with their customers, then they need to invest in their own Internet Services infrastructure. The specifics for what services need to be included would be covered in other articles. What is needed first is a team who understands how to build internet services. These are system architects who understand the TCP/IP End-to-End paradigm. They are Unix experts who will craft a telecom company’s private Infrastructure As A Service (IAAS), Software As A Service (SAAS), and Platform As A Service (PAAS). Combined, this creates the private cloud for which the “Internet Services” will be developed and deployed.
This is one area where telecom companies will need to build special teams who focus on just this service. They will be talent that would be pulled from enterprise networks in the finance and manufacturing sectors who have already deployed cloud services.
The DCN/EPC Team changes to Virtualized Core Services
For 3G/4G telecom operators, their “core team” will be pushed into a realm that is unexpected. The “core” of a mobile network is being virtualized. The combined work of ETSI’s Network Functional Virtualization (NFV) and the IETF’s Network Service Chaining (SFC)working groups are turning “specialized” hardware into modules on a “private cloud.” The 3G/4G core teams do not have the Unix skills to manage the virtualization transformation. Of all the skills gaps that will be most impactful in telecommunications, the lack of Unix skills will overshadow all other gaps.
Voice Circuit Switch Engineering turn into LTE – IMS Services Engineers
“In the future voice will be free.” When John Chambers proposed this prediction, most people in the telecommunications world dismissed it as insane. Today as data revenues on 3G/4G systems dominate the balance sheets, the prediction that voice has little to no consequence is coming to fruition. Voice is just another data service. Some would say that operators have choices. The regulatory requirements dictate that voice will be a basic service – a service essential to telecommunications. Given this, the recommended path would be to migrate to VoLTE as fast as possible. The faster a telecom company can move voice as an essential service on an IMS system, the better for the long-term cost of maintaining voice systems.
The mix of skills will be a challenge for future Voice, VoLTE, and IMS service support. IMS systems are the first to be virtualized to private clouds based on VMware or OpenStack/KVM. Future voice engineers will need to know as much about virtualized systems, elastic capacity modeling, and how that interacts with VoLTE. They will also need to know everything today’s voice engineers know. In addition, they will need the foundation Internet – TCP/IP knowledge set. This means operators should look for solid Unix systems architect who wishes to learn VoLTE/IMS to supplement the traditional voice team. This combined knowledge set will provide the best of traditional voice along with a foundation for the virtualized future.
Video Engineers – A Unique and Specialized Area
As seen throughout the industry, video dominates consumer interest. Be in a small screen, large screen, over the top (OTT), on-demand, or specialized broadcast. Video stream engineering is a specialty on its own. These engineers not only need to understand the full dynamics of a TCP/IP network, they need to understand all the interactions with transcoding, caching, transrating, buffering and adaptive bit rate (ABR). These all interact with various clients and middleware to provide the best quality video experience.
These engineers are recommended to be in a team on their own, integrated with the planning and operations team. Video engineering is unique skills that need to be cultivated over time. These skills are scarce, so finding a specialize-coach that would supplement and invest in an existing team would be the most effective way to scale.
Cyber-Security Teams are a Permanent Cost of Business
The cyber-security threat is global. A criminal in a country on one side of the planet can victimize someone on the other side of the planet. There is little to no deterrence to resolve this threat. Cyber-crime is profitable with minimal consequence. Consequently, operators have to re-think their security requirements. Security cannot be an “afterthought.” It must be a “forethought” integrated into all designs and operational plans. DDOS attacks can be commissioned in the underground market to deliver tens of gigs of attack for as little as $240 US a week. What is the cost of a telecoms business that is down for a week from a DDOS attack?
Operators are recommended to invest in security staff in their planning team, a penetration/test team, a compliance team, and an operations team. A future article will detail factors to consider when building out these specialized security teams.
Commit to a Domain Name System (DNS) Specialist Team
The DNS system is critical to the operations of all telecommunications. Lack of the core understanding of how the DNS systems works is a core Internet skills gap. 20 years ago, DNS was just needed for DNS authoritative and resolver services. Today DNS is used to find elements of an LTE system, route phone calls, interconnect voice systems between operators, and provide a way to interact with an IPv6 infrastructure. No DNS means no services – voice, video, or the Internet. Slow DNS means poor customer experience.
Investing a DNS specialize is wise. These DNS specialists could be integrated with other teams. They can also be grown from within the organization. The DNS operations community is easy to connect and participate. Forms like bind-users are open to any question from any DNS engineer at any level. What it takes from an operator is to select engineers who are focused and accountable for the robustness and low latency resolution of the DNS architecture.
Open Source – Changing the Economics of Operators
Open Source is already deployed in critical deployments throughout the Internet. The open source community is evolving, using Agile approaches and sustainable funding models to maintain and improve the code. It is logical for an operator to use open source in their production systems. The performance, feature velocity, and lower cost will drive operators to deploy open source for competitive advantage.
Operators need to prepare all their teams for the open source revolution. Find some elements of open source to use in their network in non-critical roles. Prepare the finance, procurement, and other teams to support the open source with annual maintenance contracts. More on the business advantage of open source in telecommunications will be covered in future articles.
Databases, Big Data, User Data Convergence (UDC), and Data Management
In the past, the only databases in a telecommunications organization were in the user authorization and billing. Today, data is collected throughout the network. Even as databases are converged (i.e. the UDC theme in 3G/4G), most telecom companies lack skills for basic database administration. There is an over-reliance on the vendors with their database management.
Going forward, an operator needs to consider focused data base administration and system architecture roles. These would be integrated with various parts of the organization. They should be immersed in the high-speed SQL models as well as the movement to no-SQL and other new “big data” modeling.
Network Management teams transform to modeling, simulation, and predictive planning
Yesterday managing a network was all about SNMP, Logs, Alarms, KPIs, MRTG charts, and vendor specific OSS/NMS tools that do not work with each other. Tomorrow managing a network will be about standardize APIs, XML based Netconf/Yang provisioning, complete end-to-end service modeling, service simulation over an emulated network, virtualized testing, element abstractions, orchestration test, and big data analytics.
The skills needed by the engineers today will be totally different from the skills needed to run the future network. Telecommunications companies will require “out of the box” thinking to find talent that can do this sort of modeling. This is an ideal case for building a new young team straight out of college who is lead by a seasoned professional that is more coach then “expert.”
Customer Experience – using a New Model to understand the real impact of a Network
Telecommunication business success can be summed up to two KPIs: the customer’s experience and the shareholder’s perceptions. Execution Disciple is critical to the success of any telecoms business. Effective execution disciple has to be attached to the customer’s experience. The old ways of measuring customer experience are not delivering results. The new models measure the customer’s perception of performance. It looks at rendering time, latency, and application responsiveness as more important than bandwidth and throughput. The new customer experience tools will link all end devices into a crowdsourced “cloud” application that is then linked to specific network performance tools. This analytics allows the operator to see what the customers see and then match it to action that keeps the network in an optimal balance.
The engineers needed to fulfill this role need to be cross-functional. They need device knowledge to put the tools that will measure the customer experience (cell phones, M&M, client modules in set-top boxes, etc). They will need the Unix, big data, and analytics tools to collect and process the data. They will need the cross-functional knowledge throughout the network to understand how this data is used to optimize the network. Finally, they will be the technical arm of the operators “customer experience” team. This team can be a separate unit or attached to another unity (customer service, network, sales, or product marketing).
Invest in a Labs and Tools to do the Empirical Studies
The next wave of engineers requires the tools in the lab to do the empirical investigation in all of their respective areas. Telecom companies that as seen as successful (i.e. they make money) all heavily invest in lab resources that facilitate their investigations. The new wave of technology will mean a shift in the lab tools. The traditional “Unit Under Test” UUT will be supplemented with systems/applications testing. SDN suites will need to be set up in the lab to test and certify new applications before they are loaded on production clusters. New techniques to stress test the virtualized architectures will need to be devised. It is a misnomer the think that “virtualized” means that system engineering is eliminated. Find creative lab engineers who can help all the different departments would be an idea. These would be engineers who express a desire to learn, are able to “context switch” to new technologies, and enjoy multitasking with multiple projects.