This is part “2.1” of a multipart post to help organizations take security action. Stay tuned for next week’s practical security checklist item.
Board members, CxOs, and professionals are saturated with security advice. This security advice is often confusing, contradictory, and always biased toward “buying something.” “Good security advice saturation” results in paralysis of action. No security action is the worst thing that can happen to an organization. In a way, the CxOs of the world are getting a “denial of service” based on all the good “security advice” from the industry. The “practical security checklist” series is provided the community as a tool to take action. As seen in part 1, the threat vectors that impact our industry are not going to be solved anytime soon. At the same time, organizations need to explore balanced action that mitigates risk. Balanced action means deploying security techniques that do not require substantial capital investment. For that to happen, the organization needs people to take actions.
The problem is that the industry has not kept up with fostering new security talent. It takes at least a decade of experience to have the background necessary to become an OK security professional. This security talent is not going to appear suddenly.
Hence, the security checklist series is written for the “professional” with little or no security experience. The security checklist facilitates action with any CxO, Network Director, System Leader, or IT specialist. IT provides them a tool to respond when management demands that they “do something about security.” Each module will feature the following to help teams print, plan, and use these modules within their organization.
All of the modules assumes that the talent within the organization can – if lead properly – deploy significant security improvements throughout their organization with minimal capital investment (i.e. open source and UNIX boxes).
Dedicated Security Time is the #1 CxO Security Decision
All of the security checklist items require time to get done. In fact, one of the biggest complaints heard in organizations who have heavily invested in security is the lack of time to configure, deploy and use the security tools. CxOs should consider at least one day a week where a cross function of people in all parts of the organization allocates time for “security.” Time focused on the resilience and security of the network is more critical than any other “security investment.”
Guarantee for the CxO: Instruct your teams to spend one day a week on security will be one the best investment to reduce risk and increase the security capacity/capability of your organization.
Let us start with the first of the checklist items …..
Part 2.1 Have positive control over all elements in your network. Know who is accessing, when they are accessing, and where they are accessing.
Any unknown logins on any element of your network is a warning that something is wrong. Assume a bad actor will get inside your network. Assume there are no walls on your network. Every network element that has a login in your network can be means to get primary access and then be use that access to other parts of the system.
It is important for the team to think like a bad actor. For example, if someone on your team got unauthorized access to an element in the network, what could they do break into other systems? If someone on your team got inside your network, what would they do to “scout” and look around? Could they map your internal network infrastructure? Could they figure out the configuration of the network elements? Could access to a small, unmanaged ethernet switch on the edge of the network be used as a launch pad for other miscreant activities? These “edge switches” might have been seen by the team as not needing the “secure configs.” Once you are on this switch, it becomes the first step in for the “back actor” to slowly “own” your network. The bad actor can hop around the network until they have total control.
How do you find these vulnerable/misconfigured leafs nodes? First, start with the basics. Set a policy throughout the organization that requires “secure configuration templates” for every node in the network. The secure configuration templates would be a strict policy for all elements in the network. This is not new. Many of the security standards require this practice. The problem is that most organizations do not drive this requirement. Second, find an internal scanning tool to check the each node on the network. The network/security/systems teams are recommended to start with open source tools to scan & audit. While there are good commercial scanning tools, open source tools provide the team with insight as to what they need to make their jobs easier in their environment. Surprisingly, tools like RANCID (http://www.shrubbery.net/rancid/) have been hacked and used to monitor multi-vender architectures. Lastly, consider the new functionality being developed in “orchestration.” Managing thousands and tens of thousands of elements on a network is just not feasible using the tools from the past. The new wave of orchestration tools provides new options to ensure every element is configured consistently throughout the network.
Example Requirements to Consider
- AAA for centralized login. Authentication, Authorization, and Accounting (AAA) is critical for every element on the network. The AAA solution might be TACACS+, Kerberos, or Diameter (or some other tool). Notice that Radius is not listed. If you assume that the bad actors are inside the network, then you cannot assume the path between the network element and the AAA server is “secure.” Encryption between the network element and the AAA server is a must. Radius does not encrypt the session between the network element and the server. Hence, it is a must to push your vendors to support something other than “Radius” to provide AAA support.
- Mandate SSH to log into the device. SSH support – where your path from where you connect to the network device is encrypted. End-to-End encryption is an imperative. It is more than “basic.” It is also one of the things that seen as “mandatory” for internal network communications. With today’s threats, assume the bad actors are inside the system and can sniff the links. Hence, SSH is mandatory for all internal communications.
- Require “three strikes and you’re out” password protection. Most equipment should have a configuration option to have a delay timer or a lockout for bad login attempts. These would also be logged into a Syslog or trigger some alarm.
- Set up logging for all login attempts. Logging and accounting for all access attempts a core function of the AAA system. Syslog can also be used. In fact, it should be both? Why? Think of a log level internal DDOS attack either the AAA server or the device that would “time out” the AAA server communications. Most network element AAA servers would then fail over to a local AAA function. That failover is one of the techniques an insider threat can use to get into the network device. If you are logging to the Syslog server and the AAA server, then you would capture details on both.
- Set up a VTY access list for all network elements inside the network. Some think that a VTY access list that limits that IP addresses are allowed to log into the device is something for the “outside elements” (i.e. the network devices exposed to the outside world). If you we assume the bad actors are inside the network, then one must consider that all elements must have VTY access list.
- IPv6 is your friend. Ask all the vendors to use IPv6 to manage the network elements. IPv6 opens options that create a level of difficulty for the bad actors. Please do through testing. Some vendor implementations of IPv6 are not complete – leaving exploitable holes through incomplete IPv6 features (i.e. like no access list).
- Insist on security testing for the new Orchestration Tools. Any tool that has access to all the elements in the network becomes a target of opportunity for the bad actor. Some vendors who build orchestration solutions might say “this is internal to the network, so it is secure.” Again, one cannot assume there are no bad actors already on your network. Assume they are and that they would target the orchestration solution. As a minimum, the vendor should have a penetration test report and a security architecture report that they can provide to help the organization determine the risk/benefit of the orchestration solution. This 3rd party testing would supplement the organization’s internal security testing.
- Two-factor “token based” authentication. All the AAA solutions should have options to integrate a token based two-factor authentication. Two-factor authentication systems replaces the passwords used to access elements on the network, limiting risk to the person holding the token and/or having the password to the token.
Team Exercises for Positive Control over all Network Elements
Keeping to the theme of allocating time to practice, here are some “positive control” exercises. Team leaders can use these exercises in their weekly “day of security.”
Team Exercise: Everyone is a bad actor – attack tree exercise
Tools Needed: Whiteboard with multi-color markers and an overhead projector with a computer with Freemind loaded (http://freemind.sourceforge.net/wiki/index.php/Main_Page).
Team Leader: The Team Leader is the facilitator. The leader would be at the whiteboard asking and promoting questions.
Freemind “Driver”: One member of the team would be using Freemind to transcribe that attack/threat vectors from the whiteboard into a mind map. Freeman is one of the most flexible and useful tools to quickly build attack & threat vectors.
Exercise Phase 1. The team all assumes that they have entered into the network through some point. This could be through malware, phishing, or some other source. Start with one of several starting points. For example:
Bad Actor has used spear phishing to “own” a computer in the network operations center.
Bad Actor has used a supply chain violation to load malware into new equipment installed into the network.
Bad Actor has used several techniques to “own” one of the sysadmin’s personal devices used to connect and manage the network.
More “starting points” can be considered. The key is to pick one, and then ask the team what next? What would they do if they were the bad actor? How would they expand their reach into the network? What would they do to keep from being discovered? Each of these “tricks” to expand throughout the system would be mapped on the whiteboard and transcribed into the attack tree (using Freemind). Writing it down for everyone to see enables more “dirty trick” ideas.
It may take some time to get the team into the flow of thinking like a “bad actor.” At other occasions, it would be hard to get the team to stop, as they find more and more ways to violate the system (a system that is their system). With attack tree exercises, more is better. The more items listed, the more “attack vectors” will be exposed.
Exercise Phase 2. Once the attack tree is built, the next item would be for the team to step back and explore all the ways that could be done now. The team must start with the tools that could exist now. It is too late to go out and buy new “security tools” in the middle of an incident. Open source security tools can be used. These might be quickly downloaded and deployed.
It is important to have the team think within the existing limitations. When security incidents happen on a network, remediation is conducted with the tools that exist now. There is no time to rush out and buy need equipment. At the end of this phase, the team will have a good understanding of what can be done with the tools on hand along with additional open source tools that can be downloaded and used.
Exercise Phase 3. Build a team action plan. The team has an attack tree that maps out the risk (thinking like a bad actor) The team also has a list of tools that could be used to spot the attack vectors. There is now enough information to facilitate an action plan that can be used to regain control of the network.
There one additional element to consider as part of this plan. What if your network is already owned? What if the bad actors might already be on the network? This “assume we are already owned” is a prudent frame of reference for the team to build their “recovery plans.” It would assume the bad actors can disrupt or watch the recovery plan. Any new configurations might be seen by the bad actor.
This “assumed we are already owned” is common. Outside security consultants would walk into security incident with the assumptions that a massive “advanced persistent threat” (APT) has already violated the network. This is not the end of the world. It just takes careful and creative thinking. For example, as the team deploys tools to lock down the network, they discover the “APT” actors have bypass the “lock downs.” The bypasses might range from the bad “APT” actors having privilege username/passwords, to adding rootkits, to replacing the RMON code. The key is to assume the bad actors are already on the network. It will mean more work as the action plan is deployed, but it is safer in the long run and good practice for the day when the network does get “owned.”
Team Exercise: Someone is attacking our AAA from Inside our Network!
Situation Report (SITREP): The NOC Team notices that when they try to log into a router, the username “rolls over” to the backup local username/password. They do some checking on other switches and routers and notice it is happening on all of them. When they look at the AAA servers, the notice that the system is slow and unresponsive. Further checking shows that there is a low-level DDOS attack on the AAA with several computers sending a mix of TCP & UDP packets on the AAA ports.
This exercise would happen in two phases. The first phase would be to teach the team to think like a “bad actor.” The second phase would have the team figure out ways to remediate the problem.
Phase 1: Think like a “bad actor.” The full team would walk through the AAA architecture of the organization and explore all the different ways that could be used to knock out the ability of the server to respond to valid queries. This would be an internal DOS attack. The objective is to come up with as many ways that can be used to get inside an organization, yet have a specific target in mind.
Phase 2: Find, Defend, Restore the Service, and Look for the Real attack. In this exercise, the whole team would need to work together to recover the AAA service. The team would need to deploy tools to find the source of the internal attack. They would then contain/remediate the attack, start the forensics on the attack, and then look for the real attack vectors. What many people don’t realize is that many “noisy” attacks are intentional distractions whose function is to bog down the team while the real attack is in progress.
Team Exercise: Violated Privilege Credentials
Situation Report (SITREP): John Doe, the network engineer, has been logging into the system at 3 am. The problem is that John Doe is on vacation with no network connectivity. Someone is using John’s credentials on the system!
Tracking down violated credentials is an exercise every organization should walk through. It demonstrates the painful consequences to the organization if someone loses control over an username/password. It is a valid tool to show the value of two-factor authentication and token-based systems.
Phase 1: Think like a Bad Actor Gaining Control over Privilege Credentials. The full team builds an attack tree on how they would convince someone in their organization to “give up” their username/passwords. This includes attacking the staff on their phones, homes systems, and social media. The key is to get access to a privileged account. The best team to figure this out is the current internal team. They would have the knowledge of how to fake out the culture of their organization.
Phase 2: Remediate “John Does” credentials and find the scope of the damage. The full team works together to list out the approach remediate the penetration, find the entry point of the penetration (how are they getting inside), and then map how the credentials were used throughout the organization. This last part is “mapping an iceberg” of potential violation – looking for all parts where the bad actor used the violated credentials to “explore” and further violate the system.
Where to get more Operator’s Security Information?
The Operator’s Security Toolkit is the place to start for all Operators. It provides details to help them build more security resilient networks.
….. Next Part 2.2 and the Practical Security Checklist …
≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣
➥ Barry Greene is Business Development Executive ★ Internet Technologist ★ 25 Year Veteran of Internet Security ★ Emerging Technology Mentor ★ Advisor to Innovative Startups ★ Internet in Asia Expert
➥ Barry connects to peers, colleagues and aspiring talent via Linkedin (www.linkedin.com/in/barryrgreene/). You can also follow on Barry on Twitter (@BarryRGreene) or his blogs on Senki(www.senki.org).