Vendor Security

Vendor Security – This document has been updated and maintained here: How to Demand Security from your Vendors

Demand Security from your vendors! What security questions are you asking your vendors? The Bloomberg article, “How Russian Hackers Stole the Nasdaq,” is a sobering insight into today’s risk. It should be a wake-up call for all organizations in all parts of the world to understand that even the best security teams are facing an overwhelming threat. The focused expertise used by today’s cyber-criminals is often beyond the capabilities of the normal INFOSEC Team. Today’s economically driven cyber-criminals face little real risk and huge monetary gains. The only way we, as an industry is going to overcome the threat is through more diligent collaboration. This diligence starts with the operator and the vendors. Zero-day attacks (as seen in the NASDAQ infiltration) are only prevented through diligence with both the vendor and the operator doing their part to ensure the product’s code and hardware have been pushed to its limits. What follows is a list of Security Questions to ask Vendors. It will help anyone sound like a security guru.

The list of Security Questions to ask Vendors evolved over the years as a “checklist” for the operator (and the vendor). This checklist can be used in RFPs or with any vendor. It can also be used as a conversation map with the existing vendors to shape the conversation. It will work with service providers, enterprise networks, industrial networks, etc. The checklist also provides a map for new vendors to help them know what customers would expect. Please provide feedback and questions. This checklist will be improved over time. (Article posted on Linkedin here)

Questions to ask vendors to gauge their commitment to “secure products”

Version 1.1

Does the Vendor have a Vulnerability Management Process? Never, ever expect any vendor to have a perfectly secure system. It will never happen. Brilliant economically motivated cyber-criminals are digging into systems looking for zero-days that are beyond the scope of most vendors (and operators and white hat security penetration teams). Expect the discovery of an advanced persistent threat (APT) inside your organization. When this happens, how the vendor responds is critical. Many vendors will dismiss or hide the problem. Most vendors will not have a vulnerability management process (especially new vendors and vendors outside the US and Europe). Do not wait for a vulnerability’s announcement on the news and exploited to harm customers. Understanding the vendor’s response to these crises is critical to the operability stability of the operator. Given that, the vendors are expected to meet the following:

Check the vendor’s website. See if they have a “/security” page with information about how to report a security vulnerability and their security team. Each company should have a public, easily accessible page that allows anyone easy access. For example, www.example.com/security would list the security team details, how to reach them, their phone number, and PGP keys. Too many companies have been blindsided with vulnerabilities reported via “full disclosure” (“full disclosure” is where a ‘finder’ of the vulnerability reports it publicly without coordinating with the vendor). Vendors who do not have this simple public contact list are an indication of their ability and interest in the security of their products.

Check if the vendor has a published vulnerability disclosure policy. The vendor should have a publicly accessible and policy on their corporate vulnerability disclosure process. The vulnerability disclosure process will meet industry expectations for responsible disclosure. This makes it clear to everyone what their vulnerability disclosure policy will be when someone finds a vulnerability and reports it to the vendors. The absence of a vulnerability disclosure policy is a test to understand the quality of the vulnerability response team inside the vendor.

Does the vendor have a Security and Vulnerability Response Team that is available 7×24, 360 days a year in English as the primary language of business? “Availability” shall be via E-mail (using PGP) and phone. Availability should be tested with all vendors. The operator should ask for an initial briefing call with the Security and Vulnerability Response Team. Ask for a run through with all the policies, procedures, tools, and approaches used by the vendor to ensure their products are as secure as possible. That first call will provide a gauge of the vendor’s security maturity.

Does the vendor participate in industry vulnerability and incident response teams? As mentioned, the only way the industry can respond to the aggressive cyber-criminal threat is via aggressive collaboration. No vendor can work in isolation. Collaboration with their peers, their customers, security researchers, and law enforcement has proven to be the best defense. Operators can “test” to see if their vendors are collaborative by checking on their industry participation. For example, all vendors selling equipment and software to the Internet and telecommunication industry should be part of the Forum of Incident Response and Security Teams (FIRST) www.first.org. FIRST is one key industry groups for vulnerability and industry response. Just being a member of FIRST requires a community audit of the vendor’s processes and procedures. If the vendor is not a member of FIRST, ask “why” and “when.”

Does the vendor participate in the national vulnerability and incident response teams? While many vendors might be a member of industry groups, few put the extra effort into the national Computer Emergency Response Teams (CERTs). These groups are as critical to vulnerability management as the national groups. All operators should have some relationship with their national CERTs or other vulnerability coordination bodies. These could be national CERT Teams or special interest groups like that Information Sharing and Analysis Centers (ISAC) community (see http://www.isaccouncil.org/memberisacs.html). The operator’s vendors should have some relationship with these local security and vulnerability response teams. That relationship is an excellent test of the vendor’s security maturity.

Demand the Vendor provide their Security Development Lifecycle (SDL) process. This vendor presentation will normally be under a nondisclosure agreement with the details confidential. But, each vendor with an SDL should have the ability to brief their customers, explain how the SDL process works, explain how the SDL process is evolving over time, and explain how the customer can engage with the SDL process. What should an operator look for in a mature SDL? There are books, papers, and guidelines in the industry that cover best practices for SDLs. Here are some of the core items to seek in your vendor’s SDL:

The vendor’s SDL integration to the vendor product development processes is essential. Security “after the product development” is an afterthought. It never works. Effective SDLs are an integrated part of the normal product development process. Look for the vendor’s SDL presentation to first present their product development process, then how the SDL integrates with that process.

The SDL process should allow for rapid vulnerability fixes. “Zero Day” vulnerabilities are often in the “active exploit” mode. That means these reported vulnerabilities need to have the fixes defined, integrated, tested, and deployed in hours – days – not weeks – months.

Look for the Root Cause Corrective Action (RCCA) capability in the SDL. When a vulnerability is discovered, there are two phases to the “fix.” Phase one is to get the vulnerability fixed to mitigate the potential active exploitation. Phase two uses an RCCA process to find out how the vulnerability got through all the checks. The RCCA is essential to find more potential issues. The RCCA is then used to update the tools, processes, and test to prevent a repeat of the Root Cause. The RCCA goes beyond “regression testing” and looks for the underlying causes, fixing those causes, and making impactful improvements.

Does the vendor’s SDL include regular Static Analysis Testing? How are the results of the static analysis testing used to improve the resiliency and security of the code? Static analysis testing is one of the many tests to check on the quality of the code. There are open source static analysis tools, vendor-specific tools, and commercial tools. A commercial example of static analysis is Coverity (www.coverity.com). Many vendors will use static analysis testing, get overwhelmed with the volume of errors, and take no action. Ask the vendors how many bugs were fixed each quarter; as a result, of their static analysis testing. Remember, this sort of testing is about code stability and security. It catches coding mistakes that could lead to outages.

Ask the vendor if they are using Test Driven Development (TDD) in their software deployment. TDD is an approach to coding use in Agile Development. The essence of TDD is to first code a test for the new function before coding the function. Yes, this seems backward – writing the test before the function. But, the decades of TDD application demonstrates that the disciple of first figuring out how to test the function before writing the function forces quality into the code. The result of the industry code that is higher quality, fewer bugs, and fewer security issues! Ask the vendor if they are using TDD. Seek to find out if they are using it in all their software This will include microcode for their hardware (ASIC, FPGA, NP, etc.) and elements in network equipment (control plane and management plane elements). TDD now has industry consensus to the empirical impact to the quality of software development – specifically with the prevention of security vulnerabilities.

Does the vendor use dynamic testing in the vendor’s development process? Dynamic Analysis Testing (DAST) is different from static testing. While static testing pulls in the source code to look for issues, dynamic testing pulls in the compiled software image. Dynamic testing made huge gains over the last decade. The revolution of virtualized environments makes it easier to load a compelled version of the code into a simulator designed to dynamically test the image. Issues fond via DAST cover security, stability, robustness, and for some performance. DAST will undercover issues that will not be seen with static analysis. There are problems in code that is a result of code interaction. Two modules might interact in a way that causes a vulnerability or instability. These interactions are hard to identify with static analysis but are more easily seen after the code is compelled and tested. Vendors who care about the quality of their code should have both static and dynamic analysis testing. There are now open source tools available as well as tools built specifically by the vendor for their unique environments. A good commercial explains to explore would be Veracode (www.veracode.com). They provide a baseline example of a dynamic testing tool used extensively in the industry.

Does the vendor use the industry vulnerability classification and enumeration tools? The industry puts a lot of effort into common tools to normalize the risk of a vulnerability. These tools include the Common Vulnerability Enumeration (CVE), Common Vulnerability Scoring System (CVSS), Common Weakness Enumeration (CWE), and several other tools being developed. The operator should first ask if the vendor knows about these tools. Then ask who they use these tools in their Security Development Lifecycle (SDL) processes. For example, The CVSS system has been used by several vendors to put a metric on the risk a reported vulnerability has to their customers. The CVSS risk value then drives the coding priority in the company. The higher the CVSS risk, the faster the vulnerability gets fixed, tested, and deployed to the customers. CVSS puts an empirical metric to drive the internal processes. This CVSS value is then used to determine how the vulnerability will be disclosed to operators. The higher the number, the more risk to the customer. A CVSS priority will govern the speed of field testing and deployment of the fixed code. Another example is the use of CWE in static and dynamic analysis testing. Many times these tests produce huge reports. Vendors often do not know where to start. In response, the industry has a top CWE list. CWE metrics allows the static and dynamic testing vendors to put a CWE value to each of their findings. The CWE value can then be used to triage the list of issues and prioritize limited coding resources on the most critical CWE items. Overall, vendors who are engaged and can explain how they are uses these tools in their SDL processes are must more engage with the security of their products.

Does the vendor have Regression, Stress, Performance, and Compliance System Testing? Some might ask “isn’t this basic?” Yes, it is basic. But operators would be surprised to find that major vendors do not have a structured system to test. Any operator who is serious about the resiliency and security of their operations should have details of the vendor’s regression, stress, performance, and standards compliance testing. This information would be presented under an NDA. It is a mark of a mature synergy between a vendor and their customer to have conversations around on the improvement of the testing environment based on their joint operational experiences. For example, an issue found in the network that was missed in all prior testing would be added to the vendor’s regression and stress test. This will ensure that this issue is not repeated. It is also used for other products the vendor launches. How does this test impact security? Most of these test suites are a systems test. They test the product in a production simulation. These systems oriented test is the logical follow-on to the static and dynamic testing.

Does the vendor use of fuzz testing in their quality processes? Fuzz testing pressure test an element in unexpected ways. Normally engineers who write a test are ensuring that the element under test is working as expected. Fuzz testing is the opposite. It is designed to send the unexpected. If an element in the protocol is four bytes in a specific location of a packet, then the fuzz test will send 4 bytes offset before – mismatching the location. It is interesting how systems respond when they receive information that is unexpected. Some of the most damaging security vulnerabilities have been found during fuzz testing. Others vulnerabilities have been found in crash core dumps – where the “bug” that causes the crash could if easily be found if a fuzz tester were used on the protocol. The operator should have a conversation with the vendor if fuzz testing is used and how it is used with their product.

Does the vendor have documented risk assessments for all elements, protocols, systems, and solutions? Part of an SDL is the risk assessment process. Features and functions of the product should include the capability to mitigate risk. How would the operator know if the vendor is performing these risk assessments? Ask for a review of the risk assessment. There are several approaches the vendor can use for these risk assessments. Some vendors like the ITU X.805-risk assessment process attempts to illustrate this in detail. Other vendors use simpler attack trees to explore all the vectors for risk. Some will develop their own methodology that is tuned to their solution’s environment. All of these are A-OK. The core principle the operator should look for is the existence of a risk assessment and how that impact the capabilities of the product.

Does the vendor use specific security testing tools for their Unit Under Test (UUT) testing? There are many open source and commercial security testing tools on the market. These solutions range from full organizations to commercial tools. For example, the Open Web Application Security Project (OWASP) is focused on tools practices, and best practices to promote robust web security. These tools are open source and often free to use. Contrast this to the mature market for specific commercial security testing solutions (Spirent aka Mu Dynamics, IXIA, and Codenomicon, Veracode, Qualys, Cigital, Imperva, Fortify, Trustwave, etc.). These companies are worth interacting with to see how they would test the vendor’s product in an operational environment. Operators with the means can consider using these tools in their own testing and evaluation. Others can review the tools and then ask the vendors to use them in their testing.

Once you understand the vendor’s SDL processes; it is appropriate to ask the tougher questions. These questions will determine vendors who are “OK” with their product security commitment from the vendors who are “all in” in their commitment.

Will the vendor’s Security and Vulnerability Response Team review postmortem on past vulnerabilities? Vendors who have a long track record of security will have a history of vulnerabilities. The vendor’s security team should be willing to review with customers past vulnerabilities, how they were found, what was done to fix, the code, what was learned, and how improvements were applied. This conversation is an excellent measure of the vendor’s commitment to security. As mentioned before, in today’s environment it would be impossible to have perfect security. What can be done is the continuous improvement to minimize as much of the risk as humanly possible.

Does the vendor bring in 3rd party penetration test and security auditors? All security professionals know that the biggest danger is myopic thinking. Internal security teams will fixate on what is known. The internal security team will often miss out on things that might be obvious to someone else. Some vendors mitigate this narrow thinking by investing in their teams. Sending them to a range of security conferences helps them keep a broader view of the risk. But many times this is not enough. Regular 3rd party penetration test and security audits are an effective means to review a vendor’s full SDL process. The operator should ask if the vendor pulls in these 3rd party security audits, how they are used to improve the processes, and if they will continue with the practice. It would not be appropriate for the operator to asked details of the 3rd party findings. What is more important is the level of commitment to investing in these 3rd party audits.

Will the vendor be open to a joint tabletop exercise? Tabletop exercises are tools used in the security incident response community to test the processes and procedures. Operators asking their vendors to be part of a tabletop exercise is critical to the robustness of their operations. For example, one large online website did a robust tabletop exercise with all their vendors where they simulated a combined DDOS attack with the exploitation of a zero-day attack. The exercise tested all the processes and procedures. Lessons were learned. All the processes were updated. Nine months later this large online company’s alarms were triggered a week before a major news event. They knew they were going to get hit with a major DDOS attack. They also saw the scans looking for what could be a zero-day attack. This vendor activated their emergency response procedures, pulled in all the vendors, and prepared for what was going to be a major attack during a major surge of customer traffic. As it turned out, all the preparation mitigated the attack, found a vulnerability, fixed the problem and maintained services throughout the customer surge. This is one of many examples where an operator preparing with compressive tabletop exercise were able to keep services operating at 100% in the middle of a major attack. It works. Ask the vendors if they have the willingness and capabilities to participate.

Will the vendor be willing to open their code to an operator-driven source code audit? What? Why would Operators (our customers) want to look at our source code? Why would any vendor allow that to happen? Simple. The cyber-security risk is not just a cyber-criminal risk. It is now a nation-state risk. Vendors and governments are now asking their vendors to open their internal processes to operator driven reviews. These reviews are under very tight NDAs with additional contractual agreements to protect intellectual property and liability risk. These operator driven audits are happening in the industry. The vendor should expect their customer to ask. Operators who are worried about Nation State cyber-security risk should first do all of the above, then build the capacity for this type of security audit.

Finally, look at the vendor’s Acceptance Test Plans and Solutions Certification Plans. Does the vendor include security as part of their acceptance – certification test plans? Vendor deploying their solutions are driven to get the solution up, running, tested, and in production. Testing for security is normally not part of the acceptance testing that the operator nor vendor require. Security is an “afterthought.” It might be months later when the security team come into the review and find major configuration issues that have left the solution vulnerable to attack. One of the truest tests of a vendor’s commitment to security is to see it practices in all parts of the company in all phases of the product’s lifecycle. Even with the deployment teams who are working hard to get the system up and running.