How does any organization have productive and meaningful security conversations? This guide offers a simple and meaningful security conversation guide. These conversations would help the organization determine the real security risk from their vendors. This is an updated version of a set of questions Operators (and vendors) can use to have these meaningful conversations. With it, anyone will learn how to demand security from your vendors.
Version 1.7
This document is a tool for operators to have meaningful security conversations with their vendors. Today’s ‘security reality’ is harsh. The approaches used to secure our networks are failing. If we want to make progress, we need new meaningful conversations between vendors who sell “security solutions” and their customers who deploy/operator security solutions. This document provides two conversation tools for anyone who operates a network (the “Operator”). The first is a set of questions that be the foundation of a vendor-operator dialog. The second is a list of “Request for Purchase (RFP)” questions that can be put into an RFP. Both “conversation tools” can be used by anyone with an interest in security. No security expertise is required! Anyone can use this conversation guide to have a meaningful security conversation with your hardware, software, cloud, and any other vendor.
“Vendor-Operator” Security Conversations. The first tool is a list of guided “security conversations” questions. These questions can be used with an organization’s vendors. The key is to have “meaningful conversations” that clears the path for insight, knowledge, and action.
-
- These conversations help the organization learn how their vendors would serve their security needs.
- These conversations are not one-sided “beating up on the vendor” sessions. The vendor benefits from these security conversations, creating a reason to take time to do the security work that helps all their customers.
- The security questions also act as a “guiding principles” tutorial for both parties, providing a conversation template that can be traced to international standards.
- None of the questions listed are unreasonable. They are all based on two decades of security conversations in the community. There is no point in asking vendors to do something that does not fit with their business processes. All the items listed in the guiding principles section are things that have been done in vendors who are serious about security. There is nothing in the list which is out of scope or cannot be done.
- All the items in the guiding principles are will enhance the vendor’s products and solution – benefiting all their customers. So if there are items that require new investment, that investment will benefit the vendor’s entire ecosystem.
Vendor Security RFP Questions
The second tool is a list of RFP questions. RFPs are powerful tools to drive change in a vendor. All the requirements listed in an RFP are items that vendors have deployed in their operations. Like the guiding principles, these RFP requirements are not asking vendors to do anything which is out of the scope of what they should be doing to secure their products. Any requirement that needs additional vendor investment will benefit all the vendor’s customers.
Many organizations would have a growing and evolving list of RPF security questions. This guide is a foundation to get organizations started on their own “RFP security questions template.”
Using Meaningful Security Conversations Document
This document is one chapter of a new book on practical security. This is a living document, with new versions provided to help Operators have those conversations with vendors. It can be used by clients of Senki and Barry Greene for their RFPs and vendor interactions. While this document is reviewed by the industry’s top security professionals, some of the best ideas come from new people who are using these conversations in their own organization. Feel free to send suggestions. If you have suggestions, please email Barry Greene bgreene@senki.org.
“Meaningful Security Conversations” – Conversation Guidelines
The appropriate role of these questions is for meaningful security conversations. They are designed to instigate a productive security dialog between the vendor and their customers. This security dialog will enlighten new information while uncovering the security gaps. Both parties will benefit, with the hardware, software, cloud, or application vendor learning about the Operator’s security environment. More importantly, the Operator will learn what security actions are and are not happening with their vendor ecosystem. A more resilient system can only be achieved through the dialog of discovery and action.
It is Monday morning. The CISO gets a phone call from the CEO. “Why is vendor X security vulnerability all over the news. Some hackers got into other networks. Are we next?” The CISO contacts their staff, asking “what is our risk?” The answer is uncomfortable, the vendor is tied up with phone calls and says they will get back to the team. The clock is ticking. It is only a matter of time before threat-actors have their fun and risk the business.
Security issues are not an “if,” they are a WHEN. Security conversations need to start now and continue regularly. Do not be surprised when in the middle of the conversation a big security issue is discovered. Hopefully, the security issue would be discovered through the confines of the conversation. The key is to push for the conversations now – not later after the security damage has impacted the business.
Security Discovery is the process of asking questions from all your vendors. Each vendor conversation leads to new insight. This dialog will be an on-going habit that should be made part of your normal organizational DNA.
Remember that action is the objective. Security only happens through action, both on the vendor’s part and the operator’s part. Both sides need to look at the insight gained through the dialog and take action.
Security Conversation Guiding Principles
Anyone can lead these meaningful security conversations. You do not need to know much about security to get started. Trust your knowledge of your business. Trust your “BS meter.” You can tell if your software, hardware, cloud, application, or service vendor is trying to ‘change the conversation’ away from answering the questions. Read through the guiding principles. You will see that they are common sense. Yet, through experience, these common-sense principles have been a core tool for effective security conversations with vendors. Feel free to email @ bgreene@senki.org if you have questions.
Get Executive Approval to Commit the Time to Invest in Security Conversations
The #1 security action a CxO can make is the allocation of time for their existing team to invest in security. It does not take millions of capital dollars to make a significant impact on the organization’s security posture. It does take time allocated from people’s workday to invest, converse, learn, and act on security knowledge. Get the CxO approval to have these security conversations.
“How much time does your executive team carve out for security conversations?” is a question I ask Organization under security distress (i.e. attacked). “We never get time to focus on security. There is always some other priority.” That is the essence of organizations getting breached, attacked, and penetrated.
TIME is the #1 investment needed to secure a network. Not Money. Not Security experts. Not fancy security products. TIME to have meaningful security conversations is the #1 job of leadership If leadership does not carve out the time, then leadership is security negligent.
Ask the Security Questions in Writing
Start the conversation by requesting in writing to your vendor with the first questions, the meeting time, and the meeting location. The objective should be a partnership conversation. As hinted throughout this document, there are no perfect security solutions. What is important is the dialog and action.
Be mindful that for most “security conversations” in the world, both parties would have English as their second language. Security is too critical for issues to be lost in translation. “I told the vendor that this security issue is important” was a reply I got from one of the teams reporting to me. When talking to that vendor, I found that what they understood was totally different. Once I had my team “write down the security questions,” then have the conversation, the vendor’s understanding of the concerns dramatically improved.
Non-Disclosure Agreements (NDAs) Might Be Required by Both Parties
What the Operator tells the vendor will be confidential. What the vendor tells the Operator will be confidential. The dialog and information exchanged shares details that are best controlled so that the bad threat actors do not have access. Mutual NDAs are not required but are also not to be surprised. What is key is that both parties view the exchange within the understanding that the threat vectors are real and impact each of their customers.
Security Conversations are Healthy Security Habits
Set up regular meetings with your vendors to have these conversations. These security conversations take time. Often, both parties – the vendor and the organization – needs time to prepare the materials. Other times both parties require time to think about what was presented. But, both parties need to be committed to a consistent dialog. The big organizational change comes with persistent and consistent action. It is recommended that regular weekly or bi-weekly meetings are set up until there is an agreed resolution.
Remember the goals – understanding the security risk and taking security action. For example, you sit down with one of your vendors to discuss “if I report a security vulnerability to you, what is your step by step process?” The scary thing is that most vendors will not know how to respond to those questions. They may not have a process to managed security vulnerabilities. That is OK. What you do then is ask “at what time can we expect you to build that vulnerability management process and present to us the procedure so that we can report security vulnerabilities.” That takes time. This is why it is best to turn the security conversations into regular meetings. Make them security habits.
Use all the best practices for Effective Meetings
Set a clear agenda before the meeting. Write down the security questions and make them part of the meeting invitation. Ensure the NDA is signed before the meeting. Designate a note-taker to publish notes. If possible, designate a “facilitator” whose job is to facilitate the dialog (and write on the whiteboard). Share copies of the materials before the meeting (not after). Ask the vendor to share their materials before the meeting (not after). Ask everyone to allocate time to read and review the materials provided by the vendor before the meeting. All of this and other “meeting BCPs” are important to gain the most from these security dialogs. Effective meetings are what make a dialog into a meaningful conversation. Remember, find the security action items and determine who does what by when.
Get Responses in Writing from Your Vendors
The writing process is not a burden for the vendor. The questions asked in the checklist apply to all their customers. Every question asked is a question that would be asked by other customers.
You ask, “Do you have a process for reporting security vulnerabilities?” The vendor replies “yes, we have a security vulnerability process.” You reply, “OK, can you walk through the process?” They reply “sure” and start talking. Two minutes into their explanation, you get a sense that they are “fudging” the answer. You ask clarifying questions which then has inconsistencies.
This is normal. Do not be surprised. The vendor will do their best, but for most, they have not had these meaningful security conversations. Push for written responses. Written security responses will take longer to get accurate information. But, accurate security information is much better than “security BS.”
Do not Expect Security Perfection
No vendor has every security question covered. If they do, then something is wrong! There are always big security gaps. Large mature vendors like Cisco, Juniper, IBM, and others have huge security teams covering all aspects of security and even they will not have all of these “security conversations” covered. That is why we have them as “conversations” vs “requirements.” The security conversation does not expect perfection. What it does expect is honesty under NDA that allows both parties to understand the security risk.
Remember, these conversation questions are a collection of deployed expectations based on multiple vendors, ISO standards, IETF Guidelines, FIRST guidelines, and other materials. Given this, it would be a surprise to find any vendor who has all items in this conversation list covered.
Ask for the Vendor’s Security Resolution Plan
You will find security gaps. You will be shocked and surprised at the lack of security from a huge range of software, hardware, cloud, application, and other vendors. Remember, informed security action is the desired objective with all of these conversations.
The vendors have a responsibility to meet your security expectations. The alternative is to accept the business liability for their security issues (which is not going to happen). Working with the vendor to determine how they fill in the security gap is a joint conversation.
Some gaps may take a long time to rectify, which is OK if the vendor is committed to change. The resolution plan is the most important part of the dialog. It is a document that leads to action within the vendor. If several of the vendor’s customers are asking for the same thing, the result is a business case for action within the customer.
Build an Operator Action Plan
What are you going to do in your organization? Remember, you choose these vendors. You picked the software, hardware, systems, cloud, OTT, application, and other services in your network. These security conversations with vendors are tools of discovery. Be mindful! The ultimate responsibility for the safety and security of your business is YOU – NOT YOUR VENDORS!
Use these security conversations to learn about security. The conversations will help you discover unknown risks, find solutions to those risks, and deliver a better, safer, and more secure service to your customers.
Be Ready for the Operator to Ask for Your Security Response Plan
Through all of these conversations, be aware that you also have a responsibility to build your own reaction plan. What happens if someone in your organization discovers a vulnerability? If you need to do a rapid upgrade of software, how would you do that with minimal impact to the business? How do you track software versions throughout the organization?
For example, a vendor’s security team would need to call the Operator. Who in the Operator do they call? What actions will that team take to remediate the issue? If an emergency patch is required to fix a security vulnerability, does the Operator have plans to push through “emergency patches?” These will be the sort of actions that the Operator will need to implement to leverage the full benefit of these vendor-Operator security conversations.
These meaningful security conversations are a partnership. Both sides “handshake” on knowledge, action, and process delivering a safer, more secure, more resilient service to customers.
Modeled Security Conversations
Each of these security conversations is crafted to help anyone be the “security guru” without any intimate security knowledge. The “ask” would provide the vendors with the prompted questions which will evoke a meaningful response.
Conversation 1 – Review the Vendor’s Vulnerability Management Process
The first phase of the “security conversation” discovers the vendor’s “crisis planning.” How will the vendor respond to security vulnerabilities? Remember, there is no possibility of creating a totally secure system. Vulnerabilities and risks will be found. How does the vendor respond when vulnerabilities found? Hence, the first conversation organizations need to have with their vendors revolves around their vulnerability management process.
How a vendor will react to a vulnerability reported and facilitate the deployment of the fix is a key element for all organizations. Vendors with immature vulnerability management processes are a risk to the organization. These meaningful questions will be a facilitation tool base on many national-level security requirements. There is no question meant to trap the vendor. Many vendors who have never considered these questions will be anxious, but that is part of their growth from security adolescence to security maturity.
Remember, the organization does not need security experts to ask the questions. The response from the can be measured with common sense and knowledge of the business.
Q. Does the Vendor have a Vulnerability Management Process?
Never, ever expect any vendor to have a perfectly secure system. It will never happen. Brilliant, economically motivated cyber-criminals are digging into systems looking for zero-days that are beyond the scope of most vendors (and operators and white hat security penetration teams). Expect the discovery of an advanced persistent threat (APT) inside your organization. When this happens, how the vendor responds is critical. Many vendors will dismiss or hide the problem. Most vendors will not have a vulnerability management process (especially new vendors and vendors outside the US and Europe). Do not wait for a vulnerability’s announcement on the news and exploited to harm customers. Understanding the vendor’s response to these crises is critical to the operability stability of the operator. Given that, vendors should be able to present to their customers their vulnerability management process. The first conversation starts with the vendor presenting to their customer explaining their process. This provides a baseline for all the subsequent conversations.
Q. Does the vendor have their “reporting security vulnerabilities” on their website?
See if they have a “/security” page with information about how to report a security vulnerability and their security team. Each company should have a public, easily accessible page that allows anyone easy access. For example, www.example.com/security would list the security team details, how to reach them, their phone number, and PGP keys. Too many companies have been blindsided with vulnerabilities reported via “full disclosure” (“full disclosure” is where a ‘finder’ of the vulnerability reports it publicly without coordinating with the vendor). Vendors who do not have this simple public contact list are an indication of their ability and interest in the security of their products.
Q. Does the vendor have a published vulnerability disclosure policy?
The vendor should have a publicly accessible and policy on their corporate vulnerability disclosure process. The vulnerability disclosure process will meet industry expectations for responsible disclosure. This makes it clear to everyone what their vulnerability disclosure policy will be when someone finds a vulnerability and reports it to the vendors. The absence of a vulnerability disclosure policy is a test to understand the quality of the vulnerability response team inside the vendor.
Q. Does the vendor have a Security and Vulnerability Response Team that is available 7×24, 365 days a year with English as the primary language of business?
“Availability” shall be via E-mail (using PGP) and phone. Availability should be tested with all vendors. The operator should ask for an initial briefing call with the Security and Vulnerability Response Team. Ask for a run-through with all the policies, procedures, tools, and approaches used by the vendor to ensure their products are as secure as possible. That first call will provide a gauge of the vendor’s security maturity.
Q. Does the vendor participate in industry vulnerability and incident response teams?
As mentioned, the only way the industry can respond to the aggressive cyber-criminal threat is via aggressive collaboration. No vendor can work in isolation. Collaboration with their peers, their customers, security researchers, and law enforcement has proven to be the best defense. Operators can “test” to see if their vendors are collaborative by checking on their industry participation. For example, all vendors selling equipment and software to the Internet and telecommunication industry should be part of the Forum of Incident Response and Security Teams (FIRST) www.first.org. FIRST is one key industry groups for vulnerability and industry response. Just being a member of FIRST requires a community audit of the vendor’s processes and procedures. If the vendor is not a member of FIRST, ask “why” and “when.”
Q. Does the vendor participate in the national vulnerability and incident response teams?
While many vendors might be a member of industry groups, few put the extra effort into the national Computer Emergency Response Teams (CERTs). These groups are as critical to vulnerability management as the national groups. All operators should have some relationship with their national CERTs or other vulnerability coordination bodies. These could be national CERT Teams or special interest groups like the Information Sharing and Analysis Centers (ISAC) community (see http://www.isaccouncil.org/memberisacs.html). The operator’s vendors should have some relationship with these local security and vulnerability response teams. That relationship is an excellent test of the vendor’s security maturity.
Q. How would the vendor notify our organization about a security vulnerability?
Do you get an action alert Email directly from the vendor or do you read about the vulnerability from the morning news? How your organization gets notified of the vulnerability is critical to the whole process. Some vendors will have their System Engineering (Sales Engineers) walk into the customer and brief them in person. This is an extreme example, but indicative of the level of effort needed by the vendor to communicate during a crisis. Reviewing these processes before a vulnerability happens is strongly advised. Note: Phase 4 of the security conversation will get into deep details of how the vendor interacts and notifies their customer. During the first phase of questions, the objective is to get quick answers.
Conversation 2 – How Does the Vendor Secure Their Organization?
There is no point in trying to build a secure product when the security in the organization has issues. The conversation and dialog can range from how the vendor protects against advanced persistent threats to the future of security with Beyond Corp/Zero Trust models of trust. This would a conversation where you bring in your CIO and Infosec Teams, using your organization’s experience protecting your business as a contrast to the vendor’s practices, policies, and procedures.
Q. Do you have an Information Security Policy?
An Information Security Policy outlines the vendor’s policies with respect to protecting their valuable information. The broad policies and principles set forth here are implemented in the vendor’s Information Security Program. Simple things like “what do you do with NDA agreements and contracts” when they are signed are simple conversations to gauge if the vendor has principles that protect your information.
Q. Do you have an Information Security Program?
The organization’s Information Security Program serves as both the charter for the company’s InfoSec Department and also provides overall information-security policies that apply throughout the organization. At a high level, this security program covers InfoSec generally and the department, roles, and responsibilities relating to security, risk assessment, asset management, information handling, personnel security, and other items that correspond information security framework. These information security frameworks would be part of other security compliance requirements.
Q. How do you protect the software “build” and “compilers?”
What would happen if a determined miscreant got into your organization’s software compilers and put in backdoors or other code into your software? How do you protect against this risk? What protections do you place on your compiler/build infrastructure? How do you audit this infrastructure? Do you hire teams to test your security protections? These are critical questions. We know that determined Advance Persistent Threat (APT) threat actors have targeted the organization’s compilers.
Compiler Violations are not a new threat. Ken Thompson reflected on this during his ACM TuringAward speech. Reflections on Trusting Trust should be required reading for all security professionals. Ken wrote, In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even a hardware microcode. As the level of the program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect.
Q. How do you protect the signing keys for your organization?
All compiled code would be signed with signing keys to validate their integrity. The whole business could be in jeopardy if these keys are violated by an insider threat or Determined Threat Actor. Conversations around how the vendor protects the valuable keys along with documentation that highlights that protection would indicate forethought and introspection.
Conversation 3 – Review the Vendor’s Security Development Lifecycle
The security development lifecycle (SDLs) are all the activities used during product development to minimize security risk. SDLs does not eliminate risk, but they do reduce risk. “Risk reduction” is two-fold. The first is the risk to the shareholders of the vendor. Products that have reduced security risk have less chance of impacting the company’s profitability through preventable security vulnerabilities. The second risk reductions are to the vendor’s customers. The risk is inherited from the vendor into the Operator by a selection of that vendor to be used in their operation. This “inherited risk” gives the vendor’s customers the ability to ask in-depth questions on what they are doing with their SDL to reduce inherited risk.
The following are some excellent questions for an Operator to have with their vendors to gain an understanding of the vendor’s SDL. Note that you do not have to be an expert to ask these questions. Common sense is common for everyone. Ask, listen, and learn. Then ask more questions.
Q. Do you have a Security Development Lifecycle (SDL) as part of your product development?
Demand the Vendor provide their Security Development Lifecycle (SDL) process. This vendor presentation will normally be under a nondisclosure agreement with the details confidential. But, each vendor with an SDL should have the ability to brief their customers, explain how the SDL process works, explain how the SDL process is evolving over time, and explain how the customer can engage with the SDL process. What should an operator look for in a mature SDL? There are books, papers, and guidelines in the industry that cover best practices for SDLs. Here are some of the core items to seek in your vendor’s SDL:
Q. How is the vendor’s SDL integrated into the vendor product development processes?
Security “after the product development” is an afterthought. It never works. Effective SDLs are an integrated part of the normal product development process. Look for the vendor’s SDL presentation to first present their product development process, then how the SDL integrates with that process.
Q. How does the SDL process allow for rapid vulnerability fixes?
“Zero Day” vulnerabilities are often in the “active exploit” mode. That means these reported vulnerabilities need to have the fixes defined, integrated, tested, and deployed in hours – days – not weeks – months.
Q. Does the SDL process use a form of Root Cause Corrective Action (RCCA)?
When a vulnerability is discovered, there are two phases to the “fix.” Phase one is to get the vulnerability fixed to mitigate the potential active exploitation. Phase two uses an RCCA process to find out how the vulnerability got through all the checks. RCCA is essential to find more potential issues. The RCCA is then used to update the tools, processes, and tests to prevent a repeat of the Root Cause. The RCCA goes beyond “regression testing” and looks for the underlying causes, fixing those causes, and making impactful improvements.
Q. Does the vendor’s SDL include regular Static Analysis Testing?
How are the results of the static analysis testing used to improve the resiliency and security of the code? Static analysis testing is one of the many tests to check on the quality of the code. There are open-source static analysis tools, vendor-specific tools, and commercial tools. A commercial example of static analysis is Coverity (www.coverity.com). Many vendors will use static analysis testing, get overwhelmed with the volume of errors, and take no action. Ask the vendors how many bugs were fixed each quarter; as a result, of their static analysis testing. Remember, this sort of testing is about code stability and security. It catches coding mistakes that could lead to outages.
Q. Does the vendor use Test Driven Development (TDD) in their software deployment?
TDD is an approach to coding use in Agile Development. The essence of TDD is to first code a test for the new function before coding the function. Yes, this seems backward – writing the test before the function. But, the decades of TDD application demonstrates that the disciple of first figuring out how to test the function before writing the function forces quality into the code. The result of the industry code is higher quality, fewer bugs, and fewer security issues! Ask the vendor if they are using TDD. Seek to find out if they are using it in all their software This will include microcode for their hardware (ASIC, FPGA, NP, etc.) and elements in network equipment (control plane and management plane elements). TDD now has an industry consensus on the empirical impact on the quality of software development – specifically with the prevention of security vulnerabilities.
Q. Does the vendor use Dynamic Analysis Testing (DAST) in the vendor’s development process?
Dynamic Analysis Testing (DAST) is different from static testing. While static testing pulls in the source code to look for issues, dynamic testing pulls in the compiled software image. Dynamic testing made huge gains over the last decade. The revolution of virtualized environments makes it easier to load a compelled version of the code into a simulator designed to dynamically test the image. Issues found via DAST cover security, stability, robustness, and some performance. DAST will undercover issues that will not be seen with static analysis. There are problems in code that is a result of code interaction. Two modules might interact in a way that causes vulnerability or instability. These interactions are hard to identify with static analysis but are more easily seen after the code is compelled and tested. Vendors who care about the quality of their code should have both static and dynamic analysis testing. There are now open source tools available as well as tools built specifically by the vendor for their unique environments. A good commercial example to explore would be Veracode (www.veracode.com). They provide a baseline example of a dynamic testing tool used extensively in the industry.
Q. Does the vendor use the industry vulnerability classification and enumeration tools?
The industry puts a lot of effort into common tools to normalize the risk of a vulnerability. These tools include the Common Vulnerability Enumeration (CVE), Common Vulnerability Scoring System (CVSS), Common Weakness Enumeration (CWE), and several other tools being developed. The operator should first ask if the vendor knows about these tools. Then ask who they use these tools in their Security Development Lifecycle (SDL) processes. For example, The CVSS system has been used in several vendors to put a metric on the risk a reported vulnerability has to its customers. The CVSS risk value then drives the coding priority in the company. The higher the CVSS risk, the faster the vulnerability gets fixed, tested, and deployed to the customers. CVSS puts an empirical metric to drive the internal processes. This CVSS value is then used to determine how the vulnerability will be disclosed to operators. The higher the number, the more risk to the customer. A CVSS priority will govern the speed of field testing and deployment of the fixed code. Another example is the use of CWE in static and dynamic analysis testing. Many times these tests produce huge reports. Vendors often do not know where to start. In response, the industry has a top CWE list. CWE metrics allows the static and dynamic testing vendors to put a CWE value to each of their findings. The CWE value can then be used to triage the list of issues and prioritize limited coding resources on the most critical CWE items. Overall, vendors who are engaged and can explain how they are using these tools in their SDL processes are must more engage with the security of their products.
Q. Does the vendor have Regression, Stress, Performance, and Compliance System Testing?
Some might ask “isn’t this basic?” Yes, it is basic. But operators would be surprised to find that major vendors do not have a structured system to test. Any operator who is serious about the resiliency and security of their operations should have details of the vendor’s regression, stress, performance, and standards compliance testing. This information would be presented under an NDA. It is a mark of a mature synergy between a vendor and their customer to have conversations around the improvement of the testing environment based on their joint operational experiences. For example, an issue found in the network that was missed in all prior testing would be added to the vendor’s regression and stress test. This will ensure that this issue is not repeated. It is also used for other products the vendor launches. How does this test impact security? Most of these test suites are a systems test. They test the product in a production simulation. This systems-oriented test is the logical follow-on to the static and dynamic testing.
Q. Does the vendor use fuzz testing in their quality processes?
Fuzz testing pressure test an element in unexpected ways. Normally engineers who write a test are ensuring that the element under test is working as expected. Fuzz testing is the opposite. It is designed to send the unexpected. If an element in the protocol is four bytes in a specific location of a packet, then the fuzz test will send 4 bytes offset before – mismatching the location. It is interesting how systems respond when they receive information that is unexpected. Some of the most damaging security vulnerabilities have been found during fuzz testing. Other vulnerabilities have been found in crash core dumps – where the “bug” that causes the crash could if easily be found if a fuzz tester were used on the protocol. The operator should have a conversation with the vendor if fuzz testing is used and how it is used with their product.
Q. Does the vendor have documented risk assessments for all elements, protocols, systems, and solutions?
Part of an SDL is the risk assessment process. Features and functions of the product should include the capability to mitigate risk. How would the operator know if the vendor is performing these risk assessments? Ask for a review of the risk assessment. There are several approaches the vendor can use for these risk assessments. Some vendors like the ITU X.805-risk assessment process attempts to illustrate this in detail. Other vendors use simpler attack trees to explore all the vectors for risk. Some will develop their own methodology that is tuned to their solution’s environment. All of these are A-OK. The core principle the operator should look for is the existence of a risk assessment and how that impact the capabilities of the product.
Q. Does the vendor use specific security testing tools for their Unit Under Test (UUT) testing?
There are many open-source and commercial security testing tools on the market. These solutions range from full organizations to commercial tools. For example, the Open Web Application Security Project (OWASP) is focused on tools practices, and best practices to promote robust web security. These tools are open source and often free to use. Contrast this to the mature market of specific commercial security testing solutions (Spirent aka Mu Dynamics, IXIA, and Codenomicon, Veracode, Qualys, Cigital, Imperva, Fortify, Trustwave, etc.). These companies are worth interacting with to see how they would test the vendor’s product in an operational environment. Operators with the means can consider using these tools in their own testing and evaluation. Others can review the tools and then ask the vendors to use them in their testing.
Conversation 4 – Review the Vendor’s Healthy Interaction & Transparency
Once you understand the vendor’s SDL processes; it is appropriate to ask the tougher questions. These questions will determine vendors who are “OK” with their product security commitment from the vendors who are “all in” in their commitment. It should not matter if you ask these questions to the vendor directly or through your local reseller.
Q. Will the vendor’s Security and Vulnerability Response Team review postmortem on past vulnerabilities with our organization?
Vendors who have a long track record of security will have a history of vulnerabilities. The vendor’s security team should be willing to review with customers past vulnerabilities, how they were found, what was done to fix, the code, what was learned, and how improvements were applied. This conversation is an excellent measure of the vendor’s commitment to security. As mentioned before, in today’s environment it would be impossible to have perfect security. What can be done is continuous improvement to minimize as much of the risk as humanly possible.
Q. Pen Testing. Does the vendor bring in a 3rd party penetration test and security auditors?
All security professionals know that the biggest danger is myopic thinking. Internal security teams will fixate on what is known. The internal security team will often miss out on things that might be obvious to someone else. Some vendors mitigate this narrow thinking by investing in their teams. Sending them to a range of security conferences helps them keep a broader view of the risk. But many times this is not enough. Regular 3rd party penetration tests and security audits are an effective means to review a vendor’s full SDL process. The operator should ask if the vendor pulls in these 3rd party security audits, how they are used to improve the processes, and if they will continue with the practice. It would not be appropriate for the operator to asked details of the 3rd party findings. What is more important is the level of commitment to investing in these 3rd party audits.
The UK’s National Cyber-Security Centre (NCSC) has an excellent guide on Penetration Testing with references – Guidance: Penetration Testing – Advice on how to get the most from penetration testing.
Many vendors will push back because of the cost and consequence (i.e. the pen test or security auditors finds issues). Start with the basics. Do the vendor have a pen test on their own network. Do they bring in security auditors for their own network? Then move to the next questions. Have they done 3rd party pen testing in their lab on their solutions? Have other customers done pen testing on their solutions?
The proof of concept stage is one opportunity where the vendor can bring in a 3d party pen tester in a controlled and operationally safe environment.
Q. Will the vendor be open to a joint tabletop exercise?
Tabletop exercises are tools used in the security incident response community to test the processes and procedures. Operators asking their vendors to be part of a tabletop exercise is critical to the robustness of their operations. For example, one large online website did a robust tabletop exercise with all their vendors where they simulated a combined DDOS attack with the exploitation of a zero-day attack. The exercise tested all the processes and procedures. Lessons were learned. All the processes were updated. Nine months later this large online company’s alarms were triggered a week before a major news event. They knew they were going to get hit with a major DDOS attack. They also saw the scans looking for what could be a zero-day attack. This vendor activated their emergency response procedures, pulled in all the vendors, and prepared for what was going to be a major attack during a major surge of customer traffic. As it turned out, all the preparation mitigated the attack, found a vulnerability, fixed the problem, and maintained services throughout the customer surge. This is one of many examples where an operator preparing with compressive tabletop exercise was able to keep services operating at 100% in the middle of a major attack. It works. Ask the vendors if they have the willingness and capabilities to participate.
Q. Will the vendor be willing to open their code to an operator-driven source code audit?
What? Why would Operators (our customers) want to look at our source code? Why would any vendor allow that to happen? Simple. The cyber-security risk is not just a cyber-criminal risk. It is now a nation-state risk. Vendors and governments are now asking their vendors to open their internal processes to operator driven reviews. These reviews are under very tight NDAs with additional contractual agreements to protect intellectual property and liability risk. These operator driven audits are happening in the industry. The vendor should expect their customer to ask. Operators who are worried about Nation-State cyber-security risk should first do all of the above, then build the capacity for this type of security audit.
Q. Will the vendor publish the “reporter” of a security vulnerability in their security advisory?
A leading indicator of a vendor’s security maturity is its policy on providing credit. Mature vendors with a thoughtful security policy will provide credit to the people who “report” and “find” security vulnerabilities. This “giving credit” policy is good for all parties. It encourages responsible vulnerability reporting and disclosure in the community.
Q. Does the vendor have an active bug bounty program?
Bug bounties are an evolution of a vendor or service provider working with the community to find problems before they become a problem. Bug Bounties are normally run by a company that specializes in managing the bug bounty. This “outsources” some elements of the security management, but assures that someone is looking at who is reporting the vulnerabilities to the vendor. A Keyword search on “Bug Bounty” will highlight companies who either manage bug bounties or sponsor bug bounties.
Q. Will the vendor provide the required “GPL” documentation based on requests for all open source software used in the organization?
While a vendor uses open-source forward in their product, that vendors must comply with the open-source license to use that open source. There are several different open source licenses used in the industry. The range “use it however you choose” to “if you modify it you must submit the changes publicly.” Open source compliance is something that can be used to gauge the level of vendor maturity. If they do not know how to manage the open-source software in their product, then they most likely are not managing their own code in their product. As a minimum, the vendor should provide a list of all the open-source software using throughout their product. Using an open-source package is a good thing. Using an open-source that does not match license and practice creates preventable risk.
Q. How does the vendor manage open source security vulnerabilities?
Open Source is highly encouraged, but open source is also being maintained. Part of that maintenance will fix security vulnerabilities. As the industry has seen, each vendor who uses Open Source needs to have workable plans to fix vulnerabilities with all the open-source used in their products. This question is another data point to help measure the security maturity of the vendor.
A final note to the 3rd Security Conversation
Phase 3 are some of the toughest security questions to ask vendors. It will push most vendors. Many times the vendors will not have answers because they do not have processes, procedures, nor tools to address the questions the Operator is asking. This is hard work, but it will only get done if the Operators who buy and deploy these solutions ask the questions. Otherwise, there will be no “justification” for the vendor to allocate time/resources.
Conversation 5 – Joint Vulnerability Reaction Plan
Vulnerabilities are inevitable. The speed of reaction by the vendor and the operator will determine the window of risk to the business. If an organization wishes to rapidly respond to vulnerabilities, it must proactively invest in a joint reaction plan. This phase of the security conversation is more intensive. It will get into planning that might turn into contract and service level agreement (SLA) conversations. Notice that conversation on the reaction plan is after much longer conversations that inform both parties.
Q. How will the vendor disclose a normal security vulnerability to our organization?
This is a redundant question but worth reviewing again. There will be more insight from each of the vendor conversation and within the own organization as the Operator reviews their own processes.
Q. Does the vendor have a phased disclosure process for notifying customers before the full public disclosure?
Some vendors will have segments of their customers who will be exposed to attack once the vulnerability is publicly disclosed. This window of vulnerability has posed life-impacting risk in the industry. The time it takes for a vulnerable to be “weaponized” by a threat actor is now hours vs weeks. The time it takes to weaponize is determined by the motivation of the threat actor. For example, a cyber-criminal DDOS extortionist who sees a new vulnerability useful to their DDOS extrusion racket will be highly motivated to weaponize and use the vulnerability to increase their cash flow. Knowing this “window of exposure risk,” some vendors will create a phased disclosure process. This phased disclosure will notify some customers before other customers. This process is customized per vendor and the dialog with their customers. The key to the “security conversation” is to have the conversation and understand the vendor’s process (if it exists).
Q. What are the upgrade plans for each vendor element in the organization? Can the upgrades be done with minimal impact to the organization’s business?
A fix for a vulnerability is of no use if it cannot be deployed. This means the vendor and the organization need to have an upgrade path and plan for each of the vendor’s elements in the network. This might be software, hardware, M2M elements, IoT elements, or any other part of the network that requires an “upgrade/patch” to fix a vulnerability. It is advised that the organization exercises the upgrade plans. How long will it take? Will there be impacts on the business? What will it take to “certify” the new upgrade? These are all questions that need to be answered in this conversation.
Q. We buy the vendor’s equipment through a 3rd party/reseller. How will we work with the vendors to review our reaction processes?
Many organizations cannot buy directly from the vendor. They need to go through a local reseller. The reseller should not be a barrier to the ongoing “security conversations.”
For example, if you are a mobile operator, you will have your primary vendor be the “single neck to choke.” That single “turn-key” vendor would be the one responsible for the network. That “primary integrator” will NOT HAVE A MASTER SECURITY PROCESS! If one of the boxes in their master solution has a security vulnerability, they will not be set up to 1) get notified that a security advisory has been released, 2) be able to get the software with the fix (convoluted contracts), and 3) then be able to coordinate a rapid upgrade.
It is strongly advisable for the Operators to explore all the 3rd parties software, management, hardware, Cloud, and other elements in a “turn-key” solution.
Conversation 6 – Cryptography – The Toughest Questions
How a vendor handles cryptography is some of the hardest questions for the vendor and the Operator to address. Honestly, most of the time you will have both sides talking past each other as “crypto novices” try to communicate with each other. Many do not like this truth to be pointed out. But it is needed for people to get into meaningful and productive conversations around the vendor’s use of cryptography. But, these questions need to be addressed by both parties. Not addressing them will expose both organizations to preventable risk.
Conversation Guidelines for Cryptography Conversations
Acknowledge that Cryptography is Critical. Organizations cannot assume that there is no-one sniffing the wire, accessing the files, or using other means to access information. Crypto is a critical part of an organization’s defense It will increase over time as our technology becomes more widespread.
Acknowledge that Crypto Conversations are specialized. People who have not invested in the time and effort to learn will get lost. In fact, “crypto experts” who have not spent recent time in the field will often need crypto-refresh to think within that specialized mind-frame. This should not stop vendor-operator dialogs on cryptography. Everyone needs to be asking these questions.
Constantly Revisit the Crypto Conversations. Breaking crypto is something criminal and state actors actively pursue. In addition crypto professionals proactively try to break systems before the bad actors find ways to do it. Given this level of “active investigation,” Operators should be revisiting their vendor’s “crypto strategy.” These conversations are not one-time meetings.
Cryptography Questions to Prime the “Meaningful Conversation”
Note: Version 1.4 is the beginning of “meaningful cryptographic questions.” More will be added in future versions.
What Cryptography is supported? Start with the basics. Get them to list out what the vendor things are the full list of cryptography usage. This might be a complete list it might be an incomplete list. There are times when parts of the organization do not know what other parts of the organization are doing, even though it is all on the vendor’s product.
What is the list of cryptography certifications that the vendor supports? Where is it supported by their software? This list would be the second question that would help lay the foundation for the vendor’s crypto strategy.
What is the vendor’s “Crypto Strategy?” Will the vendor support backdoors to support various demands for access to cryptographic access? How does the vendor manage the requirements from places like the US, China, UK, and other countries? It is important for the vendor to be transparent. Crypto transparency allows each organization to accurately measure risk.
How does the vendor manage the software signing keys within their organization? All the vendor’s code should be signed. How they handle the certificate within their organization is a good indicator of crypto maturity.
How does the vendor test their cryptographic implementations? As mentioned, there is concerted interest by multiple parties to break crypto implementations. These “investigations” target the cryptographic algorithm, the implementation, the vendor’s specific implementation, the supporting protocols, the configuration, the operation, and the key management. Given this, vendors should have some sort of proactive testing practice where they are one of the “investigators” on their own implementations to find problems before the bad actors (or a University grad student who is working on their thesis).
How does the vendor manage their keys wishing their implementation? Many times the “theory” of good cryptography breaks down with the key management. There are many times where poor key management allows private keys to be pushed out to the public. Asking “how the vendor manages keys” help understand what might be missing with the vendor’s solution. For example, ask what would happen if the Operator needed to push out need keys ASAP because of some type of staff change. How would that work? Can it work?
Conversation 7 – Review All Industry Certifications
Finally, look at the vendor’s Acceptance Test Plans and Solutions Certification Plans. Does the vendor include security as part of their acceptance – certification test plans? Vendor deploying their solutions are driven to get the solution up, running, tested, and in production. Testing for security is normally not part of the acceptance testing that the operator nor vendor requires. Security is an “afterthought.” It might be months later when the security team come into a review and find major configuration issues that have left the solution vulnerable to attack. One of the truest tests of a vendor’s commitment to security is to see its practices in all parts of the company in all phases of the product’s lifecycle. Even with the deployment teams who are working hard to get the system up and running.
Conversation 8 – Deep Supply Chain Security
Software, Hardware, and Solutions are never “uniquely built.” They are a collection of elements, code, APIs, ASICs, FPGAs, and other components that are weaved together into a solution. For decades, people have worried about a “State-Threat-Actor” finding a way to sneak in an element into the ‘supply chain.’ That element then gives them a tool to exploit at a time of their choosing.
First, understand, this supply chain threat vector is real! Second, be mindful that when these issues are found, they are not going to be reported in the paper. Finally, the level of paranoia should not paralyze your security activity. Given all of this, these sorts of questions to the meaningful security conversations would be useful after all the other conversations have progressed. These questions get into deep security processes, quality control, and other factors that the vendors might be doing to protect their supply chain.
Q. How do you protect your software when you ship it to me?
Hashes are common tools that are used to validate the integrity of the software that is delivered. For example, when downloading the software, a hash would be used to validate it. This conversation would be around the thinking put in by the vendor as to the threats and issues with software being shipped and loaded inside your organization.
Q. How do you protect the hardware that is shipped to my site?
Hardware can be tampered with during shipment. While perfect supply chain protection is costly, reasonable protection can be implemented. This conversation would explore the thinking, risk, and considerations the vendor has put into the shipment of hardware.
Q. How do you test 3rd party software that is used in your solution?
It is perfectly OK to use 3rd party open-source modules in solutions. It is strongly encouraged. Blindly using these modules without testing is NOT encouraged. The biggest threat is simple bugs and security vulnerabilities. Threat-Actors of all types explore these open source modules for exploit paths. The more devious threat is where a Threat-Actor intentionally adds code that seems legitimate but is really an exploit waiting to happen.
This security conversation seeks to understand how the vendor addresses these concerns and if they have ever thought of these exploit paths.
Q. How do you validate the hardware that you send to be built? How do you validate your design so that it is exactly what is delivered?
This level of supply chain threat has been in the news in the past. What people don’t realize is that this is normally a Quality Control task. Many vendors don’t consider this a security function. A few vendors will have their internal security teams part of the quality control process to add extra checks to the design and delivery reviews.
This conversation is an interesting conversation, but it would require the vendor to pulling people who never speak to customers. They work hand-in-hand with their manufacturers to ensure that is designed is delivered as specified. Remember, this is a handshake. The manufacturer is also doing their quality control before delivery to the vendor. Then the vendor does a whole series of testing to make sure everything is operating as expected.
These multiple levels of testing are one of the reasons why some question this reality threat vector. Yet, major nation-state security organizations have labs looking for their “competitors” planting violations into the supply chain. This is expensive testing. It would not be happening without some sort of evidence that the threat vector is real. It is OK to ask the question. It might be an interesting dialog with your vendors.
Time for Security Action
There is no security without action. It is like Yoda’s wisdom, “there is no try, you either do or do not.” Security is no different. It only comes from action. Talking is not action. Everyone in the Operator who is part of these vendor security dialogs is going to learn. They are going to think. They are going to reflect. They are going to think of things they can do right now to make their organization a more secure and resilient organization. It is vital for the team who are facilitating these conversations to take action. That is action should be on both sides of the conversion. The vendors need to constantly seek ways to build more resilient products. The Operator needs to find rational solutions that minimize security risk to their organization.
Need Security Advice?
If you find your organization needs help and worry about the FUD from the industry, reach out and ask for help. You can reach me at bgreene@senki.org. Help organizations leverage the talent around them to get started with their security activities. Start with the Operator’s Security Toolkit. It is the no-nonsense security for all Operators. It provides details to help them build more security resilient networks. In the meantime, stay connected to the Senki Community to get updates on new empowerment and security insights. You can sign up to the mailing list for updates here: Stay Connected with Senki’s Updates.