by Paul Morville, Founder and VP of Products, Confer
Few things are more disappointing or costly than deploying a product that fails to live up to the vendor’s claims or doesn’t meet the team’s expectations. More often than not, there is a very large grey area where it’s difficult to discern what the PowerPoint slides promise versus what the product will actually deliver. A well-structured Proof of Concept (PoC) can be extremely useful in turning this grey area into black and white. But, these PoCs can be complicated and costly to run, sapping security operations center and security analyst resources that are already spread too thin.
For endpoint security, planning and conducting a good POC is even more important than usual because security’s reputation is on the line. While improving endpoint security is essential in today’s environment, endpoint deployments can be risky. They are highly visible across the company and a failed deployment will get the security team into hot water with their end users.
By designing a solid and comprehensive PoC, you can vastly improve your chances of managing the gaggle of vendors vying for your business, make the right decision and ultimately, ensure a smooth rollout and a successful project.
Our Top 10 Do’s and Don’ts:
1: Don’t delegate the scoping and planning process
Senior security team members are typically at maximum capacity, so it’s tempting to delegate the task of planning a PoC to a more junior staff member. Don’t. The PoC is the chance to define what the organization wants from an endpoint security solution in terms of technical, operational and business requirements. In forward-thinking organizations, an experienced CISO is engaged in the upfront planning to ensure the requirements are well-defined.
2: Do ask yourself, “Will it flatten the stack?”
When testing a product, ask yourself whether it will help you flatten the endpoint security stack, thereby reducing management cost and complexity. How many items can you check off on your requirement list? How many endpoint agents can you retire?
The PoC should thoroughly evaluate every function the product claims to offer. For example, if the product blocks attacks – what kind? If the product supports incident response, does it give full visibility into the details and impact on the endpoint?
3: Do adopt the mindset of the adversary
The PoC test should serve as a proxy for the determined adversaries the organization faces. By adopting the mindset of the adversary, the CISO can emulate the types of attacker behaviors they are likely to face.
Skilled attackers can easily penetrate most networks, so the test scenarios should not focus solely on breach prevention. It’s also critical to evaluate the level of damage the attackers can do once they are inside the network, and how readily their behavior can be detected and thwarted.
4: Do form Red and Blue Teams
Conducting a PoC that most accurately reflects a real-world scenario in a specific organization requires selecting members of the security staff to mimic the attackers who are constantly trying to compromise employees’ devices and steal valuable data. These employees become the Red Team. On the flip side, staff members chosen to mimic the defenders, those who work to mitigate all threats facing the organization, become the Blue Team. If everyone knows their roles, the PoC will be as close to reality as possible.
5: Do allow those teams to work together
Often, the Red Team launches an attack and then, a month later, writes a report that says, “We got in, and here are the vulnerabilities we found.” The PoC will be far more useful if one or two key members of the Blue Team are sitting alongside the Red Team and interacting with them. The Blue team can watch how an attack unfolds, analyze how the defenses react, and evaluate what kind of information is generated by the product being tested. In turn, this gives them a better sense of how the product can actually be used, and how it will perform in a real-world environment.
6: Do testing in both the lab and the real world
A typical medium enterprise will have over 5 million executables in their environment and will see upwards of 5,000 new executables enter the environment every day. Every one of these executables has the potential to generate a false positive, but that’s impossible to simulate in a lab. Therefore, a well-designed PoC will strike a balance between bench-testing live malware in a virtual-lab setting, and testing a subset of the real-world production environment under the conditions of an actual attack. An effective PoC should include deployment on at least 20 devices from the general population to provide the real world perspective.
7: Do use a representative set of attacks
Organizations are most likely to be attacked by the same actors who have attacked them in the past, using methods that were previously successful. The goal, therefore, is not to test against the most obscure or exotic malware, but rather to focus on threats the organization has already faced. Maintaining a repository of malware samples from past incidents is a good start. Also, include malwareless attacks – such as document-based or PowerShell scripts. They are common in today’s enterprise and just as damaging as a malware-based attack.
8: Don’t blindly accept tests from your vendors
If a CISO relies on the vendor to provide malware test samples, it will be very important to ask questions about how those samples were derived. Vendors sometimes skew PoC results by repackaging known malware so it evades their competitors’ products, but is detected by their own engine (not a big surprise, since they generated it.) Ask questions and use a mixture of sources.
9: Don’t test malware on a live network
At the risk of stating the obvious, it is never wise to test live malware in a production environment. Inexperienced security personnel have actually done this, triggering a full-scale outbreak in the environment. For live malware testing, the best case is to use a segregated network consisting of virtual machines that are immediately reimaged after infection so as to avoid an actual attack.
10: Don’t test on a suspect endpoint
When conducting a PoC, it can be tempting to “kill two birds with one stone” by including real devices that are suspected of already having been compromised. This approach is not advised because it presents an incomplete picture. If the attacker has already come and gone, you often have very little to go on. Unless you plan to install the product exclusively post-incident, try to simulate the whole attack lifecycle.
Following these 10 best practices will help test how well a product addresses specific endpoint security requirements in the only environment that truly matters – yours.
About the Author
Paul Morville has been working in information security for more than 15 years. Prior to founding Confer, Paul held numerous roles at Arbor Networks, including VP Product Management and VP Corporate Business Development. Paul was an early employee at Arbor and helped take the company from pre-revenue to more than $100M in annual sales, establishing it as the leader in network security DDoS detection and prevention.
While there, Paul developed and launched Arbor’s flagship enterprise network security product line, established partnerships with ISS/IBM, Cisco and Alcatel-Lucent; managed Arbor’s Security Engineering & Response research team; acquired a company; and ultimately managed Arbor’s sale to Danaher Corporation in 2010.
Prior to entering the security industry, Paul worked for several other startups. He holds an MBA with Distinction from Michigan’s Ross School of Business.
About Confer
Confer offers a fundamentally different approach to endpoint security through a Converged Endpoint Security Platform, an adaptive defense that integrates prevention, detection and incident response for endpoints, servers and cloud workloads. The patented technology disrupts most attacks while collecting a rich history of endpoint behavior to support post-incident response and remediation. Confer automates this approach to secure millions of devices, regardless of where they are, allowing security teams to focus on more important activities.
Few things are more disappointing or costly than deploying a product that fails to live up to the vendor’s claims or doesn’t meet the team’s expectations. More often than not, there is a very large grey area where it’s difficult to discern what the PowerPoint slides promise versus what the product will actually deliver. A well-structured Proof of Concept (PoC) can be extremely useful in turning this grey area into black and white. But, these PoCs can be complicated and costly to run, sapping security operations center and security analyst resources that are already spread too thin.
For endpoint security, planning and conducting a good POC is even more important than usual because security’s reputation is on the line. While improving endpoint security is essential in today’s environment, endpoint deployments can be risky. They are highly visible across the company and a failed deployment will get the security team into hot water with their end users.By designing a solid and comprehensive PoC, you can vastly improve your chances of managing the gaggle of vendors vying for your business, make the right decision and ultimately, ensure a smooth rollout and a successful project.
Our Top 10 Do’s and Don’ts:
1: Don’t delegate the scoping and planning process
Senior security team members are typically at maximum capacity, so it’s tempting to delegate the task of planning a PoC to a more junior staff member. Don’t. The PoC is the chance to define what the organization wants from an endpoint security solution in terms of technical, operational and business requirements. In forward-thinking organizations, an experienced CISO is engaged in the upfront planning to ensure the requirements are well-defined.
2: Do ask yourself, “Will it flatten the stack?”
When testing a product, ask yourself whether it will help you flatten the endpoint security stack, thereby reducing management cost and complexity. How many items can you check off on your requirement list? How many endpoint agents can you retire?
The PoC should thoroughly evaluate every function the product claims to offer. For example, if the product blocks attacks – what kind? If the product supports incident response, does it give full visibility into the details and impact on the endpoint?
3: Do adopt the mindset of the adversary
The PoC test should serve as a proxy for the determined adversaries the organization faces. By adopting the mindset of the adversary, the CISO can emulate the types of attacker behaviors they are likely to face.
Skilled attackers can easily penetrate most networks, so the test scenarios should not focus solely on breach prevention. It’s also critical to evaluate the level of damage the attackers can do once they are inside the network, and how readily their behavior can be detected and thwarted.
4: Do form Red and Blue Teams
Conducting a PoC that most accurately reflects a real-world scenario in a specific organization requires selecting members of the security staff to mimic the attackers who are constantly trying to compromise employees’ devices and steal valuable data. These employees become the Red Team. On the flip side, staff members chosen to mimic the defenders, those who work to mitigate all threats facing the organization, become the Blue Team. If everyone knows their roles, the PoC will be as close to reality as possible.
5: Do allow those teams to work together
Often, the Red Team launches an attack and then, a month later, writes a report that says, “We got in, and here are the vulnerabilities we found.” The PoC will be far more useful if one or two key members of the Blue Team are sitting alongside the Red Team and interacting with them. The Blue team can watch how an attack unfolds, analyze how the defenses react, and evaluate what kind of information is generated by the product being tested. In turn, this gives them a better sense of how the product can actually be used, and how it will perform in a real-world environment.
6: Do testing in both the lab and the real world
A typical medium enterprise will have over 5 million executables in their environment and will see upwards of 5,000 new executables enter the environment every day. Every one of these executables has the potential to generate a false positive, but that’s impossible to simulate in a lab. Therefore, a well-designed PoC will strike a balance between bench-testing live malware in a virtual-lab setting, and testing a subset of the real-world production environment under the conditions of an actual attack. An effective PoC should include deployment on at least 20 devices from the general population to provide the real world perspective.
7: Do use a representative set of attacks
Organizations are most likely to be attacked by the same actors who have attacked them in the past, using methods that were previously successful. The goal, therefore, is not to test against the most obscure or exotic malware, but rather to focus on threats the organization has already faced. Maintaining a repository of malware samples from past incidents is a good start. Also, include malwareless attacks – such as document-based or PowerShell scripts. They are common in today’s enterprise and just as damaging as a malware-based attack.
8: Don’t blindly accept tests from your vendors
If a CISO relies on the vendor to provide malware test samples, it will be very important to ask questions about how those samples were derived. Vendors sometimes skew PoC results by repackaging known malware so it evades their competitors’ products, but is detected by their own engine (not a big surprise, since they generated it.) Ask questions and use a mixture of sources.
9: Don’t test malware on a live network
At the risk of stating the obvious, it is never wise to test live malware in a production environment. Inexperienced security personnel have actually done this, triggering a full-scale outbreak in the environment. For live malware testing, the best case is to use a segregated network consisting of virtual machines that are immediately reimaged after infection so as to avoid an actual attack.
10: Don’t test on a suspect endpoint
When conducting a PoC, it can be tempting to “kill two birds with one stone” by including real devices that are suspected of already having been compromised. This approach is not advised because it presents an incomplete picture. If the attacker has already come and gone, you often have very little to go on. Unless you plan to install the product exclusively post-incident, try to simulate the whole attack lifecycle.
Following these 10 best practices will help test how well a product addresses specific endpoint security requirements in the only environment that truly matters – yours.
About the Author
Paul Morville has been working in information security for more than 15 years. Prior to founding Confer, Paul held numerous roles at Arbor Networks, including VP Product Management and VP Corporate Business Development. Paul was an early employee at Arbor and helped take the company from pre-revenue to more than $100M in annual sales, establishing it as the leader in network security DDoS detection and prevention.
While there, Paul developed and launched Arbor’s flagship enterprise network security product line, established partnerships with ISS/IBM, Cisco and Alcatel-Lucent; managed Arbor’s Security Engineering & Response research team; acquired a company; and ultimately managed Arbor’s sale to Danaher Corporation in 2010.
Prior to entering the security industry, Paul worked for several other startups. He holds an MBA with Distinction from Michigan’s Ross School of Business.
About Confer
Confer offers a fundamentally different approach to endpoint security through a Converged Endpoint Security Platform, an adaptive defense that integrates prevention, detection and incident response for endpoints, servers and cloud workloads. The patented technology disrupts most attacks while collecting a rich history of endpoint behavior to support post-incident response and remediation. Confer automates this approach to secure millions of devices, regardless of where they are, allowing security teams to focus on more important activities.
