What cyber security should learn from the Boeing affair

The incident

After decades of steadily declining aircraft accidents, the question of how two identical new planes could simply fall out of the sky minutes after takeoff has led to intense scrutiny of the 737 Max’s systems.

Aviation safety record

The FAA points out that aviation safety certification is not a tick-box exercise — every requirement for a piece of safety-critical software must be traceable to the lines of code that implement it and manufacturers must be able to demonstrate that the code actually satisfies the requirement.

Contrast cyber security to aviation

Who tests cyber security?

Is it sufficient? In a word, no. What little testing takes place provides less assurance as a Bernie Madoff saving scheme for children. Here are four reasons why.

1) IT security testing methodology is flawed

Too often, product reviews focus on features, documentation, value for money, performance, support and ease of use. Security isn’t measured at all. When it takes place, it is focused on how well the product counters specific threats, but it does look at the security of the tool itself. Some products may even present a greater security threat than the security risk they are meant to reduce.

And neither to analysts provide much security insight. They collect data about vendors’ products through questionnaires, briefings with vendors and by speaking to customers. They do not roll up their sleeves and test the products themselves. Analysts evaluate vendors — not their products. Being a ‘Leader’ doesn’t mean that the vendor offers the best product.

2) Testing independence is an issue

Standards bodies like MITRE and AMTSO use a more collaborative, conversational product testing approach. Their aim is to improve the quality of security products, however, those with a large contingent of memberships from vendor companies are criticised for being less than transparent and objective in their testing approach.

Testing standards are for example, defined by the member organisations and vendors are notified in advance of their test plans. Some testers use the vendor to drive the tool. This may be efficient since the vendor knows how it works and is highly incentivised to get the best out of the product, but is it realistic to expect a customer to deploy and configure it in the same way? How would the product perform using default configuration settings?

3) Testing only happens if there is a market for the results

Market analysts employ a ‘pay to play’ model and use vendor selection criteria to favour those willing to pay. This favours products from mainstream vendors because analysts can shift more reports by reviewing their products.

Selection criteria such as the number of customers, revenue and feature range, are commonly used to carve out new and niche vendors from the scope of testing. For many buyers that’s OK. They are happy with mainstream products and do not seek to be different from their peers — in fact, they would hate it.

4) Relative test results are of limited use

Tests are often focused on the sensitivity of products to threats but not their accuracy: how many false results are generated for every true result?

In the absence of suitable security standards, testing results are comparative rather than absolute. They focus on differentiating products rather than their ability to address specific threats and as a consequence, they do not support risk-based decision-making, since it is not possible to calculate residual risk.

No objective security testing? Bring on the marketing mayhem

Cooperation between vendors and independent testers is apt to become a legal cat fight if the test results are not what the vendor wishes. What testing does take place, sits behind a paywall. A lack of objective security testing means nobody knows how well the products work.

There is little incentive for vendors to be transparent about the security of their product or to provide evidence of it. Some vendors maintain secrecy in an alleged effort to protect security, which means that inadequate products continue to sell and continue to fail. Buyers are forced to rely on the experience of their peers or worse: the wild claims of vendors (see above — the fruit of 5 minutes’ research). It is not the best product that wins, but the best marketed product.

The case for cyber security regulation

In cyber security, no such certification process exists and when products are tested their performance is less than stellar. A typical security testing report might say, ‘Zero-hour phishing protection ranged from 77.3% to 89.5%.’ Is it really worth the effort to implement a marginal security improvement?

Market-driven testing shuts the door on innovation since it is not possible to compare new and existing products, which is why we are losing the cyber security battle. We are being sold NERF guns disguised as Armalite rifles, to bring to the gunfight.

Only a wild-eyed optimist would claim that we are winning the cyber security battle. Would not a cyber security certification process improve the quality of cyber security products on the market or at least enable buyers to make better informed buying decisions? Very possibly, but how that is achieved is another matter.

Until vendors can evidence the security of their products, buyers will increasingly mistrust them.

Recommendations for vendors

  • Get prospective customers to lobby testers to test your product
  • Invite prospective clients to evaluate the security of your product and improve it
  • Be more honest with your marketing. No product is perfect and wise buyers now it.

Recommendations for buyers

  • Demand more of vendors. Don’t believe celebrity endorsements, people in suits or anything printed
  • When evaluating test results, look out for conflicts of interest. Determine to what extent testing was funded and performed by vendors or organisations supported by them.

Questions buyers should ask IT security product vendors:

  1. Are you willing to share your design documentation?
  2. Are you willing to share with me your internal test results?
  3. Are you willing to show me your source code?
  4. Are you willing to give me access to your test results, tools and scripts?



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store