Vendor Cyberscore Justification and Methodology

Introduction

The NCT cyberscore website seeks to design, develop, and release a novel scoring methodology to rank software/hardware vendors based upon the quality of their cyber-hygiene practices. By publishing vendor cyberscores to the public, we aim to incentivize software/hardware vendors to be more proactive in adopting better cybersecurity practices. We wish to foster a more transparent relationship between vendors and customers, one where both parties engage with the cybersecurity risks associated with the products they make and/or use.

We seek to publish vendor scores for as many vendors as is feasible. We seek to explain to vendors how their scores can be improved and inform customers such that they can make educated purchasing decisions. At procurement, a buyer could use this website to determine what vendor is best to buy from (at least, in terms of cyber hygiene and product security).

In the interest of transparency, all cyberscores published to the NCT cyberscore website rely solely on publically available data. At the time of writing, this data is primarily pulled from the National Vulnerability Database and the Known Exploited Vulnerabilities Catalog.

The version of this website you are currently visiting is still a prototype: the cyberscores present here are meant to act as a proof-of-concept, as opposed to an objective, final measure. The metrics, measures, and calculations used to generate vendor cyberscores are described below.

Overall Vendor Cyberscore

Vendor's are assigned an overall cyberscore between 0-100, with 0 indicating insecurity and 100 indicating excellent security. This score is visualized through a color gradient scheme:

• 100 is green.
• 50 is yellow.
• 0 is red.

Overall scores that fall somewhere between these three values take on a combination of colors. For example, a vendor assigned a cyberscore of 75 is assigned a yellowish-green color.

A vendor's overall cyberscore is calculated as the weighted average of a set of subscores. Each constituent subscore also falls between 0-100 and follows the same coloring scheme. Each subscore is assigned a weight. The higher the weight, the larger the impact a subscore has on the overall cyberscore. The formula for this calculation is as follows (where x's are subscores, and w's are weights):

As an example, say we have 3 subscores: 80, 20, and 10. Let’s say the weights for these scores were 3, 1, and 1, respectively:

So, the final vendor cyber-hygiene score in this example would be 54.

Subscores

At the time of writing, the NCT cyberscore website utilizes 4 subscores in calculating the final overall cyberscore for any given vendor. In the future, these subscores are subject to change, as more accurate, meaningful, and reliable scoring methods are developed.

Subscores are designed to be as objective as possible. Vendors are not measured against one another, but instead to a ubiquitous standard. This means that it is possible for all vendors to receive perfect scores of 100, or all vendors to receive terrible scores of 0. Vendors are not "graded on a curve".

The remainder of this section goes over each subscore in detail.

Exploitability

The Exploitability subscore attempts to answer the following question:

How easy is it to exploit the vendor's products?

We calculate this subscore by determining the percentage of a vendor's CVEs that are found in the Known Exploited Vulnerabilities (KEV) Catalog. Vendors with many KEV catalog entries receive low Exploitability subscores. Vendors with few KEV catalogue entries receive high exploitability subscores (higher scores are better/less exploitable).

To improve their Exploitability score, a vendor must release less exploitable software. If a vendor adopts better product security practices (e.g., QA testing, use of SAST tools, pen-testing, etc.), they are less likely to end up in the KEV catalog.

Potential pitfalls with this score include:

• Having many KEVS may not indicate a vendor's software is more exploitable, just that their software is targeted by malicious actors more often.

The Exploitability subscore has been assigned a weight of 3, meaning it accounts for 33.3% of the overall cyberscore.

Remediability

The Remediability subscore attempts to answer the following question:

Can the vulnerabilities affecting the vendor's products be easily remediated?

We calculate this subscore by determining the percentage of a vendor's CVEs that have references tagged as 'Patch', 'Mitigation', and/or 'Vendor Advisory'. In theory, CVEs with these types of references are easier to remediate.

When viewing a CVE on the NVD's website, you can scroll to the References to Advisories, Solutions, and Tools section to view a table of references. For example, CVE-2024-20421 contains a Vendor Advisory, which documents methods for patching the vulnerability.

To improve their Remediability score, a vendor should publish information on how to mitigate vulnerabilities. Additionally, vendors should release and publicize patches that fix vulnerabilities.

Potential pitfalls with this score include:

• In some cases, these tags may not always be an indicator of easy remediation. For example, a Vendor Advisory occasionally does not provide any remediation advice.

The Remediability subscore has been assigned a weight of 3, meaning it accounts for 33.3% of the overall cyberscore.

Severity

The Severity subscore attempts to answer the following question:

How severe are the vulnerabilities affecting the vendor's products?

We calculate the average CVSS score of the CVEs affecting the vendor. This score is normalized to be between 0-100 and inverted: an average CVSS score of 3.4 will result in a Severity subscore of 66.

The Common Vulnerability Scoring System (CVSS) is an open industry standard for measuring the severity of computer system security vulnerabilities. Score range from 0-10, with 10 being the most severe. To improve their Severity score, a vendor should read up on CVSS and implement better security practices to reduce the chance of releasing vulnerabilities with high CVSS scores.

The Severity subscore has been assigned a weight of 2, meaning it accounts for 22.2% of the overall cyberscore.

Explainability

The Explainability subscore attempts to answer the following question:

Are vulnerabilities affecting the vendor's products explained in a clear and concise manner?

We calculate this subscore by determining how clear the descriptions of the CVEs related to a vendor are. Clarity of description is calculated using Flesch–Kincaid readability scoring.

The Flesch–Kincaid readability tests are designed to determine how difficult a passage is to understand. It takes into account the ratio of words to sentences aswell as the ratio of syllables to words, then assigns a score between 0-100. A score of 100 is very easy to understand, a score of 0 is very difficult to understand.

To improve their Explainability score, a vendor should better explain vulnerabilities, how they work, and what systems/environments they can affect. Vendors should use clearer, more concise language when describing these vulnerabilities.

Potential pitfalls with this score include:

• Clarity is often a subjective concept: a description may be clear to one reader, but not to another.
• Some vulnerabilities are more complex than others, so their descriptions would naturally be more complicated and less clear.

The Severity subscore has been assigned a weight of 1, meaning it accounts for 11.1% of the overall cyberscore.