Vendor Cyberscore Justification and Methodology

Introduction

The NCT cyberscore website seeks to design, develop, and release a novel scoring methodology to rank software/hardware vendors based upon the quality of their cyber-hygiene practices. By publishing vendor cyberscores to the public, we aim to incentivize software/hardware vendors to be more proactive in adopting better cybersecurity practices. We wish to foster a more transparent relationship between vendors and customers, one where both parties engage with the cybersecurity risks associated with the products they make and/or use.

We seek to publish vendor scores for as many vendors as is feasible. We seek to explain to vendors how their scores can be improved and inform customers such that they can make educated purchasing decisions. At procurement, a buyer could use this website to determine what vendor is best to buy from (at least, in terms of cyber hygiene and product security).

In the interest of transparency, all cyberscores published to the NCT cyberscore website rely solely on publically available data, including the National Vulnerability Database and the Known Exploited Vulnerabilities Catalog.

Primary Vendor Cyberscores

Vendor's are assigned an primary cyberscores between 0-100, with 0 indicating insecurity and 100 indicating excellent security. This score is visualized through a color gradient scheme:

• 100 is green.
• 50 is yellow.
• 0 is red.

Primary scores that fall somewhere between these three values take on a combination of colors. For example, a vendor assigned a cyberscore of 75 is assigned a yellowish-green color.

A primary cyberscore is calculated as the weighted average of a set of subscores. Each constituent subscore also falls between 0-100 and follows the same coloring scheme. Each subscore is assigned a weight. The higher the weight, the larger the impact a subscore has on the overall cyberscore. The formula for this calculation is as follows (where x's are subscores, and w's are weights):

As an example, say we have 3 subscores: 80, 20, and 10. Let's say the weights for these scores were 3, 1, and 1, respectively:

So, the final primary score in this example would be 54.

Revisions

Each subscore is recalculated periodically, in order to pull new data and allow vendors to correct issues and improve their subscores. Each of these recalculations is referred to as a revision. Past revisions can be searched to see how a vendor's subscores have changed over time. Revisions are independent and do not affect scoring of future revisions.

Subscores

Subscores are designed to be as objective as possible. Vendors are not measured against one another, but instead to a ubiquitous standard. This means that it is possible for all vendors to receive perfect scores of 100, or all vendors to receive terrible scores of 0. Vendors are not "graded on a curve".

The remainder of this section goes over each subscore in detail.

Exploitability

The Exploitability subscore attempts to answer the following question:

How easy is it to exploit the vendor's products?

We calculate this subscore by determining the percentage of a vendor's CVEs that are found in the Known Exploited Vulnerabilities (KEV) Catalog. Vendors with many KEV catalog entries receive low Exploitability subscores. Vendors with few KEV catalogue entries receive high exploitability subscores (higher scores are better/less exploitable).

To improve their Exploitability score, a vendor must release less exploitable software. If a vendor adopts better product security practices (e.g., QA testing, use of SAST tools, pen-testing, etc.), they are less likely to end up in the KEV catalog.

Potential pitfalls with this score include:

• Having many KEVS may not indicate a vendor's software is more exploitable, just that their software is targeted by malicious actors more often.

Remediability

The Remediability subscore attempts to answer the following question:

Can the vulnerabilities affecting the vendor's products be easily remediated?

We calculate this subscore by determining the percentage of a vendor's CVEs that have references tagged as 'Patch', 'Mitigation', and/or 'Vendor Advisory'. In theory, CVEs with these types of references are easier to remediate.

When viewing a CVE on the NVD's website, you can scroll to the References to Advisories, Solutions, and Tools section to view a table of references. For example, CVE-2024-20421 contains a Vendor Advisory, which documents methods for patching the vulnerability.

To improve their Remediability score, a vendor should publish information on how to mitigate vulnerabilities. Additionally, vendors should release and publicize patches that fix vulnerabilities.

Potential pitfalls with this score include:

• In some cases, these tags may not always be an indicator of easy remediation. For example, a Vendor Advisory occasionally does not provide any remediation advice.

Severity

The Severity subscore attempts to answer the following question:

How severe are the vulnerabilities affecting the vendor's products?

We calculate the average CVSS score of the CVEs affecting the vendor. This score is normalized to be between 0-100 and inverted: an average CVSS score of 3.4 will result in a Severity subscore of 66.

The Common Vulnerability Scoring System (CVSS) is an open industry standard for measuring the severity of computer system security vulnerabilities. Score range from 0-10, with 10 being the most severe. To improve their Severity score, a vendor should read up on CVSS and implement better security practices to reduce the chance of releasing vulnerabilities with high CVSS scores.

Explainability

The Explainability subscore attempts to answer the following question:

Are vulnerabilities affecting the vendor's products explained in a clear and concise manner?

We calculate this subscore by determining how clear the descriptions of the CVEs related to a vendor are. Clarity of description is calculated using Flesch–Kincaid readability scoring.

The Flesch–Kincaid readability tests are designed to determine how difficult a passage is to understand. It takes into account the ratio of words to sentences aswell as the ratio of syllables to words, then assigns a score between 0-100. A score of 100 is very easy to understand, a score of 0 is very difficult to understand.

To improve their Explainability score, a vendor should better explain vulnerabilities, how they work, and what systems/environments they can affect. Vendors should use clearer, more concise language when describing these vulnerabilities.

Potential pitfalls with this score include:

• Clarity is often a subjective concept: a description may be clear to one reader, but not to another.
• Some vulnerabilities are more complex than others, so their descriptions would naturally be more complicated and less clear.

MFA

The Multi-Factor Authentication (MFA) subscore attempts to answer the following question:

Does the vendor utilize multi-factor authentication within their products?

We calculate this subscore by searching through vendor documentation and the Cloud Security Alliance (CSA) Consensus Assessments Initiative Questionnaire (CAIQ) to find evidence of multi-factor authentication. To find a vendor's documentation, we first identify the vendor's website, using the domain written in the vendor's name, the websites referenced in the vendor's NVD entries, or a verified guess based on the vendor's name.

To improve their MFA score, a vendor should use multi-factor authentication in all of their products to protect both administrators and users from phishing and other attacks.

Potential pitfalls with this score include:

• The scraper may report false positives or low confidence matches.
• If the vendor's website cannot be identified, the vendor receives the baseline score because no evidence could be reviewed.

Default Credentials

The Default Credentials subscore attempts to answer the following question:

Do the vendor's products contain default credentials that users are expected to change and are changing these credentials enforced?

We calculate this subscore by looking for evidence that a vendor's product contains default credentials and using a risk score based on where the evidence was acquired from. We use the Default Credentials CheatSheet to check if a vendor's product's default credentials are known.

To improve their default credentials score, a vendor must avoid using default credentials in their products, or at the very least, enforce that the user change the default credentials to prevent unwanted access to their product.

Potential pitfalls with this score include:

• Signing the CISA Secure By Design Pledge does not necessarily mean the vendor is following the pledge.
• The cheat sheet may not be properly updated and may not include every default credential.

SBOM

The Software Bill of Materials (SBOM) subscore attempts to answer the following question:

Does the vendor have any vulnerable packages or dependencies in their products?

We calculate this subscore by analyzing a vendor's GitHub repositories with Syft, Grype and OSV-Scanner to generate a SBOM and search for vulnerabilities with packages in the SBOM. We only analyze repositories that belong to the vendor: repositories are found through the links in the vendor's NVD entries, and only those owned by a GitHub account that matches the vendor are used (up to the five most frequently referenced). For each vulnerability, we take CVSS, EPSS and OSV Severity to determine the danger of the vulnerability and score based on the amount of vulnerabilities and the average severity of the vulnerabilities. Higher scores are given to vendors with product SBOMs that contain a low amount of vulnerabilities and a low average danger for those vulnerabilities.

To improve their SBOM score, a vendor should ensure that all dependencies are updated to their latest versions and unnecessary dependencies are removed. Additionally, if a dependency is vital to a project but contains unpatched vulnerabilities, vendors should try to find solutions that use different dependencies.

Potential pitfalls with this score include:

• This score only applies to vendors that have open-source products hosted on GitHub, which accounts for only a portion of all vendors.
• Repositories are found through NVD references, so a vendor's open-source projects that are never referenced there are not analyzed.
• In some cases, an alternative dependency does not exist and vendors may get bad scores for packages they are required to use. This penalizes vendors for problems that they do not have control over.
• Vendors may use vulnerable packages in contexts that avoid the vulnerability all together, as this score only checks for presence.

Compliance

The Compliance subscore attempts to answer the following question:

Does the vendor publicly show that it follows recognized secure software development guidance?

We calculate this subscore by searching for vendor documents that show alignment with two frameworks: the NIST Secure Software Development Framework (SSDF) and Federal Trade Commission (FTC) guidance. Each framework accounts for half of the score (50 points each). The documents found are analyzed to determine whether they provide evidence of following the framework. A score is only given when enough relevant documents were found and successfully read; otherwise the vendor receives no Compliance score for that revision.

To improve their Compliance score, a vendor should publish clear, public documentation of how their development practices meet the SSDF and the relevant FTC guidance.

Potential pitfalls with this score include:

• Publishing documentation does not necessarily mean the vendor follows the practices described.
• Evidence the scraper cannot find or read, such as documents that are not public, is not counted.
• Vendors for which too little evidence is found receive no score rather than a low score.

Secure Boot

The Secure Boot subscore attempts to answer the following question:

Do the vendor's products support secure boot?

We calculate this subscore, as applicable, by looking for evidence that shows whether a vendor has products that support secure boot or not. This is done by searching the internet for evidence and scraping high-confidence websites for evidence that gets classified by a classifier to determine whether the evidence shows the product supports, does not support or is unrelated to secure boot.

To improve their Secure Boot score, a vendor should support secure boot within their products and ensure that vendor documentation and other public-facing vendor websites make this clear. Additionally, vendors should strive to publicly include detail about which products support secure boot and other important details such as whether the product can prove it is in secure boot mode or information on secure boot status is shown within the product.

Potential pitfalls with this score include:

• The scraper can miss important evidence that indicates secure boot status, if the classifier is not confident or if the searcher does not identify the website while searching.
• False positives can incorrectly indicate the wrong status, especially if the website is outdated or the website has incorrect data.

Physical Tamper

The Physical Tamper Evidence subscore attempts to answer the following question:

Are the vendor's products resistant to physical tampering?

We calculate this subscore from the vulnerabilities (CVEs) published for the vendor's products. A vulnerability is counted as physical if its CVSS v3.x attack vector is Physical, or if it is tagged with a weakness related to physical access, such as improper physical access control, improperly protected debug and test interfaces, or physical side channels. The score is based on what fraction of the vendor's vulnerabilities are physical: the larger the fraction, the lower the score. The calculation is smoothed so that a single physical vulnerability does not swing the score of a vendor with only a few vulnerabilities, and a vendor needs a minimum number of published vulnerabilities (10) to receive a score.

To improve their Physical Tamper Evidence score, a vendor should reduce weaknesses that can be exploited through physical access, for example by protecting debug and test interfaces, controlling physical access to sensitive components and defending against physical side-channel attacks.

Potential pitfalls with this score include:

• The score only reflects vulnerabilities that have been reported. A vendor with no reported physical vulnerabilities is not necessarily resistant to physical tampering.
• Vendors with fewer than 10 published vulnerabilities do not receive a score.
• Results depend on how vulnerabilities are classified (CVSS attack vector and weakness tags), which is not always applied consistently.

Memory Safety

The Memory Safety subscore attempts to answer the following question:

Does the vendor use memory-safe languages and memory-safe practices within their code?

We calculate this subscore by looking at various GitHub repositories associated with the vendor and calculating the composition of languages with gcloc. Only repositories that belong to the vendor are used, found through the links in the vendor's NVD entries (up to the five most frequently referenced). We also check for various memory-safety issues such as using memory-unsafe C functions or Rust unsafe blocks within the repository. We check issues for seven languages: C++, C, Rust, Python, JavaScript, Go and Java. We calculate a language subscore for each language based on the amount of issues found and an overall subscore that shows the ratio of memory-safe to memory-unsafe languages. We calculate the final score by taking a weighted average of each subscore, with each weight reflecting how much of the language exists across the vendor's repositories.

To improve their Memory Safety score, a vendor should switch to memory-safe alternatives of memory-unsafe languages. Additionally, a vendor should rewrite code that contains memory-safety issues identified by the scraper.

Potential pitfalls with this score include:

• Memory-unsafe languages are not inherently more dangerous than memory-safe languages, if used correctly. Often vendors need to write code in memory-unsafe languages and cannot switch to memory-safe alternatives.
• This score only applies to vendors that have open-source products hosted on GitHub, which accounts for only a portion of all vendors. Open-source projects that are never referenced in the vendor's NVD entries are not analyzed.
• The checks for specific memory-safety issues only read files in the top-level folder of each repository, while the language composition covers the entire repository.
• Code that contains memory-safety issues can be code that does not execute or code that avoids the issue in a special way. Some code issues are also often required by the context and cannot be safely avoided. For instance, Rust unsafe code may be needed to perform certain operations that cannot be performed in any alternative way.

Linter

The Linter subscore attempts to answer the following question:

Does the vendor follow coding standards and best practices in their open-source code?

We calculate this subscore by running linters on vendor GitHub projects and counting the total number of linter issues discovered. Only repositories that belong to the vendor are used, found through the links in the vendor's NVD entries (up to the five most frequently referenced). More linter issues equate to a lower score. We ran the following linters on the following languages: cpplint on C and C++, ruff on Python, quick-lint-js on JavaScript, golangci-lint on Golang and checkstyle on Java. We ignored languages not present in this list.

To improve their Linter score, a vendor should fix all issues reported in their score's evidence. They should also follow coding standards and utilize linters in their workflow.

Potential pitfalls with this score include:

• This score only applies to vendors that have open-source products hosted on GitHub, which accounts for only a portion of all vendors. Open-source projects that are never referenced in the vendor's NVD entries are not analyzed.
• There are many languages that are not checked by this score that can have coding issues.
• Coding standards vary and different linters may observe different practices. A linter could easily flag an issue that may not be an issue according to the standards a vendor uses.