Inconsistencies in the Common Vulnerability Scoring System (CVSS)

Interesting research:

Shedding Light on CVSS Scoring Inconsistencies: A User-Centric Study on Evaluating Widespread Security Vulnerabilities

Abstract: The Common Vulnerability Scoring System (CVSS) is a popular method for evaluating the severity of vulnerabilities in vulnerability management. In the evaluation process, a numeric score between 0 and 10 is calculated, 10 being the most severe (critical) value. The goal of CVSS is to provide comparable scores across different evaluators. However, previous works indicate that CVSS might not reach this goal: If a vulnerability is evaluated by several analysts, their scores often differ. This raises the following questions: Are CVSS evaluations consistent? Which factors influence CVSS assessments? We systematically investigate these questions in an online survey with 196 CVSS users. We show that specific CVSS metrics are inconsistently evaluated for widespread vulnerability types, including Top 3 vulnerabilities from the ”2022 CWE Top 25 Most Dangerous Software Weaknesses” list. In a follow-up survey with 59 participants, we found that for the same vulnerabilities from the main study, 68% of these users gave different severity ratings. Our study reveals that most evaluators are aware of the problematic aspects of CVSS, but they still see CVSS as a useful tool for vulnerability assessment. Finally, we discuss possible reasons for inconsistent evaluations and provide recommendations on improving the consistency of scoring.

Here’s a summary of the research.

Tags: ,

Posted on September 5, 2023 at 7:03 AM • 17 Comments

Comments

Clive Robinson • September 5, 2023 8:43 AM

@ ALL,

The give away phrase is,

“However, previous works indicate that CVSS might not reach this goal: If a vulnerability is evaluated by several analysts, their scores often differ.”

Thus it is a “subjective system” based on the “Point of View”(PoV) and domain experience of the individual evaluator.

As has been observed in the past when people “underestimate” a threat or equivalent,

“Familiarity breeds contempt.”

So the real question is,

“Can their currently be an evaluation system that is not subjective?”

And I suspect many lean towards “no” as being the answer.

Ted • September 5, 2023 9:35 AM

Two things of note.

1) 30% of the CVSS evaluators from one study sample say they have never read the CVSS documentation.

2) Also, there is discussion about making the online CVSS calculator more nuanced and comprehensive.

If discrepancies in the CVSS scores are important, will the industry move towards a ‘certification’ for CVSS evaluators and more working group research into scoring evals and consistency?

The areas of divergence and ambiguity to CVSS evaluators are insightful.

Clive Robinson • September 5, 2023 10:38 AM

@ Ted, ALL,

Re : vulnarabilities are moving targets.

“If discrepancies in the CVSS scores are important, will the industry move towards a ‘certification’ for CVSS evaluators”

It would be both pointless and probably form a cartel.

Look at the scores as being a bit like “traffic light” systems, not exactly indicative of anything except in a specific point of time.

You can have a near “perfect 10” vulnerability today, but in 60 days it might be entirely harmless.

But it also has the effectct of making a similar vulnerability in the future only a 9.5 or a lot less even though the effects would have been the same if their discovery was the other way around.

The thing that everyone should remember is that a vulnerability is a “stepping stone”. Whilst it might be minor in of it’s self, it might put a previously unreachable steping stone within range, thus make a bridge as a part of a much more serious issue.

Now how do you rate it? The minor it actually is, or the major it alows in combination to happen?

BW • September 5, 2023 10:46 AM

I rarely pay attention to score. As my concerns are primarily related to software supply chain, any security issue for any component we use ends up in my lap.

I read the report, and then evaluate impact/risk based on our specific situation.

I think objective scoring would require some common context, which doesn’t make much sense really.

lurker • September 5, 2023 1:04 PM

@Clive Robinson
re, present CVSS are subjective

So, are people expecting them to be objective? The research hints that maybe an objective analysis would produce more consistent results. Where would we get that from, a computer following rules? Given the track record of current AI, I’m not holding my breath.

K.S. • September 5, 2023 2:21 PM

Personally, I find CVSS scores not that informative. In researching CVEs, many times I have seen CLI exploits classified as local, even when the product clearly supports remote CLI out of the box. I also have seen inflated severity given to probabilistic/theoretical attacks, such as ability to compromise 1 bit (and only that) of a private key made into a big(er) deal.

Ted • September 5, 2023 5:23 PM

@Clive, All

The thing that everyone should remember is that a vulnerability is a “stepping stone”

Great point. I’m not sure if I saw mention of daisy chaining vulns, but I did see the researchers discuss the handling of ‘security deficiencies.’

”These can facilitate an attack in combination with a vulnerability, but do not lead to a successful attack per se”

Security deficiencies are frequently assigned CVSS scores over a broad range, including None. This led the researchers to include this question:

”RQ2: Are security deficiencies considered suitable for CVSSv3.1 assessment by CVSS users?”

One example they gave are Banner Disclosures, where the particular web server and version info are disclosed in HTTP responses.

Clive Robinson • September 5, 2023 6:03 PM

@ lurker,

“So, are people expecting them to be objective? The research hints that maybe an objective analysis would produce more consistent results.”

Some, yes…

Because you get the claims “it’s algorithmic” which in this case indicates the principle of GIGO is still in play.

You will see @Lex above brought up CVE-2020-19909.

Now you would think it’s two or three years old yet it was posted anonymously less than a month back (Published: 2023-08-22 according to various sources).

And it’s caused quite some twitching.

Some are claiming it’s “malicious” others that the person is not exactly top draw.

Miter have gone into “stupid mode” rather than clean it up they’ve made it disputed with a realy realy weak some claim unsuportable argument.

Have a look for yourself as to what others are saying,

https://www.tenable.com/cve/CVE-2020-19909

As for cURL teams responce, well you will find this quote up on the web,

“filed and made public by an anonymous person due to incompetence or malice. We cannot say which and the distinction does not matter to us.”

https://cybersixgill.com/news/articles/cve-2020-19909-cybersixgill-analysis

The question is where do you go from here?

Imagine you are accused of a crime by a corporation because they can not do their job.

The court paperwork geys put into the public domain, because that’s what they do.

The case has to go up before the judge, and they decide if it should be dropped or not.

The problem is your reputation for no reason other than gross incompetence by a corporation has been trashed, and your private details will end up in all sorts of dubious databases,

http://telecom.csail.mit.edu/judy-sammel.html

At least with the court paperwork you can trace it back to a named individual.

This is not the first time Miter have been called into question over the way their system works.

It’s beyond time that it got cleaned up and made fit for the 21st century.

Clive Robinson • September 5, 2023 6:31 PM

@ ALL,

The actual question we should consider is,

“Do people care enough to act?”

The answer would appear to be in way to many cases “NO”.

https://www.theregister.com/2023/09/05/qualys_top_20_vulnerabilities/

The reasons are many[1] but you have three places you can be,

1, Vulnerable to all attacks.
2, Patched to prevent some attacks.
3, Take sensible mitigation.

Odd though it might appear, if you do it right the third place is where you actually want to be.

Why? because patches fix an instance of an attack and can open other vulnerabilities.

Mitigation on the other hand can block whole clases of attack many of which are not yet known. And importantly does not rely on the people who got it wrong in the first place to maybe get it right in some future time without messing other things up…

[1] You need to remember that corporate software developers don’t want to “fix old problems” because there is no profit in it compared to forcing you into an expensive upgrade, or worse these days an “online subscription service” where all your work becomes not just hostage but worse “third party business records” easily accessable to many without any kind of judicial oversight etc. I use some very old Microsoft software that is full of maggot holes security wise, some of which I’ve found myself. As Microsoft are never going to fix them, there is actually incentive not to report them… So my solution is to mitigate by strict segregation, and one or two other tricks I’ve mentioned before.

PaulBart • September 6, 2023 7:24 AM

This is a golden opportunity to use AI to achieve predictable scoring, like FICO.
PCGS should start using AI as well. Is it MS-65 or not?

iAPX • September 6, 2023 9:22 AM

I evaluated our bug bounty program’s entries.

This is a very difficult task, where you have to fight against your bias, I had the chance to be experienced in finding security breaches, and also had access to our systems and our code.
Thus the abilities, experience and information needed.

In fact I was the first one to entry a security breach into our (at that time) new bug bounty program, and asked for the money to be sent to our company’s well-being program. Was done by a company employee on company time.

And at some point I had to reevaluate the risks associated with every entry, assess the company’s risk behind the technical risk, still produce a CVSS, but also it happened that I had to consider multiple risks that the hacker didn’t consider as a whole, having multiple low-to-mid score CVSS that together represented a higher risk, and we distributed bonuses in this case.

I think it’s very hard to do a CVSS scoring due to our human bias, peer-pressure or company pressure, and it’s worse if you are not experienced in finding AND exploiting security breaches.

Clive Robinson • September 6, 2023 10:07 AM

@ iAPX, ALL,

“it’s worse if you are not experienced in finding AND exploiting security breaches.”

Being experienced is as our host @Bruce noted some years ago insufficient.

Look at it this way, anyone who can do basic maths can do basic accountancy, but it takes a certain mentality to make accountancy rule bendingly profitable.

It’s the ability to look at a system and spot not just the defects at a base level, but the unobvious mistakes in protocols and standards, especially when two or more standads have to dove-tail.

@Bruce called it “thinking hinky”… having suffered from not being able to look at a system without finding defects from being a pre-teen onwards I call it a curse…

People actually say only half jokingly “you’ve got witchcraft skills” or “crystal balls” etc, and whilst I don’t think I’m going to get burnt at the stake today, the way the world is heading tommorow I’m not so sure about…

Ismar • September 6, 2023 5:27 PM

There are a number of software vendors now who have developed quite sophisticated products which deal with software supply chain vulnerability scanning. They all rely on the vulnerability scores like those published by likes of cve/nist but also provide their version of vulnerability scores which may or may not fall into same rating categories. It is then up to security professionals to determine if a particular vulnerability would apply in their application context as by their nature these vulnerability ratings cannot anticipate all possible real world scenarios of their applicability.
As such, the vulnerability scores are but a starting point for evaluating their impact to a particular usage scenario (application) and should be seen as such and not as some kind of black and white measurement of how vulnerable your application is (people often search for systems which would give them black or white answers without realising such systems are by their very nature inaccurate in most situations)

Christopher Drake • October 15, 2023 10:27 PM

Someone should study the submission system – none of the severe vulnerabilities I’ve tried to submit have ever seen the light of day (the vuln’s all still exist now).

There’s some kind of cozy club of authorized gatekeepers making sure that stuff they don’t want to see never gets in at all…

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.