E-Mailing Private HTTPS Keys

I don’t know what to make of this story:

The email was sent on Tuesday by the CEO of Trustico, a UK-based reseller of TLS certificates issued by the browser-trusted certificate authorities Comodo and, until recently, Symantec. It was sent to Jeremy Rowley, an executive vice president at DigiCert, a certificate authority that acquired Symantec’s certificate issuance business after Symantec was caught flouting binding industry rules, prompting Google to distrust Symantec certificates in its Chrome browser. In communications earlier this month, Trustico notified DigiCert that 50,000 Symantec-issued certificates Trustico had resold should be mass revoked because of security concerns.

When Rowley asked for proof the certificates were compromised, the Trustico CEO emailed the private keys of 23,000 certificates, according to an account posted to a Mozilla security policy forum. The report produced a collective gasp among many security practitioners who said it demonstrated a shockingly cavalier treatment of the digital certificates that form one of the most basic foundations of website security.

Generally speaking, private keys for TLS certificates should never be archived by resellers, and, even in the rare cases where such storage is permissible, they should be tightly safeguarded. A CEO being able to attach the keys for 23,000 certificates to an email raises troubling concerns that those types of best practices weren’t followed.

I am croggled by the multiple layers of insecurity here.

BoingBoing post.

Tags: , , , , ,

Posted on March 13, 2018 at 6:31 AM • 43 Comments

Comments

Today • March 13, 2018 6:45 AM

I too am croggled by the hugger-mugger hornswoggling of these hoodwinking bamboozlers.

Brian • March 13, 2018 7:02 AM

Well, displaying the private key does demonstrate the key has been compromised. It’s not like it got any more compromised by being emailed.

Robert • March 13, 2018 7:23 AM

“Trustico allows customers to generate a Certificate Signing Request and Private Key during the ordering process,” the statement read. “These Private Keys are stored in cold storage, for the purpose of revocation.”

W…T…F…

The CA creating the key … ill advised, but probably not usually a serious problem.

But, that excuse … ALL you need to revoke a certificate is the serial number and the CA key. You NEVER need the certificate private key. THEY SHOULD KNOW THIS!

Me myself • March 13, 2018 7:35 AM

Seems like the people whose job was to make a security “product” don’t know the first thing about security after “don’t use your wife’s name as your password”.

I agree with Brian. The keys were already compromised; emailing them just made that obvious. But the CEO should know that admitting that his company had engaged in so poor practices would bring a blow to the company’s credibility.

I guess I’ll be adding Trustico to my ever-growing list of companies to be wary of.

me • March 13, 2018 7:52 AM

Commercial suicide in 3, 2, 1…

a: “we have to switch to comodo”
b: “but symantec certs expire the next year”
a: “no problem, look at this: i’ll say that they are compromised so they will revoke all them and we will issue new comodo certs!”
b: “i’m not sure is a good idea…”
a: “you think too much”
a: send 23k private keys
digicert notify customers
a: “wtf is digicert doing?”
people: “wtf trustico is doing?”
people start to search a better alternative
some people still trust them, but can’t buy new cert because the site is being ddosed by all the people trying to get a new cert

EPIC FAIL face palm

Jim • March 13, 2018 8:12 AM

I’m with Brian. The [unnamed] CEO of Trustico was demonstrating that the keys were compromised and worthless precisely by that fact that he “could” email them. It’s like proving you have someone’s SS or CC number by publishing it publicly. Rowley should have been more discrete about seeking proof and more amenable to revoking the certificates in the first place. Now DigiCert has egg on their face.

Clive Robinson • March 13, 2018 8:19 AM

@ Robert,

ALL you need to revoke a certificate is the serial number and the _CA_ key. You NEVER need the certificate private key. THEY SHOULD KNOW THIS!

Trustico were an agent of Symantic apparently required to act in the stead of a customer. Symantic would also only act on such a thing as revocation if the private key was handed over as proof…

Thus it is Symantic’s business model people should be looking at first…

Oh and also the fact that Google amongst others were going to take Symantec signed certs off of their accepted lists, so the certificatrs would have become usless fairly quickly any way…

Rick Moen • March 13, 2018 8:24 AM

@Brian, I believe you have, somehow, sailed past the point of actual interest: As @Robert suggests, the question you should have asked is, why was Trustico in possession of certificate private keys in the first place?

Nothing about being a cert reseller, nothing, nothing at all about the legitimate functions of such a business, necessitates the reseller ever, ever, ever having them. Those are not their keys; those are the customers’ private keys — keys that by acquiring and storing and doing heavens-knows-what with, Trustico immediately rendered not-private by functional definition. Ergo, the compromise did NOT occur when Trustico mailed those 23,000 private keys and could be forgiven because they were already compromised elsewhere thus OK to mail back to customers in open e-mail. The compromise occurred when Trustico wrongfully and totally inappropriately acquired them, far earlier.

This thus — familiarly to those of us who’ve followed CA and reseller follies over many years, as (more than practically anyone else) Bruce — yet another prime example of a firm fundamentally based on the assertion of trustworthiness proclaiming to the world, putting it up in lights and hanging a lampshade on it, that it was always utterly not to be trusted because either criminal or incompetent or both.

That aside, there’s additional bonus low comedy (as @me suggests) in a firm deciding that Digicert certs are insufficiently trustworthy and therefore shifting business to Comodo. That’s like deciding that investing your money with Groucho Marx was too stolid and ultra-respectable and therefore moving it to Harpo Marx. (The metaphor isn’t quite apt: I’d have trusted the Marx Brothers long before I’d have ever trusted Digicert or Comodo. Maybe it’s more like being happier that your sister is dating the Boston Strangler instead of Jack the Ripper on grounds that the Boston Strangler is, by comparison, a moderate.)

Peter A. • March 13, 2018 9:32 AM

Why they had private keys in the first place? Because customers want one-click service. They don’t even know how to generate a key pair and CSR. So the CA/reseller conveniently does it for them.

Mark • March 13, 2018 10:48 AM

Yet another reason why people who do not understand technology should not be in charge of technology. Unfortunately this is FAR too common. Most three letter executives don’t know an a–hole from an elbow, nor do they care to. There are no repercussions for them so why do they have to change.

oliver • March 13, 2018 10:56 AM

Why would any body not create the private key on that machine that it is used for?
This boggles the mind.!
The stupid, it burns!

Rodrigo • March 13, 2018 11:19 AM


I don’t know what to make of this story:

@Bruce:

May I suggest a Word Search puzzle?

It would be themed, of course… One word for each of the insecurities involced.

Matt from CT • March 13, 2018 12:05 PM

Rowley should have been more discrete about
seeking proof and more amenable to revoking
the certificates in the first place.
Now DigiCert has egg on their face.

This is entirely on Trustico.

As of today, RapidSSL’s CRL (http://crl3.digicert.com/RapidSSLRSACA2018.crl) has 1,428 out of 4,201 entries having been issued on March 1, 2018. You’ll need to use OpenSSL if you’d like to parse it out for yourself.

What’s DigiCert supposed to do with a request that increased the number of revoked RapidSSL (who Trustico resold, bought by Symantec, bought by DigiCert) certificates by 50% on a single date? Go ahead with “Alrighty, business as usual, nothing to see here.” ???

Digicert was right to ask, and had Trustico answered honestly the answer would have been, “Because we have a web interface for users to generate their private key in violation of the CABrowser forum standards on certificate resellers and therefore have a copy. Here’s the URL if you want to see how dumb we are for even dumber users.”

Matt from CT • March 13, 2018 12:12 PM

in violation of the CABrowser forum standards on certificate resellers

Which BTW, makes you wonder how they passed Comodo’s audit when they became a Comodo reseller while that web page was still up.

MB • March 13, 2018 12:47 PM

Non-technical people cannot be expected to understand the finer points of security.
Many people don’t understand that automation is anything but automated and requires constant vigilance and maintenance. They buy into it for the promise of effort, attention, and cost savings. They want technology to just work and/or to be disposable: like for a car, you take it to the shop once in a while and in the end you replace it with a newer model; you don’t need a permanent car consultant. But IT is being sold by unscrupulous salespeople as being as cheap, simple, and reliable as car ownership, which it definitely isn’t.
The promise of automation was automation, but in practice it means that every business owner needs to become something of an automation expert, hence situations like these, or needs to keep a (potentially untrustworthy, see San Francisco) automation expert on staff, something smaller business owners cannot afford and are probably strenuously resisting. Or it means effectively handing over their online business to some large conglomerate, like Paypal, 4square, or Amazon, which probably costs a lot, brings its own headaches, and is in the end no guarantee of safety (see above).
Combine this with an opaque system that makes it hard to assign responsibility and determine consequences and here we are.

Jason McNeill • March 13, 2018 1:20 PM

@Jim:

“Rowley should have been more discrete about seeking proof and more amenable to revoking the certificates in the first place. Now DigiCert has egg on their face.”

I agree. If the only proof Rowley would accept is a private key, and if furnishing such proof is itself a breach, then what exactly was Trustico supposed to do? Send hashes of the private keys? I’m not saying that it is good practice to send thousands of private keys in a file (whether the file itself was password protected or not). But when you are notified of a compromise, how should you respond? “I don’t believe you. Prove it. Show me the private key. Only the subscriber has the private key, and you’re just a reseller, so you don’t count. Therefore, for now, we will do nothing about these supposedly compromised certificates, not until you send us proof.”

Here are some quotes from DigiCert’s CEO from the Mozilla Dev Security thread; they seem to contradict themselves in my view.

========================

“Basically, our position is that resellers do not constitute subscribers under the Baseline Requirement’s definitions (Section 1.6.1).” […]

[…]

[…] “However, we insisted that the subscriber must confirm the revocation request or there must be evidence of the private key compromise.”

[…]

“At this time, Trustico has not provided any information about how these certificates were compromised or how they acquired the private keys.” […]

“Currently, we are only revoking the certificates if we received the private keys. There are additional certificates the reseller requested to have revoked, but DigiCert has decided to disregard that request until we receive proof of compromise or more information about the cause of this incident.”

[…]

“This raises a question about the MDSP policy and CAB Forum requirements. Who is the subscriber in the reseller relation? We believe this to be the key holder.“

David Rudling • March 13, 2018 1:41 PM

Seems to me that the whole TLS certificate system is worthlessly insecure.
There is surely a vacancy for a replacement for the TLS certificate system as the “next big thing”. Come on you guys in the land of opportunity – fame and fortune await.

Jesse Thompson • March 13, 2018 2:34 PM

@David Rudling

Sometimes I wonder why we don’t have better client-key options available.

So that instead of servers having their keys notarized by some race-to-the-bottom third party, clients simply create a keypair in their browser in place of or in addition to their username/password auth token.

This may not be enough to close the “when you create an account” MITM opportunity, but it’s enough to cover 100% of the remaining attack surface while eliminating “humans suck at passwords” in the same blow. So then all we need is a way to close the trust problem on account creation that doesn’t rely on numbskull notarization. DNSSEC sounds like one option there, but a marketplace of overlapping coverage options would also be nice.

MK • March 13, 2018 3:12 PM

“we are only revoking the certificates if we received the private keys”
How does this work if the keygen was from an HSM? I don’t think the private keys can be recovered.

Hmm • March 13, 2018 5:01 PM

@Jim, Jason

We don’t keep your keys – so we can’t lose them #Trustico > https://t.co/Po0AfLanUx pic.twitter.com/3Jm7oEhPqO
— GlobalSign (@globalsign) February 28, 2018

Wait, are we the keymaster or the gatekeeper or the keyholder or the subscribed reseller?
Dang these vague terms.. Let’s call the whole thing off and sell ads instead.

Clive Robinson • March 13, 2018 5:57 PM

@ MK,

Cool, I learned a new word. 🙂

I’m not sure you have…

“Croggled” appears to be a made up word from a ScFi book, in essence it was a borowed old nearly disused word which had the initial letter changed (such behaviour was quite frequent with ScFi authors of a certain vintage).

“Groggled” is however a slang word that has been in common parlance in the past though has fallen out of use and been replaced with the likes of “hung over” or less politely “shit faced”.

Groggled describes the way you feel on waking up from spending the night drinking grog (diluted dark rum) part of which is the “Oh God why have you not put me out of my misery” feeling from bad dehydration, brain shrinkage and poisons like tannins etc that give the dark rum it’s flavor. Also the potential bacterial posoning from the water that has been neither boild or purified and might well be contaminated by human waste when drunk in a tavern…

Grog came about from the Navy “Rum ration” that a sailor would receive daily. The water barrels on ship would hold at best “brackish water” thus adding dark rum helped take the taste away. Also it would help with trying to eat “Ships biscuits” / hard tack. The recipe of which was simply flour and water with some salt. Unfortunatly they were often not baked dry enough and were not propperly packed thus got infested and worse contained to much salt in an attempt to stop infestation. The biscuits would frequently be so badly infested with worms that there would be more worm and it’s wastes than actuall busicuit. But a sailor could be lashed for not eating a ration assigned to him, thus they could be used vindictively by deck officers…

Thus life as a sailor was not a happy one as Winston Churchill supposadly said when First lord of the Admiralty, Navy traditions were “Rum Sodomy and the lash”[1], in reality a brutish life with bad food and unpleasent diseases from the food like scurvy or salt poisoning. Eventually a prevention for scurvy was found which was pickled lemons (yet more salt). However a war denied the English access to lemons thus limes were used instead which is why an English sailor would be called a limey

The sailors would add their pickled limes to the grog again to try to spread some taste around.
Which is why a lot of dark rum cocktails have lime not lemon juice in them.

[1] Whilst the expression has been ascribed to Winston Churchill, when he was trying to get the admirals to accept oil rather than coal as the fuel for war ships. There is actually no record of him ever saying it.

Cris • March 13, 2018 7:09 PM

I don’t know what to make of this story… I am croggled by the multiple layers of insecurity here.

I believe you misspelled “incompetence”. And in reading today about Let’s Encrypt’s new wildcard certs, I’m really disappointed that the new ACME draft standard still does not require the CA to keep proof of a valid DNSSEC opt-out record when issuing via a non-DNSSEC method—it’s only RECOMMENED for them to even enable a DNSSEC-capable resolver! Doesn’t anybody notice this? I feel like I’m taking crazy pills!

William • March 13, 2018 8:05 PM

Let’s put our security int eh hands of third parties…what could go wrong? The current ssl system is absolutely not secure. Any CA that exists anywhere is subject to their countries government which means the keys are never secure.

Jonathan Wilson • March 13, 2018 9:20 PM

There are a number of alternatives out there that can replace the need for CAs for secure browser (EFF Sovereign Keys, DANE, probably others). What I dont understand is why no-one is interested in making these technologies more widespread. Is there something wrong with the alternatives to CAs? Are they not ready for real-world use yet?

Rick Moen • March 14, 2018 12:00 AM

@me, thank you very much for the Twitter stream. It fills in some gaps in the progression of business incompetence, greedy customer manipulation, and hapless failure — and there were also a few belly-laughs, there.

Dan H • March 14, 2018 6:52 AM

@Clive Robinson • March 13, 2018 5:57 PM

@ MK,

Cool, I learned a new word. :-)

I’m not sure you have…

It must be nice to be such a genius, to know everything about anything, to even know that someone who didn’t know a word really did know it because you say they did.

Clive Robinson • March 14, 2018 9:19 AM

@ Dan H,

Are you suffering from a lower GI tract issue? Things appear not to be sitting well with you.

There is medication you can get you know, alternatively you could stop chewing on what gives you the bloat…

Thunderbird • March 14, 2018 11:30 AM

Clive, according to this entry on “SF Citations for the OED” (apparently a project to collect science-fiction related terms for the OED), the oldest reference they have for “croggled” is from 1962. If someone was using it that long ago, I think it qualifies as “a word.” Most of the terms we use in the computer field are from later than that, at least with the meanings we assign them. And, after all, all words were originally made up somehow, even if they weren’t used by The Bard to describe his hangover.

I can see being all “get off my lawn” about “cromulent,” (which is also “a word”, even though it was created as a gag), but a nearly-sixty-year-old term seems pretty venerable (to me, at least).

I don’t vouch for the above URL since I put very little time and effort into my “research…”

justinacolmena • March 14, 2018 3:18 PM

@Bruce Schneier

I am croggled by the multiple layers of insecurity here.

There is a certain level of incompetence and stupidity which actually does descend to criminality and malice.

@Mark

Yet another reason why people who do not understand technology should not be in charge of technology.

If you understand the technology you are working on, but your boss does not, then (naturally) you have some explaining to do. If you keep certain things simple, then your explanation and documentation of the said technology can be simple and truthful as well. Otherwise your boss is not happy.

Someone with a degree or equivalent expertise in business administration or the like should not necessarily be expected to be an expert in computer programming or other highly technical field. All too often, when certain technological things are not as simple as they are supposed to be, they do not work the way they are supposed to work.

There are just too many people who appear not to be telling the truth in situations like these.

Clive Robinson • March 14, 2018 5:29 PM

@ Thunderbird,

“croggled” is from 1962. If someone was using it that long ago, I think it qualifies as “a word.

Some of us hail from before then… And can even remember the likes of “Telstar” by the Tornados in November of 62… Some are even before the Berlin Wall…

The point is though that it was an invented word for a story not a word that had evolved from something else. It’s also a “weak word” you just can not get behind it so no good for cussing either.

Seriously imagine you are up on a roof fixing the shingles and you have just welted your thumb with a hammer… Where’s the satisfaction thus pain relief of trying to execrate “croggled” it’s just not going to happen it’s got flabby vowel sounds unlike say “dag nabit” which is a good bowdlerization of a profanity.

r • March 14, 2018 7:42 PM

I’m croggled over your lack of Symantec wandering from weak word to fair prose, hysterectomy of course.

Mark • March 14, 2018 11:01 PM

@ justinacolmena I totally agree that there needs to be engineered systems in place to protect people from their ignorance or lack of understanding or use of basic security models.

FooCrypt ( without wanting to plug my own gear ) does have an getopts based interface which can be utilised from any shell ( CLI ), and hence like other products, can be included into any inbound / outbound SMTP process by an external call and forcing by DEFAULT, only encrypted emails to be sent using a generic or specific FooKey ( KEY ).

Takes about an hour to POC / and get up and running depending on the milter / etc, etc, knowledge, etc,. Saves endless embarrassments from people inappropriately disclosing security sensitive data.

RealFakeNews • March 15, 2018 12:55 AM

If SSL wasn’t broken enough.

At this rate I’ll be deleting the web browser from my computer. The whole thing is rotten, and no-one seems willing to fix it.

Who wants to go back to mailing lists? At least it notifies me of replies…

me • March 15, 2018 7:41 AM

@Rick Moen
no problem, hope is useful.
i think that this will win a “pwnie award” for the greatest epic fail.
(for who don’t know what they are; they are the equivalent “film oscar” for the best research, best vulnerability found, best epic fail, … there is great research papers in them, and much laughing)

Dan H • March 15, 2018 2:48 PM

@Clive Robinson AKA The-Know-It-All

Not that it is yours or anyones business, but yes, I do suffer from a digestive/intestinal problem from which I’ve been to the doctor and hospital numerous times for tests and they have yet to find a cause.

So, by trying to be funny, now not only are you a know-it-all, but you make fun of individuals with real and legitimate medical problems. I pity you.

Rachel • March 15, 2018 8:37 PM

@Dan H

sounds like you need a big hug. Here’s a big hug! wishing you lots of love and healing.
Now, lets drop the emotive stuff and return to all things technical xox

Doctor Science • March 16, 2018 12:02 PM

“and they have yet to find a cause.”

Maybe pull your head out? Just a suggestion.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.