Another Bloomberg Story about Supply-Chain Hardware Attacks from China

Bloomberg has another story about hardware surveillance implants in equipment made in China. This implant is different from the one Bloomberg reported on last week. That story has been denied by pretty much everyone else, but Bloomberg is sticking by its story and its sources. (I linked to other commentary and analysis here.)

Again, I have no idea what’s true. The story is plausible. The denials are about what you’d expect. My lone hesitation to believing this is not seeing a photo of the hardware implant. If these things were in servers all over the US, you’d think someone would have come up with a photograph by now.

EDITED TO ADD (10/12): Three more links worth reading.

Posted on October 11, 2018 at 6:29 AM35 Comments

Comments

Choubi October 11, 2018 7:13 AM

One thing is certain with this second story: that makes a lot of publicity for Appleboum and his business.

Stephen Tinius October 11, 2018 7:37 AM

“If these things were in servers all over the US, you’d think someone would have come up with a photograph by now”.

Or captured the traffic? Or revealed the embedded code?

KeithB October 11, 2018 7:57 AM

Unless the “chip” was just added to a corner of a huge processor chip at the wafer fab. This is why my employer requires a “Trusted” Fab.

Vesselin Bontchev October 11, 2018 7:57 AM

While the story is plausible and hardware implants exist (the NSA has been using them for years, according to the ANT Catalog), this Bloomberg story is bullshit, just like the previous one. Appelbaum is using Bloomberg to advertise his company, which sells “solution against compromised hardware”.

In general, what these two journalists publish cannot be trusted. Years ago they published an article about a cyber attack on an oil pipeline in Turkey, and the whole intelligence community was like “WTF is this nonsense?!”. They never published a retraction. And don’t even let me get started about the bullshit they have published about Kaspersky.

Wael October 11, 2018 8:52 AM

@Vesselin Bontchev,

Appelbaum is using Bloomberg to advertise his company, which sells “solution against compromised hardware”.

Seems to be a ‘marketing play’ or ‘PR stunt’.

So the video says: China doesn’t care about consumers’ data; China cares about “long term access” and access to “Intellectual Property”. Sounds like Bloomberg has access to the “intent” of the “attackers”, which leads to possible insider knowledge. Some schmuck at Supermicro sold them a Rolex.

Then again, what’s the evidence the HW wasn’t interdicted on its path?

Wael October 11, 2018 9:07 AM

@Alyer Babtu,

The institute’s semiautomated system “could have identified this part in a matter of seconds to minutes,” says Tehranipoor, an IEEE Fellow. The system uses optical scans, microscopy, X-ray tomography, and artificial intelligence to compare a printed circuit board and its chips and components with the intended design.

Of course!. It’s so easy to infer the “intended design”.

Clive Robinson October 11, 2018 10:03 AM

@ Bruce,

My lone hesitation to believing this is not seeing a photo of the hardware implant. If these things were in servers all over the US, you’d think someone would have come up with a photograph by now.

Yes and no.

First of how many NSA / GCHQ and other SigInt agencies hardware implants have been found in the wild and photographed?

Now assume some have been found, consider the reasons they might not be publically exposed as photographs or anything else for that matter…

It boils down to the target and their motivations rather than those of the attacker.

As far as I’m aware the only hardware to get photographed and made public by the MSM was the UK’s Guardian newspaper, after GCHQ’s “Tweedle dee and Tweedle dum” show went up to London from Cheltenham on a shoping trip, then drop by the Guardian’s basment to over see the destruction of the laptop that the Ed Snowden trove of documents had been on. The Guardian only did this to embarrass a certain very senior UK civil servant that should have known better decided to get into a “pissing match” with the senior at the Guardian and lost. Thus made a fool not just of himself but GCHQ personnel and revealed what might well have been “secret information” with regards what chips on the motherboaed of the laptop could have either a hardware or software implant/persistant malware.

People talk a lot about hardware and software implants in the supply line. Yes there have been a few there was IBM PC malware put on Apple iPods, and a well known supermarket chain had hardware implants conyaoning a cellular phone insert put in ePOS terminals.

By and large though the majority of what might be classified as implants have been installed by the manufacturer. Sony with the stolen FOSS they put on audio CDs, US mobile phone companies with the CarrierIQ technical support software, Lenovo with their varient of BadBIOS and just about every supplier of Android OS, MS with Win8 and above telemetary, god alone knows how many “walled Garden” apps.

The common thread beying mostly, “software” from the first link” of the supply chain on every system. Which as I’ve previously pointed out is the best solution for indiscriminately gathering data, but just about the worst for covert data collection. Which is why SigInt agencies appear to go for the last link in the supply chain and highly targeted.

Whilst a large or custom order may be known to the manufacturer, it’s actually not in their interest to implant all the motherboards but actually just a tiny fraction of them.

This is because the level of “Goods Inward Test” (GIT) required to find hardware implants involves more or less the compleate destruction of the “Device Under Test” (DUT). Not only is such GIT very very explensive it destroys the DUT. Thus the customer is going to only test at best a very very small fraction of the boards to this level. Almost certainly a lot less than 1%.

Now if you installed an implant in only one or two boards in a large order or repeate order again say a lot less than 1% then the chance of the implant actually being found is corespondingly very slight.

If the purpose of the implant is just to establish a toe hold and then exploit the other motherboarfs sideways across the network then instal a Core Memory (RAM) only item of malware as part of an “insider attack” then providing a false trail to how the malware got on the systems would not be overly difficult.

So whilst the idea of “seeing is believing” via a photograph is appealing the odds of it happening are very very slight indead.

Certainly not something I would cross my fingers for, let alone hold my breath, because the odds of it happening are way way to small.

Oh another thing is software implants are very difficult to atrribute which gives a covert operation lots of deniability. Using a chain of three “software vectors” allows what apear as “bugs” not “implants”, with the actuall “malware upload” comming across the network just as you would expect with a zero day or two. Thus just one implant in a whole server farm would be sufficient and the actual infection of the other motherboards looking like the result of a phishing attack or similar, not an actuall implant.

It’s kind of the way I would do things, and I’ve been saying just that for nearly two decades, so other people jave had plenty of time to think the same thing up…

Clive Robinson October 11, 2018 10:12 AM

@ Vesselin Bontchev,

… this Bloomberg story is bullshit, just like the previous one. Appelbaum is using Bloomberg to advertise his company…

I would not be certain Yossi Applebaum is doing that. I suspect that like Joe Fitzpatrick from the first Bloomberg story he was selected by the journalists.

Both Joe and Yossi have distanced themselves quite a bit from the respective Bloomberg stories.

Have a look at this,

https://www.servethehome.com/yossi-appleboum-disagrees-bloomberg-is-positioning-his-research-against-supermicro/

Little Lamb October 11, 2018 10:18 AM

hardware surveillance implants in equipment made in China

We are quite familiar with the adware, malware, spyware, trojans, worms, viruses, and keyloggers that plague Microsoft Windows and common off-the-shelf software such as Firefox, Chrome, and Internet Explorer, as well as, presumably, the more specific application software used to design the chips.

Same thing with, say, SCADA-controlled traffic lights that turn green all four ways.

The malicious software gets into all these systems, because of the prevailing luser porn-surfing I-don’t-care attitude among the guys who program this stuff.

Why aren’t there more women in the computer programming field? Right. The men simply haven’t had enough beer to be ready for women under Title IX blah blah blah ….

Hostile environment much? Hate to break it to you but that’s what you’ve got to program all the computers for. The honor system doesn’t apply when unauthorized intruders can hide their tracks so easily by greasing the palms of the local police.

There’s an Übermensch, and it’s definitely male no matter what because ever hooker on the block is reaching into its pocket for money.

Petre Peter October 11, 2018 10:18 AM

Remember! FTC and 32 state attorneys general penalizing Lenovo for installing spyware on their laptops. Maybe if such a team would get involved again, a picture won’t be necessary.

Paul Rain October 11, 2018 10:40 AM

I’d trust Chinese spy implants before I’d trust any ‘security’ product from Check Point.

TexasDex October 11, 2018 11:29 AM

“The denials are about what you’d expect.”

Actually, I would have expected them to be a bit less firm and unequivocal. The fact that they were so definitive in saying ‘this did not happen’ makes me a bit less sure of the report.

Matthew October 11, 2018 11:29 AM

This Bloomberg article looks like a rehash of Dragos Ruiu’s claim that his Raspberry Pi’s ethernet port contains spy implants.
https://twitter.com/dragosr/status/1001114342958317568

The Hackaday article disproving Dragos’ claim.
https://hackaday.com/2018/06/01/raspberry-pis-power-over-ethernet-hardware-sparks-false-spying-hubub/

The problem with Bloomberg’s article is that the details are vague which does not help us to identify the implant.
In today’s limited IT budget, we cannot afford to tear apart every servers just to look for a malicious implant which may not exists.
It is possible that instead of an implant, it is a hidden backdoor in the the firmware.

If the Bloomberg reporters have managed to get hold of an actual implant, they should reveal the information to help us to find other compromised servers.

Paul Rain October 11, 2018 1:01 PM

@Matthew: If the story’s true, some changes are gonna happen.

If the story’s not true, reporters are going to go to jail.

Whoever loses, we win.

Brandon October 11, 2018 2:43 PM

My problem with the story is the idea that no one has reported it publicly and no one made significant changes. You would think if the government found out about this, they would be wholesale tossing servers out of dozens of data-centers (because having the techs go in and manually remove the tiny piece would be too obvious – those techs would report it). Let alone Apple and Amazon…

On the opposing side, it makes a really great story to convince people to “buy American.” Publishing a scare-story would certainly make sense given the current trade climate.

Little Lamb October 11, 2018 5:05 PM

Donald Trump just signed the Music Modernization Act into law.

The hardware surveillance bugs are baked into our computers by law now, forevermore.

The Party has gone on too long. The whole damn symphony orchestra is out of tune, and they won’t stop playing their loud boom box in my computer.

Petter October 11, 2018 5:26 PM

Is these stories take down pieces on Supermicro or are all involved just denying and keeping shut until they have had a chance to remove the offending hardware?

There’s something that don’t add up.
Not sure where this began to wonder off but somewhere along the line there are either lies or non-denial denials.

Clive Robinson October 11, 2018 8:43 PM

@ James,

Old story, but likely still relevant

Many years ago, infact back last century by almost a couple of decades, I worked for a company that designed the first “trader tetminal” based on an 8086 CPU and 256K of RAM that did in a primitive way much of what the Bloomberg terimal does today and earns them around $25K per seat.

Even then we were concerned about what would now be called the “meta data” of traders activities.

Put simply you could see the search queries and messages sent to traders.

Having kept a watching eye on the market I heard stories around 2010 that sounded quite solid that some traders thought Bloomberg were spying on them and had been for some time…

A couple of years after that it became all to obvious that Bloomberg were indeed spying on traders and had become quite blatant about it. Specifically JP Morgan in London found that Bloomberg had done exactly that and it had probably cost the bank not just millions but billions…

So yeah, don’t trust a research database organisation that can see your searches, they might just be able to “guess ahead of you”.

And Bloomberg would be the last people on earth I would trust not to spy on it’s customers.

But under SEC rules market manipulation by deceit of the form Bloomberg appear to be trying to pull with this “China Implant” story is criminal behaviour.

Maybe it’s time Bloomberg’s “short and currlies” got a major tug, such that their toes leave the ground.

Steve October 11, 2018 9:49 PM

@Matthew: I am pretty sure gag orders do not extend to lying to the Senate.

I wouldn’t be so sure about that.

James “Least untruthful answer” Clapper is still running about loose.

As is Mr Justice Kavenaugh.

There’s clearly no penalty for lying to the Senate. Or lying from the Senate, for that matter.

Little Lamb October 12, 2018 12:51 AM

Michael Bloomberg and Brett Kavanaugh?

Almost like it ought to be a fight or something like that, but they all seem to get along splendidly well after they took over our rights, money, computers and everything else of value, and ran us off the property at gunpoint.

They go hand in hand down the aisle in Congress, the Senate or the House, like some damn wedding we weren’t invited to, and we’re supposed to forever hold our peace now.

So even yet more onerous restrictions on the life of ordinary people were enacted in the usual must-pass omnibus bill, and they like Kavanaugh’s reading of their interminably dense text.

tfb October 12, 2018 2:43 AM

I’m sure someone has suggested this, and it is anyway a conspiracy theory so inherently dubious. But the US is in a trade war with China, and has a current administration which is not, perhaps, above the odd lie or bit of underhand behaviour: wouldn’t it be convenient if a lot of stories about China doing something underhand got spread?

TimH October 12, 2018 3:59 AM

@keithB: If it’s a huge chip, even with the presumption that an extra circuit could be slid into unused silicon and not noticed during debug… how would it connect to the appropriate I/O?

Complicated designs used hardware libraries: RAM and ROM blocks, fast ADCs and DACs. Often the IP is hidden such that the chip layout designer only sees the connection points into the circuitry which looks like a blank part of the IC during layout. The standard pre-laid out circuitry itself is incorporated by the fab just before mask set manufacture.

That’s how to hide stuff… not deniably for sure, but so as not to be found.

Clive Robinson October 12, 2018 5:51 AM

@ tfb,

I’m sure someone has suggested this…

They have.

Have you ever noticed that the US only ever has on “existrntial thteat nation” at a time? And appears to select them from a short list of China, Iran, North Korea and Russia.

Well obviously those nations don’t say “OK fellers it’s my turn this month” to the others on the list, thus they are either all continuous threats or they are not. More likely than not they are actually not, they are nations that don’t think they should “toe the US line” and have built thrmselves up to the point where the US can not just “send in the boys”. Some years ago both India and Packistan were on the “US shit list” but by going nuclear they either got left alone or a seat on the “top table”. Other nations that started to look into gowing nuclear ended up getting invaded.

It’s not just US Politics there is a plan behind it. The US is it’s self a “vassal nation” to the oil producing nations, if OPEC coughs then Washington gets the sweats. But unlike many politicians and citizens some policy wonks are looking to the time when fossil fuels will be scarce. Thus the next “energy security” step for a nation is “go nuclear” yes it’s expensive and yes it is dangerous but unlike green energy it is dependable over “political time scales”. Thus the wonks have worked out that if they stop countries going nuclear then they have control over them, just as Putin currently does with his hand on the “gas tap” in winter.

Energy is becoming the new “water” that history has shown us many bitter genocidal wars were fought over (look up the history of “water rights” and “water wars”).

At the end of the day mankinds needs don’t actually change that much, but the scale of them does as does which is the scarcity at the time. Thus we see patterns in history that happen time after time, often a bit bigger in size of conflict and the fire lighting spark will be different, but the general lead up and conflict follow recognisably the same paths…

But that “one existential threat nation at a time” gag is right out of the George Orwell “play book”, and it was not new to him or various Roman Observers two thoudand years before him.

Phaete October 12, 2018 6:40 AM

From the article:
“They decided it represented a serious security breach, along with multiple rogue electronics also detected on the network, and alerted the client’s security team in August, which then removed them for analysis. Once the implant was identified and the server removed, Sepio’s team was not able to perform further analysis on the chip.”

So they have multiple implants and a team specialised in everything except chip architecture (or just plain incompetent)

Then they stop any analysis, do not consult further specialists and somehow throw away the evidence?

The same reporter as the original story also has a history of writing inaccurate and sensationalised stories while ignoring facts and feedback.

https://twitter.com/RobertMLee/status/1049617855396933632

Clive Robinson October 12, 2018 7:14 AM

@ Phaete,

Then they stop any analysis, do not consult further specialists and somehow throw away the evidence?

That’s not actually what is said in what you quoted.

One reason might be they contacted the FBI or other US National Security entity (there are quite a few) and the investigation then got taken over by the entity along with documents and other findings as “evidence”[1].

I’m not saying this is what happened but those who have started in on computer forensics have got used to “investigations getting taken over” by others as part and parcel of the job.

As for the credability of the journalists, their emoloyers don’t actually have much credability either. So… Bad apples falling close to the trees etc etc.

[1] I’ve come across this “evidence” excuse by UK Police to take away equipment to stop any further investigation. Trying to get back your equipment becomes an impossible task… When it was tried on another occasion I was prepared and gave them “dummy equipment” as there was never any come back that tells you virtually everything you need to know about their investigation.

Matthew October 12, 2018 8:48 AM

@steve

Government gag order is a get-out-of-jail document that protects the entity from legal liability of denying the truth.
Without its protection, the entity must be prepared to face criminal investigation, lawsuits and punishment.
Unless they have good lawyers and friends in high places.

An example is Apple. Tim Cook may have friends in Washington DC but other employees may not.
Apple and Tim Cook may escape punishment but whoever Apple sent to testify still faces perjury because he/she is still duty (and legal) bound to testify the truth.
So the choice is to be shafted by the company or by the government. In some cases by both.
Which is why whistleblowing exists.

Phaete October 12, 2018 8:54 AM

Let me say it in a different way.

The burden of proof is on Bloomberg and their journalists.
So far they have delivered a lot of ambiguity and hearsay and not even close to sufficient proof.

I understand why people want to believe them and provide extra possibilities(which are not explicit stated in the article) within the given ambiguity.

But any irrefutable proof is nowhere to be seen.

Weather October 12, 2018 11:04 PM

You can get rf Ic that are 1mm2 that only needs a ceramic resonator,
It wouldn’t be near the north bridge that’s linked to ram, it would be south bridge HDD etc,
The data channel’s plus about four more buses one interrupt, can’t remembered the rest,
They send a zero day over the net, and a second team read the data,mainly access keys, by rf link,
The bus speed plus mulpiter would limit were it could be installed, ram stick is out.

Clive Robinson October 13, 2018 4:11 AM

Another view point

@AlanS posted this link on the current squid page,

https://krebsonsecurity.com/2018/10/supply-chain-security-101-an-experts-view/

Around the time the first Bloomberg story was starting Brian Krebs intetviewed Tony Sager, Senior Vice President at the Center for Internet Security.

Tony is currently seen as a cyber-security evangelist and was a reviewer of code at the NSA.

He makes several comments about why some of the “perceived wisdom” in the industry is wrong, and why it realy falls down to perception of probabilities.

That is we can not audit all code, so you audit some and when you find an abnormality you make a big noise about it thus in effect scaring other coders into better behaviour.

Likewise you can not audit all hardware so instead you implement strong auditing and the equivalent of the “Two Key Launch Protocol”, where no one individual can make changes to the production process.

There is actually not much new in this for anyone who has worked in an environment with high levels of Quality Assurance.

What also gets mentioned is the thorny problem of security economics as it applies to the Internet of Things. With the suggestion that a model along the lines of the US Food and Drugs administration be adopted.

However Tony does make this comment,

    It’s now almost impossible to for consumers to buy electronics stuff that isn’t Internet-connected. The chipsets are so cheap and the ability for every device to have its own Wi-Fi chip built in means that [manufacturers] are adding them whether it makes sense to or not. I think we’ll see more security coming into the marketplace to manage devices. So for example you might define rules that say appliances can talk to the manufacturer only.

Actually demonstrates one of the major problems with “prescriptive” rule making. I for one see many problems with the “talk to the manufacturer only”.

Firstly but by no means the least, is I don’t want my potentially PII data going outside my defined perimeter. But more importantly it’s a form of unacceptable consumer lock in. We’ve seen similar with Amazon products, they sell to consumers and if there is insufficient income, not only do they stop supplying the product they turn off the “remote” asspect making all sold devices instantly useless.

But there is also the jurisdictional issue. Whilst a US rule might say “send to manufacturer only” the manufacturer might be in China which has an entirely different legal code. One of the reasons the EU brought out the EU-GDPR is because many US entities were cheating on the previous “safe harbour agreement”. The simple fact is though that unless another nation is prepared to honour the US or EU rules, once the data has left the jurisdiction there is little or no remedy.

The point is prescriptive rules have side effects many of which are undesirable, but worse are not seen at the time the rule is put in place. Thus you can end up over time with a veritable rabbit warren of tunnels made of exceptions before the original rule. Thus you end up with a great deal of complexity which we know will get exploited in one or more ways.

Mustafa Turan October 16, 2018 5:47 AM

Apple and Amazon are two companies that can calculate the risk of being proved wrong. I believe in what they say, because if they are found to be lying it would be a huge damage to their businesses. They would not take that risk.

raul October 21, 2018 5:56 AM

There might be no possible photo of the implant because the implant might be completely hidden in ICs when working under certain “out of spec” conditions. Carefully designed ICs could in theory operate in some rather different ways compared to the documented normal “within specifications” operation mode. The inspection would need to go really microscopic and with X-rays into the ICs. And still reverse engineering the exact combination of out-of-spec conditions that would bring a suspicious chip into its “backdoor” mode might be quite a challenge, even if finding something odd in the chip’s internal layout construction. What exact combination of what voltages or voltage drops (or something else in the signal) into what pins, and with which timing, to activate what?

JonKnowsNothing October 26, 2018 6:20 AM

@Clive Robinson, @Bruce, All

In the question of supply chain insecurity, there are some mentions of the TAO catalog (Tailored Access Operations) , however the ANT catalog (Advanced-Access Network Technology) might be a better Look-See for hardware exploitations. The ANT workers are what make the TAO work possible. The ANT catalog has images (for @Bruce) along with internal pricing.

The COTTONMOUTH USB series is interesting. The HOWLERMONKEY implant I would guess is much smaller these days.

HOWLERMONKEY – PCB layouts are tailored to individual implant space requirements and can vary greatly in form factor. 3 months delivery. 40 units @$750 each. 25 units @$1,000 each.

IF anyone with the propensity to create such things really wants to crack into the supply chain, it wouldn’t be too far from a possibility that it happens. The ANT/TAO folks are professionals at hiding such implants. It’s what they get paid to do.

A copy of the ANT catalog was published by SPIEGEL.de @12/29/2013.

Slightly Off Topic: The GCHQ refused to cooperate in the investigation of the Belgacom hacking inquiry.

UK refusal to cooperate with Belgian hacking inquiry condemned

ht tps://www.theguardian.com/uk-news/2018/oct/25/uk-refusal-cooperate-belgian-hacking-inquiry-condemned-gchq-belgacom
(url fractured)

It wouldn’t surprise me if the GCHQ/NSA was STILL hacking Belgacom and there are still firmware/hardware implants floating about the place.

  • COTTONMOUTH is the name of a venomous US snake.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.