Chinese Supply Chain Hardware Attack

Bloomberg is reporting about a Chinese espionage operating involving inserting a tiny chip into computer products made in China.

I’ve written about (alternate link) this threat more generally. Supply-chain security is an insurmountably hard problem. Our IT industry is inexorably international, and anyone involved in the process can subvert the security of the end product. No one wants to even think about a US-only anything; prices would multiply many times over.

We cannot trust anyone, yet we have no choice but to trust everyone. No one is ready for the costs that solving this would entail.

EDITED TO ADD: Apple, Amazon, and others are denying that this attack is real. Stay tuned for more information.

EDITED TO ADD (9/6): TheGrugq comments. Bottom line is that we still don’t know. I think that precisely exemplifies the greater problem.

EDITED TO ADD (10/7): Both the US Department of Homeland Security and the UK National Cyber Security Centre claim to believe the tech companies. Bloomberg is standing by its story. Nicholas Weaver writes that the story is plausible.

Tags: , , , ,

Posted on October 4, 2018 at 11:30 AM • 86 Comments

Comments

Sean Nienaber • October 4, 2018 11:51 AM

Bruce, while I agree we can’t trust anyone, we can reduce the risk my deploying servers which aren’t manufactured in China, or at least servers which have less components manufactured in China.

For example, Fujitsu server main boards are fully manufactured by Fujitsu in Germany and then distributed internationally.

If we can’t solve the problem, the least we can do is look to reduce the risks.

It’s not perfect but it’s a start.

tfb • October 4, 2018 12:02 PM

Whether or not this attack is real (Apple & Amazon are saying it’s not), I think it points out some of the risks we’re walking into.

Everyone wants not to run their own computers because it’s a pain, so we all want to run our stuff on infrastructure we rent from a small number of huge companies.

Those companies are running probably millions of hardware machines and I suppose tens or hundreds of millions of virtual machines on top of the tin. To operate at this scale they need to be able to do things like deploy changes across huge numbers of machines relatively easily. In the absence of magic, ‘easily’ means ‘it is easy to do bad things as well as good things’. So we’d better hope they never make a serious mistake or no-one ever gets some kind of hold over any of the people who can deploy changes at this kind of scale, because when they do, the impact is going to be catastrophic.

But, OK, let’s assume their systems people are perfect and never make any mistakes, so bad things like that never happen.

They’re still big companies, and they are deploying really large numbers of systems, all of which are identical. That means that they’re making huge orders to suppliers. And those orders can’t, really, be disguised, because they’re so big. So if their suppliers are compromised somehow, then they can specifically target orders from the large cloud companies, because they can see those orders. They can intentionally inject compromised hardware into just the places it will do the most damage.

(And of course, there are other organisations who, we assume, buy very large numbers of machines: those orders, too, are visible, and those machines, too, can have bespoke compromises added to them.)

Bad things are coming.

Jill England • October 4, 2018 12:50 PM

Most useful hacks require sending secrets from the target back to the attacker. This attack is I believe intended to be against weakening network security. If your network does not connect to public and internet you are “safe”. Don’t be blind. Biggest threat is still a thumb drive in the pocket.

TheInformedOne • October 4, 2018 1:00 PM

No one is willing to admit it but we are squarely in a cold war with China. The US has spies deep in China and visa-versa, just like we had with Russia. Each side does everything it can to subvert the other, without using conventional warfare or revealing their tactics to the free press. Would the US have agreed to allow computing hardware to be manufactured in Russia during the heart of that cold war? I think not. So how did we get to this point with China? I would say Hong Kong was a big part of it. When Hong Kong reverted back to China it was a perfect opportunity to invite the rest of the world into mainland China to conduct business with an economy of 1 billion people which had previously been perceived as a somewhat risky venture. Were the Chinese finally opening their economy up to the world for greedy corporations to plunder? Hardly!! They saw it as an opportunity to attack and enslave other economies and steal intellectual property. I’ve heard a joke that China has 1 secret weapon against the Trump tarrifs….Walmart! If you look at all the Chinese spy’s (just the ones who were caught) passing billions in US military technology to China over the years, we’d have declared conventional war on them by now if we weren’t so economically dependent (on them). You see, China has us over a barrel. Our government wages a cold war, while our economy is addicted to cheap offshore manufacturing void of environmental oversight. Tough spot to be in….

Christopher • October 4, 2018 1:10 PM

I’ve seen several comments that the reason the companies are denying it is National Security Letters. This seems very unlikely to me. In Under Seal v. Sessions it was decided that the non-disclosure requirements are constitutional, but as far as I know there is no basis for a NSL that requires a company to affirmatively make false statements. It’s possible, I guess, that they’re doing it on their own accord because there’s an investigation that they have decided is somehow helped by them making false statements, but I don’t think there’s any police power that would allow the US or state governments to compel them to make false statements.

CPC • October 4, 2018 1:47 PM

What I find fascinating is that this attack wasn’t found faster. I was expecting something embedded in chips, not an extra piece of hardware glued to the board. It seems implausible that no one at Super Micro looked at the boards they received from China and realized that they had an extra chip that wasn’t present in the design. What kind of quality controls do they apply? Did they find the chips and ignore them? Was their QA department compromised?

Winter • October 4, 2018 2:05 PM

“not an extra piece of hardware glued to the board”

Reportedly, the chips were extremely small. More like a dot.

Impossibly Stupid • October 4, 2018 2:17 PM

Our IT industry is inexorably international, and anyone involved in the process can subvert the security of the end product.

I don’t believe that’s necessarily true. The problem is mainly that of businesses that use a single supplier, often times to manufacture an entire product white label. If they had simply stuck with designs that were modular and used standard interfaces, we wouldn’t have the vulnerable mono-culture that exists today.

@Jill England

The biggest threat is that of misplaced trust, as Bruce points out. If you falsely trust that your network is safe, your greatest vulnerability might be anything (hardware or software) that, say, bridges it to an open wifi network. That way, the attacker could simply roll up into your parking lot with a mobile hot spot that gives them access to all your “secure” machines.

anon • October 4, 2018 2:33 PM

Do the British people still think it’s a good idea to construct that nuclear power plant in Britain, with China involved?

Bill • October 4, 2018 2:34 PM

In March of 2016, there were reports of an Apple supply chain attack on their servers that are consistent with what we are seeing in the Bloomberg story.

The summary of the story can be found in an article “Apple Building Own Cloud Servers To Ward Off Data Snoopers: Attackers Have Infiltrated Supply Chain, Say Insiders”. The article can be found here:

https://www.player.one/apple-building-own-cloud-servers-ward-data-snoopers-attackers-have-infiltrated-supply-521878

The source story is available here, but requires disclosing email to get access:

https://www.theinformation.com/articles/inside-apples-cloud-infrastructure-troubles

I am not sure of the relationship between the 2016 story and the current Bloomberg story. Could these articles be related?

Cvnk • October 4, 2018 2:44 PM

@CPC Have you seen a fully populated motherboard? There are so many nooks and crannies where a totally inconspicuous chip could hide unnoticed. It’s very unlikely a human could visually scan everything and spot something unusual. And while they have techniques for automating visual inspection it wouldn’t be very effective for populated boards — too many obstacles. It would be limited to verifying only certain aspects.

Once a motherboard is assembled and shipped back to the OEM I suspect the best they can do is run a functional test on it.

AlanE • October 4, 2018 3:11 PM

Is it not the case that, even if Apple et al have removed the offending servers, their enterprises can/will remain forever compromised?

The hypothesis here is that, when one of these servers is brought online, the VERY FIRST THING IT DOES is to execute code that deposits a sleeper worm in some remote corner of the enterprise it’s targeting. So, even if said back-door server is later removed, the malware remains.

I’m posing this as a question for Bruce and others. Makes sense to me; I’m wondering if such a scenario could be operative in the real world. It could definitely explain why Apple et al are denying they ever used these servers to begin with.

chris • October 4, 2018 3:41 PM

@CPC ” It seems implausible that no one at Super Micro looked at the boards they received from China and realized that they had an extra chip that wasn’t present in the design. What kind of quality controls do they apply?”

@Cvnk “Have you seen a fully populated motherboard? There are so many nooks and crannies where a totally inconspicuous chip could hide unnoticed. It’s very unlikely a human could visually scan everything and spot something unusual. ”

The graphic at the beginning of the Bloomberg article shows a stuffed motherboard, then sequentially removes layers of chips, ending with a circle highlighting the “extra” chip. Looks like it was originally underneath another, much larger chip (with six or eight pins) or at least very close to one of the pins. Would not be visible unless the board was taken apart very carefully. Of course, I have no idea whether the graphic is correct or not.

As CPC suggests in a later post, more clever would be to bury the functionality in another, nominally “standard” chip – silicon is cheap on the nation-state scale of things. Then, like VW Diesel engines programmed to pass emissions tests (( https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal )), the chip could be programmed to pass the appropriate functional tests that might be applied to that standard chip. Yes, we’re paranoid – but are we paranoid enough?

cuckleberry finn • October 4, 2018 3:45 PM

I’ll throw my insane conspiracy theory into the pot.

I think enforced denials are in play to prevent backlash from the public if the scope turns out to be much worse than admitted to. This one gets admitted to? Researchers turn their attention away from whatever else they are doing and start looking into this. Let’s say it turns out we are royally fucked…banks, cloud providers, creditors…well..

Seems to me best route from govt perspective is to stfu and run the electronic voting handbook which is to say ignore the noise and claim that it’s fine.

echo • October 4, 2018 5:15 PM

https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies

U.S. officials had caught China experimenting with hardware tampering before, but they’d never seen anything of this scale and ambition. The security of the global technology supply chain had been compromised, even if consumers and most companies didn’t know it yet. What remained for investigators to learn was how the attackers had so thoroughly infiltrated Supermicro’s production process—and how many doors they’d opened into American targets.

Unlike software-based hacks, hardware manipulation creates a real-world trail. Components leave a wake of shipping manifests and invoices. Boards have serial numbers that trace to specific factories. To track the corrupted chips to their source, U.S. intelligence agencies began following Supermicro’s serpentine supply chain in reverse, a person briefed on evidence gathered during the probe says.

Ham fisted discrete component snoopware meet ham fisted admission of espionage capability?

Every time I watch a video of TSR-2, Bluestreak, or hear anything about transputers I want to pull the ear of the nearest UK politican and beat them with my umbrella. I will also never forget how Dyson was bailed out by his own facotry staffwho worked to produce his vacuum cleaners and the second his business was secure and he sensed the opportunity he moved production to Asia and closed the factory. This egomaniac also supported Brexit and like many of his ilk he has no sense of general wellbeing for society and certainly not the social chapter elements of the EU.

I’m still wondering where the “off switch” is in the F-35 and why the software is closed to partners in the F-35 project.

Oh, it’s all a bit silly isn’t it?

Name • October 4, 2018 5:25 PM

hide an tiny chip under a large enough dip or socket?
“We give you the Spy Coprocessor[1] at no extra charge.”

  1. “For you spy on us, of course. Don’t be silly.”

Name • October 4, 2018 5:30 PM

Dyson was bailed out by his own facotry staffwho worked to produce his vacuum cleaners and the second his business was secure and he sensed the opportunity he moved production to Asia and closed the factory. This egomaniac also supported Brexit and like many of his ilk he has no sense of general wellbeing
Oligarchs are “run government like my mob” idiot savants. They tend to install Adolf Hitlers, who run over everyone, including their sponsoring oligarchs. (Wolves in sheep’s clothing get eaten by wolves)

You know my Name. Look up the number. • October 4, 2018 5:42 PM

Boards have serial numbers that trace to specific factories. To track the corrupted chips to their source, U.S. intelligence agencies began following Supermicro’s serpentine supply chain in reverse
Whaaa? Nobody was smarr enuff to file the serials off the PCBs?

Name • October 4, 2018 5:53 PM

Yes, we’re paranoid – but are we paranoid enough?
We know SOmEboDy notorious, who is paranoid enough to bury something else in his paranoia.

Thoth • October 4, 2018 8:18 PM

@all

Ever wondered why this “scandal” only surfaces now and not at other time ?

New ammunition of hurt being pumped down the line to destroy businesses at the most opportune moments ?

Political motivations ?

Petter • October 4, 2018 8:50 PM

@Harry

Regarding TDK filter featured on photo in the Bloomberg article.
I believe they simply took a good looking 0805 filter to represent the filter component the mentioned evil chip supposedly could have been mistaken for. And simply used it for the photo sessions.

Clive Robinson • October 4, 2018 8:51 PM

@ CPC,

It seems implausible that no one at Super Micro looked at the boards they received from China and realized that they had an extra chip that wasn’t present in the design.

As I mentioned over on the Friday Squid page[1], there are two ends to the supply chain, that in China and that in the last link which is the delivery to customer phase.

It is this last link that is favoured by the NSA for inyerdiction because it gets past the issue you have noted.

It’s interesting to note that the Bloomberg artical talks of unicorns jumping rainbows and sticks thrown into headwaters of a famous Chinese river washing up in Seattle but don’t go into the interdiction possabilities.

In essence there are two ways you can play this game, the first is everybody gets one of “fire and forget” style attacks, the second is the “targeted entity” attacks that the NSA ptefered last link interdiction attacks work against.

The fact that so few have been discovered tends to suggest it’s the NSA prefered method in play, which in turn suggests that maybe Bloomberg were looking at the wrong end of the supply chain for some reason…

You would expect an investigative journalist to go into this more likely aspects. The fact they did not rings little alarm bells that they have been fed this story from somewhere…

Thus I would ask “Who has most to gaon by kicking off yet another anti-China campaign in this current political climate?” I guess most people could make a reasonable guestimate…

@ Bruce,

We cannot trust anyone, yet we have no choice but to trust everyone.

Actually whilst the first point is true, the second is not.

As “the usuall suspects” know, I looked into this issue quite a long time ago and there are ways to mitigate “below CPU” attacks and lower on the computing stack. Whilst “bubbling up attacks” are impossible to deal with in the way we build commodity hardware currently there are ways both hardware and software can be organised to effectively mitigate these attacks.

I explained part of the reason why we have problems and mentioned the mitigation over on the squid page[2].

Oh and as @Thoth is only too aware some academics at UCL in the UK (George Danzis etc) have nicked the idea and have developed a commercial solution they were trying to get funding for.

[1] https://www.schneier.com/blog/archives/2018/09/friday_squid_bl_644.html#c6782859

[2] https://www.schneier.com/blog/archives/2018/09/friday_squid_bl_644.html#c6782864

Jester • October 4, 2018 9:04 PM

Setup an instrumented honeypot network with fake Internet connection while doing deep packet inspection (DPI). You know – the stuff AT&T and NSA thrive on.
Heck in the USA, even your ISP is performing continuous DPI of your data then monetizing it…

Simulate the PLA with some juicy fake treasure document files.
Or Use a known identical ‘clean’ server and run the exact same code. Filter out identical network traffic and capture what’s left.
Data analysis and reduction will further clarify.

Clive Robinson • October 4, 2018 9:28 PM

@ tfb,

Everyone wants not to run their own computers because it’s a pain, so we all want to run our stuff on infrastructure we rent from a small number of huge companies.

More fool them, they were warned before cloud computing got going it was a very very bad idea.

By the way it’s actually not “everyone” even though the trade press and marketing would have you think so.

As @Bruce notes above,

    No one is ready for the costs that solving this would entail.

Again whilst not entirely true as some of us do have an idea of what mitigation costs will be. Because there are ways some of us have developed over the years that CPU and below attacks can be mitigated and there are prototype products out there from which scaled-up cost estimates can be made.

But whilst such costs are not going to be small by any means and the performance will to put it politely “suck”, things will improve if the industry finally decides to go for secure hardware designs.

The real problem will be how best to utilize such hardware so you get as much bang for your buck as you can. Currently this is an almost empry field of endevore that will quickly fill up once it gets going as there are reputations to be made in it.

But the point our host @Bruce has not mentioned which no doubt he will is that this is just an extension or continuation of the Meltdown and Spector timeline issues. We are now moving into the so far mainly ignored realm of “Below CPU in the stack attacks” that “bubble up” the stack rendering any software or it’s tool chain compleatly vulnetable.

In the past these attacks were due to I/O DMA attacks via the likes of high speed serial busses like Firewire. Then the issues of “reach down” attacks like RowHammer became practical and more recently Meltdown and Spector have become a point on the hardware vulnerability time line.

Which has got to the point where ignoring it or trying to bolt bits on is nolonger viable and a compleate rethink on computing hardware is required. The fact that this is coincident with Moore’s law being far from relavent any longer means lots of issues are coming together at the same time which can not all be solved as seperate issues, they need to be amalgamated to form one or two solutions which almost certainly will result in significant changes to how we develop software. The future is almost certainly massively parallel from below the CPU upwards. Which means much of the human predeliction of sequential thinking will likewise need to be changed…

Clive Robinson • October 4, 2018 9:39 PM

@ Harry,

This is the “chip” they claim

That is not a “chip” but a surface mount transformer.

If you google “balun” you will find a lot of information about such “BALanced to UNbalanced” transformers made from the simple multiple windings, through coax and twisted pair transmission lines through to PCB based transmission lines.

So unless they are talking about hiding things underneath it or by disguising them as it, it’s not a “chip” in the sense most would understand.

Clive Robinson • October 4, 2018 9:51 PM

@ anon,

Do the British people still think it’s a good idea to construct that nuclear power plant in Britain, with China involved?

Actually the China involvment is the minor issue. The major issue is EDF which is a French company that is part syate owned that is in effect bankrupt. Against all EU legislation to the contrary EDF is kept in business by illegal cross subsidizing and illegal State funding.

Due to the stupidity of Thatcherism the French regard the UK energy market as a “cash cow” they can keep milking. Unfortunatly for them people are begining to wise up.

Sadly though our politicians are still “idiots on the take” so nomatter how bad the deal gets they will keep throwing money and other hidden back handers EDF’s way…

As for China it’s self Brexit is causing those Chinese investors George “gidiot” Osbourn ex-chancellor was so enamoured of to take their money else where.

It’s time for people to look back at why the Bretton-Woods Conference wad deamed necessary in the first place, before history repeates it’s self.

Skeptic • October 5, 2018 1:56 AM

Sure, China spies. So does US, UK, RUS, GER, etc.

But a 3-pin chip was capable of doing all this whilst being installed on thousands of boards running inside US military, Apple, Amazon (30 companies in total) for up to three years, and nobody noticed a thing?

And the companies in question vehemently deny this, instead of saying ‘no comment’.

And suddenly now this surfaces after a month of Supermicro being delisted and becoming ripe for takeover or bankruptcy. After a year of fairly sizeable short positions aginst it stock.

Something smells really rotten in all this. Bloomberg or not. Gitta love those ‘anonymous sources’.

tfb • October 5, 2018 2:38 AM

@Clive Robinson,

I agree, not technically everyone. But, for instance, quite probably your bank, and if not your bank then a bank which, if it became unable to trade because of a large-scale compromise, would take your bank with it. (I am not pointing fingers at any specific bank here: most of the UK banks are too big to fail still because that was a problem it was politically inconvenient to fix.)

I also agree that the sorts of hardware we currently rely on is the problem. But, further, the pervasive trick of running your code in a VM or container which you own sitting on top of an underlying OS and hardware which you don’t means you have to have complete trust in the people who do own the lower levels of the stack not to snoop on you. That’s insane, in my opinion.

me • October 5, 2018 4:12 AM

i saw from the article that apple did not share information with police.
they also did not share the compromised hardware with them.

now i’m thinking about the prism slide from nsa.
given the fact that they don’t want to collaborate do you think that they have been hacked by nsa back in 2012?

we know that many us company gave direct access to their data, the slide say “apple included 2012” but i’m starting to think that they have been hacked by nsa and they did not want this.

Same doesn’t apply to other company like google.

me • October 5, 2018 4:30 AM

@Jester

Or Use a known identical ‘clean’ server and run the exact same code. Filter out identical network traffic and capture what’s left.

it’s not that easy for some reasons:
1-the chip might not be always on, so you need to do this test for long time
2-even if you assemble two identical pc they will behave differently, even without malware chip because: hdd, cpu, mac address, serial numbers are different and also the hdd reading time is different, mouse click timing, all this is used to generate random numbers.
this is by design so it’s not that easy to spot the differences.

Clive Robinson • October 5, 2018 4:44 AM

@ tfb,

That’s insane, in my opinion.

Yup, and I’ve been telling people that for years…

The problem is fixing it, those who could most certainly don’t want to and others actively encorage them not to (DoJ/FBI -v- Apple etc).

Unfortunatly the only solutions available to “the common man” appear to be,

1, Don’t play.
2, Don’t play where people can see.

The first option is generally not open to the majority that wish to participate in society in the normal sorts of ways.

The second involves being disconnected in more ways than most can imagine. Nearly all our hardware and the OS’s and Apps that run on it are quite deliberatly designed to be promiscuous in every which way possible. Supposadly for “user convenience”.

Thus you need to seriously consider “energy gapping” and “OpSec” as the easier solutions, rather than trying to control the hardware and OS etc which is going to fight you every step of the way. Not that energy gapping is particularly easy or “OpSec” conducive to thoughtfull working. Even now people who should know better think instinctively “it’s paranoid behaviour” rather than effective self defence.

Such is the way of the world when certain types see only $$$ there are always those following close behind them who see $$$^2 as a secondary but more lucrative market…

Which is why over the years I’ve made the information available to those who do want to think about their “self defence” a little more effectively.

Cassandra • October 5, 2018 5:03 AM

As ever, The Register’ reports on this, and both the article and comments are worth a look.

One thing to note is companies can deny having received National Security Letters, but as such letters can be addressed to targetted, specifically chosen employees, who cannot then discuss the NSL with their employers, it is entirely possible for an NSL (or several NSLs) to be served and enforced without the company having any official explicit knowledge of it (or them). So the company’s PR team are not necessarily lying when saying that the company has not been in receipt of an NSL, they may just not know the whole situation, and are insufficiently cautious to realise they are assuming they know everything relevant.

It is also not unusual for large companies to have teams of people who are security cleared, who are not allowed to tell their management the details of what they are doing, as the management do not have sufficient clearance. They are a headache to manage, and you have to rely on people’s integrity.

So the fact that a company’s PR team deny something happened does not mean they are correct.

The PR team will usually include, or have access to, legal advisors, who also may well not be in possession of the full facts. A flat denial on the basis of what you know, when you are not in possession of the full facts of the case, is not necessarily lying.

Cassandra

Steve • October 5, 2018 8:54 AM

I find it interesting that when Snowden releases decade old NSA docs, everyone gobbles them up whole, but when Bloomberg reports some similar activity by a foreign power, it’s skeptic-o-rama.

Of course, given the current Administration’s latest “China did it” ploy, one does have to wonder if Bloomberg is getting played in a different way.

Just sayin’.

Clive Robinson • October 5, 2018 8:58 AM

@ Cassie,

A flat denial on the basis of what you know, when you are not in possession of the full facts of the case, is not necessarily lying.

True, that’s why it’s often called “Plausible Deniability”…

As for NSLs given to employees, apparently some work contracts actually have specific liability clauses for them now. That is if the employee gets one they get to be sued out of existance by the company for with holding it from senior managment.

Catch 22 for an employee, but if the Feds play dirty and we know they do, it’s not surprising when others start playing dirty in the “mine are steel and yours are only brass” game.

As NSL’s were and still are legaly dubious at some point it is going to get publicaly nasty big style. And due to the way the psyco’s in DoJ/FBI play I suspect they have Apple in mind for a re-play in the not to distant future. Because they know Tim Cook made theirs look like Xmas baubles under a wreaking ball and that’s an ego burner of high magnitude for some. Hopefully sensible heads will prevail, but they probably won’t…

Clive Robinson • October 5, 2018 9:09 AM

@ Steve,

Of course, given the current Administration’s latest “China did it” ploy, one does have to wonder if Bloomberg is getting played in a different way.

Opinion appears divided between that and some one trying to get the share price of a certain manufactuter down in value.

That said they do not of necesity have to be different people as they say “One hand washes the other”.

What ever people might think both the Bloomberg piece and CNN piece on the Bloomberg piece were at best shody jounalism, which means no supprise when people start looking for ulterior motives in a time of “fake news” they are likely to find smoke, be it from a gun or without fire…

vas pup • October 5, 2018 9:17 AM

@all: we should see broader question (I mean forest behind the trees): involvement of foreign manufactures/suppliers/developers into domestic critical infrastructure.
As I recall, e.g. on opposite side: Saddam and Serbian guy were both burned because Saddam build his fiber optic communication infrastructure with foreign country company more friendly to US then to Saddam’s money. This let destroy all fiber optic hubs based on layout obtained and blind Saddam’s communication. With Yugoslavia – same was with phone network which was used for Command and Control of air defense system (that was build by Germans and obtained from them). As you see, all are vulnerable on that as soon as utilized potentially not friendly supplier. My best guess that when high tech weapons supplied/sold to other countries it should be kind of protection imbedded (unknown to buyer)so that weapon could not be used against (e.g. aircrafts of seller).
On China: if somebody owns you more that trillion $$, you want to monitor his (US) ‘health’ and intentions – just common sense. I doubt China wants to destroy US economy – its largest market.

Bystander • October 5, 2018 11:12 AM

Attacking server hardware this way means that the most likely path for command & control as well as for exfiltration of data is the network. You can – of course – imagine other paths, but you do not necessarily control where the hardware will be installed and have access to the installation.

I wonder if the network traffic in large server installations can be monitored sufficiently to discover the communications generated by such an attack. I think if the communication is handled clever enough, the detection can be avoided for some time.

Now, does this mean that:
– Server hardware must be subjected to security-screening prior to installation?
– The screening must be done for each item?
– Are the server boards the only hardware to be screened?

Chris • October 5, 2018 12:06 PM

It is possible to manufacture such a tiny core, but I fail to see how this component could tap a sufficient number of signals on the board to do anything useful.
I’d be much more worried about firmware hacks.

SpaceLifeForm • October 5, 2018 5:32 PM

@chris

If one has net and root, you can hack the firmware.

At a low silicon level, you have both net and root.

You are pwned.

Phaete • October 5, 2018 5:42 PM

Story smells like cowdung.

We can decap chips and reverse engineer them to know it’s functions.
Where are the pictures of the decapped chips?
Where are the regions on that chip that posses functionality that that chip shouldn’t have?
What functionalities are those?

Never thought i’d say this, but:”Pictures or it didn’t happen”.

PS. Look up Ken Shirriff, electronupdate or many others for examples of what i mean.

bigmacbear • October 5, 2018 6:19 PM

@Chris: The Register article suggested the device would be placed inline with the serial interface between the Baseboard Management Controller (BMC, also known as iLO, iLOM, or iDRAC) and the main CPU. As such, it would be in a position to intercept firmware updates traveling that path to their ultimate destination in EEPROM or Flash memory.

Bong-Smoking Primitive Monkey-Brained Spook • October 5, 2018 10:33 PM

@Bruce:

Bloomberg is reporting about a Chinese espionage operating involving inserting …

Ahem, chop g’s foot and shift it left by one letter.

, investigators determined that the chips allowed the attackers to create a stealth doorway into any network that included the altered machines.

Seriously? So a chip on a computer is able to bypass all firewalls and other network protection controls? Tell me the chip is on a network device such as a router, switch or a gateway, and I’d be likely to believe it. Oh, wait a second! The chip creates a wireless out-of-band communication channel… Yea, I can buy← pun alert that.

@Ratio:

One more datapoint:

And that would be: “We have no clue”, yes?

Bong-Smoking Primitive Monkey-Brained Spook • October 5, 2018 11:03 PM

I wouldn’t worry too much about it. It looks like an early prototype. Got news from an acquaintance who put the chip under an electron microscope. It looks like this.

Methinks the chip communicates with a so-called “Radio Telescope” to send and receive (read: inject and infiltrate) information from/to the 母船

JG4 • October 6, 2018 8:02 AM

Thanks for the helpful discussion. @MarkH – I’m surprised that something like AI wasn’t mentioned in the context of dealing with extraordinarly dense proofs. I thought that software logic applied to proofs had made great strides and even had been mentioned here in the past couple of years. @MonkeyBrain – I thought that the explanation of the chip’s mystical powers were that it could inject arbitrary code into the proceedings. Combined with sufficient knowledge of the overall system architecture and firmware, that might allow defeat of all security – i.e., the opening of a backdoor. X-ray imaging would allow very detailed inspection of boards. I am pretty sure that x-ray imaging can get to the scale of nanometers or less and would be able to see even mask gamesmanship. Just using a CCD camera to compare an incoming board to the reference image would be quite powerful. A nice analogy to inspecting a software image to a known-good hash. Speaking of nice analogies, there is a useful comparison of climate desertification to trust desertification in one of the stories.

https://www.nakedcapitalism.com/2018/10/links-10-6-18.html
…

Stone used as a doorstep for the last 30 years is meteorite worth $ 100K Vajuu (Kevin W) […link is broken – this works]
https://vaaju.com/stone-used-as-a-doorstep-for-the-last-30-years-is-meteorite-worth-100k/

Why the next three months are crucial for the future of the planet Guardian (David L)

This Is Not A Blip The Baffler (Anthony L). Why don’t these people come up with better titles?!?

Researchers Created ‘Quantum Artificial Life’ For the First Time Vice
…

Users are Reporting Lost Data After Installing Windows 10 October 2018 Update ExtremeTech (furzy). Ugh.

Space Force seeking applications for transfer to mobile infantry Duffle Blog (Kevin W)
…

Big Brother is Watching You Watch

Instagram is testing the ability to share your precise location history with Facebook The Verge

Undercover cops break Facebook rules to track protesters, ensnare criminals NBC (furzy)

Does Your Motherboard Have a Secret Chinese Spy Chip? PC Magazine (David L)

Bloomberg’s Spy Chip Story Reveals the Murky World of National Security Reporting TechCrunch

California Bans Default Passwords on Any Internet-Connected Device engadget
…

MGD • October 6, 2018 12:42 PM

A small reality check …
Suppose the ‘chip’ in the image is about the size of the foible …
In, addition, suppose that it has a few layers of silicon …
How much computing power (processor, microcode, control pins, etc.) could be engineered on a chip of that size? … How would it gain control of the server motherboard?

Jack • October 6, 2018 3:30 PM

@Steve : The difference between Bloomberg and Snowden is “annonymous sources” vs actual documents from the horses own mouth .
I’ve seen this “Snowden said, claims”, or whatever, many times in the presstitute MSM and it’s just wrong, Snowden did not claim anything, he released the horses own papers .

Wesley Parish • October 7, 2018 4:09 AM

After reading the Register article on it, I was left with the conundrum:

https://www.theregister.co.uk/2018/10/04/supermicro_bloomberg/
What lithography is it using? 7nm?

7nm – seven nanometres or seven nautical miles? Seven nautical miles is such an impressive size for a microchip, one wonders why more chip foundries don’t copy. 🙂

And yes, gotta watch them inductors. Them pinko baluns is up to no good!

Fitwerme, I’d be inclined to put such an item inside the SoC, copying good ‘ol Intel with its various pricing schedules, where with full price you’d get a full 80×86, but with reduced pricing you’d just get somewhat less. (didn’t Sun also play such fun-n-games with core games too?) Much less trackable that way.

That way, ordinary customers would get a little extra – but unusable – silicon real estate, while the target organizations would get just that little extra in full operation, courtesy of the Whatever-They-Choose-To-Call-Themselves.

In short, Bloomberg’s anonymous sources disappoint me. I’d like to know where I can get a microchip that uses seven nautical mile lithography.

Clive Robinson • October 7, 2018 5:35 AM

@ Phaete,

We can decap chips and reverse engineer them to know it’s functions.

Err you might want to check up on that assertion you’ve made.

Because you will probably find it is actually an assumption you have made.

Bystander • October 7, 2018 6:23 AM

@Wesley Parish

I don’t know if this just a joke you made but anyway:

In short, Bloomberg’s anonymous sources disappoint me. I’d like to know where I can get a microchip that uses seven nautical mile lithography.

This might just be your perception. As can be seen in the NIST document, nm ist the correct symbol for nanometer within the SI unit system:
https://www.nist.gov/sites/default/files/documents/2016/12/07/sp330.pdf

The NOAA chose to use the same and though confusing notation for nautical mile while other organizations use M ( International Hydrographic Organization and International Bureau of Weights and Measures), NM (International Civil Aviation Organization) or mni (IEEE and GPO).

Phaete • October 7, 2018 7:36 AM

@Clive Robinson

"Err you might want to check up on that assertion you've made.

Because you will probably find it is actually an assumption you have made."

Checked up and remains solid.

I guess you haven’t checked the 2 names i gave you before assuming assumptions.
Or just even google “Reverse engineer chip” and follow the instructions, providing you have your lab.

Is this actually you Clive Robinson? Normally you have reasonable good info, not just casting doubt without any info or facts like now.

echo • October 7, 2018 10:19 AM

@Phaete @Clive

I don’t have the expertise to know but gather decapping and reverse engineering isn’t gauranteed. Not only can it fail but countermeasures are possible. Not only this but even this assumption assumes you’re not missing something else…

phaete • October 7, 2018 6:19 PM

@echo

Sure, you can hide stuff in hardware, just have a look at chips that generate keys.
You can have obscured parts, tamper proofed parts.
But they are easy identifiable as such and should not be on that part.
So show me pictures of those area with those functions on that die that should not be on a signal conditioner.
Or just a picture of the die of the original part and the ‘spy’ part.
Well, not just show me, show the world the proof for your allegations mister Bloomberg.

People often forget that a chip is just a electronic print board in small.
There is no magic, it’s all just components and connections.

Wesley Parish • October 8, 2018 4:39 AM

@Bystander

Given that I write satire and sometimes get it published, what would be the balance of probabilities?

I’m actually worried that the current Recumbent in the Oval Office might hear that the Chinese have been making and selling microchips with seven nautical mile lithography, and might slap tariffs on those as well. He’s more than capable of doing just that, and the rest of the Executive Branch are capable of backing him in so doing.

I like a laugh.

With regards to the story, I rather like one of ElReg’s commentators, who pointed out that this story looks suspiciously like the US behaviour prior to releasing Stuxnet into the wild; also the one who remarked that Intel had already done something very much like this with their Management engine – a pirated version of Tanenbaum’s Minix.

Clive Robinson • October 8, 2018 4:44 PM

@ Wesly Parish, echo,

Intel had already done something very much like this with their Management engine – a pirated version of Tanenbaum’s Minix.

The story is a bit more complicated and Andy Tanenbaum did comment on it (but I can’t find the link).

From what I remember Intel had asked Prof Tanenbaum to write some specialised memory drivers and add ifdef’s to take out other code loke floating point and other bits and pieces. From what I remember some of these were not issued under the general Minix licence others were. However Intel had been less than candid about what the specialised drivers were to be used for, thus they had in effect behaved in a less than honest fasion. As Prof Tanenbaum noted Intel used an older less secure version of the Minix microkernal which was designed for education NOT security.

The upside though is that the most popular Freeware *nix in the world is not GNU/Linux but Minux, because most Linux boxes also run Minix in the ME as do most Win OS PCs.

I’m sure Prof Tanenbaum would like a dollar for every instance of his Intel requested changes but hey what would he do with the money unless he decided to run for US President 😉

echo • October 8, 2018 6:18 PM

@Clive, @Wesley Parish

I couldn’t find a link on the driver issue. I did find an open latter by Andy Tanenbaum which highlighted a few simple issues before criticising Intel for its disingenous silence about the purposes they wanted to put Minix to and creating 1984 on a chip.

https://www.cs.vu.nl/~ast/intel/

Ratio • October 8, 2018 10:00 PM

George Stathakopoulos, Apple’s VP of Information Security, wrote a letter to Congress:

In light of your important leadership roles in Congress, we want to assure you that a recent report in Bloomberg Businessweek alleging the compromise of our servers is not true. You should know that Bloomberg provided us with no evidence to substantiate their claims and our internal investigations concluded their claims were simply wrong.

We are eager to share the facts in this matter because, were this story true, it would rightly raise grave concerns. A compromise of this magnitude, and the effective deployment of malicious chips like the one described by Bloomberg, would represent a serious threat to the security of systems at Apple and elsewhere. That’s why, ever since we were first contacted by Bloomberg’s reporters in October 2017, we have worked diligently to get to the bottom of their allegations.

[…]

In the end, our internal investigations directly contradict every consequential assertion made in the article—some of which, we note, were based on a single anonymous source.

Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server. We never alerted the FBI to any security concerns like those described in the article, nor has the FBI ever contacted us about such an investigation.

[…]

In the situation Bloomberg describes, the so-called compromised servers were allegedly making outbound connections. Apple’s proprietary security tools are continuously scanning for precisely this kind of outbound traffic, as it indicates the existence of malware or other malicious activity. Nothing was ever found.

Wesley Parish • October 9, 2018 4:31 AM

@echo

Read that three-clause BSD license again:

https://github.com/minix3/minix/blob/master/LICENSE

Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

As Prof Tanenbaum states:

https://www.cs.vu.nl/~ast/intel/

After that intitial burst of activity, there was radio silence for a couple of years, until I read in the media (see above) that a modified version of MINIX was running on most x86 computers, deep inside one of the Intel chips. This was a complete surprise. I don’t mind, of course, and was not expecting any kind of payment since that is not required. There isn’t even any suggestion in the license that it would be appreciated.

My point is that Intel did not indicate anywhere in its documentation that it was including a cut-down version of Minix as its Management Engine aka spy engine. And that was most emphatically a redistribution of Minix. As I remember, it was someone comparing the binary of the Management Engine with Minix that made it obvious.

As such, as Intel didn’t fulfill even that far-from-onerous requirement of the license, that leaves them in breach of it. So, they are Software Pirates, and the BSA should be notified, and Intel sued into the ground, according to the BSA’s standard practice.

Clive Robinson • October 9, 2018 6:00 AM

@ Ratio,

With regards Apple’s VP of Informatiin Security George Stathakopoulos, writing a letter to Congress, colour me unsuprised.

As I’ve already noted the Bloomberg artical was lacking not just testable evidence but evidence at all. Thus I was supprised the SubEds let it pass.

I’m not saying there is not evidence of some tampering somewhere but again as I indicated you would have to try and work out which end of the supply chain was affected. The NSA’s favoured end is the last link for interdiction on targeted organisations to keep their activities covert. In theory China can only effect the first links of the supply chain, which would make their optimal stratagem for information gathering one of “infect all”.

Whilst there may be exceptions (large/custom orders) the fact that others have not seen these hardware add ins suggests if they are there “covert” operation at the last links in the supply chain not the first links.

But there is a fly in the ointment, supply chain poisoning can take many many forms, with most quite difficult to find. Thus a stratagem for robustness would suggest using as many different forms of poisoning as possible but only one or two per board or shipment. If done in the right way it also gives increased “plausible deniability”.

All of that said as normal we will have to wait for more information, we might even find it was in effect “insider trading” as share prices had significant changes at what some have said is a critical time. I’m guessing that is not something Bloomberg want to hang in the air for very long, thus I guess they are likely to be quite busy currently…

Impossibly Stupid • October 9, 2018 10:51 AM

@Clive Robinson

In theory China can only effect the first links of the supply chain

I don’t believe that’s true. In the modern world, it is not unheard of to have semi-custom products drop shipped directly from the factories in which they are built. That makes it much easier not only for China to bypass the full supply “chain”, but to also target specific populations (e.g., cell phones being purchased in Washington, DC) or possibly even high-value individuals.

Regardless, it’s not like domestic companies are doing a lot of inspection on the products they’re getting made in foreign countries. It seems like there isn’t a week that goes by that doesn’t have a recall of products that still are violating ancient lead paint standards. This is what you get when you outsource everything with the sole goal of making things cheap.

echo • October 9, 2018 1:47 PM

@Wesley Parish

My point is that Intel did not indicate anywhere in its documentation that it was including a cut-down version of Minix as its Management Engine […]

Whoops! How embarassing. There’s me complaining about people not reading their own documentation and failing to follow their own rules and I miss this. It’s on one page. Right at the top.

@Impossibly Stupid

I don’t believe that’s true. In the modern world, it is not unheard of to have semi-custom products drop shipped directly from the factories in which they are built. That makes it much easier not only for China to bypass the full supply “chain”, but to also target specific populations (e.g., cell phones being purchased in Washington, DC) or possibly even high-value individuals.

This is true not to mention corruption possibilities of staff working in warehouses and delivery companies.

Clive Robinson • October 9, 2018 3:45 PM

@ Impossibly Stupid,

I don’t believe that’s true. In the modern world, it is not unheard of to have semi-custom products drop shipped directly from the factories in which they are built.

I guess you did bot read down to where I said,

    Whilst there may be exceptions (large/custom orders)…

Oh and with regards the NSA and my musings on their activities Bloomberg are now all over it big time. Have a read of,

https://www.bloomberg.com/news/articles/2018-10-09/new-evidence-of-hacked-supermicro-hardware-found-in-u-s-telecom

The thing is there is a lot of words in both Bloomberg pieces, but very little of any substance let alone evidence. Thus people are now distencing themselves from the articles,

https://9to5mac.com/2018/10/09/bloomberg/

Which is starting to leave Bloomberg “out on a limb” and potentially open to all sorts of hurt. Which is why I was initially supprised to see them double down. However with a little more reading the second piece appears to be an attempt to open up space thus potential deniability…

The fact that they have mentioned Mosad and Israeli intelligence agents into it appears even more distancing from the original article.

All most odd and definitely worth keeping an eye on. This might turn into a “three bowl”[1] –of popcorn– problem to entertain us with.

With regards,

Regardless, it’s not like domestic companies are doing a lot of inspection on the products they’re getting made in foreign countries.

No, nore are they ever likely to either.

Quite a bit of kit, especially consumer kit is not designed to be repaired let alone be taken appart these days. Often they are “shop ready” in nice boxes sealed with shrink wrap etc. Thus to check you would have to destroy finished items from China… Most managers would see that as a compleate failure of LEAN and similar protocols and processes.

So yes slipping a poisoned pill bit of hardware in is not exactly difficult, especially when correct testing would appear to be a deep chasm in the ground into which endlessly throwing money would be cheaper to do than carrying out such testing…

[1] Yes Sir Arthur’s character Sherlock Holmes used to talk of “three pipe problems” but smoking especially that of a pipe is considered so guach these days.

echo • October 9, 2018 7:53 PM

@CLive

There’s nothing wrong with a “three pipe problem”. I much prefer the emotional resonances of this to a “three popcorn bowl problem” which is altogether more snarky. As amusing as I used to find it Auberon Waugh style humour is very much out of fashion, not to mention conceited and cruel, such as his referring to those he perceived as beneath him as “people who pour their milk in first” or the lower classes. The issue being they couldn’t afford quality teaware and poured the milk in first before boiling water to avoid breakage.

Wesley Parish • October 10, 2018 3:06 AM

@usual suspects

Chinese Super Micro ‘spy chip’ story gets even more strange as everyone doubles down
https://www.theregister.co.uk/2018/10/09/bloomberg_super_micro_china_spy_chip_scandal/

The new story pointing to an Ethernet hack is clearly intended to act as support for the original story but since the details are so different, and given that the entire report is single-sourced, it has had the opposite effect among security experts who have started to doubt the credibility of the original story.

There’s an infamous put-down that springs to mind:
He reached rock bottom then started digging.

Also this little beauty of a comment:

Fitzpatrick then engages in some speculation about why the Chinese government would actually use the specific method that the story covered. “There are so many easier hardware ways, there are software, there are firmware approaches. The approach you are describing is not scalable. It’s not logical. It’s not how I would do it. Or how anyone I know would do it,” he said.

Indeed. I find it difficult to imagine what payload something as small as alleged provide. Below a certain size, you can’t have anything resembling memory; ROM iirc, is always larger than RAM; some gates in ROM get ablated iirc.

The more I think of it the more I get to thinking that someone’s been sold a bill of goods

Clive Robinson • October 10, 2018 6:32 AM

@ echo,

I’disliked Auberon Waughn from my earlist contact…

Humour is generaly based on,

1, Inflicting pain on others.
2, Shared or common interest.
3, juxtaposition.

There are various “types” that use the first to belittle others, a modern day version being the observation of,

    Thank God for Sainsbury’s, it keeps the riff raff out of Waitrose

When I first heard it I actually laughed because anyone who actually knows anything about the way Waitrose work[1] would realise it reflected more on the person making the comment than anyone else, only it was not ment to be self depreciating.

When I was young I was known for practical joking and other mild social misdeeds. Unlike others I’d worked out there were rules to the game. The first of which was never ever pick on a lesser target, the second it should not just be funny to others it should be funny to the target themselves, thus all can share the joke harmlessly. Thus if you wanted to do the “bucket over the door gag” you would use a very light weight plastic container and confetti, and pick a day like the targets birthday and stick a birthday card in with the confetti.

But appart from the occasional gag I still do, these days I appreciate situational or observational humour based on real life. That is you see something odd or juxtaposed and you not just see the humour in it you encorage others in finding it funny.

Laughter, fun and wonderment are three of the greatest possessions mankind has, and they should be used to do good. Bringing a smile, happy face or look of wonderous awe to someones face every day should be the number one goal for people. Because that is where real power lives.

[1] Waitrose panders to the bottom end of “professional” “non-U” fraction of society, that gives themselves a particular set of “airs and graces” that no other class in UK society does. They are usually of the proffesional class that think they have failed to reach a sufficient status in a desired area of their life thus they tend to be both bitter and mean inside. The most obvious being “crittics” with the worst being the “Patetnal knows best” attitude. A quite obvious one to most in the UK being Michael Gove and his wife.

Clive Robinson • October 10, 2018 9:19 AM

@ Wesley Parish and the usual suspects,

I read this just under a week ago,

https://securinghardware.com/articles/hardware-implants/

And was as usuall mulling it over, basically it said nothing that has not been said here by @The Usuall Suspects one way or another over the years and it contained an assumption that I disagreed with for various reasons.

Thus I was just going to let it pass untill @SpaceLifeForm posted the link.

So in reply I made comment that Joe Fitzpatrick was not quite on top of things here,

https://www.schneier.com/blog/archives/2018/10/friday_squid_bl_645.html#c6783004

Then the Bloomberg article kind of exploded unsuprisingly due to the fact it wiped out a lot of value in several large international companies.

The problem the article was realy bad, lacked any kind of evidence showed an incorect part and in my opinion should have been killed by the SubEd’s, Editors, or legal teams at Bloomberg (which I’ve kind of said). Because of the market loss asprct and Bloomberg sets it’s self as a “Market Organisation” others have brought up the specter of “insider trading” or “illegal market manipulation”. As some one effectively noted you don’t knock 2% of the value of the worlds largest company and not expect follow up…

My issue is the how and why. As I’ve explained in theory the supply chain has many links. China can control the first few links but not the last few which is where the likes of the NSA play with “interdiction” as this keeps things not just highly targeted it keeps things rather more covert, which in theory is generally the number one aim, even over getting data. This is because if your implanted “backdoors” get discovered you have no idea if the data you are pulling is real or what your target wants you to see as in the old school “double cross” by an agent (in this case the machine the implant is on).

Likewise in theory at the manufacturing end of the supply line their best option to “collect data” is to implant every system without exception. Obviously this makes the likelyhood of the implant being discovered much much greater than desirable.

Thus in practice the manufacturer would look at “exceptional orders” or “specific / custom orders” to implant. Further they would some how diversify the implants to increase “plausable deniability” by using multiple typrs of implant and multiple supply chain nodes such that suspicion would fall as well down stream of them as possible.

Now what we have been told by Bloomberg in the first article is that a number of companies we know actually do practice forms of supply chain tampering analysis at their goods inwards phase had been hit… These companies then denied they had been.

This is where the technical side of the game gets interesting. A full analysis for software or hardware implants is not just usually destructive of the DUTs but generally only looks for “Known Classes of Attack” or worse still “Known Instances of Attack”.

Due to this there is a very real advantage for the attacker. They know the target organisation only tests a very small percentage of DUTs to this level certainly down below 1% which means if they only implant well below 1% their odds of getting caught are below 0.01% on straight odds. Which sounds like bad attack odds unless you have a two or three stage implant. Look at the hardware implant as being a “toe hold” attack that then moves sideways to other motherboards via another “bug” that then in turn leverages another bug to bring in the attackware and spread it as a RAM only attack. To most people analysing the attack backwards they would be looking to “clean up” and might not follow the chain all the way back, esspecially if the attacker gave them a more visable but false trail to follow.

The sort of people known to be able to do this are the UK’s GCHQ and the US’s NSA. Whilst other nations (Australia, China, France, Germany, Holland, Israel, etc etc) are believed similarly capable and the US has made public some of the Dutch and Israeli capabilities. Although it’s no big secret Israel has long been suspected of getting into US and The Wests communications and computing infrastructure for years as “standard policy”. Because they like the French have used it for industrial espionage reasons… Unsuprisingly some countries ban the import of equipment with “Israeli taint” such as Motorola products etc and have done for over a third of a century because they have caught the Isreali’s returning to the scene of their crimes as a dog does unto it’s vomit.

Whilst I was expecting further comment from both Joe Fitzpatrick and Bloomberg, Joe’s was kind of what I expected. Bloomberg on the otherhand have doublrd down and started playing the distancing game by bringing up the NSA and similar big style as well as Mossad…

I’m guessing that Bloomberg realy do not have anything currently thus are going to try to slide behind the “National Security” defence unless they can get real evidence to support their original claims as time goes on. Otherwise Bloomberg could be getting their collars felt by SEC and quite a large civil suit or class action from various shareholders…

This story is growing legs quite rapidly and they are kicking Bloomberg’s ball in front of them as they run…

echo • October 10, 2018 5:50 PM

@Clive

I do agree. I believe humour should be mutually enlightening. I don’t care for rude or nasty “humour” or harmful “practical jokes”.

I liked magic and adventure. Situational and observational humour is funny. I like people watching too. It’s much more difficult today to pull the kinds of tricks in the Sherlock Holmes novels such as judging profession or class or what they have been up to. Perhaps it’s age or a change of interests but I find my focus for people watching has changed away from this to more who they are as people in a social context.

Clive Robinson • October 10, 2018 9:40 PM

@ Bruce and the usual sudpects,

Thr second Bloomberg article started trying to put distance in with mentioning both thr NSA and Mossad.

They mainly quoted from Mr. Yossi Appleboum who had apparently been “at the coal face” in Israeli Intelligence dealing woth cyber-threats at some point in time.

Like Joe Fitzpatrick, Yossi Appleboum is upset with the way Bloomberg used him and has said so in an inyerview with Mr. Patrick Kennedy of “ServeTheHome” (STH),

https://www.servethehome.com/yossi-appleboum-disagrees-bloomberg-is-positioning-his-research-against-supermicro/

The whole article is a good read, but Mr Appleboum’s main point of contention is that Bloomberg are trying to take the general problem of supply chain poisoning and incorrectly just hang it all on Supermicro, instead of the tech industry in total.

It’s a point I’ve made before for computers and it applies both for hardware and software. The two key sentances of Mr Appleboum’s comments on this issue are,

    The problem is that when you get the hardware how can you make sure the product was not compromised? Someone can replace modules that validate hardware with other modules that say it is okay.

As I’ve pointed out several times in the past, this heart of this problem predates the invention of the electronic computer and also predates Alan Turing’s work on the halting problem, that is often claimed as the seminal work on computers.

The work was by Kurt Gödel and is his two incompleteness theorems. Published back in 1931, they are theorems of mathematical logic that bring forth the inherent limitations of every formal axiomatic system capable of modelling basic arithmetic and the underlying logic.

They are important not just to mathematical logic and the philosophy of mathematics, they also have a more down to earth bearing on the logic on which all computers are based.

In short Gödel demonstrated that no consistent system of logic capable of supporting mathmatics could describe it’s self. The upshot of which is you can not ask a computer if it has been tampered with because the tampering could either mislead the computers logic, or instruct the computer to give a false output…

It is in short a bit of a problem as there is no real solution to the problem that is 100% all the time with what is in effect a single CPU system (ie whilst it might have multiple cores it can be seen as just a single CPU within the rrst of the system because the cores are not hierarchical).

However as I’ve explained before there are ways you can use state machines that are below Turing complexity to halt the CPU(s) and walk through the code memory and do range checking on registers etc. Whilst not 100% it gets you quite close especially with other hardware bounds.

As a few here are aware I explained about it as part of C-v-P some years ago on this blog. As others have noted a group of supposed researchers have effectively filched the idea along with the implementation idea by @Thoth using smart cards and have built a system a little while ago, that is actually less secure than it could be (I guess they did not read it all 😉

Oh one thing to remember about “implants” in the supply chain be they software or hardware, is that they are by definition all “insider attacks”… Which can not be prevented by any currently known means due to the “unknown unknowns” issue.

That is you can only search for known classes amd known instances, which is the point that Mr Appleboum is making, we have to be open about all instances of implant found then form them into classes of implant that can thrn be checked for. If the industry ignores the fact that implants are a general issue across the industry and instead fall for the Bloomberg line of “Supermicro” only then implants will not just get more common, but more effective.

Wael • October 10, 2018 10:48 PM

@Clive Robinson,

As a few here are aware I explained about it as part of C-v-P some years ago on this blog.

You know how to get my attention 🙂

I guess they did not read it all 😉

Clearly! Or they got lost in the metaphors and Tea-drinking stories, Shakespeare, etc… Or perchance they did not use the right pair of glasses to read between the lines to see who ‍‌‍‌‍‌‍‍‍‌‌‍‌‌‍‌‍‌‌‍‌‌‍‍‍‌‌‍‌‍‌‍‍‌‌‍‍‍‍‌‍‌‌‌‌‍‍‌‍‌‍‍‍‍‌‍‍‌‍‌‍‍‍‌‍‌‍‍‌‍‍‌‍‌‍‍‌‍‍‍‍‌‌‍‍‌‍‍‍‌‌‍‌‍‍‍‍‌‌‍‍‍‌‌‍‌‌‌‌‍‍‌‍‌‍‍‍‍‌‍‍‌‌‍‌‍‍‍‍‌‍‍‌‍‍‌‍‌‍‍‍‌‌‍‍‌‍‍‌‌‌‍‍‌‌‍‌‌‍‍‍‌‍‌‌‍‍‌‍‍‌‌‍‍‌‌‍‌‍‌‍‌‌‍‍‌‌‌‌‍‍‌‍‌‌‍‍‍‍‌‍‌‍‌‌‍‍‍‍‌‍‌‍‍‌‍‍‍‌‌‍‌‍‌‍‌‍‍‌‍‍‌‍‌‍‍‍‌‌‍‍‌‍‍‌‌‌‍‍‌‌‍‌‍‌‍‍‌‌‍‍‍‍‌‍‌‍‍‍‌‌‌‍‌‌‍‌‌‍‍‍‍‌‌‍‌‌‍‍‌‌‍‍‍‌‍‍‍‌‌‍‍‌‌‍‌‍‍‍‍‌‍‍‌‌‍‌‌‌‌‍‌‌‍‍‍‌‌‍‌‌‍‌‌‍‌‍‌‍‌‍‌‌‍‍‌‌‌‍‌‍‌‍‌‌‍‍‍‍‌‍‌‍‌‍‌‌‌‍‌‍‍‌‌‍‌‍‌‌‍‍‌‌‌‍‌‍‌‌‍‌‍‍‌‌‍‌‌‌‍‍‌‍‍‌‍‌‍‍‌‌‌‍‌‌‍‍‌‌‍‍‍‌‍‍‌‍‌‍‍‌‌‍‌‍‍‍‍‌‍‍‌‍‍‌‌‌‌‍‌‍‌‌‍‍‌‍‌‍‌‍‌‌‌‍‍‌‌‍‌‍‌‍‍‌‌‍‍‍‍‍‌‌‍‍‌‍‍‍‌‍‌‍‌‌‌‍‌‍‍‌‌‌‍‍‌‌‌‍‍‌‍‍‌‍‌‌‍‌‍‍‌‍‌‌‍‍‍‍‌‍‌‍‍‍‌‍‌‍‍‌‍‌‌‍‌‍‌‍‌‌‍‍‍‌‌‍‍‌‍‍‌‌‍‌‍‍‍‍‌‌‌‍‌‌‍‍‌‍‍‌‍‍‌‍‌‍‍‍‌‌‌‍‌‌‌‍‌‍‍‍‌‌‍‌‌‍‍‍‌‌‍‍‍‌‌‍‌‍‍‌‍‍‍‍‌‍‌‍‍‍‌‍‌‌‍‍‌‌‌‍‌‍‌‌‍‍‌‍‌‍‌‍‌‌‌‍‌‌‌‌‍‍‍‍‌‌‌‍‍‌‌‍‌‍‍‌‍‍‌‍‌‍‍‍‌‌‌‍‌‌‍‌‍‍‍‍‌‌‌‍‍‍‍‍‌‌‍‍‍‌‌‍‌‌‌‌‍‍‌‍‌‍‍‍‍‌‍‍‌‌‍‌‍‌‌‍‌‍‌‌‍‌‍‍‌‍‌‌‍‍‍‍‌‍‍‌‌‌‍‍‌‌‌‍‍‍‍‍‌‍‌‌‍‌‍‍‍‌‌‍‍‌‍‍‍‌‌‍‌‍‌‍‌‌‌‌‍‌‍‍‌‍‍‌‍‍‌‍‌‍‍‍‌‌‌‍‌‌‍‌‌‍‍‍‌‌‌‍‌‍‌‍‌‍‍‌‍‍‌‍‌‍‍‍‌‌‌‍‌‍‍‍‌‍‌‍‌‌‍‍‌‌‌‍‌‍‌‌‍‌‍‍‌‍‍‍‌‌‌‍‌‌‍‌‌‍‍‍‌‌‍‌‌‌‍‍‌‌‍‍‍‍‌‍‌‍‌‌‍‍‍‍‌‍‌‍‍‌‍‍‌‌‍‌‍‍‍‍‌‌‍‍‍‌‍‍‌‍‍‍‍‌‌‍‌‍‍‍‍‌‍‍‌‌‍‌‍‍‌‍‌‌‍‍‌‍‍‍‌‍‌‍‌‌‌‍‌‍‍‌‌‌‍‍‌‌‌‍‍‌‍‍‌‍‌‌‍‌‍‍‌‍‌‌‍‍‍‍‌‍‌‍‍‍‌‍‌‍‍‌‍‌‌‍‌‍‌‍‍‍‌‍‌‌‍‌‌‌‍‍‌‍‌‍‌‌‍‍‍‌‌‍‍‍‍‍‌‍‍‌‍‍‌‍‌‍‍‍‌‌‌‍‌‌‍‌‍‍‍‍‌‌‌‍‍‍‍‍‌‌‍‍‍‌‌‍‌‌‌‌‍‍‌‍‌‍‍‍‍‌‍‍‌‌‍‌‌‍‌‍‌‌‍‍‍‌‌‍‌‌‍‌‌‍‌‍‌‌‍‌‌‍‍‍‌‌‍‌‌‍‍‍‌‌‍‍‍‌‍‍‌‌‍‌‌‍‌‍‌‍‌‍‍‍‌‍‌‌‌‍‍‌‌‍‌‍‍‌‍‍‌‍‌‍‍‍‌‌‌‍‍‌‌‍‌‍‌‍‌‌‍‌‍‍‍‍‌‌‍‍‍‌‍‍‌‍‌‍‌‌‌‍‌‍‌‍‌‌‍‍‌‌‍‌‍‌‌‍‌‍‍‌‍‍‌‍‌‍‍‍‌‍‌‍‌‍‍‌‌‌‍‍‌‌‌‍‍‌‌‍‌‌‍‍‍‍‌‍‌‍‌‌‍‍‍‍‌‍‌‌‍‌‍‍‌‌‍‌‌‍‍‍‌‍‍‌‌‍‍‍‌‍‍‍‍‍‌‍‌‌‌‍‍‍‍‍‌‍‌‍‌‍‍‍‌‌‍‍‍‍‌‍‌‍‍‍‌‌‌‍‌‍‍‍‌‌‍‍‌‌‌‌‍‍‌‍‌‍‌‌‍‌‍‍‌‍‌‍‌‌‌‍‌‍‌‍‍‍‌‍‌‌‍‍‌‌‌‍‌‌‍‍‍‍‌‍‌‍‍‍‌‌‌‍‌‌‍‌‌‍‍‍‌‌‌‌‍‌‍‍‌‍‍‌‍‍‌‍‌‍‍‍‌‌‌‍‌‍‍‍‌‌‍‍‌‌‌‌‍‍‌‍‌‍‌‌‍‍‌‍‍‌‌‍‍‌‍‍‌‌‍‌‍‍‍‍‌‌‌‍‍‍‍‍‌‌‍‍‌‍‍‍‌‍‍‍‌‌‌‍‌‍‌‍‌‌‍‍‌‌‍‌‍‌‍‍‌‌‍‍‌‍‍‍‌‍‍‌‍‍‍‍‌‍‌‍‌‌‍‍‌‌‌‌‍‍‌‍‌‍‌‌‍‌‍‍‌‍‌‍‍‌‌‍‌‍‍‍‍‌‍‍‌‌‌‍‍‍‌‍‌‌‍‍‍‍‌‍‌‍‌‌‍‍‍‍‌‍‌‌‍‌‍‍‌‌‍‌‌‍‍‍‌‍‍‍‍‌‌‍‌‌‍‌‍‌‌‍‌‍‍‍‌‌‍‍‌‌‌‌‍‌‍‍‌‍‍‌‍‍‌‍‌‍‍‍‌‌‌‍‌‍‌‌‍‌‍‍‌‌‌‍‌‌‍‍‌‌‍‍‍‌‌‍‌‌‍‌‍‍‌‍‌‍‍‍‍‌‍‍‍‌‌‍‍‍‍‍‌‌‍‍‍‍‌‍‌‍‍‍‌‌‌‍‌‍‌‍‌‍‌‍‌‌‍‍‌‌‌‍‌‍‌‌‍‍‌‍‌‍‌‌‍‍‍‍‌‍‍‌‍‌‍‍‌‌‍‌‍‌‍‍‌‌‍‍‍‍‌‍‌‍‍‍‌‌‌‍‌‌‍‌‌‍‍‍‍‌‌‍‍‍‍‍‌‍‌‌‍‌‍‍‌‍‌‍‌‌‌‍‌‍‍‌‌‌‍‍‍‌‌‍‍‍‍‍‌‌‍‍‌‍‍‍‌‍‌‌‍‍‍‍‌‍‍‌‍‌‍‍‌‌‍‌‌‍‍‍‌‍‍‌‌‍‍‍‌‍‍‍‍‌‌‍‌‍‍‍‍‌‍‍‌‌‌‍‌‍‌‍‌‌‍‍‍‌‍‍‌‌‌‌‍‍‌‍‌‍‍‍‍‌‍‍‌‌‌‍‌‌‍‍‌‌‍‍‍‌‍‍‌‌‍‌‌‍‌‍‌‍‌‍‌‍‌‍‌‌‍‍‌‌‌‍‌‍‌‌‍‍‌‍‍‌‌‍‍‌‍‍‍‌‌‌‍‍‌‍‍‌‌‍‍‍‌‍‌‌‍‍‍‌‍‍‌‍‍‍‌‌‌‍‌‍‌‍‍‍‌‍‌‌‍‍‌‌‌‍‌‌‍‍‌‍‍‍‌‍‌‍‌‌‌‍‍‌‌‍‌‍‍‍‌‌‌‍‌‍‍‍‌‍‌‌‍‌‍‍‌‌‍‌‍‍‌‍‌‍‌‌‍‍‌‍‌‌‍‌‍‌‍‍‌‌‍‍‍‍‌‍‍‌‌‍‍‌‍‍‌‌‍‌‌‍‍‍‍‌‌‍‍‍‍put an end to this discussion!

Bystander • October 11, 2018 1:05 AM

@Wesley Parish

Given that I write satire and sometimes get it published, what would be the balance of probabilities?

It never occurred to me that this could be part of your occupations.

Having dealt with quite a few unit-impaired people at work (this includes engineers) I had my amount of Realsatire in this field and this is more on the nuisance side for me now.

I am still somehow surprised that it took these news to make people finally understand that if you are not in control of the entire production process (and chain) of your equipment anything can happen.

name.withheld.for.obvious.reasons • October 11, 2018 3:04 AM

@ Clive, Suspects, et al

Synthesis tools, those capable of producing RTL code and nets, have a series of toolchain issues. One simple modification includes modifying environment variable(s) (ACL’s are specifically applied to path, concatenated multi-value environment/set values). For example %PAWNED%;%PATH% can be useful in bypassing search and library sets that are common in toolchains.

Of course this example applies primarily to Windows PERIOD, Empire Edition.

There are some many opportunities in the long chain of most RTL sourced VHDL/Verilog synthesis tools–it is just a party waiting for the caterers.

Wesley Parish • October 11, 2018 3:35 AM

@Bystander

Not to worry. The ElReg article I was quoting said lithography, which would translate to seven nautical miles width per line per transistor. I leave it up to your imagination to picture what such a microchip would look like – given that Intel 8008 had 3 500 transistors and the 80486 had 1 180 235 …

So, I suppose this news will get certain in DC to close ranks to protect the US’s precious Olduvan industries from the upstart Chatelperronian industries of the rest of the world …

Supply has always been a major vulnerability. I would’ve thought that the US would’ve shown more class than to imagine it was immune.

Clive Robinson • October 11, 2018 5:44 AM

@ Wesley Parish,

I would’ve thought that the US would’ve shown more class than to imagine it was immune.

Ahhh but they are “exceptional” or so they thought…

Sadly having a “friendly” to the north, a “controlable” to the south and atleast a couple of thousand miles of ocean either side gave the citizens of the US a false sense of “issolation” for most of the 19th and 20th centuries.

In a sense it’s like religion, it was ingrained into US children at such a young and tender age they did not yet have the critical thinking skills to question it and it’s faux assumptions. Thus to the US mind in general “security” of the homeland and thus the citizens was a given, not the illusion it realy was…

The British proved the point it was an illusion one hundred years before WWI and Osama Bin Laden proved the point again less than a century after WWI[1]…

The schism 9/11 caused in the US psyche was felt the whole world over and unfortunatly they do not appear nearly two decades latter to realy deal with the “grieving process” and it’s five stages[2].

The US psyche appears stuck between anger and bargaining and incapable of moving forward. Arguably this is because the US MSM and the MIC find holding the US in this “continuous state of faux war” to be extreamly enrichening.

[1] World War One was considered the ultimate never to be repeated horror. It sucked not just the Western World but many other Nations World Wide into it, with horrendous losses of life and wealth. This was then followed by three flu pandemics that attacked not the young and the weak but the economically productive. The world effectively went into not just economic depression but psychological depression. The US went into “issolationism” and Europe sunk into political instability, violence and hate, which gave rise to all forms of extream politics. The Spanish Civil War lit a fuse in many tyrants heads as authoritarianism became the accepted face of politics and extream racism, eugenics, and other horrors became not just accepted but practiced widely throughout Europe. The predictable result was World War Two. It has not escaped some peoples notice that the authoritarianism and economic depression of the years between the two world wars has returned, where it will lead is not known but history gives us fairly good pointers.

[2] The 5 stages of “grieving” or “grief and loss” are,

1, Denial and isolation.
2, Anger.
3, Bargaining.
4, Depression.
5, Acceptance.

Etaoin Shrdlu • October 11, 2018 12:50 PM

“No one wants to even think about a US-only anything; prices would multiply many times over.”

Come on, Bruce, it’s not that big a price difference. The most complex parts are already mostly designed in the U.S. Fabbing mostly was until a decade or two ago; anyway, pick someplace other than a fanatically hostile and corrupt country to run the fabs in, like, say, friendly little Taiwan. Resistors aren’t going to affect anything if they’re made in China instead of the U.S. (Make the capacitors someplace where they won’t cheap out and cause them to explode every few weeks, though.) Pick-and-place machines cost the same in the U.S. as in China. Same with reflow ovens. There isn’t much labor in churning out circuit boards either; it’s mostly automated other than inspection nowadays.

The design cost is the big labor driver, and that’s a one-time cost that gets amortized over millions of units. How much does it cost to throw all the afflicted boards out and rebuild?

John Leslie • November 8, 2018 11:51 AM

Having designed a boat-load of x86 hardware (30+ CPU boards) I think this is nonsense. If the Chinese added a chip it would get spotted as soon as a U.S.A (/E.U. /etc.) repair technician got handed a faulty board and said… “hang on, this isn’t like the drawing”.

BTW:

Contract assembly – you design it and they make it. Changes by “them” trivially spotted.

OEM – they design and make it, you badge it. Changes harder to spot as they were there initially, depends on how well you checked the design.

ODM – you specify it and they design and make it. Lots of approval steps, no way anyone is slipping in anything extra. See “contract assembly” for subsequent changes.

Clive Robinson • November 8, 2018 9:57 PM

@ John Leslie,

I think this is nonsense. If the Chinese added a chip it would get spotted as soon as a U.S.A (/E.U. /etc.) repair technician got handed a faulty board…

Only that often does not happen, faulty boards hit the waste bin rather faster than the repair techs bench. They are realy not economic to repair.

To make it worse mass produced boards in FMCE are often designed to alow not just for several “product lines” but “inventory control” and “supplier independence”. The result as most people can see is the PCB has “missing components” that is there are device outlines and solder pads on the PCB that do not have componets in by design as it alows for different products and alternative function or suppliers.

With surface mount components there is often little or no PCB production overhead against doing this, as the boards have to fit a standard form factor and the increase in drilling is only for vias not component legs. Carefull design usually reduces the number of extra vias to an acceptable minimum.

If you look at the photographs of PCBs that are made for SuperMicro you will see there are indeed quite a number of unused outlines and component pads.

Thus the conversations have been about what can be done with such pads by fitting different chips, and what can be done by making components with chips hidden inside.

We know that other Governments IC entities such as those in the US and UK, have made such components and fitted them by interdiction in the last link of the supply chain.

It has always been just a “theory” based on a false assumption that you could not target and covertly poison the first links of the supply chain. Because certain “large orders” or “special orders” for the likes of Cloud Providers are known at all stages of the supply chain, they can be targeted at any point, not just the last links.

Further is the question of if you have an order for 1000 boards for a cloud provider, what percentage actually need to be poisoned?

With a little thought you can work out it could be well below 1% for certain components…

As with the US and UK IC entities, the Chinese IC would have access to the best brains and most likely the best technology available in their country. Thus if there was a way the poisoning could be done more covertly then the chances are they would not just have access to it but use it.

We know from “counterfeit components” that get into the supply chain and pass ordinary “Goods Inwards Test” (GIT) that criminals can make a good living out of fake chips and other components, and do so quite frequently thus the counterfeiting capability is out there. A company in the broadcast industry I have an association with tend to find counterfeit electronic componets a little over once a year, and that is only because they employ not just “full spec testing” but “burn in testing” and some “over spec testing” on GIT.

If nothing else the leaked TAO catalogue where “bugs” were put into USB and Video lead plugs a decade or so ago should have given plenty of people the “heads up” to start thinking about how to make supply chain poisoning as covert as possible.

But we also know that the Russian IC entities were putting highly covert implants into IBM electric typewritters that wound up in the US Embassy in Moscow half a century ago in ways that survived close visual inspection by technicians specificaly trained to spot the slightest changes…

So there is quite a bit of counter evidence to your argument.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.