Supply Chain

Episode 208: Getting Serious about Hardware Supply Chains with Goldman Sachs’ Michael Mattioli

In this week’s Security Ledger Podcast, sponsored by Trusted Computing Group, we’re talking about securing the hardware supply chain. We’re joined by Michael Mattioli, a Vice President at Goldman Sachs who heads up that organization’s hardware supply chain security program.


When we think about cyber threats to the hardware supply chain, we often think about defense contractors making missiles and fighter jets. But these days, hardware supply chain security affects a wide range of companies – not just technology giants like Intel or cloud computing providers like Amazon and Google, but banks and financial services companies, healthcare companies, consumer electronics firms and more. 

Despite media attention to the problem, the awareness of hardware supply chain risks is still low within companies. Tools and talent to address it are hard to find and expensive. What’s a company to do?

Hardware Supply Chain Is Everyone’s Problem

In this episode of the Podcast we welcome Michael Mattioli into the Security Ledger studio. Michael leads the Hardware Engineering team within Goldman Sachs. There, he is responsible for the design and engineering of the firm’s digital experiences and technologies. He is also responsible for the overall strategy and execution of hardware innovation both within the firm and within the broader technology industry.

Michael Mattioli, Goldman Sachs

Michael Mattioli

Michael is a Vice President and leads the hardware engineering team at Goldman Sachs.

“Grandma deserves to know that her iPhone is genuine in the way that a corporation deserves to know if their $30,000 server is genuine.”

Michael Mattioli, Goldman Sachs

Michael is the author of a paper Consumer Exposure to Counterfeit Hardware. In it, he notes that many of the methods used to ensure hardware supply chain integrity are fallible. Visual inspection of installed parts or open source research on sellers don’t scale and are unreliable. He’s trying to sound the alarm about the threat that hardware supply chain insecurity poses to our entire economy.

TCG Tackles Hardware Supply Chain

Michael’s part of a new working group at Trusted Computing Group and the GSA that is working to develop standards based technology and tools to enforce hardware integrity at scale. In this interview, Michael and I talk about the growing risk of hardware supply chain risk and the need for coordination throughout the industry to address hardware security threats.Goldman Sachs joined the TCG in February as it looks for partners in securing FinTech, where activities like mobile transactions are growing by leaps and bounds.   


(*) Disclosure: This podcast and blog post were sponsored by Trusted Computing Group. For more information on how Security Ledger works with its sponsors and sponsored content on Security Ledger, check out our About Security Ledger page on sponsorships and sponsor relations.

As always,  you can check our full conversation in our latest Security Ledger podcast at Blubrry. You can also listen to it on iTunes and check us out on SoundCloudStitcherRadio Public and more. Also: if you enjoy this podcast, consider signing up to receive it in your email. Just point your web browser to securityledger.com/subscribe to get notified whenever a new podcast is posted. 


Episode 208 Transcript

[START OF RECORDING]

PAUL: This week’s Security Ledger Podcast is sponsored by Trusted Computing Group.  Through open standards and specifications, Trusted Computing Group enables secure computing through its member-driven work groups.  Trusted Computing Groups enables the benefits of trust in computing devices from mobile to embedded systems as well as networks, storage, infrastructure, and Cloud security.  More than a billion devices include TCG technologies.  Check them out at trustedcomputinggroup.org.

[MUSIC]

PAUL: Hello, and welcome to The Security Ledger Podcast.  I’m Paul Roberts, Editor in Chief at The Security Ledger.  In this week’s episode of the podcast, number 208…

MICHAEL: To be clear, the people that are doing these types of things are not teenagers sitting in their basements.  It’s large, well-funded nation states.

PAUL: When we think about cyber-threats to the hardware supply chain, we often think about defense contractors making missiles and fighter jets, but these days, hardware supply chain security affects a wide range of companies and organizations; not just technology giants like Intel or Cloud computing providers like Amazon and Google, but banks and financial services companies, healthcare firms, consumer electronics makers, and more.  Despite media attention to the problem, the awareness of hardware supply chain security is still low within companies.  Beyond that, tools and talent to address it are hard to find and expensive.  What’s a company to do?  In this episode of the podcast, we welcome Michael Mattioli into The Security Ledger studio.

Michael leads the hardware engineering team at Goldman Sachs where he’s responsible for the design and engineering of that firm’s digital experiences and technologies.  He’s also responsible for the overall strategy and execution of hardware innovation at Goldman Sachs, and a recognized expert on it within the broader technology industry.  In this interview, Michael and I talk about the growing risk of hardware supply chain attacks and the need for coordination throughout the industry to address the threat that they pose.  To start off, I asked Michael to describe the work he does at Goldman Sachs and why a financial services company is employing a hardware security expert.

MICHAEL: Hey, I’m Michael Mattioli.  I’m the principle engineer within hardware engineering at Goldman Sachs.

PAUL: Michael, welcome to The Security Ledger Podcast.

MICHAEL: Thank you for having me.

PAUL: You’re on the hardware engineering team at Goldman Sachs, obviously a huge financial services firm.  Maybe we don’t think of them typically as a company that is employing hardware engineers and experts in that area, so I guess talk at a high level about some of the work you do at Goldman Sachs.

MICHAEL: A lot of the work that I do focuses on the consumer electronics space, so that’s something as simple as a keyboard or a mouse, something as complex as digital signage display, PC, a laptop, a tablet.  Then lately these days, I have my hand in a variety of different things outside the consumer electronics space, so Cloud data center, and most recently and prominently in security and trustworthy systems.

PAUL: Talk just a little bit about how you got into the field and your introduction both to cyber-security generally and particularly looking at hardware cyber-threats.

MICHAEL: Yeah, I’ve been at the firm about five and a half — almost six years now, so if you asked me in college if I thought I was gonna go for a bank as an engineer, I would have told you you were nuts, but it turns out you can have a fairly successful career in engineering at a bank so for those of you listening, if you’re wondering, I’m living proof of that.  So, a few years ago, I would say maybe eighteen, twenty-four months ago — actually, maybe not that long, I was talking to a few people in the industry about the supply chain, the looming supply chain problem ‘cause Bloomberg had published an article in — I think it was 2018 or so.  There were a variety of claims made in that article surrounding hardware implants and chips being snuck into boards from a few well-known major companies.  The scary part was that it is technically possible to do what they had alleged, and what surprised me was no one was talking about how big of a problem it actually is and no one was trying to solve it.  That kinda got me on the path of — in particular, how do you protect your hardware and how do you ensure trust in your hardware?

As the Cloud grows and people are transacting on mobile devices, when you think about it, the Cloud is just abstraction on top of abstraction on top of abstraction, so the underlying hardware which all those abstractions your software, your services are built upon, people stop and — remember, there is hardware that are supporting those things.  The hardware is the foundation for everything you do in the electronics space, so if the integrity of your hardware is a question, so is everything above it.  Then in the mobile space, a lot of — not just financial services firms, but everybody now is transacting with people on mobile devices.  When you think about it, they’re in the physical possession of somebody whom you’ve probably never met or seen, distributed all around the world in a [00:05:00] variety of different countries where there’s various legal jurisdictions.  You say to yourself, how do I know — again, to the hardware foundation piece, how do I know if the transaction I’m doing with that individual is valid?  Is it — can I really take it — word for it?

PAUL: When we talk about hardware security, I always imagined okay, some organizations do this, a Lockheed Martin or a big defense industrial base contractor; they’re probably concerned about this for weapon systems and so on.  Other nations might do it around highly-sensitive projects.  But then there’s sort of everybody else, so what has been the state of play or state of the art in terms of assessing counterfeit and cyber-risk within hardware supply chains?

MICHAEL: To your point, yes, everybody has thought about it maybe to some degree, so — but the scary part is, it doesn’t affect consumers, it doesn’t just affect large corporations; it also affects governments and people typically buy things like iPhone chargers and it turns out sometimes they’re counterfeit.  Then to your point about government military [inaudible], you could imagine that it’s a pretty big problem if somebody orders a component for a multi-million-dollar fighter jet and there’s — either it’s counterfeit, it’s been modified, it’s been tampered with.  It affects the whole gambit.  Now, in the consumer space, maybe the bad actor or the malicious actor isn’t targeting John Smith, right?  Because a lot of — I think it was very common for people to say themselves well, I’m not the target because I’m just John Smith, right?  What do people want with my information or what could people possibly want to do to harm me?  But the reality is they’re not maybe targeting John Smith specifically.

Maybe they’re targeting you as part of a broader answer; they’re targeting maybe an entire country, an entire city.  They’re not targeting just you.  It’s when they start targeting very specific companies, very specific countries.  Like, when you get to the large [inaudible] services firms or say, the large government corporations, to be clear, the people that are doing these types of things are not teenagers sitting in their basements.  It’s large, well-funded nation states that potentially could be doing these types of things.  So, it’s not uncommon or it’s not outside of the realm of possibility, I would say.  Like, let’s say someone was shipping a large number of servers to someone’s data center and the UPS label said To: Amazon or To: XYZ Financial Services Firm and someone paid the UPS guy to open the box and do something to each one.  It’s not exactly too hard to do that, so I think the interesting part is yeah, people have maybe talked about it or realized yeah, sure, it’s real, but has anyone done anything about it?

Then that’s — back in October of 2020, a few months ago, myself and a few people from Intel wrote a white paper trying to raise some awareness about this problem.  Then we actually ended up partnering with a bunch of other OEMs and ODMs and NHBs, and we said, we all agree that this is a problem.  It affects all of us.  The semiconductor supply chain is so complex that none of us can solve this one our own, not even the mighty Intel which they have their own fabs; they’re a pretty large company.  What we did was — that’s when I started to get much more involved with industry groups like the Global Semiconductor Alliance and the Trusted Computing Group.  So, right now I lead a working group with a few others specifically around supply chain security and provenance.  We’re actually in the process of forming a working group in the Trusted Computing Group to take a lot of the technologies that the TCG has developed over the last few years and apply them very specifically to supply chain.

PAUL: You’re listening to The Security Ledger Podcast.  This week’s podcast is sponsored by the Trusted Computing Group.  So, you wrote this paper Consumer Exposure to Counterfeit Hardware where you — it’s kind of like you kind of sum up some of the various manifestations of this from, again, the Supermicro compromised motherboard to the bogus, phony iPhone phone charger that might burst into flames or something.  It’s a wide range but I mean, one of the things you point out is a lot of the existing security methods rely on open-source research of where you buy it, and the seller — but also may be visual inspection for components, but that — those are all very fallible.  What are your thoughts on the way to do hardware supply chain security in a way that isn’t so fallible but that is also scalable?  ‘Cause you’re, again, talking about billions, potentially, of electronic devices that you may need to attest to.

MICHAEL: That right there is the million-dollar question, or dare I say the trillion-dollar…

PAUL: Trillion-dollar question, right.

MICHAEL: …question.  So, that’s actually the problem, is the only way to detect the — and the reason I wrote that paper was, like I said earlier, I was trying to make people aware that this affects consumers.  This affects [00:10:00] large corporations.  This affects governments.  It affects everybody.  The notion that it’s only a small segment of the population or a niche thing is just not true.  But I think that what I was trying to highlight there was the tools and methods we have to tell if something’s counterfeit [inaudible] is very primitive.  The first sign is the label misaligned or is the color yellow the color yellow it should be?  Very primitive things.  Then once you see something’s off, sure, then you dig deeper.  But then, digging deeper is x-raying it or taking it out and checking serial numbers.  So, all of those are very manual, primitive things and they’re very, very expensive to do at scale.  The truth is, is we don’t have any form of standards-based technology or tools to do these things.

There are academic methods and research being done to validate that semiconductors or entire platforms are genuine, but none of them have been adopted in practice and I think that’s largely because this just hasn’t gotten enough attention.  Not because — not just because it hasn’t gotten enough attention; no one player in the supply chain has end-to-end control.  So, let’s say XYZ fab, right, like, there’s GlobalFoundries, there’s Samsung, there’s TSMC.  Let’s say they did something to try to fix this.  Okay, great, but now everybody else in that chain has to play along, if you will, so that security model that you’ve had from the beginning carries throughout.  I think that’s really the bigger problem, is nobody uses end-to-end, and so that’s why we’re trying to partner with the TCG and the GSA, because out of those players, even though they’re, again, segmented, they all have some sort of affiliation with those two industry groups.  The goal here is how do we get everybody aligned?  Grandma deserves to know that her iPhone is genuine the same way some large corporation deserves to know that their $30,000 server is genuine.

So, I would say that there must be a QR code you could scan or an NFC, something you can scan when you get your device, and you can drill down and get very specific details like who shipped it, where it came from, date of manufacture, serial number, firmware; you can validate a whole bunch of things.  You can get as nerdy as you want.  However, I think there should be a very simple green or red — a circle or checkbox or an X or something like that, almost like — not too dissimilar to — I’m sure you’ve been paying attention to the whole vaccine distribution thing lately.  In the cold chain, as they distribute it, they put a little sticker on the box and if that sticker turns a certain color, that means at some point throughout its distribution, it either fell below a certain temperature or something happened to it where the vaccine is no longer valid.  So, something not too dissimilar from that at a very, very high level I think is something we need to get to.

PAUL: Interesting.  So, you also wrote — you’ve pointed to actually consumer gaming consoles like Xbox and PlayStation 1 and Nintendo as kind of exemplars of secure devices that have created a pretty effective ecosystem that is very resilient to compromise and attack.  Talk about that and what lessons maybe the rest of the electronics world can learn from game-makers who are obviously particularly concerned about piracy, like game piracy.  That’s kind of their motivating concern.

MICHAEL: The whole business model behind the video game industry is selling you games.  Like, when Microsoft or Sony sell you the Xbox or the PlayStation, they’re selling that console, they’re either breaking even or for the most part selling it at a loss because they know they’re gonna make the money back when they sell you games.  So, there are two motivations that I as a gamer have; one is piracy so that I can play games for free, right?  There’s a financial benefit there.  Then the other one is to cheat, so, is so that I can get an unfair advantage over other players.  So, how do Microsoft and Sony protect their business model?  Because the console is physically in their possession, right?

That console has to withstand a variety of different supply chain attacks because it also goes through the same supply chain as a PC or a laptop.  But it’s not like a corporation where that desktop or that laptop sits in a company office behind a firewall or whatever.  It’s sitting in someone’s home.  Someone can do all sorts of crazy things to it so that they can try to quote, unquote, “hack it”.  So, they have to design it to be remarkably resilient against physical attacks and they’ve been very, very successful.  I mean, when’s the last time you heard of any public vulnerabilities around PlayStation or the Xbox?

PAUL: Yeah.  It’s rare.

MICHAEL: The reason why people, I think, discount video game consoles is at first glance people go oh, well, it’s a video game console; well, of course, all it does is play games.  But very few people that — for example, the Xbox actually runs Windows.  So if you think about it, take that $60 video game, right, ‘cause that’s really what Microsoft is trying to protect; take that $60 video game and replace it with very sensitive company [00:15:00] information, and it’s the same idea.  It’s, everybody wants a device that’s high-performance, high-security, cost-effective, but they keep their data secure regardless of who has physical possession of it.

PAUL: Are we seeing any interest in that as a potential business model, do you think?

MICHAEL: It’s not a technical problem; it’s more of a — like, if you said to me is the Xbox going to start popping up in different offices around the globe, I would say it’s more of a business model problem than it is a technical problem.  But it is interesting; a lot of the hardware security that was in the Xbox is now trickling into PC, slowly but surely.  Actually, one of those technologies — Microsoft refers to it as Pluton which actually emulates a lot of the functionality that’s available in a trusted platform module or more commonly known as a TPM which is one of the core TCG technologies that I think would actually be very, very helpful in how do we secure various components in the supply chain.

PAUL: Yeah, I mean, it’s interesting compared to maybe fifteen years ago.  We now do have these very secure endpoints like iPhones and Xboxes that we use quite a bit to access data, but all that data has moved from obviously servers on our networks and into the Cloud.  So, that endpoint piece has gotten even more critical as well as the Cloud piece.  But on the Cloud side, there’s a lot of loose behavior going on.  There’s a lot of data dumps from Cloud — unprotected Cloud servers and storage containers that make headlines as well.

MICHAEL: Yeah.  I think a lot of the Cloud issues honestly are just a lack of understanding of, again, the hardware that’s underneath it.  Small anecdotal story; the other — a couple years ago we had a new group of interns and they were fresh out of college.  Some of them were still in college.  They sit down and we showed them that Outlook is where they open their e-mail and they all looked at us like, we were puzzled.  They’d never seen Outlook before, right?  Then it hit me that like, we finally crossed the threshold where people have gone their entire academic careers and have never touched things like Outlook.  They only know Gmail.

So, I think we’re — it’s the same thing with the Cloud; I think we’re finally crossing the threshold where developers are going through college and their formal training as software engineers and they are now at the point where they think that they could spin up infrastructure or do things in the Cloud, but they’ve never touched or experienced the actual hardware.  I think that’s largely due to the extractions that the Cloud providers give you, which it’s there to make your life easier.  However, there was a day when you — the only way to do those things was to get a data center and you had to know how to — I guarantee you if you ask someone today how do you rack a two-year server?  Very few people — unless they have physically done it — would know how to do that.  I think part of that is the Cloud guys are trying to make your life easier but I think everybody else needs to remember there’s hardware there.  It’s not magic.

PAUL: We’re losing survival skills in a way.  In five more years they’re just gonna ask you what e-mail is, I think, when you said those people, there.  One of the interesting tensions, I think, is — so, Bloomberg; you mentioned the Bloomberg Supermicro stories.  There have been two of them now; there was one a couple years ago and then they actually recently just kinda updated that story and doubled down on it.  As you mentioned, there’s a lot of questions around the voracity of their reporting and confirming what’s been reported.  Obviously the companies involved have really strenuously denied what Bloomberg is saying.  What do you think the practical impact of those stories has been in financial services and other industries?  What do you think is going to come from those revelations regardless of whether or not they’re actually — they actually are borne out?

MICHAEL: What they tried to do — well, they tried to raise awareness but the problem is, they did so in a way that people started to focus on the truth around their claims.  Like, they were very — to your point, adamant that what they claimed did happen and they tried, I think, to make this scandal-type thing when I think they really should have raised awareness.  I think what the reason why the initiative that we tried to start — like I mentioned earlier, I tried to start with Intel back October.  I think the reason why — thankful that’s gaining traction is because rather than focus on the OMG, this may have — may or may not have had happened, we’re focusing on the ‘this is a problem and everybody involved from the end user, consumer, government enterprise, all the way back to the fabs, the foundries, and the semiconductor designers have significant interest and stand to benefit financially from doing this.’  I think we’re — not to say that Bloomberg focused on the bad, but I feel like we’re taking a different perspective and I think that’s kind of helping us gain traction not just in the GSA but also in the TCG.

PAUL: So, what should people look for [00:20:00] from your group?

MICHAEL: Look out; I can’t say too much but I will say that we are working on proof of concepts for some of these technologies that we’re hoping to roll out sometime before the end of this year.  We are also, like I mentioned earlier, in the process of formalizing that working group into TCG which should include a lot of large tech companies which many of you have probably heard of.  I think that more broadly speaking, now that we are in a — I’ll call it a post-pandemic world with everyone working from home and devices and hardware no longer being, like I said earlier, behind the comfort and the protection of a corporate office or firewall, I think people are going to have to really rethink how they protect information because, again, hardware is subject to physical attack, supply chain attack, so can you really trust every single — I’m not saying employees are doing something malicious.  However, can you really trust every single person you give a laptop to to quote, unquote, “do the right thing”?  If you can, that’s great, but now you’re introducing the human element there, which can you really scale that?  You need to start asking yourself that.

I think you also need to start really looking at information protection, not just even in the realm of electronics, but it’s the common — say you’re at home, your kids are home, your wife is home; you’re sitting down, you have sensitive information up on your screen.  Now, yeah, sure, there may be a thousand different ways to prevent you from uploading that information to like, Dropbox or Reddit or something like that, electronic controls, but what’s stopping your wife or your kid from walking over to your monitor and taking a picture of your screen with her cell phone or his cell phone or whatever it may be, right?  Something as simple as that.  You can have all the complex technology in the world, but something as simple as taking out a phone and snapping a picture of your screen, and now what?  So, I think we need to rethink about how do we protect — or should I say more technically, how should we close those side-channels that we’ve been overlooking for so long?

PAUL: What’s on your radar for 2021 in terms of cyber-risks, concerns, priorities?

MICHAEL: I think a lot of it is — I mentioned earlier, how do we close those gaps in the side-channel space?  I think that there’s gonna be a lot more research done in the semiconductor space.  As you probably heard in the news lately, there’s a whole semiconductor shortage that affects everything from PCs to cars now.  I think that that’s very telling in a lot of ways.  There’s a whole geopolitical thing going on around there.  The current administration I think signed an executive order to go do an audit or an investigation on the current semiconductor supply chain.  So, will we see state-of-the-art semiconductor manufacturing return to the United States?  It’d be very interesting if it does.  But there’s a whole bunch of things going on there in the IP space and counterfeiting I think we need to pay attention to.

PAUL: [MUSIC] Michael Mattioli, thank you so much for coming on and speaking to us on The Security Ledger Podcast.  It’s been great talking to you.

MICHAEL: It’s been great talking to you, too.  Thank you for having me.

PAUL: This week’s edition of The Security Ledger Podcast was sponsored by the Trusted Computing Group.  Through open standards and specifications, Trusted Computing Group enables secure computing through its member-driven work groups.  Trusted Computing Groups enables the benefits of trust in computing devices from mobile to embedded systems as well as networks, storage, infrastructure, and Cloud security.  More than a billion devices include TCG technologies.  Check them out at trustedcomputinggroup.org.

[MUSIC]

[END OF RECORDING]

Transcribed by www.leahtranscribes.com