Comments
Dave • August 14, 2020 9:49 AM
The most important piece of information is missing. How does the exploit get loaded into the machine in the first place?
And no, I do not believe a simple kernel upgrade will keep out the GRU!
AAC
Another Anonymous Cananck
Tatütata • August 14, 2020 10:14 AM
The TL;DR takeaway from the DOD Executive Summary would be :
To prevent a system from being susceptible to Drovorub’s hiding and persistence, system administrators should update to Linux Kernel 3.7 or later in order to take full advantage of kernel signing enforcement.
The current Kernel level is 5.8.
Version 3.7 came out back in 2012 or thereabouts, and according to Wikipedia, was the last one to support i386.
So this advice is in practice slightly superfluous.
Additionally, system owners are advised to configure systems to load only modules with a valid digital signature making it more difficult for an actor to introduce a malicious kernel module into the system.
<a href=”https://wiki.archlinux.org/index.php/Signed_kernel_modules >Arch Linux lets you do this. But if you’re a typical out-of-the-box distro user (e.g., Debian and descendants), this bit is harder to figure out. In any case, I think the typical default use case is to install the distro with UEFI disabled. This is alas the case here, at least on some of the older hardware.
How can you trust kernel developers? And who vetted Torvalds? And what about the long run? Nobody is eternal.
In the “Detection Methodologies” section on p. 36/45, I gather from the statement “Disadvantages: Subject to evasion via TLS or if the format of messages changes.” that communication with the C&C server is in plain HTTP. The little five-eyed barnacles on oceanic cables are in a position to feed on any interesting stuff…
From a quick scan of the various links, I didn’t see anything about practical infection scenarios, or the nature of the targets.
rrd • August 14, 2020 11:53 AM
@ Dave
The most important piece of information is missing. How does the exploit get loaded into the machine in the first place?
Yeah, I couldn’t find that info yesterday, either.
@ ALL
I was able to finally install and run volatility on a Debian box yesterday using this guide:
ht tps://markuta.com/live-memory-acquisition-on-linux-systems/
Note that I had a problem with the line :
cp Debian4908.zip ../../plugins/overlays/linux/
Maybe I screwed up, but that dir didn’t exist from where I was, so I created one which was temporarily fatal to the process.
The top-level “volatility” dir itself has a “volatility” dir underneath it, under which is the “plugins/overlays/linux/” dir. Once I deleted my erroneously created path and copied it to the correct place, I was able to get everything working. Markuta’s guide is truly excellent.
Actually interpreting the data that gets generated by volatility is another chore altogether. It looks like you want to close down as many apps as possible before you dump the memory, else the process space is filled with Firefox threads and whatnot.
This was my first time doing this on a Linux box and I have to say it’s pretty daunting, especially knowing that such malware has probably morphed itself away from its stock “dr_” names. (The use of volatility’s analysis routines was fairly quick and painless, however.)
I hadn’t done any kind of forensic system introspection since WinXp64, which I had already pared down to its minimal set of services and files. Not knowing what should and should not be there makes it difficult to grok volatility’s data dumps.
Anyway, it ran successfully and generated data for the different volatility commands the NSC guide suggested I ran against my memory capture, but I wouldn’t know if the data was actually meaningful. Regardless, I only looked for the standard “dr_” names mentioned in the original notice, finding none. Note that I also did not use yara or wireshark for any deeper analysis.
My conclusion (so far) is that I learned how to install and use LiME and volatility to some extent, but I have no confidence whatsoever that the minimal data I extracted demonstrated in any way that it’s not on my box.
Note also that I didn’t trudge very far down the “kernel modules” and kernel signing mode rabbit hole so I don’t really know what my kernel’s status regarding them really is. The first few search results were ambiguous and more detailed than I was up for at that point. I do know that LiME appears itself to be a kernel module so apparently my kernel allows me to sudo insmod it, which is how the memory capture file is created.
MarkH • August 14, 2020 12:09 PM
@echo:
Purportedly, Drovorub has been propagated by a government agency. In such case, commercial/market incentives in Russia’s software industry would presumably not have direct bearing on the case. GRU could hire the expertise, even if the great majority of developers were busy with more benign developments.
@ALL:
There are some topics on which seem to excite much opinion and emotion, but little appetite to learn the facts.
In my homeland, a dreadful example is our original population (native American, or First Nations as our northern neighbors say). For as long as U.S. has existed, most white Americans have cherished their dreams about native American people, whether saturated in bigoted hate or bathed in a romantic glow … but few wish to confront the rich and heart-rending realities of a complex set of ethnic minorities, who have been abused to an almost incomprehensible extent.
A sillier example is quantum mechanics, for which many profess enthusiasm (in the sense of cartoon-style “explanations” and dubious analogies to general experiences of life) … but few wish to actually study the difficult and baffling physics.
In Western countries Russia is one of those topics: much heat, little light. It’s a fascinating (though often painful) subject area, and I encourage any who are interested to learn more!
Near at hand, we’re lucky to have contributions from Anders, who (if I understand correctly) has mastery of the language and life-long exposure to Russia at Extremely Close Range. I commend attention to his perspectives.
Anders • August 14, 2020 3:15 PM
@Dave @echo
“The most important piece of information is missing. How does the exploit get loaded into the machine in the first place?”
You forget it’s nature and purpose. It is not a network worm.
This is rootkit. Those are planted, manually, after compromising
the system. Even the advisory itself emphasizes on the page 37:
“NOTE:The mitigations that follow are not meant to protect against the initial access vector.”
There’s a million ways to hack a Linux and getting the root.
Visit again last friday’s thread where i posted link to Bejtlich
posting – any prevention ultimately fails. You must consider that
your Linux systems are already “owned”.
ps. Drovorub translates directly to Lumberjack, Woodman.
One of the Aesop’s Fables is named The Fox and the Woodman,
that is in Ukrainian Дроворуб і лисиця.
echo • August 14, 2020 5:15 PM
@Anders @MarkH
<
blockquote>ps. Drovorub translates directly to Lumberjack, Woodman.
One of the Aesop’s Fables is named The Fox and the Woodman,
that is in Ukrainian Дроворуб і лисиця.
<
blockquote>
After Marks tirade against people only interested in the pictures and too lazy to do the maths I’m still waiting for him to notice my posting about the planned solar gravitational lens telescope going into stage three planning.
When a technical topic is “duh” I’m more interested in the meta stuff and everything else surrounding the topic. Beyond a certain point technology bores me. I’m like “So what”? It’s all known stuff and anyone with half a brain can comment so there’s not much interesting in there for me. I find the technology distracts from everything else. If more technology was an answer or people and ecosystems could develop technology which solved the problem we’d have it by now. We don’t because -> “other stuff”. It’s also really difficult to discuss if people can’t see beyond their own technnological nose and some of it is really quite specialist. Some people don’t consider you worth talking to if you have less than a Masters in the topic.
Ismar • August 14, 2020 5:38 PM
@Anders
Please refrain from remarks like “you must consider your Linux system is already owned “
I have an Ubuntu box running on my home network and happy to supply my external IP to you to try and own it if you can
Ismar • August 14, 2020 5:42 PM
Also, would it not be nice if GRU was to publish details of some of the hacking tools NSA and FBI use to break into the Russian internet infrastructure
echo • August 14, 2020 6:12 PM
@Ismar
Also, would it not be nice if GRU was to publish details of some of the hacking tools NSA and FBI use to break into the Russian internet infrastructure
Yes and this is one of my points. The issue is it creates an “institution”. Assuming relatively open and consistent criteria and openess of information it is both a trust building exercise “detente” style but also helps the Russian state be more accountable as well a engaging in dialogue both internationally and with its own citizens. The Russians can be a bit klunky and out of the loop and paranoid (sometimes with good reasons) but it can be viewed as a reboot or continuation of the Gorbachev era where the Russian state as an entity is able to modernise in a way which helps normalise relations with Europe and basically disables the right wing nutters in US politics.
I would expect greater openess on security to go hand in hand with trade and human rights. I know authoritarians and conservative minded people can view these things as threats but honestly they are not and the science says so.
There are some with job titles who will always view their empire as something to beat everyone else around the head with but, myself, I think a perspective where nation states are more a guarantor within a shared space (there is no “Planet B”) of wellbeing and safety and opportunity for everyone within their borders is more the way to go.
While international governance is stumbling and we both have the pandemic and climate change to manage there is no lack of wonder and opportunity in the world.
MarkH • August 14, 2020 6:40 PM
@echo:
I did indeed read up on the gravity lens telescope, which was news to me, and fairly head-spinning. I’m indebted to you for introducing it to me.
I chose not to discuss it here, because I’ve already had enough of my comments deleted, and I want to make good use of the generous latitude Mr Schneier extends to us, without going where I feel to be “too far.”
For the record, I am not an expert in any generally recognized subject, and never earned a diploma from any school, be it high or low.
@Anders:
It cheers me a little to see Ukrainian language. Дякую!
Живэ Белорусь!
echo • August 14, 2020 7:06 PM
@MarkH
A “thank you” would have been nice. To say I was put out is an understatement. Not only that but the principles of modular systems, multi-point image capture and the maths behind everything is pretty much on topic.
I really don’t want to start throwing rocks at people for sexism again.
Anders • August 15, 2020 1:24 AM
@Ismar
You should start the question like this –
How Do I Know That’s My Home Ubuntu Is Not Yet Hacked (Owned)?
Are you sure it’s not hacked already? How can you be sure?
Today there’s no unhackable system and if THEY want to get in,
and THEY have time, resources, knowledge, money etc, THEY
get in. Period.
So actually your first task is to assume that THEY are already IN.
Next step would be find the evidence. That’s why this report was
published. And of course the little slap on the wrist for the
Russians too. And i think that this little false flag hint towards
Ukrainian is also interesting.
Anders • August 15, 2020 5:52 AM
@Dave @echo @MarkH
There can be different reasons why initial vector is not disclosed.
- They don’t know it yet. All the evidents were erased after system compromise. This happens often, is plausible, but i doubt it in this case.
- There are several different vectors observed and they don’t want to reveal which of those is exposed.
- They use same method as Russians by themselves so they don’t want to reveal it yet.
I think this is powerplay and this advisory is a message to the Russians – we know that you operate and how but we don’t disclose all the info we know. This puts Russians in little bit difficult position – they don’t exactly know what their methods and tactics are now exposed and monitored and what’s not so they must be extra careful and also possibly dump some of their developed tools and start developing new ones. This is time consuming and not exactly cheap.
So in the nutshell government doesn’t care about our systems security at all. But in the end this isn’t important either. Let’s say that initial vector was some 0day. OK, it gets revealed, gets patched. What’s next? Russians have tens more 0days in their back pocket and next time they just take another one.
Pravdorub • August 15, 2020 5:58 AM
Drovorub is not a Russian word. Lumberjack in Russian is either лесоруб (lesorub) or дровосек (drovosek). The mix of these two (дроворуб) is Ukranian.
Bruce Schneier • August 15, 2020 9:48 AM
@Ismar:
“Also, would it not be nice if GRU was to publish details of some of the hacking tools NSA and FBI use to break into the Russian internet infrastructure”
That was Shadow Brokers.
myliit • August 15, 2020 10:08 AM
I assume that booting from CD or DVD into TENS, Tails, or other linux live DVD might at least avoid, or help to avoid, Persistence.
Anders • August 15, 2020 10:20 AM
@myliit
In case of Drovorub there are files on the disk.
Reboot from live cd, mount disk and then you see
those files.
They choose high uptime machines for the persistency,
so, funny, but sometimes power outage is your best friend 🙂
For the future make list of files with checksum and keep
them separately along with the script that allows you to
quickly see the files that weren’t there next time you boot.
Not by file time but by overall existence.
echo • August 15, 2020 12:48 PM
@Anders
I think this is powerplay and this advisory is a message to the Russians – we know that you operate and how but we don’t disclose all the info we know. This puts Russians in little bit difficult position – they don’t exactly know what their methods and tactics are now exposed and monitored and what’s not so they must be extra careful and also possibly dump some of their developed tools and start developing new ones. This is time consuming and not exactly cheap.
So in the nutshell government doesn’t care about our systems security at all. But in the end this isn’t important either. Let’s say that initial vector was some 0day. OK, it gets revealed, gets patched. What’s next? Russians have tens more 0days in their back pocket and next time they just take another one.
Of course its a powerplay. Of course the USuses its status and marketing and technical knowledge to position itself as “leader” and an “authority” and the “go-to world police”. We also know next the the British the US are the biggest bandits going. This is why in a multi-polar world where human rights and internal cooperation area thing the Russians might consider different strategies politically rather than allow themselves to be painted into a corner. The Russians have nothing to lose by cleaning their act up plus make the US look stupid. This kind of scenario is played out in the courts every day.
myliit • August 15, 2020 1:59 PM
@Anders
“… For the future make list of files with checksum and keep
them separately along with the script that allows you to
quickly see the files that weren’t there next time you boot.
Not by file time but by overall existence.”
Thanks. afaik this did something similar to that.
In the past, I was sometimes able to get the burned to DVD or CD sha256 value to match the sha256 posted for the ISO file. At least for Knoppix and TENS, iirc.
At other times, the burned DVD or CD value was different from a host download site(s). Then I referred back to my first, post burn, sha256 value for reference, not the iso sha256 value(s).
Regardless, I probably still didn’t trust things.
Clive Robinson • August 15, 2020 9:45 PM
@ Anders, myliit, ALL,
In case of Drovorub there are files on the disk.
That is a pre Flash ROM stratagy, and there are ways to get around them being found, or wiped or even the hard drive being replaced…
As was pointed out by @Nick P and myself on this blog considerably more than a decade ago, there is a whole lot of other mutable memory in a PC other than the Hard Drive an attacker can use, but users do not know about, hence our recomendations for using older hardware for Internet connection.
Also when the debate about “BadBIOS” came up @RobertT and myself again mentioned this and gave fairly specific details about the dangers. I also mentioned the gaping security hole that the “I/O Driver ROM” issue[1] was, that had been in the PC BIOS since day one, and spent a day of my weekend cobbling together a “Proof of Concept” for BadBIOS and a network card ROM and some “audio networking code” that went back to the 1980’s.
Two university students then did something similar with audio networking and two laptops in a corridor and all the world was suddenly trying and building near ultra sonic comms systems malware… And most forgot about the very important problems of Flash ROM and the gaping security hole in the BIOS and OS’s the “I/O Driver ROM” mechanisum was…
Then finaly someone discovered that Lenovo who had bought out IBM’s laptop business were using the “I/O Driver ROM” mechanisum to put persistant malware on their consumer level laptops…
And again most forgot about the Flash ROM and other mutable memory issue…
This is even though due to a pissing match between some senior idiot in the UK Government and the then Editor of the Guardian over the Ed Snowden document trove. Which led to “Tweedle Dee and Tweedle Dum” from GCHQ giving the game away on their “day trip to London” with a side order of shopping. Where in the basment of the Gurdian’s offices they directed journalists as to which chips to grind off of a mother board and the Guardian then published a full page center spred photo of the motherboard… I commented on this blog at the time that it presented a wonderful opportunity not just for security researchers but also for teaching students studying IT security.
Well again an opportunity was lost and many forgot about Flash ROM and semi mutable memory issues…
@ ALL,
Please take the opportunity to not just remember but educate others about the dangers of Flash ROM and other semi-mutable memory issues, which are,
- If an attacker can change any semi-mutable memory on a computer then they own it untill you not just find the changes but remove them.
And,
- Level three attackers know what you probably do not, which is where all the semi-mutable memory is, and more importantly how to change it.
It’s why Edward Snowden upset Glen Greenwald in Hong Kong by forcing him to spend money on a new laptop and also buy it in a very specific way.
There are real security lessons in this and they are all well within living memory because they are only just coming up to a decade ago. But as I keep pointing out there is something fundamentally wrong with the ICT industry in that it realy does chose to,
- Forget it’s history, thus is condemed to relive it over and over…
Why I’ve no real idea, I guess that a big part of it must be due to “The Profit Motive”. That is they think more money can be made through insecurity…
[1] The “I/O Driver ROM” issue actually predates the IBM PC, they just stole the idea from Apple who had the same issue on the Apple][, not that there was any malware effecting personal computers back then. The problem is a “chicken and egg” one which is part and parcle of how do you “future proof” your hardware so that new I/O can be added that’s not yet been invented / considered at design time? Well the easy way is to have any plug in I/O card have it’s driver software in a ROM chip on the I/O card. The next problem is how does the computer access that new code? Well it needs some mechanism to find it when the computer starts up. But this creates a problem, which is, what if the I/O device contains a “bootable OS image on it”? That is the original BOOT code in ROM on the motherboard gets copied into RAM when the computer is reset, if it loads in a number of I/O drivers from I/O cards, any new OS image loaded from an I/O card after that will overwrite them and loose those other I/O cards driver code as well as it’s own from RAM… Unless you have some mechanism to stop it doing so. Thus you define a standard that all new OS’s respect which is the “I/O Driver ROM” specification, put simply it reserves a patch of RAM that all OS’s respect into which I/O driver code is loaded along with a data structure by which the loaded drivers can be found/used. However you also need a way for the ROMs on the I/O cards to be found in the original boot process. This is usually by putting a recognisable flag like “0xA5,0x5A” into a known memory address offset and then check sums at the end of the code. The point to realise from the security asspect is that anything loaded into the protected RAM effectively gets treated as trusted by any following OS which is one big security hole through which you can drive a herd of war elephants. It is in short a good example of a hidden “insider attack” via a non human entity.
echo • August 16, 2020 2:12 PM
@Clive
Why I’ve no real idea, I guess that a big part of it must be due to “The Profit Motive”. That is they think more money can be made through insecurity…
I think there’s a few reasons including lack of proper education and social pressures but ultimately, yes, it boils down to money. There’s too many people doing things to a price and following whatever the wave of fashion is. As things are arranged money is a key determinor behind the scenes of all this meta and micro decisions which frame and shape perception and edit institutional memory. I’ve noticed this across a number of industries who pursue broadly the same or similar economic models.
The phrase irresponsible and “plausible crap” springs to mind. It explains why Rollys Royce got rid of their mechanical clocks and changed the airconditioning switches on their newer models among other things and why Megan uses a straight line to emphasise her lower lip and not a curve.
myliit • August 18, 2020 10:46 AM
@Clive Robinson, Anders, echo, ALL,
Clive wrote: “… And again most forgot about the Flash ROM and other mutable memory issue… …”
Yeah, but if we didn’t forget, you wouldn’t have to remind us again.
Clive again: “ …This is even though due to a pissing match between some senior idiot in the UK Government and the then Editor of the Guardian over the E …”
About here I got the eerie feeling that I was about to be visited by, or hear about, again: Tweedle Dee and Tweedle Dum. ( Clive’s words, not mine. Now where’s the closest exit …)
Clive Robinson • August 18, 2020 4:01 PM
@ myliit,
Now where’s the closest exit …
Ahh well atleast what I say about security will still be relevant for the next several years or so, or untill people real understand the issue and do something about it.
But if you are on your way out don’t forget to take your Trumpian fervor with you, after all it will be compleatly irrelevant in a few months.
Subscribe to comments on this entry
Leave a comment
Sidebar photo of Bruce Schneier by Joe MacInnis.
echo • August 14, 2020 9:48 AM
I personally tune out NSA/FBI announcments because I know there’s a meta game going on here. It’s about “authority” and marketing and the Russians, quote frankly, are an easy target with their Saint Petersburg crime ecosystem. I would be more impressed if there was a roadmap to help the Russians level up and find an economic scheme which helped develop a legitimate software industry as a way out. That’s difficult when there is no quid pro quo.
This kind fo problem manifests itself in the UK, specifically England, with “authorities” and their electioneering systems not getting that some issues are not about more “boys toys” and “cracking down” and “jail” but more a social policy and economic development and rehabilitation issue. Scotland gets this much better.
Personally, I think, US doctrine/dogma of one size fits all sanctions and using human rights as a bully stick has been counter-productive. You could probably draw a straight line on a graph between US policy and Russian crime and human rights repression.
Something to ponder.