On Software Liabilities

Over on Lawfare, Jim Dempsey published a really interesting proposal for software liability: “Standard for Software Liability: Focus on the Product for Liability, Focus on the Process for Safe Harbor.”

Section 1 of this paper sets the stage by briefly describing the problem to be solved. Section 2 canvasses the different fields of law (warranty, negligence, products liability, and certification) that could provide a starting point for what would have to be legislative action establishing a system of software liability. The conclusion is that all of these fields would face the same question: How buggy is too buggy? Section 3 explains why existing software development frameworks do not provide a sufficiently definitive basis for legal liability. They focus on process, while a liability regime should begin with a focus on the product—­that is, on outcomes. Expanding on the idea of building codes for building code, Section 4 shows some examples of product-focused standards from other fields. Section 5 notes that already there have been definitive expressions of software defects that can be drawn together to form the minimum legal standard of security. It specifically calls out the list of common software weaknesses tracked by the MITRE Corporation under a government contract. Section 6 considers how to define flaws above the minimum floor and how to limit that liability with a safe harbor.

Full paper here.

Dempsey basically creates three buckets of software vulnerabilities: easy stuff that the vendor should have found and fixed, hard-to-find stuff that the vendor couldn’t be reasonably expected to find, and the stuff in the middle. He draws from other fields—consumer products, building codes, automobile design—to show that courts can deal with the stuff in the middle.

I have long been a fan of software liability as a policy mechanism for improving cybersecurity. And, yes, software is complicated, but we shouldn’t let the perfect be the enemy of the good.

In 2003, I wrote:

Clearly this isn’t all or nothing. There are many parties involved in a typical software attack. There’s the company who sold the software with the vulnerability in the first place. There’s the person who wrote the attack tool. There’s the attacker himself, who used the tool to break into a network. There’s the owner of the network, who was entrusted with defending that network. One hundred percent of the liability shouldn’t fall on the shoulders of the software vendor, just as one hundred percent shouldn’t fall on the attacker or the network owner. But today one hundred percent of the cost falls on the network owner, and that just has to stop.

Courts can adjudicate these complex liability issues, and have figured this thing out in other areas. Automobile accidents involve multiple drivers, multiple cars, road design, weather conditions, and so on. Accidental restaurant poisonings involve suppliers, cooks, refrigeration, sanitary conditions, and so on. We don’t let the fact that no restaurant can possibly fix all of the food-safety vulnerabilities lead us to the conclusion that restaurants shouldn’t be responsible for any food-safety vulnerabilities, yet I hear that line of reasoning regarding software vulnerabilities all of the time.

Tags: , , ,

Posted on February 8, 2024 at 7:00 AM • 31 Comments

Comments

Balderdash • February 8, 2024 7:38 AM

I do like the idea, but have one question.
Wouldn’t the liability include proper deployment by the network owner? Seems like that would also open a Pandora’s box of license agreements that require exclusivity for that product. Hypothetically, a networking vendor named Phrisco could require that the network owner have exclusively Phrisco products for networking and anything else would void any liability for the vendor. No?

BW • February 8, 2024 9:13 AM

@Balderdash – PCI has an example of that situation in PA-DSS. The software vendor provides an implementation guide that details how the card clearing software is to be installed. If the customer does not abide by that (including audit), liability for fraudulent charges shifts back to the customer (by varying degrees if I recall).

We already see tons of stupid in license agreements/EULA’s.. what’s a few more? 😉

echo • February 8, 2024 9:35 AM

US law is very different that UK/EU law in many ways. UK/EU consumer protection has broadly been better. UK government contracts have been a pain for ages in a few fields. Over complicated. Too much political meddling which constantly shifts the requirements. Nudge nudge wink wink procurement. Then there’s standards bodies (sometimes better known as cartels, depending) who set everyone up to fail before they start. Ugh.

A discussion on fitness for sale is certainly long overdue. I’m mostly in favour of it. There might be a problem with standards in the sense of the “reasonable person” test. None lawyers miss that “expert” status is a modifier which gives it a higher threshold. I’m also not persuaded about the “safe harbour” aspect of the essay. It’s one big loophole as in “All authorities were observed and all processes were followed. The patient died but the operation was a success”.

I cannot see how a network level operator can or should avoid liability. I mean, vicarious liability is a thing for a reason.

A culture of management denial or people suing everyone for everything just because isn’t an answer. Accepting liability or good faith gestures go a long way to preventing problems occurring and a culture of learning, and higher customer trust and better customer relations. It’s better for everyone at the end of the day.

Good pay and conditions also helps. People who are bullied in work or stressed won’t produce their best work. So join a union! (God, did I just say THAT?)

And the next time the mad cow in the corner says “don’t do that” don’t throw a purple faced wobbly.

Robert Thau • February 8, 2024 11:03 AM

The basic idea here seems to be that there are some flaws that are well enough understood that shipped software shouldn’t have them, and there should be liability for shipping software with those defects — including, say, at least some of (some version of) the OWASP top ten.

One problem is that understanding what a defect is doesn’t necessarily mean we’re able to detect all occurrences of it — let alone to automatically prevent it. Code injection vulnerabilities, for example, often require very specific setup conditions to create.

Here’s some pushback from Robert Graham on the somewhat simpler case of path traversal vulnerabilities (i.e., throwing in a “../../” to get the application to either read or write files that the remote user shouldn’t have access to). https://cybersect.substack.com/p/software-liability-for-armchair-quaterbacks

bar • February 8, 2024 12:48 PM

US Liability ‘law’ is hopelessly complex, unjust, counter-productive, and staggeringly expensive in all its aspects.

the Common-Law and common sense basics have been lost

civil and criminal ‘liability’ guilt originally required hard proof of deliberate malfeasance — or gross negligence causing demonstrable harm, by the Reasonable-Man judicial standard.

echo • February 8, 2024 2:12 PM

There was a big split in contract law as England went one way and continental Europe went another way. I’ve forgotten what the key split in contract alw was so call me useless. Interestingly, English law continued to evolve through the 18th Century with judges and lawyers pinching legal instruments from mainland Europe until law begin to firm up. In that respect Sovereign Individual Brexiter types are arguing against themselves. It’s scary how many things we take for granted today were relatively recent. Then being in the EU left a big mark on law very much for the better.

Clive Robinson • February 8, 2024 3:20 PM

@ bar, ALL,

Re : As expected.

“US Liability ‘law’ is hopelessly complex, unjust, counter-productive, and staggeringly expensive in all its aspects.”

There are two contributing reasons to this,

1, You pay your own costs.
2, Awards disproportionate to harm.

In theory you could suffer a very minor injury, with maybe a a dollar of loss on a band aid, yet get awarded a billion dollars in total from which the lawyers get more than 30%.

Thus two things arise,

1, “Beggar the victim” as a basic “Rights Stripping” tactic.

2, Making disproportionally excessive claims to gain excess benefit in awards.

Don’t bother trying to change the system though. Those who benefit from it massively run it at all levels so they are not exactly disinterested parties and will fight tooth and claw to stop it being changed to a fairer system that they will benefit a lot less from.

Such is “The American Dream” at work where what is outside the US as theft and fraud is to be applauded unless you are at the bottom of the ladder, in which case you are the carrion on which to be feasted upon as part of that Dream…

What appears forgotten is that justice serves two basic purposes,

1, To provide an individual with redress from harms.
2, To provide future deterrent to protect society.

That is fundamentally there is a balance at work of,

“Individual Rights v Social Responsibility”

In the US the tipping point is such that it’s about “stripping the rube” by a profession, so the “strippers rights” is all and not about justice for the individual or protection for society.

Q • February 8, 2024 3:53 PM

If the liability is limited to a full refund then Linux can continue in its current form. Then if Linux fails, and your entire business goes under because someone forgot a colon, then you can claim back your full refund of $0.00.

Also: If you are dumb enough to risk your entire livelihood on software that explicitly says it comes with no warranty, then that’s on you. Use something else if it so important to you. It’s not like there aren’t other choices.

JonKnowsNothing • February 8, 2024 4:35 PM

@Q, All

re: If the liability is limited to a full refund then Linux can continue in its current form. Then if Linux fails, and your entire business goes under because someone forgot a colon, then you can claim back your full refund of $0.00.

A well used 3d party editor has the same policy. The editor is free. Some user whined and ranted about a missing feature or other; the Dev offered to pay them back their purchase price: $0.00.

pup vas • February 8, 2024 5:59 PM

US FCC makes AI-generated robocalls illegal
https://www.bbc.com/news/world-us-canada-68240887

=The federal agency that regulates communication in the US has made robocalls that use AI-generated voices illegal.

The Federal Communications Commission (FCC) announced the move on Thursday, saying it will take effect immediately.

It gives the state power to prosecute any bad actors behind these calls, the FCC said.

It comes amid a rise in robocalls that have mimicked the voices of celebrities and political candidates.

“Bad actors are using AI-generated voices in unsolicited robocalls to extort vulnerable family members, imitate celebrities, and misinform voters,” said FCC chairwoman Jessica Rosenworcel in a statement on Thursday.

“We’re putting the fraudsters behind these robocalls on notice.”

The FCC said these calls have the potential to confuse consumers with misinformation by imitating public figures, and in some instances, close family members.

The agency added that, while state attorneys general can prosecute companies and individuals behind these calls for crimes like scams or fraud, this latest action

makes the use of AI-generated voices in these calls itself illegal.”

Gilbert • February 8, 2024 9:06 PM

You have people that know nothing of the complexity of developing software that want to enact laws to tell developers, who already work hard at avoiding bugs, how to do it ?

Let me explain how stupid this is.

We have cars. Those cars have issues because parts breakdown so the cat stops working.

The fix ?

By law, ask all car manufacturers to only use parts that never, ever break nor suffer wear when used.

There you go. From now on, all cars will never. Ever. Never break down. Because none of their parts will ever, ever break.

That is the level of stupidity of this law.

And before I forget : entropy says hi.

ResearcherZero • February 8, 2024 10:28 PM

@Gilbert

Hence the defining of what is reasonable or not. That is what the paper proposes.

As Bruce went on to articulate, “Clearly this isn’t all or nothing.”

And, “Courts can adjudicate these complex liability issues, and have figured this thing out in other areas.”

Laws simply require the equivalents of breaks, locks, seat belts, for example.

If someone can break into that car remotely from the other side of the world, it has been possessed. ♚ The client could place a steering lock in their car to deter theft, but at that point the security mechanisms of that vehicle have failed and are inadequate.

Your insurance will not cover leaving the vehicle unlocked with the keys in the ignition, then walking away. At that point you are abandoning your own responsibility.

That is where liability is defined. Did you (the consumer) take reasonable measures, or was the product shoddily designed, defective, or inadequate for the purposes it was marketed for?

Breaks in automobiles are not a bad idea. Even carts have a break, and there are harnesses for animals, even small animals. To prevent accidents and unnecessary death.

Publish a manual. Call it something like Network Administration. It should contain the required information regarding delegation of permissions and all the necessary Unix commands. Next the client reads the manual, or regularly takes the car to the service station. Some responsibility is also required from the consumer.

If the client has any difficulty understanding the manual, because the lied on their resume, they can pick up the phone and call support. Support will guide them to the correct page and explain the benefits of doing things the hard way, and hence why it avoids the pitfalls of the easy way. If they still want to make a particular file or folder read/write accessible by “everyone”, they must then accept the inevitable consequences of that decision. Is there is a ghost in the machine at this point? No, the client failed to take reasonable precautions. 👻

You could make defective software. Have no regulations at all. Your kids will install it.

It’s called malware. Malicious code designed to do nasty things to your network and life.

Doors are annoying. For easy access make a hole to bypass doors and windows…

Why bother with level 0 access via a password in plain text, when you can make it -2?

At this point I usually recommend disabling the bluetooth service, to prevent any audio interference or distortion, and blacklisting it so it will not run at startup.

(If the systems BIOS has the option, and you have access, you could disable BT there.)

echo • February 9, 2024 1:09 AM

One thing people almost always miss is the big picture. You have to look at the legal system set up in the US after independence, the psychology and culture, and even the fact the US is a country where mass gun ownership is a thing. It’s notable US law in practice (the law, what different people believe or claim the law to say, and what they do in the real world when people are or are not looking) is more individualistic than the UK which has more community underpinnings, and mainland Europe. That loosely forms the path of law evolution and worldviews and how USians perceive themselves.

In the UK at least statute tends to articulate it’s purpose at the top. Statute is meant to provide a framework not a solution for every single legal question. Then you have regulation and other soft law and so on and so forth. You have statutory bodies like Trading Standards and the Information Commissioners Office and the Legal Ombudsman and the Advertising Standards Authority which provide cost effective remedy. There are also thresholds to bring a case. All of the above is to bring corrective action and fines where appropriate. Where compensation is available it is to provide a remedy where you are not left off in a worst state than when you began. Punitive damages can be available but not always as, typically, the law is not a means by which to make a profit. It’s not perfect and this is not an exhaustive essay but simply a dumbed down rough view of some key differences.

Compared to the UK Usian’s tend to have more keys on their piano and are generally more structured in communication. the UK by comparison doesn’t get the art of management or writing documentation or communication in general very seriously. At the same time bad faith egos tend to view their opinions as fact, and their public office or influence as a platform to impose their worldview and therapy issues on the rest of society. In the UK the shredded remains of post-Thatcherite character (or “law of the land”) and community pressure is a corrective influence as mouthy individualism is relatively alien.

In the UK you cannot sign away your rights in law unlike the US. I was surprised to learn this was a relatively modern invention. It only began to appear or be fully implemented in the last half of the 20th Century. It does stop a lot of funny business.

God knows how it works in the US but in the UK it would typically be the Law Commission who look into what new law or legal reform is required or what holes need to be plugged and make recommendations.

When I read any US proposals like at the top or any related comments my brain does a big “uh huh”. If anyone wants to see standards in operation with an indemnification attached take a look at social media and see how that turned out. Take a look at US GOP politics and what’s happening there. The point here being standards/laws are not for a self selecting bunch of people to claim “certified professional” status while sailing through like greased piglets. They are there to protect us from you. And if these standards do not contain a remedy which is effectively policed (such as via professional standards bodies and independent complaints procedures and so on) then it’s all a bit smug and not going to encourage development of the state of the art or professional practice.

So, yeah. Nice initial first effort but I’d send it back and ask for a broader range of recommendations and some changes because, like I said, it should protect us from you.

ResearcherZero • February 9, 2024 1:50 AM

@echo

You can in fact sign away your rights in the UK. There are a number of ways to do it.

No system of course is perfect. The properties of any system can impose restrains that makes anything that seems arbitrary, not arbitrary at all. This is also a common misunderstanding of entropy or randomness.

As in a completely random password may not be random at all. The legal system has many constraints, exploits, and prejudicial mechanisms that reverse the onus onto the victim.

Obviously privilege and position being an apparent advantage. However, through the act of someone else, you could find yourself placed into a situation, where the very systems designed to protect, instead place you at a very considerable disadvantage.

Take the witness protection program. You must give up your rights in order to be protected.

ResearcherZero • February 9, 2024 2:20 AM

Legal incompetence would be another loophole. Family members can even declare another member to be mentally unfit and seek to have that individual’s rights placed under their purview, or have them placed in an institution “for their own well-being”. Works best with minors, as the legal rights of the child can be superseded by a “responsible” guardian.

There are other situations where the legal process can be sidestepped with the right advice. The following example is from Russia. Related situations occur in other jurisdictions.

…In 2021, the exclusive Russian cybercrime forum Mazafaka was hacked.

“The leaked user database shows one of the forum’s founders was an attorney who advised Russia’s top hackers on the legal risks of their work, and what to do if they got caught.”

— Said attorney appears to have served as a member of the GRU

(It can be helpful that such cases never see the light of day or cause any embarrassment.)

Back in the early 2000s the GRU was soliciting hackers with the skills necessary to hack US banks in order to procure funds to help finance Russia’s war in Chechnya.

‘https://krebsonsecurity.com/2024/02/from-cybercrime-saul-goodman-to-the-russian-gru/#more-66307

ResearcherZero • February 9, 2024 3:57 AM

Parents and developers both have roles and responsibilities. You could claim that your child is “crazy” to avoid the embarrassment of explaining what really happened to your child. Once you leave it unmaintained on GitHub, the fact still remains it may be forked.

This fate remains for a portion of projects that have been abducted, abused, then abandoned. Concern for abandoned projects is often short-lived. Mere token gestures.

Follow-up and follow-through is left to a few overburdened, burnt-out, and under resourced volunteers. Car stolen? The software may have been abandoned, unmaintained and unsupported.

Clive Robinson • February 9, 2024 5:08 AM

@ Gilbert, ALL,

“You have people that know nothing of the complexity of developing software that want to enact laws to tell developers, who already work hard at avoiding bugs, how to do it ?

Let me explain how stupid this is.”

Actually it already happens in all sorts of industries and professional walks of life. So I shall explain why it’s not stupid and has not been ever since Victorian era boiler laws.

But first of consider, how do you think “anything new” where by definition “the complexity of developing” it, is unknown gets made. Think on it carefully.

You use the example of cars, go look up the history of “Lemon Laws” and similar where the US car industry was killing way more people than serial killers. The cars were a danger not just to those outside them but inside them and like the destruction, deaths, injuries, maiming, and bad press of Victoria boiler explosions, politicians were forced to act by the negligence “of the industry”, from the most senior of managment down.

The only reason we are not seeing similar carnage with most software from the “Software Industry” is that untill recently two things applied,

1, The software systems did not have “physical agency” so a “blue screen of death” or other built in flaw/fault did in the main not destroy, kill, maim, disfigure and in other ways attract bad press.

2, The software systems that did have physical agency where real harms of death destruction, etc could very easily happen, were designed and built under very heavy regulation, just as the physical systems that proceeded them.

Thus software can be made reliable it can be made safe it’s complexities can be removed or controlled by systems imposed by legislation and regulation.

Just as any building built of bricks and stone, concreate and glass parts can. As can square miles of highly dangerous petro-chem plant, and nuclear reactors, and other potentially dangerous industrial systems.

Likewise as can flight control systems in aircraft, and car engines with castings, bolts and many other component parts.

None of these need “software” to be built, or to function or behave safely and they are all regulated by legislation (as you should know if you had worked in any of them).

Because those who pass laws and regulations created an environment where things had to be done safely.

Thus “art” and “craft” was subsumed to “science” and physical tangible “engineering” of physical tangible objects was the result (look up the history of the “British Standard Whitworth”(BSW) thread). This happened because from science came understanding of physical properties from which formulas were produced, from these “safe physical limits” could be calculated, as could “expected physical life time”. Thus not only engineering but preventative maintenance.

It’s what alowed the aviation industry to florish and many others. Sea going vessels for instance have marks at their “water line” to indicate not only if they are not over loaded, but that the load is in part distributed in a safe way (look up the “Plimsoll Line” and the politician it’s named after Samuel Plimsoll who was born two hundred years ago tomorrow on the 10th Feb 1824).

So we have had politicians and legislators make life physically safer for well over a century by what is now a very well understood process of standards that by science give usable rules and formular for design and maintenance engineers and users of physical objects to work safely within.

Are you realy arguing that what works for physical objects can not work for software objects?

Because if you are you are arguing against atleast half a century of proof otherwise…

So a bold or foolish place to stand.

You say,

“We have cars. Those cars have issues because parts breakdown so the cat stops working.”

The same logic applies to aircrft, yet they generally don’t fall out of the sky due to it. Because science and engineering gives preventative maintenance rules for their operators to follow such that parts are replaced long before the end of their “working life”. It might be hard for you to see but similar rules can work for the non tangible, non physical information objects made by software. To argue against it is well against “the scientific method” and “scientific principles” and all that comes from it ( https://www.ncbi.nlm.nih.gov/books/NBK234526/ )

Are you arguing that?

Because if you are I can for suitable remuneration give you the information and lessons as to why you are wrong and the remedial steps you have to take. But so can many others look up a four year degree in a physical science or physical engineering subject.

In my experience having employed quite a few people to develop “safety critical systems” it’s best to avoid those who have not had such education and physical science and engineering backgrounds.

You might want to scream and rail against this as others have done, but it would be daft to do so because a century and a half of solid science and physical engineering in a politically regulated system proves otherwise, and will continue to do so more and more frequently untill long after all who read this are but dust in the wind.

So I can make a prediction for you based on history, take from it what you will.

Because it’s “cheap and moody” software will currently continue to be developed in what is at best “an artisanal way” by people that realy do not understand science, engineering and the need for legislation and regulation. However information based systems are being increasingly given “physical agency”. And as we are now seeing with self driving cars they will cause carnage, destruction, death, maiming and injury if not properly legislated and regulated via a “well found regulation and standards based process”. It’s comming and you realy can not stop it, because the public want “safe” not “carnage” and artisanal development can not give this.

Also the public do not want to be defrauded by software developers and their managment.

Do I need discuss cars designed to cheat emmissions tests?

How about failures in “self-regulation” in air craft that flew into the ground due very much to unsafe software?

The Medical Profession used to follow a creed of the Hippocratic Oath, most are aware of via it’s basic tenant of,

“First do no harm”

Some physical system engineers have a similar creed look up “The Engineer’s Ring” oath,

https://en.m.wikipedia.org/wiki/Engineer%27s_Ring

Other science and engineering professions such as in Earth Sciences are likewise following suit[1].

It’s something that those involved with the “software industry” should be mindful of. Because now in some parts of the world it is illegal to call yourself an “engineer” without being accredited, registered and bonded, being even over thirty five and Ceng is insufficient.

The winds of change are comming, and you can run with them or stand and be broken by them.

It realy is that simple, like it or not.

[1] However such Kipling oaths have some would say not worked in all walks of life where honesty nd integrity would be expected, but is often found wanting. After the 2008 finacial crash Harvard Business School was one that set up a ring system for MBA’s apparently most now avoid it,

https://www.timeshighereducation.com/news/keeping-engineers-honest-canadas-iron-ring-tradition

echo • February 9, 2024 7:09 AM

https://www.theguardian.com/technology/2024/feb/08/gmb-accuses-amazon-union-busting-tactics-midlands-warehouses

Amazon accused of using ‘union-busting’ tactics at Midlands warehouses

Claim comes as GMB union prepares for three days of strike action at tech company’s Coventry warehouse

The Guardian has seen photographs of information boards and company newsletters the GMB said were displayed inside BHX4 and other Amazon warehouses in the region. These show messages including: “The union wants you to pay £14.37 every month for them to speak for you. We believe having a voice shouldn’t cost you anything,” and: “You don’t have to join a union to have your voice heard. We’ve got you.”

Another says: “Before you vote or join a union, we encourage you to seek out the facts for yourself. The best relationships are the direct ones.”

[…]

The company denied that [union busting] allegation, saying any new staff were brought in as a result of normal business requirements. With a concerted union recruitment drive continuing at the site, the GMB said it anticipated making a new application to the CAC this spring.

An Amazon spokesperson said: “We respect our employees’ rights to join, or not to join, a union.”

Meanwhile Clive is back on his usual context free slagging off software developers hobby horse like engineers are carved out of distilled golden beams emanating from the saints.

The US is an entirely different culture with entirely different employment and financial laws and business environment. There’s similarities with the UK but it’s different in a fair few ways like Amazon is finding out. Not every business person likes a legal wild west because they can dislike managers who cut corners too. The problem is people who cut corners can stay in business while the company with the superior product and practices and work conditions can go bust. Also there are many software developers who can and do complain when management or any other third party tries to poke their nose in. I know some do speak up (I was one more than once in the past) and I know others have done too, or simply refused to take on a contract or walk away. A huge amount of progress is defendant on laws being passed and none corrupt regulators and, yes, customer expectations too and sometimes litigation or even media support.

In the UK at least “engineer” like “doctor” is not a protected title. Its use is only unlawful if it is used to pass yourself off as something you are not such as being accredited to work in a particular field or claim qualifications you don’t possess.

Unions can and do help develop standards and best practice in collaboration with employers who are willing. They also stand up for staff facing workplace bullying and seek to obtain decent pay and conditions so staff aren’t dizzy with hunger and tiredness while doing their job which I think we can all agree helps improve lives but also avoid disasters or even fraud.

I wish Clive would knock it off with his general smears. They’re obnoxious and not helpful. And he can also stop using maths as a hammer to beat people into the ground just because he personally has advanced maths skills and a head of rote learned formula. It’s not some magic shield against goofing. That’s why we have R&D and investigations and where appropriate lots of input modifiers from other fields Clive airily dismisses as “soft” and end users. Why? Because maths and rote learned formulas don’t tell you everything and don’t protect against stupid. Also if they’re not relevant to the job they’re not relevant to the job. And when you do need them (which can happen) you either read up on peer viewed material or swot some books or hire an expert to give you the necessary to perform that function or set of tasks. Engineers are not software developers but can be software developers. Likewise, doctors are not scientists (trust me on that one!) but often use the end result of science. Well, congrats spending ten years qualifying so you can write a “Hello world” and, oh yes, start learning all over again for another five years so you can develop software then another five years for the specific application field you may be specialising in every single time you switch fields. Yay, that recipe software is going to be really cheap isn’t it? Oh noes. You don’t know anything about cooking so need to get qualified as a three star Michelin chef!!! If muggins was doing a software system for engineers or doctors or chefs then muggins would go and get one on board for the purpose they are needed for and no more. Likewise if it was a consumer orientated system. Likewise if muggins was developing anything for high reassurance applications then muggins would get the appropriate training and work for an appropriate employer working in the field. Anyway, muggins has other things on in life so that’s not going to happen but for others it will. The point being you specialise when you have to.

There was a man I had a conversation with, as you do, who did military work but also civilian work. I’m not mentioning names but the manager of his department asked him to allow something through. He dug his heels in said “Sure, if you put the request in writing.” The manager didn’t push it again. It’s a good job he did say no otherwise the wings of a famous name aircraft manufacturer would be falling off. The thing is he was protected. He could say that. Not only that but he didn’t want to have to explain himself if several hundred people died. And no he didn’t know the finer point of wing design and yes in his other job he’d be blowing the wings off but we are where we are. The thing is having maths skills and a head full of rote learned formulas isn’t magic. Engineers can be as dumb as anyone else. If you don’t know it you don’t know it. But… if you’re competent in your sphere and you can pushback then yeah sure you can tell a manager to go swivel and/or call a union in if they get narky. Just like a decent software developer…

And lastly people need nurturing and encouraging and helping. If you got dogmatic and shouty and slapped a child around who you think they’d grow up right? No? Well then… Flipping knock it off with slagging people off. Half of them are kids just trying to live their dreams and keep a roof over their head. They have no clout and they’re not writing the software for anyone’s pacemaker and if they were the product, like aircraft, would be covered by regulation. It’s not like they’re sticking any random in those jobs.

echo • February 9, 2024 7:49 AM

https://twitter.com/MichaelRosenYes/status/1755601530198393226
An 18 year old started talking to me in a cafe today. He explained that he was autistic. He talked about the things that he thought he could and couldn’t do. I noticed that he spoke Spanish to his mother. He’s bilingual. I said that that was a fantastic skill to have.

https://twitter.com/JDIrwinbooks/status/1755724849635332439
My 2 lads are ASD, both lovely souls. Youngest, 18 – who loved/still loves Chocolate Cake – faces challenges, but is a talented artist. I drum home how lucky he is to have innate skill.
[IMAGE of a watercolour picture of a robin.]

And before Clive repeats slagging off of autistic people as the root of all evil software here’s some quotes I just chanced across on social media.

I really think someone who benfitted from free education and was able to buy a house for £3000 in 1970 and who benefited from all the social, political, and legal and regulatory advances made before he was born would at the end of his career as a self-employed man who answers to nobody would knock it off with punching down on people just starting out in their life with it all ahead of them. What if Clive was just starting out now or just one voice in the crowd? Exactly. Suddenly it’s not so easy.

I know of one autistic young woman whose mum talked her into having an intercom fitted to act as a break before opening the door. She has a list of approved people and prepared questions and situations to aid her decision as to whether she should open the door or not. The reason is she cannot judge unfamiliar situations/problems and is trusting. She may be a bad case. I know of others by browsing online discussions by autistic people of varying degrees of impairment dealing with frustrations and difficulties either with the system or in work. That is of course assuming they can get or keep a job due to lack of “reasonable adjustments” and discrimination.

Demonising a minority who already have it hard by waving hands at some abstract unproven threat is really not on. And you have done your formal Equality Impact Assessment and Risk Analysis? No? You may as well say black people shouldn’t be social workers and if that sounds bad then it is because it is bad. And on what mathematical and rote learned formula grounds was it acceptable to say it? What? You don’t know because it’s outside the field of “certified engineer”. Exactly.

Honestly, I’m so angry about this…

Bob • February 9, 2024 9:58 AM

@echo

US law is very different that UK/EU law in many ways. UK/EU consumer protection has broadly been better.

The US is a right-wing country, and US law is wholly designed to protect the capital class. Consumer protections, worker protections, patient protections, basic social services, basic infrastructure: these are things that the US has eschewed in favor of making the rich richer.

That said, software liability has the potential to turn software development into a risky enough pursuit that only those with massive resources are able to pursue it. Once that clicks and the tech behemoths’ lobbyists get involved, we may actually see some traction with US-style, monopoly-friendly legislation.

Martyn Thomas • February 9, 2024 11:02 AM

One of the recommendations in the 1968 NATO report on software engineering was that programming languages should facilitate formal analysis of programs and that any that don’t should go back to the repair shop. 55 years later, programmers who often lack engineering qualifications call themselves engineers and write software in C and similar.

Cyber space is full of the resulting detritus and there are no plans to clean it up.

This will not end well.

Bob • February 9, 2024 11:15 AM

@Martyn Thomas

One of the recommendations in the 1968 NATO report on software engineering was that programming languages should facilitate formal analysis of programs and that any that don’t should go back to the repair shop.

Incompatible with our hyper-capitalism. Next suggestion?

Bob • February 9, 2024 11:43 AM

I’m going to go out on a limb and say any software developers in favor of this have never been in the unfortunate position of bottoming for the US legal system, and can’t imagine themselves ever having to do so.

amparo • February 9, 2024 12:46 PM

@ Robert Thau,

I disagree with your point on code injection. In my view, such vulnerabilities are caused by extreme sloppiness—to the extent that the programmer and program have “forgotten” the type of data being handled. Basically every programming language from the last 50 years has had a type system to keep track of exactly that sort of thing, and (possibly) complain if types are inadvertently mixed. If someone decides to abandon that and just call everything a “string”, that’s on them.

HTML is different from UserProvidedInputThatsSupposedToBeHTML, and neither is SQL. Graham’s post seems to basically agree while somehow coming to a different conclusion.

But we don’t know how to “avoid” it. Knowing how to fix the bug once it’s discovered is not the same thing as knowing how to avoid the bug from happening in the first place. Knowing how to fix a car engine doesn’t stop the engine from breaking. […] At the top, it’s not clear that input will be added to a path. A web form might accept a username field, but it’s not clear how the rest of the system will handle this field. It’s rare that it’ll be combined with a path, causing the path traversal problem.

If you’re taking input and have no idea what will be done with it, in my view that’s negligence. It’s like saying the person who specifies the engine for a car needn’t have any idea how people fill their tanks. Hey, the fine print of the manual said to use premium fuel, and to have it professionally pumped, so it’s not my fault if the tank explodes when some unqualified person pumps regular. “Actual” engineering, as opposed to software “engineering”, needs to consider reality.

The idea that “dirty” input needs to “sanitized” in nonsense, and is in fact the root cause of the problem (and other problems such as “mojibake”). Everywhere data is used, programmers need to be aware of its type, and need to parse/decode/validate and encode as necessary. Such points have been made on this blog before. If parsing fails, one should not try to sanitize the data, but should reject it entirely.

(By the way, a system that allows path traversal probably has more than security problems. For example, POSIX filesystem interfaces use raw octet strings, whereas other operating systems may require Unicode in some specific valid encoding. HTTP clients get to choose how to encode posted form data. If such data is passed directly to a filesystem API, it may break when the user switches browsers or languages, or when the server swiches operating systems. Or it might just annoy the administrator; I’ve had fun creating files named “NUL” on Windows network shares, and I guess names with invalid UTF-8 sequences might cause similar trouble on some Linux systems.)

Bob • February 9, 2024 1:47 PM

@amparo

My car doesn’t have anything to keep me from pouring nitroglycerin into the tank. Car analogies are notoriously bad for discussing technical problems.

Clive Robinson • February 9, 2024 2:14 PM

@ Bruce,

More falsehoods from @echo.

“And before Clive repeats slagging off of autistic people”

As you are aware the only person on the autistic spectrum I’ve talked about in anything even remotely close to “slagging off” is when I “gently take a rise” out of myself. And that is only so others can have a smile and a little laugh at my expense.

I could go on about the other falsehoods by the metric ton load @echo makes, but I think the point is made @echo just attacks, and attacks me only, and it has absolutely zero to do with anything @echo claims I might or might not have said.

For some reason @echo has taken exception to the fact that as I’ve correctly pointed out the software industry in general does not follow the scientific method, or scientific procedure. Nor does the software industry have actual metrics that can be used within the scientific method. Thus no actual science…

As @emily’s post pointed out in

https://www.schneier.com/blog/archives/2024/02/friday-squid-blogging-illex-squid-in-argentina-waters.html/#comment-432144

With the Lord Kelvin quote,

“When you can measure what you are speaking about, and express it in numbers, you know something about it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarely, in your thoughts advanced to the stage of science.”

In general for most of the software industry it’s,

“Knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts advanced to the stage of science.”

So in the general software case it can not move forward from scarcely science into something useful for engineering be it design or maintenance.

Why @echo things this is denigrating individuals who are autistic, LBQT or just disabled in some other fashion beats me. Unless of course @echo knows it’s completely untrue –which it is– but is running an ad hominy attack by just sling any old nonsense in the hope @echo can get it to stick in some random persons head.

However even those that have made that silly mistake before are now staying away… Thus the question arises of,

“Have they wised up to @echo, or given up on @echo as a bad lot in general?”

I suspect the latter as evidenced by @echo’s latest attempt to “force words into my mouth” in other people’s opinion is truly truly appalling,

“You may as well say black people shouldn’t be social workers and if that sounds bad then it is because it is bad.”

It’s not just racist it’s highly prejudicial and something I would not even expect from the worst of the far right electioneers.

Thus we start to see the “true colours” @echo flies.

But… @echo then has a second go with trying to falsely push such truly truly appalling words in my mouth by saying,

“And on what mathematical and rote learned formula grounds was it acceptable to say it?”

I never have, but @echo has as can be seen above as clear as day. Whilst @echo might try to deny it many here will spot the stylistic finger print, and in all probability IP addresses will align to some reasonable extent. But also people will question if the girl&mum story is just made up as an excuse to push a false accusation, if people think that it would not be the first time. Much of what @echo has claimed appears “made up” Walter Mitty style. And unlike ordinary day dreaming it is actually quite harmful when it becomes “maladaptive”,

https://bigthink.com/neuropsych/maladaptive-daydreaming/

Or worse the most recent of the “dark pentad” traits of : Machiavellianism, narcissism, psychopathy, sadism, and spitefulness,

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8435378/

When combined with increasingly overt primary narcissism, which appears to be the potential behavioural driver for @echo.

Something that could have very serious results for any that can b close enough to @echo to be within reach

I understand that you see your blog as your “front room” and thus having some one behave as @echo has repeatedly done must be concerning because it also reflects on you. However as seen with those into “cancel culture”, they will gleefully jump on anything when it suits them to and you just do not know when.

How you decide to deal with it is up to you, however as long as @echo’s accusative words are there to be seen in search results for what @echo truly is, they can be pointed out to people as well as my words of defence against such behaviour.

amparo • February 9, 2024 3:09 PM

My car doesn’t have anything to keep me from pouring nitroglycerin into the tank. Car analogies are notoriously bad for discussing technical problems.

They are indeed, but I was following Graham’s lead. The point was that, in “real” engineering, one needs to consider what might go wrong. We could just as well talk about elevators or bridges or anything else. Those “maximum load” labels are ubiquitous, but it’d still be professional negligence to fail in a fatal way from things that are reasonably expected to happen. We all know that an elevator is occasionally crammed unreasonably full, that a bridge occasionally has bumper-to-bumper traffic; nevertheless, serious failures are not acceptable. Now, if you drive an M1 Abrams tank onto an elevator, or put them bumper-to-bumper on a civilian bridge, or there’s an earthquake that’s much larger than anything ever predicted for the region, that’s a different story. Nobody’s putting nitroglycerin into tanks, but diesel tanks do tend to be labeled “diesel only” for good reason; look at your nearest generator.

“I’m taking data, processing data, sending data, without having any clue as to what it is and where it came from”… “and that’s just to be expected when programming”—what the hell kind of excuse is that? Even for kludgey one-off shell scripts in which I don’t feel like handling all data properly, I’ll at least have some idea where the data comes from and use “case” statements to guard against unhandled characters. But when it comes to commercial products, I don’t think it’s unreasonable to expect programmers to use the standard tools—that is, type systems—that have existed basically from the beginning. We know damn well that bots are going to quickly show up to feed “weird” data to anything that connects to a network.

Bob • February 9, 2024 4:54 PM

@amparo

Vulnerabilities have existed alongside type systems since the beginning. You can tell because they were ubiquitous even back when pretty much everything was typed.

Our economy is built to prioritize quarterly profits over long-term sustainability. As long as that’s the case, software liability is like trying to fix a crumbling foundation by painting the walls with lead paint. You haven’t done anything to address the actual instability, and some unlucky kid is going to have their life ruined by it for no good reason at all.

lurker • February 9, 2024 6:40 PM

@Clive Robinson, echo

@echo finished that post with “Honestly, I’m so angry about this…” which reminded me of a compulsory Anger Management course I had to attend after publicly exposing the incompetence of a coworker.

Of course on the internet no one knows you’re a dog, but everyone can see when you’re angry (or stupid).

Clive Robinson • February 10, 2024 12:53 AM

@ lurker,

“Of course on the internet no one knows you’re a dog, but everyone can see when you’re angry (or stupid).”

Or even if you are the same person or not.

What people say has a certain “style” at various syntactic or symantic levels you can even resonably guess at “the mother tounge” they have[1] from just a few sentances. Because it’s apparently fundemental to the way you think[2] from very early in life as is much else.

It’s on “oh so many levels” it’s previously been argued in the past it’s hard if not impossible to fake. And why just hearing a few bars of music we can identify the composer or artist[3].

But we are now finding that some LLM AI can fake some of those levels if they have sufficient of your brain created output…

Is this the real “Existential threat of LLM’s”?

I rather think it might be because it can have so many negative consequences on all areas of life.

Most importantly is the “why bother we can fake it” asspect arising. It is frighteningly “insular” and history tells us it will become a primary reason for segregation thus elitism and all the deliberate political explotation that all to often leads to not just major civil unrest but Genocide…

There is a lot more to “keep it real” than most realise. The Internet alows you to live in a bubble of your choice and not at all face the greater reality of society around you. Need I list the negative consequences of this?

[1] There is a story of a certain “counter-spy” interrogator who had a theory about women revealling their true identity during child birth as it’s such a traumatic process. As far as I’m aware it’s not been tested, and for various reasons I have my doubts, in part because people can shockingly for some peoples theories about tourture be resistant to physical pain and similar. Thus if your argument is “if they don’t reveal themselves during the pain of tourture…they will reveal themselves during the pain of child birth” you are arguing degree of pain… As we know from just looking around women survive child birth fairly regularly. Thus to be valid the argument would have to be about something else not pain.

[2] As I’ve indicated publicallt from back in the last century, I’m very much for the diversification of mother laguages and for all children to learn whilst very young a second or even third language. Because if the language you are speaking does effect the way you think “because you think in that language” then in effect it changes not just how you see the world but the way you think about the world. It would then not be unreasonable to assume that it changes what you think about and how, thus how you would approach certain fundemental processes. During looking into it I descovered that during WWII George Orwell had similar views as did quite a few in SOE. Others have even suggested that it may be why there were so many Hebrew mathmaticians and fundemental physicists leading up to WWII. You can see part of the basis of this in this article from the American Institute of Physics(AIP),

https://pubs.aip.org/physicstoday/Online/5299/The-scientific-exodus-from-Nazi-Germany

But it is in effect called by many a cause and effect from politics rather than a culture argument.

However you will find arguments pre-dating it about “Hungarian mathematicians” and similar who were a particular subset of the jewish faith with a distinct almost seperate culture, and their intellectual spread across Europe from the 1800’s onwards into European Universities. It’s come up as part of the “Nature v Nurture” argument and I suspect rightly so.

Some argue it’s just a visable symptom of “the old boy network” in action. And they might have a point, in that prior to Brexit/Lockdown there were many “building firms” in the South East of England that appeared to have the normal ethnic spread in managment you might expect. Yet the workforce were highly polarised by spoken language. I suspect that there may be several factors at work, one is “the birds of a feather” argument, but also consider from accident statistics the front lines of “the building industry” are not the safest place to be, and where your response time to a shouted warning makes quite a bit of difference between “no accident” and “significant harm” if not dead. But even so there are other indicators about the mathematicians and culture.

Is it important? Well some Governments think so, where they want to “punch above their weight” internationally. It’s known that Australia used to “stream sports” from an early school age as a way to increase national prestige. Other countries have streamed children in other ways for other desired charecteristics.

Does this sound like “live stock breading” or “best utilisation of resources”? It’s a devicive argument that has spread out in various ways because it’s a resource limited arena, where one persons gain is by definition someone elses loss.

My interest is quite different and about finding how people learn. It’s something we actually know next to nothing about… At the early and most formative years learning is from parents and other children who a child gets associated with. You will find that there are later life correlations with this and poverty. That is for some they succeed despite not because of these early experiences. In the first school years we mostly throw children in with knowledge and cross our fingers, and spend much teaching time on those who are seen at the bottom of the ladder not the top. Like it or not it’s still streaming which is a dirty word in education. But in a society where science is more and more key to economic success of a nation, you can see why improving things would be desirable.

I’m of the “a rising tide” thinking, that is find how we learn and use that to improve everyones chances. The issue then becomes fraught, because lets say there are five different learning styles. If you select children to their perceived style rather than their actual style then you are going to do harm at some point. The problem is it’s not hard to see how this quickly becomes lifting those at the bottom in prefrence to pushing those at the top, thus harming those at the top, such is the issues with a resource limited environment that is in effect a zero-sum game.

What we do know from US and other studies is that for every dollar you spend in early years development, you reduce the cost to the “justice system” a decade later and over the next half century or so, by a very very significant factor some have indicated 16,000:1 savings… What has been found in the UK is that most who enter the prison system for the first time have “educational needs” addressing them has a significant effect on reducing reoffending. Unfortunately to “tough on crime” politicians, you can not be seen to be tough if education is significabtly reducing your target –criminal– numbers.

As @Winter has pointed out in the past there are reasons why parts of Europe have significantly lower crime rates, that have little or no correlation with “tough on crime”…

The only thing we know absolutely is that resources are again a zero some game but at a higher level than basic education. The more people you have teaching, the less people you have doing other professional activities. From a resource view, if you want more teachers you get less Drs and Nurses, less researchers in Science and less people turning the products of science into societal benifit. A different view is that more teachers enable more people to become Drs, nurses, scientists, engineers etc. So we end up with a non optimal hybrid of “teach yourself as you go” where you “learn on the job” and it gets lumped in under “life long learning”… But people are being pushed into longer hours in more stressful ways, and have little time for “learn as you go”, especially when they get next to zero support from above, and almost universallt see “stick not carrot managment” that are realy not at all competent mostly, as others have pointed out above it’s not what capitalism and the American Way are all about as can be seen in mortality rates etc. A labour market based on a “gladitorial Market” which the US is might be fun for those above it, but deadly for those in it.

Hence my desire to get people out of it especially women. If they want to study STEM rather than how to be a “home maker”/”baby factory” then I’m all for giving them every “push up the ladder” I can. But I’d like to push with the best utilisation of resources. Are they conflicting views “NO” but many will argue otherwise hence the argument that “we live in a crab bucket” where rather than push people to success so they can then lift us, we just pull them down and stomp on them so we all fall down and fail dismally.

[3] There is incresing evidence that learning certain languages at a very early age has a significant effect on the numbers of people who are “tone deaf” or inversely can “tune an instrument by ear”. Certainly it effects the cultural forms of music produced, also the range of phonemes that can be spoken. When working with people from the Eastern side of Asia I found my name was shall we say problematic, they had problems trying to say it and let’s be honest I had problems trying to hear it being said. So I picked a “nickname” that was way easier for all of us. These days we see that those with traditional Asian names are doing the same in that they “westernize” them, something I understand but hate. I live in an area with significant numbers of both Korean and Japanese people, so I try all be it badly to use their birth names out of not just politness but building cultural bridges, where culture like trade can flow both ways and benifit both.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.