Schneier on Security

Clive Robinson • September 23, 2022 10:31 AM

@ Bruce, ALL,

“This is an interesting attack I had not previously considered.”

I guess the reason is like nearly all people you’ve not had cause to.

One of the signs of a new and important idea is that you almost instantly grasp it’s fundementals and then your mind goes

“Oh… That means… And…”

And so a whole world of new posabilities opens up.

We’ve actually known that this sort of thing is possible since the late 1960’s and we’ve failed to solve it in a reliable way.

There are two very fundemental problems involved and we have no real idea how to solve them simultaniously.

1, Data is code and code is data[1].
2, In band signalling is unreliable without modifying input and causes serialisation and timing issues[2].

Without an absolutly reliable way to signify when the “instructions” have stopped and the “data” started then as an attacker you will always be able to pull off this kind of trick.

[1] It should be obvious but it bears repeating, all a computer understands at the base layer is basic logic and unsigned integers. Signed integers use “in-band-signalling” and data encoding (usually 2’s complement). Both of which effect the range of data and cause it to be unbalanced.

In a von Newman architecture there is only one memory space and it can store both “instruction” code and “data”. There is no way for the base computer to distinquish them, thus it has to be “told” in various ways. At the heart of things all computers are “interpreters” you give them bus wide integers in a number of sequences that determine if they are instructions or data by the type of “signalling” used. If the signalling goes wrong in some way then data becomes instructions and a vulnerability to data input attack happens (look for “Smashing the stack for fun and profit” for more details). The problem is the higher you go up the computing stack the harder it is to differentiate between instructions and data. Natural language mixes instructions and data freely as humans are expected to be smart enough to know the difference implicitly. The thing is we don’t so how the heck we implicitly expect a computer to be able to do it…

[2] Some here are old enough to remember the debate between “C-Strings” and “Pascal-Strings” all most know is “Pascal-Strings have been confoned to history”. What is important is that Pascal-Strings used “Out-of-Band Signaling” where a number was stored in memory that contained the length of the string. Pascal-Strings are therefore a more complex “Abstract Data Type”(ADT) importantly of “known characteristics” one of the most important being that all values in a string were alowable. C-Strings were however just a linear array of consecutive bytes, with the only thing known was where the start address was, the end had in every occurrence had to be found by searching which was at best inefficient and usually problematic. The reason being the “null terminator” tacked on the end. This is “in-band Signaling” and it’s always a problem, primarally because one or more values that can be stored in a string have “special meaning”. Any one who has been involved with poorly designed serialisation –much of it– knows how much of an issue this can be. The sad thing is actual communications engineers coming up with standards were only too aware of the problem. There is a protocol called HDLC and it can sit on a physical layer that is either synchronous or asynchronous. To alow for the issues synchronous communications created they had to come up with a “bit stuffing” protocol,

https://en.m.wikipedia.org/wiki/Bit_stuffing

Bit stuffing creates two problems, the first is unknown length of time to send a given amount of data. The second is it can cause “packet fracture” that has ripple back effcts up the data comms stack and this can make things problematic in high speed data networks.

[]

Clive Robinson • September 25, 2022 7:19 PM

@ lurker, winter,

Re : Forcing Inteligence.

“There have been various experiments over the years attempting to nurture “intelligent” birds, and chimpanzees, in a human environment“

The big problem with this as I’ve highlighted is,

“in a human environment”

There is a saying of,

“We are a product of our environment.”

Whist true it leaves a lot out. For instance I have survived quite harsh circumstances in many environments that would be “alien” to most in their late adolescent to mid twenties. There are a couple of reasons for this when I was very young (ie first 8years),

1, My environment was still post war non technical.
2, All the adults around me had lived through WWII and the harshness it created.

Thus I had to “learn to stand on my own two bare feet” and know how to do things even without the most basic of technology like matches or even cooking pots. Which os why it comes as a big surprise to many that you can actually heat water in a paper bag over an open fire IF and it’s a very important IF you know either,

1, The skill as a learned trade.
2, Work it out via knowledge of science.

With the first one you have a “pattern” or “artisanal secret” you can trade on. You can sort of scale it but not inovate with it. Inovation as such is by accident or by trial and error, and all you end up with is another “pattern”. It’s effectively “found knowledge” not “reasoned knowledge” and it does not realy take you forward.

With the second you can pull the same trick with any type of container including plastic bottles or leather bags, that could be even a leather boot at a pinch (I’ve demonstrated it with a triangular piece of sewn leather used as the seat of a three legged stool). The reason is you know that the temprature on the outside of the container if thin is primarily due to the temprature of the liquid inside, and if pourous due to the latent heat from evaporation. This knowledge enables you to work out where to place the container with respect to the heat comming off the fire and it’s reduction due to convection effects. Whilst you still might need to “experiment” this is guided by knowledge and reason and is much faster than dumb luck or trial and error.

Understanding about the difference between “artisanal” and “science” behaviours gives the practical real world difference between having a “trade” and being an “engineer”.

Whilst a “trade” took more than five centuries to give us the “coach wheel” enginering gave us efficient heat engines in less than half a century, likewise electrical communications over vast distances, oh and likewis got us to the moon in around a quater of a century. So ten to twenty times as fast to radically change our environment. Computers and other technologies are changing it at an even faster rate. It’s why “technology” is forcing an effective environment evolutionary change in some humans, by “continuous learning” from “reasoning”. Whilst the majority tend to hold to a moment in time by the mechanical skills they learn whilst still adolescent, then spend the rest of their lives honing those mechanical skills.

We want to find the former in these “animal tests” but mostly get the latter. Which brings us to another saying of,

“A fish out of water.”

If you take an average adult human out of an environment it is comfortable in, the chances are for most they won’t survive let alone thrive and will be dead in just a few days to a few months depending on the level of environmental change. They are creatures of the environment they never learned to live outside of whilst they were still adaptable enough to do it.

If we apply similar logic to animals their infant to end of adolescent time is a very small fraction of that of humans, often measured in a handfull of weaks not a couple of decades.

So I don’t expect the average animal to have the time to learn to reason, they are hard enough pushed just to get environmental mechanical skill sets done.

If I am right or wrong on this can be tested by “back to the wild” experiments trying to teach creatures effectively “fed by the hand of man” to learn wild environment skill sets. We know they have not been very successfull in “big cat” trials in the past.

Clive Robinson • September 26, 2022 1:13 AM

@ SpaceLifeForm, JonKnowsNothing, lurker, Winter, All,

Re : Stable Confusion

About the only two things I would say they have correct are,

1, AI is an unfortunate term.
2, Nobody knows what the technology goals are long term.

So the only thing true is that the domain has two parts. Firstly “research” that is mostly “aimless” in the true meaning of the word (ie has no inherant/relavant goal). Secondly “Production” that is often ainless in of it’s self but reflects what “people will purchase”.

Thus the “market” for production is one that is in theory “needs driven”. But what are those needs?

Perhaps if we examined those we would come to better naming. Without going through it “Scapegoat” or “Excuse” systems might be the highest truthful answers, though “Agenda Following” might be better. That is curently people buy systems that follow “preconcieved notions” of what they want. When conected to codifing covert or overt latent prejudices these systems are extreamly dangerous as they provide,

1, “Computer says” excuses.
2, “Black Box” excuses.

Neither of which are factually correct but easily alow “arms length discrimination” by people with “Agendas”.

But how did we get hear?

Well historically,

In the early days it was as I’ve explained before “expert systems” or basic fixed “rule following” based on “facts” thus was an electronic version of a paper book of flow charts.

This was problematic because the “question / answer” flow was still “opinion” rather than “fact based”. For instance the question and answers might be entirely based on facts, but is the question the right one? That is the question might be related to colour perception, or, to “touch perception” both in of themselves fact based, but which is the correct question to ask at a given point in time. It effectively is still “opinion” rather than “fact” based though one step removed. Currently we are trying to make such systems appear to be “fact” based rather than “opinion” based but are only adding layers to hide the “opinion” thus providing easier excuses to hide “prejudice”.

The main difference today is that Machine Learning –also an inappropriate name– is still an “Expert System” but these days uses statistical meathods to find the “rules” or “questions answers” to build the expert system with.

In essence when you boil down these ML systems we have had so far, you end up with the same “rule following” behaviours, but with the number of rules beyond most human comprehension. More important is they have so many “dimensions” these too are beyond most comprehension. The result is that we can explain individual rules, like we can the nuts and bolts of a complex machine like a 747 aircraft. Yet our understanding of the nuts and bolts provides no answers as to what the machine does or how it behaves. That is we have to “look beyond” the individual rules, and even related collections of rules and try to see the system and it’s actual operation. When designing an aircraft this is comparitively easy because we know what it is we want to do so we “top down design” which is many orders of magnitude easier than just looking at the parts of the system and trying to “bottom up reason” the functionality (in actuallity it’s the same issue as with “one way functions” used in crypto algorithms).

When expressed this way it’s easy to see that an “artfully chosen” set of data will have the prejudice “built in” statistically, thus the resulting rules will follow the “prejudice”.

Less obvious but sadly true, as some have now practically demonstrated the “early data taints the rules”. Thus you can have a statistically neutral data set, but the order you present the data weights the rules in favour of the “prejudice” contrary to the over all data set…

But also the statistics all the systems use suffer from the “early late” issue at all stages. So using tainted-order data-sets still moves the out come in desired ways. In essence the statistical systems behave like “leaky integrators” so have the equivalent of atleast two time constants of charge and discharge. Knowing what these time constants are alows you to “pump the system” to achive a desired outcome.

This is effectively the same as in signal processing where we have the notion of “matched filters” where it’s not just “time” but “order” that can be selected for but also “amplitude” of each component. That is each dimention of the signal can be used to “pattern match” to produce a desired result. The real fun comes when you start to look at how “Adaptive Matched Filters” work. In essence they are “unstable” using “positive feed back” but are stabalised by using “negative feed back” in certain ways.

All the current ML systems can be seen in this way only the “weights” that control the stabalising feedback are effectively hidden behind complexity as complex as that used in cryptographic systems. So we as “observers” outside the box with the “Chinese Door” have to take ask many questions and analyse the answers to try to deduce the rules. Something that in effect takes to long to do…

But this undecidability issuse in ML systems is highly desirable by those that wish to hide prejudice and keep it at more than arms length…

There is what was a fiction story at the time it was written about a weapon that used “facial recognition” it was used to perform a political assasination. In the story they can not find the face pattern used by the assasin and it was believed that no photo or other image of the target existed. The story explains it away as it was “discovered” that the assasin knew the target would wear an item of cloathing that was “black and white checked”. So rather than match the face it matched the item of cloathing.

The problem is this is actually exactly what you can do with modetn ML systems. They have no definition as to what a face is or what an item of cloathing is just a “pattern” so a pattern for a black and white check is just as valid to an ML system is, as is one of a face. You as an observer have to find out what the signal is without knowing what type it is. So if you are looking for “facial features” in your interogation search by using modified face photographs you will probably fail.