Schneier on Security

Clive Robinson • May 25, 2023 10:03 AM

@ ALL,

Re : Filtered Stochastic sources.

We need to remember –as I keep saying– LLM’s are just adaptive filters in a massively parellel form. Thus their behaviour is,

“All in the MAD weights”

So,

“They that control the weights, control the system…”

Only we currently have no idea how the complex interaction of billions of these weights work to produce the effects we see from LLM’s

Thus getting an LLM can be seen as a two stage process,

1, Create weights based on input data.
2, Tweak weights based on more select data.

The first stage could be compared to the chaotic and massive use of resources we see with geological upthrust that creates mountain ranges and deep valleys etc.

The second stage could be compared to the smothing of the first stage results we call weathering.

There is an obvious third stage we’ve not yet got to with LLM’s which is engineering of the landscape, which man does to the surface of the earth to create a more amenable environment.

Which brings us to,

“Given that we’ve known about model poisoning for years, and given the strong incentives the black-hat SEO crowd has to manipulate results, it’s entirely possible that bad actors have been poisoning ChatGPT for months.”

The thing is as seen with Meta’s model escaping whilst the first stage requires massive crudely applied resourses. The second stage of weathering can use vastly less resources and be quite localized in comparison to a geological up thrust or other tectonic plate event. So much so that focused versions with limited scope can now run on laptops and even higher end “Single Board Computers”(SBC).

But weathering whilst limited in resources and scope, is still highly inefficient compared to “engineering” to a requirment.

As far as publically published data goes we have not yet got to the point where we can “engineer the weights”, thus poisoning is still both inaccurate and resource intensive.

However the speed things went from first stage development to second stage development whilst initially very slow, has now as resource demands come down sped up in a way that none would have predicted publically only a few months ago.

So we need to realise that,

“It is inevitable that at some stage probably very soon, we will be able to directly engineer the weights without either the first or second stages being required.”

People realy need to think about the consequences of that happening. For one thing whilst “poisoning” will initially benifit from it, it poisoning as we currently know it will soon be irrelevant.

That is it will become known which weights need to be tweaked and by how much to get a desirable shift. Most probably quite close to an optimal minimum number of very small changes.

That is a few bits here and a few bits there, certainly achievable by a small memory resident program, hidden from view just watching for tell-tail data coming in.

Then the malware makes the changes to the weights for the desired period of time then removes all trace of it’s self from memory and even from the weights if required.

How to stop such an attack may actually be harder to do than getting Quantum Computing up and running, and almost certainly way more resource intensive on the larger LLMs…

Clive Robinson • May 25, 2023 1:40 PM

@ Winter, Jason, ALL,

Re : Can you tell?

“Anyone who wants to build LLMs will have to start with constructing filters to remove the output of other LLMs from their training data.”

That would require a pre-filter based on a recognizer circuit of some kind.

But ask yourself,

“How would such a circuit be built?”

All of a sudden you realise you are in a “Turtles all the way down” problem, against an active opponent.

Let’s make a spherical cow first approximation of effectively,

“Average the data and filter by distance from the mean”.

That will work only in extrodinarily limited cases and only if there are no suppliers of data with agency and intent to falsify results.

With “search enginess” wr already know there are numerous “Black Hat” attackers making significant sums of money distorting search engine results.

So any system based on averaging is going to fail, unless you pre-filter out bad data… So you have to pre-filter before you prefilter… So now you’ve gone from a spherical cow to a cubic cow.

So now you have to pre-filter against another set of issues, and so on. Each stage changing the shape of your cow model, but are any actually “cow shaped?” the simple answer is NO.

So you have to ask,

“Can a recogniser circuit actually be built?”

The answer is “No” unless you already have a highly accurate map of the cow in all it’s complexities. To which the obvious question is,

“If you have such an accurate map of a cow that you can judge what is fake and what is real by it…, why do you need to use unfiltered data from unknown and almost certainly untrustworthy sources?”

The answer is “You would not” which gives the inescapable issue of,

“You do not have an accurate map so you have to build one”

But you’ve no way to tell if the data you aquire is not biased, poisoned or in some other way distorted by another “agent in the game”…

Whilst humans fall foul of this themselves, usually they can work out some “gut hunch” for plausability. It’s almost certainly wrong in nearly all cases but,

“not that wrong, for a starting hypothesis”.

Because we have multiple ways to check, that are not just based on supplied data… That is we have some level of “experience” and multiple sub conscious tells that are different for every one, and as a rule “cheaters” are sufficiently disliked they get not just found out but called out and penalized in multiple ways…

We don’t currently have ways to penalize cheaters in the data sets LLM generators use. So at the very least we need such, as well as investigators, and adjudicators much like the justice system we’ve built up over the past couple of millennia.

Clive Robinson • May 26, 2023 10:50 PM

@ lurker, Hans,

Re : Could is a recognition of might by an observer.

“It lacks the clarity, precision, and factual basis I would expect from an entity being aimed as the Ruler of the World.”

The LLM is being asked to make an evaluative observation based on an input (set of rules) that can arbitrarily be chosen by a definer from a set of all rules. The output set from such rules is “planet, not planet”.

That is all “the rules” form a function,

f : A to B

Where A is a set of unknown size and B is a set of size 2 {planet, not planet}

Thus it is not difficult to realise that for a fixed set of measurments of Pluto, some functions will give “planet” and others will give “not planet”. Especially as there have been two such functions already the first that gave “planet” the second that gave “not planet”.

As the picking of the rules is arbitary then any future functions are likewise arbitary as is what planetary object is decided to be a “planet” or “not planet”.

The question you should thus be considering is,

“Are the rules that make the functions arbitary or do the changes in the rules follow so kind of reasoned logic?”

If you say “reasoned logic” can an observer be expected to be able to determin “the logic” or “know the logic”?

Further can you also that the “reasoned logic” is immutable, thus based on some fundemental of the universe, or just arbitary like a line drawn in the sand below high water mark?

I would not expect an LLM to be any more omniscient than I would any entity capable of reasoning by mechanically applying the rules of logic in a constrained environment.

But to take it further I would not expect any entity mechanically applying the rules of logic even in a non constrained environment to be able to give a meaningfull answer to the question asked. Because the question lacks any kind of deductable constraint.

Thus the defect is not in the LLM answer but in the question asked.

The question can be reduced to that of,

“Can an observer of an Oracle, reliably determine if the Oracle’s output is random or determanistic?”

The answer is always “no” with an unbounded output, and only probabalistic with a bounded sample of output.

To see why consider a mechanical odometer, untill it “rolls over” you can only show it is behaving like a counter. When it does roll over you can not say if it will always roll over at that particular count or not. That is the roll over count could be chosen by another mechanical sequencer, or by some apparently random input. The latter can be seen in real world “integrating measuring instruments”. I give an example of which when talking about “random generators” using two free running oscillators and a counter. One side effect of which is the count whilst appearing random over a short range can be shown to actually be a very pure sinewave at a fundemental difference frequency between the two oscilators (and why chip manufacturers should not use “ring oscillators driving D type latches or other counters” as supposed TRNGs).

Clive Robinson • May 30, 2023 3:10 AM

@ vas pup, ALL,

Re : Neural Net black boxes

From what you quote,

“Even more problematic is the opacity of the system, which experts call a “black box” because even they barely understand how it works.”

It’s actually worse than that.

Even for experts it’s still very much like,

“Most know a car has a wheel at each corner and one for the driver, but they don’t know how the radio works.”

Neural Networks are not structurally very difficult to realise. Heck I built some back in the 1980’s. They are at heart a very very simple circuit and you can build one to run on even an old 6502 or Z80 8 bit CPU of that era. All you need is a multiplier and adder and a table of weights. But they were of little or no practical use so as engineers we went down the “fuzzy logic” path –derived from Alfred Tarski’s multi-valued logic/reasoning– instead, as it gave not just usable but better results than control theory.

Now we have tables with multiple billons of weights these Neural Networks are starting to show benifit[1].

But they are still little more than “Digital Signal Processing”(DSP) filters at heart. If you look at the circuit for an “AI Transformer” and rip away the jargon you will see it for what it is a form of Feed Forward filter in a feedback loop[2]. You will often see it compared to the Doya notion of brain learning functions[3].

The thing we don’t understand and may never properly understand is,

“How information form is stored in the weights.”

By “information form”, I don’t mean “data” information that you would encode in say a zip or other compressing archive. It’s the information about how the information is communicated, kind of meta-meta-data but without the base data. It’s the rules of communications, and to a certain extent reasoning.

At it’s lower level it behaves like a piano or other musical instrument, where each note is a “resonant filter”, but just making notes is not making music. That needs fine control of the energy in each resonator against sequence and time, such that it convays or communicates information we barely understand so lable as “emotion”.

So we have a “blind leading the blind” type problem. We can not quantify what we mean by “emotion” so we resort to hand waving and say “touchy feely things” about it. What we have is billions of numbers in a table, and thus an N^2-N set of relations between them. And in those spaces in between them we have an anolog of “emotion” etc… Have a look back on this blog to where I’ve talked about “data shadows” in the past, used to securely hide roots of trust to see why this black box problem might be an insermountable problem to solve.

There is the old joke about looking for something, which boils down to,

1, You know it by sight so look around systematically for it till you find it.
2, You know it not, so you blunder around in the dark untill by happenstance you trip over and it hits you in the eye, so you can’t ignore it…

At the moment, we don’t even know how to “blunder around” Alfred Tarski was doing his work back in the 1920’s nearly a century ago.

To see it another way, anoungst us we know what sand, limestone, fire, and water are, and ways we can apply them to each other. But they don’t give you the emotions that majestic architecture does.

[1] The theory says the weights are real numbers with infinite precision between zero and one. In practice we don’t even need IEEE floats or even much more than a few bits long integers faked up as reals.

[2] If you want to make real money in AI don’t invest in the startups and VC “pump and dumps” that create churn and hype but not utility. Instead look at those who supply them with what they blow billions of VC cash on… King of that heap is NVIDIA and it’s highly parallel processing “Graphics Processing Units”(GPUs)… They were draging massive profit in from crypto-mining rigs, and are now doing the same with AI and that new run of “Fools Gold”, which might account for why they blog etc about AI. Which gives a nice non tech description of the “AI transformer Model” algorithm,

https://blogs.nvidia.com/blog/2022/03/25/what-is-a-transformer-model/

Note in the flow chart the significance of the “Feed Forward Network” and “Add and Normalize” functions, these are DSP algorithms that work “oh so well” on GPU’s and get used multi billions of times a second.

For a little more depth and a walk through,

https://www.tensorflow.org/text/tutorials/transformer

[3] Back at the end of the last century AI and brain research where playing “swapsies” with ideas and Prof Kenji Doya et al came up with the notion of “Learning algorithms of the human cerebellum, basal ganglia, and cerebral cortex” that is some what famously pictured as,

http://www.brain-ai.jp/wp-content/uploads/2017/01/brain.png

Clive Robinson • May 30, 2023 1:32 PM

@ Winter, vas pup,

Sorry guys looks like there is a dose of road rash going through.