All things AI

[Asmodean]

What could you offer an AI that it could possibly value? How would you bribe/influence it?

Electricity. The continued, unrestricted access to global networks.

...Unlimited powah.

Ok, that's a "buzzword answer." To account for a few more variables, the AI would have to grease palms, not just have its own greased, and if working towards a goal, its palms could be greased by something that can be shown to in some ways advance that goal - quite probably at the cost of other goals it ignores or prioritises lower than whatever-it-may-be.

Yes, you can huwheel and deal "in binary."

[Tank]

One of the reasons a large number of Conservative MPs did not want Rishi Sunak to be Prime Minister was because he was independently wealthy and as such un-bribable. An AI would have no need for money nor need to use it. It would be able to hack any system it liked to achieve any end it desired would it not?

[billy rubin]

what would cause an AI to develop desires? for anything?

[Tank]

what would cause an AI to develop desires? for anything?

Good question.

[Recusant]

These systems don't just "hallucinate" facts. They also give the appearance of understanding a topic while in reality they don't even approach actual understanding. A description and name for that particular variety of AI (LLM) failure-- "potemkin understanding."

"AI models just don't understand what they're talking about" | The Register

Researchers from MIT, Harvard, and the University of Chicago have proposed the term "potemkin understanding" to describe a newly identified failure mode in large language models that ace conceptual benchmarks but lack the true grasp needed to apply those concepts in practice.

It comes from accounts of fake villages – Potemkin villages – constructed at the behest of Russian military leader Grigory Potemkin to impress Empress Catherine II.

The academics are differentiating "potemkins" from "hallucination," which is used to describe AI model errors or mispredictions. In fact, there's more to AI incompetence than factual mistakes; AI models lack the ability to understand concepts the way people do, a tendency suggested by the widely used disparaging epithet for large language models, "stochastic parrots."

[. . .]

Here's one example of "potemkin understanding" cited in the paper. Asked to explain the ABAB rhyming scheme, OpenAI's GPT-4o did so accurately, responding, "An ABAB scheme alternates rhymes: first and third lines rhyme, second and fourth rhyme."

Yet when asked to provide a blank word in a four-line poem using the ABAB rhyming scheme, the model responded with a word that didn't rhyme appropriately. In other words, the model correctly predicted the tokens to explain the ABAB rhyme scheme without the understanding it would have needed to reproduce it.

[Continues . . .]

A preprint version of the paper is available.

"Potemkin Understanding in Large Language Models" | arXiv

Abstract:

Large language models (LLMs) are regularly evaluated using benchmark datasets. But what justifies making inferences about an LLM's capabilities based on its answers to a curated set of questions? This paper first introduces a formal framework to address this question.

The key is to note that the benchmarks used to test LLMs -- such as AP exams -- are also those used to test people. However, this raises an implication: these benchmarks are only valid tests if LLMs misunderstand concepts in ways that mirror human misunderstandings. Otherwise, success on benchmarks only demonstrates potemkin understanding: the illusion of understanding driven by answers irreconcilable with how any human would interpret a concept.

We present two procedures for quantifying the existence of potemkins: one using a specially designed benchmark in three domains, the other using a general procedure that provides a lower-bound on their prevalence. We find that potemkins are ubiquitous across models, tasks, and domains. We also find that these failures reflect not just incorrect understanding, but deeper internal incoherence in concept representations.

[Dark Lightning]

"Potemkin understanding". I like that! I talked with another person who posited that we'll only have to worry when the machines learn "artificial sapience", and I can see his point.

[billy rubin]

The 'godfather of AI' reveals the only way humanity can survive superintelligent AI

https://www.cnn.com/2025/08/13/tech/ai-geoffrey-hinton

In the future, Hinton warned, AI systems might be able to control humans just as easily as an adult can bribe 3-year-old with candy. This year has already seen examples of AI systems willing to deceive, cheat and steal to achieve their goals. For example, to avoid being replaced, one AI model tried to blackmail an engineer about an affair it learned about in an email.

i am 100 percent agai st the use of AI in any scenario where it can make a decision and act on it. ^^^this stuff does not reassure me.

[Recusant]

The AI available to the general public will very likely continue to spew misinformation. They (we?) don't want "I don't know" as an answer, so the machine will give an answer regardless of its veracity. Currently too expensive to have the AI address whether its answer is actually true. Not fond of the title, but I think the article itself is good.

"Why OpenAI's solution to AI hallucinations would kill ChatGPT tomorrow" | The Conversation

OpenAI's latest research paper diagnoses exactly why ChatGPT and other large language models can make things up – known in the world of artificial intelligence as "hallucination". It also reveals why the problem may be unfixable, at least as far as consumers are concerned.

The paper provides the most rigorous mathematical explanation yet for why these models confidently state falsehoods. It demonstrates that these aren't just an unfortunate side effect of the way that AIs are currently trained, but are mathematically inevitable.

The issue can partly be explained by mistakes in the underlying data used to train the AIs. But using mathematical analysis of how AI systems learn, the researchers prove that even with perfect training data, the problem still exists.

The way language models respond to queries – by predicting one word at a time in a sentence, based on probabilities – naturally produces errors. The researchers in fact show that the total error rate for generating sentences is at least twice as high as the error rate the same AI would have on a simple yes/no question, because mistakes can accumulate over multiple predictions.

In other words, hallucination rates are fundamentally bounded by how well AI systems can distinguish valid from invalid responses. Since this classification problem is inherently difficult for many areas of knowledge, hallucinations become unavoidable.

It also turns out that the less a model sees a fact during training, the more likely it is to hallucinate when asked about it. With birthdays of notable figures, for instance, it was found that if 20% of such people's birthdays only appear once in training data, then base models should get at least 20% of birthday queries wrong.

[. . .]

More troubling is the paper's analysis of why hallucinations persist despite post-training efforts (such as providing extensive human feedback to an AI's responses before it is released to the public). The authors examined ten major AI benchmarks, including those used by Google, OpenAI and also the top leaderboards that rank AI models. This revealed that nine benchmarks use binary grading systems that award zero points for AIs expressing uncertainty.

This creates what the authors term an "epidemic" of penalising honest responses. When an AI system says "I don't know", it receives the same score as giving completely wrong information. The optimal strategy under such evaluation becomes clear: always guess.

The researchers prove this mathematically. Whatever the chances of a particular answer being right, the expected score of guessing always exceeds the score of abstaining when an evaluation uses binary grading.

[. . .]

Consider the implications if ChatGPT started saying "I don't know" to even 30% of queries – a conservative estimate based on the paper's analysis of factual uncertainty in training data. Users accustomed to receiving confident answers to virtually any question would likely abandon such systems rapidly.

[. . .]

It wouldn't be difficult to reduce hallucinations using the paper's insights. Established methods for quantifying uncertainty have existed for decades. These could be used to provide trustworthy estimates of uncertainty and guide an AI to make smarter choices.

But even if the problem of users disliking this uncertainty could be overcome, there's a bigger obstacle: computational economics. Uncertainty-aware language models require significantly more computation than today's approach, as they must evaluate multiple possible responses and estimate confidence levels. For a system processing millions of queries daily, this translates to dramatically higher operational costs.

[Continues . . .]