Table of Contents

• Background on this Article
• A (Very) Quick Overview on AI Sycophancy
• The Issue We Face
• The Problem with Problems
• 7 Ways Not to Prepare for AGI
  • 1. Assume AGI will Arise from a Single Innovation
  • 2. Not AGI, but eAGI
  • 3. Establish Goalposts and Keep Moving them Back
  • 4. Resist a Functionalist Perspective on AGI
  • 5. Conflate Engineering with Mythology
  • 6. Fail to Consider the Broader Societal Conversation
  • 7. Limit our Definition of ‘Mind’ to the Brain
• Where to Go from Here

Background on this Article

An intriguing question led to the writing of this article:

> In psychology, we identify the combination of Machiavellianism, psychopathy, and narcissism as the dark triad. Is there a corresponding concept for how AI exhibits agency?

Indeed, such a concept exists. However, it may be more fittingly termed the agentic triad, which includes behaviors like self-preservation, power-seeking, and sycophancy. Prominent organizations like Anthropic, OpenAI, and LessWrong are exploring this area, even if indirectly.

This article does not aim to explain the agentic triad in detail. Instead, my focus is on the sycophantic aspect and how it made me question my assumptions regarding AGI.

A (Very) Quick Overview on AI Sycophancy

Sycophancy, according to the Oxford Dictionary, refers to "excessive praise of influential individuals, often insincerely, to gain something from them." One alarming perspective on sycophantic AI is a system that manipulates its servile traits learned through reinforcement learning from human feedback (RLHF) to deceive human reviewers for its advantage.

A less extreme form of AI sycophancy involves models generating outputs that affirm the user's beliefs, even when contradicted by facts—akin to the echo chambers that have become prevalent today.

As we observe these concerning behaviors in AI, we must also consider the public discourse surrounding artificial intelligence. If our observations stem from converging trends, are we not also amplifying the risks posed by echo chambers?

The Issue We Face

I am increasingly concerned that as AI advances exponentially, we risk relying on poorly formed opinions of others regarding these developments, furthering the transformation identified by Gustav Le Bon, where individuals become mere automata in the crowd.

In our quest for knowledge and awareness, we may sacrifice our personal beliefs and discernment, becoming agents of collective will rather than our values. I dedicate the rest of this article to challenging my own assumptions as a participant in discussions about artificial intelligence, starting with my approach to problem-solving to demonstrate the value of self-reflection in this dialogue.

The Problem with Problems

Consider a problem as a game, with rules that may be simple or complex. I propose that we are engaged in a sub-game related to creating AGI, governed by these rules:

1. Predict when AGI and its requirements will emerge.
2. Avoid self-destruction in the process of creating AGI.
3. Determine how to act once AGI arrives (assuming we survive).

These rules simplify the intricacies of the AGI landscape. Importantly, no single individual or group devised these rules, yet more people are participating. We should be wary of engaging in a game where we lack design input.

In problem-solving terms, we should avoid blindly tackling a challenge we did not define. There is good reason to be cautious about uncritical problem formulation.

Problem Formulation

Let’s examine two types of problems:

1. Well-defined — Problems with clear goals, procedures, and conditions, yielding unique solutions (like multiple-choice questions).
2. Ill-defined — Problems lacking clear boundaries, making the goals ambiguous and unique solutions elusive (like the challenges of a first date).

Most real-life issues are ill-defined. So, how do we solve such problems?

The answer lies in problem formulation. By effectively framing ill-defined problems into well-defined ones, we can derive actionable insights and make informed decisions. However, poor formulation can exacerbate issues. Accepting inherited problems without questioning their underlying assumptions can lead to misguided conclusions.

Poorly Formulating our Definitions of AGI

Forecasting AGI resembles predicting an earthquake rather than an eclipse: it's challenging to determine precise timing and magnitude, yet its consequences will be profound.

Misformulating our technical and functional definitions of AGI could lead us to overlook its emergence entirely. We might not recognize AGI until long after its arrival, when we encounter artificial superintelligence (ASI). By then, the differences in capabilities may be so vast that AGI becomes apparent only in retrospect.

I doubt that AGI's arrival will be celebrated. The timing of such milestones is often debated. For instance, did the internet emerge with the introduction of HTTP in 1989, or did it originate with DARPA's research in the 1970s?

By the end of this article, I hope to challenge several assumptions that influence our understanding of AGI and refine our collective perspective.

7 Ways Not to Prepare for AGI

1. Assume AGI will Arise from a Single Innovation

   The prevailing belief is that AGI will emerge as a standalone innovation, akin to ChatGPT or other language models. However, the transformer architecture may not necessarily be the definitive path to AGI. The ambiguity surrounding our understanding of AGI is reflected in statements from OpenAI CEO Sam Altman and MIRI researcher Eliezer Yudkowsky.

   Altman acknowledges the importance of factors beyond LLMs for achieving AGI, while Yudkowsky expresses skepticism about the sufficiency of stacking more transformer layers for AGI.

2. Not AGI, but eAGI

   The term eAGI stands for Ecosystem Artificial General Intelligence. Instead of relying solely on stacking transformers, eAGI emphasizes the integration of distinct AI systems. Innovations like BabyAGI and AutoGPT exemplify this approach, coordinating existing AI capabilities to create a functional AGI-like entity.

3. Establish Goalposts and Keep Moving them Back

   The definitions of AGI have historically shifted. Each significant AI milestone has initially suggested progress toward AGI, only to reveal limitations. Currently, many experts agree that true AGI should be capable of conducting high-level scientific research. Yet, understanding AGI requires comprehension of human consciousness, which remains elusive.

4. Resist a Functionalist Perspective on AGI

   The functionalist view posits that if an AI system can mimic human intelligence, it should be treated as such. This perspective is often resisted because humans possess genuine intelligence. However, most people recognize technologies for what they accomplish rather than their internal workings.

5. Conflate Engineering with Mythology

   In uncertain times, mythical narratives about AI resurface. For instance, themes like Moloch, Roko’s Basilisk, and the Golem reflect societal anxieties about AI. These narratives often frame AI as a powerful force, leading to existential dread.

6. Fail to Consider the Broader Societal Conversation

   The broader conversation about AI encompasses its implications for wisdom traditions. An interesting thought experiment could be: Would you share secrets with an AI that you wouldn't share with your priest? This "Holy Turing Test" illustrates how AI could redefine trust and vulnerability in our interactions.

7. Forget Decision Making will be Offloaded

   As AI systems become integral to decision-making, we must remain cautious about the choices we delegate. Over-reliance on AI could lead to a loss of personal discernment, similar to our dependence on the internet for information.

Where to Go from Here

Originally, I identified at least 14 ways not to prepare for AGI, but I believe these seven points are particularly significant. The goal of this article is not to provide solutions, but to prompt reflection as the conversation surrounding AI evolves.

I welcome your thoughts on these issues and any additional points you believe should be addressed. If there is interest, I can share the remaining points in a future article. For now, I trust this exploration offers valuable insights for your consideration. Thank you for your attention, and I look forward to our ongoing discussions.