Nish Tahir
Nish Tahir

Notes on arxiv.org 🧵 Massive claims coming out of this paper. A framework for Artificial superintelligence by having an LLM based agent loop that automatically optimizes and discovers new NN architectures. Let's see if it holds up.

So the main claim that this paper makes is that they have hit AI's Alpha Go moment (meaning designing completely new unorthodox architectures that no human would ever make). For context see en.wikipedia.org

AlphaGo versus Lee Sedol - Wikipedia

en.wikipedia.org

Their system they call ASI-ARCH is focused on using LLMs as autonomous agents to do research and write up their own NN architectures and evaluate them for performance. They claim to have discovered 106 novel state of the art architectures using the system.

Their agent roles seem to be * Researcher - look at the most recent 50 attempts and propose changes * Engineer - train, evaluate * Analyst - mine data for insights * A fitness function - mimics a genetic algorithm to pick the best design * rinse and repeat Prompts are provided in the appendix Their training ground was on improving deltanet. They provide some tables that show the architectures developed by the model outperforming the baseline on a 20M and 340M implementation

There's some interesting research here, but calling this Alpha Go levels of game changing is kind of much. IMO they are really just presenting an agentic workflow, it's interesting but not exactly earth shattering It doesn't address what I think is the main elephant in the room. How does a system trained to reproduce stuff like humans (standard LLMs) create something that no human would even dream of - which is what AlphaGo did for Go? Essentially, I think they'd need to show that their process can break out of the constraints described in arxiv.org

Assessing and Understanding Creativity in Large Language Models

arxiv.org

1 / 3
While AI systems demonstrate exponentially improving capabilities, the pace of AI research itself remains linearly bounded by human cognitive capacity, creating an increasingly severe development bottleneck. We present ASI-ARCH, the first demonstration of Artificial Superintelligence for AI research (ASI4AI) in the critical domain of neural architecture discovery—a fully autonomous system that shatters this fundamental constraint by enabling AI to conduct its own architectural innovation. Moving beyond traditional Neural Ar- chitecture Search (NAS), which is fundamentally limited to exploring human-defined spaces, we introduce a paradigm shift from automated optimization to automated innovation. ASI-ARCH can conduct end-to-end scientific research in the challenging domain of architecture discovery, autonomously hypothesizing novel architectural concepts, implementing them as executable code, training and empirically validating their performance through rigorous experimentation and past human and AI experience. ASI-ARCH conducted 1,773 autonomous experiments over 20,000 GPU hours, culminating in the discovery of 106 innovative, state-of-the-art (SOTA) linear attention architectures. Like AlphaGo’s Move 37 that revealed unexpected strategic insights invisible to human players, our AI-discovered architectures demonstrate emergent de- sign principles that systematically surpass human-designed baselines and illuminate previously unknown pathways for architectural innovation (Fig. 2). Crucially, we establish the first empirical scaling law for scientific discovery itself—demonstrating that architectural breakthroughs can be scaled computationally, transforming research progress from a human-limited to a computation-scalable process. We provide comprehensive analysis of the emergent design patterns and autonomous research capabilities that enabled these breakthroughs, establishing a blueprint for self-accelerating AI systems. To democratize AI-driven research, we open-sour…
Our primary contributions establish a blueprint for self-accelerating AI systems and advance this paradigm: • ASI4AI Framework: We design and build the first demonstration of Artificial Superintelligence for AI research through a highly autonomous, tool-centric multi-agent system that enables AI to independently conduct the entire scientific research process—from hypothesis generation to empirical validation—in neural architecture discovery. • Emergent Design Intelligence: Through comprehensive analysis, we identify novel design patterns that emerge from AI-driven discovery, demonstrating qualitatively different architectural intelligence that expands beyond human design paradigms and establishes new principles for attention mechanism innovation. • Computational Scaling of Discovery: We discover 106 novel, state-of-the-art linear attention architectures and establish the first scaling law for automated scientific breakthroughs, proving that research progress can be scaled with computational resources rather than human expertise. We open-source the complete framework, discovered architectures, and cognitive traces to democratize AI-driven research.

Share this Page