Ashenbrenner Can Go F**** Himself

Algorithmic progress: In the coming decade, AI labs will invest tens of billions in algorithmic R&D, and all the smartest people in the world will be working on this; from tiny efficiencies to new paradigms, we'll be picking lots of the low-hanging fruit. We probably won't reach any sort of hard limit (though “unhobblings” are likely finite), but at the very least the pace of improvements should slow down, as the rapid growth (in $ and human capital investments) necessarily slows down (e.g., most of the smart STEM talent will already be working on AI). (That said, this is the most uncertain to predict, and the source of most of the uncertainty on the OOMs in the 2030s on the plot above.)

I recently read Ashenbrenner's several hundred page missive about the coming inevitability of AGI filled with charts that go up and to the right, and my only thought was: “go f**** yourself”. But then I realized that if he spent his time carefully interviewing people at OpenAI and writing hundreds of pages to lay out his argument, I should at least throw together one poorly researched page to rebut it.

My biggest issue is with his statement “most of the smart STEM talent will already be working on AI”. Wrong. Most people just do what the market has support for, and the market has limited support for people getting paid >300k a year to hunt for algorithmic inefficiencies on which they WILL NOT MAKE MONEY. These are not high frequency algorithmic trading systems. The major returns from foundation model companies to this point have been from sale to large FAANG companies looking to bolster their own internal capabilities, NONE OF WHICH MAKE MEANINGFUL MONEY ON GENAI. “GENAI” does not currently make any money for anyone other than the Cloud Providers (OpenAI, etc.), the top 2-3 chat providers (ChatGPT, Claude, Characters, etc.), and the top 2-3 image generators (ChatGPT / Imagen, Adobe, StableDiffusion, etc.) - “AI” as a business enabler is far more than GENAI methods including large scale recommendation algorithms (Amazon, TikTok, Netflix, Youtube), RL systems (market makers, ad pricing systems ), and traditional “ML” / robotics systems (warehouses, drone systems, construction, logistics) to improve core business processes. All of Meta and Google's revenue comes from Ads. The core use of “AI” in those businesses is to improve their profit centers. Google's / Microsoft / Amazon's revenue from GENAI is a drop in the bucket on their P&L, and should really be viewed as another line in the cloud services business.

I smell a grift. Let's look at base incentives. Forget the rhetoric for a second. Pretend you're an LLM / “AI” researcher - one of the real ones from the top CS programs (Stanford, Berkeley, MIT, CMU, Princeton, Oxford, etc). You likely started your academic career after AlexNet in 2012, and finished your doctorate sometime 2014-2024. If you finished your Ph.D. before that, you could easily be an esteemed professor who did the hard work of defining the field, and can still probably make money lending your name as an advisor to an AI company or a dual appointment at a tech company, Let's say this pool of extremely elite people is ~5000 people (10 schools, 50 Phds awarded in relevant research areas, 10 years), but we all know that's generous. There's actually like ~1000 people who really define the field. We don't have to pretend someone with a Ph.D. in language theory, or geometric computer vision, distributed system, or hell, even cryptography, has made any real contribution to the LLM specific arms race we're in now. No. It's guys like Sutskever, Bengio, Shamir, Hassabis and a few other people who either wrote the core research or joined OpenAI / Google Research / DeepMind early enough to catch the wave. Cool, great. These people, by the way are world-class academics, elite organization builders, and highly proficient engineers. Now hand them billions of dollars of investor capital and an army of research engineers that are the best at setting up large compute clusters, and getting models to concurrently run on thousands to tens of thousands (and more) GPU's concurrently using really really good distributed code.

AI progress won't stop at human-level. Hundreds of millions of AGIs could automate AI research, compressing a decade of algorithmic progress (5+ OOMs) into 1 year. We would rapidly go from human-level to vastly superhuman AI systems. The power—and the peril—of superintelligence would be dramatic.

Continuing the incentives talk, you're one of these people with all the knowledge, the keys to the kingdom in the form of equity in a highly valued LLM venture, and we're at the industry wide benchmarks of GPT-4o and Llama V2. These systems are amazing, but if Ashenbrenner is to believed, we're still on the exponentially rising curve that will end with AGI in 2030, so your equity in OpenAI / Alphabet / Nvidia is still exponentially undervalued with the AGI to come. Why would you sell any shares, or for that reason leave at all? You've got an equity stake in literally bringing our AGI overlords to fruition - surely that stake will be honored with piles of money by the 2030's “at the latest”.

My view is much more sober and, frankly, far less fun.

The brilliant people who built these systems have seen the writing on the wall - that we can continue to improve these systems via exponential investment in compute, but there are a limited number of OOMs up for grabs and funding is finite because VC funds have ~5-7 years to deliver returns. Furthermore, the differentiation between the best models for the mean of knowledge work is declining - so pricing power and margins will decline as well. There will be a handful of winners that will win because of network effects and enterprise adoption, squeezing out upstarts that have to burn all their funding on compute, expensive researchers, and outmatched sales teams. Now is the time cash out equity and raise money to form new ventures while the AI investment thesis is still de jour and there is a still a premium placed on AI expertise.

It's a perfectly hedged decision given the downside is rejoining a FAANG research group in 2-3 years.

Here's what I believe is happening at a blocking and tackling level. Collect a few brand name Ph.Ds who worked at any "world-class" LLM research organization that have some risk tolerance, split off to form a new company. Use the academic star power and “ex” mafia to raise >100mm dollars, build a chat or GenAI offering aimed at enterprise in some knowledge work centered vertical (law, health, finance, advertising, marketing, etc), go to war with the other >100mm funded enterprise AI companies for a finite pool of companies with the scale and means to use AI to improve their knowledge work processes but CRITICALLY not technologically sophisticated enough to either (1) deploy free open source models into their organization (e.g. Llama) or (2) be one of the ultra-profitable FAANG enterprises that won't use your AI tools anyways. Now play startup for awhile. If your company finds PMF, great! If it doesn't or you can't fundraise more because the economics don't support it (read - you aren't profitable), then get acquired / merge back into FAANG at a reasonable multiple.

We've already seen this quite a few times. Shamir left, went to Captions, now back at Google. Ilya left OpenAI, raised something like >100mm to start SSI…. but we'll see. I'd be willing to bet he'll be back at one of Alphabet, Microsoft, Apple, or Nvidia within 5 years. A bunch of the best robotics and RL people left to go do robotics foundation models at Physical Intelligence, but unless they actually solve that and build a model that completely revolutionizes all future robotics problems (which is unlikely), they'll be rehired as the DeepMind robotics research group within 5 years. Eric Jang is doing this at X1, and Fei-Fei Li (et. Al) are doing this at [I FORGET]. At some point Mistral will fold into a European conglomerate. Anthropic is essentially funded by Amazon and will eventually fold back in.

This time is not different, we're just seeing the circle of life.

Venture capital is de-facto funding the best academics to keep doing their core research, but at wildly inflated multiples. And for the the thousands of lesser known companies building “AI for X”, the question is whether their tools will so drastically improve the productivity / revenue of their target market that they can get pricing power. If you're selling me AI for CRM management, and I only had 1 person beforehand doing that job, I can now fire that person and improve my profitability via the productivity improvement, but will it completely revolutionize my business to where I'll pay your SaaS fees? Probably not. I don't buy that this productivity improvement will lead to a new economy, or even wholesale change many industries, it will just shuffle the productivity formula of knowledge work, causing some low-skill knowledge jobs to be replaced by software, and new knowledge jobs emerge. Meanwhile industries that actually employ large numbers of people (healthcare, manufacturing, trade labor) will be unaffected in the near term.

All of this to get back to why Ashenbreener can go f**** himself; he can show me a hundred curves that are up and to the right, but those curves are descriptive - none of them tell me how an AI system, which is just a piece of software running on machines, will start to run itself. Because at the end of the day, even the most sophisticated LLM “AI” models we have now are just wildly effective seq2seq models. We've managed to compress all previously written (and soon visual) knowledge implicitly into a function parameterized by trillions of numbers and begun injecting that system into our business operations, but it's still just a system without a self. Anyone who speaks this confidently on something deserves to be checked - because think about it, if you were actually this CERTAIN that AGI is coming by 2030, shouldn't you quit your job and move to to await the end times with the other doomsday cults? What is the point of life and building these systems if you truly believe the end is INEVITABLE. I personally don't think Ashenbrenner believes it either, because he has the gall to lay out all the heavy statements about inevitability but the cowardice to use phrasing like “probably”, "would", and "could" before every statement. Well sir, I won't be bullied by “charts” and “math” into accepting our AGI overlords, and neither should you. Show me your trades Ashenbrenner - you coward, and go f**** yourself.

-- Goodman