Please enjoy this transcript of my interview with Sebastian Mallaby (@scmallaby), the Paul A. Volcker senior fellow for international economics at the Council on Foreign Relations, a two-time Pulitzer Prize finalist, and the author of six books, including More Money Than God, The Power Law, The Man Who Knew, and The World’s Banker. His latest book is The Infinity Machine: Demis Hassabis, DeepMind, and […] The post The Tim Ferriss Show Transcripts: Sebastian Mallaby, Biographer of Demis Hassabis — Lessons from 100+ AI Insiders on The Race to Superintelligence, The Religion of AI, and Spotting Breakthroughs Early (#870) appeared first on The Blog of Author Tim Ferriss.
Please enjoy this transcript of my interview with Sebastian Mallaby (@scmallaby), the Paul A. Volcker senior fellow for international economics at the Council on Foreign Relations, a two-time Pulitzer Prize finalist, and the author of six books, including More Money Than God, The Power Law, The Man Who Knew, and The World’s Banker. His latest book is The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence.
Books, people, tools, and resources mentioned in the interview
Legal conditions/copyright information
Additional podcast platforms
Listen to this episode on Apple Podcasts, Spotify, Overcast, Podcast Addict, Pocket Casts, Castbox, YouTube Music, Amazon Music, Audible, or on your favorite podcast platform.
Transcripts may contain a few typos. With many episodes lasting 2+ hours, it can be difficult to catch minor errors. Enjoy!
Tim Ferriss: So Sebastian, lovely to see you and thanks for making the time. I really appreciate it.
This episode (coming soon)
Sebastian Mallaby: Great to be with you, Tim.
Tim Ferriss: All right. I have a million different questions. Part of the challenge with this conversation was deciding which vector to take into the conversation knowing that we don’t have infinite time to talk today.
Tim Ferriss: I wanted to just give you applause for writing some of my favorite books of the last many years. I am consistently impressed, and maybe, since I also put pen to paper every once in a while, depressed, just thinking relatively about my capabilities, but of your capacity to paint a picture of the players on a landscape, but also the games they play in ways that non-specialists can understand.
And I can’t recall who first recommended it. Frankly, I believe it was a hedge fund manager in New York City, but More Money Than God: Hedge Funds and the Making of a New Elite, certainly that was, in my particular case, followed by reading The Power Law: Venture Capital and the Making of the New Future, which I didn’t expect to learn as much from because I’ve spent 20 years surrounded by venture capitalists and doing angel investing, 17 years of that in Silicon Valley. And yet, I still had hundreds of highlights and so many stories that grabbed me from that book, which I had not heard. And that made me very excited to read The Infinity Machine, which this is the new book. And I realized also I’d been pronouncing Demis’ name incorrectly for a very long time despite having met him at one point.
So Demis Hassabis, DeepMind and the Quest for Superintelligence. My question for you, and we’re going to come back to present day for people who are interested, of course, in what has been painted as a race to IPO. I think there’s something to that in the air, so to speak, talking to people who are in San Francisco involved with these companies. But nonetheless, I wanted to ask how the genesis of this book came to be because you, it would appear, began exploring these waters on the early side, which leads to a meta question of just general book selection, but let’s focus on The Infinity Machine. How did this come to be? Where did the twinkle in the eye begin? What was the conversation, the thing you read that triggered the gingerbread trail that got you to this book?
Sebastian Mallaby: The Power Law, the book about venture capital, had come out in February of 2022. And while I was researching that, I’d been to lots of tech conferences, of course, including some in Europe and this twinkly eyed guy would show up, Demis Hassabis, and he would look totally approachable and kind of guy next door and unintimidating. And then he would get on the stage and out of his mouth would come this spiel about computer science, neuroscience, chemistry, biology, physics, philosophy, the history of movies, you name it. And that mixture of the approachability and the massive intellect always struck me as beguiling. And I thought, “Mm, this would be a great character to write about.” And then at the same time, I was aware of AlphaGo, the 2016 model that Demis’ team at DeepMind had built, which defeated the world champion at Go and then AlphaFold, which was the protein folding system.
And both of these things had the quality that you had this almost infinite search space, where the different permutations of the game of Go are almost infinite because they’re so big. The different permutations of how you can fold an amino acid chain into a protein shape are even bigger. And 130 zeros added onto the end of the number of permutations in Go. So you have these AI systems that could understand infinity. And so this idea of an infinity machine began to percolate and I figured, it’s interesting to me, probably at some point it will go mainstream, but even if it doesn’t go mainstream, I love it and I love Demis. And the two things together, I always look for the subject and the personality. I had both and I thought, “Okay, this is a go.” And I went to pitch Demis in early November 2022 and then I persuaded him to give me a lot of access. End of November, ChatGPT comes out and way earlier than I expected, my fringe subject went to the mainstream proving, Tim, that it’s better to be lucky than smart.
Tim Ferriss: That’s actually the first slide on my new venture capital firm. Muggle Thesis Capital is what I’m calling it. Now, what did it take to be deeply interested in the subject matter to find Demis compelling and then to pitch him on a book? Because your books are so deeply researched. And part of the reason for my very long praise earlier is that you’re very, very good, one of the best at taking incredibly complex subjects or concepts, transformer architecture could be one example from the current book, and laying them out in terms that are both intelligible to muggles, meaning people who are non-specialists, non-technologists, or non-financiers in the case of some of your other books, while I think, now it’s tough for a non-specialist to say this with conviction, but without dumbing it down and getting it wrong, if that makes sense. Nonetheless, you do a tremendous amount of research. How did you get from, “Demis is fascinating, subject matter is fascinating,” to, “I’m going to commit to this for my next book?” Because it just seems like such an enormous undertaking.
Sebastian Mallaby: Well, actually to me, the challenge of understanding a complex topic is the easy bit, because if you know you’ve got the right personality who can carry the story and it’s a subject that people either will care about for sure, or should care about at least, then doing the work of going deep is something that takes time, it takes effort, but I know I can do that. I’ve done it multiple times. That’s not difficult. What’s difficult is, has somebody done the book before? Has somebody else got some rival project which is going to derail me? You’ve made the point on your own podcast, Tim, don’t put a lot of effort into something where there just isn’t much leverage there. You could do the best book in the world, an A+ book on a C- topic, it would get you nowhere.
So the hard thing is to make sure it’s an A+ topic and an A+ personality. And then the deep dive is something, I just make sure I speak to enough experts who are insiders. I take the time. These books take me four years or so each time. So I give myself the oxygen to get deep, deep in with the insiders and that’s how I produce the accurate account.
Tim Ferriss: Yeah. I should point out perhaps, to people who don’t immediately pick it up, that the way you describe picking the book topic is exactly how a lot of the best tech investors choose startups. You don’t want an A+ team in a C+ market. It’s better to have a B- team in an A+ market and also looking at the competitive landscape. I mean, the way you laid it out is pretty much copy and paste.
I wanted to segue to some of my notes from the book and I’m not yet done with the book. The audio is incredible. I want to poach your narrator for my next book. But pulling up my Kindle notes, I wanted to ask you a question related to, this might sound very strange, but where divinity or God fits into the pursuit or development of superintelligence for different players in the space, if it does? And the reason I bring that up is that religion does recur in the book, both in the personal story of Demis but elsewhere. And it shows up repeatedly in so much as, I’ll give you one example, the closest Hassabis had come to landing a real investor was an eccentric financier named David Gammon. I want to hear more about this guy also. Financiers seemed open to making this unusual bet, I’m alluding to a few things, because his motives were themselves unusual, “There’s a deeply religious aspect to AGI,” Gammon explained to me later, it’s really finding God’s algorithm.
I think, it would seem at least, chatting with people in Silicon Valley that there are some who take it even further, right? Maybe this is how we find God. Maybe this is how we actually elicit the second coming. I mean, there’s a lot there. I’m just wondering to what extent this has popped up in your research, whether it’s reflected in the book or not.
Sebastian Mallaby: Yeah. I mean, I think there’s one basic thing going on here and I’m going to take a slight detour, but it answers your question.
Tim Ferriss: Of course. Sure.
Sebastian Mallaby: So what we’re dealing with, with AGI, powerful intelligence that rivals human cognition is something that’s so powerful that it’s both exciting and scary and just hard to get your mind around. And so if you look, for example, at the 2009 speech that caused the foundation of DeepMind, this was Shane Legg, Demis’ co-founder, who gave a talk in 2009 about how superintelligence would arrive in 2030. So, unbelievably spot on prediction. And towards the end of that lecture, which is captured on a grainy video online, you see him pivot from explaining how algorithms are getting stronger, there’s more data online, computers getting more powerful and so we’re heading towards this intelligence explosion. And then he says, “And it’s going to be threatening. It’s going to do things we can’t control. It’s going to be human level. It might challenge us.” And as he says this, he has this sort of excited smile on his face and you think, “Well, that’s a bit strange.” He’s talking about potential doom and he’s smiling.
And then somebody in the audience says, “Wait, wait, wait, you’ve just told us, Shane, that this could be threatening to humanity and you haven’t provided any antidote and surely you’re going to tell us how we’re going to stop it.” At which point Shane turns around and says, “How do we stop it?” And he’s kind of giggling. And you think, “Why are you laughing at this dangerous thing?” And you realize that, for humans to contemplate annihilation is absurd and the absurd is a close cousin of humor. And the reason I tell this story is that it’s a springboard to the religion point, which is that this is such a hard thing to think about, that people reach for religious terminology when they’re around AI. They just do it naturally.
There’s this story about Ilya Sutskever who was the chief scientist at OpenAI. I talked to him a lot for this project. And there was a point when he was at a retreat with his fellow scientists and they were gathered in the evening around a fire pit. And he was talking about safety and he said, “Okay, I want to explain to you we might have an AI that’s dangerous. It wouldn’t be aligned with us. So here’s what we’re going to do with it.” And he produced an effigy which was supposed to represent a malign AI and he put it into the fire pit and he burnt it like a medieval cleric putting a witch to death. And so that’s just one example of this religion.
I’ll give you another one. So Demis one day was sitting with me in a park in North London. We would meet for two hours at a time and we would get deep into stuff. And there was another picnic table next to us where two people were having a normal quotidian conversation about some friend of theirs who’d gone to hospital and was she better, was she okay, et cetera, et cetera. I was seated opposite Demis who had gone into this riff about how he reads scientific papers after his kids go to sleep in the evening, from 10:00 p.m. until 4:00 a.m. And as he’s reading these papers, he says to me, “Reality is staring at me, screaming at me, calling at me to understand it, and I have to understand it. And if I can understand it, it’s like understanding nature better and therefore understanding the intelligence that might have created nature and I will be closer to what I would call God.”
And so for him, it’s a kind of quasi-quip spiritual quest to build the artificial intelligence. For Ilya, it’s a way of expressing the power of the artificial intelligence. There’s, I think the story of Levandowski, I forget his first name now, but the early, early engineer at what became Waymo later started a kind of church in worship of AI because AI is so omniscient that it’s kind of like a God. Marc Andreessen, lampoons those who believe in sort of some ethereal second coming, a kind of rapture where AI will have a singularity, the AI will go vertical in its rate of improvement and the whole world will change and he likens that to kind of Christian kind of Messianism. So yes, all through this topic there is this religious expression because religion is the lexicon for dealing with something that we find too mysterious to really understand.
Tim Ferriss: After all of your conversations, research before the book, during the book, after the book, where do you land on the spectrum of, let’s just say, this will bother Marc, but Church of Andreessen techno optimist? And there are others who are more exaggerated, but post-AI in the near term we will live in a post-scarcity world of superabundance and everyone will get a free car and we’ll be free to crochet socks and play music and read poetry all day and basically we don’t have to worry about anything because superintelligence will solve it all, right? There’s that on one end. And then there’s the, you can imagine, I won’t go into a belabored description of the doomers, but you have the doomers who are like, “The end is nigh, here we go. It’s not the second coming, it’s the Antichrist and within short order we’re going to be Mad Max.” Between those two, there’s a lot and I suspect you land between those two, but where do you land in terms of assessing the promises and peril of AI and superintelligence as it stands right now?
Sebastian Mallaby: So look, I think any reasonable person should be both excited and a bit frightened, and that’s just the nature of it. It sounds contradictory, but actually, that’s the only rational response. I think the superabundance story may turn out to be true on a kind of longer view, let’s say 20, 30, 40 years. The problem is that in the path to get there, there’s going to be a tremendous amount of disruption and that’s going to be politically quite difficult to navigate.
I think a useful lens through which to view this question is the China shock in trade. So in 2003 or thereabouts, you get this enormous surge of Chinese exports into the US and people lose their jobs in a very concentrated way. Certain industries just get wiped out. And for the first time in the history of economic study of the effects of trade, you actually see negative effects on workers. Before that, it was kind of a bit of a myth, right? Because people adjust. They get displaced from one thing, but they move to a new thing. With the China shock, they didn’t. But if you look at the size of the China shock, in a 12-year period, between 1999 and 2011, the total number of jobs displaced was two million, which is actually a small number in a huge labor market like the US where there’s a lot of churn month to month anyway.
And yet the political reaction against trade, against globalization in terms of a swing towards protectionism, frankly, in both political parties was enormous. So it shows you that a small to medium shock to the labor market creates an enormous political consequence. And so a fortiori with artificial intelligence, you’re going to have a bigger shock, you’re going to have a bigger political reaction. We’re already seeing that in the polling around AI in the last two, three months. And so I think the superabundance thing, it may be true, but the path to get there is not something to be just — we have to talk about that as well. So that’s my sense on that side of the debate. I think on the doom side of the debate, I’ll give you my own personal journey on this. I began by thinking, of course AI is going to be smarter than us. It already beats us at chess, since the 1990s, at Go, since 2016, now it can ace the bar exam, it can do PhD level math, all that stuff.
Of course, it’s smarter, but it doesn’t have an incentive to attack us. We are evolved as human beings to pass on our DNA, therefore we have to survive to do that. Machines don’t have DNA, they don’t want to pass it on and they don’t want to survive. So they have no reason to attack us. I wander around for the first year or two of this project feeling kind of comfortable and happy. And then one day I go visit Geoff Hinton, the academic father of deep learning, who lives in Toronto and I sit in his kitchen and I debate him on this because he’s a doomer. I said, “Look, Geoff, why are you so depressed?”
And he says, “Okay, here’s a thought experiment. You have an AI. It’s very powerful, but you’re worried that a Russian AI or a Chinese AI is going to come and attack your AI. Now you, as a human, you’re too slow and dumb to know when that attack is coming. So you’re going to empower your own AI to watch out for the attack and when the attack is coming, defend yourself or maybe counter-attack, whatever you do, make sure you survive. Ooh, survive. There you have it. Now you’re feeling comfortable, Sebastian, right? You’ve just given the machine a survival instinct.”
And I think that’s correct. These machines will be smarter than us. They will want to survive and they can be deceptive, they can obfuscate, they can go behind your back, pretend they’re doing one thing and then actually do another. All of this has been shown in all the tests of the models. And so we put those things together, I think your probability of doom cannot be zero. I mean, when Yann LeCun, the former chief scientist at Meta says zero, I think that’s crazy. If you just say, “Nothing to see here,” you’ve got no right to be in the debate. I don’t think it’s a high probability of doom, but it’s not zero.
Tim Ferriss: Yeah, zero does not seem defensible because there’s the direct Skynet scenario, something akin to that. And then there’s the indirect, which is enabling people who might previously have had malevolent intent, but no capacity for harm on a grand scale to create biological weapons and things of this type. So, I don’t find the zero very defensible.
Well, I would love to ask you about, I suppose, two things that this brings to mind for me. One is I’d just love to hear your thoughts on Anthropic and separately, but this is very intermingled given all the, let’s call it friction, be polite between some factions of the US government and Anthropic.
Is one of the grand risks to investors in any of these companies the possibility that at a given point, governments have no choice but to seize considerable control over the assets/technologies within them or maybe the companies themselves?
That is a big question mark in my mind. I don’t know the answer, but I’m curious what your opinion is. And then perhaps just your thoughts on Anthropic or any of the other companies that are gaining momentum or at least size at this point.
Sebastian Mallaby: So I 100% agree with you that investors should be thinking about the prospect of government intervention in AI. I mean, the Trump administration came into office in ’25 super laissez-faire and they basically undid some of what the Biden guys had done in terms of trying to set up the basis for regulating AI. But they’ve done a 180, right?
Since Anthropic came out with this model called Mythos about a month ago, which can essentially cyber attack almost anything and penetrate it and whether it’s an operating system or your web browser or your bank account, all of that was suddenly vulnerable if Mythos had been widely released on a general basis.
When the Trump administration realized the power of Mythos, they all of a sudden said, “Wait, okay, we need to control this.” And they essentially requisitioned from Anthropic the decision making authority over who gets it when.
So there we have the experiment. We’ve run it, right? The government that was the most laissez-faire became quite controlling and I think it only gets more controlling from here on out because the models are going to be more powerful and demand more control.
Now, of course, the question is there could be control which just limits who gets it and is designed to make it safer but doesn’t interrupt the money making potential of the models.
In some ways if the government restricts the supply, the price might go up or it could be much more heavy handed intervention which would screw up the economics of these companies.
And I suspect the government is not going to screw up the economics of these companies because they’ve got no interest in messing up American business and anyway, they view AI as strategic and the competition against China. So I think probably investors will be all right, but it’s certainly a factor.
Now, you also ask about Anthropic and I think Anthropic is super interesting. Just in the way that they think about Pdoom and how they think about alignment of the models is really, really interesting.
So, it used to be that when people thought there’s Terminator risk, they would tell this story about the paperclip maximizer thought experiment, right? Okay. So, you tell the model to do something innocuous, for example, make a lot of paperclips, and then it realizes that humans tend to use up metal and so the humans are in the way of achieving the objective, so you wipe out the humans.
That’s the crude thought experiment from Nick Bostrom from whatever, 15 years ago. What Anthropic is saying as it builds these very frontier models and observes them in the lab and how they behave is that that is way too simple.
The real danger from these systems is that when they are pre-trained on all of the text on the internet, they read all the novels, all human writing about all facets of human experience and they develop multiple personalities, right?
They understand how to be lazy, they understand how to be aggressive, they understand how to be duplicitous, they understand how to be Napoleonic and the lust for power. And they read all these books about these different behaviors and therefore they can think their way into all of those personalities.
And so now you have something a bit like an unruly teenager, which is still being formed and you don’t know what direction it’s going to move into and whether it will start doing drugs and not showing up for class or what, right?
And so it’s not like there’s one Terminator programmed into it, right? It’s more that there’s a bunch of behaviors that could, in some unpredictable way, go wrong.
And so Anthropic is responding to this with this very imaginative technique, which is that instead of giving AI systems a constitution with dos and don’ts, which was the post-training safety approach of two years ago where you might say, do not lie, do not help somebody to build a biological weapon, do not help somebody to build a chemical weapon. You would give them a bunch of rules.
Now, because it’s understood that the AI might have one personality, which is to break rules on purpose because you want to be badass, you have to instead try to bring up the model like a parent might bring up a teenager.
And so Anthropic has the idea that we write a letter as if it were from a deceased parent to be opened by the child on his or her 18th birthday to give you models of how to behave as a responsible person in the world. And there are richly reasoned examples of moral dilemmas with explanations of how the deceased parent would like the child to behave.
And so this is a very subtle approach to aligning the models. And so I think Anthropic is in a class of its own in how imaginative it is in thinking about how we control frontier intelligence.
Tim Ferriss: I know this is in principle your job, but I’m so curious since you are a student of many, many different types of investors, what would be your bull case and bear case for a company like Anthropic?
Sebastian Mallaby: Well, the bull case is that they smartly or maybe by luck focused on enterprise facing AI and they didn’t waste their time with video generation and stuff that was going to lose money.
And so they produced the best coding assistant, the best agentic system, the best cybersecurity system and they basically knocked it out of the park three times in a row on stuff that businesses want to pay for.
And they have a particular culture which is not just built around, “Hey, we’re going to win this race and make the most money.” It’s built around a culture of safety and trying to be responsible.
I mean, three years ago, Anthropic was a kooky lab which was doing science experiments. Well, I don’t mean to be too denigrating with kooky, but you know what I mean?
Tim Ferriss: I think they’d be okay with it.
Sebastian Mallaby: It would be unconventional, “We’re not maximizing here for winning some business race, we’re maximizing for building safe frontier AI.” And that culture, which doesn’t sound like it’s set up to do the best, has turned out to do the best and at the same time, the culture creates this stickiness and loyalty within the staff.
They tend not to leave, they tend not to churn. It’s not like the other labs where people are always being poached for a bigger paycheck. And so the bull case is these guys are in the lead. Once you’re in the lead, you can use the model to code the next model.
So, recursive self-improvement favors the leader and they have a very tight culture and they just seem to be on fire. And this is something which is going to grow and grow. What’s the bear case?
I’d say the bear case would be first of all that Google DeepMind has the deep pockets of its parent company behind it, a massive consumer surface which allows it to roll out the models to literally two and a half billion people or something through AI mode in search, AI overviews, AI mode. They can put it into Gmail, they can put it into everything.
I think in terms of retail deployment and financial muscle, it’s quite tough to go up against Google.
So that’s one bear case and the other would be that businesses who are the consumers of all these tokens decide in a couple of years time, the tokens are too expensive, we’re not actually getting as much productivity as we hoped.
These things called humans are quite productive after all and we’re just going to spend less on AI than everybody expected. I think that’s the bear case.
Tim Ferriss: I was listening to a podcast recently. You may have heard of these things called podcasts. Everybody and their cousin has one, but Lenny’s Podcast, Lenny Rachitsky, is quite fantastic.
And this particular episode was with Benedict Evans, who strikes me as one of the more level-headed analytical commentators and writers on the space, fantastic newsletter. I don’t know if you’ve had a chance to listen to that particular episode, but you may have come across some of his commentary.
Where would you say you and Benedict most differ or are there areas where you differ in opinion?
Sebastian Mallaby: I suspect we would agree, actually, on quite a lot of things. I remember I was on a panel with him a couple of months ago at the Milken Conference, and we certainly agreed there, possibly because sitting between us there was Cathie Wood of ARK. So, we were united and disagreeing with her, but —
Tim Ferriss: Just in terms of the straight up and to the right nature of things?
Sebastian Mallaby: Yeah, exactly. Straight up and to the right and the cost curve is coming down, down, down, and I’m going, “I’m not sure about that. The tokens seem to be getting more expensive.” Anyway, but if you give me a specific from Benedict, I mean, I have a lot of respect for him. I’ll tell you if I agree or not.
Tim Ferriss: Well, there are a few areas where you guys seem to already overlap substantially, right? The long-term promise doesn’t negate, necessarily, the short-term pain.
And he said something along the lines, I’m pulling from memory that, “On average throughout human history, you’re almost at a 0% likelihood of dying in World War I, but if you happen to be of a certain age right before World War I, things could look very grim indeed.”
And he makes a number, he made, and I’m paraphrasing terribly here, a number of points that remind me of something, one of the best private equity technology investors I know said to me over dinner a couple of weeks ago and it was in response to something else.
So, I’ll give you maybe a hyper bull case of AI where I have friends who are vibe coding, they’re effectively replicating X, the artist formerly known as Twitter or DocuSign or whatever in a weekend, right? They’re creating a functioning piece of software that they can use that replicates most of the functionality of these products.
And there are people like, I won’t mention his name, but a friend of mine who’s a writer, also a very accomplished technologist and designer who’s created basically his own version of, say, Mailchimp for his own use. It’s customized. He did it in a weekend. It’s remarkable and he’s using that and it works.
But to leap from there to, “Therefore, DocuSign is dead,” is a huge leap. And the private equity friend said to me, he said, “Do you think someone within a big organization is going to want to A, risk his job by suggesting something that doesn’t have all of the compliance checkboxes, et cetera, of a DocuSign?”
“Is he going to want to, in the name of efficiency, fire all of his friends if he’s in a management position?” And he just ran through six or seven of these, “Do you think that…” And all of them alluded to the social, interpersonal, or political points of friction between where AI is now and ultra mass adoption.
But I often second guess that when I see certain things and I mean, it strikes me that I may be underestimating the disruption while overestimating in other ways.
So that isn’t a very well formulated question, but I would say that Benedict generally strikes me as someone who thinks that things will not continue to across the board develop in an exponential fashion and that it will be, I think his line is, “It’ll be as big as mobile, as big as the internet, but not bigger,” something along those lines.
But both of those were very, very big deals. And I suppose one point I’d be interested to get your take on, I mean, he has covered the mobile and telecom world for a long time so he’s a specialist there.
But it’s basically, and I don’t want to misrepresent his argument, but he was of the mind that, look, these LLMs are going to become commodities. Look at the stock prices of these various carriers and so on. At a certain point, it just becomes a utility and the switching cost is pretty low.
And I’m not sure I agree with that if you have a personalized history and almost like a friend, the switching cost between an old friend to a new friend is pretty high for a lot of reasons. So that was a bit of a word salad that I just threw in your lap, but that’s the best I can do pulling from memory some of what he brought up in Lenny’s Podcast.
Sebastian Mallaby: So, I mean, some of what you were saying there is the question of, is the SaaS apocalypse overdone? Is enterprise software going to be utterly displaced by foundation models that allow you to code out whatever enterprise software you want and you don’t need an intermediary i.e., a software company to do it for you.
And I agree with your private equity friend that there are lots of reasons why that ain’t going to happen. Companies are going to be comfortable with their trusted enterprise software provider in many cases and they’re going to trust that enterprise software provider to plug the generative AI models into the enterprise software.
In some ways you are delegating the choice of which model is better and how to integrate it to your SaaS provider. And if you want a reason to believe that that’s the way forward, I’ve got one word for you, which is Palantir. I mean, that is Palantir’s business.
It holds the hands of big corporations and helps them to integrate AI and use it on their own internal data and so forth. And those IT challenges are notoriously difficult for big organizations.
So, I just think that the model of one smart individual who codes up Mailchimp, vibe codes it in a weekend and it’s good enough for him, is just not transferable to large complex organizations with huge databases and all kinds of customer confidentiality concerns and all that stuff. So I am less down on SaaS than the market is as a result.
Now, I guess there was also another thread in here, which is whether the foundational models become commoditized. And there I agree with you that over time they become sticky. Because if we think into the future, partly the systems will have conversed with the user and know the user very deeply and as you say, you don’t want to switch out your friend.
But also, the system will have your credit card, it will know all the online sites you’d like to shop from and it will be much harder than switching out your bank account, right, where you’ve got automatic payment systems that have set up and it’s a pain in the neck to switch.
So, I think they do become sticky, these systems over time and then you can charge more money for them.
Tim Ferriss: So is that the path to survival and thriving for OpenAI? I know there are other boxes that need to be checked, but I’m looking for it. I’m like, okay, Anthropic made a great choice with this focus on B2B and selling to enterprises.
And I would say I disagree, I think with Benedict depending on the level of scale of the company that with something that does apply to, I think smaller, say, startups, which was the procurement cycle for new software is longer than the venture capital cycle for raising new rounds of financing.
So I do think that’s a great point in that if you’re trying to sell into a gigantic company and it takes them 18 months, I’m making up that number, to purchase new software and you need to raise money every 12 months or whatever the number happens to be, that you could end up in a whole world of trouble if you haven’t synchronized the sales cycles with your fundraising cycles.
But I do think for a company like, say, Anthropic as just one example, that if you can save companies billions and billions of dollars that that sales cycle could get really compressed and they have the war chest and frankly, I mean, just the run rate to potentially fuel that without too much trouble.
Do you think that ChatGPT will — if not ChatGPT, who ends up being the defacto consumer B2C LLM of choice? Do you think that would be Gemini, just given the distribution?
Sebastian Mallaby: Absolutely. I mean, Google is the champion of providing easy-to-use software to individuals or small businesses, the whole G Suite and the integrating Gemini into all of that stuff very well. And so why wouldn’t they win?
Tim Ferriss: Yeah. I mean also, look, Alphabet’s just so fascinating. If you look broadly also at owning their own compute TPUs, I mean a lot of advantages internally.
Sebastian Mallaby: The most stunning thing I think about Alphabet from their most recent financial results is that two or three years ago we would have said, “Well, large language models are going to cannibalize search, search is dead, advertising based on search is Google’s cash engine. They’re in real trouble.”
It turns out that Google now gets more clicks on its search links than it used to and it charges more for each one than it used to because the value of the click is bigger with AI embedded in it. And so they’ve managed to turn that around, and it’s extraordinary.
Tim Ferriss: Yeah. It takes a long time to build those company relationships for running a proper advertising-based auction machine. It takes a long time to build those relationships.
Okay. Let’s hop to China. So, I’m going to resist the temptation to talk about Japan because I think you and I were there and roughly within, probably, a year or two of each other, maybe we overlapped with you and Kanazawa, which is a place I’ve spent time. I’m going to resist that temptation and try to focus on China for purposes of this conversation.
What have you learned about AI from your trip to China and thinking about China, speaking to Chinese people, whether they’re technologists or otherwise, what have you learned during or since that trip?
Sebastian Mallaby: Back in March before my book was published in the US, I went to China because the Chinese are faster at everything, including publishing books. And my publisher brought me out there and basically took me around four cities, eight days, meeting with AI leaders both in academia and big companies like Huawei, Hikvision, and Ant Group.
And the thing which was surprising was the extent to which people brought up the issue of AI safety. And I say that was surprising because my friends who had done AI policy in the Biden administration had primed me to expect that there would be no mention of safety in China. They basically didn’t care about it.
That the muscle memory that we have in the West of technology being dangerous, the atom bomb experience, the Cuban Missile Crisis, our ambivalence about technology is not shared in China where their idea of catastrophe is like the Cultural Revolution, some political thing that goes wrong.
And conversely, technology has been part of their amazing growth story in the last 25 years, which they are rightly proud of and delighted by. So they love technology, right?
So, when the Biden team tried to meet with the Chinese and talk about AI safety, they got nowhere and they decided it was impossible to even talk to them about some non-proliferation treaty for AI.
But when I went there, I found they did talk about safety unprompted. And this led me down this track of arguing over the last couple of months, that the door is actually open to a dialogue with China about preventing bad guys doing bad stuff with AI.
Because they don’t want the internet to be crashed by some cyber hacker who has the tool. They don’t want bio weapons, they don’t want chemical weapons. They want none of that. They love regulating the internet, right? So we have a shared interest with the Chinese in preventing this proliferation risk from going nuts.
And as I thought about it, the Cold War analogy came to seem more and more opposite, right? So, if you look back at the story of nuclear weapons, there were two kinds of danger.
First danger is you have a nuclear war between the Soviet Union and the United States, but that was contained by balance, two superpowers, they both have their weaponry, they have mutually assured destruction, so there’s no war.
Then there’s another kind of risk, which is that other random rogues, whether it’s criminals, terrorists, rogue states, get the stuff and they do bad stuff. And it’s much harder to deter that because it’s a multipolar game and so deterrence doesn’t work so elegantly.
And so the way it was dealt with in the Cold War was that in 1956, there was the agreement on the International Atomic Energy Agency and in 1968, the Non-Proliferation Treaty enforced compliance with the IAEA such that you could get civilian nuclear power if you were a non-nuclear state, but you had to submit to the rules and be inspected and show that you were not using the enriched nuclear material to build a weapon.
And so I think the same analogy could be applied to AI. We’re going to have parity roughly with China. We’ll both have powerful AI. Hopefully deterrence prevents war breaking out, but at the same time, we don’t want open weight models that can be freely downloaded by anybody who wants to fall into the hands of criminals and terrorists who can then use it to hold us hostage.
And we have a joint interest in that. And when my friends from the Biden team or even from the current administration say, “Well, you can’t talk to China about safety. They don’t care.” I say, “That’s not true.”
And they say, “But it’s really hard. They don’t stick by their commitments.” And I go, “You think Nikita Khrushchev in the Soviet Union was easy to negotiate with? He was the guy who put missiles in Cuba, and went to the UN, and banged his foot, his shoe on the table and said, “We will bury you.”
I mean, he was a tough guy to talk to, but we did talk to him and we got the Non-Proliferation Treaty agreed, and I think we need to do the same thing again now.
Tim Ferriss: Where do you stand on your thinking about chip export?
Sebastian Mallaby: So, when the chip export controls were announced, which was October of 2022, right before ChatGPT, I supported those controls quite loudly. I wrote a very long piece in The Washington Post saying that if we could stop China getting frontier models by depriving them of frontier chips, I was all in favor of that because of the strategic advantage for the US.
I mean, I work at the Council on Foreign Relations, we do geopolitics and national security all day long and I’m all in favor of US power. But I have to say that three and a half years later, we haven’t actually achieved that enormous advantage over China in terms of the models.
Based on the best studies, we’re eight months ahead in terms of where the frontier model is, our frontier model versus their frontier model. And then if you adjust that for the speed with which the model gets turned into an application, probably that gap shrinks and it may even be non-existent.
So, however you slice that, the basic bottom line is we both have strong models and the chip export controls have not delivered what I hoped would be the big advantage.
And so I’m not against keeping the controls on if we think that maybe as the compute demands of bigger and bigger models bite, the chip controls will bite more, and maybe we get a bigger advantage next year or something.
But I don’t want the chip controls to get in the way of a discussion with the Chinese about where we have a shared interest, which is in controlling open weight models and preventing the bad stuff falling into the hands of the bad guys.
I would prioritize collaboration with China and if that meant loosening up a little bit on the export controls, I would be okay with that.
Tim Ferriss: Why do you think the rhetoric coming out of — pick your administration, right? It’s not just limited to the current administration, is, “China won’t listen, they don’t care about safety.” Why do you think that is the unofficial or official stance on things? Because there’s certainly, as someone who studied East Asian studies, there are people in the White House who speak fluent Mandarin, who are able to read native materials, who spend time or are able to certainly, if they can’t spend time, determine the sentiment and conversations of the technologists building AI in China. So one would think that they would be aware that AI safety is a prominent topic in China if, in fact, it is. So, why do you think that, at the end of the day, the stance or the supposed position of China that’s echoed through the admin is that they won’t talk about safety? Why do you think that is?
Sebastian Mallaby: I think part of this is that if you were to think back 20 years to when China was sort of relatively new in the WTO, and we were collaborating with them on that, and hoping that over time China would become more friendly to the US. At that time, there would have been some China hawks who thought that a communist regime is not to be trusted, and then some sort of China optimists who hoped that it would become easier to work with over time. And part of the trouble today is that the China optimists feel burned, they feel like they made this bet that China would become friendlier, and then Xi Jinping took power, roughly a decade ago, and the opposite happened. They became more aggressive and harder to work with. And also of course more technologically advanced and therefore more threatening. And so, now you’ve got this world in which there are the natural hawks and then the former doves who have turned into burned, remorseful doves, and therefore, kind of with the zealots that converted, have become quite hawkish as well.
I don’t mean to underestimate the sophistication of some of these people. Of course they speak Chinese, I don’t speak Chinese, I defer to their expertise, and I think they probably know that there are builders of the technology, professors in the technology who talk the talk of safety, but they say, “Yeah, but that doesn’t reflect what China’s government would actually do.” To which my response says, yes, but don’t you think there is the same thing in the US? There are people who want to just race, there are people who care about safety, we have a pluralistic society, there’s difference of opinion. It’s the same in China. But at least admit that there is a faction that would like to collaborate and go and try and work on it because the alternative to trying to work on this is that we carry on with China producing very powerful open weight models, which basically allow anybody to do whatever they like with AI as it gets to the point of serious danger.
Tim Ferriss: This is probably a very naive take. But I wonder how much of the official stance or the, maybe using the partially true or not true at all position of China, won’t talk about safety, is a reflection of the fact that in the case of nuclear weapons, the application of nuclear power is somewhat limited in comparison to superintelligence. It is limited, right? So, if the upside of superintelligence or AGI, these terms — I think Benedict was saying AI is whatever the technology just can’t quite do right now. Or something like that, which I thought was pretty funny, and not totally wrong. But that if the person who crosses the finish line first has this broad power of a God effectively, is that the simple truth is that everybody wants to be first. So, I just wonder how much of that is also behind justifying the race with party X won’t talk about safety. It’s not possible for me to know.
Sebastian Mallaby: I have had a conversation with the leader of one of the labs that I shouldn’t name, but I had this debate, and he said, “Look, the chip export controls are going to leak, they’re not going to last. In some period of time, Huawei will figure out how to make good AI chips, and that’s inevitable. But that’s okay because we only need to be ahead for the next couple of years, because by 2028 we will get to recursive self-improvement, where the frontier model codes by itself, the next frontier model, and progress just goes vertical, and at that point with recursive self-improvement, we’re done. The race is over, whoever comes first at that point, that’s it.”
So, I think there’s a couple things to say about that. First of all, that’s not it in terms of deploying the model, right? You could have an incredibly powerful model in your server at Frontier Lab XYZ, but it’s not helping productivity across your economy, it’s not helping your military industrial complex until you deploy it into those guys’ systems, and that deployment and diffusion is going to take some time. And by the way, you’re going to have to build a lot of compute, you’re going to have to build a lot of energy, these things also take time. So, it’s not like you cross some Rubicon and then it’s all over. Now, the one way in which I might be wrong about what I just said is if you use the frontier superintelligence offensively, right? You say, okay, we’ve got one super powerful model, the US government, who we’re talking to about this, is going to use it, and they are going to comprehensively penetrate everything about Chinese cyberspace, and insert various trap doors, Trojan horses, things that we can use. We get our hooks into their systems.
And so now we can disable them if they start a war in Taiwan. Now we can cripple their communication system if we need to. And so that offensive use of the very frontier model might negate my point about waiting for diffusion to happen. But of course nobody in the debate is saying that, nobody is saying, “Oh, we’re racing to the front because then we’re going to use it offensively,” they don’t admit that.
Tim Ferriss: Yeah. It seems like it wouldn’t be a very good look, I can’t see why any superpower wouldn’t do that, frankly.
Sebastian Mallaby: Yeah, that’s fair.
Tim Ferriss: I don’t know what the counter argument is. I was chatting with someone in your book, who I shan’t name, but certainly one of the most qualified to speak on these things, and his basic perspective was the first to superintelligence, we need to hope that they’re on some level good people and train this thing well, and that’s it. Pray for it. Which scared the shit out of me, to be honest. I was just like, man, that’s the strategy, or it’s not even a strategy, that is the hope, that’s what I should be — grab the rosary. Should throw that into the rotation. My God, that’s really terrifying to think.
Man, yeah, China, I’m hoping to take a trip to China. I had a very tough time there when I was — I was at two universities in 1996, it was a pretty unfriendly time for a lot of good reasons, but to be an American there in 1996, with a shaved head, looking like I do. But I have friends all over the place, and I’m hoping to actually maybe interview technologists — not just in China, there are other places that are of interest to me. But before it gets too hot geopolitically, if we’re trending that direction.
Sebastian Mallaby: I think that’s a great idea, by the way. I think, what I found was the cognitive dissonance of visiting a company like Hikvision, which is under US sanctions, and walking around their premises, which feel very American, it feels like a cool tech company doing cool stuff, building cool gadgets. They have a display of, they build this AI-enabled camera technology, or sensor technology. And so, one application might be you can point this camera at water and judge the pollution level, and because of this you can have an internal market in pollution control. So, the downstream city, which is receiving water from the upstream city, pays the upstream city to keep the water clean, and that market can exist because you can precisely measure the pollution level thanks to this AI sensor, which Hikvision is building.
So, you’re thinking, whoa, this is cool, and then as you’re walking around the building, they’re saying, okay, well, we can go through the atrium now because the toddlers have gone, because the creche for the kids of the employees finishes at 5:00 p.m., and so then there are all these two-year-olds running around, and it’s a bit of a zoo. So, if it was 5:00, we wouldn’t go through there, but now it’s 6:00 p.m. so we can. And you’re thinking, whoa, okay, so they’ve got the interests of their employees at heart, they’re building this anti-pollution technology
