Will Fithian, associate professor of statistics at UC Berkeley, had never researched artificial intelligence before 2022. Back then, he thought the AI tools on the market were limited and didn’t pose risks to the public.
Then OpenAI released ChatGPT and his attitude changed. While AI models weren’t new, ChatGPT was the first publicly-available generative AI product that suggested just how powerful and potentially harmful a technology it could be.
Since then, disturbing stories have surfaced about chatbots attempting to seduce users, engaging in blackmail, even abetting suicides.
ChatGPT has gotten more powerful with each version and competitors have come out with comparable products, including Microsoft’s Copilot, Anthropic’s Claude and Google’s Gemini, which UC Berkeley made available to all students earlier this year.
These developments deeply concern Fithian, who says he can no longer imagine his daughter’s future given how quickly AI is progressing. In response, he started Berkeley AI Risk, a reading group and speaker series, six months ago with philosophy professor Wes Holliday. Over the summer, the reading group hosted weekly discussions of scholarly papers examining the relative dangers of AI.
The group has since amassed over 1,000 subscribers, with the involvement of professors in philosophy, statistics, applied math, anthropology, as well as computer science and data science—a range of disciplines Fithian said he was surprised were “open to thinking hard about these topics.” The group began hosting speakers in the fall, covering topics like AI governance and the early effects of AI on employment.
California asked Fithian how he views AI risk and what should be done about it.
This interview has been edited for length and clarity.
What have you been hearing from faculty who engaged in the reading group?
The dominant conversation among faculty is just how AI is hitting higher education like a train. So, for example, in statistics, data science, computer science, we worry about take-home assignments not being as meaningful anymore, because students can use AI products to write code for them or help them solve problems.
In my favorite class in college, we would have one problem set every week, and I would spend almost all of Saturday sitting and staring at these problems, feeling like I was making no progress. Then I’d wake up on Sunday, finally having some ideas about where to start.
That experience, I think, more than any other, prepared me for a life of doing research.
But now, chatbots can cut time on assignments. For example, you can prompt AI to write an essay. And if learning how to write essays is not your priority as a student, you might feel more inclined to use AI to help.
And I know the students are so busy that they probably feel like they’re being punished, right? And punishing themselves if they don’t use AI to at least help them structure an essay or brainstorm.
It’s hard for faculty to say that that’s against the rules, but those elements of the assignment are some of the most challenging and valuable. For me, when I had writing assignments in college, the hardest thing wasn’t writing, but trying to figure out where to start and how to start.
Based on your discussions in the reading group, do you think AI is currently an existential threat to society?
The AI that we have now doesn’t feel like an existential threat. But it’s rapidly getting more powerful. We don’t know whether it’s going to keep getting more powerful at the same rate that it has been, but if it does I think there are a number of existential threats that it can pose.
Much is said about the risks of, say, rogue AIs, or the risks of misuse of AIs. I don’t discount those risks, but what worries me the most is just what is going to happen to the fabric of society when people are not useful to each other anymore.
If you look at, for example, Open AI’s business mission, they say that they want to automate the most economically valuable work. And there are other AI companies that say much the same thing. I don’t know how long it will take for them to do that, but it seems very plausible that it would happen within the next decade.
People in the AI industry often say it will happen a lot faster. I don’t know whether they are overhyping things. But nowadays, when you listen to technical people argue about AI progress, it’s often taken as the pessimistic view that it’ll take a whole 10 years to have AI that is human level.
Even if it’s 15 years, even if it’s 20 years, I think we have a lot of preparation to do.
It’s hard to understand why people who think that there is a 10 to 20 percent chance of human extinction would think that we should go ahead the way that we’re going.
Given that an artificial general intelligence could be here soon, why isn’t there greater alarm?
I think we have a very strong presumption that people who are building technology can just build whatever they can, and if there are negative consequences, we’ll deal with it later. I think that presumption makes sense for many technologies. But I don’t think that makes sense for AI, and I think that members of the public should exercise agency in how such a socially and politically and economically transformative technology will be built. And also on what timeframe it will be built, and whether it should be built at all.
I think that we’re kind of reflexively treating it as a technocratic issue, but I don’t think that is concordant with the level of risk that both experts and members of the public generally believe is there. In a poll that I saw recently, more than 50 percent of Americans said that they believe it’s at least somewhat likely that AI is going to destroy humanity. When you say that, it sounds kooky. But people have also polled AI researchers, and I think roughly 50 percent of them gave a 10 percent chance that we’ll have a human extinction or a similar existential catastrophe.
So, the public’s fear is justified?
The heads of the top four AI frontier labs in the world are giving humanity roughly Russian roulette odds to survive the technology that they’re building. If someone wants to build a nuclear power plant and they tell the regulator the chance that it’s going to melt down was 10 to 20 percent, they would ask you to come back with a better plan, right? The actual risk has to be much lower than that for you to build it.
They also just throw these numbers out as their personal estimates. They don’t actually have any justification for why it couldn’t be higher. The people who are building these AI models don’t understand how they work. They understand how to build it, or really, how to grow it, but the entity that’s actually created, they don’t understand how it operates. And so it will do things that they don’t understand.
Some of these examples are funny. This Microsoft AI model, a couple years ago, would suddenly fly into fits of rage and threaten its users with blackmail, even though it didn’t have any information to blackmail with.
It sounds like you can’t just extract unwanted behaviors from the large language models.
That’s right. There’s a big technical area of AI unlearning, where they try to get models to forget certain dangerous information. Ideally, you want your model to just refuse if somebody asks it to help, say, synthesize a bioweapon. But it’s hard to do that reliably because you can’t just code in the rule “refuse when somebody asks you to help them build a bioweapon.” You have to kind of train it, so you show it many examples of it refusing to build a bioweapon.
But people who are clever about writing prompts, they can get it to sort of migrate in a high dimensional space to some other persona that’s perfectly happy to help you build a bioweapon. This is called model jail breaking.
What worries you most about how AI is affecting society?
Even as chatbot products, AI can be very harmful to individual people, but as they become more powerful, they’re going to take on more responsibility, and the product form factor will be different. So you might have AIs commanding military units, or you could have AIs running companies, AIs engaged in running political campaigns …. And the ethical quandaries that you find in areas like that are so much more difficult than the ones you encounter with chatbots.
Another thing that’s worrying is the concentration of power. Who is going to be in charge of these AI models?
Also, what is it going to do to individual human empowerment? If we can’t support ourselves by earning a living wage, it could be that the political system will reorient itself to give a generous universal basic income to everyone. Or that might not happen.
I think that the level of risk that regular people, that experts, that leaders of companies, assign to the present course that our society is taking is way out of proportion to what I think any reasonable person would believe is a sane risk to take.
I can understand why people who feel that there is no existential risk would want to go ahead, but it’s hard to understand why people who think that there is a 10 to 20 percent chance of human extinction would think that we should go ahead the way that we’re going.
So what should be done?
It doesn’t have to be all or nothing, that we either have the current, all-out race, or that we stop and never build. It could make sense to pause indefinitely, but at the very least, I think it makes sense to pause until we have some confidence that we can build AI without the current level of risk.
But it’s very difficult for rival countries like the US and China, especially, to trust each other to implement these sorts of surgical, technocratic regulations.
The political system is really starting to take notice of what’s happening, and I think that that’s also likely to change rapidly over the next few years. I think it’ll become a much more salient issue. And so it wouldn’t surprise me if several years from now, there’s a strong movement to slow down the development of AI, or have more public control over how AI is developed.
But by that time, the amount of leveraged investment could be such that if we moved to a more reasonable way of doing things, it could cause a market crash. Or the companies could be too big to fail. And that will make it much harder.
So do people need to start marching in the streets?
I think protesting probably would help. I think writing letters to their congressmen would help. Following political discussions and becoming engaged helps.
What I would like people at Berkeley to do is bring their scholarly expertise to bear on the public discussion about AI in society.
Frontier AI companies are presently racing as fast as they can to build as powerful AI as they can, despite the fact that they don’t understand how to control current AI, and certainly have no way of understanding how they’re going to control the more powerful AI they’re trying to build.
This situation poses severe risks to society. Regular people recognize this. AI experts recognize this. Leaders of frontier labs recognize this. It follows that there needs to be democratic decision-making processes for the big questions about whether we are going to build this, when, on what timeframe, and how.
There currently isn’t any mechanism for that kind of democratic input, and I think that’s urgently needed.
