Vinay Rao | Deploying AI safety from Anthropic to Google
Vinay has spent twenty years in security at the world's largest technology companies. Most recently he served as Head of Safeguards at Anthropic, where he led key initiatives to ensure the safe deployment of frontier models. He is now VP Engineering for Cloud AI at Google, heading up model safety and quality.
A deeply technical profile with a Stanford PhD and a background in statistical modelling, Vinay walks us through how he first got into security at Google, solving click fraud with statistics. Airbnb, and Stripe came next, each with an entirely different set of safety challenges.
Together we unpack how AI is moving out of the experimental phase and into strategic territory, with direct consequences for product, risk and business model. We get into specifics, from jailbreaks that get a model to reveal its own training data, to prompt injection in agents running inside a company, to scams that play out across several platforms so that none of them ever sees the whole attack.
Vinay explains how safety and security have converged as agents operate inside the organisation. We talk about what this means for a startup facing the same threats as a large enterprise without any of the structure, and how much of your security stack is actually worth building in house for the next generation of AI systems.
We close by widening the lens. What role governments should play over technology they do not control, where national security ends and mass surveillance begins, and what happens when state actors stop shaping narratives and start going after companies directly.
A conversation about how security will determine how far AI goes and how fast, with someone who has spent two decades on that frontier.
00:00:00 - 00:04:09 - Vinay Rao, 20 years inside trust and safety
00:04:09 - 00:06:27 - The three kinds of bad actors on any platform
00:06:27 - 00:08:11 - How AI collapses the safety team into a few people
00:08:11 - 00:10:29 - Why safety and security can no longer sit apart
00:10:29 - 00:17:53 - Scams and where accountability breaks down
00:17:53 - 00:24:52 - Open source safety tools for teams too small to build their own
00:24:52 - 00:30:00 - Deciding what to buy, build or borrow, and the open weights danger
00:30:00 - 00:35:24 - The open stack and the model race ahead
00:35:24 - 00:40:11 - Moats and where the value really accrues
00:40:11 - 00:47:08 - Geopolitics, sovereign data and trusting governments over labs
00:47:08 - 00:54:10 - State actors, coordinated attacks across platforms and fragmented defence
Transcript:
Samaipata host
So today we have with us Vinay Rao for a very special edition of our podcast, recorded live in our founder summit in Mallorca. For those of you that haven't had the chance to talk to him, he has over 20 years of experience in AI and specifically leading trust and safety teams in companies like YouTube, Stripe, Airbnb, Anthropic, Google. I mean, too many to name them all. And this is the Samaipata Podcast. So Vinay, welcome. Most of the room knows you for your work on AI safety. But the first decade of your career, it was something completely different, right? So you studied engineering at IIT in Bombay. And then you did a PhD in Stanford in applied mechanics. Then years of modeling, fluid dynamics, and semiconductors. How did that engineer end up running trust at YouTube and safety at Anthropic?
Vinay Rao
Yeah, firstly, thank you for having me. It is definitely a privilege. Going all the way back to my IIT days, and this was... I joined IIT in 1987. So many of the things we're doing today did not exist in their current form. So it clearly was not planned. So I would say a lot of things that I ended up doing are just sheer luck, right? So going through school and going to the graduate program was almost kind of autopilot. I felt like it was expected. So I just went and did it. Then went to England to work just because I got an opportunity to work in Europe. The big shift, I think, was when I went to England, it was 1997. And the dotcom boom took off in America. And I thought, oh, that's terrible luck. Like all my friends are making so much money. Then I said, OK, I should participate in it. So in 2000, a friend started a company. I came back to America. And in 2001, it all crashed. I said, oh, that is terrible luck. I must be really unlucky. But turns out, pretty much, most of the things that ended up happening are because of luck. One of the biggest changes was in 2006, Google had a class action lawsuit for click fraud. So advertisers claiming that they were paying for clicks they shouldn't. And click fraud, unlike many other kinds of safety issues, there is no ground truth. You can't look at a single click and say if it is good or bad. So you had to do this in aggregate, and you needed statistics. And in my semiconductor modeling work, I was doing statistics. So it just happened to be the kind of person that they needed, and I got in. And that was transformational for my career. So since then, I have been in the safety space. As Google grew, I ended up working on ads integrity, publisher integrity. Google introduced a payment platform. I worked on payment fraud. Each of these things just helped me set up for the next stage. Then went to Airbnb handling yet newer kinds of abuse offline. Property damage and physical safety. Came to Stripe to build the first credit risk model. And so by the time AI took off, I was just one of those guys in the right place, right time, and right-ish skills. And was just fortunate to be able to get into an Anthropic at a very, early stage.
Samaipata host
It's like you've watched every platform shift from the past 25 years or so from the trust and safety perspective. Like click fraud, you mentioned, payment fraud. Now all the safety challenges are related to AI. What is the pattern you keep seeing throughout these cycles?
Vinay Rao
So, you can... This can be grouped in many ways. And they're all valid ways of grouping because it's just a whole bunch of uncoordinated activities that happen. One way to think about it that has helped me is looking at who are the actors doing the bad things and why are they doing it. So I think broadly you can categorize it as in three forms. Number one is where the actor does something and it is completely accidental. You record something horrific like an accident and upload it on YouTube. Or you're testing what does Claude know and you ask it to generate the recipe for a bomb. These are accidental. And then second is people who are idealistic, who believe what they are saying and doing is right, but it does not necessarily align with the policies of the platform. So ethical hackers, for example, is one. A lot of violent terrorism is the other. They believe they are freedom fighters and they will believe that their methods are okay, even though the rest of the world disagrees. And these are idealism based and so they will continue to do. These groups typically are not very adversarial. They will continue to just push their propaganda. And the third one is the hardest ones to handle. They're malicious actors. They're in it for profit. They make money sometimes. They don't make money sometimes. But the longer they stay on the platform, the more likely is that they will create enough ROI with some schemes that keeps them going for other schemes. These are people who are running scams, who are doing fraud, who launch botnets. And in many cases, like you will see, even with AI, very systematic misuse of AI across a large number of the users.
Samaipata host
If you had to point to one category where AI changes the game so much as to modify the roadmap of a team, what would that be?
Vinay Rao
So I think it depends on which part of the world you're talking about. If I were to look at the safety world, which is the part that I'm most familiar with, historically, we've seen the safety teams in YouTube or Airbnb or any other place was that you had an engineering team, you have a product management team, you have a policy team, you have analyst teams and so on. Now, you don't need all of that because you can have an ML engineer who can iterate on the policy of your platform to be able to optimize both the policy and the detection in one go. Similarly, you don't need an enforcement agent to go and reach out and talk to an analyst to be able to run the SQL query because AI can do that for you. And so you can start by looking at a single entity and then go do your own group by with like, give me all the other people who are doing the same thing or give me everything else this user has done in the last 30 days and so on, and you can run those queries, etc. So I think the biggest move is this kind of ability to bring all of the problems in one place and enable the small number of people within the group who know how abuse happens and give them all the tools they need. They are an analyst, an ML developer, an engineer, and an abuse expert all in one, and that suddenly increases their impact.
Samaipata host
It's interesting because for most of the internet era, safety and security teams have been quite separate as organizations, separate functions, different type of tools, different organizations and mindsets within the company. But with AI and within the agentic era, does that separation still make sense?
Vinay Rao
So I will posit that it does not. And this is the following is the reason why. So if you go back to the standard internet era, the simple definition, this is just like very broad sweep, is safety protected the users and security protected the enterprise. So all the data, the code, etc. And in the agentic world, what ends up happening is you have AI inside the enterprise doing a bunch of things, interacting with the outside, whether it is through the customers or external data sources, etc. And I'll give you two examples where these two come together. Let's say you have a customer service agent, and that customer service agent has likely speaker guardrails. And there is a jailbreak that you can use to have the model regurgitate its training data. This is well documented. Detecting a jailbreak is a safety problem. Protecting training data is a security problem. Or in another example, if you have a mechanism to get a prompt injection into an agent that is working inside an enterprise, get it to write some piece of malicious code, and then execute it. That again, detecting prompt injections is a safety part. Preventing malicious code from being executed inside your corporate network is the security part. So these two are coming together. And time after time, when we were working on writing threat models, we saw this, the same pattern come up more and keeping the two teams separate was much worse than actually having them come closer and work together.
Audience
On the security side, sorry, the safety side, where do you see the accountability ending with the company and then also being taken by the end user themselves? Because scammers are getting so sophisticated now. It feels like inevitable, but my 80-year-old mother is going to get taken for a ride anytime soon. Or at the worst end of the spectrum, a young child and so forth, all those vulnerable. But at the end of the day, where does the accountability lie with the platform and where does it lie with the user, do you think?
Vinay Rao
I think the way the scams work let's talk about support scams we'll come back to child safety I think child safety is actually a more tractable problem. So the way scams work, it's actually hard to pin down security. Because it's usually a multiplatform attack. So the DM you may receive on LinkedIn and the conversation goes off to Signal or WhatsApp and then the first few trades happen on KrakenD or Coinbase. And then the last trades happen on this fake exchange, but the money is being transferred into that exchange from your bank. So now you have LinkedIn, you know whatever WhatsApp, KrakenD, and your bank. Somehow all of them need to understand that there are scams happening and these are the signs of the scams and this is how I need to protect. So putting accountability in any one accountability in anyone's place is hard. Banks are doing some amount of work. The US, again, I know the most about the US. US's banks are becoming aware. Now, every time I send money it says like, "Are you sure it's not a scam?" And do you know the user and the person that is receiving it, etc. But from what I hear, Singapore is doing this really well. They have this kind of scam task force and this is people who are crypto experts or banking experts, platform experts, etc. All sitting in one room and looking around and asking "Hey is this happening?" So that is I think the pinning down accountability is just hard. With regard to child safety, I think children are less platform they don't hop platforms as much. And we have done a profound service to the kids by not holding platforms accountable for harm to children. The reason airlines are so much safer than pretty much any form of transport was because every time an air crash happened the airline was responsible for it And that liability forced them to make it safe. But every time a child has something bad happen Facebook is not paying for it most of the time it is not even reported and I think that has been a problem that has compounded over time. And now children are spending all their time online, a lot of the time online. And even in cases that you could think of as closed calls, but not necessarily direct harm have an additive impact on the child. And I think that in my mind, the first two lawsuits that were both Facebook and YouTube were held responsible for in California might actually open the floodgates where responsibility will shift to the accountability shifts to the platform. And I think that is right. I also think I'm now speaking from the outside of my house, trying to solve it with like saying, "Oh, we have put age verification, all problems gone." I think it's also false. Kids are very smart. They will get around age verification They will be back on the platform. The platform needs to make structural changes, like how it responds to kids.
Samaipata host
So, moving to idiosyncratic examples of AI safety challenges, I think one that comes up very often is the pig butchering scam, which probably comprises a lot of the aspects of AI safety. Maybe people in the room have not followed this. I think it would be interesting for you to share, especially how it shows novel risks versus acceleration of pre-existing ones.
Vinay Rao
Correct. A pig butchering scam, it is an unfortunate name because the victim is the pig and the victim is fattened as part of the scam and eventually sacrificed. I prefer the term "financial grooming", but it's not particularly...
Samaipata host
We're trying to explain to be honest.
Vinay Rao
Yeah, okay. Particularly popular. So, it has basically four steps. Step number one is the attacker spreads their network wide to be able to find this candidate that they're eventually going to scam. And this is basically the financial mechanics of spam. You reach out to a thousand people and one or two fall for it. And typically, this reach out has some kind of urgency or some kind of hook. So, it could be an unpaid bill or it could be very popularly a romance scam. You're reaching out to someone who's looking for someone to be romantic with. Stage two is trust building, where these two now are talking generally on an encrypted platform like Signal or Telegram or even WhatsApp now. And they exchange a lot of conversations. So, this is fairly deep, fairly involved and you're exchanging your interests, your vacations, your desires, etc. And in this time, the attacker is trying to understand what is the next hook that they can get into you to be able to extract money from you. And a lot of the examples that I have read about is wanting to figure out how to invest money, how to make a lot of money with that investment. And so, that's the third stage where the first investment is done. In the example that I know, and which I think if you haven't already heard it, the Scam Inc podcast on The Economist, it's really nice. So, it's generally a crypto investment where they will invest a small amount of money under the direction of the attacker and end up making money. And so, they'll invest some more, they'll make some more money. And this goes on for many weeks, even months. And then they may go in for the big move where they create this urgency saying, "Oh, there is this big thing happening. You really need to now get into it. This is guaranteed many, percent return." And then people move in a large amount of money. Sometimes they even borrow money and invest in it. And by the time they realize this is a scam, it's all gone. They're unable to get back in touch with the attacker. They realize that the crypto exchange that they were trading on was fake. And there is no contact number or anything, and they're completely at a loss. And this is basically the crypto scam. And it is surprisingly high ROI as soon as they get in. Recently, there was a bust where it was a $15 billion dollar recovery of cryptocurrency.
Samaipata host
So we were discussing the other day or yesterday at lunch with some of the founders about how the requirements for security, safety, and trust are becoming more and more relevant for our portfolio companies, but in general for most AI startups. And I'd like to ask you, what are learnings you think from enterprise safety teams that these startups can replicate? And they try to work with SOC to give some level of degree of trust, but these requirements are not enough today.
Vinay Rao
So yeah. And this is a great question. This truly is the challenge of today's founders and people who are building companies, which is AI can bring enormous advantages to your business in the way it can do things. AI can be unpredictable in many ways, and sometimes it works well, sometimes it goes catastrophically wrong. So how do you build the trust that is needed? I think the use of things like standard compliance and standard regulations is valuable, but it is not complete. It will provide a base level of trust to your end users, but anyone within the company will probably think, "Is this complete if it is not where it is it?" So the mechanisms that we need to come up with are still very much in flux. So within the organization, putting sufficiently good guardrails, and guardrails by this side, I don't mean complex detections and such, but basically limiting the kind of tasks that AI can do, the kind of data that it can access, the kind of code that it can run, the kind of changes that it can make, like does AI, your coding agent allow you... Do you allow your coding agent to submit code into your GitHub repository, etc.? So each of these are relatively lightweight mechanisms. They're not too different from what you were doing with your human employees, but it's slightly different. They're probably more time bound, they're probably more contextual. And I think those are good starting points to start building trust within yourself that I can now safely launch this. And once you can, then you can talk about it externally for your users.
Samaipata host
I think this is actually a very good segway to your work at Roost, since a lot of founders in the room and watching are structurally constrained at the early stage, and then some resources and safety is not always like the first priority versus just surviving and growing, etc. I think it opens up a very interesting debate of buying versus building, also in the context of fast moving landscape at the moment. So we'd love to learn a little bit more about your work at Roost and the open source approach and how that can benefit a lot of early stage companies?
Vinay Rao
Yeah, So Roost stands for robust open online safety tools. It was built on three theses. Thesis number one was safety tooling requires engineering, and that is generally not in line with the product you're building. So it is taking people away. And so that could potentially have some kind of, you know, if not minimal staffing, less staffing than safety teams generally need. Number two is with the onset of AI, there will be many small or even like in using Sam Altman's words, like two engineer companies that can quickly get up to high revenue levels. And these companies simply won't have the ability to build their own safety systems. And thesis number three is because of the availability of AI, particularly open weights models to the attackers, the kind of risks and attacks that they will see will also change really fast. So one company cannot fix and make changes to their safety systems across this entire range of attacks. So that's where open source comes in. And the advantage of open source is you have now a tool that is available for you. It is neither built nor buy. It is free. You have the general advantages of open source. That is any bug that was found or any vulnerability that was found was fixed and you get the fix for free. And number three is as it becomes more robust against different kinds of attacks different organizations are seeing, you get the robustness for free as well. So that was the thesis. And we started this about a year ago, launched it at the Paris Action Summit.
Samaipata host
What type of tools are currently available for startups?
Vinay Rao
Yeah, so we started with two of the most common tools. So there was a period of about six months where we went and talked to a variety of trust and safety teams. And the two tools we have is one is a very standard first tool that every team needs. That is it takes a JSON blob, which could contain a piece of content that has been posted on your platform. It could be an event like a login or a post or anything. And you could pretty much put anything in it. You can attach arbitrary classification and detection information. And now you have enriched the data. So you now know that this user has a reputation X. This post is y% likely to be bad, etc. Now you have enriched it and you can then build a whole bunch of business rules. If the score is greater than this, do this. If not, send it for manual review. And for human review, we have the human review tool. Most recently, as of last week, we even launched a blog about now how you can build agents on top of this to be able to now use AI in different forms to help do the human reviews. The other one is a more complex. What you can think of as a rules engine and safety teams have used it, security teams have used it for their detection and response capabilities where you have this basically think of it as a stream of events coming through your logs. And you can then decide to do what you want with it based on fairly complex descriptive logic. So you can say, should I allow this login or not? Should I allow this refund or not? And each of these things you can write reasonably complex logic and you can have it run in real time.
Samaipata host
So pulling up from the specific case of safety to other areas in AI, what do you think is the right framework to think about buy versus build versus open source and in different layers of the AI stack? What are you seeing around in the market?
Vinay Rao
So I think there are two parts to this, right? I think some of the choices that the safety teams make can be used by other parts of the team, right? So each of these defenses that we build are essentially risk-based choices. And if you think the risk is so much, you have a binary choice, like do this or don't do this, right? And many of us know, person A does not have access to this data, so they can never run this query. Or person B does not have access to this code base, they can never clone that part of the repo. So from there to completely probabilistic, how likely is this supposed to be a scam? And initially, if you see scam is generally a scale-based approach, the first few you see, you can let through and it's fine. And what you don't want is the full scale of the scam happen. So I think once you understand the risk-based approach that safety and security teams will generally put in place, you can then say, "Okay, what tools and what systems do I need to be able to manage this risk?" It is entirely possible that you already have the tools and you have the systems and you don't need it. But if you don't, I would definitely say, look at open source first. Open source has some profound advantages in terms of its transparency, its ability to fix bugs, its ability to funnel the improvements that are coming from various places. So as a random example, if you look at the top contributions to Kubernetes, it actually comes from pretty much every major cloud provider. But Kubernetes has zero employees and it does not pay any salary at all. And this is the best, what you can think of as the optimal system for open source where every person in Google or Azure or AWS who's making a change or an improvement to Kubernetes knows that person is changing the tool for everyone else, but everyone else is also changing the tool for them. So it is this 1x5, 1x10 ROI in terms of improvement to the tool and that works really well in open source. And if you don't have it, of course, you should consider the overall ROI of buy versus build.
Samaipata host
Are there scenarios where the open source model doesn't make as much sense?
Vinay Rao
Yes, that's a good question. One of the areas that I am beginning to get concerned with is the enormous benefits that open source, open weights models that are increasingly available and are increasingly capable. And that's great because inference costs and API costs are high. And if you have highly capable open weights models, you can then decide how you inference them and save money. On the other hand, it is also well established that these models can be... They're launched with a bunch of safety training, which happens as part of the fine tuning stage, but you can also make them unlearn their safety training. And once you do that, what you have is a highly capable model that will not refuse any kind of harmful request. And wide availability of that would just make all the platforms risky because now attackers can use this highly capable model, whether it is coding, whether it is planning a big attack. And with increasingly reasoning models, they can also reason what the safety systems look like in teams so they can then create adversarial models to just redeem you. And that I think is concerning.
Samaipata host
This is the reason why Mythos has been kept in a little niche club.
Vinay Rao
That's very much the case. Yeah. And I think it's a very wise decision. The world of cybersecurity, I don't know very much. Is it a symmetric problem versus asymmetric?
Symmetric... What I mean by that is, the model's ability to find bugs as expensive as its ability to fix the bugs? If it is symmetric, then holding back makes some sense. So figure out all the major or critical infrastructure areas, find the bugs and fix them. If it is not, then I think we will have increasing pressure because companies... And Anthropic will not be the last company to build a model of this size. Others are, I'm sure, building these models. They will be under increasing pressure to launch it. These are incredibly capable models. They can charge 3X the price from tokens. But once it is out, it will be very hard for us to then roll back the compromises that they highlight.
Samaipata host
Up until this point, AI has been built in public to a great extent. I would say all the founding papers are out there, one after the other. And even before the awareness was widespread, all the data was there. But are you saying looking forward, do you see a change in tenancy?
Vinay Rao
I think that's a good question. I would actually say, firstly, AI has been built in public and AI has been more open as a technology than pretty much any other technologies we've built before. The transformer based neural nets was open, RLHF was in the open. Even some of the inference software like VLLM is open, SGLang is open, CUDA is not, but XLA, which is accelerated linear algebra from Google, is open. So a lot of AI, PyTorch is open. A lot of AI and their stack is open. The magic sauce really is in the art and science of that. How do you... What is the right mix for pre-training? What is the architecture you use? What's the right mix for post-training, etc. I do think just by throwing random world data in the models won't make them particularly clever. And that has been figured out and so on. So yes, you can take all the data in Common Crawl and train a model. I don't think it will be that clever. These companies have now a historical institutional knowledge about what works and what doesn't. What are the exact RL environments that you need to place them in and to be able to build certain skills in the models. So I do think there is a little bit of defense there. And secondly, training is very expensive. It's a lot cheaper to fine-tune an existing model to do something harmful than to pre-train a model from the start. So I think the existing framework is reasonably robust against building absolutely new models. But the open-weight models being fine-tuned to remove their safety features, I think continues to be a vulnerability and a problem.
Samaipata host
You spent years inside a frontier lab, right? And I'd like to get your view on how you see the model race in the coming years. Do you still think it's going to be an oligopoly where they keep taking some headway over the others and back and forth?
Vinay Rao
I think if I were to think about this in two different ways, and I'll talk briefly about them, one is the science of model building and number two is the economics of model building. So the science of model building, I think, is that the transformer-based neural nets and especially the decoder models, so these generative models, turned out to be enormously successful. It was unreasonably successful. And particularly when associated with scale, the larger the model, the more clever it is and emergent capabilities, it has been so successful that one almost feels like this is the end of it, like AI is LLM. I actually don't think that is the case. I think there are profound areas of improvement that have not been explored yet. AI does not have an inherent concept of the world. It does not have an inherent concept of predictive decision making, saying, "If I do this, will happen. So I have these choices. And based on those outcomes, I will make the right choice." It doesn't do that. Yann LeCun talks about it as part of his Joint Embedding Predictive Architecture. So these, I think, are still exploratory areas. And science, as it develops, will potentially change the nature of AI itself. And if that happens, it will just bring more players into the mix. It's not just the Google's, Anthropic's, and OpenAI's, and these companies could potentially now push the frontiers from a different direction, towards a different direction, just with the science. Number two is the economics of this. Even anyone who's now using AI to a large extent within their own companies are beginning to see, "Okay, this does cost a fair amount of money." And the new models are, if anything, going towards a more expensive proposition than cheaper, which is kind of unintuitive, because one would think scale makes the cost less, but GPT 5.5 is more expensive than 5.4. Claude has always been more expensive. So it is still going in that direction. So one can argue that the sheer market pressure to build alternatives that are cheaper to win back market share could bring in new players as well, where you have similarly clever models, but in terms of parameter count, they're not as big. So they may need additional data sources to refer to, but they have the intelligence. They may not have all of the inbuilt knowledge, and that could bring other players in. So in both of these scenarios, I think others will come in, and we are just kind of waiting for the developments to happen in that space.
Samaipata host
I think it's also probably useful to throw into that reflection the elements of defensibility, right, of what all of these players are building. So I wonder if you see any moats being built in any of these layers, whether it is the model of the data, even the trust posture, branding, or more commercial elements that can really tilt the scale in favor of one of our other players.
Vinay Rao
Yeah, the more I think about it, I come increasingly to the conclusion that no one really has a mode right now. The moat is just the expense. That's it, the price, the cost. But from a purely technology point of view, there isn't one, right? So you can see the, like I've already said, the transformer-based architecture is fairly common, fairly commoditized. The nature of RL that you do is fairly common, fairly commoditized. It's not like any of these companies are doing magically different things, even in the world of safety, not just the safety systems we talk about in place today. That is things like fine tuning, things like building audio classifiers, or even some of the newer things like classifier probes where you have a value head trained inside the model. They're also fairly generic. There isn't one thing that people are doing that is significantly different and unable to replicate. Same with hardware. It's a surprising number of chip manufacturers now in place, right? So there was Nvidia with its GPUs. Google has its TPUs. Amazon has its own now. And many other smaller companies are now coming up with GPUs. Interconnect also getting to be more common, right? So whether it's NVLink or ICI from Google and so on, it's growing. And as is the inference software itself, like the part where you do the matrix multiplication, Google has even open source theirs. And so it just makes it a lot easier for people to reach. So I don't think there is an existing moat here that might be in attaching these capabilities to a business problem where you are adding something that is completely complementary, meaning something that LLMs just cannot do. So that will largely be within an application, but not in the LLM itself.
Samaipata host
So basically, you're saying that the model layer isn't going to capture or accrue any of the value in the coming years. It's all going to go to the application layer.
Vinay Rao
I think that... So a model layer will do important things. It's just that no single model provider can do things that others can't replicate in six months.
Samaipata host
So where does that drive the value capture or does it go towards the end client?
Vinay Rao
Yeah, so the value capture is people who are building systems on top of these models and they become fairly agnostic to what models they use. So one way of thinking about this is models are very good at understanding language. So semantic understanding. But they're not very good at system level understanding and figuring out what is optimal for your organization, especially if it involves complex computations and your IP when it comes to some kind of clever algorithm or data manipulation, etc. And if anyone who builds an application that takes advantage of the semantic skills of the model with that additional IP that you build, that is what becomes harder to replicate. The ability to combine the language complexity, which historically has been hard to do in machine learning, like an LP was like this idea that promised more than what it delivered. LLMs are delivering it. But the other thing that we've always been good at, like why does Google's ad system work better than everyone else? Why does Facebook's or TikTok's recommendation system work better than everything else? That is not because of semantic understanding. It is some fundamental algorithmic stuff that they have. And if they can combine that with semantic understanding, that's where the cap value capture is.
Samaipata host
So model layer will capture some value in the application layer by being able to do additional things. Exactly. But they won't be able to capture value from the model itself.
Vinay Rao
Purely from the model, I don't see how that can happen. And signs have shown that that's largely true. All the models give or take a few points on most of the benchmarks that we care about.
Audience
Yes, so we've talked about the tension between proprietary and open source models. As an Indian who has worked in the US and Europe, how do you see the geopolitical considerations here, especially since those regions have their own data and privacy frameworks, and how does that change the future?
Vinay Rao
Yeah, I think this is a great question. With regard to particularly sovereign, government-related information data or data that is very highly specific to a country, there are two ways we can go. One is the AI model developers, people who are offering closed source models, are able to create the right framework where the data stays within the country, the inference happens elsewhere, but there is no recording of that information. And that would be a very sort of geopolitically friendly way of approaching it. The alternative is for these countries to rely on open source models till they build their own models. Those investments are quite high. Even if India gets it, there are many countries that are below India in terms of available capital for this that may struggle to get there. So we probably have an intermediate solution where at least some of the closed source model offers are able to provide that and then eventually get to a place where countries have the right level of AI available. But the larger question remains when there are parts of the world that still don't have clean water. And that's because clean water is a little more expensive than they can afford. Many places don't have electricity because that's a little more expensive than what they can afford. And AI is that much more expensive than what they can afford. And we don't want to create a permanent underclass in this. And we need to find a way to solve it.
Audience
Well, I understand that in the safety department you pursue to defend ethics within the platform. I think we need to take two things into account. On the one part, the tension between Anthropic and the government, wanting to take over control of these kind of technologies, or the close relationship of OpenAI with the government, and how people use these kind of models so that they almost obey it and take them as a fundamental truth. How do you cope with that from the safety department? Are you involved with that? Or how do you think it can shape the future of society? Because I think you play a huge role there and it's kind of fearful.
Vinay Rao
Yeah, so let me separate. Safety departments actually don't deal with the ethics of the model, meaning the classic ethics question, "Do I sacrifice one person for the betterment of five people?" etc. Those are deeply philosophical questions. And at least in Anthropic, we actually had a philosopher to help decide that. was not a safety question. But the safety teams do look at some of the more systematic population-level problems. Things like persistent bias, etc. We were testing for that to make sure that the models are relatively well calibrated to be able to serve the full diversity of population of the world in general. The question that you are asking with regard to the government, it really comes down to relative trust, I think. So I'm fairly old, and I come from a fairly conservative view that the government that is elected knows what is best for the country than an unelected CEO of a small company. And the government will have better processes in place to evaluate if their use of a certain technology is right or wrong. And so in that regard, relative trust modulo that, I would expect the government to do the right thing. And this whole conversation has become so fraught that you have to pick sides. I don't know why we should pick sides. We should agree on values and then say, "Are the values maintained?" "Who cares?" I don't expect Anthropic to ensure there is no mass surveillance happening across the board. All they're saying is, "Don't do mass surveillance with my tool." But the government should say, "We don't do mass surveillance because that's how we build an open society where people can talk freely." So I think that's kind of become very fraught in the way it has been portrayed. And I think we're actually asking the wrong questions. Should we support Anthropic or should we support the government? No, we should support human rights. And that is the more important problem. And if that doesn't work, go to the elections and vote someone who supports human rights. So at least that's my take. It has nothing to do with AI at all.
Audience
Absolutely. Yeah. It seems like a beautiful world, though.
Vinay Rao
It is. It is. In a way, AI is probably causing the uncertainty because of this. It is the belief that it can do everything and then in many ways it can do things that are better than what humans can do. And again, coming from a safety world, my whole reason for existence is AI can go wrong and AI can be misused. So I come from a very skeptical point of view. And AI, more than anything else, even in the way you do threat modeling, how you do threat modeling with humans, you need to treat AI like a human in this. AI is a platform. Chat GPT is a platform. AI is an intermediary, like how in an agent or something you use LLM to summarize something and then you make a decision based on that. And in the not-too-distant future, AI will be an actor where it will have agency and a misaligned AI is a problematic actor. So I think thinking about it that way, you then realize that AI causes a change in the way we think about problems and that makes us a little less capable of wrapping our heads. So we take mental shortcuts to say, "Yeah, it's good or AI is bad, but AI is a little bit of everything."
Audience
Yeah, just I think it's interesting that you were talking about the Anthropic versus the government, because my question was when you mentioned malicious actors, I was thinking about state actors, right? Because you didn't mention you said ideology, etc. But right now the states are using AI. So first to defend to develop offensive capabilities. So when you see in Ukraine, So when you see Ukraine, there's this autonomous drones, etc. But also to manipulate the information. So this is the two risk I see, the one that basically is used for defense, but also for offense. And then in a way to manipulate the information, we're mentioning YouTube example, etc. But we know that, for example, in the recent election in Bulgaria, Romania, Moldova, Russia was spraying a lot of misinformation, right, to try to trick the election. So I'm not sure that democracy these days are winning. So there's more and more actors that are going to act offensively and defensively. And you need also to protect your own defense sector against these offensive capabilities that rogue states are building.
Vinay Rao
Yeah, Very good point. I think we also, some of the AI companies have put out their threat intel reports where the cyber capabilities are also being used by state actors to go and probe various kinds of companies, you know, corporate defenses and seeing if they can get into it and get data. And this actually raises a more interesting question. So AI model in itself is, especially if you think of it as a chatbot, is a one on one conversation. Like, what the AI tells you has actually no audience other than you in general. But it can then be amplified by, you know, putting it on any of the social media platforms or any amplification platform. And now, so this was one of my profound, like, learnings in anthropic is that I see a statement that seems vaguely political and, but not against any of my policies. And I say, Oh, I can't actually stop the model from doing this. But then I can also see in the back of my head, if this goes viral, this is a problem. But it can't go viral on anthropic, but it'll go viral, let's say on Instagram, but I have no control over Instagram. So the advantage that your state actors have is that they can use different platforms for different tasks and they move very smoothly across all of them. But we as defenders, we are so siloed that I cannot tell or it's very hard for me to have Instagram respond to an alert that I send them. And also, I don't know how many people I need to send to because before it actually happens, I don't know which platform it's going to go viral on. So from a structural perspective, the safety teams, the security teams need to come together a lot better and share intel. And that will at least level the ground. And if we want to stop the state level like astroturfing campaigns, etc, we have to be able to do that. Otherwise, it's kind of a one way or an imbalance that it's hard to fight.
Audience
Hi, I agree with you that the security risks have increased for companies because of the capabilities of AI. Should we modify the fees that the companies face when there are data breaches. Should we modify the policies, no? Because for example, as a startup founder, let's say, currently I don't expect a state actor or a malicious actor to attack me, to be targeted, let's say, in an individual way. But if the capabilities are there to attack thousands of companies at the same time, there is more risk. So how does that balance? Because as we mentioned before, we sold invest, but that's an investment, that's a trade-off, but the risk is higher. And then the fees, especially in Europe, the fees are really high if there are a huge data breach. So how do you see that balance of investment, risk, policies?
Vinay Rao
Yeah, I think so. Anyone now building systems needs to have a tighter tolerance to security based vulnerabilities than before without question. Anthropic produced a report called GTG 1002. It was how the Chinese state actors used Claude code to breach into multiple companies. And from the perspective of Mythos, it was a brute force attack. It was not a particularly clever attack. It was not actually using Claude's ability to read code. But what it was doing is Claude's ability to organize the attack. So they let it loose on like 30 companies, and then it tried various systems. And eventually it found a, I think, cross-site scripting attack or something and got in. Once it got in, it was extremely organized in testing every possible resource internally. It generated its own payloads. It generated its own queries. It figured out which databases within the company were unprotected or did not have role-based access controls and then started running queries against them and then started exfiltrating data. So it tried on 30 companies and in four of them, it basically exfiltrated data. And it did all of this in like three days. And the companies did not even recognize it till Anthropic highlighted it to them saying, "Hey, you were a part of an attack." What this highlights is what I said earlier, the security, obscurity-based security will not work. So which means you have to close all the attacks. What I also hope is this knowledge is now transformed into some companies that will do exhaustive security checks for you, and that will also become a bit of a commodity. And that will help you build those defenses. So anyone who's very good at finding security vulnerabilities does not always have to steal it. They can just make a business out of it.
Samaipata host
Well, thanks Vinay for taking the stage today. I can speak for myself, but I'm sure for all of us here, we'll be rooting for you in your coming endeavors.
Vinay Rao
Thank you.
Samaipata host
Thank you so much, Vinay.
Vinay Rao
Thank you very much.
