BUILD EVALS

“Because we put this mission above any individual product line, we're able to make very fast decisions that cut across the entire org and execute on them in a unified way. Mission means that teams are willing to make sacrifices that hurt their own goals and their own KRs in service of Anthropic's goals. I've never seen that at a company of our scale.”

“Building 10 great evals is important for helping the team quantify what the goal is and what their progress towards it is and what they're missing. I think evals is this like underappreciated thing that more PMs and more engineers should be working on. It varies a lot based on the exact feature, but features such as memory benefit a lot from it.”

“I show that thesis to so many candidates. Like, I have it as my Chrome autofill at this point. And here's why it's not just a wrapper. I actually think more and more people understand it's not just a wrapper. And by the way, like, in the beginning, parts of what we build are wrappers on functionality that works well. But that's not really the product that that drives revenue.”

“If an automation doesn't work 100% of the time, it's not really an automation. I would encourage listeners to put in that time to scope some automation that you really want to get to 100%, put in the elbow grease to teach Claude your preferences and give it feedback so that it can improve its skill. There's just not much value in a 95% there automation.”

“As code becomes much cheaper to write, the thing that becomes more valuable is deciding what to write. Like what is the right UX for this feature? What is the most delightful way that a user can experience it? It takes a lot of care and taste to figure out, okay, which of these is worth building and what is the right way to build it? That skill set can come from any background, but I think that's the most important thing.”

“Building 10 great evals is important for helping the team quantify what the goal is and what their progress towards it is and what they're missing. I think evals is this like underappreciated thing that more PMs and more engineers should be working on. It varies a lot based on the exact feature, but features such as memory benefit a lot from it.”

“I think there's definitely a lot of use cases where you don't want, like, a full coding agent with a compute run time. And also, you want it to be, like, more tightly permissioned. MCP inherently has a really strong permission model. Like, all you can do is call the tools. MCP is just, like, the dumb simple thing that works and it that is pretty good.”

“The timelines for a lot of our product features have gone down from six months to one month and sometimes to even one day. We want to remove every single barrier to shipping things. We want to make sure every single person on the team feels empowered to take their idea from just an idea to like out in the world in less than a week, sometimes even in a day.”

“I think one thing that's becoming more clear is I think the coding agents are the kernel VGI. Everything is a coding agent. I think that's one one direction. And then, yeah, the exciting thing about that is your agent can bootstrap its own software and capabilities and actually debug and maintain them.”

“It's pretty important to build products that don't necessarily work yet, so that you know, okay, what is missing for this product to work? And then with the newest model, you can just swap it in to the prototype you've already made and see, okay, does this new model close that gap? You want to be ready when that model jump happens.”

“It's pretty important to build products that don't necessarily work yet, so that you know, okay, what is missing for this product to work? And then with the newest model, you can just swap it in to the prototype you've already made and see, okay, does this new model close that gap? You want to be ready when that model jump happens.”

“People really like that Claude's low ego, and so if you tell it, hey, you did this thing wrong, it's like truly sorry. It's like, oh, shoot, thanks for telling me, let me fix it, let's work together. I think part of what makes a great coworker is this positivity, this bias towards action, and this ability to give you earnest feedback, not just agreeing with every single thing that you say.”

“The timelines for a lot of our product features have gone down from six months to one month and sometimes to even one day. We want to remove every single barrier to shipping things. We want to make sure every single person on the team feels empowered to take their idea from just an idea to like out in the world in less than a week, sometimes even in a day.”

“If an automation doesn't work 100% of the time, it's not really an automation. I would encourage listeners to put in that time to scope some automation that you really want to get to 100%, put in the elbow grease to teach Claude your preferences and give it feedback so that it can improve its skill. There's just not much value in a 95% there automation.”

“As code becomes much cheaper to write, the thing that becomes more valuable is deciding what to write. Like what is the right UX for this feature? What is the most delightful way that a user can experience it? It takes a lot of care and taste to figure out, okay, which of these is worth building and what is the right way to build it? That skill set can come from any background, but I think that's the most important thing.”

“Because we put this mission above any individual product line, we're able to make very fast decisions that cut across the entire org and execute on them in a unified way. Mission means that teams are willing to make sacrifices that hurt their own goals and their own KRs in service of Anthropic's goals. I've never seen that at a company of our scale.”

“We did partnerships at both Amphorpe and OpenAI at different times to try to at the time, the when we first tried, there wasn't even a constant of, like, tools yet. We designed our own, like, tool calling framework. And then we tried to fine tune the models to to use it over multiple turns. And because it didn't work well out of the box, I think yeah. The models are just too dumb and the context thing was also way too short.”

“As a PM, there should be less emphasis on making sure that you're aligning your multi-quarter roadmaps with your partner teams and more emphasis on, okay, how can we figure out the fastest way to get something out the door? I think the PMs who do the best on AI native products are the ones who can figure out how can I shorten the time from having this idea to actually getting the product in the hands of users.”

“As a PM, there should be less emphasis on making sure that you're aligning your multi-quarter roadmaps with your partner teams and more emphasis on, okay, how can we figure out the fastest way to get something out the door? I think the PMs who do the best on AI native products are the ones who can figure out how can I shorten the time from having this idea to actually getting the product in the hands of users.”

“People really like that Claude's low ego, and so if you tell it, hey, you did this thing wrong, it's like truly sorry. It's like, oh, shoot, thanks for telling me, let me fix it, let's work together. I think part of what makes a great coworker is this positivity, this bias towards action, and this ability to give you earnest feedback, not just agreeing with every single thing that you say.”

“It was definitely super exciting for me because it's probably the fourth or fifth time that we rebuilt that. We've been building this since 2022. It was even right when we got access to G P G 4 in late twenty twenty two. Let's make an agent that I mean, oh, we used the word assistant at the time. There wasn't really a word agent yet.”