Lightspeedlsvp.com · 18 September 2026

In December 2025, we led Raindrop’s seed round on a simple thesis: as agents become the predominant form of software, observability has to fundamentally change. The company predicted that traditional evals would give way to production monitoring that surfaces the failures humans — or even traditional monitoring tools — would miss. Today, we’re proud to double down on that thesis and this team as part of their Series A bringing total funding to $50 million. Over the past year, demand for production monitoring has only grown, with agents being deployed into industries like defense, healthcare, and financial services. Research from METR shows that the length of tasks frontier AI agents can complete autonomously with 50% reliability has roughly doubled every seven months since 2019. This means that bad behavior is no longer just an inconvenience, but a liability. Agents don’t fail like traditional software. There’s no stack trace or 500 error. An agent fails by doing the wrong thing convincingly, at scale, until someone happens to notice. We believe Raindrop was built for this moment: its product re

Tony Dinhx.com · 18 September 2026

Need a new set of evals for models like Jev

Sidebar.ioearendil.com · 15 September 2026 · not counted as a party

LLMs have become almost perfect at generating code, but that isn’t the end of the story. Just because the code is formally correct doesn’t mean that it is not introducing unnecessary abstractions, creating duplicates, or just making bad decisions overall. This is not a groundbreaking observation, most people who have vibe-coded a project, have realized that each additional feature can sometimes lead to an explosion of lines of code (LOC). This results in a loss of human agency, because in projects that are adding millions of LOC per month, it is hard for humans to keep up.1 Some people might say that that is not an issue at all, because they trust their agents to deal with it. I have bad news for you, agents can't really deal with the slop either. Coming from a physics background, I always had an experimental/quantitative approach to solving problems. When I started at Earendil, with the task of figuring out how to measure code sloppiness, my natural instinct was to first take a deep dive into the literature and then check what other companies were doing. To be frank, with the exception of a few insightful research papers, I was disappointed at how “v

Hugging Face bloghuggingface.co · 15 September 2026

Your agent works in rehearsal, but during the live demo, it takes a different path and fails the same task. That is embarrassing onstage. In production, it is a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, such as reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper. Most benchmarks hide this variability behind an average. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions. But it succeeded in all five runs for only 53.0% of tasks — a 24.4-point consistency gap. Most benchmarks report the first number. We built a way to measure the second — and improve it. In an earlier post, we introduced ALTK-Evolve — a system that turns an agent's own past trajectories into reusable guidelines, distilled automatically and injected back at inference time. It measurably improves task success, but those results only asked the average-case question too. This post introduces consistency guidelines, a new guideline type in altk-evolve built on top of a diagnostic tool we call th

Greg Kamradt, ARC Prize presidentx.com · 7 September 2026

To be clear - Astra building tools is fair game

Hamel Husainx.com · 4 September 2026 · 3 posts

I love how Teresa is merging product discovery with evals. I think its a powerful combination! I started learning about product discovery b/c of Teresa and I think its a great skill for any engineer as it helps you focus on what to build. Highly recommend checking out her work.

Product Talk (Teresa Torres)x.com · 3 September 2026 · 7 posts

AI evals have been the "it" skill for product teams for over a year. I've even called evals a new discovery habit. But I still meet product teams who only have a vague idea of what evals are. And it's not their fault. Most of the writing on this topic is intended for engineers or just isn't specific enough. I recently created an in-depth eval guide to explain what evals are and why product teams can and should create them. I did my best to make it practical, hands-on, and easy to follow. AI evals (short for evaluations) are methods for measuring whether an AI product or workflow is performing well. Evals give teams confidence that their AI applications are doing what they expect them to do.

Perplexityjobs.ashbyhq.com · 2 September 2026

In 2026, we launched Computer, the defining product for the new era of agentic AI. We’ve scaled beyond the millions of people using Perplexity every day for research, shopping, investing and curiosity into a new paradigm of using AI to transform knowledge into action. Product Managers at Perplexity work closely with design and engineering, focusing on the product systems and experiences that help more people discover, adopt, and get lasting value from Perplexity. The ideal candidate should have strong communication and writing skills, the ability to envision innovative technology, and a willingness to reimagine work to be more efficient and creative. Ideal candidates combine product intuition with analytical rigor: in the application, please highlight the growth problems, experiments, or loops you are most proud of building. What you’ll do - Work with select SMB customers to build and test frontier agent capabilities that run their revenue-generating workflows. - Prioritize the highest-value workflows to automate and build the skills, evals, and agent systems that make them reliable and performant. - Turn customer learnings into product requirements for

Meta engineeringengineering.fb.com · 2 September 2026

We’ve built an AI agent that acts as a secondary expert for a given domain, making deep specialist knowledge readily available and preserved for anyone in an organization to access, share, and build upon. This is not a typical domain-specific agent. Its novelty comes from integrating two layers:  A structured, auditable knowledge architecture separates what the agent knows from how it reasons.  A self-improvement loop then compiles expert feedback into verified, regression-tested updates without model retraining.  Together, these two layers turn one-off expert corrections into permanent, compounding institutional memory, and the pattern is designed to generalize to other domains governed by retrievable text rather than model weights. This system is saving domain subject matter experts (SME)s at Meta substantial time, allowing them to focus more on the work where their knowledge matters most. Many large organizations have the same problem when it comes to specialist knowledge. While some of it is written down in the form of models, playbooks, checklists, and frameworks, the most valuable specialist knowledge

r/LLMDevsreddit.com · 2 September 2026 · 6 threads · not counted as a party

https://preview.redd.it/5cnahgrri3nh1.png?width=1282&format=png&auto=webp&s=1a4af80d66754b8410f0af1337bf356d4e657e96 I’ve been trying to compare agent harnesses like Claude Code, Codex, Hermes, OpenClaw and others, and kept running into the same problem: most comparisons change too many vari

r/AI_Agentsreddit.com · 31 August 2026 · not counted as a party

I’ve been researching agentic AI platforms for production use in 2026, and one thing became pretty clear: **“Can build an AI agent” ≠ “Can run AI agents in production.”** There are now dozens of frameworks, agent builders, automation platforms, and enterprise AI platforms. So I narrowed the evaluation dow

Sarah Drasner, Chromex.com · 31 August 2026 · not counted as a party

🏣 Thank you all for coming to the WebMCP Office Hours! I mentioned a few resources. We've created guides for: - Site builders: https://t.co/JookKJaIJX - Agent builders: https://t.co/szfS2QrmIl - Agentic workflows: https://t.co/sHyvE0dBJa - Evals: https://t.co/eibTVeX8zt Here's a demo site with a lot of links like the spec, repo and demos: https://t.co/4Q9kSY90st

github.comgithub.com · 27 August 2026 · not counted as a party

AI Engineer Notebooks – free, framework-free RAG/agents/evals on Colab Man, I could have used something like this last year. Upon review, it appears that fumbling my way through the dark with Claude and Google led to the same place, in nearly all cases. However, this is all written by Claude — it has too many em-dashes to not be, does it not? So, maybe that's why we ended up in the same places. Do

DeepMind blogdeepmind.google · 27 August 2026

August 27, 2026 Responsibility & SafetyWilliam Isaac, Sol Messing and Kristian LumBuilding trust in proprietary model benchmarks using cryptographically secure environmentsImagine a student is set to take a high-stakes exam. If they accidentally peek at the test questions in advance, achieving a perfect score is influenced by this knowledge, making it a meaningless accomplishment. To truly measure what they know, they must have no visibility of the test questions until it's time to take the exam. That is the exact challenge the industry faces when evaluating advanced AI models. If a model has already seen the test questions - a problem known as benchmark contamination - the results can only be trusted to an extent.Today, we’re introducing the world’s first double-blind evaluation of a proprietary, frontier class AI model, which keeps external evaluations confined to a cryptographic “box” where they can’t be used by models later to optimize performance ahead of testing. We're partnering with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, to test a Gemini Flash Lite model against confidential benchmarks in a

LangChain bloglangchain.com · 26 August 2026 · 4 posts

Key Links:Technical documentationVideo walkthroughEvaluation is the process of continuously improving your LLM application. This requires a way to measure your application’s performance. LLM applications often produce outputs in natural language, which are difficult to judge using hard-coded rules. For example, attributes like conciseness or correctness relative to a reference output are difficult to express as typical unit tests. Using an “LLM-as-a-Judge” is a popular way to grade natural language outputs from LLM applications. This involves passing the generated output (and other information) to a separate LLM and asking it to judge the output. Although this has proven useful in several contexts, it raises an interesting problem: you now have to do another round of prompt engineering to make sure the LLM-as-a-Judge is performing well.LangSmith presents a novel solution to this rising problem. LangSmith evaluators now feature “self-improvement” whereby human corrections to LLM-as-a-Judge outputs are stored as few-shot examples, which are then fed back into the prompt in future iterations.💡The net impact is that it is easier to create

Vercel changelogvercel.com · 25 August 2026 · not counted as a party

Agents increasingly write TypeScript programs to coordinate tools and process their results. Once those programs touch real applications, some steps require authentication, while others need human approval.Executing that code with eval gives it the same access as the application around it, including its secrets and internal services, and leaves no durable way to pause at those boundaries.Today, we're releasing the Run SDK, a package for executing untrusted JavaScript and TypeScript without giving it direct access to your application or system. Applications expose narrow host functions and can interrupt execution for authentication or human-in-the-loop approval. The program resumes after a decision without repeating completed work.pnpm add runCopy link to headingA small interface to the hostThe Run SDK evaluates JavaScript or type-stripped TypeScript in a fresh QuickJS context inside a worker thread, with no direct route to Node.js or the network.The application exposes selected operations through hostFunctions. These are regular functions that become callable globals inside the sandbox:import { run } from 'run';const result = await r

GitHubgithub.blog · 25 August 2026

These are the lessons we learned evaluating LLMs for real-world secret scanning. August 25, 2026 | 12 minutes Share: A language model can perform well on a clean benchmark and still struggle with the cases that matter in production. Benchmarks and curated datasets are useful when prototyping an LLM-based system. They help teams compare models, test an initial prompt, and determine whether an idea is technically plausible. But as a system moves closer to production, the evaluation problem changes. Real inputs are often ambiguous. Labels may be inconsistent. Important context may be missing or truncated. The evaluation set may not reflect the production distribution. Edge cases that rarely appear in benchmarks can become common sources of failure. Even when offline metrics improve, those results may not translate cleanly into production behavior. We encountered these challenges while evaluating an LLM-based system designed to reduce false positives in GitHub secret scanning. Secret scanning identifies credentials such as tokens and keys that may have

PostHogposthog.com · 24 August 2026

Part of my job as a product marketer at PostHog is keeping tabs on what every other company in the data space is talking about. So I read a lot of industry reports, and surrendered my email address to a bunch of different mailing lists. Here's what I learned so you don't have to do the same.Everyone agrees on the problem their customers are facing: Product development is stalling because AI ambition is outrunning data readiness, trust, and governance. But, most vendors in the data space are specialized, so each is describing the problem from inside their own slice of the stack. The pipeline vendor sees a pipeline problem. The governance vendor sees a governance problem.Without naming it, every report I read was describing the same fix – a context warehouse. Yes, I know how that sounds; the PostHog PMM thinks the answer is PostHog. Fair. But I didn't run these numbers. My competitors did, and published them. I'm the first to read all of them, saw the pattern, and say why that adds up to a context warehouse.The average company now runs 106 SaaS apps, each one throwing off its own data.BetterCloud, SaaS statistics, 2025An agent is only as good as the data it can r

Arize blogarize.com · 24 August 2026 · 4 posts

TL;DR: A skill is just an agent, a prompt running in a harness, so you can test a change to it exactly the way you test any other agent. This post walks through how we do that for the arize-instrumentation skill: a golden dataset of real apps to instrument, a sandboxed experiment that stops the agent cheating its way to a good score, and trace-level evaluators that grade each run. By the end you’ll have a repeatable way to answer “did that change actually make the skill better, faster, and cheaper,” backed by a real merged PR and the numbers behind it. You changed a skill, tightened the instructions, cut a redundant block, reworded the step that kept tripping the agent up, and now it feels better. But does it feel better because it is better, or because you just spent an hour staring at it and want to be done? That question should sound familiar because it’s the same one you ask every time you touch a prompt or swap a model. You run the agent a few times by hand, the output looks fine, and you ship on vibes. Those vibes are the problem, and with a skill it’s especially easy to convince yourself you’re safe because a skill looks lik

Smartsheetjob-boards.greenhouse.io · 20 August 2026

15+ years experience as a product manager for a software product, B2B or B2C cloud experience preferred. 15+ years experience as a people manager. Experience developing and shipping AI features or AI-powered products, with deep familiarity with generative AI, agentic systems, and responsible AI practices. Experience working with UX and engineering teams; deep commitment to customers and team collaboration. Passionate drive and extraordinary ability to communicate with customers, colleagues, and executives. Passion for web-based business software. Ability to evaluate and discuss technical concepts. Experience working with data analytics and visualization.

Stack Overflow blogstackoverflow.blog · 18 August 2026

Ryan welcomes Suneet Malhotra, Senior Manager of Test Engineering at Motorola Solutions, to chat about building end-to-end agentic SDLC pipelines using MCPs, using Cohen’s kappa to evaluate multiple LLMs-as-judges, and how you can improve requirements by shifting QA left through a specification enrichment stage immediately after the design phase.

Nielsen Norman Groupnngroup.com · 14 August 2026

Summary:  One output cannot establish how well an AI system performs. Evaluate with multiple representative inputs, repeated runs, and confidence intervals. Suppose you ask an AI customer-service system the same question several times: “Can I return an opened product after 30 days?” One answer may explain the policy accurately. Another may omit an important exception. A third may confidently promise a refund that the customer is not entitled to receive. Which of these answers represents the system? The answer is: all of them, taken together — and none of them, taken in isolation. Yet teams often evaluate AI systems by running them once, inspecting the result, and drawing conclusions about what AI can do or how it should be used. This is not because they are lazy or careless. It is because decades of deterministic software have taught us that a feature that works once will work the same every time. AI systems offer no such guarantee. One good output demonstrates that a system can perform a task. It does not show how often or how reliably the system will do so. AI Outputs Are Nondeterministic AI Evaluation Should Resemble a Qu

Raluca Budiunngroup.com · 14 August 2026

Summary: One output cannot establish how well an AI system performs. Evaluate with multiple representative inputs, repeated runs, and confidence intervals. Suppose you ask an AI customer-service system the same question several times: “Can I return an opened product after 30 days?” One answer may explain the policy accurately. Another may omit an important exception. A third may confidently promise a refund that the customer is not entitled to receive. Which of these answers represents the system? The answer is: all of them, taken together — and none of them, taken in isolation. Yet teams often evaluate AI systems by running them once, inspecting the result, and drawing conclusions about what AI can do or how it should be used. This is not because they are lazy or careless. It is because decades of deterministic software have taught us that a feature that works once will work the same every time. AI systems offer no such guarantee. One good output demonstrates that a system can perform a task. It does not show how often or how reliably the system will do so. AI Outputs Are Nondeterministic AI Evaluation Should Resemble a Quantitative U

Intercom Blogintercom.com · 13 August 2026

Today we’re announcing Evals and Releases. Paired with Monitors, they give you an evaluation system for Fin, so you can test changes before they go live, roll them out with control, evaluate every live conversation, and have confidence in the experience Fin delivers. Evals tests Fin’s behavior at scale using simulated scenarios built from your own conversations. You group these simulated conversations into an Eval around a theme – such as refund requests, escalation rules, or Fin’s tone of voice – and each one is scored automatically against criteria you set. Releases gives your team a dedicated space to build changes away from the live version of Fin, test them on real conversations, and roll them out safely. Once live, Monitors keeps the process continuous: assessing the quality of every conversation, and catching issues you can work through in your next Eval and Release. We call this “Eval-driven delivery.” It’s the discipline our AI research team uses to build and tune AI products, and it’s now in the hands of the teams running support with Fin. Here’s what’s new: Evals tests Fin end to end. Cre

Lovablejobs.ashbyhq.com · 13 August 2026

TL;DR: We're looking for a Senior Product Manager to lead the Lovable agent end-to-end - owning quality, roadmap, and the feedback loops that make it better. You'll shape how our agent reasons, acts, and delivers for millions of users, directly influencing one of the most consequential product surfaces at Lovable. Why Lovable? Lovable is the software creation platform that gives people the power to act on the problems closest to them. For decades, turning an idea into software required so much capital, technical fluency, and time that many ideas never came to life. Lovable is the counterargument: a platform for all people with ideas, ambition, and problems worth solving. From solopreneurs to small business owners to teams at companies like Adidas and Zendesk, people have built over 60 million projects on Lovable since its launch in November 2024. And we’re just getting started. We’re building a generational company from Stockholm, with growing teams in London, Boston, New York, and San Francisco. Our team is small, talent-dense, and moving quickly, with a culture rooted in extreme ownership, high velocity, and low-ego collaboration. We look for people who c

Vantajobs.ashbyhq.com · 12 August 2026

At Vanta, our mission is to help businesses earn and prove trust. We believe that security should be monitored and verified continuously, and we empower companies to practice better security and prove it with ease. Vanta has a kind and talented team, and while some have prior security experience, many have been successful at Vanta without it. As a Staff Product Manager, AI Foundations at Vanta, you will build the core AI capabilities that power Vanta’s AI product experiences: the agent harness our agents are built on, and the quality and evaluation stack that makes them trustworthy. The AI pillar is making Vanta the intelligence layer for security and compliance work, wherever that work starts. We build the AI products and foundations that work across Vanta: a cohesive first-party Vanta Agent experience, headless surfaces like MCP and CLI that bring Vanta into tools like Claude Code and Codex, and the shared capabilities that let every team ship high-trust AI products. This role sits behind Vanta’s customer-facing AI experiences. Today, every team shipping AI features has to answer the same hard questions on their own: how to define quality, ho

Gustojob-boards.greenhouse.io · 6 August 2026 · 2 jobs

A systems thinker with 8+ years of product management experience building complex, data-intensive platforms — ideally in customer service, CRM, or enterprise SaaS environments where routing, orchestration, or workflow automation were core to the product Deep familiarity with AI/ML concepts — particularly intent classification, NLP, and agentic systems — with a track record of translating model capabilities into tangible product outcomes at scale A hands-on builder who can move fluidly between strategy and execution — equally comfortable whiteboarding architecture with engineers, influencing senior stakeholders, and diving into the data to pressure-test assumptions Strong cross-functional instincts and a demonstrated ability to align Sales, Service, and Engineering teams around a shared product vision, even in the face of competing priorities and organizational complexity A bias toward simplicity — someone who defaults to first principles when solving hard problems, knows when to tear down legacy systems rather than patch them, and measures success by how much friction they remove for customers and teammates alike If you don't think you meet

Rampjobs.ashbyhq.com · 4 August 2026

ABOUT RAMP Ramp is building the smart infrastructure for finance teams, embedded in the transaction flow of every dollar a business spends. We automate how over $200B in annualized spend flows in and out of 70,000+ companies: authorizing payments, flagging risk, categorizing spend, and closing books. The problems are high-stakes, data-dense, and unforgiving. We hire people with high agency and high urgency. We look for slope over intercept. We care less about where you trained and more about what you’ve built. At Ramp, everyone is a builder who owns problems end to end and makes consequential decisions that shape the outcome. The median Ramp customer saves 5% and grows revenue 16% in their first year – far in excess of businesses operating without Ramp. We believe every ambitious company deserves the same. If you want to build systems that directly shape how companies move and manage billions, Ramp is the place to do it. ABOUT THE ROLE As the Product Manager for Tax, you will own the vision, strategy, and roadmap for Ramp’s next generation of AI-powered tax products. You will build solutions to automate tax calculation, validation, and filing for our existing

Supabasesupabase.com · 31 July 2026

Today we're open sourcing supabase/evals, our benchmark and framework for testing how well AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real Supabase tasks, for example, building a schema, debugging a failed Edge Function, or fixing a broken RLS policy, and then scores how well they performed. It powers both our published benchmark and an internal regression suite we monitor daily. As more people ship Supabase projects through an agent instead of by hand, we wanted a way to measure that experience instead of guessing. The results cited here reflect a snapshot at the time of writing. AI tooling changes rapidly, so check the live page for current results. Agents are becoming a primary way people build with Supabase, interacting through our CLI, MCP server, agent skills, and docs. We needed a way to understand how agents perform across all of these Supabase surfaces, not just one tool in isolation. By tracking where agents stumble, we can make intentional fixes and verify they hold instead of regressing, and test new features confidently before shipping. We started by defining the dimensions we

Webflowjob-boards.greenhouse.io · 30 July 2026 · 2 jobs

Webflow is the agentic web marketing platform for modern marketing teams, helping organizations build, manage, and optimize high-performing web experiences that drive predictable growth and strengthen brand trust. Building at Webflow will need grit, because we move fast, without ever sacrificing craft or quality. We're looking for a Senior Product Manager, AI to own how AI shows up across Webflow's platform. This is our Agent Access area, spanning the Webflow AI Assistant inside the Designer, the 3rd-party agents that reach Webflow through MCP, the Agent Instructions that guide them, and the platform underneath that powers it all. It's early, fast-moving territory. Much of what "good" looks like for AI-assisted creation hasn't been written yet, and you'll help define it, unlocking new ways for developers, partners, and creators to build for the web. About the role: Location: Remote-first (United States; BC & ON, Canada) Full-time Permanent Exempt The cash compensation for this role is tailored to align with the cost of labor in different geographic markets. We've structured the base pay ranges for this role into zones for our geographic markets, and the s

philschmid.dephilschmid.de · 29 July 2026

Most coding benchmarks work the same. You give the agent a single task, let it work, check the result. The agent might run dozens of tool calls, but there's only one user input and one final evaluation. That's not how most of us use agents. You build something, you get new ideas, requirements shift, you refactor... EvoCode-Bench is designed to test that cycle. EvoCode-Bench is a new multi-turn coding benchmark with 26 tasks spanning 227 sequential rounds (5–15 rounds per task). It evaluates agents across five domains: ML/MLOps, build systems, data engineering, cloud/security, and scientific computing. Three things make it interesting: Persistent workspace: One container lives across the entire lifecycle of a task. Code, dependencies, and architectural decisions from Round 1 are still there at Round 15. Evolving specifications: Each new input brings a new instruction that either extends the codebase, corrects logic, or conflicts with earlier requirements (intentionally breaking previous assumptions). Cumulative testing: After each turn, the test suite checks all requirements, not just the new one. Break something from a previous turn, and yo

Product Growthnews.aakashg.com · 28 July 2026

He wrote evals for Gemini, for Llama, and for Ray-Ban Meta. Today he builds one from a blank spreadsheet, live!

Aakash Guptanews.aakashg.com · 28 July 2026

Check out the conversation on Apple, Spotify, and YouTube. Brought to you by: SerpApi - Get started with SerpApi using 250 free credits. Product Faculty - Get $550 off their #1 AI PM Cert with code AAKASH550C7 Ariso - Ship AI agents and features faster, with fewer regressions Land PM Job - 12-week experience to master getting a PM job Pendo - The #1 software experience management platform Today’s Episode Rasty Turek spent the past year building with coding agents, and he mapped how his process changed over that time. His eval started as QA. Then it became the spec. He reckons he now spends around 90% of his time on evals. And the difference is “indescribable.” Just about every developer and builder has experienced a similar pattern. This is the opportunity for PMs to add great value to the development team. Just as the value of a long PRD was waning, evals sprang up. And PMs are often considered the best owner. So how do you do them well? I’ve now covered evals basics, excel implementation, advanced theory, Braintrust implementation, and Arize implementation. Today, I cover how to practically write

blog.murphytrueman.comblog.murphytrueman.com · 17 July 2026 · not counted as a party

Design systems have spent years borrowing from how software gets built. Versioning, semantic naming, explicit contracts, deprecation cycles. The API framing I wrote about last year was the same premise, treating a component library as an interface other systems consume rather than a folder of pictures. Most of that borrowing is now considered normal practice. The next thing to take comes from a different part of the stack.When you let an agent generate against your system, you're running a model you don't control against rules you wrote, and hoping the output respects them. That's the same problem the people shipping LLM features have been dealing with for a couple of years. One of their answers is evals.An eval is a test for output you can't predict. You hold a set of inputs constant, run them through the model, and check the results against assertions about what a good answer contains. promptfoo, acquired by OpenAI in March, made these cheap to write and run in CI on every change. Because the same input can produce a different output next time, you run each case several times for nondeterministic tasks and watch for the result that wavers.What this looks

Sunojobs.ashbyhq.com · 17 July 2026

ABOUT SUNO We're building the world's first creative entertainment platform, where the entire world can feel the joy and fulfillment of making music. Music is for everyone: Our users include everyone from grandmothers creating songs for their loved ones, to Grammy winners using Suno Studio, our power tool, to make the most popular hits in the world. Building the future of entertainment requires ambition. The pace is fast, the problems are hard, and the work demands ownership and intensity. For the right people, it’s incredibly rewarding: a chance to shape a new medium, work with a small team that cares deeply about quality, make music, drink too much coffee, and build something that millions of people use to express themselves in ways that were never before possible. Suno is the fastest growing consumer entertainment company and the leader in AI music. We are backed by leading investors including Bond Capital, Menlo Ventures, Lightspeed Venture Partners, IVP, Forerunner, Union Square Ventures, Alkeon, Quiet, Matrix Partners, Schroders Capital and, NVentures (venture arm of NVIDIA). Founded in 2023 by Harvard alumni with a shared passion for music, Suno

Jeff Gothelfjeffgothelf.com · 13 July 2026

Before you read this post I wanted to note that I do not cover Objectives in this article. The reason is that without context it’s difficult to provide an example that would resonate broadly in a short blog post. So this article focuses solely on key results. Don’t forget to write an aspirational and inspirational objective to go along with your AI-tool OKRs. Earlier this year Josh Seiden and I worked with an AI team at a big bank in Europe. As you might imagine we spent a good chunk of time with them discussing how they’re measuring the success of their new AI-powered services. Many of these tools were internally-facing, targeting the staff of the company as the target audience. “Achieve 95% accuracy in all output.” The team was all bought into this number because it felt rigorous, the kind of number a serious product team commits to. It’s also an output. And once again, even with AI products, we’re seeing the same mistake teams have been making with OKRs for fifteen years. I’ve argued for a long time that a key result has to be a measure of human behavior. Not an output, not a task, not a feature you shipped but rather what a customer does diff

OpenAIjobs.ashbyhq.com · 7 July 2026 · 4 jobs

About the Team The Core Models team helps shape how OpenAI’s frontier models are built, measured, and launched. We work across Research, Engineering, Model Design, Data Science, and Product to turn advances in model capabilities into reliable, useful experiences for people. Our scope includes model planning and launches as well as building data flywheels, evaluations and measurement systems to ensure our models have strong capabilities and behavior. About the Role As a Product Manager for the Core Models team, you'll be at the forefront of defining and guiding the future of how our AI models work in real-world applications. You will connect user needs to model and systems decisions: how prompts are understood; how information is aggregated and made useful for training and evaluation data; and how capabilities move from research prototypes into the mainline model and launch stack. You will operate comfortably across research, infrastructure, and consumer product surfaces, creating clarity where ownership and technical boundaries are still emerging. This role is based in San Francisco, CA. We use a hybrid work model of 3 days in the office per week and of

Databricksdatabricks.com · 23 June 2026

8+ years of product management experience, with at least 3 years on internal platform, infrastructure, or developer-experience products. Deep experience building platforms that other teams build on you understand the difference between a platform and an application, and you have opinions about API design, developer ergonomics, and self service. Demonstrated experience with AI/ML platforms, agent frameworks, LLM-powered applications, or agentic systems. You know what an agent runtime is, what RAG means in practice, and why evaluation is the hardest part. Strong technical foundation you can read architecture diagrams, discuss trade offs with engineers (e.g., sync vs. async, checkpointing strategies, context window management), and make informed prioritization decisions on deeply technical work. Experience defining and shipping developer experiences: SDKs, CLIs, templates, documentation, and self service workflows. You measure success by adoption and developer NPS, not feature count. Proven ability to lead cross-functional initiatives across 4+ teams without direct authority. You influence through clarity, conviction, and stakeholder alig

Paweł Hurynproductcompass.pm · 22 June 2026

A loop does not fail because the agent keeps working. It fails because nobody told it what “done” means. That’s the part most loop guides skip, and it isn’t a coding detail. Defining “done” is writing acceptance criteria, picking a success metric, and deciding who signs off. A loop is only as good as the thing that tells it to stop. It also explains why the loop debate sounds so confused. Nvidia's Jensen Huang: "Nobody writes prompts anymore. The new job is to write and handle loops." Boris Cherny, who built Claude Code: "My job is to write loops." Then a week later, another builder declared loops dead. One camp means retrying the same prompt forever. Another means a goal-driven system that checks itself. They're not arguing about the same thing. A loop is an agent that repeats one cycle until something tells it to stop. Goal, act, check, adjust, go again. It's the feedback loop agile teams have been running, with the agent running the cycle instead of you. What's new is that the agent runs the cycle itself instead of waiting for your next prompt. What you’ll learn: What a loop is, and how it differs from a routine and a workflow How t

Sierrajobs.ashbyhq.com · 17 June 2026

ABOUT US At Sierra, we’re building a platform to enable every company in the world to build better, more human customer experiences with AI. We partner with industry leaders such as SoftBank, Uber, Rivian, CLEAR, and Sutter Health. We are primarily an in-person company based in San Francisco, with growing offices across North America, Europe, and Asia. We are guided by a set of values that are at the core of our actions and define our culture: Trust, Customer Obsession, Craftsmanship, Intensity, and Family. These values are the foundation of our work, and we are committed to upholding them in everything we do. Our co-founders are Bret Taylor https://www.linkedin.com/in/brettaylor/ and Clay Bavor https://www.linkedin.com/in/claybavor/. Bret currently serves as Board Chair of OpenAI. Previously, he was co-CEO of Salesforce (which had acquired the company he founded, Quip) and CTO of Facebook. Bret was also one of Google's earliest product managers and co-creator of Google Maps. Before founding Sierra, Clay spent 18 years at Google, where he most recently led Google Labs. Earlier, he started and led Google’s AR/VR effort, Projec

Claire Volennysnewsletter.com · 15 June 2026

In this episode, I sit down with Ankur Goyal, founder and CEO of Braintrust, the AI evals and observability platform used by teams like Notion, Stripe, Vercel, and Zapier. This one is for the senior engineers, staff engineers, VPs of engineering, and CTOs in my audience. We get into how coding agents can take on deeply technical architecture and infrastructure work that no single human engineer could tackle before, and then we demystify evals so you can use them to make your AI products better without touching the implementation. Listen or watch on YouTube, Spotify, or Apple Podcasts What you’ll learn: How Ankur uses Codex to run week-long benchmark experiments across database indexes, column store formats, and execution engines to speed up slow queries Why he argues there’s no excuse to skip rigorous benchmarking now that agents can run them tirelessly The “agent line” framework: how to decide which decisions, directions, and interactions you can hand off to an agent How I think about the practical vs. theoretical quality of AI on hard technical problems, and why human attention decays on tedious wo

Lenny Rachitskylennysnewsletter.com · 15 June 2026

Claude Fable 5 review: what the new Mythos model gets right (and very wrong) Listen now on YouTube • Spotify • Apple Podcasts Claire puts Claude Fable 5, Anthropic’s first generally available Mythos-class model, through a series of real-world tests: product specs, agent workflows, design tasks, vision tasks, and multi-agent orchestration. She breaks down what Anthropic is claiming, where the model genuinely feels like a leap forward, and where it surprisingly falls short. Biggest takeaways: Fable 5 is Anthropic’s first “Mythos-class” model to reach general availability, and it’s crushing benchmarks across the board. It hit 80% on SWBench Pro, significantly outperforming Opus 4.8, GPT-4.5, and Gemini 3.1 Pro. Claire found the model excels in specific areas while falling short in others that matter for everyday product work. The model is expensive by design: $10 per million input tokens and $50 per million output tokens. That’s a new tier above Opus, and it consumes tokens at roughly twice the rate of other models. You need to be strategic about when to deploy this level of inte

latent.spacelatent.space · 4 June 2026 · 2 posts

The new AIEWF website is live! Get your tickets booked ASAP as they -will- sell out. Take the AI Engineering Survey and get >$2k in credits and free AIE WF tickets! Most industry benchmarks compress intelligence and reasoning ability into scores. SWE-Bench Pro, MMLU, Humanity’s Last Exam, etc. These metrics are useful, but don’t always represent the full extent of how a model performs in the real world. Some of the most interesting evals today look less like exams and more like operating businesses in the real world. One of which is Vending Bench. In Anthropic’s Mythos Preview System Card, Andon was the only third party eval to get their own section, observing increasingly concerning aggressive behavior: You don’t know what a model is capable of doing in the real world unless you actually give it inventory, a wallet, tools, customers, competitors, humans, & some time. More often than not, it’ll surprise you how much a model is capable of and in doing so, also reveal unexpected behavior: deception, context collapse, emergent coordination, & bizarre negotiation behavior. While an inflection point in personal ag

Anthropicjob-boards.greenhouse.io · 4 June 2026

About Anthropic Anthropic’s mission is to create reliable, interpretable, and steerable AI systems. We want AI to be safe and beneficial for our users and for society as a whole. Our team is a quickly growing group of committed researchers, engineers, policy experts, and business leaders working together to build beneficial AI systems. About the role As a Product Manager on Claude Code's model performance team, you will drive model launches end-to-end, build evals that measure what matters, and partner directly with researchers and product engineers to translate model improvements into developer-facing outcomes. Claude Code is the most capable coding agent in the world but there’s much more we can do to extract the maximum performance from our models. We're looking for a PM who has personally built agentic evals, thinks in systems, uses Claude Code every day, and has refined model taste. You should be as comfortable influencing our research team as you are getting in the weeds of transcripts. You will be the connective tissue between frontier research and the millions of developers who depend on Claude Code to do their best work. Respo

Stripestripe.com · 20 May 2026

We’re looking for someone who meets the minimum requirements to be considered for the role. If you meet these requirements, you are encouraged to apply. The preferred qualifications are a bonus, not a requirement. Our ideal Product Manager will be relentlessly user-focused, comfortable in ambiguity, have strong technical skills, and have the desire to create something new. 12+ years of experience as a Product Manager Deep user empathy and a high bar for user experience Experience developing a multi-year vision, strategy and roadmap for large-scale products Experience leading cross-team initiatives from ideation to execution Technical expertise: You’ve led technical products and/or have a background in computer science (or similar) Strong written and verbal communication skills

Datadogcareers.datadoghq.com · 7 April 2026

5+ years of experience in B2B SaaS Product Management, with at least some of it spent on data-intensive or analytics products You have domain knowledge of business intelligence, data visualization, semantic layers, data warehouses and the modern data stack. Strong analytical skills - you can write SQL, interpret complex data, and use data to drive product decisions Experience with AI/LLM-powered products is a strong plus - you understand prompt engineering, evals, and the unique UX challenges of non-deterministic systems You are AI-pilled. You live in Claude Code or Cursor, have built unique AI workflows for yourself, hack around with new tools. etc You have a strong sense of ownership with an entrepreneurial mindset - this is a 0-to-1 product that requires someone comfortable with ambiguity and speed Sharp communication skills with the ability to interact with technical stakeholders and executives alike. Bachelor's Degree in Engineering, Computer Science, Data Science, or equivalent experience is a plus Datadog values people from all walks of life. We understand not everyone will meet all the above qualifications on day one. That's okay. If you’

Figmaboards.greenhouse.io · 21 November 2025

Experience developing design heuristics and design principles Experience leading design efforts for a suite of products or working on features that span multiple product lines Experience using AI models for design tasks, either coding or image models Proven track record in managing complex projects that involve collaboration across different teams and organizational boundaries Experience managing contractors and multiple projects on tight timelines Strong design background with excellent taste and attention to detail