Sidebar.ioearendil.com · 15 September 2026 · not counted as a party

LLMs have become almost perfect at generating code, but that isn’t the end of the story. Just because the code is formally correct doesn’t mean that it is not introducing unnecessary abstractions, creating duplicates, or just making bad decisions overall. This is not a groundbreaking observation, most people who have vibe-coded a project, have realized that each additional feature can sometimes lead to an explosion of lines of code (LOC). This results in a loss of human agency, because in projects that are adding millions of LOC per month, it is hard for humans to keep up.1 Some people might say that that is not an issue at all, because they trust their agents to deal with it. I have bad news for you, agents can't really deal with the slop either. Coming from a physics background, I always had an experimental/quantitative approach to solving problems. When I started at Earendil, with the task of figuring out how to measure code sloppiness, my natural instinct was to first take a deep dive into the literature and then check what other companies were doing. To be frank, with the exception of a few insightful research papers, I was disappointed at how “v

Hugging Face bloghuggingface.co · 15 September 2026

Your agent works in rehearsal, but during the live demo, it takes a different path and fails the same task. That is embarrassing onstage. In production, it is a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, such as reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper. Most benchmarks hide this variability behind an average. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions. But it succeeded in all five runs for only 53.0% of tasks — a 24.4-point consistency gap. Most benchmarks report the first number. We built a way to measure the second — and improve it. In an earlier post, we introduced ALTK-Evolve — a system that turns an agent's own past trajectories into reusable guidelines, distilled automatically and injected back at inference time. It measurably improves task success, but those results only asked the average-case question too. This post introduces consistency guidelines, a new guideline type in altk-evolve built on top of a diagnostic tool we call th

Meta engineeringengineering.fb.com · 2 September 2026

We’ve built an AI agent that acts as a secondary expert for a given domain, making deep specialist knowledge readily available and preserved for anyone in an organization to access, share, and build upon. This is not a typical domain-specific agent. Its novelty comes from integrating two layers:  A structured, auditable knowledge architecture separates what the agent knows from how it reasons.  A self-improvement loop then compiles expert feedback into verified, regression-tested updates without model retraining.  Together, these two layers turn one-off expert corrections into permanent, compounding institutional memory, and the pattern is designed to generalize to other domains governed by retrievable text rather than model weights. This system is saving domain subject matter experts (SME)s at Meta substantial time, allowing them to focus more on the work where their knowledge matters most. Many large organizations have the same problem when it comes to specialist knowledge. While some of it is written down in the form of models, playbooks, checklists, and frameworks, the most valuable specialist knowledge

DeepMind blogdeepmind.google · 27 August 2026

August 27, 2026 Responsibility & SafetyWilliam Isaac, Sol Messing and Kristian LumBuilding trust in proprietary model benchmarks using cryptographically secure environmentsImagine a student is set to take a high-stakes exam. If they accidentally peek at the test questions in advance, achieving a perfect score is influenced by this knowledge, making it a meaningless accomplishment. To truly measure what they know, they must have no visibility of the test questions until it's time to take the exam. That is the exact challenge the industry faces when evaluating advanced AI models. If a model has already seen the test questions - a problem known as benchmark contamination - the results can only be trusted to an extent.Today, we’re introducing the world’s first double-blind evaluation of a proprietary, frontier class AI model, which keeps external evaluations confined to a cryptographic “box” where they can’t be used by models later to optimize performance ahead of testing. We're partnering with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, to test a Gemini Flash Lite model against confidential benchmarks in a

LangChain bloglangchain.com · 26 August 2026 · 4 posts

Key Links:Technical documentationVideo walkthroughEvaluation is the process of continuously improving your LLM application. This requires a way to measure your application’s performance. LLM applications often produce outputs in natural language, which are difficult to judge using hard-coded rules. For example, attributes like conciseness or correctness relative to a reference output are difficult to express as typical unit tests. Using an “LLM-as-a-Judge” is a popular way to grade natural language outputs from LLM applications. This involves passing the generated output (and other information) to a separate LLM and asking it to judge the output. Although this has proven useful in several contexts, it raises an interesting problem: you now have to do another round of prompt engineering to make sure the LLM-as-a-Judge is performing well.LangSmith presents a novel solution to this rising problem. LangSmith evaluators now feature “self-improvement” whereby human corrections to LLM-as-a-Judge outputs are stored as few-shot examples, which are then fed back into the prompt in future iterations.💡The net impact is that it is easier to create

Vercel changelogvercel.com · 25 August 2026 · not counted as a party

Agents increasingly write TypeScript programs to coordinate tools and process their results. Once those programs touch real applications, some steps require authentication, while others need human approval.Executing that code with eval gives it the same access as the application around it, including its secrets and internal services, and leaves no durable way to pause at those boundaries.Today, we're releasing the Run SDK, a package for executing untrusted JavaScript and TypeScript without giving it direct access to your application or system. Applications expose narrow host functions and can interrupt execution for authentication or human-in-the-loop approval. The program resumes after a decision without repeating completed work.pnpm add runCopy link to headingA small interface to the hostThe Run SDK evaluates JavaScript or type-stripped TypeScript in a fresh QuickJS context inside a worker thread, with no direct route to Node.js or the network.The application exposes selected operations through hostFunctions. These are regular functions that become callable globals inside the sandbox:import { run } from 'run';const result = await r

GitHubgithub.blog · 25 August 2026

These are the lessons we learned evaluating LLMs for real-world secret scanning. August 25, 2026 | 12 minutes Share: A language model can perform well on a clean benchmark and still struggle with the cases that matter in production. Benchmarks and curated datasets are useful when prototyping an LLM-based system. They help teams compare models, test an initial prompt, and determine whether an idea is technically plausible. But as a system moves closer to production, the evaluation problem changes. Real inputs are often ambiguous. Labels may be inconsistent. Important context may be missing or truncated. The evaluation set may not reflect the production distribution. Edge cases that rarely appear in benchmarks can become common sources of failure. Even when offline metrics improve, those results may not translate cleanly into production behavior. We encountered these challenges while evaluating an LLM-based system designed to reduce false positives in GitHub secret scanning. Secret scanning identifies credentials such as tokens and keys that may have

PostHogposthog.com · 24 August 2026

Part of my job as a product marketer at PostHog is keeping tabs on what every other company in the data space is talking about. So I read a lot of industry reports, and surrendered my email address to a bunch of different mailing lists. Here's what I learned so you don't have to do the same.Everyone agrees on the problem their customers are facing: Product development is stalling because AI ambition is outrunning data readiness, trust, and governance. But, most vendors in the data space are specialized, so each is describing the problem from inside their own slice of the stack. The pipeline vendor sees a pipeline problem. The governance vendor sees a governance problem.Without naming it, every report I read was describing the same fix – a context warehouse. Yes, I know how that sounds; the PostHog PMM thinks the answer is PostHog. Fair. But I didn't run these numbers. My competitors did, and published them. I'm the first to read all of them, saw the pattern, and say why that adds up to a context warehouse.The average company now runs 106 SaaS apps, each one throwing off its own data.BetterCloud, SaaS statistics, 2025An agent is only as good as the data it can r

Arize blogarize.com · 24 August 2026 · 4 posts

TL;DR: A skill is just an agent, a prompt running in a harness, so you can test a change to it exactly the way you test any other agent. This post walks through how we do that for the arize-instrumentation skill: a golden dataset of real apps to instrument, a sandboxed experiment that stops the agent cheating its way to a good score, and trace-level evaluators that grade each run. By the end you’ll have a repeatable way to answer “did that change actually make the skill better, faster, and cheaper,” backed by a real merged PR and the numbers behind it. You changed a skill, tightened the instructions, cut a redundant block, reworded the step that kept tripping the agent up, and now it feels better. But does it feel better because it is better, or because you just spent an hour staring at it and want to be done? That question should sound familiar because it’s the same one you ask every time you touch a prompt or swap a model. You run the agent a few times by hand, the output looks fine, and you ship on vibes. Those vibes are the problem, and with a skill it’s especially easy to convince yourself you’re safe because a skill looks lik

Stack Overflow blogstackoverflow.blog · 18 August 2026

Ryan welcomes Suneet Malhotra, Senior Manager of Test Engineering at Motorola Solutions, to chat about building end-to-end agentic SDLC pipelines using MCPs, using Cohen’s kappa to evaluate multiple LLMs-as-judges, and how you can improve requirements by shifting QA left through a specification enrichment stage immediately after the design phase.

Nielsen Norman Groupnngroup.com · 14 August 2026

Summary:  One output cannot establish how well an AI system performs. Evaluate with multiple representative inputs, repeated runs, and confidence intervals. Suppose you ask an AI customer-service system the same question several times: “Can I return an opened product after 30 days?” One answer may explain the policy accurately. Another may omit an important exception. A third may confidently promise a refund that the customer is not entitled to receive. Which of these answers represents the system? The answer is: all of them, taken together — and none of them, taken in isolation. Yet teams often evaluate AI systems by running them once, inspecting the result, and drawing conclusions about what AI can do or how it should be used. This is not because they are lazy or careless. It is because decades of deterministic software have taught us that a feature that works once will work the same every time. AI systems offer no such guarantee. One good output demonstrates that a system can perform a task. It does not show how often or how reliably the system will do so. AI Outputs Are Nondeterministic AI Evaluation Should Resemble a Qu

Raluca Budiunngroup.com · 14 August 2026

Summary: One output cannot establish how well an AI system performs. Evaluate with multiple representative inputs, repeated runs, and confidence intervals. Suppose you ask an AI customer-service system the same question several times: “Can I return an opened product after 30 days?” One answer may explain the policy accurately. Another may omit an important exception. A third may confidently promise a refund that the customer is not entitled to receive. Which of these answers represents the system? The answer is: all of them, taken together — and none of them, taken in isolation. Yet teams often evaluate AI systems by running them once, inspecting the result, and drawing conclusions about what AI can do or how it should be used. This is not because they are lazy or careless. It is because decades of deterministic software have taught us that a feature that works once will work the same every time. AI systems offer no such guarantee. One good output demonstrates that a system can perform a task. It does not show how often or how reliably the system will do so. AI Outputs Are Nondeterministic AI Evaluation Should Resemble a Quantitative U

Intercom Blogintercom.com · 13 August 2026

Today we’re announcing Evals and Releases. Paired with Monitors, they give you an evaluation system for Fin, so you can test changes before they go live, roll them out with control, evaluate every live conversation, and have confidence in the experience Fin delivers. Evals tests Fin’s behavior at scale using simulated scenarios built from your own conversations. You group these simulated conversations into an Eval around a theme – such as refund requests, escalation rules, or Fin’s tone of voice – and each one is scored automatically against criteria you set. Releases gives your team a dedicated space to build changes away from the live version of Fin, test them on real conversations, and roll them out safely. Once live, Monitors keeps the process continuous: assessing the quality of every conversation, and catching issues you can work through in your next Eval and Release. We call this “Eval-driven delivery.” It’s the discipline our AI research team uses to build and tune AI products, and it’s now in the hands of the teams running support with Fin. Here’s what’s new: Evals tests Fin end to end. Cre

Supabasesupabase.com · 31 July 2026

Today we're open sourcing supabase/evals, our benchmark and framework for testing how well AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real Supabase tasks, for example, building a schema, debugging a failed Edge Function, or fixing a broken RLS policy, and then scores how well they performed. It powers both our published benchmark and an internal regression suite we monitor daily. As more people ship Supabase projects through an agent instead of by hand, we wanted a way to measure that experience instead of guessing. The results cited here reflect a snapshot at the time of writing. AI tooling changes rapidly, so check the live page for current results. Agents are becoming a primary way people build with Supabase, interacting through our CLI, MCP server, agent skills, and docs. We needed a way to understand how agents perform across all of these Supabase surfaces, not just one tool in isolation. By tracking where agents stumble, we can make intentional fixes and verify they hold instead of regressing, and test new features confidently before shipping. We started by defining the dimensions we

philschmid.dephilschmid.de · 29 July 2026

Most coding benchmarks work the same. You give the agent a single task, let it work, check the result. The agent might run dozens of tool calls, but there's only one user input and one final evaluation. That's not how most of us use agents. You build something, you get new ideas, requirements shift, you refactor... EvoCode-Bench is designed to test that cycle. EvoCode-Bench is a new multi-turn coding benchmark with 26 tasks spanning 227 sequential rounds (5–15 rounds per task). It evaluates agents across five domains: ML/MLOps, build systems, data engineering, cloud/security, and scientific computing. Three things make it interesting: Persistent workspace: One container lives across the entire lifecycle of a task. Code, dependencies, and architectural decisions from Round 1 are still there at Round 15. Evolving specifications: Each new input brings a new instruction that either extends the codebase, corrects logic, or conflicts with earlier requirements (intentionally breaking previous assumptions). Cumulative testing: After each turn, the test suite checks all requirements, not just the new one. Break something from a previous turn, and yo

Product Growthnews.aakashg.com · 28 July 2026

He wrote evals for Gemini, for Llama, and for Ray-Ban Meta. Today he builds one from a blank spreadsheet, live!

Aakash Guptanews.aakashg.com · 28 July 2026

Check out the conversation on Apple, Spotify, and YouTube. Brought to you by: SerpApi - Get started with SerpApi using 250 free credits. Product Faculty - Get $550 off their #1 AI PM Cert with code AAKASH550C7 Ariso - Ship AI agents and features faster, with fewer regressions Land PM Job - 12-week experience to master getting a PM job Pendo - The #1 software experience management platform Today’s Episode Rasty Turek spent the past year building with coding agents, and he mapped how his process changed over that time. His eval started as QA. Then it became the spec. He reckons he now spends around 90% of his time on evals. And the difference is “indescribable.” Just about every developer and builder has experienced a similar pattern. This is the opportunity for PMs to add great value to the development team. Just as the value of a long PRD was waning, evals sprang up. And PMs are often considered the best owner. So how do you do them well? I’ve now covered evals basics, excel implementation, advanced theory, Braintrust implementation, and Arize implementation. Today, I cover how to practically write

blog.murphytrueman.comblog.murphytrueman.com · 17 July 2026 · not counted as a party

Design systems have spent years borrowing from how software gets built. Versioning, semantic naming, explicit contracts, deprecation cycles. The API framing I wrote about last year was the same premise, treating a component library as an interface other systems consume rather than a folder of pictures. Most of that borrowing is now considered normal practice. The next thing to take comes from a different part of the stack.When you let an agent generate against your system, you're running a model you don't control against rules you wrote, and hoping the output respects them. That's the same problem the people shipping LLM features have been dealing with for a couple of years. One of their answers is evals.An eval is a test for output you can't predict. You hold a set of inputs constant, run them through the model, and check the results against assertions about what a good answer contains. promptfoo, acquired by OpenAI in March, made these cheap to write and run in CI on every change. Because the same input can produce a different output next time, you run each case several times for nondeterministic tasks and watch for the result that wavers.What this looks

Jeff Gothelfjeffgothelf.com · 13 July 2026

Before you read this post I wanted to note that I do not cover Objectives in this article. The reason is that without context it’s difficult to provide an example that would resonate broadly in a short blog post. So this article focuses solely on key results. Don’t forget to write an aspirational and inspirational objective to go along with your AI-tool OKRs. Earlier this year Josh Seiden and I worked with an AI team at a big bank in Europe. As you might imagine we spent a good chunk of time with them discussing how they’re measuring the success of their new AI-powered services. Many of these tools were internally-facing, targeting the staff of the company as the target audience. “Achieve 95% accuracy in all output.” The team was all bought into this number because it felt rigorous, the kind of number a serious product team commits to. It’s also an output. And once again, even with AI products, we’re seeing the same mistake teams have been making with OKRs for fifteen years. I’ve argued for a long time that a key result has to be a measure of human behavior. Not an output, not a task, not a feature you shipped but rather what a customer does diff

Paweł Hurynproductcompass.pm · 22 June 2026

A loop does not fail because the agent keeps working. It fails because nobody told it what “done” means. That’s the part most loop guides skip, and it isn’t a coding detail. Defining “done” is writing acceptance criteria, picking a success metric, and deciding who signs off. A loop is only as good as the thing that tells it to stop. It also explains why the loop debate sounds so confused. Nvidia's Jensen Huang: "Nobody writes prompts anymore. The new job is to write and handle loops." Boris Cherny, who built Claude Code: "My job is to write loops." Then a week later, another builder declared loops dead. One camp means retrying the same prompt forever. Another means a goal-driven system that checks itself. They're not arguing about the same thing. A loop is an agent that repeats one cycle until something tells it to stop. Goal, act, check, adjust, go again. It's the feedback loop agile teams have been running, with the agent running the cycle instead of you. What's new is that the agent runs the cycle itself instead of waiting for your next prompt. What you’ll learn: What a loop is, and how it differs from a routine and a workflow How t

Claire Volennysnewsletter.com · 15 June 2026

In this episode, I sit down with Ankur Goyal, founder and CEO of Braintrust, the AI evals and observability platform used by teams like Notion, Stripe, Vercel, and Zapier. This one is for the senior engineers, staff engineers, VPs of engineering, and CTOs in my audience. We get into how coding agents can take on deeply technical architecture and infrastructure work that no single human engineer could tackle before, and then we demystify evals so you can use them to make your AI products better without touching the implementation. Listen or watch on YouTube, Spotify, or Apple Podcasts What you’ll learn: How Ankur uses Codex to run week-long benchmark experiments across database indexes, column store formats, and execution engines to speed up slow queries Why he argues there’s no excuse to skip rigorous benchmarking now that agents can run them tirelessly The “agent line” framework: how to decide which decisions, directions, and interactions you can hand off to an agent How I think about the practical vs. theoretical quality of AI on hard technical problems, and why human attention decays on tedious wo

Lenny Rachitskylennysnewsletter.com · 15 June 2026

Claude Fable 5 review: what the new Mythos model gets right (and very wrong) Listen now on YouTube • Spotify • Apple Podcasts Claire puts Claude Fable 5, Anthropic’s first generally available Mythos-class model, through a series of real-world tests: product specs, agent workflows, design tasks, vision tasks, and multi-agent orchestration. She breaks down what Anthropic is claiming, where the model genuinely feels like a leap forward, and where it surprisingly falls short. Biggest takeaways: Fable 5 is Anthropic’s first “Mythos-class” model to reach general availability, and it’s crushing benchmarks across the board. It hit 80% on SWBench Pro, significantly outperforming Opus 4.8, GPT-4.5, and Gemini 3.1 Pro. Claire found the model excels in specific areas while falling short in others that matter for everyday product work. The model is expensive by design: $10 per million input tokens and $50 per million output tokens. That’s a new tier above Opus, and it consumes tokens at roughly twice the rate of other models. You need to be strategic about when to deploy this level of inte

latent.spacelatent.space · 4 June 2026 · 2 posts

The new AIEWF website is live! Get your tickets booked ASAP as they -will- sell out. Take the AI Engineering Survey and get >$2k in credits and free AIE WF tickets! Most industry benchmarks compress intelligence and reasoning ability into scores. SWE-Bench Pro, MMLU, Humanity’s Last Exam, etc. These metrics are useful, but don’t always represent the full extent of how a model performs in the real world. Some of the most interesting evals today look less like exams and more like operating businesses in the real world. One of which is Vending Bench. In Anthropic’s Mythos Preview System Card, Andon was the only third party eval to get their own section, observing increasingly concerning aggressive behavior: You don’t know what a model is capable of doing in the real world unless you actually give it inventory, a wallet, tools, customers, competitors, humans, & some time. More often than not, it’ll surprise you how much a model is capable of and in doing so, also reveal unexpected behavior: deception, context collapse, emergent coordination, & bizarre negotiation behavior. While an inflection point in personal ag