Sidebar.ioearendil.com · 15 September 2026 · not counted as a party

LLMs have become almost perfect at generating code, but that isn’t the end of the story. Just because the code is formally correct doesn’t mean that it is not introducing unnecessary abstractions, creating duplicates, or just making bad decisions overall. This is not a groundbreaking observation, most people who have vibe-coded a project, have realized that each additional feature can sometimes lead to an explosion of lines of code (LOC). This results in a loss of human agency, because in projects that are adding millions of LOC per month, it is hard for humans to keep up.1 Some people might say that that is not an issue at all, because they trust their agents to deal with it. I have bad news for you, agents can't really deal with the slop either. Coming from a physics background, I always had an experimental/quantitative approach to solving problems. When I started at Earendil, with the task of figuring out how to measure code sloppiness, my natural instinct was to first take a deep dive into the literature and then check what other companies were doing. To be frank, with the exception of a few insightful research papers, I was disappointed at how “v

Hugging Face bloghuggingface.co · 15 September 2026

Your agent works in rehearsal, but during the live demo, it takes a different path and fails the same task. That is embarrassing onstage. In production, it is a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, such as reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper. Most benchmarks hide this variability behind an average. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions. But it succeeded in all five runs for only 53.0% of tasks — a 24.4-point consistency gap. Most benchmarks report the first number. We built a way to measure the second — and improve it. In an earlier post, we introduced ALTK-Evolve — a system that turns an agent's own past trajectories into reusable guidelines, distilled automatically and injected back at inference time. It measurably improves task success, but those results only asked the average-case question too. This post introduces consistency guidelines, a new guideline type in altk-evolve built on top of a diagnostic tool we call th

Meta engineeringengineering.fb.com · 2 September 2026

We’ve built an AI agent that acts as a secondary expert for a given domain, making deep specialist knowledge readily available and preserved for anyone in an organization to access, share, and build upon. This is not a typical domain-specific agent. Its novelty comes from integrating two layers:  A structured, auditable knowledge architecture separates what the agent knows from how it reasons.  A self-improvement loop then compiles expert feedback into verified, regression-tested updates without model retraining.  Together, these two layers turn one-off expert corrections into permanent, compounding institutional memory, and the pattern is designed to generalize to other domains governed by retrievable text rather than model weights. This system is saving domain subject matter experts (SME)s at Meta substantial time, allowing them to focus more on the work where their knowledge matters most. Many large organizations have the same problem when it comes to specialist knowledge. While some of it is written down in the form of models, playbooks, checklists, and frameworks, the most valuable specialist knowledge

DeepMind blogdeepmind.google · 27 August 2026

August 27, 2026 Responsibility & SafetyWilliam Isaac, Sol Messing and Kristian LumBuilding trust in proprietary model benchmarks using cryptographically secure environmentsImagine a student is set to take a high-stakes exam. If they accidentally peek at the test questions in advance, achieving a perfect score is influenced by this knowledge, making it a meaningless accomplishment. To truly measure what they know, they must have no visibility of the test questions until it's time to take the exam. That is the exact challenge the industry faces when evaluating advanced AI models. If a model has already seen the test questions - a problem known as benchmark contamination - the results can only be trusted to an extent.Today, we’re introducing the world’s first double-blind evaluation of a proprietary, frontier class AI model, which keeps external evaluations confined to a cryptographic “box” where they can’t be used by models later to optimize performance ahead of testing. We're partnering with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, to test a Gemini Flash Lite model against confidential benchmarks in a

LangChain bloglangchain.com · 26 August 2026 · 4 posts

Key Links:Technical documentationVideo walkthroughEvaluation is the process of continuously improving your LLM application. This requires a way to measure your application’s performance. LLM applications often produce outputs in natural language, which are difficult to judge using hard-coded rules. For example, attributes like conciseness or correctness relative to a reference output are difficult to express as typical unit tests. Using an “LLM-as-a-Judge” is a popular way to grade natural language outputs from LLM applications. This involves passing the generated output (and other information) to a separate LLM and asking it to judge the output. Although this has proven useful in several contexts, it raises an interesting problem: you now have to do another round of prompt engineering to make sure the LLM-as-a-Judge is performing well.LangSmith presents a novel solution to this rising problem. LangSmith evaluators now feature “self-improvement” whereby human corrections to LLM-as-a-Judge outputs are stored as few-shot examples, which are then fed back into the prompt in future iterations.💡The net impact is that it is easier to create