Manav Singhal

A Survey of Agent Evaluation Frameworks: Benchmarking the Benchmarks

A Survey of Agent Evaluation Frameworks: Benchmarking the Benchmarks

In recent months, we've witnessed an explosion in the development of AI agents. Autonomous systems powered by large language models (LLMs) can perform complex tasks through reasoning, planning, and tool usage. However, as the field rapidly advances, a critical question emerges: how do we effectively measure and compare

OpenAI’s BrowseComp: Redefining How We Benchmark Web-Browsing Agents

OpenAI’s BrowseComp: Redefining How We Benchmark Web-Browsing Agents

As language models become increasingly agentic, including browsing the internet, reasoning across sources, and acting on user instructions, it is imperative that our methods of evaluating their capabilities must evolve too. OpenAI’s BrowseComp introduces a fresh benchmark for this paradigm, offering a challenging, realistic, and carefully curated evaluation framework

Agent Evaluation: Metrics for Evaluating Agentic Workflows

Agent Evaluation: Metrics for Evaluating Agentic Workflows

This is Part 2 of our Agent Evaluations series. Here are Part 1 and Part 3 in this series. As AI agents start to gain traction across industries, driving innovation in tasks ranging from customer support to automation of tasks like booking requires their real-world performance evaluation to go beyond

Agent Evaluation: Understanding Agentic Systems and their Quality

Agent Evaluation: Understanding Agentic Systems and their Quality

This is Part 1 of our Agent Evaluations series. Here are Part 2 and Part 3 in this series In today’s rapidly advancing world of artificial intelligence (AI), agentic systems are becoming an integral part of numerous industries, powering everything from customer support to robotics. But what exactly are

BrowserGym: Technical deep dive into web agent automation

BrowserGym: Technical deep dive into web agent automation

The field of web automation faces significant challenges in standardizing agent development and evaluation. BrowserGym, a Gym environment for web automation tasks by ServiceNow, addresses these challenges by providing a unified framework that standardizes the development, testing, and evaluation of web agents. In addition, they also design AgentLab, a complementary

Image generated using Meta AI

Ensuring responsible AI: An overview of DeepMind’s FACTS framework

Introduction DeepMind introduces the FACTS Grounding, an online leaderboard and benchmark designed to evaluate the factual accuracy of language models' long-form responses based on a given context in user prompts. The benchmark requires models to generate text grounded in a provided document, which can be up to 32,000

Image credit: Meta

Innovative training of LLMs in continuous latent spaces, by Meta AI

LLMs have made significant progress in tasks based on language processing and understanding. However, their reasoning capabilities, particularly in complex scenarios, often fall short. Traditional approaches like Chain-of-Thought (CoT) reasoning, which asks the model to reason before answering by thinking step-by-step, face inherent limitations. A recent paper introduces a novel