Manav Singhal

VGBench: Evaluating Vision-Language Models in Real-Time Gaming Environments

VGBench: Evaluating Vision-Language Models in Real-Time Gaming Environments

Introduction Vision-Language Models (VLMs) have achieved remarkable success in tasks such as coding and mathematical reasoning, often surpassing human performance. However, their ability to perform tasks that require human-like perception, spatial navigation, and memory management remains underexplored. To address this gap, the paper titled "VideoGameBench: Can Vision-Language Models complete

Base vs. Aligned: Why Base LLMs Might be Better at Randomness and Creativity

Base vs. Aligned: Why Base LLMs Might be Better at Randomness and Creativity

Introduction As large language models (LLMs) continue to improve in tasks ranging from education to enterprise automation, alignment techniques like Reinforcement Learning from Human Feedback (RLHF) have become the standard. These methods make models safer, more helpful, and generally better at following instructions. However, recent findings challenge the assumption that

From Turn 1 to Turn 10: How LLMs Get Lost In Multi-Turn Conversations

From Turn 1 to Turn 10: How LLMs Get Lost In Multi-Turn Conversations

Real-world interactions between humans and LLMs are rarely single‑shot. Rather, users start with vague requests, iterate, clarify, and refine over multiple turns. Yet, most LLM benchmarks assume a fully‑specified, single‑turn setting which is different from how people actually chat. Prior analyses of conversation logs confirm that underspecification

SuperBPE: Rethinking Tokenization for Language Models

SuperBPE: Rethinking Tokenization for Language Models

In the domain of language models, tokenization i.e. the process of breaking down text into manageable units plays a pivotal role. Traditionally, models rely on subword tokenization, where words are split into smaller units. However, this approach often overlooks the semantic significance of multi-word expressions and varies across languages.

Can We Trust What AI Models Say They're Thinking? A Deep Dive into Chain-of-Thought Faithfulness

Can We Trust What AI Models Say They're Thinking? A Deep Dive into Chain-of-Thought Faithfulness

Chain-of-Thought (CoT) based reasoning has exploded across the AI landscape. Modern large language models (LLMs) like Claude 3.7 Sonnet and DeepSeek R1 no longer just give answers but also generate natural language explanations that walk through their decision-making process. This transparency isn’t just about UX but it has

The Era of Experience: Vision for the Next Frontier in AI

The Era of Experience: Vision for the Next Frontier in AI

In this recent paper, The Era of Experience, David Silver and Richard Sutton articulate a vision for artificial intelligence where there is a shift from reliance on static, human-generated data to dynamic, self-generated experiential learning. This paradigm aims to propel current models beyond their limitations, developing systems capable of continuous

Building Robust Evaluation Workflows for AI Agents

Building Robust Evaluation Workflows for AI Agents

Through the first two blogs (Part 1 and Part 2) of the AI agent evaluation series, we explored AI agents and the key performance metrics for evaluating them. Now, we focus on building end-to-end evaluation workflows. A structured AI evaluation process encompassing both pre-release and post-release phases is crucial for