Imagine having an assistant that can manage complex research tasks, interpreting conflicting information, sourcing data from across the web, and synthesizing it into actionable insights. That\u2019s what large language models (LLMs) have been evolving into. No longer limited to answering simple factual queries, developers are marketing them as tools adept at carrying out \u201cdeep research.\u201d And in the AI world, this capability seems to have a lot of names. OpenAI brands it as \u201cDeep Research,\u201d Anthropic prefers \u201cExtended Thinking,\u201d for Google\u2019s Gemini, it’s \u201cSearch + Pro,\u201d and Perplexity uses phrases like \u201cPro Search\u201d and \u201cDeep Research.\u201d<\/p>\n
A FutureSearch<\/a> study called the Deep Research Bench (DRB)<\/a> gave these systems a thorough examination, providing the most comprehensive insights yet.<\/p>\n The Deep Research Bench (DRB) developed by FutureSearch is an evaluation tool designed to assess how well AI agents can handle complex, web-based research tasks. Think of it as a simulated arena for the vexing problems encountered by researchers, analysts, and decision-makers in the real world. The benchmark includes an array of real-world tasks, like “Find Number” problems or tasks that require to “Validate Claim” or “Compile Dataset.”<\/p>\n To make comparisons fair and consistent, human-verified answers accompany each task, and RetroSearch, a static archive of web pages, is employed. This tactic eliminates the unpredictable nature of live web data, providing an even footing for different AI agents.<\/p>\n In the heart of DRB is the ReAct (Reason + Act) architecture. It replicates how a human researcher operates\u2014contemplating a problem, taking action such as a web search, scrutinizing the results, and then iterating. Although newer LLMs have integrated this loop into a smoother process, the ReAct setup still offers a valuable structure for AI reasoning. For tasks like \u201cGather Evidence,\u201d RetroSearch, used in DRB, includes up to 189,000 web pages, all frozen in time to ensure repeatability, thanks to tools such as Serper<\/a>, Playwright<\/a>oraz ScraperAPI<\/a>.<\/p>\n When it came to performance, OpenAI\u2019s o3 led the pack with a score of 0.51 out of 1.0. This might seem low, but considering the benchmark’s complexity, it’s a significant achievement. The researchers estimate that even a perfect agent would likely top out around 0.8 due to the inherent vagueness in task definitions and scoring.<\/p>\n Trailing close behind were Anthropic\u2019s Claude 3.7 Sonnet and Google\u2019s Gemini 2.5 Pro; Claud demonstrated adeptness in structured and flexible thinking, while Gemini excelled in step-by-step planning tasks. Interestingly, DeepSeek-R1 exhibited performance almost matching GPT-4 Turbo indices, indicating a closing gap between open and closed-source models.<\/p>\n Despite enormous progress, AI models do struggle with certain aspects. Memory degradation is a significant issue; as tasks become longer, models tend to overlook crucial details, lose sight of targets, and provide disjointed or irrelevant responses. Other common weaknesses include repetitive tool use, unproductive search queries, and jumping to premature conclusions.<\/p>\n Even the top-performing models suffer certain susceptibilities. For example, GPT-4 Turbo often drops earlier steps from memory, and DeepSeek-R1 has a tendency to generate false yet plausible-sounding insights. A common fault across all models is their frequent failure to confirm findings or cross-reference sources, which is critical in serious research tasks.<\/p>\n The report also looked at \u201ctoolless\u201d agents\u2014language models that rely merely on their internal training data, devoid of any external tools like search tools. Surprisingly, these agents managed almost as well as tool-enabled ones on some tasks. This finding suggests that some LLMs have robust internal priors and can efficiently judge the plausibility of common claims. However, their limitations become transparent on more challenging tasks where-up-to-date, exhaustive information is indispensable.<\/p>\n The comprehensive Deep Research Bench report underscores one thing: while today\u2019s AI agents are gaining ground, they are still playing catch-up with skilled human researchers, especially in tasks that demand strategic planning, flexible thinking, and subtle reasoning.<\/p>\n These gaps become particularly noticeable during longer or more complex research sessions, where agents often lose coherence or wander off track. Yet, DRB\u2019s beauty lies in its ability to evaluate not just rudimentary knowledge but also the deeper interplay of memory, reasoning, and tool use. As LLMs continue integrating into professional workflows, tools like the DRB from FutureSearch<\/a> will be crucial for measuring real-world AI performance.<\/p>\nEvaluating AI’s Research Prowess: Deep Research Bench and its Findings<\/h3>\n
The Hurdles Hitherto Unsurmounted: Where AI still lags<\/h3>\n