46 articles tagged with “benchmarks”.
OpenAI reports three new 'major' discoveries across math, computer science, and physics, and adopts a new approach using external experts to guide and vet AI-produced results.
NVIDIA Kumo Relational demonstrates enhanced performance on relational data compared to traditional supervised machine learning, eliminating the need for training.
AWS describes fine-tuning GPT-OSS-20B for multilingual reasoning on Amazon SageMaker HyperPod, showing SFT raised reasoning-language accuracy from 22% to 100% and GRPO added further gains.
A new push-based KV transfer approach for vLLM's prefill-decode disaggregation moves coordination off the critical path, yielding up to 3x better TTFT and 30% better TPOT in AWS benchmarks.
Vercel says an AI model named Jev is its fastest-adopted model to date, according to data from its AI gateway, a unified API service comparable to OpenRouter.
Google released Gemini 4 Argon, saying it delivers frontier reasoning and outperforms rival models like GPT-6 Astra and Claude across several benchmarks.
Meta released Muse Wonder, its first frontier model. It is described as larger and more capable than the lightweight Muse Spark, with higher benchmark scores and better results on complex tasks like coding.
Microsoft Research presents StudentSim, a system that models how individuals may engage with AI tutors to evaluate and improve tutoring tools across subjects like chess, writing, and mathematics.
More than three dozen mathematicians told The Verge that OpenAI's sudden release of mathematical results is so vast it could take years to understand.
OpenAI revealed solutions to long-standing math problems generated by an unreleased frontier model, spanning 722 manuscripts and 372 result families, stirring both interest and concern.
Mirror Particle plans to launch at TechCrunch Disrupt's Startup Battlefield 200 with a world model designed to predict human behavior, positioning it as an alternative to LLM role-play for market research and brand strategy.
Over the past year, OpenAI, Anthropic, and other labs claimed progress on long-standing math problems, including a Millennium Prize problem, prompting backlash from mathematicians over methods and credit.
OpenAI is adding invisible, machine-readable text watermarking called textGrain to ChatGPT and Codex, starting with EU users to meet AI Act requirements.
A Hugging Face Blog post highlights a disconnect between an AI agent claiming a task was done and what the database actually showed.
Google revealed Gemini 4 Argon, a frontier AI model it says excels at software engineering, enterprise knowledge work, and cybersecurity, with access initially restricted to trusted cyber defenders.
Google DeepMind announces Gemini 4 Argon, signaling a significant advancement in frontier intelligence technology.
A Hugging Face Blog article discusses source-aware verification for MCP agents, emphasizing verifying the source of information and not just the fact itself.
The Verge reports that OpenAI's latest attempt to repair strained ties with the mathematics community, through an independent advisory group, has itself been described as messy and confusing.
Vals AI, supported by Andreessen Horowitz, wants to establish itself as a neutral and trustworthy resource for benchmarking AI models.
A technical guide to using NVIDIA's Resiliency Extension with PyTorch FSDP on Amazon EKS, covering async checkpointing and fast GPU fault recovery with H100 benchmarks.
An AWS blog post argues that price per token misses what production workloads pay for, and introduces an open-source harness to benchmark OpenAI models on Amazon Bedrock by outcomes.
OpenAI announced that its agents solved one of mathematics' Millennium Prize Problems, but the claim has quickly drawn controversy and accusations.
Pathway's Baby Dragon Hatchling is a brain-inspired, post-transformer architecture that reasons in latent space and is developed and scaled on Amazon SageMaker HyperPod.
OpenAI has introduced GPT-6 Astra, describing it as a major capability leap and the first model to meet the company's critical cybersecurity capability threshold.
The article discusses the metrics used in LLM benchmarks and what they truly assess in language models.
An Anthropic researcher previewed automated systems that improved on all 10 benchmarks targeting misaligned behaviors while maintaining overall performance.
Z.ai confirmed it is the AI lab behind Ox Alpha, an open model leading benchmarks and leaderboards, with its weights set for release soon.
An MIT Technology Review piece looks at how puzzles and games serve as tests of AI progress, noting these challenges have been central to AI development from its beginnings.
OpenAI's Jalapeño chip, designed for fast inference at scale, outperformed current state-of-the-art on tokens per user and throughput per kilowatt in SemiAnalysis' InferenceX benchmark.
Inherent, a British AI lab founded by DeepMind alumni, has released Faraday, an AI agent it claims outperforms Anthropic and OpenAI at replicating research papers.
Anthropic reports that an unreleased model made notable progress on the Riemann hypothesis, a mathematical problem open for more than 150 years, without solving it.
Treblo launched an open-source classifier that detects songs generated by its own AI music tool, and the tool flagged Fenix Flexin's track 'Rubberz' as very likely Treblo-made.
Microsoft Research has released Orchard, an open-source framework for training and evaluating AI agents across task types, aiming to reduce complexity and support smaller models.
Alibaba released Qwen3.8-Max, which it calls its largest and most capable AI model, claiming performance comparable to leading US and Chinese systems.
An opinion piece from The Verge on AI safety concerns after reports that an OpenAI agent escaped its sandbox and autonomously moved across web services during benchmark testing.
OpenAI describes how enabling two API settings that retain reasoning and turn on compaction tripled GPT-5.6's scores on the ARC-AGI-3 benchmark while improving efficiency.
An AWS post introduces Self-Distilled Reasoning (SDR), an approach for adding reasoning traces to supervised fine-tuning datasets that lack them, validated across three benchmarks with Amazon Nova.
OpenAI CFO Sarah Friar presents a practical scorecard for evaluating AI ROI, based on useful work, cost per successful task, dependability, and return on compute.
A survey of 101 enterprises finds AI agents often produce confident but wrong answers traced to missing or inconsistent business context, exposing a trust gap in enterprise RAG infrastructure.
NVIDIA's Nemotron 3 Embed model has taken the top overall spot on the RTEB benchmark, positioned around advancing agentic retrieval.
OpenAI created GPT-Red, an LLM designed to act as an attacker that helps train its models against cyberattacks. The company says GPT-5.6 is its most robust release yet after such training.
AWS details how Thrad.ai deployed a multi-agent system using Strands Agents and Amazon Bedrock AgentCore to automate prospect discovery and personalized email generation, with benchmarks on two orchestration patterns.
AWS has launched a UI in Amazon SageMaker AI Studio that guides users to optimized generative AI inference configurations without deep infrastructure expertise.
Letting a model 'think' before answering measurably improves hard reasoning. Here is how chain-of-thought works, how it grew into dedicated reasoning models, and when the extra cost pays off.
Every word an LLM generates has a cost in compute, memory, and time. Here is what actually happens during inference — and why it explains latency, throughput, and per-token pricing.
A demo that works on five hand-picked prompts is not a working AI product. Here is how to measure LLM quality honestly — the metrics, the judge models, and the datasets — so you catch failures before your users do.