Jev is Vercel’s fastest-adopted AI model ever
Vercel says an AI model named Jev is its fastest-adopted model to date, according to data from its AI gateway, a unified API service comparable to OpenRouter.
#inference
85 articles tagged with “inference”.
Introducing Reflection Beam: an open-weight model for agentic coding on AWS
Reflection AI's Beam is an open-weight model for agentic coding and long-horizon tasks, with AWS guidance for deploying it on SageMaker JumpStart and HyperPod Inference.
A week with RTX Spark: everything we’ve learned using Nvidia’s new laptops
Nvidia's RTX Spark N1X, an Arm-based laptop SoC with RTX Blackwell graphics and large unified memory for running local AI models, ships in Windows laptops from six manufacturers.
Meta launches an open-weight AI model that runs on Macs
Meta has released Muse Spark Open, an open-weight AI model built in collaboration with Apple to run locally on Apple silicon.
Meta and Apple team up to make Muse AI models run better on Macs
Meta announced the Muse Spark Open AI model, calling it the most capable US open-weight model to date, and partnered with Apple so it can run on two M5 Ultra Mac Studios using the MLX framework.
Meta’s Muse Spark Open now available in Amazon SageMaker JumpStart
Meta's coding-optimized open-weight model, Muse Spark Open, is now available in Amazon SageMaker JumpStart and can be deployed to a SageMaker AI endpoint.
Open From Data Center to Desktop: NVIDIA Expands Nemotron Family With Highest Efficiency Super VL and …
NVIDIA released Nemotron 3.5 Super VL, an open 120-billion-parameter model, alongside two new Nemotron 3.5 Bolt edge models aimed at local agentic workloads.
Cerebras CEO says we’re nearing a future where everyone has a personal AI assistant at home
Cerebras CEO Andrew Feldman forecasts a future where personal AI assistants are as common as household appliances, operating partly on local devices and partly in the cloud.
All Systems Go: Global PC App Ecosystem Ready for NVIDIA RTX Spark
NVIDIA introduces RTX Spark, a superchip-based PC platform offering up to 128GB of unified memory and 1 petaflop of AI compute, with a ready global PC app ecosystem.
Accessing OpenAI GPT models on Amazon Bedrock with global cross-Region inference in Thailand, Malaysia…
AWS explains how to access OpenAI GPT-6 Sol, Luna, and Astra on Amazon Bedrock through global cross-Region inference from Singapore, Jakarta, Thailand, Malaysia, and Taipei.
Run multi-node inference workloads using Ray LLM Deep Learning Containers
AWS describes how to run multi-node LLM inference on Amazon EKS using pre-built Ray LLM Deep Learning Containers, splitting prefill and decode across two GPU nodes behind one OpenAI-compatible endpoint.
Microsoft’s new AI chip makes it less reliant on Nvidia’s GPUs
Microsoft announced Maia 300, its third-generation in-house AI chip built as its primary AI inference system, with production rollout slated for 2027.
Asana cuts model costs 76x in browser tests with GPT-6.1 Sol
Asana says it made its browser agent 76x cheaper and 5x faster in tests by using GPT-6 Astra in Codex, aiming to deliver more capable models to customers.
How Qlik built grounded, enterprise-scale AI with Amazon Bedrock
Qlik built Qlik Answers on Amazon Bedrock to deliver grounded, sourced answers across structured and unstructured enterprise data for its 40,000+ customers.
New agent skill: Amazon SageMaker optimized generative AI inference for your coding agent
AWS introduces the aws-ai-ml skill through the Agent Toolkit for AWS, enabling coding agents like Kiro, Claude Code, and Codex to generate SageMaker code for inference optimization and benchmarking.
How NVIDIA GPUs Help Accelerate OpenAI’s GPT-6 Astra Ultrafast
GPT-6 Astra Ultrafast, running on NVIDIA Blackwell GPUs, is now available in the OpenAI API and to eligible ChatGPT Work and Codex users, with up to 8x faster token generation than Astra Standard.
Amazon Bedrock expands Claude model availability to in-country inferencing in India
Anthropic's Claude Opus 5, Sonnet 5, and Haiku 4.5 are now accessible in India through Amazon Bedrock, with data processed within the India Regions.
Introducing Anthropic models on Amazon Bedrock for in-region inference in Seoul and Singapore
Amazon Bedrock now offers in-region inference for Anthropic's Claude models in Seoul and Singapore, keeping data processing within the called Region.
Build real-time voice applications with vLLM-Omni on SageMaker AI – Part 1
AWS walks through deploying the Qwen3-TTS model on SageMaker AI using the vLLM-Omni Deep Learning Container, streaming generated speech over a persistent bidirectional connection via a Gradio application.
Introducing GPT-6 Sol and Luna
OpenAI has announced GPT-6 Sol and Luna, a pair of models aimed at everyday work, each with a different trade-off between capability and cost.
OpenAI launches GPT-6 Sol and Luna, boasting lower cost and fewer mistakes
OpenAI introduced two new models, GPT-6 Sol and Luna, which it says deliver lower cost and fewer mistakes and are related to its Astra model.
xAI’s Grok 4.6 is now available in Amazon Bedrock
xAI's Grok 4.6 is now available in Amazon Bedrock, featuring a 500K token context window, four reasoning effort levels, and support for long-running agents, coding, and knowledge work.
Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem
A Hugging Face blog post explores pruning large language models by treating block removal as an Ising optimization problem, drawing on a physics-inspired framing.
Amazon SageMaker Inference: 2026 year-to-date launches in review
AWS reviews 13 inference launches shipped year-to-date in 2026 for Amazon SageMaker AI, spanning managed endpoints and SageMaker HyperPod Inference.
Introducing Kimi K3 on Amazon Bedrock
Moonshot AI's Kimi K3 is now available on Amazon Bedrock as an open-weight option for coding and knowledge work, featuring native vision, a 1M-token context window, and prompt caching.
Introducing Amazon SageMaker HyperPod Inference Gateway
AWS introduced SageMaker HyperPod Inference Gateway, a Kubernetes-native, GPU-aware routing add-on for Amazon EKS that routes inference requests using real-time GPU signals.
Huawei plans Q1 2027 launch of new AI chip as it takes on Nvidia
Huawei is accelerating the launch of its Ascend 960DT AI chip, targeting a Q1 2027 release as it seeks to compete with Nvidia and close China's AI computing gap with the U.S.
Kimi-maker Moonshot AI targets $2 billion in annual revenue
Moonshot AI, maker of the Kimi models, is aiming for $2 billion in annual revenue, with OpenRouter data showing up to 300 billion tokens generated daily by K3 models despite a slight usage decline.
Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference
Amazon SageMaker Inference introduces prefix-aware routing, directing requests with shared prompt prefixes to the same instance to keep KV caches warm and reduce latency.
Reduce inference cold starts on Amazon SageMaker HyperPod with model caching
SageMaker HyperPod now supports model caching for inference, pre-loading model weights and container images onto cluster nodes so pods read from local NVMe instead of downloading over the network.
OpenAI puts Pro subscriptions on hold due to Astra demand
OpenAI has temporarily halted new Pro subscription sign-ups, saying Pro tiers strain its systems most, and is pausing while it expands capacity.
d-Matrix Adopts NVIDIA NVLink Fusion for Rack-Scale XPU Deployment
d-Matrix announced it will adopt NVIDIA NVLink Fusion to link its next-generation Raptor XPUs to NVIDIA's scale-up and scale-out networking and MGX rack architecture.
Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM
AWS details how to deploy the open-weight Qwen3.8-2.4T-A95B model on Amazon SageMaker HyperPod with vLLM, covering cluster setup, quantization, and an OpenAI-compatible endpoint.
ICYMI: What landed for AI builders in August 2026
AWS summarizes its August 2026 releases for AI builders spanning Amazon Bedrock, Bedrock AgentCore, and Strands, including million-token context, cross-Region inference, and long-running agents.
Take on your most ambitious work with GPT-6 Astra on Amazon Bedrock
OpenAI's GPT-6 Astra is generally available on Amazon Bedrock, offering deeper reasoning on Bedrock's inference engine built for performance, security, and scale.
Run agent-driven Amazon SageMaker HyperPod operations with InstantStart
AWS introduced HyperPod InstantStart, an open source control plane combining Amazon EKS with SageMaker HyperPod to run guarded operations via a web interface and an AI agent.
Microsoft’s Project Zenith is a ‘distraction-free Windows experience’ for developers
Microsoft has named its developer-focused Windows experience Project Zenith, built for new devices with 64GB or more of unified memory and preconfigured for local AI model work.
Nvidia launches free tool that links idle computers into a personal AI data center
Nvidia announced Personal AI Router (PAIR), free open-source software that discovers and connects compatible PCs on a network to run local AI inference and agentic workflows.
Accessing OpenAI models on Amazon Bedrock from Australia with global cross-Region inference
Amazon Bedrock now offers Australian teams access to OpenAI GPT-5.6 Sol, Terra, and Luna models with global cross-Region inference from the Sydney and Melbourne Regions.
Google says its new Gemini 3.8 Flash model ‘works harder’ but might cost more
Google released Gemini 3.8 Flash weeks after 3.7 Flash, claiming it 'works harder' with more reasoning and iterative tool calls, though higher token use could raise costs.
Anthropic launches Claude Fable 5.1 and says it’s up to 45 percent cheaper for agentic work
Anthropic released Fable 5.1 and Mythos 5.1, saying the new models cost less—up to 45 percent cheaper for complex agentic tasks—while improving performance and addressing customer concerns.
Anthropic’s new Fable release is cheaper, less restrictive
Anthropic released Fable 5.1, an update focused on reducing token cost and limiting false-positive restrictions from the model's safety mechanisms.
GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundati…
Microsoft Research presents GigaPath-Flash and GigaTIME-Flash, pathology foundation models that reduce computational demands while maintaining strong performance to enable larger-scale studies.
How Decathlon runs demand forecasting at scale with Chronos-2
Sporting goods retailer Decathlon uses Chronos-2 on AWS to forecast weekly demand across tens of thousands of products, reporting accuracy gains of 11-15 points and low-cost CPU inference.
OpenAI’s Jalapeño chip is built for fast inference at scale, benchmarks show
OpenAI's Jalapeño chip, designed for fast inference at scale, outperformed current state-of-the-art on tokens per user and throughput per kilowatt in SemiAnalysis' InferenceX benchmark.
OpenAI says its Jalapeño chip can power faster AI responses than the competition
OpenAI says its Jalapeño chip, an ASIC built with Broadcom for AI inference, delivers lower latency and higher throughput than rival systems.
The full stack behind abundant intelligence
OpenAI's CFO describes how progress across chips, compute, models, and products compounds to deliver more useful intelligence at larger scale and lower cost.
Introducing new Ray capabilities on SageMaker HyperPod
SageMaker HyperPod now offers managed Ray support on Amazon EKS, letting users create and monitor Ray clusters, connect notebooks, and run distributed training and accelerated inference.
With Groq 3 LPX in Full Production, NVIDIA Extends Vera Rubin Inference for Agents
NVIDIA announced an extension of its Vera Rubin NVL72 rack-scale system aimed at faster token generation for agentic AI systems, stressing coordination across all layers of the AI factory.
Reduce RAG costs on Amazon Bedrock with query-aware compression
AWS describes a query-aware compression approach on Amazon Bedrock that uses a smaller model to filter retrieved chunks before answering, reducing input tokens and cost in RAG.
Introducing cross-Region inference for OpenAI GPT-5.6 models on Amazon Bedrock
Amazon Bedrock now supports OpenAI GPT-5.6 models with cross-Region inference in more than 25 AWS Regions, routing requests for higher throughput via US geographic and global inference profiles.
Up to 3.2x Faster Inference with LFM2.5-DSpark
The introduction of LFM2.5-DSpark enhances inference speed by up to 3.2 times, benefiting developers in AI.
How Much Memory Does Your Agent Actually Need?
This article discusses the memory demands of AI agents, emphasizing the importance of optimizing memory usage for enhanced performance.
NVIDIA Nemotron 3.5 Lightning now available in Amazon SageMaker JumpStart
NVIDIA Nemotron 3.5 Lightning, an open 30B Mixture-of-Experts model with 3B active parameters, is now available in Amazon SageMaker JumpStart for high-volume agentic workloads.
Building agentic workflows with SageMaker AI and Bedrock AgentCore
AWS describes building multi-agent workflows using OpenAI-compatible SageMaker AI endpoints and Bedrock AgentCore runtime, with each agent using a model suited to its task.
Writer introduces new AI model and upgraded harness to contain token costs
Writer introduced a new AI model and an upgraded harness designed to contain token costs, built as a post-training variation on Z.ai's open source GLM-5.2 model.
Previewing Ultrafast mode: GPT-5.6 Sol at up to 14X the speed
OpenAI is previewing Ultrafast, an API service tier that runs GPT-5.6 Sol up to 14x faster, reaching up to 750 output tokens per second using Cerebras hardware.
Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine
An AWS blog post details building a tiered KV cache on Amazon SageMaker HyperPod with Curvine, extending the cache into a shared NVMe pool so replicas reuse cache on cost-efficient instances.
NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI
NVIDIA has added Nemotron 3.5 Lightning to its Nemotron 3 model family, alongside NeMo Switchyard, aimed at efficient, long-running agentic AI workloads with open models.
Anthropic is hiring an AI chip design team
Anthropic, the maker of Claude, is building a team to design its own custom AI chips, aiming to co-design hardware and models for improved speed and efficiency.
As AI Increases Demands on Memory, Storage Steps Up
NVIDIA highlights how growing AI demands push datasets and context windows past system memory, requiring efficient, secure storage architectures rather than added capacity alone.
Deploy local agents everywhere with LFM2.5-2.6B
A Hugging Face Blog post introduces LFM2.5-2.6B, presented as a way to deploy local AI agents across environments.
How we built a realtime system for responsive voice AI in six months
OpenAI details GPT-Live, a realtime voice AI system built over six months that uses a turnless speech model and low-latency architecture for continuous, more natural voice interaction.
Building abundant intelligence
OpenAI describes a full-stack strategy intended to make advanced AI more capable, more affordable, and more broadly useful.
Introducing explicit prompt caching for OpenAI GPT-5.6 models on Amazon Bedrock
OpenAI GPT-5.6 Sol, Terra, and Luna are generally available on Amazon Bedrock, now with explicit prompt caching that lets users choose which prompt parts to cache and reuse.
Advancing the price-performance frontier with GPT-5.6
OpenAI announced lower GPT-5.6 pricing for its Luna and Terra offerings, positioning more efficient models to support enterprise AI workflows at scale.
How enabling two settings tripled our scores on the ARC-AGI-3 benchmark
OpenAI describes how enabling two API settings that retain reasoning and turn on compaction tripled GPT-5.6's scores on the ARC-AGI-3 benchmark while improving efficiency.
The OlmoEarth Platform: Geospatial inference at planetary scale
The OlmoEarth platform is designed for geospatial inference, enabling advanced data analysis across the globe.
Satya Nadella says companies that trust one AI for everything may not survive
Satya Nadella cautions that businesses depending on one AI for everything risk trouble unless they build their own models or use AI gateways to separate prompts from the underlying model.
Introducing Claude Opus 5 on AWS: Anthropic’s most capable Opus model
AWS introduces Claude Opus 5, Anthropic's most capable Opus model, with practical guidance for engineers building agentic systems and production inference workloads on Amazon Bedrock.
Get started with OpenAI GPT-5.6 Sol, Terra, and Luna on Amazon Bedrock
Three OpenAI GPT-5.6 models—Sol, Terra, and Luna—are now generally available on Amazon Bedrock, with support for the Responses API, prompt caching, and the Codex coding agent.
Google releases three new Gemini models — but no 3.5 Pro
Google has released Gemini 3.6 Flash, 3.5 Flash-Lite, and Flash Cyber, while notably not releasing a Gemini 3.5 Pro model.
NVIDIA Vera Rubin Driving Performance Per Watt, Lowest Token Cost for Partners Worldwide
NVIDIA's Vera Rubin is moving into production, with NVL72 racks deployed at major cloud partners and backed by a supply chain spanning over 350 factory sites in 30 countries.
NVIDIA Vera Rubin Maximizes Intelligence per Dollar for Post-Training Workloads — a Key Metric for Age…
NVIDIA's Vera Rubin is positioned to reduce cost per token for post-training workloads through codesign, maximizing intelligence per dollar for agentic AI.
Why the first GPU financiers are turning to inference chips in a $400 million deal
A $400 million chip-backed loan highlights a move by early GPU financiers toward inference chips as part of the next wave of AI infrastructure financing.
NVIDIA Introduces New Jetson Thor Computers to Advance Mainstream Robotics and Edge AI
NVIDIA unveiled the T3000 and T2000 modules based on its Thor architecture, aimed at bringing compact, power-efficient AI computing to mass-market robotics and edge AI.
Why Performance per Watt Is the Ultimate Metric for AI Infrastructure Efficiency
NVIDIA frames performance per watt as the defining efficiency metric for AI infrastructure, arguing power is the main constraint on how many tokens an AI factory can generate.
OpenAI GPT-5.6 Sol, Terra, and Luna are now generally available on Amazon Bedrock
OpenAI's GPT-5.6 Sol, Terra, and Luna models are now generally available on Amazon Bedrock, accessible through its inference engine.
Launching UI for generative AI inference recommendations in Amazon SageMaker AI
AWS has launched a UI in Amazon SageMaker AI Studio that guides users to optimized generative AI inference configurations without deep infrastructure expertise.
How LLM Inference Works: Tokens, KV Cache, and Why It Costs What It Costs
Every word an LLM generates has a cost in compute, memory, and time. Here is what actually happens during inference — and why it explains latency, throughput, and per-token pricing.
Deploying quantized models on Amazon SageMaker AI with Unsloth
An AWS blog post explains four deployment patterns for serving Unsloth-quantized models on AWS infrastructure, covering EC2, SageMaker AI endpoints, EKS, and ECS.
Enhancing enterprise inference on Amazon SageMaker HyperPod with data capture, Hugging Face, NVMe, and…
AWS details five new inference capabilities for SageMaker HyperPod, including multi-tier data capture, direct Hugging Face Hub deployment, NVMe model loading, Route 53 DNS, and pod-level IAM.
GPT-5.6: Frontier intelligence that scales with your ambition
OpenAI introduces GPT-5.6, a frontier model it says delivers more intelligence per token, better performance per dollar, and additional capacity for demanding tasks.
Inside the AI Chip Race: Why Compute Became the Real Bottleneck
Modern AI runs on a scarce resource: specialized compute. This feature unpacks why GPUs became the bottleneck, why everyone is building custom chips, and what it means for the balance of power in AI.
Small Language Models: Why Smaller Is Suddenly Smarter
The race is no longer only about bigger models. Small language models that run cheaply — even on a phone — are one of AI's most important trends. Here is why, and what to watch.