AI PCs: The NPU, Emblem of a Market Still Finding Its Way

“AI PCs? No, I don’t know what that is.”

We heard it—quite often—during the B2B days at the Viva Tech trade show. We chose the event to “take the temperature” of this concept as old as ChatGPT or thereabouts.

A few weeks after the chatbot’s launch, CES 2023 indeed gave a first push. For example with the Ryzen AI announcement. The movement accelerated in 2024 when Microsoft launched the label “Copilot+ PC.” To benefit from it, a machine must, in particular, carry a sufficiently powerful NPU.

AMD, Intel, Qualcomm: The TOPS Battle

NPUs (Neural Processing Units) are processors specialized in the multiplications of vectors and matrices—fundamental operations inside neural networks. They perform these tasks more efficiently than GPUs, but at the cost of reduced flexibility and, above all, a tooling ecosystem that is still far less mature.

Their performance is traditionally reported in TOPS. That is, thousands of billions of operations per second in simple precision (8-bit integers), whereas GPUs typically work in floating-point. The Copilot+ PC label sets a threshold of 40 TOPS. In the beginning, only Qualcomm hit it, with its Snapdragon X chips. AMD and Intel would align a few months later with, respectively, Strix Point and Lunar Lake-V.

AMD at 60 TOPS with the Ryzen AI 400

The Strix Point family was announced in July 2024. It mainly comprises the Ryzen AI 300 line, aimed at notebooks, with a base TDP of 28 W. AMD integrated the second generation of its XDNA NPUs, based on Xilinx technology (acquired in 2022). They market them at 50 TOPS. With the Ryzen AI 400 (Gorgon Point, 28 W also), announced in January 2026, the threshold rose to 60 TOPS.

Intel at 45 TOPS with the Core Ultra 200V

The Lunar Lake-V family, launched in September 2024, also targets notebooks (17-30 W), under the commercial name Core Ultra 200V. Intel integrated a 45 TOPS NPU, also the outcome of a (Movidius, 2016) acquisition. It remains for now an exception in its catalog. Its other Core Ultra Series 2 processors stay at:

• 12 TOPS NPU for Arrow Lake-U (15 W)
• 13 TOPS NPU for Arrow Lake-H (28-45 W)
• 13 to 36 TOPS NPU for Arrow Lake-S (35-125 W)

Qualcomm at 45 TOPS with Snapdragon X… and 85 with X2

With the Snapdragon X (Elite announced in October 2023, then Plus in April 2024), Qualcomm is also at 45 TOPS, for base TDPs between 23 and 35 W. Since then, it moved to 80 with the Snapdragon X2 Plus (January 2026) and 85 with the X2 Elite (September 2025).

Recap of the latest mobile processor generations:

Unified Memory, Adopted but Not Yet Standardized on AI PCs

Beyond raw NPU power, running AI models depends on memory quantity as well as bandwidth. To boost this bandwidth, unified architectures have emerged: similar to Apple Dawn Liphardt, RAM sits on the SoC and is shared among all processors. AMD implemented this on its Halo chips; not on Point. Intel has only integrated it in Lunar Lake-V… perhaps Razor Lake, aimed at competing with Halo.

Qualcomm’s Snapdragon X all feature unified memory. Over generations, theoretical bandwidth grows with the adoption of faster memory. The X Plus and Elite, with their LPDDR5X-8448, reach 135 Go/s on paper. The X2 moves to 152 Go/s with its LPDDR5X-9523. The X2 Elite Extreme even does better (228 Go/s) thanks to a 192-bit bus.

As for memory capacity, AMD set the bar at 128 GB on Strix Halo, 192 GB on Gorgon Halo, and 256 GB on Strix Point / Gorgon Point. Intel tops at 32 GB for Lunar Lake-V and 96 GB on Arrow Lake. Qualcomm sits at 32 GB for the X Plus, 64 GB for the X Elite, and 128 GB for the others.

For all these platforms, the amount of memory that can be used as VRAM depends on BIOS and operating systems.

Summary:

OpenVINO, Ryzen AI, QAIRT… As Many Toolkits as NPUs

What varies a lot too is the programming model for the different NPUs: each has its own toolbox, whereas a “GPU” is largely synonymous with “CUDA.”

Intel includes NPUs alongside CPUs, GPUs and FPGAs in the OpenVINO toolkit. This kit has been extended with models tailored to generative tasks (quantization, key-value cache encapsulation, speculative execution…). Intel also ships a catalog of ready-to-use models for its NPUs. You’ll find classic ML models (MobileNet for object detection, AlexNet for image classification…) and LLMs (generally under 10B: Gemma 7B, Llama-3.2-1B, Phi 2…). In addition to Windows compatibility, a Linux driver is available from kernel 6.6.

AMD’s Ryzen AI Software stack (for Windows and Linux) relies on the ONNX runtime, also equipped with GenAI extensions. The Lemonade SDK offers an abstraction layer, with a Python API and a REST server interface. At a lower level, there is a C++ API enabling hybrid execution of generative models (using the NPU for pre-filling and the GPU for decoding). AMD also provides a catalog of ready-to-use models. Some are optimized for long context windows (16k), others for performance (at the cost of a 4k window). The Lemonade server can attach a backend dedicated to the XDNA NPUs: FastFlowLLM. Linux support was added recently.

Qualcomm develops its NPUs under the Hexagon brand, a progression from its DSP lineage since 2007. The move is not insignificant: for Qualcomm, the former are descendants of the latter. It’s simply “the architecture that changed” (fusion of scalar, vector and matrix compute units, with shared memory). The QAIRT (Qualcomm AI Runtime) toolkit handles programming: at the lowest level, C and C++ interfaces; at the highest level, a Neural Processing SDK. In between, a more granular SDK with accelerator-specific libraries, but still abstracting network partitioning. An Amazon SageMaker bridge exists for fine-tuning before local deployment.

Foundry on Windows, the Promise of a Common Foundation for Local AI

Microsoft is trying to unify things with Foundry on Windows (formerly Copilot Runtime). Under this banner, it offers three local-AI options: APIs, ready-to-use LLMs, and the Windows ML inference framework.

The APIs—about ten—enable speech recognition, OCR, image processing (description, segmentation, upscaling…), and access to Microsoft’s Phi Silica LLM—with optimizations for NPUs, such as request compression. Primarily reserved for Copilot+ PC, they have begun opening to CPUs (speech-to-text and video upscaling) and to GPUs (Phi Silica). The corresponding models are downloaded at runtime and then shared among applications. These apps are not required to bundle runtimes or drivers.

The second option, Foundry Local, enables fine-tuning Phi Silica with LoRA. It also provides access, including on Windows 10 PCs, to a handful of open LLMs pre-optimized. Some are designed for NPUs; these are mainly Phi models (3 Mini, 4 Mini, 4), Qwen (Coder, Instruct) and DeepSeek (R1-Distill-Qwen-7B). Others run on (Web)GPU and possibly on CPU (Ministral, Nemotron, Olmo…).

Windows ML is a distribution of the ONNX Runtime, succeeding the DirectML API. It automates chip selection for inference, retrieval of execution providers (backend abstractions) and keeps everything up to date via Windows Update. The execution of generative models is officially still in preview.

Strengths and Weaknesses of NPUs

In 2024, shortly after the first Copilot+ PC launches, a USC study on an Intel SoC highlighted the strengths and weaknesses of NPUs. It compared them against CPUs and GPUs across two tasks. On one side, linear algebra (matrix, scalar, and matrix-vector products). On the other, neural networks, with video classification (MobileNetV2), time-series analysis (an LSTM network, not specified), and natural language processing (TinyLlama).

Direct memory access delivers an edge…

Direct memory access gives the NPU a handsome lead in matrix-vector products, with data reuse kept to a minimum. The same goes for scalar products, especially given the cost of final synchronization.

By contrast, for matrix operations, GPUs pulled ahead once the matrix size exceeded a certain threshold—the computation became increasingly dependent on raw processing power.

… and drawbacks

For video classification, in single or double precision (INT8 or FP16), the NPU has the advantage when processing sequential data. In batches, the balance shifts starting from a batch size of 8.

With TinyLlama, the NPU regains the advantage. It isn’t at ease during the prefill phase, dominated by matrix multiplications. But most of the operations lie in the decoding phase, which is rich in matrix-vector products.

NPUs, not fans of batch processing

The irregular memory accesses reappear in a more recent comparison (June 2026). Test setup: an embedded system with a Hexagon NPU and an Ada GPU. Running Qwen-2.5-7B and Llama-3-8B (as well as a 3B distilled variant). When inputs are fixed, the NPU performs better or uses less power for the same throughput (around 60% less energy versus the GPU at equal throughput). Things change with variable-length inputs (32 to 2048). In INT8, GPU throughput drops by 12%… versus 41% for the NPU. The effect is magnified with batching: with a batch size of 16, the GPU retains 78% of its sequential throughput, while the NPU falls to 34%. The culprit is the inefficient handling of padding: with heterogeneous tensors, execution becomes sequential, with internal reconfiguration between requests.

Text generation on NPUs: how many tokens per second?

FastFlowLLM performed its own measurements of text generation on a Ryzen AI 350 AP with 32 GB of RAM. Context windows were limited because the NPU could not access more than 50% of system memory. A token equals roughly three-quarters of a word, by OpenAI’s counting.

Results for the decoding phase:

Audio, Graphics, Cybersecurity, Accessibility… Software Takes Hold of NPUs

All these results show one thing: with a pipeline that is more rigid than that of GPUs, NPUs are best suited to workloads whose patterns are predictable. Software makers have accordingly worked in this direction. For example:

• Norton for deepfake audio detection
• Adobe for categorizing audio content in Premiere Pro

• Moises for voice-instrument separation

• CapCut for subtitling, speech synthesis and background removal
• Blender for a 2D ↔ 3D conversion plugin

• Capture One for masking, facial retouching and color grading
• Cephable for its assistive technologies (voice and gesture control)

Microsoft followed the same approach, focusing on real-time tasks sensitive to latency. Notably for communications (Windows Studio Effects). Some features are reserved for Copilot+ PC (portrait blur, vocal focus, creative filters…). Others are available with the 10 TOPS-generation NPUs (background blur, auto framing, gaze correction…)

AI PC Makers Integrate Their Own Services—Sometimes Based on NPUs

Acer’s “local AI” catalog also includes image creation and a generalist assistant based on an LLM. But these features require a GPU.

ASUS has adopted a similar segmentation. Features like noise suppression and adaptive brightness rely on the NPU. While StoryCube (media management) and MuseTree (image generation) require at least a GeForce RTX 4050.

Development Studios…

On its dedicated AI PC microsite, ASUS communicates the GPU TOPS in addition to the NPU TOPS. It also highlights its AI SuperBuild inference toolkit, developed with Intel and validated on its NUCs. It enables model conversion, deployment and evaluation, including knowledge-base management and a marketplace of MCP servers.

Dell also has its toolkit: Dell Pro AI Studio. It exposes OpenAI-compatible APIs and provides access to a catalog of optimized models. Some for GPUs (Devstral Small, Gemma 3, Granite 4…), others for NPUs (CLIP, Whisper, Phi 4…). Two reference guides for NPU development accompany it. One covers image-based speech search with CLIP and Whisper. The other, on RAG with LLaMA, LangChain, Chroma and Nomic.

… and “Local ChatGPTs”

HP isn’t showing such a payload. But for the end user, there is a reference service: AI Companion. It brings together three elements: a general-purpose chatbot, knowledge management, and a performance optimizer (with parameter-tuning assistance and troubleshooting). Reserved for Copilot+ PC, it has cloud mode (GPT-4o) and a local mode (Phi 3.5) that requires 32 GB of RAM.

Lenovo’s AI companion is called AI Now. Built on LLaMA 3 for the local portion, it requires a GPU (integrated with 24 Go of VRAM or dedicated with 16 Go). It carries the same triad—chat, knowledge base, and PC configuration. A number of AI services rely on it, such as Learning Zone (aid to learning: transcription, synthesis, quiz creation…).

NVIDIA, a Fourth Contender Expected This Fall in AI PCs

The llama.cpp inference server now supports Intel NPUs via the OpenVINO backend. Snapdragon X NPUs remain experimental, while AnythingLLM support is official, starting with the default embedding model.

YouTube content embedded below:

NVIDIA is working with llama.cpp to enable specificity in decoding optimizations (multitoken prediction) … ahead of its arrival in the AI PC segment. This is expected in fall 2026 with the RTX Spark chip. Born out of a collaboration with MediaTek, it blends the CPU Grace, the GPU Blackwell RTX and unified memory (up to 128 GB). No dedicated NPU per se, hence. This movement echoes Apple’s approach, which, for its M5 chips, integrated a neural accelerator into each GPU core.

NVIDIA claims a theoretical petaflop performance (4-bit precision) and promises a “local autonomous AI.” It has tied security primitives to its OpenShell runtime and expanded the NemoClaw blueprint.

* Microsoft also pushes WebNN API with Intel. It uses ONNX Runtime Web or LiteRT.js. Minimum configuration: an 11th-generation Core processor (Tiger Lake) and 8 GB of memory.