News

Same Chip, Different AI Speed: How GMKtec EVO-X5 Pro Reaches 75 Tokens per Second

If you’ve been following the world of local LLM inference, you may have heard a common criticism of AMD’s Strix Halo platform: with 256 GB/s of memory bandwidth, it can run large language models, but don’t expect spectacular speeds There’s some truth to that—but only half the story. Hardware sets the limits. Software determines how much of that potential you can actually unlock. On the GMKtec EVO-X5 Pro, a specialized inference engine called Halogen demonstrates just how much difference software optimization can make. Running the same model on the same underlying hardware, Halogen delivers a substantial performance boost over a general-purpose inference engine. Here’s what makes the difference. 1.  The Numbers: 23 Tokens/s vs. 75 Tokens/s Let’s start with the results. Running Qwen3.8-Flash-Next on the AMD Strix Halo platform, we compared the performance of a general-purpose inference engine with Halogen on the GMKtec EVO-X5 Pro. Inference Engine Sustained Generation Speed llama.cpp, general-purpose engine Around 23 tokens/s Halogen on GMKtec EVO-X5 Pro Around 75 tokens/s   That’s more than 3× the sustained token generation speed under the tested conditions. And the gap can become even more pronounced when working with long contexts. In practical terms, tokens per second (tok/s) measures how quickly an AI model generates its response. At 75 tok/s, the model can produce text at a pace that feels much closer to a real-time conversation, making it easier to work through coding tasks, analyze documents, and build local AI workflows without constantly waiting for the next sentence. The key point? Both configurations use the same underlying chip. The difference comes from how the software puts that hardware to work. Note: These figures reflect GMKtec's internal testing under the specified configuration. Actual performance varies with model settings, context length, quantization, and workload. 2.  The Secret Isn’t New Hardware. It’s a Different Engine. First, let’s clear something up: AMD provides the hardware. GMKtec hasn’t modified the silicon or changed the physical memory bandwidth. The real variable is Halogen, a specialized inference engine designed to optimize AI workloads on AMD’s Strix Halo platform. Unlike general-purpose engines built to support a wide range of hardware and model architectures, Halogen takes a much narrower approach. It focuses on a specific GPU architecture and a specific model family to extract more performance from the hardware. That specialization is the foundation of its advantage. 3.  Built for One Platform, Not Every Platform General-purpose inference engines are designed to work across a broad ecosystem. They need to support different GPU vendors, hardware generations, model architectures, and execution paths. That flexibility is valuable. It also introduces complexity. Halogen takes the opposite approach: ● Target hardware: AMD Strix Halo, using the gfx1151 GPU architecture. ● Target models: The Qwen3.8-Flash-Next model family. ● Optimization strategy: Hardware-specific kernels and model-aware execution paths. In its current implementation, Halogen is deliberately specialized rather than a universal inference solution. Why does that matter? Every layer of generality creates engineering trade-offs. A general-purpose engine must accommodate different hardware capabilities, maintain compatibility paths, and handle alternative execution strategies. A specialized engine can focus its resources on the exact hardware and workload it is designed to run. Think of it as the difference between a Swiss Army knife and a precision-engineered tool. One can do almost everything; the other is built to do one job exceptionally well. 4.  Optimization No. 1: Removing Unnecessary Abstraction A general-purpose inference engine such as llama.cpp must accommodate a wide range of GPUs and computing environments. Its implementation needs to remain flexible enough to support different architectures and execution requirements. Halogen can take a more direct route. By developing kernels specifically for the gfx1151 architecture, the engine can optimize critical operations around the platform’s actual hardware characteristics, including its compute resources, memory behavior, and cache hierarchy. This reduces reliance on generic execution paths and allows performance-critical operations to be tuned for the target platform. The advantage isn’t simply about removing a few compatibility checks. It’s about engineering the execution path around the hardware from the ground up. When those optimizations are applied to operations that run repeatedly during inference, even small improvements can compound into a meaningful increase in generation speed. 5.  Optimization No. 2: Smarter Execution for Mixture-of-Experts Models Qwen3.8-Flash-Next uses a Mixture-of-Experts (MoE) architecture, with close to 180 billion total parameters. MoE models work differently from conventional dense models. Instead of activating every parameter for every token, they route each input through a selected subset of specialized experts. Imagine a team of specialist doctors. Not every doctor needs to examine every patient. The system identifies the relevant specialists and brings them into the process. This architecture can make very large models practical to run locally—but it also creates specific challenges for inference engines. Expert routing, weight access, and memory movement all affect performance. On a platform where memory bandwidth is a critical resource, unnecessary data movement can become particularly expensive. Halogen is designed around the target model's execution patterns, optimizing expert routing and data movement so that the inference process focuses on the weights and operations needed for the current computation. The goal is straightforward: move less unnecessary data, reduce avoidable overhead, and keep the hardware working efficiently. Fine-Grained Quantization: 5.53 Bits per Weight Halogen also uses a 5.53 bits-per-weight (bpw) quantization configuration. Quantization reduces the amount of memory required to store model weights by representing parameters with fewer bits. This can ease memory pressure and reduce the amount of data that needs to move during inference. But more aggressive quantization isn't automatically better. Pushing precision too low can affect model quality, depending on the model and workload. A 5.53 bpw configuration offers a different balance: compact weight storage without simply forcing the entire model into a uniform 4-bit representation. Combined with model-aware execution, this helps manage the memory demands of large local models while maintaining a practical balance between efficiency and output quality. 6.  Optimization No. 3: Speculative Decoding Another important technique is speculative decoding. In conventional autoregressive generation, a language model generates tokens sequentially, with each new token depending on the preceding context. Speculative decoding changes the process. A smaller draft component proposes several candidate tokens in advance. The main model then verifies those candidates in a more parallelized step. Tokens that pass verification can be accepted together, allowing the system to generate multiple tokens for the computational cost of fewer sequential decoding steps. The result depends on how accurately the draft component predicts the main model’s output. When predictions are accepted at a high rate, generation can become substantially faster. For workloads with repetitive or predictable output patterns—such as code completion, structured text generation, and document rewriting—this technique can be especially effective. In GMKtec's internal testing, Halogen reached a peak of 94 tok/s in repetitive-content scenarios. That is a peak result for a specific workload, rather than a universal generation speed. Actual gains depend on the task, context, and draft acceptance rate. Think of it as having an assistant prepare a draft while you review it. When the draft is right, you can approve several words at once instead of composing every word from scratch. 7.  The Combined Result: Fast Prompt Processing and Sustained Generation The real benefit comes from combining these optimizations rather than relying on a single trick. On the GMKtec EVO-X5 Pro, equipped with up to 192GB of unified memory, Halogen delivers the following results in GMKtec's internal testing: <| |> ● Around 75 tok/s sustained generation speed ● Up to 2,000 tok/s prompt-processing speed ● Up to 94 tok/s peak generation speed in repetitive-content scenarios These figures describe different stages or conditions of inference and should not be treated as directly interchangeable. Prompt processing, also known as prefill, is the stage in which the model processes the input context. Generation, or decoding, is the stage in which it produces the response token by token. Fast prompt processing helps when you submit a long document, a large codebase, or an extensive conversation history. Fast sustained generation makes the response itself feel more immediate. Together, they help deliver a smoother experience across long-context AI workflows. What Does This Mean for a 128K-Token Context? A 128K-token context can accommodate a substantial amount of material—potentially hundreds of pages, depending on the language, formatting, and content. With prompt processing reaching up to 2,000 tok/s under the tested conditions, the EVO-X5 Pro can ingest large inputs quickly. Its sustained generation performance then helps turn that context into useful answers, summaries, analysis, or code. For developers and professionals who work with large documents or complex AI workflows, both stages matter. A model that generates quickly but takes too long to process the prompt can still feel sluggish. Optimizing the entire inference pipeline makes a more meaningful difference. 8.  The EVO-X5 Pro: More Than a Powerful Chip The GMKtec EVO-X5 Pro is built for users who want to run demanding AI workloads locally, with substantial compute resources and memory capacity available on the desktop. Paired with Halogen, it offers more than the hardware alone: a specialized software stack designed to extract additional performance from a specific combination of GPU architecture and model family. This is particularly relevant for developers running local AI agents, coding assistants, and long-context workflows where responsiveness directly affects productivity. Local inference also gives users greater control over where their data is processed. For organizations handling proprietary code, internal documents, or sensitive business information, keeping compatible AI workloads on local hardware can help reduce reliance on cloud-based inference services. The trade-off is deliberate. General-purpose engines prioritize broad compatibility. Halogen prioritizes performance on its target platform and supported models. Neither approach is universally better. But if your workload matches what Halogen is designed to accelerate, specialization can deliver a tangible advantage. Because being able to run a large model and being able to run it smoothly are two very different things. Memory bandwidth is determined by the hardware. How much performance you unlock depends on the software. With the EVO-X5 Pro and Halogen, GMKtec is pushing local AI inference toward a more responsive, desktop-based experience—without requiring a different chip to make the difference. Experience Local AI at Full Speed Discover the GMKtec EVO-X5 Pro and explore what a specialized AI inference engine can bring to local LLM workloads. GMKtec EVO-X5 Pro — Built for the Next Generation of Local AI.