Evaluating Energy Efficiency and Performance in LLM Inference and Introducing DheyoGPUpd

Introduction

Large Language Models (LLMs) are increasingly integrated into production systems across industries. These models require significant computational resources while serving, especially when deployed at scale.

According to the International Energy Agency (IEA), global data center electricity consumption reached approximately 460 TWh in 2022, accounting for about 1.5% of global electricity demand. This is projected to rise significantly by 2026, with AI workloads, especially large-scale inference, being a key contributor.

In today’s AI/ML clusters, HPC workloads, and media processing environments, workloads increasingly run across heterogeneous accelerator fleets that combine NVIDIA and AMD GPUs within the same data center and, in some cases, even within the same node. This trend is driven by supply constraints, cost–performance trade-offs, and workload specialization.

For tasks such as bottleneck detection, dynamic tuning, and preventing thermal or hardware failures, a robust telemetry system is essential. Telemetry, however, is more than just a performance tool — it plays a critical role in energy management.

For example, the total estimated energy consumed by H100 GPUs sold in 2023 was 13,797 GWh, enough to power approximately 4 million people (equivalent to the population of Georgia, USA).

The urgency of energy efficiency in computing is further emphasized by ARM CEO Rene Haas, who predicts that:

“By 2030, the world’s data centers are on course to use more electricity than India, the world’s most populous country.”

Just as fuel efficiency became a defining factor in the automotive industry, energy efficiency in compute infrastructure will soon be a central concern. Controlling GPU power consumption is therefore a critical step toward mitigating the massive and growing energy demands of modern compute workloads.

For data center operators, this trend introduces challenges related to infrastructure scaling, operational expenditure, and sustainability. LLM inference in particular involves complex trade-offs between latency, throughput, and power consumption. As energy costs rise, evaluating models not only by throughput and latency but also by efficiency metrics such as energy per request and tokens per joule (TPJ) becomes increasingly important.

This study analyzes how batch size and GPU frequency impact the energy efficiency and performance of LLM inference using Meta’s LLaMA-3.1-8B model.

The Challenge of Energy-Efficient LLM Serving

Inference with large language models (LLMs) introduces distinctive challenges due to their autoregressive decoding behavior, where tokens are generated sequentially. Unlike conventional inference, the computational workload during LLM decoding evolves over time and depends on parameters like batch size and decoding length.

This study focuses specifically on the decoding stage of inference. Each input consists of a single token, and the model is tasked with generating 512 output tokens. This controlled setup allows consistent evaluation of compute load and power draw across different runs while excluding variability introduced by prompt length.

Traditional power management techniques such as static power capping are often suboptimal for LLM workloads. A central metric in this study is Tokens per Joule (TPJ), defined as the number of generated tokens divided by the energy consumed during decoding. TPJ captures both throughput and power efficiency, making it a useful composite indicator of overall inference cost-effectiveness. Since energy is computed as power integrated over time, TPJ also implicitly accounts for how fast tokens are generated and how much power the system uses during that time.

Understanding how TPJ responds to different batch sizes and GPU frequencies is critical for identifying configurations that deliver high generation throughput while minimizing energy expenditure.

While Tokens per Joule (TPJ) is the primary metric for assessing energy efficiency, it does not capture all relevant performance trade-offs. Depending on the deployment objective—whether minimizing power costs, reducing response time, or maximizing throughput—different metrics become more appropriate.

Challenges in Heterogeneous GPU Environments

However, in heterogeneous GPU environments, telemetry itself becomes a major operational challenge:

• Vendor-specific telemetry stacks: NVIDIA, AMD, and other accelerators each expose different telemetry interfaces (e.g., DCGM, ROCm SMI), requiring separate exporters for each vendor.
• Inconsistent metric schemas: Different exporters use divergent naming conventions and labeling for similar metrics, complicating PromQL queries, alerting rules, and cross-vendor comparisons.
• Lack of node-level integration: GPU telemetry is often isolated from system-level signals (CPU, memory, PCIe bandwidth, power draw), hindering root-cause analysis for performance regressions or resource contention.
• Limited data delivery models: Most exporters rely solely on pull-based scraping, limiting flexibility for real-time or push-driven telemetry use cases.
• Operational overhead at scale: Large deployments must manage, configure, and update multiple exporters per node type, plus separate system-level exporters, leading to higher DaemonSet counts, complex scrape configurations, and increased change management burden, especially in Kubernetes and managed Prometheus environments.
• Lack of integrated control capabilities: Existing systems are primarily read-only; they provide metrics but cannot proactively adjust GPU parameters (e.g., power limits, clock speeds) in response to performance or energy considerations.

Metrics and Evaluation Goals

This study reports multiple performance and efficiency metrics to guide configuration selection based on three common serving goals:

These modes represent trade-offs. Understanding the relationship between batch size, GPU frequency, and each metric helps guide configuration choices depending on the specific LLM use case.

• Energy-Efficient ServingKey metrics: Tokens per Joule (TPJ), Energy per Request, Average Power Consumption. This mode is suitable for cost-sensitive LLM deployments, where minimizing energy use is a priority. In such cases, response time is less critical, so settings that optimize for energy per token are preferred. Ex: For lowering electricity and cooling costs in data centers.
• Latency-Sensitive ServingKey metrics: End-to-End Latency, Time Between Tokens (TBT) This mode applies to interactive/streaming applications, where fast output tokens are needed to maintain responsiveness. Low latency ensures fluid back-and-forth interaction. Ex: Conversational AI Chatbot where users expect near real-time output token streaming.
• Throughput-Driven ServingKey metric: Tokens per Second (TPS) This mode is relevant for high-throughput inference pipelines, where the objective is to process large volumes efficiently. Here maximizing tokens per second becomes critical, even if latency or power usage is higher. Ex: LLM inference behind an API for large-scale users, such as coding assistants or enterprise SaaS tools with many parallel requests.

Results and Discussion

4.1 Energy Efficiency Analysis

Tokens per Joule (TPJ) improves with both batch size and GPU frequency, peaking around Batch 32 at 1000-1400 MHz. Energy per request drops significantly with larger batches, making them more efficient overall. However, average power also increases as batch size and frequency rise, which may limit how aggressively these settings can be used in practice. So from the trend, large batch size at moderate GPU frequencies (1000-1400 MHz) offers the best energy efficiency while keeping power draw manageable.

Figure 1: Energy efficiency metrics across batch sizes and GPU frequencies, showing trade-offs in tokens per joule, energy per request, and power consumption.

4.2 Latency Analysis

Latency decreases with smaller batch sizes and higher GPU frequencies. Batch 1 consistently shows the lowest Time Between Tokens (TBT) and End-to-End latency across all settings. Larger batches incur higher latency, which can be mitigated but not eliminated by increasing the frequency. For batch sizes 1 and 2, latency is already low, and increasing GPU frequency beyond 700-800 MHz offers minimal improvement making these lower frequencies a good trade-off point for reduced power consumption.

For applications that prioritize fast response time, using Batch 1 or 2 with high GPU frequencies (≥1200 MHz) is the most effective setup.

Figure 2: Latency analysis showing impact of batch size and GPU frequency on Time Between Tokens and end-to-end latency for low-latency applications.

4.3 Throughput Analysis

Throughput increases steadily with both batch size and frequency. Batch 32 around 1600 MHz achieves the highest tokens-per-second rate. Even at lower frequencies, larger batches outperform smaller ones in throughput.

For throughput focused scenarios, Larger batch at high GPU frequencies (1400-1600 MHz) is the most suitable configuration.

Figure 3: Throughput analysis showing how tokens per second scale with batch size and GPU frequency.

Summary of Trade-offs

Note: These observations are based on the LLaMA 3.1–8B model. While exact values may vary across models, the overall trends with respect to batch size and GPU frequency are expected to remain consistent. The goal is to understand these trends so they can guide configuration choices for different LLMs.

Introducing DheyoGPUpd

DheyoGPUpd is a robust and efficient solution for monitoring and controlling GPU resources in real-time.

Features

1. Comprehensive MonitoringFull support for NVIDIA and AMD GPUs, along with system-level and CPU metrics for complete observability.
2. Dual Control PlaneActively manage GPU performance parameters, such as clock speeds, via either a REST API or an MQTT message bus.
3. Flexible Metrics DeliverySupports push-based metric publishing over MQTT for event-driven or real-time use cases, and pull-based scraping via a native /metrics endpoint for seamless Prometheus integration.
4. High PerformanceDesigned with a concurrent, asynchronous architecture using Rust’s tokio runtime and a multi-threaded C++ backend for efficient AMD metrics collection.
5. Memory SafetyCore application logic is implemented in Rust, ensuring reliability and eliminating common memory-related vulnerabilities.
6. Container-ReadyIncludes a production-ready Dockerfile and Kubernetes DaemonSet configuration for easy deployment in containerized and cloud environments.
7. Graceful DegradationAutomatically detects the absence of GPUs and continues to operate by providing system-only metrics, ensuring monitoring continuity.

Comparison

This way it’s not just a table of facts, but also explains what each exporter is good for and why DheyoGPUpd stands out.

System Overview

GPUpd consists of two main components; Telemetry Module and Power Control Module

1. Telemetry Module

The Telemetry Module is responsible for real-time collection, aggregation, and forwarding of GPU and system metrics. It ensures high availability and flexibility for monitoring in both standalone and distributed GPU environments.

Key Features

• Multi-vendor support: Collects metrics from NVIDIA, AMD, and system GPUs in parallel.
• Flexible data export:

– Prometheus – Exposes metrics via /metrics endpoint for scraping.

– MQTT – Publishes metrics to the dheyo/metrics topic (configurable broker endpoint).

• Alerting & diagnostics:

– Built-in alerting system for threshold-based notifications.

– Differentiates between hardware-level faults and system-level errors.

• Scalability: Designed for real-time monitoring across single or multiple GPUs in a cluster.

2. Power Control Module

The Power Control Module enables fine-grained GPU power and performance management, giving administrators precise control over GPU behavior.

Key Features

• Clock speed management: Dynamically tune GPU clock frequencies to optimize performance or reduce power consumption.
• Power caps: Define maximum power limits for GPUs to balance performance and energy efficiency.
• Scheduling: Automate power and clock adjustments based on workloads, time-of-day, or policies.
• Granular scope: Apply control policies to individual GPUs or clusters of GPUs simultaneously.
• Flexible control methods:
  • Publish control commands to the dheyo/power-control topic via MQTT.
  • Send JSON-based requests to the /power-control HTTP endpoint.

Figure 4: High level design of the DheyoGPUpd system.

Conclusion

DheyoGPUpd unifies GPU telemetry and power control across heterogeneous environments, offering a single lightweight solution for monitoring and managing NVIDIA, AMD, and system-level resources. By combining real-time observability with fine-grained control, it simplifies operations, improves efficiency, and helps reduce the growing energy footprint of modern compute workloads.