Efficient AI Across Edge, Near-Edge, and Cloud
Overview
Modern applications — smart cameras, self-driving cars, AR/VR headsets, on-device assistants — depend
on increasingly large AI models. The catch is that the appetite of these models is growing far faster than
the hardware meant to run them, especially at the edge. Closing that gap is not just a matter of building
bigger chips; it requires rethinking where each part of a model executes across the devices a user actually
has access to.
Our work develops two complementary frameworks that address this directly. DONNA decides how to
split a model across heterogeneous devices — CPUs, GPUs, and emerging Compute-In-Memory (CIM)
accelerators — to balance throughput and energy. HiDist extends the idea to the full vertical stack,
distributing inference across edge devices, near-edge servers, and the cloud, with per-tier energy control
and a search formulation that stays tractable as systems scale.
ــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــ
Motivation: Model Demand Is Outpacing Device Capacity
State-of-the-art models are growing roughly 8.4× per year in compute demand, while compute TFLOPs
capacity grows at only about 2.3×/year for edge HW and 1.9×/year for none-edge HW. The result is a
widening gap, shown in Fig.1, between what models need and what hardware can actually deliver — and
that gap is a central problem when deploying modern AI outside a datacenter.
[CAPTION: Fig. 1: Compute demand of state-of-the-art AI models versus the capacity of the hardware meant to run
them, 2017–2025. Model FLOP requirements (green triangles) have scaled at 8.4×/year through the recent LLM
era, up from roughly 3×/year in the pre-Transformer phase. Over the same window, non-edge hardware capacity
(red dots, FP16) — datacenter GPUs from P100 to B200 — has grown at only 1.9×/year, while edge hardware
capacity (blue dots, FP16) — devices such as the Edge TPU and Jetson AGX Orin — has grown at 2.3×/year. The
result is a widening gap between what models need and what hardware can deliver. Partial data source: Rakka et
al., "Mixed-Precision Quantization for Language Models," 2025.]
Two complementary strategies push back against this gap. Compression techniques — quantization,
pruning, distillation, LoRA — raise the ceiling on what a single edge device can run , but not indefinitely.
Real-time multimodal agents and AR assistants reasoning over long contexts, for example, routinely
demand capability above this ceiling, and compressing further only makes the model too inaccurate to be
useful. Compression buys headroom; it does not close the gap. Distributed inference, which is the focus
of our work, places each layer of a model on the tier best suited to execute it, spanning a continuous
spectrum from the user’s device capabilities to the cloud abundant resources.
That spectrum is no longer best understood as three competing options. It is a single continuum along
which latency, privacy, and compute constraints determine where each workload — or each sub-task
within a workload — should sit:
Personal AI (edge / on-device): with properties of one user scope, full data privacy, no network
latency, offline-capable, but bounded by device compute.
Enterprise AI (near-edge / on-premises): with properties of organization members scope, data
sovereignty, LAN latency, intranet-bound, with cluster-scale compute.
Public AI (cloud): with properties of global reach scope, large-scale storage, internet RTT,
internet-dependent, with elastic compute.
A hybrid distribution lets a model split across these tiers at runtime based on the user’s current network
conditions, battery state, and privacy context as illustrated in Fig.2.
[CAPTION: Fig. 2: The edge-to-cloud spectrum as a single continuum. Personal AI (on-device), Enterprise AI (on-
premises), and Public AI (cloud) differ in user scope, data privacy, network latency, connectivity, and available
compute. Two strategies operate across this spectrum: compression raises the ceiling on what the edge tier alone
can run, while distributed inference (DONNA, HiDist — our work) places each layer on the tier best suited to it and
spans the full continuum at runtime.]
ــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــ
DONNA: Distributed Optimized Neural Network
Allocation
Reference: AlShams, Smagulova, Fahmy, Fouda, and Eltawil — 2024 IEEE EDGE.
DONNA addresses a basic deployment question: given a pool of available devices with very different speed
and energy characteristics, where should each layer of a model run? Rather than pushing the entire model
onto whichever device happens to be largest, DONNA splits the model into pieces and assigns each piece
to the device best suited for it. Its contribution is a partitioning framework that jointly optimizes
throughput and energy under a single user-controllable parameter and first to integrate emerging energy-
efficient CIM devices into distributed setups.
Devices are heterogeneous.
Some devices are optimized for latency, others for energy. A ResNet-152 profile across a GPU, a CPU, and
a CIM accelerator makes this clear (Fig. 3): the GPU is fastest per layer, the CIM consumes the least energy
per layer, and no single device wins both metrics. Real deployments combine devices with mixed strengths
like these, and the question is how to take advantage of that mix rather than picking one and ignoring the
rest.
CAPTION: Fig. 3: ResNet-152 per-layer profile across three devices (Nvidia GPU, Intel CPU, CIM
accelerator). Left: latency per layer on a log scale — the GPU is roughly an order of magnitude faster than
the CPU, which is in turn faster than the CIM. Right: energy per layer on a log scale — the CIM is four to
five orders of magnitude more energy-efficient than the GPU and CPU.]
How DONNA works.
DONNA captures this with a two-stage framework (Fig. 4). A profiler measures each device's per-layer
latency and per-layer energy on the target model, producing DNN profiles. The profiles along with the list
of available devices, their communication characteristics, and a user-supplied tuning parameter 𝛼, feed
into a search algorithm. The algorithm minimizes a tunable cost function:
where 
is the normalized time component (the slowest pipeline stage) and 
is the normalized energy component. The parameter 
exposes the trade-off to the user: = 1.0 optimizes purely for throughput, = 0 purely for energy, and intermediate values balance the two. The output is an optimal partitioning of the model into pipeline stages, each mapped to the device best suited to execute it.
[CAPTION: Fig. 4: DONNA framework architecture. A profiler builds per-layer DNN profiles from the target
model and available devices; the partitioning algorithm consumes those profiles together with the tuning
parameter !, available devices, and communication characteristics, then effeciently searches for the layer-
to-device assignment that minimizes the joint cost function. The output is an optimal pipeline-stage
allocation.]
Key result.
Across device configurations on ResNet-152 (Fig. 5), the value of joint optimization becomes concrete. A
homogeneous 4-CPU setup produces a tight cluster of overlapping operating points at high energy and
low throughput — the trade-off space is constrained. Adding GPUs (2 CPUs + 2 GPUs) provides flexibility
along the throughput axis but leaves energy optimization limited; conversely, adding CIMs (2 CPUs + 2
CIMs) provides flexibility along the energy axis but leaves throughput limited. Only the fully
heterogeneous configuration (2 CPUs + 1 GPU + 1 CIM) opens both dimensions at once, tracing a
Pareto-like curve toward the "best" corner of the plot — high throughput, low energy — as α is swept
from 0 to 1.
[Fig. 5: Part of ResNet-152 distribution results. Inference energy consumption vs. estimated throughput
across four device configurations (4 CPUs; 2 CPUs + 2 GPUs; 2 CPUs + 2 CIMs; 2 CPUs + 1 GPU + 1 CIM).
The fully heterogeneous configuration traces a Pareto-like curve toward the bottom-right "best" corner as
! sweeps from 0 to 1.]
The takeaway: heterogeneity is where distribution flexibility lives. Different applications want different
priorities, and a heterogeneous device pool gives DONNA the room to satisfy each. The richer the device
mix, the more material the search has to work with, and the further DONNA can push the throughput–
energy frontier.
ــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــ
Modern applications like smart cameras, self-driving cars, and VR devices rely on powerful AI models. Running these models quickly and efficiently across phones, edge devices, and cloud servers is a tough challenge.
Our work develops two frameworks to make this possible:
- DONNA finds the best way to split and run AI models across different types of devices, from traditional CPUs and GPUs to new Compute-In-Memory (CIM) accelerators, so they use less energy while staying fast.
- HiDist takes the idea further by looking at the whole system: edge devices near the user, stronger near-edge servers, and powerful cloud machines. It decides where each part of the model should run to save energy and boost performance, instead of simply sending everything to the cloud.
Fig.1 shows this idea in action: AI models are broken into pieces and distributed across device tiers, with the system automatically choosing the best balance between speed and efficiency.
Fig. 1: A typical multi-tier edge-to-cloud network with DNN offloading guided by a search-based optimal distribution strategy.
Inside DONNA: Approach & Key Results
The Distributed Optimized Neural Network Allocation (DONNA) framework introduces a smarter way to run AI models across different devices. Instead of pushing the entire model to a single machine, DONNA carefully splits the model into pieces and assigns each part to the device best suited for it, whether that’s a CPU, GPU, or newly-emerging technology CIM.
As Fig.2 shows, DONNA uses a profiler to understand how fast and energy-efficient each device is. Then, it searches for the best distribution strategy that balances two goals at once: high throughput and low energy use. Unlike multiple earlier approaches that focused on only one of these goals, DONNA achieves both with user-controllable parameter, making it more efficient and adaptable across a variety of hardware setups.
Highlighted Results
DONNA consistently finds the sweet spot between speed and energy. As shown in Fig.3, it produces smooth trade-offs, or “Pareto curves”, across devices and networks. With strong communication links and heterogenous devices (Fig. 3a), DONNA maps out a clear Pareto curve, while under weaker links (Fig. 3c), it still spreads optimization points to reflect user preferences and maintain flexibility.
HiDist: Smarter Distribution Across Tiers
While DONNA showed how to balance throughput and energy, it used a single energy-aware parameter to treat all devices the same. In reality, devices behave very differently. Some scale well with larger workloads and deliver big speedups (Fig.4a), while others are far more energy-efficient for the same task (Fig.4b).
Fig. 4: HiDist leverages heterogeneity and tiered resource availability across real-world edge environments.
Highlighted Results
HiDist paper is still on-going, but even in its current form it clearly outperforms traditional “full offloading” strategies. As shown in Fig.5, instead of pushing all work to the cloud or near-edge servers, HiDist distributes it smartly across tiers. This leads to dramatic improvements, up to 7.7× faster throughput and 1.4× better energy efficiency compared to the best full offload.
What stands out is that HiDist doesn’t just chase one goal. It creates a balanced trade-off, or Pareto front, where users can benefit from both speed and efficiency, while full offloading gets stuck with poor compromises. This shows HiDist’s potential as a next-generation framework for real-world AI systems.
Watch Our Presentation
Copyright
The data and results presented in this work are protected by copyright and may only be used with proper citation. Any use of this work should reference the following papers:
