Reducing Address Arithmetic Overheads: New Compiler Techniques for Programmable Dataflow-based AI Accelerators

Dataflow accelerators have emerged as a dominant architecture for modern AI workloads, combining massively parallel compute fabrics with hierarchical on-chip memories to achieve high performance and energy efficiency. Many such systems adopt a decoupled access–execute model, where programmable load/store engines orchestrate data movement between memory and compute units. While this design improves utilization, it exposes efficient address generation as a performance concern in the compiler. In these units, address arithmetic is typically realized as scalar operations interleaved with memory operations, forming dependence chains along the critical data supply path. Under stringent hardware constraints, such as in-order execution, these computations introduce latency, stall the serving compute units, and degrade overall performance.

In this work, we propose a compiler approach to reduce address arithmetic overheads in programmable load/store units. Our approach consists of two complementary techniques: (1) \emph{Address arithmetic absorption}, a local transformation that incorporates scalar address updates into composite memory operations by leveraging their ${\tt pre-incr}$ and ${\tt post-update}$fields, and (2) $\emph{Address arithmetic hoisting}$, a global transformation that restructures loop-carried address computations to simplify updates across iterations.

We demonstrate the effectiveness of our approach in the context of IBM Spyre accelerator and its compiler. Our results over a range of AI workloads spanning transformer, convolution-style models along with object-detection models demonstrate a geometric mean speedup improvement of $1.65\times$ in execution cycles over a strong baseline with state-of-the-art optimizations.