Faster-MoA: Low-Latency Tree-Structured MoA Serving with Early Exit and Agent-Aware Prefill-Decode Overlap

Mixture-of-Agents (MoA) improves output quality by letting multiple LLM agents collaborate, but its communication-heavy workflow makes serving slow. Faster-MoA introduces a novel tree architecture for local aggregation, agent-aware overlap of the prefill and decoding stages, KV-cache management for consecutive layers, and PEFT-tuned proposers (via LLaMA-Factory) that diversify exploration directions. A semantic-similarity-based early-exit scheme prunes unnecessary inferences, achieving 10× faster serving with only ±1% accuracy variation.

Recommended citation: Z. Wang*, Y. Qi*, H. Chen, and Z. Wan. "Faster-MoA: Low-Latency Tree-Structured MoA Serving with Early Exit and Agent-Aware Prefill-Decode Overlap." Design Automation Conference (DAC), 2026. (*Equal contribution)
Download Paper