Reinforcement Learning for RDMA Traffic Scheduling in Distributed Data Centers

Abstract

Modern distributed data centers increasingly rely on Remote Direct Memory Access technologies to achieve high throughput and microsecond scale latency, particularly for demanding workloads such as large language model training and distributed storage operations. Conventional congestion control and traffic scheduling mechanisms, largely reliant on heuristic thresholds and reactive priority based flow control, struggle to handle the highly dynamic, bursty traffic patterns inherent in scale out architectures. These traditional systems frequently experience structural inefficiencies, including priority based flow control deadlocks, congestion spreading, and suboptimal link utilization. To address these limitations, this paper proposes a novel multi-agent reinforcement learning framework for dynamic Remote Direct Memory Access traffic scheduling in spine leaf data center networks. By formulating the network scheduling challenge as a decentralized Markov Decision Process, the proposed system enables intelligent edge switches and network interface cards to autonomously optimize rate limiting and priority marking based on real time telemetry data. The methodology leverages a multi-agent proximal policy optimization algorithm, which stabilizes training in highly concurrent environments while maximizing global network utility. Extensive evaluations conducted in packet level simulators demonstrate that the reinforcement learning based approach significantly reduces flow completion times for delay sensitive mice flows while maintaining high throughput for background elephant flows. Furthermore, the intelligent scheduling policy successfully mitigates queue buildup and eliminates the risk of cascading flow control pauses. The results underscore the potential of deploying continuous learning algorithms to manage the growing complexity of low latency data center interconnects.