Efficient Resource Allocation for Data Centers with Dynamic Optical Network Infrastructure

Abstract

We investigate resource orchestration in a data center interconnection network, which relies on hybrid electro-optical top-of-rack switches to interconnect servers over multi-wavelength optical rings. The bandwidth of the rings is shared, and an efficient utilization of the infrastructure calls for coordination in the time, space, and wavelength domains. To this end, we present offline and incremental dynamic resource assignment algorithms. The algorithms are suitable for implementation in a software defined network control plane, achieving efficient, collision-free, and on demand capacity use. Our simulation results indicate that the proposed algorithms can achieve high utilization and low latency in a variety of traffic scenarios that include hot spots and/or rapidly changing traffic. Furthermore, we evaluate the effect of the control plane delay and traffic estimation policies, using the OMNET++ packet-level simulator with realistic MapReduce traffic.

Next, we propose a DCN fabric that relies on a “Lean” optical switch design with limited but scalable configurability. This design offers high reconfiguration speeds, real-time scheduling, efficient network control, and a low number of switching elements. To achieve these objectives, we relax the non-blocking network requirement and opt for partially configurable switching modules, limiting the control capability of the scheduler and reducing control overhead. We compare our proposed network with the RotorNet architecture, which operates with fully distributed control, and the Mordia architecture, which operates with centralized control. Each architecture achieves varying levels of functionality and offers distinct advantages. The proposed solution lies in the middle of the other two approaches and combines the benefits of both of them. Additionally, we analyze the crosspoint complexities of the proposed and the aforementioned reference architectures, and evaluate their throughput and latency performance through simulations. Finally, we enhance RotorNet using breakout to control latency, partial configurability with centralized control, and an adaptive scheduling policy that learns and optimizes resource allocation dynamically.

Finally, we examing the problem of the secure distributed storage in heterogeneous cloud-edge environments. Distributed storage systems spanning across different cloud data centers have substantially improved availability and flexibility for data storage and retrieval operations. However, stringent latency requirements of emerging applications necessitate optimized selection of storage resources that exhibit smaller delay. Introducing edge resources into distributed storage systems enables data placement closer to its source, but simultaneously increases the complexity of decision-making and orchestration processes for optimal data placement. In this work, we develop mechanisms for storing data across an infrastructure that includes both edge and cloud resources. Our approach focuses on optimizing data integrity, longevity, security, and cost, while leveraging erasure coding when performing the resource allocation. We first present a comprehensive mixed integer linear programming formulation of the storage resource orchestration problem. As the search space for the optimal solution can be vast and the execution time prohibitively large for real size problems, we also propose an innovative multi-agent heuristic approach that uses the rollout, a reinforcement based policy, to balance performance and execution time efficiently. Through various simulation experiments, we evaluate the developed mechanisms and trade-offs involved in our approach. By incorporating data from a multi-cloud provider, we further enhance the validity of the simulations and the conclusions drawn.