40G LAN Project

Project Concluded:   September 30, 2020

DB Reference:   dc_net_40g_lan

Last Updated:   September 26, 2023

Part of the Genetic Analysis Project

picture of arista 7050-QX in the rack.

Introduction

The purpose is to provide sufficient fast inter-connect network among computing and storage clusters.

The 40G LAN consisits of a 40G switch and 40G NIC on each client.

In addition, RDMA was configured for faster file access.

Specification & Structures

As of April 2023, the system has the following specification:

Hardware: 1 * ARISTA DCS 7050QX-32 switch

Hardware: 26 * Finisar FTL410QE2C-G1 40G Fiber Transceiver.

Hardware: 11 * Mellanox MCX354A HP 40G/56G NIC

Hardware: N * MPO 12 core fiber cable.

Upper Link: 10G

26 Finisar transceivers

Detailed Description

The 40G LAN project was initiated to provide a high-speed interconnect for major servers and hypervisors, which was achieved by configuring a data center-grade switch. As a newcomer to this level of network configuration, I had to start from the beginning and learn how to operate EOS, which is Arista's operating system for switches. What makes it worse is that I need to choose the compatible equipment and parts before I even knew how to operate them. With a through research online as well as browsing available and affordable options, I have successfully settled the devices needed.

Configuring the network was also a time-consuming process, particularly for functions like RDMA. There is not much documentation available for different operating systems, and different versions of RDMA exist on different OS. Consequently, I had to test different parameter combinations among servers and the switch. However, after extensive testing, I was able to successfully set up RDMA. This greatly improved the access speed to the storage cluster since data no longer had to be processed by the CPU before being sent out via the network.

11 mellanox 40G nics

At present, the network runs in Ethernet mode up to 40G with an upper link of 10G. Any equipment that does not support 40G network is connected to the data flow from there. The traffic is then sent to the gateway, which has a 2G upper link via 3 fibers.

Currently, the gateway is an OpenWRT router installed in a R630 server. Static DHCP was used to assign IP addresses, which means that servers would use DHCP to obtain their IP addresses, and the DHCP server responds with a pre-defined static IP with an infinite lease.

In the near future, I am planning to implement a more strict VLAN policy to improve security in the near future, which will be part of the Datacenter Security Project. Additionally, I am also considering adding an Infiniband Network, in addition to Ethernet, for storage purposes.