Datacenter Power (Electricity)

Part of the Datacenter Project

Electricity is vital for data centers, as a power outage can cause catastrophic failure of all hosted servers. To prevent this, most official data centers have at least two power supplies from different sources, as well as battery backup and emergency generators.

Having dual power input may not be practical for me. While a generator is a viable option, it may not be started in time since there is no dedicated person to monitor the data center's status. However, if I have a stable power source, I can focus on ensuring safe power distribution and forget about power outages for the time being.

The goal is to safely deliver sufficient electricity to all equipment for the long-term operation.

Since we only have one power source, upgrading while the data center is running is unacceptable as it would require a complete shutdown. Therefore, it is crucial to ensure that the current power input is sufficient with an expandable design.

The current power source is a 380V cable with three hot wires and a neutral wire, and each pair of hot and neutral wires has a voltage of 220V. Each wire has an area of 25mm². After conducting research and calculations, it has been found that this specification can safely handle up to 70KW. However, to account for other purposes such as air conditioning, heating, or safe redundancy, only 50KW can be used for servers.

With the increasing power consumption of CPUs and GPUs, it is not uncommon for a 2U server to consume 700W or more. For GPU servers, a single server with 4 GPUs can consume up to 3KW. In contrast, storage servers have relatively low power consumption. Assuming an average power consumption of 500W per 2U server, 50KW can support approximately 100 * 2U servers, which is only the capacity of 5 racks. As we have 10 racks, it is necessary to upgrade the entry line first.

Several options for upgrading are available, including 35mm², 50mm², or 70mm² cables, which can support 85KW, 125KW, and 160KW, respectively. It is evident that a 50mm² cable will be sufficient for our needs, providing a capacity of 125KW, with 105KW solely available for servers. This will allow for approximately 200 * 2U servers and can fully load 10 racks.

After completing the necessary paperwork, the State Grid has dispatched workers to upgrade our entry line.

Each Rack

Now that the entry line has been upgraded, it is time to distribute power to each rack. The racks are divided into three categories: computing racks that host high-performance servers, storage racks that host storage equipment, and one network rack for networking.On average, each rack should receive approximately 10KW (105KW/10). However, computing servers consume much more power than storage servers or network equipment, so we need to increase the power capacity for computing racks and reduce it for the other two..

PDU(Power Distribution Unit)

Before a server can start, it needs to be plugged into an outlet. In other places, this is usually done using a power strip, but in the rack, a dedicated device called a Power Distribution Unit (PDU) is used for this purpose.

After reviewing several models, I have found that 8000W is currently the maximum power capacity available in retail PDUs. These PDUs do not have a switch and require a separate circuit breaker to control. I have decided to install two of these PDUs for the computing rack (up to 16KW) and one for the storage and network rack (up to 8KW).

Cable & Circuit Breaker Specification

An 8KW PDU requires suitable cables to operate safely. After some calculations, it has been determined that power cables with a wire area of at least 6mm² would be sufficient.

An 8KW circuit requires slightly less than 40A, as China's electricity voltage is 220V instead of 110V in the US. Therefore, the ideal circuit breaker for each PDU should have a specification of about 40A.

Initially, there are plans for four racks in the starting phase. We have two primary breakers for servers. Each primary breaker with a specification of 125A controls three hot wires and a neutral wire. To ensure balance between servers, it should be safe for each hot wire to handle up to 4 * 40A PDUs. This results in a total of 12 circuit breakers being installed under one of the primary breakers. Even though there are only four racks initially, all 12 breakers have been fully installed to accommodate for potential future growth. This should be sufficient for the time being and near future. In case of additional demand in the future, I can do the same for the second distribution breaker without turning off the first one, which would require a complete shutdown of all servers.

To distribute power to individual racks, the 50mm² primary cables needed to be split into smaller cables with proper protection. This was done using a power distribution cabinet. It should be noted that the calculation formula for all amperage or voltage was taken from trusted sources and was corss-validated several times. For the sake of simplicity, the calculations will not be shown here.

Structure

It has been decided to install 5 primary circuit breakers under the main line. Two breakers will be dedicated solely to the servers, two will be used for other buildings and purposes, and a smaller breaker will be used for other equipment within the data center.

Furthermore, the three hot wires will be distributed and used evenly across all appliances including servers.

The Main Breaker

The State Grid has provided a 160A breaker, which is insufficient for our needs. Based on the estimated requirements and cable dimensions, it has been replaced with a 250A model.

Primary "Cable" and Circuit Breaker Specification

As someone new to electrical engineering, I began my research by looking for useful information online. It did not take me long to realize that for higher specification, conventional cables are no longer used in distribution cabinets. Instead, copper bars are used, which require customized production. I decided to use copper bars between the entry breaker and five primary breakers.

The primary circuit breakers were selected based solely on the dimensions of the cables, as their job is to protect the circuit and wire. Therefore, for the two main breakers for the servers, each will have a specification of 125A. For the other two breakers used for other buildings, 125A and 100A were used due to their cable sizes of 35mm² and 25mm², respectively. For the last breaker, which will be used for non-server purposes in the data center, such as cooling, a 63A breaker was used, as 16mm² cable will be used.

Circuit Breaker Specification - PDU

It has been determined that an earth-leakage circuit breaker (ELCB) should be used for each PDU, rather than a conventional breaker. This is because servers, like any other appliance with metal covers, can leak voltage, which is dangerous for maintenance staff. An ELCB can provide proper protection against shock. Therefore, a 40A ELCB will be used for each PDU.

Circuit Breaker - Cooling System

The cooling system comprises two fans, which primarily consist of motors. Unlike other appliances such as servers (which require C-type breakers), motors have a very different startup current curve. Therefore, a special type of circuit breaker (D-type) is required for motors, as the breaker will not disconnect due to the surge at the starting phase.

Although there are no immediate plans to install air conditioning, two breakers have been assigned for them just in case. This is because it would not be practical to modify the distribution cabinet later while the data center is operational. Therefore, two 4P D-type ELCBs with three hot wires of 32A have been installed for the air conditioning units. This should provide about 20KW capacity for each A/C unit, which is more than sufficient.

Surge Protection

To prevent damage to the servers from lightning or electrical surges, a surge protector has been planned. The CHNT surge protector selected has a capacity of 65kA, which should be more than enough to protect against general surges.

Model Section

To limit the scope of potential problems, 4P breakers (which break the neutral line as well) will be used whenever possible.

For the primary breakers, there is a frame current specification in addition to the rated current. This is primarily for modularized circuits where parts may need to be replaced. However, as that is not a concern at present, the cheapest model among those available can be chosen. Based on price, availability, brand, reliability, safety considerations, and other specification not showing above, the following model has been selected.

Wires

There are different national standards for wires in China, each with its own requirements and purposes. The most appropriate model has been selected based on factors such as suitability, availability, and price at the time. For the sake of simplicity, the specific wire types and their purposes will not be discussed here, but there are many publicly available sources online for those who are interested.

Note: The table below does not include the 50mm² entry cable provided by the State Grid, nor the 50mm² copper bar. For colors such as blue (which represent the neutral line), more length were purchased than for other colors. Cable lengths were calculated based on the design graph, and cables for wall outlets and lights were not included as they were present before the project began.

Tools & Material

After setting the specifications for the breakers, I calculated the required supporting materials and tools for the construction phase. This included connection terminals of different sizes and colors, wire splitting terminals for different breakers, connectors, heat shrinking tubing of various sizes, screws, insulating tape, and other necessary items.

The necessary tools for the project included wire cutters for different wire sizes, a wire stripper, heat gun, vernier caliper, multimeter, crimper, power meter, hydraulic tongs, and several multi-use power tools.

Cabinet Installation

The size of the cabinet was determined using the design graph, and it was mounted on the wall due to the location of the entry cable. To make it easier, I divided the electrical parts into several blocks. The blocks were pre-assembled outside and were mounted into the cabinet with the help of several my friends.

Ventilation & PDU

The PDU installation was completed later. For flexibility in future adjustments, I first attached one connector to each PDU, then connected another connector to the cabinet, before finally joining the two connectors together.

The installation of the ventilation system followed the same principle. Due to the height of the working area, it took around 2 days to complete, because I had to move and adjust the scaffold to different positions and heights.

UPS

Providing UPS for the entire datacenter is currently not feasible due to its cost. However, I have optimized the server structure to prepare for a potential power outage. The computing servers store only temporary data locally, while the important data, such as source and results, are stored remotely in the low-energy storage servers. The high-speed 40G LAN and RDMA ensure satisfactory I/O speed for both read and writes. In case of a power loss, the only impact is restarting the current job, which is not that severe in most cases.

For more information about the storage, you may want to take a look at the Hybrid Storage Project.

The entire distribution cabinet works like a charm.

Two plans are currently being considered: providing UPS to more servers, starting with hypervisors and potentially expanding to a wider range of servers, and utilizing solar energy to reduce energy costs with the addition of a battery system.

These plans need a more in-depth research before turning into actual project to implement.

Email

• [email protected]
• [email protected]