Methodology to Characterize Row Manifolds for High Power Direct to Chip Liquid Cooling Data Centers

Air cooling, the most common and traditional approach to cooling Information Technology Equipment (ITE), is encountering limitations as the heat fluxes of the components increase. Consequently, data center facilities and researchers have extended the cooling capabilities of air cooling through various methods such as in-row cooling [1,2], enhancing the mixing effectiveness of air at the mixing chamber [3], implementing aisle containments [4], and making design modifications at the server level [5]. While these methods can effectively dissipate large heat loads, there is also a significant increase in the cost of cooling infrastructure [6–10]. Recently, researchers and the industry have researched liquid cooling technologies, and their capabilities to cool high-power density racks [11–13], this includes liquid cooling on processors (i.e., single phase and two-phase cold plate design) [14,15], subimmersed cooled servers, or immersion cooling [16], feasibility study on rear door heat exchangers [17].

Direct-to-chip liquid cooling has been one of the promising and efficient solutions for taking away substantial amounts of concentrated heat loads from the primary components in a server, due to higher volumetric specific heats and high heat transfer coefficients of the fluid [18]. To attain the highest possible efficiency, in liquid cooling the heat transfer from the Information Technology Equipment (ITE) to the liquid is performed at its highest possible temperature [19]. Thus, direct liquid cooling can significantly increase the percentage amount of savings for both the operational as well as capital expenditure costs, especially for high-performance computing equipment.

In direct-to-chip liquid-cooled data centers, there are two heat transfer loops, a primary loop, and a Technology Cooling System (TCS), the TCS loop is also called the secondary loop, where the cooling fluid is circulated by the Cooling Distribution unit. This secondary loop consists of several devices/components such as the row manifolds, rack manifolds, and cooling loops where the cold plates are mounted on top of the heat-dissipating components. On the secondary side, the design of the components must meet operating parameters such as the flowrate, system pressure, and temperature [20]. The Row manifold is an essential part of the secondary loop as it distributes the fluid from the CDU to all the racks in the data center. Dynamic liquid cooling approach is being implemented for current liquid-cooled data centers as it increases the pumping power savings [21–23], thus for dynamic cooling, it is important that the components used to build this type of infrastructure are carefully designed to handle the varying flowrate caused by the change in pressure drop and at different fluid temperatures. Thus, before installing these components in the cooling infrastructure, they must be characterized.

In this study, experiments are performed on the row manifolds used in the liquid-cooled data center and are characterized for understanding the pressure drop and the distribution of the fluid to the different rack manifolds for designing an energy-efficient data center. For this set of experiments, propylene glycol-25% was used as the coolant and 21 °C inlet temperature was provided. The results obtained from the experiments were validated with a one-dimensional (1D) flow network modeling simulation for efficient prediction of the flow rates, pressures, and temperature in a complete liquid-cooling system.

The experimental setup consists of eight 52 U racks, two-row manifolds of six ports each, eight rack manifolds, a Coolant distribution unit, and flow, temperature, and pressure sensors to measure data as shown in Fig. 1 below. The CDU used in this study was 450 kW, which can provide a maximum flowrate of 500 lpm at an external pressure drop of 3.4 bar. The hose connecting the CDU with the Y connectors had having pressure sensor to measure pressure drop across the row manifold. The hoses connecting row manifold ports with rack manifolds have pressure sensors and flow sensors to measure the supply, return pressure, and flowrate, respectively.

The coolant used in this study was propylene glycol 25% (PG-25). The coolant was maintained at a set temperature by the CDU, this coolant then passes through the Y connectors in the setup and is distributed among two supply row manifolds. Supply row manifolds then distribute coolant to supply rack manifolds and row manifolds are connected to rack manifolds with the help of Eaton valve FD-83. From the supply rack, manifold coolant is then delivered to the cooling loops, where coolant captures heat from the IT and follows the path back to the CDU using the return rack manifold, row manifold, etc.

This paper focuses on the hydraulic characterization of row manifolds. For the experiments and characterizing the row manifold, ports of row manifolds were short-circuited using FD 83 valves. Out of 6 ports as shown in Fig. 2 below only 4 ports on the row manifolds were considered for this experiment, the first two and last two ports. Out of two row manifolds (one of copper and another one was steel) in parallel, one row manifold was kept open and allowed the coolant to pass through it at a time while another one was kept closed. The row manifolds were characterized at 21 °C inlet coolant temperature and the flowrate of the coolant varied from 40 to 165 lpm. During the experiment, pressure drop across the row manifold, and each port supply pressure and the flowrate were measured.

In the experiments, pressure sensors and flow sensors were used. For the calibration of the pressure sensors Fluke P5510-2M Pneumatic Comparison Test pump was utilized as shown in Fig. 3, pressure sensors used were Keyence GP-M010. The pressure was increased in the test rig and at different pressure readings the sensor's values, and the reference pressure gauge readings were taken for error analysis. Similarly, for the flow sensor calibrations, Coriolis flowmeter was utilized to calibrate the Keyence FDQ-32C flow sensor. The K-type thermocouples used to measure the fluid temperature were calibrated with Fluke 7109 A calibration bath. A two-point calibration method was used between temperatures of 0–100 °C as shown in Fig. 3, Table 1 shown below shows the details of the sensors used in this study. The calibration of the pressure sensors showed that factory-calibrated sensors were very precise and closely aligned with the reference pressure gauge. The calibration equation obtained was directly used as input in the DAQ software as gain and offset values. To calibrate the ultrasonic flow sensors, a calibrated Electromagnetic flow sensor was used by placing the flow sensor in the same closed-loop along with the Electromagnetic flowmeter. Tables 2–4 shows the error calculation quantified from the calibration process for pressure sensors, temperature sensors, and flow sensors [24].

where K is loss coefficient, ρ is coolant density which is PG25 in this study, Q is coolant flowrate, and A is flow area. The bulk temperatures of each cooling stream are calculated from the component-defined heat which transfers to flow streams and mixes the flow streams at different nodes. The Nusselt number of each component is predicted using Eq. (1).

The simulation process starts with processor-level CFD modeling to measure the thermal and hydraulic performances of the cooling system. In direct-to-chip liquid cooling, the processor-level cooling system is the cold plate. Then, the performance curves are used as the input for the FNM model of a server-level cooling loop. All the components (tubes, QDs, valves, Tees, etc.) are included in the cooling loop FNM model. Pressure drop and thermal resistance data of the cooling loop are used as the input for the next step which is the rack-level modeling. In the rack FNM model, the elevation of each server is specified inside the rack, and the uniformity of flow distribution is checked. Finally, the flow network of a data center can be modeled to measure the thermal performance and pressure drop of the whole secondary loop for the required flowrate. CDU's performance will be evaluated for the designed loop in terms of having enough cooling capacity and pressure budget. Here, we used Macro-Flow to model the flow network of the studied loop.

As explained in the experimental setup and procedure section, two-row manifolds were characterized in this study, but the results of the one-row manifold are discussed below because of the similarities in the results. The results in this section are divided into two parts, in the first part experimental results will be discussed and in the later part, experimental results will be compared with 1D flow network simulation results.

In the first set of experiments copper, material-based row manifold was characterized. The flowrate varied from 40 lpm to 167 lpm with a coolant supply temperature of 21 °C. Figure 4 shown below shows the flowrate going to the entire row manifold and each individual port. When the system flowrate was around 166.2 lpm the flowrate going to each individual port was around 41.2 lpm, where the maximum difference between the two ports' flowrate was 0.66 lpm, which lies in the range of flow sensor accuracy. Similarly, when the low flowrate of 40 lpm passed through the system, the flowrate going to each individual port was around 9.8 lpm, with the maximum difference in flowrate between the two ports being 1 lpm.

The graph shown below in Fig. 5 shows the copper row manifold pressure drop versus flowrate graph. The row manifold had shown a pressure drop of negligible at the flowrate of 40 lpm and of 4.5 psi (31.02 kPa) at the flowrate of 166 lpm. Figure 6 shows the supply pressure at the inlet of each port of the row manifold, the average supply pressure at 40 lpm is 15.5 psi (106.87 kPa) and at 166 lpm was 19.2 psi (132.38 kPa). Though each port shows some variation in the supply pressure at each port, but it was between the pressure sensors' accuracy range.

In this section, a flow network model is used to demonstrate the pressure drop in each component and verify the experimental results at the maximum tested flowrate (166 lpm). The network representation of the deployment is depicted in Fig. 7. Including the piping, valves, sensors, and all the fittings that are involved. The flow characteristic of each component is introduced to the model to accurately measure the pressure drop of the whole loop. The results are presented in Table 5 showing the contribution of QD, Belimo valves, ball valves, sensor connections, and all other fittings. As is demonstrated in Table 5, the total pressure drop for the whole row manifold is 4.07 psi (28.06 kPa) which is approximately 0.35 psi (2.41 kPa) less than the measured pressure drop across the whole loop between point 1 and 2 indicated in Fig. 6. The coefficient factor (Cv) for rack valve and sensors are 8.7 and 389, respectively. For FD-8 and ball valves, a PQ curve has been used as an input for the model based on the experimental results shown in Fig. 8. Then, the supply pressure is calculated at each port of the row manifold with the flowrate of 40 lpm and presented in Table 6. The uniformity of the flow distribution can be understood by comparing the pressure results with having less than 0.05 psi (0.34 kPa) difference. The FNM model indicates more uniformity in comparison with test results which have around 0.5 psi (3.44 kPa) difference between the minimum and maximum pressures.

In this paper, the methodology is developed to characterize the row manifolds as no literature was found related to liquid-cooled data center row manifolds. The six-port row manifolds' P-Q curves were generated, and the value of supply pressure and the flowrate were measured at each port, no IT hardware with cooling loops or row manifolds was included in the loop to add heat loads. The results obtained from the experiments were validated by a technique called flow network modeling (FNM). FNM is a 1D simulation suited for the analysis of flow distribution in liquid cooling systems. The designing and characterization of row manifold and its sizing impact the multiple factors like selection of CDU's and dampening of fluctuation of supply coolant temperature at low loads. Which will be addressed in future papers. During the experiments and FNM modeling, it was observed that similar coolant flowrates were achieved through all the ports of the row manifolds with the variation of 1 lpm, which lies under the flow sensor accuracy. The pressure drop across the row manifold was around 4.4 psi (30.34 kPa) at the flowrate of 166 lpm and a negligible pressure drop was observed at the flowrate of 36 lpm. The results obtained from the FNM match the experimental results with a maximum variation of 10%.

The authors attest that all data for this study are included in the paper.