Context-aware reinforcement learning for cooling operation of data centers with an Aquifer Thermal Energy Storage  

作　　者：Lukas Leindals Peter Grønning Dominik Franjo Dominković Rune Grønborg Junker 

机构地区：[1]Department of Applied Mathematics and Computer Science,Technical University of Denmark,Denmark

出　　处：《Energy and AI》2024年第3期233-246,共14页能源与人工智能（英文）

基　　金：the project titled ‘Cool-Data Flexible Cooling of Data Centers’ and was financed by the Innovation Fund Denmark (nr. 0177-00066B).

摘　　要：Data centers are often equipped with multiple cooling units. Here, an aquifer thermal energy storage (ATES) system has shown to be efficient. However, the usage of hot and cold-water wells in the ATES must be balanced for legal and environmental reasons. Reinforcement Learning has been proven to be a useful tool for optimizing the cooling operation at data centers. Nonetheless, since cooling demand changes continuously, balancing the ATES usage on a yearly basis imposes an additional challenge in the form of a delayed reward. To overcome this, we formulate a return decomposition, Cool-RUDDER, which relies on simple domain knowledge and needs no training. We trained a proximal policy optimization agent to keep server temperatures steady while minimizing operational costs. Comparing the Cool-RUDDER reward signal to other ATES-associated rewards, all models kept the server temperatures steady at around 30 °C. An optimal ATES balance was defined to be 0% and a yearly imbalance of −4.9% with a confidence interval of [−6.2, −3.8]% was achieved for the Cool 2.0 reward. This outperformed a baseline ATES-associated reward of 0 at −16.3% with a confidence interval of [−17.1, −15.4]% and all other ATES-associated rewards. However, the improved ATES balance comes with a higher energy consumption cost of 12.5% when comparing the relative cost of the Cool 2.0 reward to the zero reward, resulting in a trade-off. Moreover, the method comes with limited requirements and is applicable to any long-term problem satisfying a linear state-transition system.

关 键 词：Data centers ATES Smart energy systems Reinforcement learning Delayed rewards 

分 类 号：TP30[自动化与计算机技术—计算机系统结构]