Research Papers: Design for Manufacture and the Life Cycle

Automated Design of Energy Efficient Control Strategies for Building Clusters Using Reinforcement Learning

[+] Author and Article Information
Philip Odonkor

Department of Mechanical and
Aerospace Engineering,
University at Buffalo-SUNY,
Buffalo, NY 14260
e-mail: podonkor@buffalo.edu

Kemper Lewis

Department of Mechanical and
Aerospace Engineering,
University at Buffalo-SUNY,
Buffalo, NY 14260
e-mail: kelewis@buffalo.edu

1Corresponding author.

Contributed by the Design Automation Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received July 6, 2018; final manuscript received September 21, 2018; published online December 20, 2018. Assoc. Editor: Harrison M. Kim.

J. Mech. Des 141(2), 021704 (Dec 20, 2018) (9 pages) Paper No: MD-18-1541; doi: 10.1115/1.4041629 History: Received July 06, 2018; Revised September 21, 2018

The control of shared energy assets within building clusters has traditionally been confined to a discrete action space, owing in part to a computationally intractable decision space. In this work, we leverage the current state of the art in reinforcement learning (RL) for continuous control tasks, the deep deterministic policy gradient (DDPG) algorithm, toward addressing this limitation. The goals of this paper are twofold: (i) to design an efficient charged/discharged dispatch policy for a shared battery system within a building cluster and (ii) to address the continuous domain task of determining how much energy should be charged/discharged at each decision cycle. Experimentally, our results demonstrate an ability to exploit factors such as energy arbitrage, along with the continuous action space toward demand peak minimization. This approach is shown to be computationally tractable, achieving efficient results after only 5 h of simulation. Additionally, the agent showed an ability to adapt to different building clusters, designing unique control strategies to address the energy demands of the clusters studied.

Copyright © 2019 by ASME
Your Session has timed out. Please sign back in to continue.


Amin, M. , 2013, “ Energy: The Smart-Grid Solution,” Nature, 499(7457), p. 145. [CrossRef] [PubMed]
Odonkor, P. , Lewis, K. , Wen, J. , and Wu, T. , 2016, “ Adaptive Energy Optimization Toward Net-Zero Energy Building Clusters,” ASME J. Mech. Des., 138(6), p. 061405.
Korpaas, M. , Holen, A. T. , and Hildrum, R. , 2003, “ Operation and Sizing of Energy Storage for Wind Power Plants in a Market System,” Int. J. Electr. Power Energy Syst., 25(8), pp. 599–606. [CrossRef]
Perez, R. , David, M. , Hoff, T. E. , Jamaly, M. , Kivalov, S. , Kleissl, J. , Lauret, P. , and Perez, M. , 2016, “ Spatial and Temporal Variability of Solar Energy,” Found. Trends Renewable Energy, 1(1), pp. 1–44. [CrossRef]
Sommerfeld, J. , Buys, L. , and Vine, D. , 2017, “ Residential Consumers Experiences in the Adoption and Use of Solar PV,” Energy Policy, 105, pp. 10–16. [CrossRef]
Crabtree, G. , 2015, “ Perspective: The Energy-Storage Revolution,” Nature, 526(7575), pp. S92–S92. [CrossRef] [PubMed]
Lund, P. D. , Lindgren, J. , Mikkola, J. , and Salpakari, J. , 2015, “ Review of Energy System Flexibility Measures to Enable High Levels of Variable Renewable Electricity,” Renewable Sustainable Energy Rev., 45, pp. 785–807. [CrossRef]
Ted, C. , 2016, “ Tesla Powerwall: Not Just for Solar,” Hanley Wood Media, Inc., Washington, DC, accessed Oct. 12, 2018, https://goo.gl/QudGAR
Le, H. T. , and Nguyen, T. Q. , 2008, “ Sizing Energy Storage Systems for Wind Power Firming: An Analytical Approach and a Cost-Benefit Analysis,” IEEE Power and Energy Society General Meeting-Conversion and Delivery of Electrical Energy in the 21st Century, Pittsburgh, PA, July 20–24, pp. 1–8.
Business Wire, 2014, “ Sonnenbatterie Enters us Market With First Distribution Deal,” Business Wire, San Francisco, CA, accessed Oct. 12, 2018, https://goo.gl/hFd7ix
Deign, J. , 2015, “ Sungevity and Sonnenbatterie to Undercut Tesla and SolarCity on Storage Pricing?,” Greentech Media, Boston, MA, accessed Oct. 12, 2018, https://goo.gl/Yr9kpj
GMP, 2017, “ Green Mountain Power,” GMP, Colchester, VT, accessed Oct. 12, 2018, https://goo.gl/wVhhzq
Alharbi, H. , and Bhattacharya, K. , 2018, “ Stochastic Optimal Planning of Battery Energy Storage Systems for Isolated Microgrids,” IEEE Trans. Sustainable Energy, 9(1), pp. 211–227. [CrossRef]
Chakraborty, S. , Senjyu, T. , Toyama, H. , Saber, A. , and Funabashi, T. , 2009, “ Determination Methodology for Optimising the Energy Storage Size for Power System,” IET Gener., Transm. Distrib., 3(11), pp. 987–999. [CrossRef]
Bhandari, R. , and Stadler, I. , 2009, “ Grid Parity Analysis of Solar Photovoltaic Systems in Germany Using Experience Curves,” Sol. Energy, 83(9), pp. 1634–1644. [CrossRef]
Kou, P. , Gao, F. , and Guan, X. , 2015, “ Stochastic Predictive Control of Battery Energy Storage for Wind Farm Dispatching: Using Probabilistic Wind Power Forecasts,” Renewable Energy, 80, pp. 286–300. [CrossRef]
Divya, K. , and Østergaard, J. , 2009, “ Battery Energy Storage Technology for Power systems - An Overview,” Electric Power Syst. Res., 79(4), pp. 511–520. [CrossRef]
Hadjipaschalis, I. , Poullikkas, A. , and Efthimiou, V. , 2009, “ Overview of Current and Future Energy Storage Technologies for Electric Power Applications,” Renewable Sustainable Energy Rev., 13(6–7), pp. 1513–1522. [CrossRef]
Wang, X. , Vilathgamuwa, D. M. , and Choi, S. S. , 2008, “ Determination of Battery Storage Capacity in Energy Buffer for Wind Farm,” IEEE Trans. Energy Convers., 23(3), pp. 868–878. [CrossRef]
Venu, C. , Riffonneau, Y. , Bacha, S. , and Baghzouz, Y. , 2009, “ Battery Storage System Sizing in Distribution Feeders With Distributed Photovoltaic Systems,” IEEE Bucharest PowerTech, Bucharest, Romania, June 28–July 2, pp. 1–5.
Castillo-Cagigal, M. , Caamano-Martín, E. , Matallanas, E. , Masa-Bote, D. , Gutiérrez, A. , Monasterio-Huelin, F. , and Jiménez-Leube, J. , 2011, “ PV Self-Consumption Optimization With Storage and Active Dsm for the Residential Sector,” Sol. Energy, 85(9), pp. 2338–2348. [CrossRef]
Guille, C. , and Gross, G. , 2009, “ A Conceptual Framework for the Vehicle-to-Grid (v2g) Implementation,” Energy Policy, 37(11), pp. 4379–4390. [CrossRef]
Wu, X. , Hu, X. , Moura, S. , Yin, X. , and Pickert, V. , 2016, “ Stochastic Control of Smart Home Energy Management With Plug-in Electric Vehicle Battery Energy Storage and Photovoltaic Array,” J. Power Sources, 333, pp. 203–212. [CrossRef]
Haines, G. , McGordon, A. , Jennings, P. , and Butcher, N. , 2009, “ The Simulation of Vehicle-to-Home Systems—Using Electric Vehicle Battery Storage to Smooth Domestic Electricity Demand,” International Conference on Ecologic Vehicles and Renewable Energies, Monaco, Mar. 26–29. https://www.researchgate.net/publication/228696724_The_Simulation_of_Vehicle-to-Home_Systems-Using_Electric_Vehicle_Battery_Storage_to_Smooth_Domestic_Electricity_Demand
Jian, L. , Xue, H. , Xu, G. , Zhu, X. , Zhao, D. , and Shao, Z. , 2013, “ Regulated Charging of Plug-in Hybrid Electric Vehicles for Minimizing Load Variance in Household Smart Microgrid,” IEEE Trans. Ind. Electron., 60(8), pp. 3218–3226. [CrossRef]
Fares, R. L. , and Webber, M. E. , 2017, “ The Impacts of Storing Solar Energy in the Home to Reduce Reliance on the Utility,” Nat. Energy, 2(2), p. 17001. [CrossRef]
Itron Consulting, 2016, “ 2016 SGIP Advanced Energy Storage Impact Evaluation,” Itron Consulting, Davis, CA, accessed on Oct 9, 2018, http://docs.cpuc.ca.gov/PublishedDocs/Published/G000/M194/K599/194599448.PDF
Fridgen, G. , Gründler, A. , and Rusic, M. , 2015, “ Energy Cooperatives as an Application of Microgrids: Multi-Criteria Investment Decision Support,” 36th International Conference on Information Systems (ICIS), Fort Worth, TX, Dec. 13–16, pp. 1–20. https://aisel.aisnet.org/cgi/viewcontent.cgi?article=1653&context=icis2015&usg=AFQjCNHTIeza7NOWDj1vLJpsn5szRfm9PQ
Hu, M. , Weir, J. D. , and Wu, T. , 2012, “ Decentralized Operation Strategies for an Integrated Building Energy System Using a Memetic Algorithm,” Eur. J. Oper. Res., 217(1), pp. 185–197. [CrossRef]
Odonkor, P. , and Lewis, K. , 2016, “ Optimization of Energy Use Strategies in Building Clusters Using Pareto Bands,” ASME Paper No. IDETC2016-59963.
Ren, Y. , Bayrak, A. E. , and Papalambros, P. Y. , 2016, “ Ecoracer: Game-Based Optimal Electric Vehicle Design and Driver Control Using Human Players,” ASME J. Mech. Des., 138(6), p. 061407. [CrossRef]
Sexton, T. , and Ren, M. Y. , 2017, “ Learning an Optimization Algorithm Through Human Design Iterations,” ASME J. Mech. Des., 139(10), p. 101404. [CrossRef]
Sutton, R. S. , and Barto, A. G. , 1998, Reinforcement Learning: An Introduction, Vol. 1, MIT Press, Cambridge, UK.
Duch, W. , and Mandziuk, J. , eds., 2007, Challenges for Computational Intelligence, Vol. 63, Springer Science & Business Media, Berlin.
Silver, D. , Schrittwieser, J. , Simonyan, K. , Antonoglou, I. , Huang, A. , Guez, A. , Hubert, T. , Baker, L. , Lai, M. , Bolton, A. , Chen, Y. , Lillicrap, T. , Hui, F. , Sifre, L. , Van den Driessche, G. , Graepel, T. , and Hassabis, D. , 2017, “ Mastering the Game of Go Without Human Knowledge,” Nature, 550(7676), p. 354. [CrossRef] [PubMed]
Barth-Maron, G. , Hoffman, M. W. , Budden, D. , Dabney, W. , Horgan, D. , TB, D. , Muldal, A. , Heess, N. , and Lillicrap, T. P. , 2018, “ Distributed Distributional Deterministic Policy Gradients,” CoRR, e-print arXiv:1804.08617. https://arxiv.org/abs/1804.08617
Pardo, D. , Möller, L. , Neunert, M. , Winkler, A. W. , and Buchli, J. , 2016, “ Evaluating Direct Transcription and Nonlinear Optimization Methods for Robot Motion Planning,” IEEE Rob. Autom. Lett., 1(2), pp. 946–953. [CrossRef]
Böhme, T. J. , and Frank, B. , 2017, “ Direct Methods for Optimal Control,” Hybrid Systems, Optimal Control and Hybrid Vehicles, Springer, Berlin, pp. 233–273.
Lillicrap, T. P. , Hunt, J. J. , Pritzel, A. , Heess, N. , Erez, T. , Tassa, Y. , Silver, D. , and Wierstra, D. , 2015, “ Continuous Control With Deep Reinforcement Learning,” e-print arXiv:1509.02971. https://arxiv.org/abs/1509.02971
Mnih, V. , Kavukcuoglu, K. , Silver, D. , Rusu, A. , Veness, J. , Bellemare, M. G. , Graves, A. , Riedmiller, M. , Fidjeland, A. , Ostrovski, G. , Petersen, S. , Beattie, C. , Sadik, A. , Antonoglou, I. , King, H. , Kumaran, D. , Wierstra, D. , Legg, S. , and Hassabis, D. , 2015, “ Human-Level Control Through Deep Reinforcement Learning,” Nature, 518(7540), p. 529. [CrossRef] [PubMed]
Silver, D. , Lever, G. , Heess, N. , Degris, T. , Wierstra, D. , and Riedmiller, M. , 2014, “ Deterministic Policy Gradient Algorithms,” International Conference on Machine Learning, Beijing, China, June 21–26, pp. 387–395. http://proceedings.mlr.press/v32/silver14.pdf
Uhlenbeck, G. E. , and Ornstein, L. S. , 1930, “ On the Theory of the Brownian Motion,” Phys. Rev., 36(5), p. 823. [CrossRef]
Keskar, N. S. , Mudigere, D. , Nocedal, J. , Smelyanskiy, M. , and Tang, P. T. P. , 2016, “ On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima,” CoRR, e-print arXiv:1609.04836 https://arxiv.org/abs/1609.04836.
City of Austin, 2017, “ Residential Electric Rate Schedules,” City of Austin, Austin, TX, accessed Oct. 12, 2018, https://goo.gl/Atd66W
Andrychowicz, M. , Wolski, F. , Ray, A. , Schneider, J. , Fong, R. , Welinder, P. , McGrew, B. , Tobin, J. , Abbeel, P. , and Zaremba, W. , 2017, “ Hindsight Experience Replay,” CoRR, e-print arXiv:1707.01495 https://arxiv.org/abs/1707.01495.
Tobin, J. , Fong, R. , Ray, A. , Schneider, J. , Zaremba, W. , and Abbeel, P. , 2017, “ Domain Randomization for Transferring Deep Neural Networks From Simulation to the Real World,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada, Sept. 24–28, pp. 23–30.


Grahic Jump Location
Fig. 1

Architecture of the proposed shared battery dispatch algorithm

Grahic Jump Location
Fig. 2

Basic agent-environment interaction in reinforcement learning

Grahic Jump Location
Fig. 3

Architecture of the proposed DDPG and battery setup

Grahic Jump Location
Fig. 4

Energy demand profiles of all four homes considered

Grahic Jump Location
Fig. 5

Training results for cluster 1

Grahic Jump Location
Fig. 6

Effect of dispatch strategy on cluster 1's demand profile

Grahic Jump Location
Fig. 7

Training results for building cluster 2

Grahic Jump Location
Fig. 8

Effect of dispatch strategy on cluster 2's demand profile

Grahic Jump Location
Fig. 9

Training results for building cluster 3

Grahic Jump Location
Fig. 10

Effect of dispatch strategy on cluster 3's demand profile

Grahic Jump Location
Fig. 11

Training results using increased episode length



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In