Optimal Control of Hybrid Energy Storage System Based on Deep Reinforcement Learning

doi:10.16628/j.cnki.2095-8188.2021.11.007

Abstract

Abstract:

Under the islanding operation mode of DC micro-grid,when the photovoltaic output or load changes suddenly,the DC bus voltage fluctuates greatly.To stabilize the DC voltage,hybrid energy storage system charges and discharges quickly,but traditional proportional integral (PI) control is difficult to achieve good control effect.Therefore,an optimal control method based on deep reinforcement learning is proposed.Firstly,the nonlinear characteristics of DC voltage control are analyzed; the deep reinforcement learning algorithm framework and learning process based on input/output data are given.The state space,action space,reward function and neural network are designed to realize the optimal control of DC/DC converter in hybrid energy storage system.Finally,the simulation in DC micro-grid system shows that the proposed method can reduce the DC bus voltage fluctuation and improve the stability of the system compared with PI control.

Key words: DC micro-grid, hybrid energy storage system, deep reinforcement learning, optimal control

CLC Number:

TM912

LI Hong, ZHU Liwei, GUO Zhaohui. Optimal Control of Hybrid Energy Storage System Based on Deep Reinforcement Learning[J]. LOW VOLTAGE APPARATUS, 2021, 0(11): 42-49.

Figures/Tables 14

References 23

[1]	张怀宇, 徐超然, 黄志龙, 等. 考虑光伏发电不确定性的源-网-荷-储协调调度方法[J]. 电器与能效管理技术, 2021(5):86-92.
[2]	倪爽, 崔承刚, 郑庆荣, 等. 基于多风电相关性场景生成法的配电网随机多目标无功优化[J]. 浙江电力, 2020, 39(11):103-111.
[3]	刘颖明, 陈亮, 王晓东, 等. 混合储能参与风电集群一次调频的容量配置优化[J]. 电器与能效管理技术, 2020(10):55-63.
[4]	吴仁光, 郑立, 李凯鹏, 等. 面向综合能源配电网的储能系统优化配置方法[J]. 广东电力, 2020, 33(3):42-50.
[5]	孔昱凯, 温步瀛, 朱振山, 等. 含储能的孤岛微电网日前-日内协调优化运行[J]. 电器与能效管理技术, 2021(2):65-74.
[6]	徐虹卿, 张宇献, 邢作霞. 考虑风电不确定性的风电储能混合系统协调优化计算[J]. 电器与能效管理技术, 2019(9):53-59,70.
[7]	栾德利, 张宇献. 计及可靠性的风光储混合系统优化配置策略[J]. 电器与能效管理技术, 2020(11):95-102.
[8]	沈乃君, 高杰, 何新. 一种基于混合储能系统的风光功率平滑控制策略研究[J]. 电器与能效管理技术, 2019(1):62-68.
[9]	林崇, 陈佳桥, 王金友, 等. 新型混合储能功率变换器及控制策略研究[J]. 电器与能效管理技术, 2019(20):26-31.
[10]	刘文康, 谷志锋, 葛孟超, 等. 混合储能系统反演自适应变结构控制[J]. 电力电容器与无功补偿, 2020, 41(1):215-220.
[11]	KOLLIMALLA S K, MISHRA M K, NARASAMMA N L. Design and analysis of novel control strategy for battery and supercapacitor storage system[J]. IEEE Transactions on Sustainable Energy, 2014, 5(4):1137-1144. doi: 10.1109/TSTE.2014.2336896
[12]	ZHU F Q, YANG Z P, XIA H, et al. Hierarchical control and full-range dynamic performance optimization of the supercapacitor energy storage system in urban railway[J]. IEEE Transactions on Industrial Electronics, 2018, 65(8):6646-6656. doi: 10.1109/TIE.2017.2772174
[13]	王坦坦, 孙树敏, 王楠, 等. 基于混合储能的微电网控制策略研究[J]. 现代电子技术, 2020, 43(21):119-121,126.
[14]	DUAN J J, YI Z H, SHI D, et al. Reinforcement-learning-based optimal control of hybrid energy storage systems in hybrid AC-DC microgrids[J]. IEEE Transactions on Industrial Informatics, 2019, 15(9):5355-5364. doi: 10.1109/TII.9424
[15]	TANG Y H, HU W H, XIAO J, et al. Reinforcement learning based efficiency optimization scheme for the DAB DC-DC converter with triple-phase-shift modulation[J]. IEEE Transactions on Industrial Electronics, 2021, 68(8):7350-7361. doi: 10.1109/TIE.2020.3007113
[16]	ZHOU L, SWAIN A, UKIL A. Reinforcement Learning controllers for enhancement of low voltage ride through capability in hybrid power systems[J]. IEEE Transactions on Industrial Informatics, 2020, 16(8):5023-5031. doi: 10.1109/TII.9424
[17]	王之伟, 陆晓, 刁瑞盛, 等. 基于深度强化学习的电网自主控制与决策技术[J]. 电力工程技术, 2020, 39(6):34-43.
[18]	WU S, LI G Q, DENG L, et al. L1-Norm batch normalization for efficient training of deep neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(7):2043-2051. doi: 10.1109/TNNLS.2018.2876179
[19]	AWATE Y P. Policy-Gradient based Actor-Critic algorithms [C]//WRI Global Congress on Intelligent Systems, 2009:505-509.
[20]	SUTTON R S, MCALLESTER D A, SINGH S P, et al. Policy gradient methods for reinforcement learning with function approximation [C]//Neural Information Processing Systems, 1999:1057-1063.
[21]	LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning [C]//International Conference on Learning Representations, 2016:219-225.
[22]	GHEISARNEJAD M, FARSIZADEH H, KHOOBAN M H. A novel nonlinear deep reinforcement learning controller for DC-DC power Buck converters[J]. IEEE Transactions on Industrial Electronics, 2021, 68(8):6849-6858. doi: 10.1109/TIE.2020.3005071
[23]	SUTTON R S, BARTO A G. Reinforcement learning:An introduction [M]. Cambridge: MIT Press, 1998.

参数	数值
开关频率/kHz	10
R₁_f/Ω	0.2
L₁_f/mH	5
R₂_f/Ω	0.2
L₂_f/mH	5
C/μF	4 700

参数	数值	参数	数值
迭代次数/次	1 000	经验回放区长度	1×10⁶
采样时间/s	1×10^-4	更新系数τ	1×10^-3
动作学习率	1×10^-3	目标平滑因子	1×10^-3
批评学习率	1×10^-3	目标策略平滑噪声方差	0.3
折扣因子	0.92	目标策略平滑上限	0.5
小批量大小	64	目标策略平滑下限	-0.5

控制方法	电压波动/V
控制方法	光伏突变	负荷突变
PI控制	36 (5.5%)	35 (5.4%)
DDPG控制	22 (3.4%)	20 (3.1%)