[1]杨继阳,欧阳权,丛玉华,等.基于 Transformer 改进强化学习的无人机电力巡检规划[J].机械与电子,2024,42(10):54-60.
　YANG Jiyang,OUYANG Quan,CONG Yuhua,et al.UAV Power Inspection Planning Based on Transformer Improved Reinforcement Learning[J].Machinery & Electronics,2024,42(10):54-60.
点击复制

基于 Transformer 改进强化学习的无人机电力巡检规划()
分享到：

《机械与电子》[ISSN:1001-2257/CN:52-1052/TH]

卷:
42
期数:
2024年10期
页码:
54-60
栏目:
机电一体化
出版日期:
2024-10-30

文章信息/Info

Title:
UAV Power Inspection Planning Based on Transformer Improved Reinforcement Learning
文章编号:
1001-2257 ( 2024 ) 10-0054-07
作者:
杨继阳; 欧阳权; 丛玉华; 王瑞群; 王志胜
南京航空航天大学自动化学院,江苏 南京 210016
Author(s):
YANG Jiyang ;  OUYANG Quan ;  CONG Yuhua ;  WANG Ruiqun ;  WANG Zhisheng
( College of Automation Engineering , Nanjing University of Aeronautics and Astronautic , Nanjing 210016 , China )
关键词:
无人机; 电力巡检; 轨迹规划;  Transformer ; 强化学习
Keywords:
drone ;  power inspection ;  trajectory planning ;  Transformer ;  reinforcement learning
分类号:
TP273
文献标志码:
A
摘要:
为实现无人机电力巡检过程的全自主决策,针对传统强化学习轨迹规划存在的收敛速度慢、易陷入局部最优的问题,基于 Transformer 模型改进深度强化学习,设计了电量约束下的无人机充电巡检决策算法。首先建立对电力巡检任务场景的能耗模型和马尔可夫决策模型。然后分别设计了基于图神经网络的静态编码器和基于门控循环的动态编码器以提取不同类型环境数据,同时设计了基于多头注意力机制的解码器,输出不定长的全局充电巡检策略序列以预测未来奖励。最后对收敛后的推理模型在电力巡检仿真环境进行验证。仿真结果表明,相比于传统强化学习,所提算法可以提取地图深层状态特征,路径能耗降低了 26.61% ,并具有更好的收敛性。
Abstract:
To achieve autonomous decision making in the process of drone power inspection and solve the issues of slow convergence and susceptibility to local optima in traditional reinforcement learning trajectory planning , this paper propose an improved deep reinforcement learning approach based on the Transformer model , which designs a drone charging inspection decision making algorithm under the constraint of battery capacity.Firstly , an energy consumption model and a Markov decision model are established for the power inspection task scenario.Then , static and dynamic encoders based on graph neural networks ( GNN ) and gated recurrent units ( GRU ) are designed to extract different types of environmental data.The multi-head pointer network is employed to plan a global charging inspection strategy and predict future rewards.Finally , the converged inference model is validated in a power inspection simulation environment. Simulation results demonstrate that compared to traditional reinforcement learning , the proposed algorithm can extract deep-level map features , path energy consumption reduced by 26.61% , while achieving better convergence.

参考文献/References:

[ 1 ] 毛建旭,贺振宇,王耀南,等 . 电力巡检机器人路径规划技术及应用综述[ J ] . 控制与决策, 2023 , 38 ( 11 ): 3009-3024.
[ 2 ] 胡腾,倪航,方力,等 . 智能巡检机器人研究现状与发展方向分析[ J ] . 机器人产业, 2022 ( 3 ): 30-35.
[ 3 ] TAKAYA K , OHTA H , KROUMOV V , et al.Development of UAV system for autonomous power line inspection [ C ]// 2019 23rd International Conference on System Theory , Control and Computing ( ICSTCC),2019 : 762-767.
[ 4 ] 冯新江,林泽科,陈岳贤,等 . 无人机电力巡检系统研究 [ J ] . 机电信息, 2021 ( 5 ): 1-4.
[ 5 ] 李冬 . 输电杆塔无人机巡检路径规划技术研究[ D ] . 长沙:湖南大学,2020.
[ 6 ] 徐其春,郭晨晨,刘志明,等 . 无人机线路自主巡检的动态轨迹规划方法[ J ] . 电力系统及其自动化学, 2022 , 34( 10 ): 24-31.
[ 7 ] 田晓红 . 四旋翼无人机输电杆塔巡检的能耗模型构建及应用研究[ D ] . 南宁:广西大学,2021.
[ 8 ] 牟治宇,张煜,范典,等 . 基于深度强化学习的无人机数据采集和路径规划研究[ J ] . 物联网学报, 2020 , 4 ( 3 ):42-51.
[ 9 ] 田子玉 . 基于强化学习的无人机数据收集路径规划策略研究[ D ] . 长春:吉林大学,2023.
[ 10 ] 唐馨铭 . 基于强化学习的固定翼无人机控制与路径规划[ D ] . 重庆:重庆大学,2022.
[ 11 ] VASWANI A , SHAZEER N , PARMAR N , et al.Attention is all you need [ J ] .Advances in neural information processing systems , 2017 , 30 : 5998-6008.
[ 12 ] BENGIO Y , LODI A , PROUVOST A.Machine learning for combinatorial optimization : a methodological tour d ’ horizon [ J ] .European journal of operational research , 2021 , 290 ( 2 ): 405-421.
[ 13 ] WANG C , CUI Y L , DENG D H , et al.Trajectory optimization and power allocation scheme based on DRL in energy efficient UAV-aided communication networks [ J ] .Chinese journal of
electronics , 2022 , 31 ( 3 ):397-407.
[ 14 ] LIU Z L , SENGUPTA R , KURZHANSKIY A.A power consumption model for multi-rotor small unmanned aircraft systems [ C ]// 2017 International Conference on Unmanned Aircraft Systems ( ICUAS ), 2017 : 310-315.
[ 15 ] KARIMI-MAMAGHAN M , MOHAMMADI M , MEYER P , et al.Machine learning at the service of metaheuristics for solving combinatorial optimization problems : a state of the art [ J ] .European journal of operational research , 2022 , 296 ( 2 ):393-422.
[ 16 ] YANG X Y , WANG Z Y , ZHANG H X , et al.A review : machine learning for combinatorial optimization problems in energy areas [ J ] .Algorithms , 2022 , 15( 6 ): 205.
[ 17 ] GASSE M , CH?TELAT D , FERRONI N , et al.Exact comb-inatorial optimization with graph convolutional neural networks [ J ] .Advances in neural information processing systems , 2019 , 32 : 15580-15592.
[ 18 ] VINYALS O , FORTUNATO M , JAITLY N.Pointer networks [ J ] .Advances in neural information processing systems , 2015 , 28 : 2692-2700.
[ 19 ] WU Z H , PAN S R , CHEN F W , et al.A comprehensive survey on graph neural networks [ J ] . IEEE Transactions on neural networks and learning systems , 2020 , 32 ( 1 ): 4-24.
[ 20 ] GULCEHRE C , CHO K , PASCANU R , et al.Learnednorm pooling for deep feedforward and recurrent neural networks [ C ]// Joint European Conference on Machine Learning and Knowledge Discovery in Databases , 2014 : 530-546.
[ 21 ] FUKUI H , HIRAKAWA T , YAMASHITA T , et al. Attention branch network : learning of attention mechanism for visual explanation [ C ]// 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR ),2019 : 10705-10714.
[ 22 ] JAMSHIDI V , NEKOUKAR V , REFAN M H.Analysis of parallel genetic algorithm and parallel particle swarm optimization algorithm UAV path planning on controller area network [ J ] .Journal of control , automation and electrical systems , 2020 , 31 ( 1 ): 129-140.

相似文献/References:

[1]魏志强,葛 珊.无人机载光电吊舱双路视频压缩存储系统设计与实现[J].机械与电子,2019,(06):32.
　.Design and Realization of Dual-channel Video Compression Storage System for UAV Electro-optical Pod[J].Machinery & Electronics,2019,(10):32.
[2]陈斯昀１,蒋 刚２,留沧海２.六足机器人协同无人机大气数据分析系统[J].机械与电子,2019,(05):71.
　,The Atmospheric Data Analysis System of Hexapod Robot Collaborated with the Unmanned Aerial Vehicle[J].Machinery & Electronics,2019,(10):71.
[3]冷献春,曾俊杰.某型无人机卫通设备的环境适应性设计[J].机械与电子,2019,(02):45.
　.Environmental Adaptability Design of a Certain Type of UAV Satcom Equipment[J].Machinery & Electronics,2019,(10):45.
[4]许 军,张军红,荣海春.小型无人机火箭助推发射动态响应研究[J].机械与电子,2018,(01):15.
　XU Jun,ZHANG Junhong,RONG Haichun.Research on Dynamic Response of Rocket Booster Launching for Small Unmanned Aerial Vehicle[J].Machinery & Electronics,2018,(10):15.
[5]李晓萌,王道波,郭继凯,等.基于某种生物启发式算法的无人机航迹规划[J].机械与电子,2018,(11):15.
　LI Xiaomeng,WANG Daobo,GUO Jikai,et al.Path Planning of UAV Based on a Biological Heuristic Algorithm[J].Machinery & Electronics,2018,(10):15.
[6]明瑞晨,李吉宽,李 刚,等.无人机纵向着舰下滑轨迹控制技术研究[J].机械与电子,2018,(08):44.
　MING Ruichen,LI Jikuan,LI Gang,et al.Research on Longitudinal Automatic Carrier Landing Control for Unmanned Aerial Vehicle[J].Machinery & Electronics,2018,(10):44.
[7]吴 斌,荣海春,王呈顺,等.某无人机载雷达挂架结构设计[J].机械与电子,2019,(12):26.
　,,et al.Structure Design of the Bracket Used in an Unmanned Aerial Vehicle Carried Radar[J].Machinery & Electronics,2019,(10):26.
[8]许 哲,张少帅,郭 璐,等.无人机深度学习去雾算法[J].机械与电子,2021,(04):13.
　XU Zhe,ZHANG Shaoshuai,GUO Lu,et al. Deep Learning Defogging Algorithm for UAV[J].Machinery & Electronics,2021,(10):13.
[9]黄 莺 １,张少帅 ２,黄 鹤 ２,等.无人机遥感山区公路图像去噪方法[J].机械与电子,2021,(09):66.
　HUANG Ying,ZHANG Shaoshua,HUANG He,et al.A Denoising Method of Remote Sensing Mountainous Road Image by UAV[J].Machinery & Electronics,2021,(10):66.
[10]王瑞群,欧阳权,段朝伟,等.基于强化学习的无人机全自主电力巡检[J].机械与电子,2021,(12):34.
　WANG Ruiqun,OUYANG Quan,DUAN Chaowei,et al.Autonomous Power Inspection of UAV Based on Reinforcement Learning[J].Machinery & Electronics,2021,(10):34.

备注/Memo

备注/Memo:
收稿日期: 2024-04-02
基金项目:国家自然科学基金资助项目( 61473144 )
作者简介:杨继阳 ( 1999- ),男,安徽淮北人,硕士研究生,研究方向为优化控制、强化学习等;欧阳权 ( 1991- ),男,湖北仙桃人,博士,硕士研究生导师,研究方向为新能源系统集成与控制、无人机飞行控制等,通信作者;丛玉华 ( 1981- ),女,山东烟台人,博士,讲师,研究方向为跨域协同、无人机飞行控制等;王瑞群 ( 1999- ),男,河北邯郸人,硕士研究生,研究方向为深度学习、飞行控制等;王志胜 ( 1970- ),男,湖北荆门人,博士研究生导师,研究方向为智能机器人技术、智能感知与信息融合等。
更新日期/Last Update: 2024-10-31