[1]兰海麟,吉琳娜,杨风暴,等.基于生成对抗网络的红外与可见光视频语义特征驱动融合方法[J].机械与电子,2026,44(01):35-44.
　LAN Hailin,JI Linna,YANG Fengbao,et al.Semantic Feature Driven Fusion Method for Infrared and Visible Videos Based on Generative Adversarial Networks[J].Machinery & Electronics,2026,44(01):35-44.
点击复制

基于生成对抗网络的红外与可见光视频语义特征驱动融合方法()
分享到：

《机械与电子》[ISSN:1001-2257/CN:52-1052/TH]

卷:
44
期数:
2026年01期
页码:
35-44
栏目:
智能检测
出版日期:
2026-01-27

文章信息/Info

Title:
Semantic Feature Driven Fusion Method for Infrared and Visible Videos Based on Generative Adversarial Networks
文章编号:
1001-2257(2026)01-0035-10
作者:
兰海麟; 吉琳娜; 杨风暴; 刘延东
(中北大学信息与通信工程学院,山西 太原 030051)
Author(s):
 LAN Hailin; JI Linna; YANG Fengbao; LIU Yandong
 (College of Information and Communications Engineering,North University of China,Taiyuan 030051,China)
关键词:
双模态视频; 语义特征; 目标检测; 深度学习
Keywords:
bimodal video; semantic features; object detection; deep learning
分类号:
TP391
文献标志码:
A
摘要:
针对现有视频融合方法过度关注视觉质量提升而忽略下游检测任务需求、且未实现目标区域差异化处理导致的性能瓶颈问题。提出了一种基于生成对抗网络的红外与可见光视频语义特征驱动融合方法。设计了语义特征嵌入模块(SFEM),通过动态生成的语义引导卷积核与计算源图像与目标图像在特征表示空间中的相似性,将语义特征有效融入融合网络中。构建了双路特征提取架构,分别通过纹理特征提取模块(TFEM)和对比度特征提取模块(CFEM)提取红外与可见光图像的对比度与细节信息,实现纹理、对比度与语义特征的深度融合,进而提升下游任务的表现。在训练阶段,集成了一个目标检测子模块,将检测损失反向传播到融合网络中,从而实现面向任务的融合过程优化。实验结果表明,该方法在平衡像素强度和保持目标物体纹理方面优于其他对比方法,并且在下游目标检测任务中表现出显著优势。
Abstract:
To address the performance bottleneck in existing video fusion methods—which often prioritize visual quality improvement while neglecting the requirements of downstream detection tasks and fail to achieve differentiated processing of target regions,a semantic feature driven fusion method for infrared and visible light videos based on GAN is proposed.A semantic feature embedding module (SFEM) is designed to effectively integrate semantic features into the fusion network.This module calculates the similarity between source and target images in the feature representation space by using dynamically generated semantic guided convolutional kernels and attention mechanisms.A dual path feature extraction architecture is constructed,comprising a texture feature extraction module (TFEM) and a contrast feature extraction module (CFEM).These modules separately extract contrast and detail information from infrared and visible light images,achieving deep fusion of texture,contrast,and semantic features,thereby enhancing downstream task performance.During the training phase,a target detection sub module is integrated.The detection loss is backpropagated into the fusion network,enabling task oriented optimization of the fusion process.Experimental results demonstrate that this method outperforms other comparative approaches in balancing pixel intensity and preserving target object texture,while also exhibiting significant advantages in downstream target detection tasks.

参考文献/References:

 [1] 邢静,张艺驰,高俊钗.基于HSV 色彩空间的改进红外与可见光图像融合算法研究[J].机械与电子,2022,40(12):15-19.
[2] 高训东,陈辉,姚亚宁,等.基于双分支特征分解的红外与可见光融合网络[J].激光与光电子学进展,2025,62(14):482-492.
[3] ZHANG Y,LIU Y,SUN P,et al.IFCNN:a general image fusion framework based on convolutional neural network[J].Information fusion,2020,54:99-118.
[4] ZHANG H,XU H,XIAO Y,et al.Rethinking the image fusion:a fast unified image fusion network based on proportional maintenance of gradient and intensity[C]∥Proceedings of the AAAI Conference on Artificial Intelligence,2020,34(7):12797-12804.
[5] HAO S,HE T,AN B Y,et al.VDFEFuse:a novel fusion approach to infrared and visible images[J].Infrared physics and technology,2022,121:104048.
[6] LI H,WU X J,DURRANI T.NestFuse:an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J].IEEE Transactions on instrumentation and measurement,2020,69:9645-9656.
[7] MA J Y,YU W,LIANG P W,et al.FusionGAN:a generative adversarial network for infrared and visible image fusion[J].Information fusion,2019,48:11-26.
[8] MA J Y,LIANG P W,YU W,et al.Infrared and visible image fusion via detail preserving adversarial learning[J].Information fusion ,2020,54:85-98.
[9] MA J Y,XU H,JIANG J,et al.DDcGAN:a dual discriminator conditional generative adversarial network for multi resolution image fusion[J]IEEE Transactions on image processing,2020,29:4980-4995.
[10] TANG L F,YUAN J T,MA J Y.Image fusion in the loop of high level vision tasks:a semantic aware real time infrared and visible image fusion network[J].Information fusion,2022,82:28-42.
[11] MA J Y,CHEN C,LI C,et al.Infrared and visible image fusion via gradient transfer and total variation minimization[J].Information fusion 2016,31:100-109.
[12] BAVIRISETTI D P,DHULI R.Fusion of infrared and visible sensor images based on anisotropic diffusion and Karhunen Loeve transform[J].IEEE Sensors journal,2015,16(1):203-209.
[13] TANG L F,XIANG X Y,MA J Y,et al.DIVFusion:darkness free infrared and visible image usion[J].Information fusion,2023,91:477-493.
[14] TANG L F,ZHANG H,MA J Y,et al. Rethinking the necessity of image fusion in high level vision tasks:a practical infrared and visible image fusion network based on progressive semantic injection and scene fidelity[J].Information fusion,2023,99:101870.
[15] LIU J,FAN X,HUANG Z,et al.Target aware dual adversarial learning and a multi scenario multi modality benchmark to fuse infrared and visible for object detection[C].Conference on Computer Vision and Pattern Recognition,2022:58025811.
[16] MA J Y,TANG L F,XU M L,et al.STDFusionNet:an infrared and visible image fusion network based on salient target detection[J]IEEE Transactions on instrumentation and measurement,2021,70:1-13.
[17] SRIVASTAVA S,DIVEKAR A V,ANILKUMAR C,et al.Comparative analysis of deep learning image detection algorithms [J].Journal of big data,2021,8(1):66.
[18] LI S T,YANG B,HU J W.Performance comparison of different multi resolution transforms for image fusion [J].Information fusion,2011,12(2):74-84.
[19] PETROVIC V,XYDEAS C.Objective image fusion performance measure [J].Electronics letters,2000,36(4):308-309.
[20] HAN Y,CAI Y Z,CAO Y,et al.A new image fusion performance metric based on visual information fidelity[J].Information fusion,2013,14(2):127-135.
[21] WANG Z,BOVIK A C.A universal image quality index[J].IEEE Signal processing letters,2002,9(3):81-84.
[22] MA J Y,MA Y,LI C.Infrared and visible image fusion methods and applications:a survey [J].information fusion,2019,45:153-178.
[23] PIELLA G,HEIJMANS H.A new quality metric for image fusion[C]∥ Proceedings 2003 International Conference on Image Processing,2003:173-176.
[24] ROBERTS W,AARDT J A W,AHMED F.Assessment of image fusion procedures using entropy,image quality,and multispectral classification[J].Applied remote sensing,2008,2(1):023522.

备注/Memo

备注/Memo:
收稿日期:2025-07-26
基金项目:山西省基础研究计划项目(202203021221104);中北大学研究生科技立项项目(20242029)
作者简介:兰海麟 (1999-),男,河北张家口人,硕士研究生,研究方向为红外与可见光信息处理;吉琳娜 (1988-),女,山西临汾人,博士,副教授,研究方向为多模态信息融合与识别、不确定性处理理论等。
更新日期/Last Update: 2026-03-09