[1]赖金水,万中英,曾雪强*.基于情感轮和多任务卷积神经网络的图像情感分布学习[J].江西师范大学学报(自然科学版),2022,(04):363-371.[doi:10.16357/j.cnki.issn1000-5862.2022.04.06]
　LAI Jinshui,WAN Zhongying,ZENG Xueqiang*.The Image Emotion Distribution Learning Based on Emotion Wheel and Multi-Task Convolutional Neural Network[J].Journal of Jiangxi Normal University:Natural Science Edition,2022,(04):363-371.[doi:10.16357/j.cnki.issn1000-5862.2022.04.06]
点击复制

基于情感轮和多任务卷积神经网络的图像情感分布学习()
分享到：

《江西师范大学学报》（自然科学版）[ISSN:1006-6977/CN:61-1281/TN]

卷:
期数:
2022年04期
页码:
363-371
栏目:
信息科学与技术
出版日期:
2022-07-25

文章信息/Info

Title:
The Image Emotion Distribution Learning Based on Emotion Wheel and Multi-Task Convolutional Neural Network
文章编号:
1000-5862(2022)04-0363-09
作者:
赖金水; 万中英; 曾雪强*
江西师范大学计算机信息工程学院,江西 南昌 330022
Author(s):
LAI Jinshui; WAN Zhongying; ZENG Xueqiang*
School of Computer & Information Engineering,Jiangxi Normal University,Nanchang Jiangxi 330022,China
关键词:
Mikel's情感轮;  多任务卷积神经网络;  情感分布学习;  情绪分类;  标记分布学习
Keywords:
Mikel's emotion wheel;  multi-task convolutional neural network;  emotion distribution learning;  emotion classification;  label distribution learning
分类号:
TP 391
DOI:
10.16357/j.cnki.issn1000-5862.2022.04.06
文献标志码:
A
摘要:
图像情感分布学习可以对多种情绪同时进行建模,但现有的模型缺乏有效的方法直接考虑情绪之间的相关性.针对这一问题,该文提出一种基于情感轮和多任务卷积神经网络(EW-MTCNN)的图像情感分布学习模型,通过先验知识模块将心理学情感知识直接引入到深度神经网络中.基于Mikel's情感轮定义成对情绪之间的相关性,EW-MTCNN模型采用多任务卷积神经网络学习情绪之间的相关性信息,同时优化情感分布预测和情绪分类任务.EW-MTCNN模型由3个模块组成,3个模块分别是图像特征提取层、情感轮先验知识层和多任务损失层.在情感分布数据集(Emotion6)和单标签数据集(Artphoto)上进行对比实验的结果表明:EW-MTCNN模型在情感分布预测与情绪分类任务上比其他情感分布学习方法具有更优的性能.
Abstract:
Image emotion distribution learning can model multiple emotions simultaneously,but existing models lack effective methods to directly consider the correlation between emotions.In response to this problem,the emotion wheel enhanced multi-task convolutional neural network for image emotion distribution learning(EW-MTCNN)model is proposed.Psychological emotional knowledge is directly introduced into the deep learning network.Based on Mikel's emotion wheel to define the correlation between paired emotions,the EW-MTCNN model adopts a multi-task convolutional neural network to learn the correlation information between emotions to jointly optimize the emotion distribution prediction and emotion classification tasks.The EW-MTCNN model consists of 3 modules,namely the emotional wheel prior knowledge layer,the visual feature extraction layer and the multi-task loss layer.Comparative experiments on emotion distribution dataset(Emotion6)and single-label dataset(Artphoto)show that the EW-MTCNN model has better performance than other emotion distribution learning methods on emotion distribution prediction and emotion classification tasks.

参考文献/References:

[1] 王仁武,孟现茹.图片情感分析研究综述[J].图书情报知识,2020(3):119-127.
[2] JIA Jia,WU Sen,WANG Xiaohui,et al.Can we understand van gogh's mood?learning to infer affects from images in social networks[EB/OL].[2021-12-12].https://doi.org/10.1145/2393347.2396330.
[3] BORTH D,JI Rongrong,CHEN Tao,et al.Large-scale visual sentiment ontology and detectors using adjective noun pairs[EB/OL].[2021-12-11].https://dl.acm.org/doi/pdf/10.1145/2502081.2502282.
[4] ZHAO Sicheng,GAO Yue,JIANG Xiaolei,et al.Exploring principles-of-art features for image emotion recognition[EB/OL].[2021-12-11].https://dl.acm.org/doi/epdf/10.1145/2647868.2654930.
[5] PLUTCHIK R.A general psycho evolutionary theory of emotion[M].Pittsburgh:Academic Press,1980:3-33.
[6] ZHANG Minling,ZHOU Zhihua.A review on multi-label learning algorithms[J].IEEE Transactions on Knowledge and Data Engineering,2013,26(8):1819-1837.
[7] ZHOU Ying,XUE Hui,GENG Xin.Emotion distribution recognition from facial expressions[EB/OL].[2021-12-10].https://dl.acm.org/doi/10.1145/2733373.2806328.
[8] JIA Xiuyi,ZHENG Xiang,LI Weiwei,et al.Facial emotion distribution learning by exploiting low-rank label correlations locally[EB/OL].[2021-12-04].https://ieeexplore.ieee.org/document/8953790.
[9] ZHOU Deyu,ZHANG Xuan,ZHOU Yin,et al.Emotion distribution learning from texts[EB/OL].[2021-12-12].https://www.researchgate.net/publication/311990212_Emotion_Distribution_Learning_from_Texts.
[10] XIONG Haitao,LIU Hongfu,ZHONG Bineng,et al.Structured and sparse annotations for image emotion distribution learning[J].Proceedings of the AAAI Conference on Artificial Intelligence,2019,33(1):363-370.
[11] MIKELS J A,FREDRICKSON B L,LARKIN G R,et al.Emotional category data on images from the International Affective Picture Sysem[J].Behavior Research Methods,2005,37(4):626-630.
[12] GONG Rui,WANG Qing,HAI Yan,et al.Investigation on factors to influence color emotion and color preference responses[J].Optik,2017,136:71-78.
[13] MATTHEWS T,NIXON M S,NIRANJAN M.Enriching texture analysis with semantic data[EB/OL].[2021-12-15].https://dl.acm.org/doi/10.1109/CVPR.2013.165.
[14] YAO Lei,SURYANARAYAN P,QIAO Mu,et al.Oscar:on-site composition and aesthetics feedback through exemplars for photographers[J].International Journal of Computer Vision,2012,96(3):353-383.
[15] COLOMBO C,DEL BIMBO A,PALA P.Semantics in visual information retrieval[J].Ieee Multimedia,1999,6(3):38-53.
[16] MACHAJDIK J,HANBURY A.Affective image classification using features inspired by psychology and art theory[EB/OL].[2021-12-19]. https://dl.acm.org/doi/10.1145/1873951.1873965.
[17] RAO Tianrong,XU Min,XU Dong.Learning multi-level deep representations for image emotion classification[J].Neural Processing Letters,2020,51(3):2043-2061.
[18] YANG Jufeng,SHE Dongyu,SUN Ming.Joint image emotion classification and distribution learning via deep convolutional neural network[EB/OL].[2021-12-19].https://dl.acm.org/doi/10.5555/3172077.3172345.
[19] GENG Xin.Label distribution learning[J]. IEEE Transactions on Knowledge and Data Engineering,2016,28(7):1734-1748.
[20] 曾雪强,罗明珠,陈素芬,等.基于自适应多重多元回归的人脸年龄估计[J].江西师范大学学报(自然科学版),2019,43(1):68-75.
[21] ZHANG Yuxiang,FU Jiamei,SHE Dongyu,et al.Text emotion distribution learning via multi-task convolutional neural network[EB/OL].[2021-12-12].https://dl.acm.org/doi/abs/10.5555/3304222.3304409.
[22] HE Tao,JIN Xiaoming.Image emotion distribution learning with graph convolutional networks[EB/OL].[2021-12-12].https://dl.acm.org/doi/pdf/10.1145/3323873.3326593.
[23] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].[2021-12-18].https://arxiv.org/pdf/1409.1556.pdf.
[24] GENG Xin,YIN Chao,ZHOU Zhihua.Facial age estimation by learning from label distributions[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(10):2401-2412.
[25] PENG Kuanchuan,CHEN Tsuhan,SADOVNIK A,et al.A mixed bag of emotions:model,predict,and transfer emotion distributions[EB/OL].[2021-12-12].https://ieeexplore.ieee.org/document/7298687/.
[26] MACHAJDIK J,HANBURY A.Affective image classification using features inspired by psychology and art theory[EB/OL].[2021-12-12].https://dl.acm.org/doi/10.1145/1873951.1873965.
[27] CAMPOS V,SALVADOR A,GIR-I-NIETO X,et al.Diving deep into sentiment: understanding fine-tuned CNNs for visual sentiment prediction[EB/OL].[2021-12-12].https://arxiv.org/pdf/1508.05056.pdf.
[28] YANG Jingyuan,LI Jie,LI Leida,et al.A circular-structured representation for visual emotion distribution learning[EB/OL].[2021-12-12].https://arxiv.org/abs/2106.12450.
[29] YANG Jufeng,SUN Ming,SUN Xiaoxiao.Learning visual sentiment distributions via augmented conditional probability neural network[EB/OL].[2021-12-12].https://dl.acm.org/doi/10.5555/3298239.3298273.
[30] GAO Binbin,XING Chao,XIE Chenwei,et al.Deep label distribution learning with label ambiguity[J].IEEE Transactions on Image Processing,2017,26(6):2825-2838.
[31] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenet classification with deep convolutional neural networks[J].Communications of the ACM,2017,60(6):84-90.
[32] HE Kaiming,ZHANG Xiangyu,REN Shaoqing,et al.Deep residual learning for image recognition[EB/OL].[2021-12-12].https://ieeexplore.ieee.org/iel7/7776647/7780329/07780459.pdf.

备注/Memo

备注/Memo:
收稿日期:2022-01-13
基金项目:国家自然科学基金(61866017)资助项目.
通信作者:曾雪强(1978—),男,江西南昌人,教授,博士,博士生导师,主要从事标记分布学习、自然语言处理和数据降维的研究.E-mail:xqzeng@jxnu.edu.cn
更新日期/Last Update: 2022-07-25