Action recognition and localization in untrimmed videos is important for many applications and have attracted a lot of at-tention.Since full supervision with frame-level annotation places an overwhelming burden on manual labeling effort,learning with weak video-level supervision becomes a potential solution.In this paper,we propose a novel weakly supervised framework to recognize actions and locate the corresponding frames in untrimmed videos simultaneously.Considering that there are abundant trimmed videos publicly available and well-segmented with semantic descriptions,the instructive knowledge learned on trimmed videos can be fully leveraged to analyze untrimmed videos.We present an effective knowledge transfer strategy based on inter-class semantic relevance.We also take advantage of the self-attention mechanism to obtain a compact video representation,such that the influence of background frames can be effectively eliminated.A learning architecture is designed with twin networks for trimmed and untrimmed videos,to facilitate trans-ferable self-attentive representation learning.Extensive experiments are conducted on three untrimmed benchmark datasets(i.e.,THUMOS14,ActivityNetl.3,and MEXaction2),and the experimental results clearly corroborate the efficacy of our method.It is espe-cially encouraging to see that the proposed weakly supervised method even achieves comparable results to some fully supervised meth-ods.

Xiao-Yu Zhang;Hai-Chao Shi;Chang-Sheng Li;Li-Xin Duan

Institute of Information Engineering,Chinese Academy of Sciences,Beijing 100093,China;School of Computer Science,Beijing Institute of Technology,Beijing 100081,China;School of Computer Science and Engineering,University of Electronic Science and Technology of China,Chengdu 611731,China

机器智能研究（英文）

[1]Xiao-Yu Zhang;Hai-Chao Shi;Chang-Sheng Li;Li-Xin Duan-.TwinNet:Twin Structured Knowledge Transfer Network for Weakly Supervised Action Localization)[J].机器智能研究（英文）,2022(03):227-246

TwinNet,untrimmed,trimmed,ferable,ActivityNetl,MEXaction2

Structured,Knowledge,Transfer,Network,Weakly,Supervised,Action,Localization,recognition,localization,videos,important,many,applications,have,attracted,lot,Since,supervision,level,annotation,places,overwhelming,burden,manual,labeling,effort,learning,becomes,potential,solution,In,this,paper,novel,weakly,supervised,framework,recognize,actions,locate,corresponding,frames,simultaneously,Considering,that,there,are,abundant,publicly,available,well,segmented,semantic,descriptions,instructive,knowledge,learned,can,fully,leveraged,analyze,transfer,strategy,inter,class,relevance,also,take,advantage,self,attention,mechanism,obtain,compact,representation,such,influence,background,effectively,eliminated,architecture,designed,twin,networks,facilitate,attentive,Extensive,experiments,conducted,three,benchmark,datasets,THUMOS14,experimental,results,clearly,corroborate,efficacy,method,It,espe,cially,encouraging,see,proposed,even,achieves,comparable,some,ods

0.561715