Comments
Description
Transcript
スポーツ映像からのコミック風コマ割りダイジェスト画像の 自動生成
ĈÓšãŶžÓÅ
IPSJ SIG Technical Report
Vol.2012-DCC-1 No.15
2012/5/18
$2&
A)6&
9*5.
ŋǖĎÖ† äĶ·ą††
ǘīǚ%
4*OgwVƮ¿Ī!'ǯ&Ʈ¿ƈǎ;ŵīǚ!Ĕğ7"!7S@NBOY&őţÔǣÛ
ǰWr`&OgwV[mwO$#!'ǯƮ¿&ǕƢ$Mwu;Ǡ1 ƍǠS@NBOYĪ4ţ587
ǯOgwVħƓ3 Web L@Y$#!'S@NBOY'Ą2Ť!'$ǰ!ĵŶž!'ǯOgwVĪ
5S@NBOYŤ;Ɨũ%Ţď7đŐ;ĝľ7ǰ S@NBOYŤ'ǯê&ŭČũ$Ĕğ;Ŭũ"
JiUHǫ&Jh©6;Ę7ǰĝľđŐ!'ǯǕƢ$Mwu&ĖǯMwu;Ơ7ǢŅŤŢďǯJh©
6Ţď$#&š%46ǯJiUHǫS@NBOYŤ;Ɨũ%Ţď7ǰĵƵĥ!'LUFw}ƉĪ;šë
ƺ" ǯĝľđŐ&ݬć;Ŀƭ7ǰ
Automatic Generation of an Image Digest with a Comic-like Layout
from a Sports Video
SHINJI MIZUNO† TOMOYUKI USUGI††
A digest of a long sports match video is useful to catch the progress of the match in a short time. Video digests which include
important scenes are often used in sports news programs. However, image digests are useful in sports papers and web pages. In
this paper, we propose a method to generate an image digest from a sports match video. An image digest generated by our
method has a comic-like layout, and it is easy to catch the progress of the match from the image digest intuitively. The process of
the proposed method is extracting important scenes, generating images which represent the scenes, and arranging the images in a
comic-like layout. We use videos from soccer TV programs in experiments and evaluate our method.
1. gwVĪ}&Mwu;Ĝå
Dž÷ǯġLjTlu\q&Õª3\UYtwHēƟ&ŧǏ
|,7"!S@NBOYŤ;Ţď7"!7ǰ
%
ǯŬ%7`XE$#&ĪĈÓǗ'Üõ
ĵŶž!'Ť&ǔƏĨŐ%JiUH!ţ587Jh
7ǰŜ%OgwV}Ɖ$#ǘīǚ%
4*Ī
©6%ŲŬ7ǰJiUH!'ǯ@oOY&Ü3ǔƏ
!'ǯ&ê;ŵīǚ!Ĕğ"ƢIJí$
ĨŐ;Ǒ¡%Ù¯7"!ǯê3ƈǎ;ŭČũ%Ĕ
$ǯOgwV[mwO$#!ƣ5874%ǕƢ$M
ğ!74%ðÝ
wu;Ǡ1S@NBOY;őţ7ÔǣÛ$
ƅ3@duY>q^j&ƗŢď%JiUHǫJh©6;
%ÕÜ
ǯMwu}3Mwu&¡
6Į:6Ǔ &Ť;Ţď7ǰ85&Ť;īƄ£%
7ǰ&1ǯ`XEĪ&Ƣ
7. &4$ƕĭ&}ǯOgwV}ƉƁ&S@NBOY
Ǒţ7ŶžÓÅ8
Ī;Ɨũ%Ţď7Ŷž2ÓÅ8
Ī5Ţď8Mwu;Ơ7Ť;JiUHǫJ
7
7[3][4]ǰĵŶž!'OgwV
[1][2]ǰ(ǯƹŎ5'ǖŠ&}ƉĪ5Ī&¡8Ŭ
h©6!|,S@NBOYŤ;Ţď7ǰ
"$7FUYř3ĿǯĕŠ3WsUc$#;đ
ĵƵĥ!'Wr`&LUFw}ƉĪ;šëƺ"
6%
ĝľđŐ&ݬć;Ŀƭ7ǰ
ľ
S@NBOYĪ;Ɨũ%Ţď7đŐ;ĝ
7[1]ǰ85&đŐ!'ǯ~%Ť%47S@N
BOY;Ţď
7ǰǯOgwVĪ&S@NB
2. ǯ
A)
OY!'ŵīǚ!Ʈ¿&ƈǎ;Ĕğķ7"Ō1
Jh©6"'JiUH$#%ţ587ƠşēŐ!7ǰ
58
xx&ļ;Jh"Ç)ǯ&ǔƏĨŐ%4
71ǯĄ2ŤƠşįǑ!7"'ǜ5
Ôǣ&
$ǰħƓ3Űnj¢ƯǯWeb ewN&4%ǯìĤĺ&
ǁĞ3īǚ&Œ8ǯüƴ$#;Ʋƒ%ĉƾ7"!
ǢŅŤ!Łď8S@NBOY%4
7.
Ʈ¿ê;Ơ
JiUH&Jh©6'Ì 1 %Ÿ4% 1 ewN'
ş7Ô¿2Ûǰ
58 Jh¨Ā!Łď87ǰ.ʼnĤ' 3 ʼn 4 ʼn!7
!ǯĵŶž!'ǢŅŤ;ţOgwVĪ&S
"ÛǰƲƒ&Ƥƌ'¼5ñ+Œ8
@NBOY&ƗŢďđŐ;Ŀƫ7ǰĝľđŐ!'ǯO
!&Òĵ!7ǰJh"Jh&ǚǞ'Ń&ǚǞ46
Ǝ&ǚǞ;õø¸7ĨƲƒ%"
†
ċŴðŀÜãĈÓŹãǓ Faculty of Information Science, Aichi Institute of Technology
††
XmcsƻØ(Ľ) Duplo Sales Co., Ltd.
2012 Information Processing Society of Japan
z&ʼn+Ǎ<
Ʋ/3"8
7ǰǯ85&Jh©6qwq'xƙũ$2&
!ǯÚ2ÛáÍ7ǰ
1
ĈÓšãŶžÓÅ
IPSJ SIG Technical Report
Vol.2012-DCC-1 No.15
2012/5/18
JiUH&Jh©6%'ł$ēŐáÍ7ǰ(
/7ǰ
Ì 1 & 1 ʼnŬ&4%ǯJh"Jh&ŃǚǞ;ܸ7
Wr`&LUFw}Ɖ!'ǯpcr@Ī;Ě7ǟ
"!īǚ&ƈǎ3ÔǣǁĞ;Ơş7Ô¿7ǰlj%
%ǯ&¨Ā%X?Qqb$#&ŜňάĻ;ţ
ǚǞ;ì7"!ǯơĤ&ƺÀī%ƾ
cr@Mwu!7";ĩŷ%Ÿ"ÛǭÌ 2Ǯǰ
p
7ŝč;Ơş7Ô¿7ǰ2 ʼnŬ!'Ħƌ!Jh©6
!ǯĵŶž!'LUFw}ƉĪ%Â.87X?Qq
;ƞ
b;Ŀ7"!pcr@Mwu;Ė7ǰ
7ǰ8'OgwZČ;Õ¬Ļ7ǰ3 ʼn
Ŭ!'Ôǣ;üƴ71%Ü$Jh;ţ
7ǰ
X?Qqb&°ǚ&¨Ā!'ǯĪ&ǂù²ƴ
8'īǚ;ǘƣ71%2ţ587ǰ
%ÜÙ¯7"ÛǯǘÎ5'&ćƽ;§ţ
&4%ǯJiUH!'Ʋƒ%Ôǣ; 63³
X?QqbĿđŐ;ĝľ
ƺũ%71%ǯJh©6ĨŐðÝ8
đŐ;Ćţ
7ǰ
X?Qqb&Ŀ;ƞǰĵŶž&ũ
'ǯŤ&şÍ&arwj5ǎµ 8 arw
$š"
j%
7[5]ǰĵŶž!'&
ǯ¾arwj&öÏǂù;ţ
arwjǚ
D i ²ƴÙ¯7"
&ǂùò D i ;Ō17ǰ
"2%ǯDi & ģz&Ô¿%X?Qqb!
7"¥å7ǰ
Ì 2 X?Qqb&ǰ
Ì 1 JiUH&Jh©6&ǰ
Figure 2 An example of the dissolve effect.
Figure 1 An example of a layout in a comic book.
pcr@'KwqMwuÚ!2Ě871ǯĖ
3. ' 4 / @
pcr@ĪKwqMwu!'$ºƖć27ǰ
Wr`&LUFwƮ¿&}ƉĪ5JiUHǫ&Jh
"9ǯKwqMwu!'ł$>uIq5&pcr
©6;ĘƮ¿&S@NBOYŤ;Ţď7%'ǯz
@;ƞ1ǯ&Mwu%Ŋ,
&đǦĄƢ!7"Ƒ7ǰ
ǘ$7":ǭƠ 1Ǯǰ!ǯX?Qqb%Ò
Ɲƒ5ơĤ&LUFw}ƉĪ&pcr@Mwu;ƴ,
pcr@īǚÜõ%
v Ī}5&ǕƢ$Mwu&Ėǰ
v ǕƢ$Mwu;Ơ7Ť&Ţďǰ
;ƿ7Ô¿%KwqMwu!7"¥å
v Ţď8Ť&JiUHǫJh©6ǔƏǰ
Ā&š%ţ7""7.
z%88&š%
Ėpcr@Ī%ë
dž,7ǰ
ǯ&pcr@īǚ
ǯ
Ơ 1 pcr@īǚǰ
3.1 ! " , ? ; $ # 0 (
LUFw&Ʈ¿' 2 īǚŻù!6ǯ&}5Ǒ¡%
Table 1 Lengths of replay videos.
pcr@Ĥ
ǕƢ$Mwu;Ė7ĄƢ7ǰĵƵĥ!'ǕƢ$M
KwqMwu
wu"
&&Mwu
Ʈ¿}&KwqMwu;Ė7""7ǰ
öÏīǚǭźǮ
6
30.33
22
9.14
Wr`&LUFw&}ƉĪ!'ǯTluO3Kwqǯ
a=Aq$#&Ʈ¿}&ŜĂũ$Mwu!pcr@Ī;
3.2 $ # % : 6 & 5 .
Œ"ÛǰŜ%KwqMwu!'Ą"Ʃ
ĖKwqMwu5S@NBOYŤ&¾Jh%
-#pcr@Ī;Œǰ!ǯĵŶž!'ǕƢMwu
ţ7Ť&Ţď;ƞǰ&1ǯį¤%KwqMwu
!7KwqMwu;pcr@Ī5Ė7";Ʈ
%
7Fko>uIq; 4 żǪ% Ǫ7ǰ4 żǪ&F
2012 Information Processing Society of Japan
2
ĈÓšãŶžÓÅ
IPSJ SIG Technical Report
Vol.2012-DCC-1 No.15
2012/5/18
ko>uIq'ǯ(a)ñ¼KwqCp>ǯ(b)Kwq¨ǯ(c)
ǐĭǯ(d)&ǯ!7ǰ
¥å7ǰ
3. Su ĴŖ!ǯS d y!6ǯ5%
L y!7Ô¿ǯŤ}zǓ&/%ƛǧÑ
áÍ
ǯŭƌ± ÛĿ8ŝč!71ǯ
Kwq¨>uIq"¥å7ǰ!$Ô¿'
&>uIq"¥å7ǰ
Ì 5ǯÌ 6 %š&łß;ŸǰŤ}&ŨǧÑĖ
(a)ñ¼KwqCp>ǰ (b)Kwq¨ǰ
8ƛǧÑ!7ǰ
ñKwqCp>>uIq!'Ť
!'$6Ü$ǣŽ!ƛǧÑĪ
7ǰ
ǯŤ}&ñ¼ǧÑ&ƛǧÑ;Ŋ,7"ǯ¼ǧÑ%Ŋ,
ñǧÑ&ƛǧÑ$6ì$
7"ŷư!
7ǭÌ 5Ǯǰ.Kwq¨!'ǯŤ}&zǧÑ%ƛ
ǧÑŷư!7"%ǯKwqgOY3Ʀç£$#%4
(c)ǐĭǰ (d)&ǰ
Ť%Ŕ
ŭƌĿ8
7":7
ǭÌ 6Ǯǰ
Ì 3 4 żǪ&Fko>uIqǰ
Figure 3 Four types of camera angles.
Fko>uIq& Ǫ'ǯŤ}&ƛǧÑ& óŝčǯ
4)Ť}&ŭƌĿ$#ǯŤ&ŜĂǗ%Ò
ƞǰ
į¤%ǯHSV ýú%ÙоarwjŤ&ƚŮď ;
ţ
ƛǧÑ&Ė;ƞǰ
Ė8ƛǧÑǣ
Ž SǯŤ}&ñ¼ǧÑ%Â.87ƛǧÑǣŽ Sl, Sr ǯŤ
}&yzǧÑ%Â.87ƛǧÑǣŽ Su, Sd ;ƪƂ7ǭÌ
4Ǯǰ.ǯKwqņǣ&>uIq!'ǯKwqgOYǯK
wq\UYǯůĸǯƦç£$#%4
.871ǯHough ÙĞ%4
ŭƌď ÛÂ
Ť}&ŭƌ;Ŀ
ǯ
Ì 5 ñKwqCp>&ƛǧÑǰ
Figure 5 The grass region around the left goal area.
&ĵĤ L ;ƪƂ7ǰ
Su
Sl
Sr
S
Sd
(a)Kwq¨&ƛǧÑ
Ì 4 ƛǧÑ&ǣŽƪƂǰ
Figure 4 The area of grass regions.
z&đǦ%4
4 żǪ&Fko>uIq;¥
å7ǰ
1. S &Ü!ƛ&ƣĨ;¥å7ǰS y
&Ô¿'đǦ 2 %Ǎ0ǰ!$Ô¿'đǦ 3 %Ǎ0ǰ
2. Sl " S r "&ŊŞ;ƪƂ
ǯñ¼&ò46Ü
(b)Hough ÙĞ!Ŀ8ŭƌǰ
Ô¿%'ǯñ¼8&KwqCp>>uI
Ì 6 Kwq¨ǰ
q"¥å7ǰ!$Ô¿%'ǐĭ>uIq"
Figure 6 In front of the goal.
2012 Information Processing Society of Japan
3
ĈÓšãŶžÓÅ
IPSJ SIG Technical Report
Vol.2012-DCC-1 No.15
2012/5/18
KwqMwuĪ&}!ǯFko>uIq¡6Į:
!ǯfwqKwq%ǀ6DŽ.8Ôǣ;üƴ7
"3À>uIqǘƊÔ¿%ǯ`XEĪ;J
1ǰKwq¨Ī"ñ¼KwqCp>Ť'ºƖ$Ô¿
ǭÌ 7Ǯǰ8%4
%'ÜJh©6!ǔƏ7ǰǯJh;ǔƏ7
6ǯ¡68Ť'ł$Fko>uIq5ű1
Əñ&Ô¿%'ÜJh©6;$1ǯK
KwqMwuÂ.8
wq¨!
h©6ţǢŅŤ"
¡6
6ǯÀżǪ&>uIqƊ
2ì$Jh©6!ǔƏ7ǰ.ǯNjƊ
Ô¿%'åijũ%Ť;¡6"!Kwq%Ƙ7ƈǎ
Ʀŕ!7"ijÿ87ǰ
$Jh©6!ǔƏ8
Kwq
Dž;ǔƏ7Ô¿%'ǯ!%¨&ŤÜ
7"71ǯì$J
h©6!ǔƏ7"%7ǰ
4. - B
ĝľđŐ&ݬć;ŷư71ǯ&Wr
`&LUFw}ƉĪ%ë
đŐ;Ǒţ
ǯJiUH
ǫS@NBOYŤ;Ţď7æǬ;ƞǰPaYAB
>&Ǚŧ'ǯMacOS X10.6 y% C++" OpenCV o@bop
;ţ
Ì 7 Jh©6ţǢŅŤ&¡6ǰ
Figure 7 Picking up images for a comic-like image digest.
ƞǰ$
ǯX?Qqb¬Ļ'LUFwpwI
3ġĪWr`î%4
żǪŦ$6ǯƛǧÑ&ƚ%
2ORN>j%4
Ŧ$71ǯpcr@Ťǣ&Ė
3Fko>uIq&¥å;ƞ1&_okwR3
3.3 '88!įǑ%$74%đ!Ƭåǰ
A)>7
Ì 9 %ƗŢď8JiUHǫJh©6;ĘS@
ǰJh©6&Łď'ìJh"ÜJh&żǪ"
NBOYŤ;Ÿǰfwq;Gwc7ǒđ&ǒđ
¡6Ť;JiUH&Jh©6&4%|,
7ǰ
Ì 8 &4%ǯ¼y5ñz%Á
Ǧ%Ť¾ʼn! 12 ĺǯīƄ£%ǔƏ
¡6
%Ë.8$52Kwq;ō17łß;ƠJiUHǫJ
ǰ
h©6S@NBOYŤŢď8
7ǰ
Ì 8 ƛǧÑ&ǣŽƪƂǰ
Figure 8 The area of grass regions.
ƀ 2 ſ!dž,4%ǯJiUH!'Jh©6ĨŐ%4
ł$Ơş;ƞǯĵŶž!'.üƴÔ
ǣ&Jh©6;Ü7ēŐ;æşǰpcr@!Œ
Ì 9 Ţď8JiUHǫJh©6S@NBOYŤ 1ǰ
87KwqMwu5¡68Ť'ǯ¨ƃ!dž,
Figure 9 A comic-like layout image digest #1.
đŐ%46 4 &Fko>uIq% Ǫ8
7ǰ&
ǯæǟ%fwqKwq%ǀ6DŽ.87Ôǣ'Kw
¦&LUFw}ƉĪ5ƗŢď8S@NBOY
q¨Ťñ¼KwqCp>Ť%Â.87"Ûǰ
Ť;Ì 10 4)Ì 11 %ŸǰÌ 10 !'3'6fwq
2012 Information Processing Society of Japan
4
ĈÓšãŶžÓÅ
IPSJ SIG Technical Report
Vol.2012-DCC-1 No.15
2012/5/18
;ĘǒđKwqņǣ
Dž!&ǒđ%Ë.87ǯ
&Ā%Kwq;ō1"S@NBOYŤ5:7ǰ
ǯfwqKwq\UY;Ġ54$Ðũ$ų
$ǰxĨǯÌ 11 &S@NBOYŤ%
ǚ'Â.8
'KwqMwuÂ.8
ǒđX?Qqb"
$MwuƱ
$ǰ8'ǯÜ8
ƱĿ8
.ǯƮ¿}&ö
KwqMwu"
Ė8
.
"´Ê!7ǰ.Fko&>uIqzƏ5
&Ô¿%'ǯKwqCp>
Dž!'$
Æǃ"¥å8
2KwqCp>
.Ô¿7ǰ
5. ĵŶž!'ǯWr`&LUFw}ƉĪ5JiUHǫ
&Jh©6;ĘƮ¿S@NBOYŤ;Ɨũ%Ţď
7đŐ;ĝľǰĝľđŐ'ǯpcr@Ī}%Ʈ¿&
į2ŜĂũ$Mwu!7KwqMwuÂ.87'!
7"¨ĝ%Òǯ}ƉĪ5pcr@Ī
;Ė
ǯ5Fko>uIq&¡6ĮR@iu
I%Ò
Ť;Ė
&Jh©6%4
ǯ85&Ť;JiUHǫ
ǔƏǰ&"ǯpcr@Ī&
Ė%'X?Qqb&Ŀ;§ţǰ.Fko>uI
q'ǯĪ&ƚĈÓ3ŭƌĿ$#&ŤŜĂǗ;ţ
4 żǪ&Fko>uIq;¥åǰ
ǯ
¡6Ť
'ǯJiUH!ţ587Jh©6WH[UH;¶Ƒ%
$5ǯFko>uIq%4
Ì 10 Ţď8JiUHǫJh©6S@NBOYŤ 2ǰ
Figure 10 A comic-like layout image digest #2.
Jh&Ü;Ù¯
$5ǔƏ7"!ǯįƇũ$Jh©6S@NBOY
Ť;Ţďǰ
Ţď8S@NBOYŤ'ǯKwqMwu&Ī
īƄ£!Jh©6ƠŸ8
6ǯƮ¿&ƣĐMwu;ŵ
īǚ%Ƥƥũ%Ĕğ7"!ǰKwq¨!'Jh
ėÜ8
ǯfwqKwq\UY;Ġ5ŝč;ŷư
7"2!ǰǯWsUcĈÓ$#'Ŝ%ĊƸ
š;
7:!'$1ǯāř&Ù¯3Kw
q;ō1ǒđ$#&ĈÓ'ǯĄ2ě7"!
$ǰ
Ā&ƳǨ"
'ǯMwu&Ĕğ%ŤŜĂǗÚ&
Ŵũ$ĈÓ;ţ7"!ǯ46Ǒ¡%Ʈ¿&ǕƢ$Mw
u;Ė6Ť;¡667"ę587ǰ
(ǯWsUcĈÓǯǒđ&ǩ3ƕť½ǯn[aDwj
ƚ$#;§ţ&Ŝåǯæŏ3ń×$#&Ǥ×ĈÓ
&§ţ$#;Ŀƫ7ǰĵƵĥ!'Jh&Ü;Ù¯
7&²Ɔ$Jh©6;æşǯŴũĈÓ;
§ţ7"!ǯīǚƈǎ3>uIq&Ŧ$7ÀīMwu
;|£%¸6Ē4$Jh©6;æş6ǯS@NB
OYŤ%JiUH&Ã&4$ĥàĈÓ;LJª
67"2Ŀƫ7ǰ.ǯÉ'LUFw;ë
ƺ"ǯWr`!}Ɖ87&OgwV%
đŐ&Ǒţ;Ǎ1
26!7ǰ
2012 Information Processing Society of Japan
2
Ì 11 Ţď8JiUHǫJh©6
S@NBOYŤ 3ǭÞĢǮǰ
Figure 11 A comic-like layout image digest #3.
5
ĈÓšãŶžÓÅ
IPSJ SIG Technical Report
< = ĵŶž;Ǎ17%6ǯŚă%ƫƵ
Vol.2012-DCC-1 No.15
2012/5/18
ǥ
ċŴðŀÜãĈÓŹãǓŘǖûĩƶô"ǯŋǖĎÖŶžè
4)ŘǖŶžè&Ūł%œČƷ7ǰĵŶž&xǓ'
ŹŶƼÒū(C)ǭ23500139Ǯǯ
4)ċŴðŀÜãŜ¦Ŷž
«ď%47ǰ
+813
1) ƹŎǝ», {ƧƔ, Áþ, ìĶ: ĕŠMwu&ƨĹ"W
sUcĈÓ;ţǖŠ}ƉS@NBOYś&ƗŢďŐ, Ī
ĈÓkX?>ãēƟÓÅ, 28(71), pp. 27-30 (2004).
2) éïƜÄ, ÜǛÈÝ: LUFwŤ%
7Ťš%47
]@o@YŢď, ĪĈÓkX?>ãēƟÓÅ, 35(8), pp. 17-19
(2011).
3) Myodo, E., Ueno, S., Takagi, K., Sakazawa, S.: Automatic
Comic-like Image Layout System Preserving Image Order and
Important Region, Proc. of the 19th ACM International Conference on
Multimedia, pp. 795-796 (2011).
4) Calic, J., Gibson, D.P., Campbell, N.W.: "Efficient Layout of
Comic-like Video Summaries", IEEE Transactions on Circuits and
Systems for Video Technology, Vol. 17, No. 7, pp. 931-936 (2007).
5) ǘÎĬı, éŇâĥ, ƹ¹®Ɛ: ǂùÙ¯&²ƴć%ŲŬ
XQqbĿđŐ, ǡßĈÓNJãƋ¿ÜƶŗƵĥǠ,1996
÷, ĈÓvMOWj(2), p. 403 (1996).
2012 Information Processing Society of Japan
6