Comments
Description
Transcript
スポーツ映像からのコミック風コマ割りダイジェスト画像の 自動生成
ĈÓšãŶžÓÅ IPSJ SIG Technical Report Vol.2012-DCC-1 No.15 2012/5/18 $2& A)6& 9*5. ŋǖĎÖ† äĶ·ą†† ǘīǚ% 4*OgwVƮ¿Ī!'ǯ&Ʈ¿ƈǎ;ŵīǚ!Ĕğ7"!7S@NBOY&őţÔǣÛ ǰWr`&OgwV[mwO$#!'ǯƮ¿&ǕƢ$Mwu;Ǡ1 ƍǠS@NBOYĪ4ţ587 ǯOgwVħƓ3 Web L@Y$#!'S@NBOY'Ą2Ť!'$ǰ!ĵŶž!'ǯOgwVĪ 5S@NBOYŤ;Ɨũ%Ţď7đŐ;ĝľ7ǰ S@NBOYŤ'ǯê&ŭČũ$Ĕğ;Ŭũ" JiUHǫ&Jh©6;Ę7ǰĝľđŐ!'ǯǕƢ$Mwu&ĖǯMwu;Ơ7ǢŅŤŢďǯJh© 6Ţď$#&š%46ǯJiUHǫS@NBOYŤ;Ɨũ%Ţď7ǰĵƵĥ!'LUFw}ƉĪ;šë ƺ" ǯĝľđŐ&ݬć;Ŀƭ7ǰ Automatic Generation of an Image Digest with a Comic-like Layout from a Sports Video SHINJI MIZUNO† TOMOYUKI USUGI†† A digest of a long sports match video is useful to catch the progress of the match in a short time. Video digests which include important scenes are often used in sports news programs. However, image digests are useful in sports papers and web pages. In this paper, we propose a method to generate an image digest from a sports match video. An image digest generated by our method has a comic-like layout, and it is easy to catch the progress of the match from the image digest intuitively. The process of the proposed method is extracting important scenes, generating images which represent the scenes, and arranging the images in a comic-like layout. We use videos from soccer TV programs in experiments and evaluate our method. 1. gwVĪ}&Mwu;Ĝå Dž÷ǯġLjTlu\q&Õª3\UYtwHēƟ&ŧǏ |,7"!S@NBOYŤ;Ţď7"!7ǰ % ǯŬ%7`XE$#&ĪĈÓǗ'Üõ ĵŶž!'Ť&ǔƏĨŐ%JiUH!ţ587Jh 7ǰŜ%OgwV}Ɖ$#ǘīǚ% 4*Ī ©6%ŲŬ7ǰJiUH!'ǯ@oOY&Ü3ǔƏ !'ǯ&ê;ŵīǚ!Ĕğ"ƢIJí$ ĨŐ;Ǒ¡%Ù¯7"!ǯê3ƈǎ;ŭČũ%Ĕ $ǯOgwV[mwO$#!ƣ5874%ǕƢ$M ğ!74%ðÝ wu;Ǡ1S@NBOY;őţ7ÔǣÛ$ ƅ3@duY>q^j&ƗŢď%JiUHǫJh©6; %ÕÜ ǯMwu}3Mwu&¡ 6Į:6Ǔ &Ť;Ţď7ǰ85&Ť;īƄ£% 7ǰ&1ǯ`XEĪ&Ƣ 7. &4$ƕĭ&}ǯOgwV}ƉƁ&S@NBOY Ǒţ7ŶžÓÅ8 Ī;Ɨũ%Ţď7Ŷž2ÓÅ8 Ī5Ţď8Mwu;Ơ7Ť;JiUHǫJ 7 7[3][4]ǰĵŶž!'OgwV [1][2]ǰ(ǯƹŎ5'ǖŠ&}ƉĪ5Ī&¡8Ŭ h©6!|,S@NBOYŤ;Ţď7ǰ "$7FUYř3ĿǯĕŠ3WsUc$#;đ ĵƵĥ!'Wr`&LUFw}ƉĪ;šëƺ" 6% ĝľđŐ&ݬć;Ŀƭ7ǰ ľ S@NBOYĪ;Ɨũ%Ţď7đŐ;ĝ 7[1]ǰ85&đŐ!'ǯ~%Ť%47S@N BOY;Ţď 7ǰǯOgwVĪ&S@NB 2. ǯ A) OY!'ŵīǚ!Ʈ¿&ƈǎ;Ĕğķ7"Ō1 Jh©6"'JiUH$#%ţ587ƠşēŐ!7ǰ 58 xx&ļ;Jh"Ç)ǯ&ǔƏĨŐ%4 71ǯĄ2ŤƠşįǑ!7"'ǜ5 Ôǣ& $ǰħƓ3Űnj¢ƯǯWeb ewN&4%ǯìĤĺ& ǁĞ3īǚ&Œ8ǯüƴ$#;Ʋƒ%ĉƾ7"! ǢŅŤ!Łď8S@NBOY%4 7. Ʈ¿ê;Ơ JiUH&Jh©6'Ì 1 %Ÿ4% 1 ewN' ş7Ô¿2Ûǰ 58 Jh¨Ā!Łď87ǰ.ʼnĤ' 3 ʼn 4 ʼn!7 !ǯĵŶž!'ǢŅŤ;ţOgwVĪ&S "ÛǰƲƒ&Ƥƌ'¼5ñ+Œ8 @NBOY&ƗŢďđŐ;Ŀƫ7ǰĝľđŐ!'ǯO !&Òĵ!7ǰJh"Jh&ǚǞ'Ń&ǚǞ46 Ǝ&ǚǞ;õø¸7ĨƲƒ%" † ċŴðŀÜãĈÓŹãǓ Faculty of Information Science, Aichi Institute of Technology †† XmcsƻØ(Ľ) Duplo Sales Co., Ltd. 2012 Information Processing Society of Japan z&ʼn+Ǎ< Ʋ/3"8 7ǰǯ85&Jh©6qwq'xƙũ$2& !ǯÚ2ÛáÍ7ǰ 1 ĈÓšãŶžÓÅ IPSJ SIG Technical Report Vol.2012-DCC-1 No.15 2012/5/18 JiUH&Jh©6%'ł$ēŐáÍ7ǰ( /7ǰ Ì 1 & 1 ʼnŬ&4%ǯJh"Jh&ŃǚǞ;ܸ7 Wr`&LUFw}Ɖ!'ǯpcr@Ī;Ě7ǟ "!īǚ&ƈǎ3ÔǣǁĞ;Ơş7Ô¿7ǰlj% %ǯ&¨Ā%X?Qqb$#&ŜňĪ¬Ļ;ţ ǚǞ;ì7"!ǯơĤ&ƺÀī%ƾ cr@Mwu!7";ĩŷ%Ÿ"ÛǭÌ 2Ǯǰ p 7ŝč;Ơş7Ô¿7ǰ2 ʼnŬ!'Ħƌ!Jh©6 !ǯĵŶž!'LUFw}ƉĪ%Â.87X?Qq ;ƞ b;Ŀ7"!pcr@Mwu;Ė7ǰ 7ǰ8'OgwZČ;Õ¬Ļ7ǰ3 ʼn Ŭ!'Ôǣ;üƴ71%Ü$Jh;ţ 7ǰ X?Qqb&°ǚ&¨Ā!'ǯĪ&ǂù²ƴ 8'īǚ;ǘƣ71%2ţ587ǰ %ÜÙ¯7"ÛǯǘÎ5'&ćƽ;§ţ &4%ǯJiUH!'Ʋƒ%Ôǣ; 63³ X?QqbĿđŐ;ĝľ ƺũ%71%ǯJh©6ĨŐðÝ8 đŐ;Ćţ 7ǰ X?Qqb&Ŀ;ƞǰĵŶž&ũ 'ǯŤ&şÍ&arwj5ǎµ 8 arw $š" j% 7[5]ǰĵŶž!'& ǯ¾arwj&öÏǂù;ţ arwjǚ D i ²ƴÙ¯7" &ǂùò D i ;Ō17ǰ "2%ǯDi & ģz&Ô¿%X?Qqb! 7"¥å7ǰ Ì 2 X?Qqb&ǰ Ì 1 JiUH&Jh©6&ǰ Figure 2 An example of the dissolve effect. Figure 1 An example of a layout in a comic book. pcr@'KwqMwuÚ!2Ě871ǯĖ 3. ' 4 / @ pcr@ĪKwqMwu!'$ºƖć27ǰ Wr`&LUFwƮ¿&}ƉĪ5JiUHǫ&Jh "9ǯKwqMwu!'ł$>uIq5&pcr ©6;ĘƮ¿&S@NBOYŤ;Ţď7%'ǯz @;ƞ1ǯ&Mwu%Ŋ, &đǦĄƢ!7"Ƒ7ǰ ǘ$7":ǭƠ 1Ǯǰ!ǯX?Qqb%Ò Ɲƒ5ơĤ&LUFw}ƉĪ&pcr@Mwu;ƴ, pcr@īǚÜõ% v Ī}5&ǕƢ$Mwu&Ėǰ v ǕƢ$Mwu;Ơ7Ť&Ţďǰ ;ƿ7Ô¿%KwqMwu!7"¥å v Ţď8Ť&JiUHǫJh©6ǔƏǰ Ā&š%ţ7""7. z%88&š% Ėpcr@Ī%ë dž,7ǰ ǯ&pcr@īǚ ǯ Ơ 1 pcr@īǚǰ 3.1 ! " , ? ; $ # 0 ( LUFw&Ʈ¿' 2 īǚŻù!6ǯ&}5Ǒ¡% Table 1 Lengths of replay videos. pcr@Ĥ ǕƢ$Mwu;Ė7ĄƢ7ǰĵƵĥ!'ǕƢ$M KwqMwu wu" &&Mwu Ʈ¿}&KwqMwu;Ė7""7ǰ öÏīǚǭźǮ 6 30.33 22 9.14 Wr`&LUFw&}ƉĪ!'ǯTluO3Kwqǯ a=Aq$#&Ʈ¿}&ŜĂũ$Mwu!pcr@Ī; 3.2 $ # % : 6 & 5 . Œ"ÛǰŜ%KwqMwu!'Ą"Ʃ ĖKwqMwu5S@NBOYŤ&¾Jh% -#pcr@Ī;Œǰ!ǯĵŶž!'ǕƢMwu ţ7Ť&Ţď;ƞǰ&1ǯį¤%KwqMwu !7KwqMwu;pcr@Ī5Ė7";Ʈ % 7Fko>uIq; 4 żǪ% Ǫ7ǰ4 żǪ&F 2012 Information Processing Society of Japan 2 ĈÓšãŶžÓÅ IPSJ SIG Technical Report Vol.2012-DCC-1 No.15 2012/5/18 ko>uIq'ǯ(a)ñ¼KwqCp>ǯ(b)Kwq¨ǯ(c) ǐĭǯ(d)&ǯ!7ǰ ¥å7ǰ 3. Su ĴŖ!ǯS d y!6ǯ5% L y!7Ô¿ǯŤ}zǓ&/%ƛǧÑ áÍ ǯŭƌ± ÛĿ8ŝč!71ǯ Kwq¨>uIq"¥å7ǰ!$Ô¿' &>uIq"¥å7ǰ Ì 5ǯÌ 6 %š&łß;ŸǰŤ}&ŨǧÑĖ (a)ñ¼KwqCp>ǰ (b)Kwq¨ǰ 8ƛǧÑ!7ǰ ñKwqCp>>uIq!'Ť !'$6Ü$ǣŽ!ƛǧÑĪ 7ǰ ǯŤ}&ñ¼ǧÑ&ƛǧÑ;Ŋ,7"ǯ¼ǧÑ%Ŋ, ñǧÑ&ƛǧÑ$6ì$ 7"ŷư! 7ǭÌ 5Ǯǰ.Kwq¨!'ǯŤ}&zǧÑ%ƛ ǧÑŷư!7"%ǯKwqgOY3Ʀç£$#%4 (c)ǐĭǰ (d)&ǰ Ť%Ŕ ŭƌĿ8 7":7 ǭÌ 6Ǯǰ Ì 3 4 żǪ&Fko>uIqǰ Figure 3 Four types of camera angles. Fko>uIq& Ǫ'ǯŤ}&ƛǧÑ& óŝčǯ 4)Ť}&ŭƌĿ$#ǯŤ&ŜĂǗ%Ò ƞǰ į¤%ǯHSV ýú%ÙоarwjŤ&ƚŮď ; ţ ƛǧÑ&Ė;ƞǰ Ė8ƛǧÑǣ Ž SǯŤ}&ñ¼ǧÑ%Â.87ƛǧÑǣŽ Sl, Sr ǯŤ }&yzǧÑ%Â.87ƛǧÑǣŽ Su, Sd ;ƪƂ7ǭÌ 4Ǯǰ.ǯKwqņǣ&>uIq!'ǯKwqgOYǯK wq\UYǯůĸǯƦç£$#%4 .871ǯHough ÙĞ%4 ŭƌď Û Ť}&ŭƌ;Ŀ ǯ Ì 5 ñKwqCp>&ƛǧÑǰ Figure 5 The grass region around the left goal area. &ĵĤ L ;ƪƂ7ǰ Su Sl Sr S Sd (a)Kwq¨&ƛǧÑ Ì 4 ƛǧÑ&ǣŽƪƂǰ Figure 4 The area of grass regions. z&đǦ%4 4 żǪ&Fko>uIq;¥ å7ǰ 1. S &Ü!ƛ&ƣĨ;¥å7ǰS y &Ô¿'đǦ 2 %Ǎ0ǰ!$Ô¿'đǦ 3 %Ǎ0ǰ 2. Sl " S r "&ŊŞ;ƪƂ ǯñ¼&ò46Ü (b)Hough ÙĞ!Ŀ8ŭƌǰ Ô¿%'ǯñ¼8&KwqCp>>uI Ì 6 Kwq¨ǰ q"¥å7ǰ!$Ô¿%'ǐĭ>uIq" Figure 6 In front of the goal. 2012 Information Processing Society of Japan 3 ĈÓšãŶžÓÅ IPSJ SIG Technical Report Vol.2012-DCC-1 No.15 2012/5/18 KwqMwuĪ&}!ǯFko>uIq¡6Į: !ǯfwqKwq%ǀ6DŽ.8Ôǣ;üƴ7 "3À>uIqǘƊÔ¿%ǯ`XEĪ;J 1ǰKwq¨Ī"ñ¼KwqCp>Ť'ºƖ$Ô¿ ǭÌ 7Ǯǰ8%4 %'ÜJh©6!ǔƏ7ǰǯJh;ǔƏ7 6ǯ¡68Ť'ł$Fko>uIq5ű1 Əñ&Ô¿%'ÜJh©6;$1ǯK KwqMwuÂ.8 wq¨! h©6ţǢŅŤ" ¡6 6ǯÀżǪ&>uIqƊ 2ì$Jh©6!ǔƏ7ǰ.ǯNjƊ Ô¿%'åijũ%Ť;¡6"!Kwq%Ƙ7ƈǎ Ʀŕ!7"ijÿ87ǰ $Jh©6!ǔƏ8 Kwq Dž;ǔƏ7Ô¿%'ǯ!%¨&ŤÜ 7"71ǯì$J h©6!ǔƏ7"%7ǰ 4. - B ĝľđŐ&ݬć;ŷư71ǯ&Wr `&LUFw}ƉĪ%ë đŐ;Ǒţ ǯJiUH ǫS@NBOYŤ;Ţď7æǬ;ƞǰPaYAB >&Ǚŧ'ǯMacOS X10.6 y% C++" OpenCV o@bop ;ţ Ì 7 Jh©6ţǢŅŤ&¡6ǰ Figure 7 Picking up images for a comic-like image digest. ƞǰ$ ǯX?Qqb¬Ļ'LUFwpwI 3ġĪWr`î%4 żǪŦ$6ǯƛǧÑ&ƚ% 2ORN>j%4 Ŧ$71ǯpcr@Ťǣ&Ė 3Fko>uIq&¥å;ƞ1&_okwR3 3.3 '88!įǑ%$74%đ!Ƭåǰ A)>7 Ì 9 %ƗŢď8JiUHǫJh©6;ĘS@ ǰJh©6&Łď'ìJh"ÜJh&żǪ" NBOYŤ;Ÿǰfwq;Gwc7ǒđ&ǒđ ¡6Ť;JiUH&Jh©6&4%|, 7ǰ Ì 8 &4%ǯ¼y5ñz%Á Ǧ%Ť¾ʼn! 12 ĺǯīƄ£%ǔƏ ¡6 %Ë.8$52Kwq;ō17łß;ƠJiUHǫJ ǰ h©6S@NBOYŤŢď8 7ǰ Ì 8 ƛǧÑ&ǣŽƪƂǰ Figure 8 The area of grass regions. ƀ 2 ſ!dž,4%ǯJiUH!'Jh©6ĨŐ%4 ł$Ơş;ƞǯĵŶž!'.üƴÔ ǣ&Jh©6;Ü7ēŐ;æşǰpcr@!Œ Ì 9 Ţď8JiUHǫJh©6S@NBOYŤ 1ǰ 87KwqMwu5¡68Ť'ǯ¨ƃ!dž, Figure 9 A comic-like layout image digest #1. đŐ%46 4 &Fko>uIq% Ǫ8 7ǰ& ǯæǟ%fwqKwq%ǀ6DŽ.87Ôǣ'Kw ¦&LUFw}ƉĪ5ƗŢď8S@NBOY q¨Ťñ¼KwqCp>Ť%Â.87"Ûǰ Ť;Ì 10 4)Ì 11 %ŸǰÌ 10 !'3'6fwq 2012 Information Processing Society of Japan 4 ĈÓšãŶžÓÅ IPSJ SIG Technical Report Vol.2012-DCC-1 No.15 2012/5/18 ;ĘǒđKwqņǣ Dž!&ǒđ%Ë.87ǯ &Ā%Kwq;ō1"S@NBOYŤ5:7ǰ ǯfwqKwq\UY;Ġ54$Ðũ$ų $ǰxĨǯÌ 11 &S@NBOYŤ% ǚ'Â.8 'KwqMwuÂ.8 ǒđX?Qqb" $MwuƱ $ǰ8'ǯÜ8 ƱĿ8 .ǯƮ¿}&ö KwqMwu" Ė8 . "´Ê!7ǰ.Fko&>uIqzƏ5 &Ô¿%'ǯKwqCp> Dž!'$ Æǃ"¥å8 2KwqCp> .Ô¿7ǰ 5. ĵŶž!'ǯWr`&LUFw}ƉĪ5JiUHǫ &Jh©6;ĘƮ¿S@NBOYŤ;Ɨũ%Ţď 7đŐ;ĝľǰĝľđŐ'ǯpcr@Ī}%Ʈ¿& į2ŜĂũ$Mwu!7KwqMwuÂ.87'! 7"¨ĝ%Òǯ}ƉĪ5pcr@Ī ;Ė ǯ5Fko>uIq&¡6ĮR@iu I%Ò Ť;Ė &Jh©6%4 ǯ85&Ť;JiUHǫ ǔƏǰ&"ǯpcr@Ī& Ė%'X?Qqb&Ŀ;§ţǰ.Fko>uI q'ǯĪ&ƚĈÓ3ŭƌĿ$#&ŤŜĂǗ;ţ 4 żǪ&Fko>uIq;¥åǰ ǯ ¡6Ť 'ǯJiUH!ţ587Jh©6WH[UH;¶Ƒ% $5ǯFko>uIq%4 Ì 10 Ţď8JiUHǫJh©6S@NBOYŤ 2ǰ Figure 10 A comic-like layout image digest #2. Jh&Ü;Ù¯ $5ǔƏ7"!ǯįƇũ$Jh©6S@NBOY Ť;Ţďǰ Ţď8S@NBOYŤ'ǯKwqMwu&Ī īƄ£!Jh©6ƠŸ8 6ǯƮ¿&ƣĐMwu;ŵ īǚ%Ƥƥũ%Ĕğ7"!ǰKwq¨!'Jh ėÜ8 ǯfwqKwq\UY;Ġ5ŝč;ŷư 7"2!ǰǯWsUcĈÓ$#'Ŝ%ĊƸ š; 7:!'$1ǯāř&Ù¯3Kw q;ō1ǒđ$#&ĈÓ'ǯĄ2ě7"! $ǰ Ā&ƳǨ" 'ǯMwu&Ĕğ%ŤŜĂǗÚ& Ŵũ$ĈÓ;ţ7"!ǯ46Ǒ¡%Ʈ¿&ǕƢ$Mw u;Ė6Ť;¡667"ę587ǰ (ǯWsUcĈÓǯǒđ&ǩ3ƕť½ǯn[aDwj ƚ$#;§ţ&Ŝåǯæŏ3ń×$#&Ǥ×ĈÓ &§ţ$#;Ŀƫ7ǰĵƵĥ!'Jh&Ü;Ù¯ 7&²Ɔ$Jh©6;æşǯŴũĈÓ; §ţ7"!ǯīǚƈǎ3>uIq&Ŧ$7ÀīMwu ;|£%¸6Ē4$Jh©6;æş6ǯS@NB OYŤ%JiUH&Ã&4$ĥàĈÓ;LJª 67"2Ŀƫ7ǰ.ǯÉ'LUFw;ë ƺ"ǯWr`!}Ɖ87&OgwV% đŐ&Ǒţ;Ǎ1 26!7ǰ 2012 Information Processing Society of Japan 2 Ì 11 Ţď8JiUHǫJh©6 S@NBOYŤ 3ǭÞĢǮǰ Figure 11 A comic-like layout image digest #3. 5 ĈÓšãŶžÓÅ IPSJ SIG Technical Report < = ĵŶž;Ǎ17%6ǯŚă%ƫƵ Vol.2012-DCC-1 No.15 2012/5/18 ǥ ċŴðŀÜãĈÓŹãǓŘǖûĩƶô"ǯŋǖĎÖŶžè 4)ŘǖŶžè&Ūł%œČƷ7ǰĵŶž&xǓ' ŹŶƼÒū(C)ǭ23500139Ǯǯ 4)ċŴðŀÜãŜ¦Ŷž «ď%47ǰ +813 1) ƹŎǝ», {ƧƔ, Áþ, ìĶ: ĕŠMwu&ƨĹ"W sUcĈÓ;ţǖŠ}ƉS@NBOYś&ƗŢďŐ, Ī ĈÓkX?>ãēƟÓÅ, 28(71), pp. 27-30 (2004). 2) éïƜÄ, ÜǛÈÝ: LUFwŤ% 7Ťš%47 ]@o@YŢď, ĪĈÓkX?>ãēƟÓÅ, 35(8), pp. 17-19 (2011). 3) Myodo, E., Ueno, S., Takagi, K., Sakazawa, S.: Automatic Comic-like Image Layout System Preserving Image Order and Important Region, Proc. of the 19th ACM International Conference on Multimedia, pp. 795-796 (2011). 4) Calic, J., Gibson, D.P., Campbell, N.W.: "Efficient Layout of Comic-like Video Summaries", IEEE Transactions on Circuits and Systems for Video Technology, Vol. 17, No. 7, pp. 931-936 (2007). 5) ǘÎĬı, éŇâĥ, ƹ¹®Ɛ: ǂùÙ¯&²ƴć%ŲŬ XQqbĿđŐ, ǡßĈÓNJãƋ¿ÜƶŗƵĥǠ,1996 ÷, ĈÓvMOWj(2), p. 403 (1996). 2012 Information Processing Society of Japan 6