Wolfram Blog
Shadi Ashnai

Computational Video Premieres in Wolfram Language 12.1

2020年5月19日 -Shadi Ashnai,声音和图像,算法R&d经理

Version 12.1在Wolframanbet万博appm语言介绍的期待已久Videoobject. TheVideoobject is completely (and only)out-of-core; it can link to an extensive list of video containers with almost anycodec. Most importantly, it is bundled with complete stacks forimageandaudio processing,machine learningandneural nets,统计andvisualization和许多更多的功能。这已经使得Wolfram语言强大的视频计算平台manbet万博app,但仍有更多的功能探索。

The Video Object

A video file typically has a video and an audio track. Here is aVideo对象链接到一个视频文件:

Video

Video["ExampleData/Caminandes.mp4"]

在12.1版本,by default, theVideoobject is displayed as a small thumbnail and can be played in an external player. There are other appearances to enable in-notebook players, like theVideo一个基本的播放器对象:

Video

视频[ “ExampleData / Caminandes.mp4”,外观 - > “基本”]

Now you can inspect theVideoobject:

Duration

Duration[Video["ExampleData/Caminandes.mp4", Appearance -> Automatic, AudioOutputDevice -> Automatic, SoundVolume -> Automatic]]
Information

Information[ Video["ExampleData/Caminandes.mp4", Appearance -> Automatic, AudioOutputDevice -> Automatic, SoundVolume -> Automatic]]

Most video containers support multiple video, audio and subtitle tracks. Having multiple audio or subtitle tracks in a single file is more common than having more than one video track.

这是一个的例子Videoobject linking to a file with multiple audio and subtitle tracks:

Information

Information[Video["ExampleData/bullfinch.mkv"]]

Accessing Parts of a Video

There are several parts of a video you may be interested in extracting. UseVideoFrameListandVideoExtractFramesto extract specific video frames. You can also useVideoFrameListto sample the video uniformly or randomly with框架:

VideoFrameList

VideoFrameList [视频[ “ExampleData / Caminandes.mp4”,外观 - >自动,AudioOutputDevice  - >自动,SoundVolume  - >自动],3]

Use this function to create a thumbnail grid (a group of smaller images that summarizes the whole video):

VideoFrameList

VideoFrameList [视频[ “ExampleData / Caminandes.mp4”,外观 - >自动,AudioOutputDevice  - >自动,SoundVolume  - >自动],12] // ImageCollage

You can also trim a segment of a video:

VideoTrim

VideoTrim[ Video["ExampleData/Caminandes.mp4", Appearance -> Automatic, AudioOutputDevice -> Automatic, SoundVolume -> Automatic], {30, 60}]

或提取只能从视频中的音轨进行分析:

Audio

Audio[Video["ExampleData/Caminandes.mp4", Appearance -> Automatic, AudioOutputDevice -> Automatic, SoundVolume -> Automatic]]
Spectrogram

Spectrogram[%]

Performing Analysis

在12.1版本中,我们已经介绍了VideoTimeSeries,它工作在一个视频文件的帧来一次执行在任何时间计算,任何一个或多个帧的列表中的所有。这是能够分析像在下面的例子中的有力工具。

Compute the mean color of each frame over time:

VideoTimeSeries

VideoTimeSeries[Mean, Video["ExampleData/Caminandes.mp4", Appearance -> Automatic, AudioOutputDevice -> Automatic, SoundVolume -> Automatic]] // ListLinePlot[#, PlotStyle -> {Red, Green, Blue}] &

Count the number of objects (cars, for example) detected in each frame of a video:

V =视频

V =视频["http://exampledata.wolfram.com/cars.avi"];
TS = VideoTimeSeries

TS = VideoTimeSeries[Point[ImagePosition[#, Entity["Word", "car"]]] &, v]

Plot the number of objects (again, using cars as an example) detected in each frame:

TimeSeriesMap

TimeSeriesMap [长度@@#&,TS] // ListLinePlot

Highlight the position of all detected objects (cars) on a sample frame:

HighlightImage

HighlightImage[ VideoExtractFrames[v, 1], {AbsolutePointSize[3], Flatten@Values[ts]}]

We can also use the multiframe version of the function to perform any analysis that requires multiple frames.

通过观察从连续的帧pixabay视频和计算四个视图之间的差别,我们可以发现从一个视图中的过渡时间到另一个,然后使用这些时间,以提取每个场景中的一个帧:

V =视频

V =视频[ “Musician.mp4”]
diffs = VideoTimeSeries

diffs = VideoTimeSeries[ImageDistance @@ # &, v, Quantity[2, "Frames"], Quantity[1, "Frames"]]
ListLinePlot

ListLinePlot[diffs, PlotRange -> All]
times = FindPeaks

times = FindPeaks[diffs, Automatic, Automatic, 150]["Times"]
VideoExtractFrames

VideoExtractFrames[v, Prepend[times, 0]]

Process a Video

The Wolfram Language already included a variety of image and audio processing functions.VideoFrameMapis a function that takes one frame or a list of video frames, filters them and writes them to a new video file. Let’s use the bullfinch video:

V =视频[ V =视频[

V =视频["ExampleData/bullfinch.mkv"]; VideoFrameList[v,3]

We can start with a color negation as a simple “Hello, World!” example:

VideoFrameMap

VideoFrameMap[ColorNegate, v] // VideoFrameList[#, 3] &

Or posterize frames to create a cartoonish effect:

f = With

f = With[{tmp = ColorQuantize[#, 16, Dithering -> False]}, tmp - EdgeDetect[tmp]] &;
VideoFrameMap

VideoFrameMap[f, v] // VideoFrameList[#, 3] &

Use a neural net to perform semantic segmentation on the previously used video of cars:

V =视频

V =视频["http://exampledata.wolfram.com/cars.avi"];
segment

segment[img_] := Block[{net, encData, dec, mean, var, prob}, net = NetModel["Dilated ResNet-38 Trained on Cityscapes Data"]; encData = Normal@NetExtract[net, "input_0"]; dec = NetExtract[net, "Output"]; {mean, var} = Lookup[encData, {"MeanImage", "VarianceImage"}]; Colorize@ NetReplacePart[ net, {"input_0" -> NetEncoder[{"Image", ImageDimensions@img, "MeanImage" -> mean, "VarianceImage" -> var}], "Output" -> dec}][img]]
VideoFrameList

VideoFrameList[VideoFrameMap[segment, v], 3]

Next is a video stabilization example, which is a vastly simplified version of this Version 12.0product example. The input video is another pick frompixabay:

V =视频

V =视频[ “soap_bubble.mp4”]

Here is the mask over the ground to make sure the shaking soap bubble movement does not affect our stabilization algorithm:

mask = CloudGet

mask = CloudGet["https://wolfr.am/Mt580rl0"];

其次是要找到每两个连续的帧之间的对应关系和几何变换,与先前的变换迭代组成获得稳定一路初始帧例行:

f = Identity;

f = Identity; VideoFrameMap[ Module[{tmp}, tmp = Last@ FindGeometricTransform[##, TransformationClass -> "Rigid"] & @@ ImageCorrespondingPoints[Sequence @@ #, Sequence[ MaxFeatures -> 25, Method -> "ORB", Masking -> mask]]; f = Composition[tmp, f]; ImagePerspectiveTransformation[#[[2]], f, Sequence[ DataRange -> Full, Padding -> "Fixed"]]] &, v, Quantity[2, "Frames"], Quantity[1, "Frames"]];

From Manipulate to Video

让我们将话题切换到代视频的。操纵已经超过十年创建的Wolfram语言动画的核心方式。manbet万博app在12.1版本,操纵表达式可以容易地被转换成视频。

This is a操纵来自Wolfram Demonstrations Project:

米= ResourceData [“演示项目:日夜世界时钟”]

And a video generated from it:

Video

Video[m]

A video can also be generated from a操纵and aSoundorAudioobject:

出口

出口["file.mp4", {"Animation" -> m, "Audio" -> ExampleData[{"Audio", "PianoScale"}]}, "Rules"] // Video

一张纸条,约支持的编码解码

The Wolfram Language by default uses the operating system as well as a limited version ofFFmpegfor decoding and encoding a large number of multimedia containers and codecs.$VideoEncoders,$VideoDecoders,$AudioEncoders等列表支持的编码器和解码器。

Codec support can be expanded even further by installing FFmpeg (Version 4.0.0 or higher). This is the number of decoders and the list of MP4 video decoders on macOS with FFmpeg installed:

Length /@ $VideoDecoders

Length /@ $VideoDecoders
$VideoDecoders

$VideoDecoders["MP4"][[All, 1]]

More to Come

在Wolfram语言视频的计算只在起步阶段。manbet万博app这里展出的新功能,只是一个已经强大的集合的一部分video basics, and we are actively designing and developing updates to existing functions and additional capabilities for future versions, with machine learning and neural net integration at the top of the list. Let us know what you think in the comments—bugs, suggestions and feature requests are always welcome.

可以完全访问最新的Wolfram语言的功能与manbet万博app数学12.1orWolfram|One审判。

Leave a Comment

一个评论


埃莱亚萨J.

优秀的职位和一个愉快的阅读沙堤!

Thanks,
埃莱亚萨

发贴者埃莱亚萨J. 2020年6月9日在下午9点09分


Leave a comment