Play Videos in R

This bite has some quick code to parse video files and render them to the console using only vanilla methods. (and the av package to parse video)

Code for this can be found in video-player.R in the repo or in the main r-bites zip.

vid = load.video('XcQ.mp4')
play.video(vid)

The console must be resized (and zoomed in or out) until the video renders properly.

Squinting helps a lot.

Big Picture

To render a video, we need:

  • a way to convert high-res, color images to something viewable in the console

  • a way to load and display each frame at the right time

Functions

render.matrix()

Renders an integer matrix to the console as a single wrapped line, overwriting the previous render.matrix() output. This allows us to animate video frames by drawing them in succession.

render.image()

Converts an image (e.g. from jpeg::readJPEG()) into a matrix suitable for render.matrix. This grayscales and quantizes it to only a few shades of gray.

load.video()

Reads a video file and generates an image for each frame using av::av_video_images(). The video is also downscaled and set to a specified framerate for ease of rendering.

play.video()

Reads a list of images as generated by load.video() or av::av_video_images() and renders each frame at the appropriate time. If rendering takes too long, frames will be skipped to preserve the overall video speed.

Code Details

library(av)
library(jpeg)

We use the standard video processing package av to generate JPEG images from a video file, and the jpeg package to read these images.

av uses FFmpeg to process videos, which you might have to install.

render.matrix = function(M,palette=c('▓',' ')) {
    cat(
        #clear previous render
        '\r',       
        #index palette to convert matrix to characters
        c('\t',palette)[
            #transpose to orient matrix properly 
            #pad matrix with tab characters
            t(cbind(M,matrix(-1,nrow(M),10)))+2
                #add two to make it 1-index
        ], sep='')
}

Console rendering is achieved by calling base::cat() on a single-line string which wraps to match the dimensions of the matrix we feed in.

Additionally, we:

  • use \r to clear the previous frame rendered by the function

  • use \t tab characters to wrap each row onto a new line, without having to use actual \n newlines (which would break \r)

The input matrix consists of positive integers (and 0) corresponding to the index in palette of the character that should be drawn for each cell in the matrix; this function essentially renders a simple bitmap matrix as unicode characters.

render.image = function(img,shades=2){
    #luma greyscale
    img = 0.2126 * img[,,1] + 
        0.7152 * img[,,2] + 
        0.0722 * img[,,3]
    
    #filter
    floor(img * shades) 
}

Images are prepared for render.matrix() by flooring them to only a few integer values (determined by shades) to be used to index the character palette. Before this, the image is greyscaled by combining the color channels into one matrix.

load.video = function(path,size=128,framerate=25) {
    #size = desired horizontal resolution
    
    cat('Downscaling...\n')
    #get resolution metadata
    info = av_media_info(path)$video 
    
    scale = c(info$width,info$height)
    scale = scale * (size/scale[1]) #scale to size
    scale[2] = scale[2]/2 #halve vertical scale
    scale = floor(scale/2)*2 #make resolution even
        #the encoder hates odd resolutions
    
    #create temporary directory for downscaled video 
    temp = tempdir()
    temp = paste0(temp,'downsampled.mp4')
    av_encode_video(
        path,
        output = temp,
        framerate = framerate,
        vfilter = paste0("scale=",scale[1],':',scale[2])
    ) #downscale video
    
    cat('Extracting frames...\n')
    av_video_images(temp,fps=framerate) 
        #automatically creates its own temporary directory
}

Before using av::av_video_images() to generate a list of image files for play.video(), we use av::av_encode_video() to downscale the image to the desired resolution for rendering.

We could manually downsample the image files to get the desired resolution, but it’s a much better idea to precompute this step, since generating the image files in the first place already requires precomputation.

The downscaled video is saved to a temporary file, and av::av_video_images() generates a temporary directory to store the image frames.

play.video = function(
        video_data,
        target_framerate = 25,
        palette=c('▓','∏','░',':',' ')
){
    frame = 1
    while (frame < length(video_data)){
        #get time before rendering frame
        t = as.double(Sys.time()) 
        
        #render frame
        render.matrix(
            render.image(
                readJPEG(video_data[frame]),
                shades=length(palette)
            ),
            palette=palette
        )
        
        #how long that took to render
        dur = as.double(Sys.time()) - t 
        
        #how long that took in frames
        dur_frames = dur/(1/target_framerate) 
        
        #advance to next frame
        frame = frame + ceiling(dur_frames) 
        
        #time until the next frame
        buff = (dur_frames - floor(dur_frames)) * 
            1/target_framerate 
        
        #sleep to the start of the next frame
        Sys.sleep(1/target_framerate - buff) 
    }   
}

play.video attempts to render each image every 1/target_framerate seconds.

The rendering is done with render.matrix(render.image(readJPEG(file))), where file is the path to the image file supplied by load_video().

In order to play at the desired framerate, the video player records the time it takes to read and render the frame (which is not insignificant, especially at higher resolutions) and then sleeps for the remainder of the frame.

The function also has code to skip frames if the time it took was greater than the intended duration of the frame.

Notes

Palette

I chose this default (recommended) palette after quite a bit of experimenting. It aims to provide high contrast while still giving good tone resolution without using too many shades; it’s quite a bit more interpretable than many of the other palettes I tried.

The supply of good-quality characters for palettes is severely limited: many characters (notably, the full block) render too small or not monospaced1 in the RStudio console, and even those that don’t still have significant gaps between lines. This is one of the few combinations with good contrast that doesn’t introduce distracting shapes and patterns.

Pure R

This is intended to run in RStudio; it actually does work in R.app, but R.app renders much less frequently than RStudio and therefore can’t support a very good framerate.

Footnotes

  1. Of the default uniform block shading elements █▓▒░, RStudio renders █ at half height and ▒ as thinner than a regular monospaced character. It’s absurd!↩︎