One of the projects that I am working “CloudBank” required that I somehow capture the frames from a video game that was displayed in a separate window on the screen. I needed to do this so that I could analyse and process the frames using OpenCV. A few searches on google shows that this is one of the most frustrating things to accomplish so in this post, I am going to show you how I used the media streaming framework GStreamer to accomplish this. By using GStreamer I was able to accomplish the following:
-
- Capture the video game frame that is being played in a separate window and then save that frame into a file of any format that can then be analyzed and processed by an OpenCV program.
- Capture the video game frame in a way that DOES NOT require the window to be placed in any specific position of the screen.
- Control the frame capture rate.
GStreamer command and options.
Let us analyze the GStreamer cmd that I used for my project:
gnome-terminal -x gst-launch-1.0 ximagesrc xname=Transistor use-damage=0 ! tee name=t ! queue ! videoconvert ! videorate ! "video/x-raw,framerate=30/1" ! autovideosink t. ! queue ! videoconvert ! jpegenc ! multifilesink location=/home/sheun/Cloudbank-DEV_Trans/game_vision/current_game_frame.jpg
- ‘gnome-terminal -x ‘: this opens a new terminal that we will launch GStreamer and ‘-x’ will execute the remainder of the cmd line in the terminal.
- gst-launch-1.0: launch GStreamer (version 1.0)
- ximagesrc, xname, use-damage:
- ximagesrc: Capture your desired window and creates a raw RGB video. (default 25 frames per second)
- xname: The name of the window you want to capture. (IMPORTANT NOTE: This will capture any window that possesses the name of the window you want to capture. Ensure that your desired window is THE ONLY ONE with that name on your screen )
- use_damage: enable XDamge (default is true)
- tee: Needed when you want to capture a video on the screen and save it into a file.
- queue: make sure that the data is queued. I’ve found this very useful to ensure that a frame is completely saved in a file before another frame is written into that same file.
- videoconvert: convert video frames into different formats.
- videorate: control the video capture framerate. (In my example I use ‘”video/x-raw,framerate=30/1″‘ to state that I want to capture them the raw video output at 30 frames per second )
- autovideosink: detects the video sink (which is a type of media sink) to use. We place the data that we split from ‘tee’ which we named “t”.
- jpegenc: encode jpeg images (the file format that I will be saving the frames as).
- multifilesink: write frames into a file at a specified location.
Running the cmd
On Linux, you can run a command in a variety of different ways. In my project, I run the Gstreamer cmd (written within a .sh file) within my C++ using the function “system” and pass in the cmd to run the script as the argument. For example to run the script/file that holds the GStreamer cmd mentioned before I used:
system("<path_to_file> &");
the “&” symbol runs the command in a terminal in the background.
The image below shows an example of the game “Transistor” that is being run on a Steam and a separate video output produced by GStreamer. each frame is saved into a separate file at a location that I have specified.
Sheuns blog 