Introduction of GStreamer

A few months ago if you’ve asked me what GStreamer is I would’ve said that I don’t know anything about it, or maybe knew what it is in a very high level.

I’ve had the opportunity to work on a project which demanded a very deep understanding of GStreamer, and it’s inner parts, and it still something I’m in the progress of learning because it’s like learning a programming language.

Something that I’ve did experienced in using it is that it isn’t that easy to understand GStreamer unless you go deep regarding the thing you’re trying to build, and also understand how OS, Networking, and especially Real-Time communications work.

So while doing that project, I’ve also learned more deeply about OS and Networking so I wanted to give a good understanding of GStreamer for anyone who wants to use it or simply just to understand what it does, and how it can be used.

The reason for that is because while doing that project I remember the days and nights that I got frustrated by not achieving the result I wanted, and I simply got stuck a lot of times catching up with missing information from various fields.

That’s why I wanted for this post to have a good understanding of GStreamer in short, so anyone else would have an easy beginning ☺

So what is it all about?

GStreamer allows us to manipulate, send, and receive data on our machine in various protocols.

For example, we can generate on our machine RTP data and also read it.

We can also create an RTSP server that uses some source of data like RTP or a file(WAV/MP3 for example) and allows RTSP clients to access it.

For anyone who doesn’t know what is RTSP, it’s a streaming protocol that allows a VCR like functionalities with networking sources like security or home cameras, media servers, encoders, and more…

GStreamer allows one to achieve those functionalities by adding their libraries to an application that we write or simply by a one-line command to execute.
In order to understand the differences, we’re going to understand later on.

The most basic part of GStreamer is that we’re building pipelines.
A pipeline is describing a flow of operations that start from a source, goes through some operations we wish all along the way to the destination of it.

An example you ask?
Let’s say we wish to take our cool podcast recording file which is in wav format, but we wish to convert it to mp3.
We can define the file as the source -> parse out of the wav file the raw audio -> encode it with mp3 -> save that raw audio to its destination which is a new mp3 file.

Few basics semantics

Before we start talking about examples of implementations, let’s see what actually GStreamer is composed of first.

Element

An element is a GStreamer module that does some operation.
Regarding the previous example, we can consider parsing the wav file to raw audio as an element and also parsing raw audio to mp3 as an element, which we will see both later.

An element could be one of the following :

• Source element — Only accepts a source of data and doesn’t get any input in our pipeline, an example could be defining a file source to use.
• Manipulation element(filter, converter, codec, decoder, and more) — These elements are the ones that decide the outcome from our pipeline, and we can use them to do a lot of things.
  The challenge with them is actually understanding efficiently how to bind them together to work.
• Destination(Sink) element — This actually describes what will happen with the data at the end of the pipeline.
  The destination could be a lot of things but just for example they could be network, files, other pipelines, and more…

Bins

This point is here by no mistake.
When you must’ve read the ‘other pipelines’ in sink elements(above) your eyebrow must have risen up like Dwayne Johnson iconic look.

We can define a pipeline that reads from a file, decodes the raw data, and then queues the data to a destination that could be read by other pipelines.
We can then use that sink destination to do multiple functionalities on that data like playing it, sending it to a networking location, and more…

The pipeline that reads from the file and decodes it is a bin.
From what you’ve must understand is that we can build “sub pipelines”, and use them as an element.

Bus

Like a networking bus or simply an entity that allows receiving a message from what goes on in our pipeline.

It allows us to do a lot of things, but for the simplest thing in order to understand, we can use it to get a message if the pipeline received an error of stopped doing his work.

Pads

Pad is a little complicated to understand at first so if we will use the definition from the original website, you might be having Dwayne Johnson's iconic eyebrow again, so I’ll try in another way to describe it.

Let’s say we’re using an element that decodes raw data to an MP3 stream that could be saved into a file destination.
You might be saying that the source is raw data and the destination is MP3 stream data.

Well, you’re right and we talked about that elements could be a source, manipulation, or a sink element and for each one there is a pad to understand what the element does.

Each pad describes what its capable of doing for its element.
If we take the wav parser element, and we look under the “Pad Templates” section, we can see for the sink it receives a wav and RF64 format for our stream.
For the src, we can see that it has multiple outputs of data it give.

If we will not follow the pads description we will receive an error when we will try to run our pipeline.

One-line command vs GStreamer based application

When will use the GStreamer as a one-line command, we will get the result we wish out of our pipeline and that is cool and all, but what about getting more intel from our pipeline to understand better what happened and how to continue our application behavior regarding the pipeline.

If you’ve asked that then you’ve asked a good question and there’s an answer.
If we will create a GStreamer based application of our own then as described before, we can receive errors, re-play our pipeline, play another pipeline instead when receiving another kind of data from a network source, and a lot more…

When we’re developing a GStreamer based application we can build or execute our pipeline in two ways.

We can either take the command we execute in a one-line command and ask GStreamer library to execute that pipeline and that will work like a charm.

Another way would be building a pipeline using the GStreamer library one after another in a more programmatic way, and that will work as well.

As you might already understand, if we will use the one-line command it could be used in our GStreamer application and also in the command line or other GStreamer-based applications that are written in other programming languages.

The programmatic way could only be used in the same programming language due to library use that matches the programming language runtime.

Give me a good example

Now what I wish to show you is a good and simple example of a line-command.

If we will take the following command and execute it in the same location of wav audio file, and access to an installed GStreamer on our PC then the result would be hearing the wav file.

gst-launch-1.0.exe filesrc location=some-audio-file.wav ! wavparse ! audioconvert ! audioresample ! autoaudiosink

This command would work on windows.
In case for dear Linux or Mac users execute this command:

gst-launch-1.0 filesrc location=some-audio-file.wav ! wavparse ! audioconvert ! audioresample ! alsasink

In general, those pipelines start from the source which is the filesrc element.
For each element, we can also define properties to configure them which is in this case the location property, that allows us to tell filesrc element where to get the file from.

Now the file binary data is being processed by another element which is wavparse.
What it does it takes the wav encapsulation from the raw audio and gives that out of it.

Afterward, we do further manipulation to allow hearing the content much better using audioconvert and audioresample(mainly audioresample).

In the end, we send the raw audio that went resampling to the sink element which actually makes the pipeline allowing us to hear the file content.

In conclusion

GStreamer is a great tool for many cases regarding multimedia and networking applications.

Today, as far as I know, it’s even being used to develop encoders which allow getting analog streams from cameras by RTSP, and many more cases.

I hope you had a great time reading this piece, and if you have any more further questions I would be delighted to answer them.
Also, if you have any opinions or suggestions for improving this piece, I would like to hear :)

Thank you and have a great journey on your journey for knowledge! :)