Setting up a simple Streaming Performance test with Jmeter

Introduction

Streaming tests are a mandate in today’s digital landscape. The critical challenge is to ensure seamless and quality live streaming depends on a large number of factors like network, server capacity, live-streaming protocol, and notably the number of users streaming concurrently.

In this blog, we learn a very simple yet efficient way to performance test the streaming of a video with the HTTP Live Streaming(HLS) protocol.

Pre-Requisites

• Install Jmeter in your system. You can download the latest version from this link
• Once the Jmeter is installed, the next comes the installation of the Jmeter plugin manager. Please follow this link
• Once you restart Jmeter, you can find the plugin manager under Options

• Open the Plugin manager and then install “Blazemeter – HLS Plugin”
• Once done, you should be able to see it in the installed plugins list.
• The most important pre-requisite is to have a video that you are going to stream.

Video Pre-Processing

With the video, execute this command. 

ffmpeg -i filename.mp4 -codec: copy -start_number 0 -hls_time 10 -hls_list_size 0 -f hls filename.m3u8

With this command, you will see the video split into n number of videos.

The user should also be able to see a m3u8 file.

We split the video into n number of segments. When streamed, each of these segments is fetched sequentially and it looks like the video is being streamed in real time.

Once this is done, upload the video to S3 and get the m3u8 URL.

Note: In this blog, we will only be considering how to run in a test with the objects being public. So if you are following this blog, please make sure to have all the above files public. 

For this test, we are going to use this video

KPI:

• Buffer time
• Startup duration
• Errors

Buffer time – 

• Each segment will be fetched sequentially one after another.
• The easiest way to know if there will be a lot of buffering is to measure the time it took for each segment to be fetched and then calculate the average. This average needs to be less than the duration of the videos. In our case, each video is 10s and if the average response time to get these videos is less than 10s, then it is good to assume that the backend performance is on par.

Startup Duration – 

It is a measure of how long it takes for the application to start streaming the video.

APIs:

There are 3 main APIs involved in this test.

• Media Playlist – This API returns the information on the master playlist associated.
• Master Playlist -For the sake of performance testing the app, we only had 1 supported resolution and bitrate.
• Media Segment – This API request gets all the chunks of the video(i.e streams)

The Media playlist is called and then it returns the master playlist information.

The master playlist contains the path to all the chunks and their order.

Each chunk is then obtained via the Media Segment API call.

The media playlist/master playlist does not contain any video data. All the video data is stored in the Media segment API.

In our test, there is no master playlist file present since there is only 1 config we are running this test at.

Configuring the test:

• Open Jmeter from the Bin folder.
• Now, the user sees a test plan on the left navigation tree.
• Add a thread group by right-clicking on the Test-Plan.
• Clicking on the thread group should open the below window in Jmeter.

• It is important to understand and configure the Thread group since these define the parameters of the test.
  • Number of Threads(users):

This represents the number of simultaneous threads that will be doing an action/ calling an API. Each thread is independent of another and every step in the test plan gets executed separately in them. This is just equivalent to how each user needs to log in independently inside their application.

• Ramp-up Period(seconds)

Imagine the number of threads being 5. It does a little extra good to start the users from 1 and then ramp it up to 5. This enables us to validate the performance of the test app with different thread counts. If you set your Ramp-up period to 1, then after 1s, the Jmeter increases the number of threads to 2 and then so on till the number reaches 5.

• Loop Count:

It is the number of times you want to test to be executed. It is detached from ramp-up and number of threads. For example, it will keep looping even if the requests are already sent.

For ease, I am setting the Thread group as above. 1 user will live stream a video 1 0 times in loop. 

• Now, Right click on the Thread Group -> Add -> Sampler -> bzm – Streaming Sampler.

• Configuring this is a piece of cake.
  • Please the m3u8 file’s URL in the URL field.
  • We are testing the streaming of videos with the HLS protocol. So, please choose HLS there.
  • In the case of Duration, I entered Whole video since the video is of a smaller size. It would make sense to fetch the video by the duration of 2s if it is 4k since that makes for faster streaming.
  • We do not have any tracks in this video, we leave them empty.
  • Bandwidth information is useful for test streaming applications like Youtube which allows multiple resolutions of the same video. In our test, there is only 1 resolution so we choose Max bandwidth available and Max resolution available.
• Finally, Add a Summary report to the thread group. You can do that by right-clicking on Thread Group, Add -> Listeners -> Aggregate Report. The Aggregate report contains all the relevant test information.
• Run the test and once done, look at the Aggregate Report.

• We are looking at 3 different APIs. The first one is the Master playlist, the second one is the Media playlist and the third one is the Media segment.

Results

• Buffer time Calculation

From the results, it is seen that each of the media segment requests has an average throughput of 82ms. The minimum duration of all the videos in the segment is 4s so we are likely to see little to no buffering in this load.

• Lag Calculation:

The video is not split into different channels like Audio, video, and subtitles.

Each of the chunks contains all of the components.

No lag can be seen with this setup.

• Video Startup Duration:

The average response time of the Media playlist is 49ms. Since this API translates to how quickly the video starts playing, the performance of this is extremely good.

With this, you have successfully run your streaming performance test. Thanks for reading, Watch out for more blogs that cover more than just the basics of performance tests.