Witbe QoE Snapshot - Who is the best mobile operator for video apps in Canada?

Introduction & setup

The environment for this Snapshot consisted of a Witbe “Video & Media” Mini Robot hosted in the Witbe Montreal Office in Canada and connected to four Apple iPhone 7 Plus running iOS version 12.1.2.

The four mobile operators tested were Bell Mobility, Fido, Telus Mobility and Videotron (in 4G LTE). Fido and Videotron advertise 60Mbps download speed, Bell and Telus advertise 85Mbps download speed.

In order to test the mobile operators, we used five of the most popular OTT video streaming apps in Canada: YouTube, Netflix, Tou.TV, CBC Gem, and Crave. Each mobile operator was tested with every OTT app. Each OTT app was playing the same video across network.

We measured and verified the following KPIs:

• App Launch Time: time between the Robot clicking on the app icon and the app being ready to be used (the apps are “force-closed” after every run)
• Login Time: time between the Robot clicking on the login button (after credentials were input) and it being successfully logged-in (the apps are logged-out before every run)
• Video Initial Buffering Time: time between the Robot clicking on the play button and the video starting to play
• Witbe VQ-MOS Score: a score representing the video quality of the first 30 seconds of video, measured with our famous Video Quality Mean Opinion Score algorithm

Global Overview

By looking at the medians of the four KPIs that were measured, we see that the overall Quality of Experience when viewing videos through mobile networks in Canada is good. Performances KPIs (App Launch Time, Login Time, and Video Initial Buffering Time) are all within the optimal range that we determined based on our expertise. No major discrepancies were noticed between the four mobile operators.

This shows that in the center of Montreal, all tested mobile operators are providing an equivalent level of Quality of Experience. Still, it is interesting to observe that even though all the operators are advertising a download speed above 50Mbps, the Witbe VQ-MOS score is averaging at 3, which is decent but not at the value of around 4 that would be expected for HD content.

We can see that the top 4 performing operators for this channel change test are North American, with a Channel Change Time under 2 seconds. It can partially be explained by the fact that European operators have more 1 and 2-digit channel codes in their test lineup.

Mobile operator performances during peak hours

For three of the four mobile operators, performances measured over the 24-hour period in Montreal are not really impacted by traffic peak hours and are pretty much constant. We noticed an increase of the App Launch Time for Videotron starting at 6pm.

By drilling down into the App Launch Time KPI for Videotron, grouping the KPI by OTT app, we can see that there is a problem with Netflix only and not with the other apps.

The video above shows the excessive time it took the Witbe Robot to launch Netflix on one of the viewing session. (Witbe Robots record video traces for every test, in order to replay the scenario and understand what the Robot measured — this video was re-encoded and compressed).

A 24-hour study does not allow us to draw further conclusions regarding this issue. In order to understand the root cause, it would be interesting to re-run the same Netflix test on Videotron mobile network for several days, and to activate the new network tracing capabilities of Witbe Robots 6.2 (since November 2018, Witbe Robots are indeed capable of recording the network PCAP traces of any iOS device, without jailbreak, and through any 3G/4G mobile or WiFi network).

A surprising video availability rate

While analysing video availability for each mobile operators, we were surprised that Crave was the only app without a perfect 100% Video Availability Rate. 4% of Crave’s viewing sessions revealed a problem during the start of the video.

The screenshot above shows the error experienced by the Witbe Robot during one of the incomplete viewing sessions.

It seems this availability problem is not due to a network or service issue but to a bug in the app where a parental controls popup is mistakenly displayed at random when requesting a video to be played.

This type of errors is the perfect example of why it is so important to test video apps as close as possible to real end-user conditions: with real apps, on real devices, and testing the service from one end to the other. Replicating users’ behaviors with active but non-intrusive Robots has been Witbe’s methodology since the beginning.

A quick comparison between the heavyweights

As a side note, by focusing on the performance for every mobile operator on the most popular Apps (Netflix and YouTube), we can see that YouTube has the fastest Video Initial Buffering Time and the lowest Witbe VQ-MOS score, and Netflix has the slowest Video Initial Buffering Time and the highest Witbe VQ-MOS score. This highlights once again the challenges faced by OTT app’s engineers when balancing load times and video quality on unmanaged networks (the same observation was noted during the first Witbe QoE Snapshot, comparing mobile OTT apps in the UK).

Because the tests on Netflix and YouTube were using the same video content (Bird Box Trailer), it is a good opportunity for us to compare VQ-MOS scores on the very same frame:

The two images above are encoded in PNG without loss and uploaded on this page without further compression. For a deeper analysis of Blur scores, and another comparison of Witbe VQ-MOS scores please refer to Witbe QoE Snapshot #2.

Conclusion

To conclude, the result of this Snapshot is quite encouraging for mobile operators in Canada, with good and constant performances. One area where they could improve appears to be the overall video quality which is lower than the one measured (on WiFi) in the UK, during Snapshot #1.

Providing end-users with enough bandwidth to watch HD quality videos on their mobile devices in an area with a high population density is one of the many challenges operators face today, as well as covering 100% of a very large territory or keeping up with the ever evolving technologies (3G, 4G, and soon 5G). Witbe “Video On-the Go” technologies can help operators prepare themselves by enabling them to map the true quality of their mobile networks with video KPIs.

For more details about the results of this QoE Snapshot, don't hesitate to contact us!

See you soon for the next Witbe QoE Snapshot!

About Witbe QoE Snapshots

In the same way that a consumer report tests a product and publishes an analysis of its overall quality, the Witbe QoE Snapshots test digital services to make available to the market information on the true Quality of Experience delivered internationally. These QoE Snapshots should not serve as benchmarks nor as rankings of operators by service or by device. Rather, the goal of these QoE Snapshots is to provide a global overview of digital services with multiple configurations, and in various environments. The public will thus be able to better understand the technological complexity inherent to today’s services, like the distribution of video content. It is quite a technical feat – considering the efforts and means implemented – to broadcast videos on different devices and networks, with a quality that is acceptable by consumers with high expectations.

Since its origin, Witbe relies on a non-intrusive technology, based on Robots measuring the quality truly delivered. The Witbe Robots are placed at the edge of distribution, and connected to test devices, the same ones as those used by real users. The Robots measure the Quality of Experience actually delivered to the end-users by providing KPIs on the availability, performance and integrity of the service.

Each snapshot is composed of several analytical frames, highlighting interesting findings about the KPIs that were measured. In our last QoE Snapshot, we looked at who was the best OTT app on Apple TV in the United States. This time, we are publishing a look at the quality of four mobile operators in Canada: Bell Mobility, Fido, Telus Mobility and Videotron.