Introduction & setup
The environment for this QoE Snapshot consisted of a Witbe Video & Media on-the-go robot, connected to two Apple iPhone 8 running the latest version of iOS.
Installed in a lab van or a regular car, the Video & Media on-the-go robot controls mobile devices, connected to the operator’s mobile network, in drive-test conditions. Instead of measuring the quality of the network by looking at QoS indicators, it is connected to real end-user test devices, and looks at true QoE KPIs. This allows operators to map out the true quality of experience of their mobile networks.
Our specially chartered Witbe lab van drove four times around north London, on Friday April 5th, from 12pm to 8pm. Four runs were performed using two phones at the same time to compare a British mobile operator with one of its associated MVNOs, by playing the same video in the BBC iPlayer app.
The following carriers and MVNOs were tested:
- Run #1: Vodafone and Lebara Mobile
- Run #2: EE and BT Mobile
- Run #3: O2 and Sky Mobile
- Run #4: Three and ID Mobile
We measured and verified the following KPIs:
- App Availability: percentage of viewing sessions for which the application successfully launched
- Video Availability: percentage of viewing sessions for which the video both successfully started to play & successfully played for a minute
- Video Initial Buffering Time: time between the Robot clicking on the play button and the video starting to play
- Witbe VQ-MOS Score: a score representing the video quality of the first 60 seconds of video, measured with our famous Video Quality Mean Opinion Score algorithm
All graphs and other visual representations of KPIs in this QoE Snapshot are generated with Witbe Datalab, our restitution interface for fault finding, root cause analysis and advanced analytics.
Global Overview
First of all, because each run was done at a different time of the day, it is important to state that they cannot be compared directly. We can only compare each pair of results, and this QoE Snapshot was designed mostly to analyze the performance of British mobile operators with their associated MVNOs.
The first conclusion that can be drawn, by looking at the Datalab dashboard is that the MVNOs compare quite nicely to their associated carrier.
Indeed, there were no significant difference in performance or quality between each carrier and its matching MVNOs: every time a degradation was reported on an MVNO, it was also observed on its associated carrier. Both carriers and MVNOs have a Video Initial Buffering Time lower than 5s and a reasonable Witbe VQ-MOS Score, averaging at 3.2 out of 5.
As we already observed in our very first QoE Snapshot, customers in the United Kingdom appear to be benefiting from an overall satisfactory video viewing experience on mobile devices.
Performance during peak time
Run #4 (Three and ID Mobile) was performed at around 6pm, during peak hour, which resulted in a very congested network with lower median KPIs than previous runs. The chart above displays the Video Initial Buffering Time KPI for each run and shows a noticeable degradation on the 4th run.
Because the video on-the-go robot is used for drive tests and records its location as it makes its measurements, we are able to display the same KPI on a map of London. This allows us to identify the geographical areas were the performance was the most affected by peak hour usage (Finsbury Park and Crouch End).
We can also create a heatmap for video quality KPIs: the Witbe VQ-MOS score and the duration of Blurriness events which usually represent upscaling artefacts.
Witbe VQ-MOS is a ten-year R&D effort in psycho-acoustic and psycho-visual analysis of video streams. It works without referential, or previous knowledge of the video, and in real-time. It is based on three main artifacts: jerkiness, blurriness and blockiness.
Because all tests were using the same video content, it is another opportunity for us to play our favorite game of “spot the difference” and compare the video quality of the same frame on two different viewing sessions.
Influence of network type & signal strength
When looking at the influence of network type & signal strength on Video Initial Buffering Time & Witbe VQ-MOS scores, an interesting thing we noticed is that a 4G connection with a poor signal (1 or 2 bars) outperformed on average a 3G connection with a good signal (3 or 4 bars).
By looking at the performance per signal strength for viewing sessions in 4G, it appears that signal strength itself did not really affect the quality of experience. Both Video Initial Buffering Time & Witbe VQ-MOS scores were not correlated to signal strength (i.e. number of bars).
This can be explained by the fact that nowadays, mobile video applications are built to deliver the best service possible even in poor connectivity conditions. This can explain why the quality of experience was roughly the same with a 1-bar signal strength and a 4-bar signal strength. This shows that using only QoS tools will not inform on the true QoE delivered to end users and is not enough for operators and MVNOs to provide the best possible service to their end-users.
The Blind Spot
It is interesting to note that our App availability measurements revealed a blind spot.
In a specific geographical area, the BBC iPlayer app couldn’t load properly, during peak or off-peak time, and with two different carriers. This blind spot would require a closer look to identify the cause of the problem.
We could speculate that it is due to the presence of a park nearby, reminding us that environmental factors (trees, tunnels, etc) can also influence the quality of experience perceived by end users.
Conclusion
To conclude, it was interesting to measure no major difference between a mobile carrier and its associated MVNO, in terms of quality of experience.
This QoE Snapshot also shows, once again, a good overall quality of experience when watching videos on mobile devices in the UK, highlighting the continuous efforts and massive investments made by mobile operators to accommodate new user behaviors.
For more details about the results of this QoE Snapshot, don't hesitate to contact us!
See you soon for the next Witbe QoE Snapshot!
About Witbe QoE Snapshots
In the same way that a consumer report tests a product and publishes an analysis of its overall quality, the Witbe QoE Snapshots test digital services to make available to the market information on the true Quality of Experience delivered internationally. These QoE Snapshots should not serve as benchmarks nor as rankings of operators by service or by device. Rather, the goal of these QoE Snapshots is to provide a global overview of digital services with multiple configurations, and in various environments. The public will thus be able to better understand the technological complexity inherent to today’s services, like the distribution of video content. It is quite a technical feat – considering the efforts and means implemented – to broadcast videos on different devices and networks, with a quality that is acceptable by consumers with high expectations.
Since its origin, Witbe relies on a non-intrusive technology, based on Robots measuring the quality truly delivered. The Witbe Robots are placed at the edge of distribution, and connected to test devices, the same ones as those used by real users. The Robots measure the Quality of Experience actually delivered to the end-users by providing KPIs on the availability, performance and integrity of the service.
Each snapshot is composed of several analytical frames, highlighting interesting findings about the KPIs that were measured. In our last QoE Snapshot, we looked at what was the best way to watch Super Bowl LIII. This time, we are publishing a look at the quality of British mobile carriers and their associated MVNOs.