Creation & Assessment of a Video Quality Ruler

Abstract (Swedish)

Image quality has always been a matter of great importance for anyone

working with or consuming images and videos. Images and videos are, in

general, for human consumption. Thus, image quality is essentially a

subjective attribute of an image. Though there exist both objective and

subjective approaches to assessing image quality, objective results may

not correlate well with subjective results. On the other hand, pre-existing

subjective methods for evaluating video quality can be time-consuming,

resource-consuming, or unreliable.

The aim of this thesis was to create and analyse a method for sub-
jectively determining video quality. Based on a previously established

method for assessing still image quality prescribed in ISO 20462, this

method, the Video Quality Ruler, establishes an absolute scale of per-
ceptive video quality. The Video Quality Ruler consists of a series of 31

ordered video clips, varying solely in sharpness. Each of the 31 video clips

are one perceptual unit apart. Users are able to determine the overall level

of quality of a video through comparison against the Ruler. The method

provides an easy, universal analysis of video quality that correlates well

to human opinions, and is not as time- nor resource- consuming as many

other subjective methods.

The creation of the Video Quality Ruler was, on the whole, successful.

The calibration of the ruler videos differs slightly from the calibration of

the Image Quality Ruler from ISO 20462, particularly for images on the

blurrier end of the Ruler. Assessment of the Video Quality Ruler deter-
mined that a few levels of the Ruler may need to be adjusted, and that

the software and laboratory used for the experiment should be improved

slightly, to eliminate bias and reduce the variance of results.

Popular Abstract

Image quality has always been a matter of great importance for anyone working
with or consuming images and videos. There exist both objective and subjective
approaches to assessing image quality. Example metrics of the former include
PSNR and SSIM, both of which mathematically compare pixels of the original
image with the equivalent pixels of the distorted or changed image. Objective
methods are fast and easy to implement, but may not always be well correlated
with human perception of image quality. Images and videos are, in general, for
human consumption. Thus, image quality is essentially a subjective attribute
of an image.
Pre-existing subjective methods for evaluating image quality can be time- and
resource-consuming, or unreliable. Examples of such methods include the Mean
Opinion Score (MOS) test, whereby participants are asked to rate an image
along a scale, which may be numerical or descriptive. The results of all participants
are averaged to obtain the mean opinion score, and thus the quality level
of the image. A problem with the MOS test is the lack of frame of reference,
resulting in ambiguous and unreliable scores. A subjective method which solves
this particular issue is the Paired Comparison test, in which participants select
the preferred image out of a pair of images. However, this method can only
provide information about relative quality, and not absolute quality.
A subjective method that eliminates the problems of both of these methods is
the Image Quality Ruler (IQR) method, outlined in ISO 20462. An IQR is a
series of 31 images, varying only sharpness. Each image is separated in one perceptual
unit of quality, where one perceptual unit is defined as 75% agreement
amongst participants of the preferred image, between a pair of images. The
unit is known as a just noticable difference, or JND. Sharpness is chosen as the
varying attribute, as it is easily changed using image processing, has low scene
and observer variability, and a strong influence on image quality . Sharpness can
be measured by a function known as the Modulation Transfer Function (MTF).
The MTF of the images in the IQR mimic the MTF of a diffraction-limited
lens.
The aim of this thesis was to develop the IQR method for video purposes,
thus establishing a Video Quality Ruler (VQR). The method of creating a wellcalibrated
VQR followed roughly four steps. Firstly, capture an adequate scene
to be used for the video clips of the VQR. A 12 second scene containing a variety
of information, including text, people, and buildings was recorded. Secondly,
the sharpness of the video clip was altered to 62 different levels, spanning the
1
same range of sharpness of the IQR. The MTFs of the 62 distorted video clips
mimicked MTFs of a diffraction-limited lens, as in the IQR. The third step
was to implement a Paired Comparison test to determine which of the video
clips were one JND apart. All possible permutations of video clips that were
one JND apart were found. The weighted average of the permutations was computed,
where the weights were dependent on the empirical percentage agreement
amongst participants in each pair of videos. The fourth and final step was to
implement the VQR to validate the results found in the Paired Comparison test
in the previous step, and to analyse the VQR methodology itself.
The calibration of the VQR differed slightly from that of the IQR, particularly
on the blurrier end of the spectrum. Here, it was determined that smaller
changes in objective sharpness were needed to create the same perceived change
in quality.
The validation experiment found that the levels of the VQR were calibrated
accurately, with the exception of a few clips which are in need of adjustment.
Implementation of the VQR also determined parts of the methodology that
should be adjusted for reliability and accuracy. These include randomisation of
the initial positioning of the ruler to avoid bias and changing of the non-uniform
computer screen used to display the video clips. The VQR does not cover the
same range of quality as the IQR, and thus it is advised to extend the range of
the former to match that of the latter.

• author: Shah, Maya
• supervisor: Johan Lindström
• organization: Mathematical Statistics
• course: MASM01 20161
• year: 2016
• type: H2 - Master's Degree (Two Years)
• subject: Mathematics and Statistics
• language: English
• id: 8886025
• date added to LUP: 2016-06-30 15:50:41
• date last changed: 2016-06-30 16:04:51