Image and Video Quality Metrics: A Comprehensive Reference

Description: Most widely used objective metric measuring pixel-wise difference between reference and distorted images.

How it works:

\[\text{PSNR} = 10 \log_{10} \left( \frac{\text{MAX}^2}{\text{MSE}} \right)\]

where MAX is maximum possible pixel value (255 for 8-bit images), MSE is mean squared error.

Limitations: Poor correlation with human perception.

Libraries:

• Python: skimage.metrics.peak_signal_noise_ratio, cv2.PSNR
• MATLAB: Built-in psnr() function

Open Source:

• scikit-image metrics module

Datasets:

• TID2013
• LIVE Image Quality Database

References:

• Wikipedia: Peak signal-to-noise ratio

Description: Perceptual metric considering luminance, contrast, and structure similarities.

How it works:

\[\text{SSIM}(x,y) = \frac{(2\mu_x\mu_y + C_1)(2\sigma_{xy} + C_2)}{(\mu_x^2 + \mu_y^2 + C_1)(\sigma_x^2 + \sigma_y^2 + C_2)}\]

where μ is mean, σ is variance/covariance, C are constants.

Libraries:

• Python: skimage.metrics.structural_similarity, pytorch-msssim
• MATLAB: Built-in ssim() function

Open Source:

• SSIM PyTorch implementation
• Original MATLAB code

Datasets:

• LIVE IQA Database
• CSIQ Database

References:

• Wang, Z., et al. (2004). “Image quality assessment: from error visibility to structural similarity”
• IEEE Xplore paper

Description: Extension of SSIM evaluating structure at multiple scales via downsampling.

How it works: Applies SSIM at multiple resolutions, combines results with weights.

Libraries:

• Python: pytorch-msssim, piq
• MATLAB: Available via File Exchange

Open Source:

• MS-SSIM PyTorch

References:

• Wang, Z., et al. (2003). “Multiscale structural similarity for image quality assessment”

Description: Machine learning-based video quality metric developed by Netflix, fusing multiple elementary metrics.

How it works: Combines VIF, DLM, motion, and temporal features via SVR (Support Vector Regression).

Libraries:

• C: libvmaf (Netflix)
• Python: vmaf (pip installable)
• FFmpeg: Built-in VMAF filter

Open Source:

• Netflix VMAF GitHub

Datasets:

• VMAF Dataset (NFLX-TEST)
• VideoSet

References:

• Li, Z., et al. (2016). “Toward a practical perceptual video quality metric”
• Netflix Tech Blog

Description: Information-theoretic metric quantifying shared information between reference and distorted images.

How it works: Models image as natural scene statistics passing through distortion channel, computes mutual information.

Libraries:

• Python: piq library
• MATLAB: VIF Toolbox

Open Source:

• PyIQA implementation

References:

• Sheikh, H.R., & Bovik, A.C. (2006). “Image information and visual quality”

Description: Low-level feature-based metric using phase congruency and gradient magnitude.

How it works: Extracts phase congruency (PC) and gradient magnitude (GM) as features, computes similarity.

Libraries:

• Python: piq, sewar
• MATLAB: FSIM Code

Open Source:

• FSIM MATLAB code

References:

• Zhang, L., et al. (2011). “FSIM: A feature similarity index for image quality assessment”

Description: Uses wavelet-based entropy features transmitted as side information.

How it works: Extracts entropy of wavelet subbands from reference, compares with distorted version.

Libraries:

• MATLAB: LIVE RR-IQA toolbox

References:

• Soundararajan, R., & Bovik, A.C. (2012). “RRED indices: Reduced reference entropic differencing”

Description: Reduced-reference metric based on spatial and spectral entropies.

How it works: Transmits entropy statistics of DCT blocks and edges as reference features.

References:

• Chandler, D.M., & Hemami, S.S. (2007). “A57 database and VSNR metric”

Description: No-reference metric using natural scene statistics (NSS) features and SVR.

How it works: Extracts locally normalized luminance coefficients, fits to generalized Gaussian distribution, trains SVR on features.

Libraries:

• Python: libsvm, piq, imquality
• MATLAB: BRISQUE Code

Open Source:

• BRISQUE MATLAB
• imquality Python

Datasets:

• LIVE IQA Database
• TID2013

References:

• Mittal, A., et al. (2012). “No-reference image quality assessment in the spatial domain”

Description: Opinion-unaware (no training on human scores) metric based on NSS model.

How it works: Models pristine natural images with multivariate Gaussian (MVG) in NSS feature space, measures distance of test image.

Libraries:

• Python: piq, scikit-image (limited)
• MATLAB: NIQE Code

Open Source:

• NIQE Python implementation

References:

• Mittal, A., et al. (2013). “Making a ‘completely blind’ image quality analyzer”

Description: No-reference metric analyzing blockiness, noise, and spatial activity.

How it works: Divides image into blocks, evaluates distortion using perceptual features (noticeably distorted blocks, noise, spatial activity).

Libraries:

• MATLAB: Built-in piqe() function

References:

• Venkatanath, N., et al. (2015). “Blind image quality evaluation using perception based features”

Description: CNN-based aesthetic and technical quality predictor trained on AVA dataset.

How it works: Fine-tunes pre-trained CNN (e.g., MobileNet, Inception) to predict distribution of human ratings.

Libraries:

• Python: TensorFlow/Keras, PyTorch

Open Source:

• NIMA PyTorch
• TensorFlow implementation

Datasets:

• AVA (Aesthetic Visual Analysis)

References:

• Talebi, H., & Milanfar, P. (2018). “NIMA: Neural image assessment”

Description: Vision Transformer-based no-reference IQA handling arbitrary resolutions and aspect ratios.

How it works: Uses multi-scale image representation fed to Transformer encoder, predicts quality score.

Open Source:

• MUSIQ GitHub

References:

• Ke, J., et al. (2021). “MUSIQ: Multi-scale image quality transformer”

Description: Average absolute pixel-wise difference.

How it works:

\[\text{MAE} = \frac{1}{N} \sum_{i=1}^{N} |x_i - y_i|\]

Libraries:

• Python: numpy.mean(np.abs(x - y))

Limitations: Does not correlate well with perceived quality.

Description: No-reference metric combining Ma’s and NIQE scores to measure perceptual quality.

How it works:

\[\text{PI} = \frac{1}{2} \left( 10 - \text{Ma} + \text{NIQE} \right)\]

References:

• Blau, Y., & Michaeli, T. (2018). “The perception-distortion tradeoff”

Description: Measures image sharpness via Laplacian operator variance.

How it works: Applies Laplacian filter, computes variance (higher = sharper).

Libraries:

• Python: cv2.Laplacian() + numpy.var()

Open Source:

• OpenCV Laplacian sharpness

References:

• Pech-Pacheco, J.L., et al. (2000). “Diatom autofocusing in brightfield microscopy”

Description: Quantifies perceived colorfulness based on opponent color space.

How it works: Computes standard deviation and mean of rg and yb channels in opponent space.

Open Source:

• Colorfulness Python implementation

References:

• Hasler, D., & Suesstrunk, S. (2003). “Measuring colorfulness in natural images”

Description: Measures luminance contrast.

How it works:

Michelson: \(C = \frac{L_{\max} - L_{\min}}{L_{\max} + L_{\min}}\)

RMS: Standard deviation of pixel intensities.

Libraries:

• Python: Custom with numpy

References:

• Peli, E. (1990). “Contrast in complex images”

Description: Detects compression artifacts like blocking and blur.

How it works: Analyzes edge discontinuities (blocking) and high-frequency attenuation (blur).

Open Source:

• Part of BRISQUE, PIQE implementations

References:

• Wang, Z., et al. (2002). “A universal image quality index”

Description: PSNR adapted for equirectangular projection, accounting for latitude-dependent sampling density.

How it works: Weights pixel errors by spherical area (cos of latitude).

Open Source:

• 360Lib (spherical video tools)

References:

• Yu, M., et al. (2015). “A framework to evaluate omnidirectional video coding schemes”

Description: Improved S-PSNR with uniform sampling on sphere via resampling.

How it works: Projects equirectangular to uniform spherical grid, computes PSNR.

Open Source:

• 360Lib GitHub

References:

• Sun, Y., et al. (2017). “Weighted-to-spherically-uniform quality evaluation for omnidirectional video”

Description: Uses Craster projection minimizing area distortion for quality measurement.

How it works: Converts to Craster parabolic projection before computing PSNR.

References:

• Yu, M., et al. (2017). “A framework to evaluate omnidirectional video coding schemes”

Description: Evaluates quality based on user’s viewing direction/viewport.

How it works: Samples viewports according to head movement patterns, computes quality per viewport.

Datasets:

• 360° Video Head Movement Dataset

References:

• Xu, M., et al. (2018). “Predicting head movement in panoramic video”

Description: See “Non-Intrusive Metrics” section above.

Description: Improved NIQE with local quality assessment and integration.

How it works: Computes local NIQE scores in patches, aggregates for global score.

Open Source:

• IL-NIQE MATLAB

References:

• Zhang, L., et al. (2015). “A feature-enriched completely blind image quality evaluator”

Description: Full-reference metric modeling human visual system for HDR images.

How it works: Applies luminance masking, contrast sensitivity, spatial frequency channels.

Libraries:

• MATLAB: HDR-VDP-2 Toolbox

Open Source:

• HDR-VDP-2 SourceForge

Datasets:

• ESPL-LIVE HDR Database

References:

• Mantiuk, R., et al. (2011). “HDR-VDP-2: A calibrated visual metric for visibility and quality predictions”

Description: Extends HDR-VDP to video with temporal modeling.

How it works: Adds temporal contrast sensitivity and motion modeling to HDR-VDP.

References:

• Narwaria, M., et al. (2015). “HDR-VQM: An objective quality measure for high dynamic range video”

Description: Perceptually uniform color space for HDR images.

How it works: Transforms HDR pixel values to perceptually uniform encoding before computing differences.

Open Source:

• PU21 MATLAB/Python

References:

• Mikhailiuk, A., et al. (2021). “A perceptually uniform color space for HDR imaging”

Description: Measures content and style preservation in neural style transfer.

How it works: Computes content loss (feature difference in deep layers) and style loss (Gram matrix difference).

Open Source:

• Neural Style Transfer PyTorch

References:

• Gatys, L.A., et al. (2016). “Image style transfer using convolutional neural networks”

Description: Predicts aesthetic appeal using deep learning trained on AVA dataset.

How it works: CNN extracts features correlating with aesthetic ratings (composition, color harmony, etc.).

Open Source:

• NIMA (see No-Reference section)

Datasets:

• AVA Dataset

References:

• Murray, N., et al. (2012). “AVA: A large-scale database for aesthetic visual analysis”

Description: See “Intrusive Metrics” section above. Widely used for streaming video (Netflix, YouTube).

Description: Extends SSIM to temporal dimension for video.

How it works: Computes SSIM across spatial and temporal patches (3D blocks).

References:

• Wang, Z., et al. (2004). “Video quality assessment based on structural distortion measurement”

Description: ITU standard for broadcast video quality.

How it works: Extracts perceptual features (spatial/temporal activity, edge degradation, chroma spread), combines via linear model.

References:

• ITU-T J.144: Objective perceptual video quality measurement

Description: No-reference metric for videoconferencing and VoIP video.

How it works: Detects packet loss artifacts, blockiness, blurring specific to real-time video transmission.

References:

• Reibman, A.R., et al. (2004). “Quality monitoring of video over a packet network”

Description: Models predicting quality-of-experience considering bitrate switching, stalling, resolution changes.

How it works: Combines video quality metrics with buffering/stalling penalties.

Open Source:

• QoE Models GitHub

References:

• ITU-T P.1203: Parametric bitstream-based quality assessment