Patch Quality Scanner
A deep learning model that scans a camera image in small patches and gives each patch an overall quality score plus a severity score for five distortion families: geometry, color, light, contrast and focus.
- Credit
- 7 points
- Team
- Up to 3 students
- Partners
- HIT and Samsung
- Area
- Computer vision, image quality
A research project for an industry customer. There is no ready recipe: the team defines the details of the method, reads the literature, tries ideas, measures, and learns from what does not work. The project requires independent work, proactive exploration and a lot of trial and error, with regular meetings with the supervisor and periodic meetings with the Samsung team.
Description
Students build a model that takes an image, cuts it into overlapping patches (for example 128×128 pixels), and returns for every patch a vector of six numbers. The result is a set of quality maps over the image: where it is sharp or blurry, where the color is off, where it is over or under exposed, where contrast is flat, and where the geometry is bent.
Input: an RGB image (smartphone photo, any resolution).
Output per patch: [quality, geometry, color, light, contrast, focus], each in 0 to 1, where 0 means no distortion of that family.
Output per image: six heat maps and an overall score pooled from the patches.
The model is trained and tested on three sources of data:
- Synthetic patches: clean, high quality images degraded with distortions of a known type and strength, so every patch gets an exact label for free.
- Real public patches: public quality datasets with human opinion scores, including patch-level scores (PaQ-2-PiQ) and per-attribute ratings of smartphone photos (SPAQ, KonIQ++).
- Proprietary Samsung dataset: real camera images provided by Samsung, used to adapt and validate the model on the target cameras and distortions.
The five distortion families
| Family | Distortions to synthesize | Example implementations |
|---|---|---|
| Geometry | Barrel and pincushion lens distortion, perspective tilt, local warping, rolling shutter skew | cv2.initUndistortRectifyMap + cv2.remap, cv2.warpPerspective, kornia RandomFisheye, RandomPerspective, albumentations OpticalDistortion |
| Color | White balance shift, color cast, saturation change, color quantization, chromatic aberration | kornia RandomPlanckianJitter, RandomSaturation; ARNIQA color_shift, quantization; Unprocessing random_gains |
| Light | Under and over exposure, uneven illumination, vignetting, lens flare | ARNIQA brighten, darken; kornia RandomLinearCornerIllumination; albumentations Vignetting, LensFlare; Google flare dataset |
| Contrast | Low and high contrast, gamma and tone curve changes, mean shift | kornia RandomGamma; ARNIQA non_linear_contrast_change, mean_shift; torchvision adjust_gamma |
| Focus | Defocus (lens) blur, motion blur, Gaussian blur | ARNIQA lens_blur, motion_blur; imagecorruptions defocus_blur; kornia RandomMotionBlur |
Hard part: geometry is not visible inside a flat, textureless patch. No public dataset labels geometric distortion, so students generate it themselves and should study how much image context the model needs (patch size, neighbor patches, straight line cues) to detect it.
Motivation
Camera teams test new sensors, lenses and image processing pipelines on thousands of photos. A single quality number per image does not say what went wrong or where. A per patch, per distortion map points engineers directly at the problem: soft corners from the lens, a color cast in shadows, clipped highlights in one region. The same model can serve as an automatic judge in regression tests, a filter for training data, and a feedback signal for tuning the camera pipeline.
Recent research is moving in the same direction (models that locate a distortion, name its type and rate its severity), which makes this a current topic with room for a publishable result.
Goals and deliverables
- A distortion synthesis library in Python: one function per distortion, each with a strength parameter mapped to a 0 to 1 severity, able to mix several distortions and apply them locally (to a region) as well as globally.
- A training set generated from clean images, with per patch labels.
- A multi-head model that outputs the six scores per patch, and a script that produces heat maps for a full image.
- An evaluation report: accuracy on synthetic test data, correlation with human scores on public datasets, and results on the proprietary Samsung dataset.
- Optional: an on-device version (ONNX or TFLite) with measured run time on a phone.
Suggested approach
- Backbone: start from a pretrained CNN or vision transformer (ConvNeXt, EfficientNet, ViT-S), or from an IQA pretrained encoder such as ARNIQA or CONTRIQUE.
- Heads: one small regression head per family plus one for overall quality; try adding a distortion type classifier as an auxiliary task (as in LIQE).
- Losses: regression (L1 or MSE) on severity, a ranking loss so that stronger distortion scores higher, and correlation based losses when fitting human scores.
- Context: compare a pure patch model with a fully convolutional or transformer model that sees the whole image and outputs a dense map.
- Calibration on human data: fine-tune the quality head on KADID-10k and PaQ-2-PiQ patch scores; check the light, color, contrast and focus heads against SPAQ and KonIQ attribute ratings.
What students will need
A map of the concepts, methods and tools this project touches. Students are not expected to know all of it at the start; the first weeks are for learning the essentials.
Concepts
- Image formation and the camera pipeline (ISP)
- Color spaces (sRGB, linear RGB, Lab), white balance, gamma
- Lens distortion models (radial, tangential) and homographies
- Blur and the point spread function, convolution
- Full-reference vs. no-reference quality assessment
- Mean opinion score (MOS), subjective testing
Algorithms and techniques
- Synthetic data generation with known labels
- Multi-task learning with several output heads
- Regression, ranking and correlation losses
- Self-supervised and contrastive pretraining
- Transfer learning and fine-tuning
- Sliding window inference, dense prediction, patch pooling
- Data augmentation and domain shift handling
Models
Evaluation
- SRCC and PLCC (rank and linear correlation)
- MAE of predicted severity
- Type classification accuracy, confusion matrix
- Cross-dataset generalization tests
- Run time and model size on device
Tools and frameworks
- Python, PyTorch, torchvision, timm
- OpenCV, kornia, scikit-image, NumPy
- Distortion code: ARNIQA, imagecorruptions, albumentations
- pyiqa (IQA-PyTorch) for baseline metrics
- Weights & Biases or TensorBoard for experiments
- ONNX, TFLite for deployment; Git, GitHub
Datasets
Synthetic distortions with type labels
| Dataset | Content | Labels | Use here |
|---|---|---|---|
| KADID-10k | 81 reference, 10,125 distorted; 25 types × 5 levels | DMOS, distortion type and level | Main human-labeled set; its type groups match four of the five families |
| KADIS-700k | 140k reference, 700k distorted; MATLAB code | Distortion type and parameters (no MOS) | Large scale pretraining for type and severity |
| TID2013 | 25 reference, 3,000 distorted; 24 types × 5 levels | MOS | Contrast, saturation, color quantization, chromatic aberration |
| PIPAL | 250 reference, ~29k distorted; 40 types | Elo scores from pairwise votes | Includes spatial warping and restoration artifacts |
| LIVE IQA, CSIQ | Classic small sets (blur, noise, compression, contrast) | DMOS | Standard baselines |
| Diag-Bench (2026) | ~21k samples; 12 types × 5 levels | Region box + type + severity | Closest match to per-region output; release announced |
Patch-level and smartphone photos
| Dataset | Content | Labels | Use here |
|---|---|---|---|
| PaQ-2-PiQ (LIVE-FB) | ~40k images, ~120k patches | MOS per image and per patch | The only large human patch-score set |
| SPAQ | 11,125 photos from 66 smartphones | MOS + brightness, colorfulness, contrast, noisiness, sharpness + EXIF | Real-world test for light, color, contrast, focus heads |
| KonIQ-10k | 10,073 images | MOS + brightness, colorfulness, contrast, sharpness | Large in-the-wild set with attributes |
| KonIQ++ | Same images | Defect scores: blur, artifacts, color, contrast | Real defect-type labels |
| LIVE Challenge | 1,162 mobile photos | MOS | Standard in-the-wild test set |
| SQAD | 3,017 images from 29 phones | Lab measurements + quality attributes | Smartphone camera benchmarking |
Focus, exposure, and clean source images
| Dataset | Content | Use here |
|---|---|---|
| DPDD | 500 scenes with real defocus and all-in-focus pairs | Real defocus test data |
| CUHK blur detection | 1,000 images with hand-drawn blur masks | Tests whether focus maps find the blurred regions |
| Afifi exposure set | FiveK RAW photos rendered at several exposures | Real exposure errors |
| DIV2K, LSDIR | 1,000 and ~85,000 high-quality images | Clean source images for synthesis |
| Waterloo Exploration | 4,744 clean images | Clean source pool |
| Google flare set | 5,001 flare-only images (CC BY 4.0) | Realistic lens flare overlays |
Libraries for creating distortions
| Library | Families | Status and license |
|---|---|---|
| kornia | All five, differentiable, runs on GPU | Apache-2.0 active |
| ARNIQA distortions | Python port of KADID distortions (focus, color, light, contrast) | Apache-2.0 |
| OpenCV | Geometry: lens distortion, perspective, remap | Apache-2.0 active |
| torchvision v2 | Geometry, color, contrast, blur | BSD active |
| imagecorruptions | ImageNet-C corruptions: blur, brightness, contrast, saturation | Apache-2.0 |
| BSRGAN degradations | Realistic blur, noise and compression chains | Apache-2.0 |
| Unprocessing | Color via a simple camera pipeline model (white balance and color matrix) | research code |
| albumentations / AlbumentationsX | All five, very rich | original archived 2025; successor AGPL or commercial |
| pyiqa | Ready IQA models and metrics for baselines | non-commercial license |
| imgaug | All five | unmaintained since 2020 |
License check: several datasets are "research only" and some libraries have non-commercial or copyleft licenses. Agree with Samsung early which ones may be used.
Relevant papers
Patch-based quality
- Kang, Ye, Li, Doermann. Convolutional Neural Networks for No-Reference Image Quality Assessment. CVPR 2014. First patch-input CNN for IQA.
- Bosse et al. Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment (WaDIQaM). IEEE TIP 2018. Weighted pooling of patch scores.
- Ying et al. From Patches to Pictures (PaQ-2-PiQ). CVPR 2020. Human patch scores and local quality maps.
No-reference quality models
- Su et al. HyperIQA: Blindly Assess Image Quality in the Wild. CVPR 2020.
- Zhang et al. DBCNN: Blind IQA Using a Deep Bilinear CNN. Pretrained on distortion type and level.
- Ke et al. MUSIQ: Multi-scale Image Quality Transformer. ICCV 2021.
- Yang et al. MANIQA. Multi-dimension attention for IQA.
- Madhusudana et al. CONTRIQUE: Image Quality Assessment using Contrastive Learning. IEEE TIP 2022.
- Saha et al. Re-IQA. CVPR 2023.
- Agnolucci et al. ARNIQA: Learning Distortion Manifold for Image Quality Assessment. WACV 2024. Reusable distortion code.
- Chen et al. TOPIQ: A Top-down Approach from Semantics to Distortions.
- Zhang et al. LIQE: Blind IQA via Vision-Language Correspondence, a Multitask Approach. CVPR 2023. Joint quality, scene and distortion type.
- Wang et al. CLIP-IQA: Exploring CLIP for Assessing the Look and Feel of Images. AAAI 2023.
- Wu et al. Q-Align. ICML 2024. Large multimodal model as quality judge.
- You et al. DepictQA: Depicting Beyond Scores. ECCV 2024.
- Li et al. Q-Insight. NeurIPS 2025. Reinforcement learning with score and degradation labels.
Distortion type and severity
- Moorthy, Bovik. DIIVINE: Blind Image Quality Assessment, From Natural Scene Statistics to Perceptual Quality. IEEE TIP 2011. Identify the distortion first, then score it.
- Zhai et al. PrISM-IQA. arXiv 2026. Severity level for 53 ISP-related smartphone issues.
- Zhang et al. From Glance to Scrutiny (GS-IQA). arXiv 2026. Locate, classify and rate each distortion; introduces Diag-Bench.
- Nguyen. A Distortion Aware Image Quality Assessment Model. WACV Workshops 2025.
- Kim et al. pMOS: per-pixel quality prediction. arXiv 2022.
Datasets and robustness
- Lin, Hosu, Saupe. KADID-10k: A Large-scale Artificially Distorted IQA Database. QoMEX 2019.
- Fang et al. Perceptual Quality Assessment of Smartphone Photography (SPAQ). CVPR 2020.
- Hendrycks, Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (ImageNet-C). ICLR 2019.
- Brooks et al. Unprocessing Images for Learned Raw Denoising. CVPR 2019. Camera-pipeline-aware color distortions.