RAW Sensor Digital Twin
A neural model that predicts the exact RAW color filter array (CFA) data a given camera sensor would record, before demosaicing, from an input image, the capture conditions, and the sensor and lens parameters.
- Credit
- 7 points
- Team
- Up to 3 students
- Partners
- HIT and Samsung
- Area
- Computational imaging, deep learning
A research project for an industry customer. There is no ready recipe: the team defines the details of the method, reads the literature, tries ideas, measures, and learns from what does not work. The project requires independent work, proactive exploration and a lot of trial and error, with regular meetings with the supervisor and periodic meetings with the Samsung team.
Description
A camera sensor does not record a color image. Each pixel sits under one color filter (red, green or blue, in a Bayer pattern) and counts photons, adding noise from the physics of light and from the electronics. The camera's image signal processor (ISP) later turns this RAW mosaic into the photo we see.
Students build a digital twin of this first stage: a model that answers "what would this sensor have recorded?" for a scene, under given settings, before any processing.
Input: an image of the scene (sRGB or linear), plus
• Capture conditions: exposure time, ISO (analog gain), f-number, focal length, illuminant and color temperature, scene brightness.
• Sensor parameters: CFA pattern, pixel pitch, spectral sensitivity, full-well capacity, read noise, dark current, black level, bit depth.
• Optics parameters: lens blur (PSF or MTF), vignetting, chromatic aberration.
Output: RAW CFA data: a single-channel Bayer mosaic with realistic signal, clipping, quantization and noise, matching what the real sensor produces. Each pixel holds one number (typically 10 to 14 bits), the light measured through its own red, green or blue filter; the other two colors at that pixel do not exist yet and are only filled in later by demosaicing.
The project combines two worlds: a physics model of the camera (light falling off at the edges, the response of each color channel, shot and read noise, clipping) and a learned model that corrects what the physics misses. The physics part keeps the output plausible for unseen settings; the learned part makes it match real data.
Motivation
- Training data for camera AI. Denoising, demosaicing and low-light models need large amounts of RAW data from a specific sensor. Capturing it is slow and expensive; a twin can generate it from ordinary photos.
- Designing the next sensor. Engineers can ask "how would images look with smaller pixels, more read noise, or a different color filter?" before the silicon exists.
- Testing the ISP. A twin lets the full camera pipeline be tested on thousands of virtual captures with controlled conditions.
- Current research. Models that convert RGB back to RAW using camera settings, such as ParamISP (CVPR 2024, with Samsung co-authors), show the field is active and has open questions.
Goals and deliverables
- A physics-based simulator in PyTorch: inverse tone and color processing, lens effects, spectral to CFA mapping, signal-dependent noise, clipping and quantization, all controlled by parameters.
- Sensor calibration: noise parameters (per ISO) estimated from public data or from flat-field and dark frames, following the EMVA 1288 method.
- A learned twin: a network conditioned on the capture and sensor parameters that refines the simulator output (or replaces part of it) to match real RAW captures.
- An evaluation on held-out real RAW images: pixel accuracy, noise statistics, and a downstream test (a denoiser trained on twin data vs. real data).
- Optional: a small "what-if" tool that shows how one scene changes as sensor parameters change.
Suggested approach
- Start from Unprocessing (Brooks et al.): invert tone mapping, gamma, color correction and white balance to get linear RGB, then mosaic it.
- Add physics: lens vignetting and blur, exposure scaling from exposure time, ISO and f-number, then a Poisson-Gaussian noise model (shot noise plus read noise), and later the richer ELD model (row noise, quantization, dark current).
- Condition a network (U-Net with FiLM conditioning or a small transformer) on the parameter vector, as ParamISP does, to learn the residual between simulated and real RAW.
- Learn the noise with a generative model (normalizing flow as in Noise Flow, or diffusion as in RAW-Diffusion and NoiseDiff) so that the noise distribution matches, not just its mean.
- Train and test across ISO and exposure so the twin works at settings that were not in the training set.
What students will need
A map of the concepts, methods and tools this project touches. The first weeks are for learning the essentials.
Concepts
- Image formation: photons, quantum efficiency, exposure
- CMOS sensor operation, Bayer CFA, quad Bayer
- The camera ISP: black level, white balance, demosaicing, color matrix, tone curve, gamma
- Linear vs. display color spaces, spectral sensitivity
- Noise sources: shot, read, dark current, row noise, quantization
- Optics: PSF, MTF, vignetting, chromatic aberration
- EXIF and DNG metadata
Algorithms and techniques
- Inverse ISP (unprocessing)
- Poisson-Gaussian and physics-based noise models
- Noise calibration: photon transfer curve, noise level function
- Conditional networks (FiLM, parameter embeddings)
- Generative modeling of noise: normalizing flows, diffusion
- Differentiable simulation, hybrid physics + learning
- Image alignment for paired data
Models
Evaluation
- PSNR and SSIM in the RAW domain
- KL divergence of noise histograms per ISO and channel
- Noise level function and photon transfer curve match
- Downstream test: denoiser trained on twin vs. real data
Tools and frameworks
- Python, PyTorch, NumPy
- rawpy (LibRaw) for reading RAW and DNG
- colour-science, colour-demosaicing
- kornia
rgb_to_raw/raw_to_rgb - simple-camera-pipeline (SIDD rendering)
- ISETCam (MATLAB) as a reference simulator
- exiftool, Git, Weights & Biases
Datasets
Paired RAW and RGB with capture metadata
| Dataset | Sensors | Content | Use here |
|---|---|---|---|
| SIDD | 5 phones incl. Samsung Galaxy S6 Edge | Noisy and clean RAW + sRGB, noise level functions, metadata; ~24k images in the full set MIT | Main smartphone set; noise calibration |
| S7-ISP (DeepISP) | Samsung Galaxy S7 | 110 scenes, RAW + JPEG, normal and low-light | Samsung RAW to RGB pairs |
| Zurich RAW-to-RGB | Huawei P20 + Canon 5D IV | ~10k aligned patch pairs CC BY-NC-SA | Phone RAW with high-quality RGB |
| SID | Sony a7S II, Fujifilm X-T2 | 5,094 short-exposure RAW + long-exposure references | Low-light, exposure scaling |
| ELD | 4 cameras | Low-light RAW at several ISO, released noise parameters | Physics noise calibration |
| Cao 2023 (LLD) | - | Noisy/clean pairs + flat-field and bias frames over a wide ISO range | ISO-dependent noise modeling |
| MIT-Adobe FiveK | Several DSLRs | 5,000 DNG RAW + expert retouches | Large RAW pool with EXIF |
| RAISE | 3 Nikon cameras | 8,156 uncompressed RAW + metadata | Varied scenes and settings |
| Raw-to-raw | Galaxy S9, iPhone X | Paired and unpaired RAW between two phones | Mapping between sensors |
| Fujifilm UltraISP, MAI 2021 | Phone sensors (incl. Sony IMX586 quad Bayer) | RAW with reference RGB | Learned ISP challenge data |
Color and spectral data
| Dataset | Content | Use here |
|---|---|---|
| Camera spectral sensitivity database (Jiang 2013) | 28 cameras incl. mobile, 400 to 720 nm | Spectral to CFA response |
| NUS 8-camera | 1,736 linear RAW images with illuminant ground truth | Illuminant conditioning, cross-camera color |
| Cube+, Gehler-Shi | RAW with measured illuminant | White balance modeling |
Challenges with ready data and code
- AIM 2022 Reverse ISP (Samsung S7 and Huawei P20 Pro tracks).
- NTIRE 2025 RAW reconstruction from smartphone RGB, NTIRE 2025 RAW restoration, NTIRE 2024 RAW super-resolution.
- AIM 2025 real-world RAW denoising (trained on synthetic noise).
- Organizer code: mv-lab/AISP.
Samsung data: no public dataset of ISOCELL sensor RAW captures exists. Calibration captures from a Samsung sensor, if Samsung can provide them, would make the twin far more valuable.
Relevant papers
Reverse ISP: from RGB back to RAW
- Brooks et al. Unprocessing Images for Learned Raw Denoising. CVPR 2019. The standard starting point.
- Zamir et al. CycleISP: Real Image Restoration via Improved Data Synthesis. CVPR 2020.
- Xing, Qian, Chen. Invertible Image Signal Processing. CVPR 2021.
- Conde et al. Model-Based Image Signal Processors via Learnable Dictionaries. AAAI 2022.
- Nam, Punnappurath, Brubaker, Brown. Learning sRGB-to-Raw-RGB De-rendering with Content-Aware Metadata. CVPR 2022. York University and Samsung AI Center Toronto.
- Kim et al. ParamISP: Learned Forward and Inverse ISPs using Camera Parameters. CVPR 2024. Closest prior work; conditions on exposure, ISO, aperture, focal length.
- Reinders et al. RAW-Diffusion: RGB-Guided Diffusion Models for High-Fidelity RAW Image Generation. WACV 2025.
- Yu et al. ReconfigISP. ICCV 2021. Differentiable ISP modules.
Sensor noise modeling
- Foi et al. Practical Poissonian-Gaussian Noise Modeling and Fitting for Single-Image Raw-Data. IEEE TIP 2008.
- Abdelhamed, Lin, Brown. A High-Quality Denoising Dataset for Smartphone Cameras (SIDD). CVPR 2018.
- Abdelhamed et al. Noise Flow: Noise Modeling with Conditional Normalizing Flows. ICCV 2019.
- Wei et al. A Physics-based Noise Formation Model for Extreme Low-light Raw Denoising (ELD). CVPR 2020; TPAMI.
- Zhang et al. Rethinking Noise Synthesis and Modeling in Raw Denoising. ICCV 2021.
- Monakhova et al. Dancing under the Stars: Video Denoising in Starlight. CVPR 2022.
- Feng et al. Learnability Enhancement for Low-light Raw Denoising (PMN). ACM MM 2022.
- Cao et al. Physics-Guided ISO-Dependent Sensor Noise Modeling for Extreme Low-Light Photography. CVPR 2023.
- Zhang et al. Towards General Low-Light Raw Noise Synthesis and Modeling (LRD). ICCV 2023.
- Jin et al. Lighting Every Darkness in Two Pairs (LED). ICCV 2023.
- Lu, Achddou, Süsstrunk. Dark Noise Diffusion (NoiseDiff). 2025.
- Li, Jiang, Iso. Noise Modeling in One Hour. arXiv 2025.
Simulators, optics and standards
- ISETCam (Wandell, Farrell et al., Stanford). Full camera simulator in MATLAB; a reference to compare against.
- Sitzmann et al. End-to-end Optimization of Optics and Image Processing (Deep Optics). SIGGRAPH 2018.
- Chen et al. Extreme-Quality Computational Imaging via Degradation Framework. ICCV 2021. Spatially varying lens PSFs of mobile cameras.
- Schwartz, Giryes, Bronstein. DeepISP: Toward Learning an End-to-End Image Processing Pipeline. IEEE TIP 2019.
- EMVA 1288 Release 4.0. Standard for measuring sensor parameters (quantum efficiency, gain, dark noise).
Prerequisites
- Required: prior knowledge of deep learning and PyTorch, from a completed course or a significant project (training a network, writing a Dataset and DataLoader, a training loop).
- Required: good Python and NumPy.
- Recommended: a course in image processing or signal processing; basic probability (noise distributions). Interest in physics and cameras helps.