Self-Driving Camera Lab
Software that remotely controls a camera test lab (lights, motorized stages, focus motors and the camera under test), finds the best setup on its own, and searches the space of test conditions for the cases where the camera performs worst.
- Credit
- 7 points
- Team
- Up to 3 students
- Partners
- HIT and Samsung
- On-site work
- Required, at the lab
A research project for an industry customer. There is no ready recipe: the team defines the details of the method, reads the literature, tries ideas, measures, and learns from what does not work. The project requires independent work, proactive exploration and a lot of trial and error, with regular meetings with the supervisor and periodic meetings with the Samsung team.
On-site visits required: the equipment is in a test lab on site. Most development is done remotely against simulated (mock) devices, but integration and testing with the real hardware require regular visits to the lab. Students should plan for scheduled lab days during both semesters, with more of them in the integration phase.
Description
Camera image quality is measured in labs with test charts, calibrated lights and precise positioning. Today, much of this setup is manual: moving the camera, aligning it to the chart, setting the light, focusing, and capturing. This project turns the lab into a programmable, self-driving system in three steps.
(a) Unified API. One software interface for every device in the lab: light sources (intensity, color temperature, flicker), motorized positioning (linear and rotation stages, pan-tilt, robot arm), focus motors, and the camera or phone under test. Every capture is logged together with the full state of all devices.
(b) Automatic best setup. Algorithms that find the right settings on their own: align the camera to the chart (position and angle), make the lighting uniform, and find the best focus.
(c) Smart exploration and failure hunting. A strategy that scans the space of configurations with as few captures as possible, first to get a representative sample of camera behavior, then to actively hunt for configurations where the camera under-performs. Both static conditions and dynamic ones (moving targets, camera shake, changing or flickering light) are covered.
Tasks to automate
The lab team has defined the first tasks. Each action must be fitted to the specific chart being shot, so every chart gets a definition (its markers, regions of interest and target pose) that the routines read.
| Task | What the software must do | Main technique |
|---|---|---|
| Sensor arm alignment | Move the sensor so it faces the chart at the right distance and orientation (height, pitch, roll, yaw) | Fiducial marker detection, pose estimation (PnP), closed-loop correction |
| Image focus | Find the sharpest focus position for the chart and distance | Focus measure on chart regions, coarse-to-fine search |
| Light source adjustment | Set intensity and color temperature; check that the chart is evenly lit | Measure gray patches across the chart, adjust lights |
| Exposure and white balance | Choose exposure time and gain so the chart is well exposed without clipping, and estimate white balance from neutral patches | RAW histogram and clipping checks, gray-patch statistics |
| Chart movement | Bring the requested chart in front of the sensor | Motorized chart track, position check with markers |
Main challenges
- Computer vision on RAW CFA data. The sensor delivers a RAW Bayer mosaic, not a finished color image. Marker detection, focus and exposure measurements have to work directly on this data: subtract the black level, split the mosaic into its color planes (or use the green plane at half resolution), and measure there, without running a full camera pipeline.
- Controlling hardware through multiple APIs. Motion stages, lights, power switches and the sensor capture software all come with different interfaces. The project wraps them in one consistent API.
The lab and its hardware
Based on photos of the lab, the setup includes the equipment below. The interfaces listed are our best guess from the visible hardware and will be confirmed during the first lab visit.
| Equipment | What it does | Likely interface and API |
|---|---|---|
| Long linear stage (track on the table) | Moves the sensor arm toward and away from the charts (distance), controlled from the lab computer | Zaber motorized stage; Zaber Motion Library (Python), Zaber ASCII protocol over USB/serial |
| Zaber sensor mount: vertical lift plus rotation stages | Height, pitch, roll and yaw of the sensor | Zaber stages on the same controller chain; same library. Some pitch/roll stages look manual (scales and knobs) and may need motorizing |
| Sensor board on the mount | The camera sensor under test, delivering RAW CFA frames | Vendor capture software or SDK; the project wraps it with a capture service (command line or library call) |
| Chart wall on sliding tracks | Several large test charts that can slide sideways in an aluminum profile frame | Currently manual; can be motorized with a belt or linear drive (a Zaber stage, or a stepper motor with an Arduino / GRBL controller) |
| Test charts with fiducial borders | Multi-target charts (Siemens stars, slanted edges, color patches, texture) surrounded by square markers | Passive; the markers make automatic alignment possible with OpenCV ArUco-style detection |
| Ceiling spotlights (two, NANLITE) | Main lighting for the charts | Professional LED fixtures usually support DMX512; control from Python through a USB-DMX interface (for example ENTTEC DMX USB Pro with DMXEnttecPro or OLA), or the maker's wireless app protocol |
| Standing light panels (light boxes on wheels) | Diffuse, uniform illumination or backlit targets | Depends on model: DMX, USB/serial, or manual dimmer; to be checked on site |
| Ceiling LED panels | Room light, a source of stray light | Building control; at least on/off should be scripted or switched off for tests |
| Web power switch (remotely switched outlets) | Turns equipment on and off, resets devices remotely | Digital Loggers Web Power Switch, HTTP/REST API; Python dlipower |
| ATEN unit on the rig | Probably a USB or KVM extender carrying signals from the rig to the lab computer | Transparent to software, but affects cabling and latency |
| Small camera above the chart wall; ceiling dome camera | Overview of the lab | Can serve as a remote "eye" for safe operation (check that the path is clear before moving) |
Safety: remote motion needs software limits for every axis, a collision check between the arm and the chart wall, and an emergency stop that works from the remote interface.
Motivation
- Speed and repeatability. Automated setups run day and night and repeat a test exactly, which a manual setup cannot.
- Better test coverage. The number of combinations of light, distance, angle, motion and camera settings is huge. A grid search is far too slow; smart sampling covers the space with a fraction of the captures.
- Finding weak spots early. Actively searching for the conditions where image quality drops finds problems before users do, and points engineers to what to fix.
- Data for other projects. The lab produces labeled test data for models such as the patch quality scanner (Project 1) and the RAW sensor twin (Project 2).
Goals and deliverables
- A device layer with a common interface (connect, set, read, move, capture), drivers for each lab device, and a mock version of each device for development without hardware.
- A lab service: a documented REST or gRPC API, an event bus for device status, and a database of every capture with its full configuration.
- Auto-setup routines: chart alignment with fiducial markers, autofocus search, and lighting uniformity optimization.
- An exploration engine: space-filling sampling, Bayesian optimization, and failure search, with a report that maps where the camera performs well and badly.
- Dynamic tests: synchronized runs with motion and changing light.
- A demonstration in the lab showing a full unattended test campaign.
Suggested approach
- Learn from Android Camera ITS. Google's Camera Image Test Suite already drives phones over ADB with Python, shows charts on a tablet, rotates phones with an Arduino-controlled rig, and has an automated "ITS-in-a-box" setup. It is a good template for the API and test flow.
- Hardware abstraction. Follow the device model of Bluesky and ophyd (or ophyd-async): every device exposes readable and settable signals, and a run engine records every step.
- Simulate first. Build a simple simulator of the rig (render a chart under a given pose, light and focus) so that the search algorithms can be developed and tested before the lab visits.
- Measure image quality. Use slanted-edge sharpness (MTF Mapper), noise, distortion and color metrics as the scores the search optimizes.
- Search in stages: Sobol or Latin hypercube sampling for coverage; Bayesian optimization (BoTorch / Ax) for best settings; level-set estimation to map the boundary between good and bad quality; quality-diversity search (MAP-Elites with pyribs) to collect many different failure cases.
What students will need
A map of the concepts, methods and tools this project touches. The first weeks are for learning the essentials.
Concepts
- Camera image quality metrics: sharpness (SFR / MTF), noise, distortion, color accuracy, flare
- RAW sensor data: Bayer CFA, black level, color planes, clipping
- Exposure, gain and white balance
- Camera calibration, camera pose, homography
- Light: color temperature, illuminance, PWM flicker
- Motion control: stepper and servo motors, stages, kinematics
- Rolling shutter, motion blur, image stabilization
- Design of experiments, search spaces
Algorithms and techniques
- Fiducial markers (ArUco, ChArUco, AprilTag) and pose estimation (PnP)
- Visual servoing for alignment
- Computer vision directly on RAW CFA frames
- Auto exposure from histograms; gray-world and gray-patch white balance
- Autofocus: focus measures, hill climbing, golden-section search
- Quasi-random sampling: Sobol, Latin hypercube
- Gaussian processes and Bayesian optimization
- Active learning, level-set estimation
- Falsification and adversarial testing, quality-diversity (MAP-Elites)
Software engineering
- Hardware abstraction layers, device drivers
- REST and RPC API design, OpenAPI
- Asynchronous programming, event buses
- Mock devices and automated testing
- Logging, data provenance, experiment databases
Evaluation
- Alignment repeatability (reprojection error, angle)
- Autofocus accuracy and number of steps
- Share of a campaign run unattended
- Coverage vs. number of captures
- Failures found per capture budget vs. random sampling
Tools and frameworks
- Python, asyncio, FastAPI or gRPC, MQTT or ZeroMQ
- Bluesky, ophyd-async, PyVISA, PyMeasure
- ADB, Camera2, scrcpy, gphoto2, Harvester (GenICam)
- Zaber Motion Library (the lab's stages); other SDKs: Thorlabs Kinesis, PI pipython
- DMX lighting control (USB-DMX, OLA); web power switch API
- rawpy, NumPy for RAW frames
- Arduino, GRBL for low-cost motion; ROS 2 and MoveIt 2 for robot arms
- OpenCV, MTF Mapper
- SciPy (qmc), BoTorch, Ax, Optuna, pyribs, VerifAI
Reference test systems
Existing automated test suites, characterization standards and commercial systems show how industry camera labs are built.
| System | What it covers |
|---|---|
| IEEE 1858 (CPIQ) | Camera phone image quality metrics |
| IEEE 2020 | Automotive camera quality, including LED flicker |
| EMVA 1288 | Sensor and camera characterization |
| Android Camera ITS | Automated camera test suite with open rig designs (ITS-in-a-box) |
| Image Engineering LE7 | Programmable uniform light source with API |
| Image Engineering STEVE-6D | 6-axis shake robot for stabilization tests |
| Imatest, DXOMARK | Commercial image quality analysis and test protocols |
Useful Software and Libraries
| Tool | Role in the project |
|---|---|
| Bluesky, ophyd-async | Experiment orchestration and hardware abstraction, from large science facilities |
| bluesky-adaptive | Adaptive scans driven by a search agent redesign announced |
| pycro-manager | Python control of cameras, stages and lights via Micro-Manager |
| PyVISA, PyMeasure, pylablib | Instrument drivers |
| ADB, Camera2, scrcpy | Remote control of the phone under test |
| gphoto2, Harvester, Aravis | Control of DSLRs and machine vision cameras |
| Thorlabs Kinesis, Zaber Motion Library, PI pipython | Motorized stage SDKs |
| GRBL, Arduino | Low-cost motor control |
| ros2_control, MoveIt 2 | Robot arm control and motion planning |
| Open Lighting Architecture | DMX control of studio lights |
| FastAPI, gRPC, MQTT, ZeroMQ | Lab service API and messaging |
| OpenCV ArUco / ChArUco, ViSP | Pose estimation and visual servoing for alignment |
| MTF Mapper | Open-source slanted-edge sharpness measurement |
| scipy.stats.qmc | Sobol and Latin hypercube sampling |
| BoTorch, Ax, Optuna | Bayesian optimization |
| pyribs | Quality-diversity search (MAP-Elites) for diverse failure cases |
| VerifAI + Scenic | Falsification toolkit for systems that use perception |
Avoid: GPyOpt (archived in 2023). Consumer smart bulbs are not repeatable enough for measurement lighting.
Relevant papers
Self-driving labs and adaptive experiments
- Häse, Roch, Aspuru-Guzik. Next-Generation Experimentation with Self-Driving Laboratories. Trends in Chemistry 2019.
- Abolhasani, Kumacheva. The Rise of Self-Driving Labs in Chemical and Materials Sciences. Nature Synthesis 2023.
- Pinkard et al. Pycro-Manager: open-source software for customized and reproducible microscope control. Nature Methods 2021.
Testing and failure search
- Dreossi et al. VerifAI: A Toolkit for the Formal Design and Analysis of Artificial Intelligence-Based Systems. CAV 2019.
- Abdessalem, Nejati, Briand, Stifter. Testing Vision-Based Control Systems Using Learnable Evolutionary Algorithms. ICSE 2018.
- Gotovos et al. Active Learning for Level Set Estimation. IJCAI 2013. Finds the boundary between good and bad regions.
- Tian et al. DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars. ICSE 2018.
- Pei et al. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. SOSP 2017.
- Mouret, Clune. Illuminating Search Spaces by Mapping Elites (MAP-Elites). 2015.
- Baek et al. Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains (ImageNet-ES). CVPR 2024. About 202k photos taken while physically varying light and camera settings; very close in spirit to this project.
Optimization, sampling and focus
- Balandat et al. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. NeurIPS 2020.
- Shahriari et al. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proceedings of the IEEE 2016.
- Pertuz, Puig, García. Analysis of Focus Measure Operators for Shape-from-Focus. Pattern Recognition 2013.
- Garrido-Jurado et al. Automatic Generation and Detection of Highly Reliable Fiducial Markers under Occlusion (ArUco). Pattern Recognition 2014.
Prerequisites
- Required: prior knowledge of computer vision or image processing, from a completed course or a significant project, and experience with OpenCV and NumPy.
- Required: strong Python and software engineering skills.
- Required: availability for scheduled on-site lab visits.
- Recommended: interest in hardware and hands-on work; a course in optimization or machine learning.