Summary
Modern cameras don't end at the lens. After the sensor, image processing takes over—denoising, demosaicing, deblurring, and increasingly, AI-driven enhancement. When a lens is to be paired with one of these pipelines, reaching optimal performance requires the optical design to be optimized with the pipeline in the loop: this is processing-aware optical design.
In practice, this rarely works as well as it should. Optical designers have spent decades polishing Levenberg-Marquardt (LM)—a robust pseudo-second-order solver that underlies commercial lens design packages. But LM needs a vector of residuals to fit against; in contrast, the losses that drive processing-aware design are scalars (like a single PSNR or a single perceptual score). With nothing to least-squares against, LM collapses, and the field falls back on underperforming first-order optimizers like Adam—leaving performance on the table.
Conventional spot-radius optimization, the workhorse of textbook lens design, already solves a version of this problem: it turns a scalar objective (RMS spot size) into a vector of transverse ray aberrations (TRA). We generalize that idea. Any differentiable scalar loss can be lifted into a high-dimensional set of ray-level residuals—generalized transverse ray aberrations (GTRA)—that LM can work with. The result is a single solver that combines the robustness of conventional design with the flexibility of end-to-end pipelines.
Across 100+ design instances spanning smartphone telephoto lenses, microscope objectives, and C-mount cameras, our LM-GTRA optimizer decisively outperforms first-order baselines and yields designs that match or beat their spot-radius-optimized counterparts in image quality or form factor.
Optimization in Action
Our LM optimization engine is robust across complex optical landscapes. The video below shows it generating designs from rudimentary starting points using conventional spot-radius optimization, accounting for glass materials in hybrid lenses with aspheric and diffractive surfaces. This foundational robustness is what makes the engine effective when the objective shifts to the more intricate processing-aware setting demonstrated in the sections that follow.
Generalized Transverse Ray Aberrations
The scalar nature of typical end-to-end losses excludes effective optics optimization with the Levenberg-Marquardt (LM) algorithm. We introduce GTRA as a method to lift any differentiable scalar loss into the intermediate, high-dimensional spot-diagram space, enabling its use with least-squares solvers. The GTRA objective generalizes the transverse ray aberrations (TRA) formulation that underlies industry-standard spot-radius optimization to the processing-aware setting.
End-to-End Image Restoration
We address E2E image restoration by jointly optimizing the lens design and an image restoration model (IRM), minimizing the difference between the original image and the restored image. Spot diagrams serve as the intermediate representation that links the lens variables to the downstream image simulation and restoration processes—and, crucially, satisfy the conditions for GTRA.
A Toy Problem: Conventional vs. Task-Driven Design
Before turning to real lenses, the two-element setup below isolates what changes when the objective shifts from conventional (minimize the spot radius via TRA) to task-driven (minimize the MSE between the restored and the original image via GTRA). The input is a concentric-circle target—a pattern whose dominant orientation we expect the task-driven design to exploit by learning anisotropic PSFs aligned with the circle gradients. That is exactly what happens, and the restored image gains +4.8 dB over the conventional design.
The convergence behavior also splits sharply. On the conventional objective, our LM solver with TRA reaches the best solution almost immediately and outperforms Adam and SGD, which are common in prior E2E work. On the task-driven objective, the margin widens: LM with GTRA finds dramatically lower MSE than any first-order alternative, while a naive LM formulation that compresses the loss into a single residual stagnates—direct evidence that the high-dimensional residual structure of GTRA is what unlocks LM in the processing-aware setting.
Smartphone Telephoto Lenses
We apply our method to smartphone telephoto lenses, including hybrid configurations that pair refractive elements with a diffractive surface. Across a range of telephoto ratios (TRs), the LM-GTRA optimization decisively beats both spot-radius baselines and first-order (Adam) E2E baselines—achieving better image quality or smaller form factors.
The PSFs reveal a subtle effect. Our E2E-optimized lenses tend to produce PSFs with narrower, rounder central peaks but longer tails. These tails inflate the RMS spot radius, so a spot-radius optimizer would actively discard such designs—even though the narrow peak is exactly what a downstream restoration model can recover from. This is a concrete example of the optical–processing mismatch that processing-aware design is meant to fix.
BibTeX
@article{cote2026generalized,
author = {C\^{o}t\'{e}, Geoffroi and Tseng, Ethan and Heide, Felix},
title = {Generalized Aberrations for Processing-Aware Optical Design},
journal = {ACM Transactions on Graphics},
year = {2026},
doi = {10.1145/3817055},
}