Physics-based machine learning sharpens images taken through through fog, tissue, cloudy liquids

A UCLA- and Rochester-led team has shown that combining an existing scattered-light imaging technique with a physics-constrained machine learning framework can more than double image clarity in near real time, with potential applications from surgical guidance to autonomous vehicle sensing

A research team led by University of California Los Angeles (UCLA) and the University of Rochester has demonstrated a novel evolution of an imaging system that can capture detail within ‘complex media’, environments that scatter light, ranging from biological tissue to heavy fog. The system applies physics-based machine learning to improve an existing imaging technique.

In tests using standard calibration images obscured by complex media, the novel system more than doubled the signal-to-noise ratio achieved by a previous generation of the technology. It has also proven able to generate images in close to real time – within thousandths of a second.

Conventional approaches to imaging through complex media rely on expensive cameras able to detect light just beyond the visible range, into the near-infrared. By contrast, the underlying method that the researchers set out to improve can use comparatively inexpensive silicon-based cameras, of the type found in smartphones. First introduced ten years ago by study co-authors from the University of Rochester, this technique uses a specialised film that allows some photons to pass through while blocking others, to convert scattered light from the near-infrared into the visible range.

However, the method has tended to produce a vignetting effect, in which shadows darken the edges of an image and reduce the field of view. Images have also been prone to artefacts, which appear as lighter or darker splotches.

To address these limitations, the researchers combined the existing imaging technique with a machine learning framework named DeepTimeGate. This operates in two stages. The first uses an algorithm trained to reconstruct images mathematically. The second, developed at UCLA, performs a rapid consistency check, to constrain the results according to the fundamental rules of physics.

The ability to sense inside complex media in near real time, using silicon-based cameras, could benefit biomedical imaging. DeepTimeGate may enable less expensive and more effective imaging to guide surgical procedures, including endoscopy. Laboratories that test cloudy fluids such as blood for dangerous microbes or anomalous cells could use a technology of this kind to analyse samples without the need to dilute or filter them first.

A further potential application lies in cameras for autonomous vehicles, to help detect surroundings through rain, fog, dust or sand. In industry, the imaging system might in future support quality control in manufacturing processes involving cloudy liquids or frosted packaging, as well as in waste-removal plants.

The study was carried out through a collaboration between UCLA, the University of Rochester, Stanford University, the University of Ottawa in Canada, the Air Force Research Laboratory, Clemson University and the University of Central Florida.

The study’s leading authors are Dr. Sergio Carbajo, an associate professor of electrical and computer engineering at the UCLA Samueli School of Engineering and of physics and astronomy at the UCLA College, and a member of the California NanoSystems Institute at UCLA, and Dr. Robert Boyd of the University of Rochester. Hao Zhang, a doctoral candidate at UCLA who also serves as the corresponding author, and Dr. Yang Xu of the University of Rochester, are the study’s co-first authors.

For further reading please visit: 10.1038/s41377-026-02375-6