I build low-cost sensors that turn everyday signals into real data, and I spend just as much time finding out where they get it wrong.
An AI sensor that reads the stove.
A small camera that watches the stove and works out when someone's cooking, what is in the pan, and how it's being cooked. That record is exactly what air-quality researchers need, and it costs a tiny fraction of what proper lab instruments do.
Cooking is one of the biggest sources of indoor air pollution, and it's surprisingly hard to track. To model it, researchers need to know how often a home cooks, on which burner, how, and for how long. Asking people to write it down doesn't really work, and the lab-grade gear that measures it properly is way too expensive to put in a lot of homes. StoveCam started as a side question from our earlier air-quality work at LBNL: could a cheap camera and an AI model just figure the cooking out by watching?
Each unit is a Raspberry Pi 5 sitting above the stove with two cameras: a normal RGB camera and an Adafruit thermal camera. Every 60 seconds it takes a picture with both and saves the pair. The two cameras are doing different jobs on purpose. The normal photo shows what's on the stove. The thermal photo shows what the normal one can't — heat under a lid, a pan that's been left on, and the actual temperatures.
The most common thing the model saw across the whole dataset was a covered pot. Those are exactly the moments the thermal camera pulls its weight: you can't see the food, but you can still see the heat.
Running a top-end AI model on every photo would cost a fortune. So each capture goes through three steps, and only the photos that matter make it as far as the expensive one.
A quick check on the device itself throws out dark photos — lights off, nobody home — before any AI runs. This alone drops most of the captures.
A small vision model on LBNL's CBorg service answers one yes/no question: is anyone actually cooking?
Only the photos that pass step two reach the best model (Gemini), which writes down the food, how it's cooked, the cookware, the burner, and how sure it is.
Step three writes one record per cooking session, and more than one when several burners are going at once. It covers the method (boil, simmer, sauté, pan-fry, deep-fry, bake, broil), which burner, and the cookware.
The model doesn't really "see" the way we do. It shrinks each photo, cuts it into small patches, and turns every patch into a list of numbers. Those numbers sit in the same space as words, so a round metal shape with a handle ends up near "skillet," and a bright ring in the thermal photo ends up near "hot burner." Because it was trained on huge piles of labelled food photos, overhead cooking clips and thermal images, it can even read the little temperature numbers printed on the IR frame.
To see whether the labels were any good, we went through 506 random photo pairs by hand, scored each out of 10, and noted what kind of mistakes turned up. The model averaged 8.1/10, with 66% scoring 8 or higher and 32% we couldn't fault. When it does get it wrong, it usually thinks cooking is happening when it's really just leftover heat — and for an air-quality study, that's the safer way to be wrong.
Two real captures that scored a perfect 10/10 when we graded them. Each one shows the two photos that went in and the record the model wrote out, including the notes it left for itself.
“French toast is actively cooking in a skillet on the front right burner. The thermal image confirms high heat under the pan, with cooler spots where the bread absorbs the heat.”
“A person is handling the lid of a large pot on the right burner. Raw chicken is visible inside. The thermal image confirms a strong, ring-shaped active heat signature under this pot. The left burner shows residual heat but is empty in the RGB image, so it is excluded.”
The second one is the interesting case. There's motion blur, a hand in the shot, and a covered pot — but the model still puts the two photos together, using the normal photo to rule out the leftover heat the thermal camera picks up on the empty left burner.
The main point of the project is the method — building cheap sensors from this kind of paired data for air-quality and building research. But once you have a record of what's being cooked, it's useful for a lot of other things too:
Quantify cooking frequency, duration, and method to model indoor pollutant emissions without intrusive lab gear.
A daily heartbeat summary — "Mom used the stove for 20 minutes this morning; it's now off."
Left in a commercial kitchen, it can help verify food is cooked to adequate temperatures.
Compare ambient heat wasted by gas vs. electric stoves using the thermal channel.
A heat-aware alternative to photo-only calorie apps, using the cooking process as signal.
Cooking habits — the ratio of meat to vegetables, when households cook — as a ground-level indicator.
An intensive fellowship on the fabrication and characterization of nanoscale devices — processing silicon wafers in a Class 100 cleanroom with advanced lithography.
