Most of this is one project: a cheap camera over a stove that works out what's cooking, and a lot of checking where it gets that wrong.
An AI sensor that reads the stove.
A small camera that watches the stove and works out when someone's cooking, what is in the pan, and how it's being cooked. That record is what air-quality researchers need, and the parts cost a fraction of what lab instruments do.
A Raspberry Pi above the stove with two cameras, one normal and one thermal. It photographs the cooktop every minute.
Three AI stages: throw away dark frames for free, screen for cooking cheaply, then have a top model write down food, method, burner, and cookware.
Cooking is one of the biggest indoor air pollutants. Exposure researchers need a record of it, and lab-grade gear costs too much to put in many homes.
Cooking is one of the biggest sources of indoor air pollution, and it's surprisingly hard to track. To model it, researchers need to know how often a home cooks, on which burner, how, and for how long. Asking people to write it down doesn't really work, and the lab-grade gear that measures it properly is way too expensive to put in a lot of homes. StoveCam started as a side question from our earlier air-quality work at LBNL: could a cheap camera and an AI model just figure the cooking out by watching?
Each unit is a Raspberry Pi 5 sitting above the stove with two cameras: a normal RGB camera and an Adafruit thermal camera. Every 60 seconds it takes a picture with both and saves the pair. The two cameras are doing different jobs on purpose. The normal photo shows what's on the stove. The thermal photo shows what the normal one can't: heat under a lid, a pan that's been left on, and the actual temperatures.
The most common thing the model saw across the whole dataset was a covered pot. That's the case the thermal camera is there for. The food is hidden and the heat isn't.
Running a top-end AI model on every photo would cost a fortune. So each capture goes through three steps, and only the photos that matter make it as far as the expensive one.
A quick check on the device itself throws out dark photos (lights off, nobody home) before any AI runs. This alone drops most of the captures.
A small vision model on LBNL's CBorg service answers one yes/no question: is anyone actually cooking?
Only the photos that pass step two reach the best model (Gemini), which writes down the food, how it's cooked, the cookware, the burner, and how sure it is.
Step three writes one record per cooking session, and more than one when several burners are going at once. It covers the method (boil, simmer, sauté, pan-fry, deep-fry, bake, broil), which burner, and the cookware.
Two real captures that scored a perfect 10/10 when we graded them. Each one shows the two photos that went in and the record the model wrote out, including the notes it left for itself.
"French toast is actively cooking in a skillet on the front right burner. The thermal image confirms high heat under the pan, with cooler spots where the bread absorbs the heat."
"A person is handling the lid of a large pot on the right burner. Raw chicken is visible inside. The thermal image confirms a strong, ring-shaped active heat signature under this pot. The left burner shows residual heat but is empty in the RGB image, so it is excluded."
The second one is the interesting case. There's motion blur, a hand in the shot, and a covered pot, but the model still puts the two photos together, using the normal photo to rule out the leftover heat the thermal camera picks up on the empty left burner.
To see whether the labels were any good, we went through 506 random photo pairs by hand, scored each out of 10, and noted what kind of mistakes turned up. The model averaged 8.1/10, with 66% scoring 8 or higher and 32% we couldn't fault. When it does get it wrong, it usually thinks cooking is happening when it's really just leftover heat, and for an air-quality study that's the safer way to be wrong. And in all 506 pairs, it never once mixed up the burner or the cooking method.
The model doesn't really "see" the way we do. It shrinks each photo, cuts it into small patches, and turns every patch into a list of numbers. Those numbers sit in the same space as words, so a round metal shape with a handle ends up near "skillet," and a bright ring in the thermal photo ends up near "hot burner." Because it was trained on huge piles of labelled food photos, overhead cooking clips and thermal images, it can even read the little temperature numbers printed on the IR frame.
The point of the project is the method: building cheap sensors out of paired image and sensor data, for air-quality and building research. Once you have a record of what's being cooked, though, it's useful for other things too:
Quantify cooking frequency, duration, and method to model indoor pollutant emissions without intrusive lab gear.
A daily heartbeat summary: "Mom used the stove for 20 minutes this morning; it's now off."
Left in a commercial kitchen, it can help verify food is cooked to adequate temperatures.
Compare ambient heat wasted by gas vs. electric stoves using the thermal channel.
A heat-aware alternative to photo-only calorie apps, since the cooking itself is signal.
Cooking habits as a ground-level indicator: the ratio of meat to vegetables, and when households cook.
Three weeks at Penn's Singh Center for Nanotechnology, working in the Class 100 (ISO-5) cleanroom. The course covers the whole fabrication loop: pattern a wafer, build something on it, then measure whether what came out matches the design.
