A research directionThermal Array–Based Spatial Sensing
Physics-guided and learning-based frameworks that elevate low-resolution thermal arrays into 3D spatial representations for human detection, ranging, reconstruction, interaction, and building-scale occupancy analytics, with strong privacy and cost advantages.
Principle and property
Fully passive sensing
The array never lights the scene and never sends a probe. Many of them can share a room, through the day and the night, without a lamp, a radio collision, or an emission hazard.
Long wavelength (8–15 μm)
The recording is heat, so a face is not resolved. The same wavelength is why glare, smoke, and darkness do not take the person off the map.
Multi-physics encoding
One pixel mixes temperature, how the surface emits, and a cue tied to distance. Pulling those apart is the problem the techniques below take on.
Current research overview
Techniques turn the map into spatial primitives: a range for each person, then a body-shaped cloud. Applications read behavior from that geometry, from a hand beside a device to sleep, a loss of balance, and a blink. Systems leave the same sensing running in real rooms and record how those rooms are used.
1Techniques
A thermal pixel is a temperature, not a distance. The first two steps put a range on that heat, then a body.
1.1TADAR
Thermal Array-based Detection and Ranging for Privacy-Preserving Human Sensing
Motivation and challenges. Human sensing needs a range, and a thermal array is the modality that keeps both resolution and privacy. The map it returns is only two-dimensional. It has lost the complete 3D spatial information, so it cannot say where each person stands, and earlier work was left mounting the sensor on the ceiling and reading the pixels as a flat picture.
Motivation. From a camera to a radio, resolution falls as privacy rises. A thermal array is where both still hold.
Key insight. Air attenuates thermal radiation, so a closer person reads warmer than a farther one. Bare skin at the face and neck is the stable reference: its emissivity barely changes, and clothing only makes the other pixels cooler.
Insight. The same person reads cooler, and smaller, as the range grows from 1 m to 5 m.
Model. For each person, the region of interest is pooled, flattened, and sorted so the hottest pixels lead the feature, and the region’s center is appended to correct readings near the edge of the frame. A histogram gradient booster maps that feature to a range, and a Kalman filter smooths it over time. A spatial detector and a temporal tracker find and follow several people in the textureless map, without a trained segmentation network, and the two streams are fused.
Model overview. Detection and tracking feed the ranging model.
Results
88.8%multi-user detection F1
32.0 cmmean ranging error
20.1 cmwithin 3 m
Several people, each with a range, from one passive array.
Motivation and challenges. A human point cloud is the intermediate representation that carries pose and shape into later tasks, and the bridge that aligns one modality with LiDAR, radar, and depth. Depth cameras raise privacy concerns, LiDAR is costly and emits a laser, and radar returns a sparse cloud and interferes with its neighbors. From a temperature map the remaining challenges are depth mixed with emissivity and surface temperature, warm objects that look like people, and people who overlap along the line of sight.
The same heat, as a body-shaped cloud, beside depth, LiDAR, and radar.
Key insight. A pixel’s temperature is not a depth. Depth is entangled with emissivity and surface temperature. Those two are what mark a person, and people who overlap along the optical axis often separate in a bird’s-eye view.
Pilot study. A direct learner misplaces depth, takes a warm refrigerator for a person, and merges two people into one cloud.
Physics model. Each pixel collects radiation from a patch of body surface. Air attenuates that radiation on the way to the lens, and warm objects nearby can add a reflection. The patch projects onto the array through a pinhole, so the temperature a pixel reports mixes depth, emissivity, and surface temperature. The forward model rebuilds the map from those quantities. The network then solves the inverse.
Physics model. A pixel receives the person’s emission, attenuated along the projection line, plus radiation reflected from ambient objects.
Model. A U-Net-style inverse solver reads one temperature map and jointly estimates depth, emissivity, surface temperature, reflected radiation, and which person each pixel belongs to. The forward thermal model above rebuilds the map from those estimates, so the solver trains without labels for emissivity or surface temperature. Geometric perspective fusion then cleans the result: a detector on the optical-axis view uses temperature and emissivity to reject ambient heat, and differentiable index mapping lifts the depth into a bird’s-eye view so adjacent people come apart. The two views fuse into one cloud per person.
Model overview. Primitives are estimated, then refined from two views.
Results
4.60 cmdirected Chamfer distance
0.95detection F1
21.86 cmtracking MAE
4.87 cmmesh error
91.46%fall from the cloud
160K samples, 8 rooms, 11 people.
Case: the cloud is dense enough to generate a body mesh while walking, standing, and sitting.
Motivation and challenges. More IoT devices ship without a screen or a keyboard, so they need flexible gesture control beside the device, beyond a fixed menu of gestures. Wearables ask to be worn, cameras threaten privacy and cost compute, and radio or acoustic sensing needs a cumbersome setup. A privacy-preserving thermal array can reconstruct the 3D hand for that control, yet the map is low-resolution and textureless, and the model must still run on a small, low-power board.
Key insight. Captured radiant power falls as the hand moves away, but not by the same curve at every keypoint. Different poses also leave different two-dimensional patterns even at the same depth, so the pose can be retrieved from that pattern rather than regressed from a blank map.
Insight. Temperature falls with depth, and the edge of the hand falls faster than the center.
Model. Three modules. A thermo-space encoder turns the temperature map, with its two-dimensional hand context, into a three-dimensional space of depth and possible poses. A keypoint feature extractor reads that spatial pattern together with the structure of a hand. A position-retrieval decoder uses those features as queries and picks each keypoint’s three-dimensional position out of that space. Heterogeneous distillation compresses this model into NanoTAPOR, which runs on an ESP32-class board from one MLX90640, about $20 and under 0.08 W.
Model overview. Pose is retrieved from a thermo-space, then distilled for the device.
Results
2.26 cmjoint error
3.26 cmon device
2.69 cmfinger tracking
377×less compute
Case: gesture and pose recovered from the temperature map of one hand.
Motivation and challenges. Falls are common among older adults, and many of them happen in bedrooms and bathrooms, where a camera is hard to accept and a wearable is easy to leave off. Radio sensing avoids the image, yet it still reads a fall through a change in height or speed, so sitting down quickly, bending, or a passing pet raises alarms that wear out trust. On a thermal map the further challenges are a textureless pose under motion smear, a loss of balance that is brief and hard to label, and daily motions the model has never seen.
Key insight. A fall is a loss of balance, not a spike in speed. On a thermal map the head and exposed skin stay hotter than the clothed torso, and the smear of a fast movement is a cue for velocity rather than a defect. Balance itself is a continuous margin, not a binary label.
Stable, loss of balance, ground. The decision follows the margin, not a velocity peak.Insight. Balance is the margin between the center of mass and the base of support. Fast motion leaves a smear the array can read as velocity.
Model. Two phases. Appearance–motion fusion uses a dual path: one path reads the temperature layout, the other reads the smear, and together they estimate a pose sequence. A balance-aware head is trained on both the continuous signed margin of balance and the discrete fall label. Before that, the pose network is pretrained on large public pose data with cross-view contrast, so unseen activities and people raise fewer false alarms.
Model overview. Pose from appearance and motion, then the margin of balance.
Results
98.26%detection
0.65%false alarms, lab
3.6 / weekfalse alarms, homes
25 / 25bathroom falls
More than 3,000 falls, 35 people. Then 27 days in four homes: 0.00126% of about 0.29 million windows, and one false alarm in the bathroom pilot.
Motivation and challenges. Sleep posture and motion are health signals, from bedsores to positional apnea, and a clinical lab with contact sensors is the wrong place to watch them for a week. Cameras need light and enter the bedroom; radar is coarse. A thermal array is private and works in the dark, but a blanket hides the body and heat left in the mattress looks like a person who has already gone.
Key insight. A blanket hides the body from the array, and heat left in the mattress looks like a person who has already gone. Depth from a co-located time-of-flight sensor supplies the occupancy the heat map cannot, and a state machine stops leftover warmth from flipping presence on and off.
Insight. The blanket hides the torso. Only the limbs still read as body heat.Insight. After the person leaves, the bed keeps a body-shaped patch for about a minute.
Model. Three modules on synchronized thermal and depth frames. Presence scores summary statistics of the temperature map and of consecutive depth frames with a random forest each, averages them, and smooths the result with a state machine whose transitions depend on motion. Motion is a masked difference of thermal frames, summed inside the body. Posture resizes that body region and classifies it with a ResNet-18, then a majority vote assigns one posture to each low-motion stretch.
Model overview. Presence, motion, and posture from the thermal map and a small depth sensor.
Results
98.7%presence detection
96.1%posture
0.718motion MAE
170 minutes, 5 people, 6 postures, 3 rooms.
The in-bed heat pattern the array still sees.Case: one week, 45 hours. Posture around the clock, from absent to soldier, starfish, log, yearner, foetus, and freefaller.
2.4ThermalEye
Fully Passive Eye Blink Detection on Smart Glasses via Low-Cost Thermal Sensing
Motivation and challenges. Blink rate is a marker for fatigue, dry eye, and interaction, and a visible-light camera on the face is lighting-sensitive, power-hungry, and hard to justify. On a thermal array the blink is a small, slow dip. It is easy to lose in sensor noise, uneven across the pixels, and different from person to person.
Key insight. A blink is a small, slow dip in the heat over the eye. It is easy to lose in sensor noise, the dip is uneven across the array, and the resting pattern differs from person to person.
Insight. One person’s sessions agree. Four people do not: each rests at a different temperature.
Model. Three stages. Preprocessing filters, normalizes, and raises the contrast of the 12×16 map. A ResNet-18, taking a six-channel frame, scores whether the eye is closed. Temporal post-processing applies a median filter and groups frames into blink events, which is what cuts isolated false alarms.
Model overview. A frame score, then a blink event.
Results
0.89F1 per frame
0.80F1 per blink
190 minutes, 4 people.
The array, worn. One small passive sensor on the frame.Case: temperature falls between blink start and blink end. The maps are eyes open and eyes closed.
3Systems
3.1BuildingPulse
Privacy-First Occupancy Monitoring with Thermal Arrays
Occupancy from a ceiling mesh
Under review
We are commercializing this system at SenseAI, where Xie Zhang is Chief Scientist, for privacy-first occupancy sensing in human-aware buildings.
Motivation and challenges. Buildings need continuous occupancy for crowding, evacuation, and heating and cooling, and cameras, LiDAR, and Wi-Fi each fail privacy, cost, or spatial precision. One thermal array covers only part of a room. At building scale the nodes are mounted differently, their views overlap, their clocks disagree, and sunlight, monitors, and leftover heat look like people.
Key insight. A ceiling view makes an occupant a small, stable signature, so nearby people can be split by region size. Nodes should send a detection, not the raw map. Short tracks reconcile overlapping views, and the history of a location separates a person from a heater that never moves.
Insight. From the ceiling, two nearby people merge into one warm region. A size cue splits them back apart.
Model. Each node runs Extended-TADAR: a nonlinear contrast transform, then a region-size-guided search that splits merged occupants. The cloud projects those boxes onto one floor, fuses asynchronous tracks so the same person is not counted twice, and applies a posterior filter that marks persistent static heat as non-human.
Model overview. Compact detections leave the node. Fusion and filtering happen once, for the whole floor.
Results
0.933F1, one node
0.16counting MAE, one node
0.96recall, many nodes
1.30counting MAE, many nodes
The three rooms, and where the nodes sit. Office, 60 m², 8 nodes, 198 days, on a routine. Student activity center, 180 m², 15 nodes, 256 days, spiking with events. Gym, 300 m², 31 nodes, 62 days.Case: the gym floor over a month, a week, and a day. It shows where people stand and which machines they use.
Open questions for future exploration
I am open for any discussions and collaborations. If you are interested in the following research problems, or in other thermal-related questions, please feel free to contact me with zhangxie289 at gmail dot com.
From TADAR to TAP3D. TADAR puts a range on the heat. TAP3D turns that heat into a body-shaped cloud, and the heat itself marks the person. The same physical model could extend to targets that are not people, toward a panoptic cloud of the whole scene.
Density. TAP3D raises what a thermal array can resolve by extending TADAR from a range to a point cloud. The cloud is still one point per pixel, hundreds of points on a person: denser than a compact radar cloud, and still thinner than an RGB-D cloud. Interpolation or learned super-resolution is the way past that per-pixel limit.
Primitives. An open question is whether the other physical quantities can be read from the same heat signature: emissivity, surface temperature, and reflected radiation. One model might estimate each quantity on its own, or one model might estimate them jointly.
Several sensors. Multi-sensor fusion matters. BuildingPulse has begun it for occupancy, placing a ceiling mesh on one floor. Aligning body-shaped clouds across arrays, in one coordinate frame and for tasks beyond counting, still needs research.
Several modalities. A thermal array is fully passive, long-wave, and encodes several physical quantities in one pixel. Fusion with LiDAR, radar, and depth cameras can open applications and meet challenges that one modality leaves, by using the complementary nature of the sensors.
Power. The sensor itself draws milliwatts. Battery-powered and battery-free nodes, bringing the whole node down toward microwatts, still need research.
Outdoors. Sunlight heats the scene and flattens the contrast between a foreground and the background. Shallow placements, night-only operation, and uses such as wildlife monitoring are ways to begin.
References
Xie Zhang and Chenshu Wu. 2024. TADAR: Thermal Array-based Detection and Ranging for Privacy-Preserving Human Sensing. In The Twenty-fifth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing (MobiHoc ’24), October 14–17, 2024, Athens, Greece. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3641512.3686357
Xie Zhang, Chengxiao Li, Xuan Liu, and Chenshu Wu. 2026. TAP3D: Thermal-Assisted 3D Human Point Clouds. In The 32nd Annual International Conference on Mobile Computing and Networking (MobiCom ’26), October 26–30, 2026, Austin, TX, USA. ACM, New York, NY, USA. https://doi.org/10.1145/3795866.3844491
Xie Zhang, Chengxiao Li, and Chenshu Wu. 2025. TAPOR: 3D Hand Pose Reconstruction with Fully Passive Thermal Sensing for Around-Device Interactions. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 9, 2, Article 63 (June 2025), 25 pages. https://doi.org/10.1145/3729499
Chengxiao Li, Xie Zhang, Wei Zhu, Yan Jiang, and Chenshu Wu. 2026. TaFall: Balance-Informed Fall Detection via Passive Thermal Sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 10, 4, Article 232 (December 2026), 29 pages. https://doi.org/10.1145/3857959
Xuan Liu, Xie Zhang, and Chenshu Wu. 2025. Privacy-Preserving Non-Contact Sleep Monitoring via Multimodal Thermal-Depth Sensing. In ACM International Workshop on Thermal Sensing and Computing (MobiCom ’25), November 4–8, 2025, Hong Kong, China. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3737905.3769281
Yuhan Chen, Jingwei Song, Xie Zhang, Jianqi Zhang, and Chenshu Wu. 2025. ThermalEye: Fully Passive Eye Blink Detection on Smart Glasses via Low-Cost Thermal Sensing. In ACM International Workshop on Thermal Sensing and Computing (MobiCom ’25), November 4–8, 2025, Hong Kong, China. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3737905.3769280
Xie Zhang, Tsun Ho Wu, and Chenshu Wu. Privacy-First Occupancy Monitoring with Thermal Arrays. Under review.
Xie Zhang. 2026. From Body Heat to Behavioral Horizons: Spatial Sensing Intelligence with Thermal Arrays. In Proceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services Companion (MobiSys Companion ’26), June 21–25, 2026, Cambridge, United Kingdom. ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/3812835.3814826
