Aligning and fusing vision, thermal, RF, acoustic, and inertial signals into unified representation spaces that expose richer physical structure than any single modality. The modalities are complementary, so together they can answer sensing problems that no single one of them can solve, supported by multi-modal datasets, physics-grounded models, and a shared sensing platform.
The analogy
Vision–language models, and the action models built on them, are the nearest analogy. One embedding is taken from a camera, another from language, and aligning the two serves recognition, retrieval, and action. Multi-modal sensing looks like the same problem: many streams, one embedding.
Three strategies now in the literature. Fuse modalities into one embedding: X-Fi (ICLR 2025), MASTER (IMWUT 2025). Align a sensing embedding with a language or vision foundation model: FM-Fi, mmCLIP (SenSys 2024), TENT (TMC 2026). Align sensing modalities with each other by contrastive learning on paired streams: BABEL, MMBind (SenSys 2025).
The analogy breaks in two places. The signal is physical. The sensing modalities are diverse. Each break leaves a set of open problems.
The signal is physical
A pixel records appearance. A phrase encodes a concept. Wi-Fi, radar, thermal, and IMU observe the world through different physical principles.
Is the Transformer the right backbone for these modalities, or how should it be adapted to each sensor’s physics?
How do we combine the generalizability of physics with the fitting power of deep learning?
How should proven signal-processing methods enter a data-driven model?
What are effective learning tasks and objectives for sensing foundation models?
……
The sensing modalities are diverse
Sensing spans far more modalities than the vision–language setting.
Is aligning a sensing embedding to a language or vision foundation embedding a good approach, as in CLIP-style methods?
Is contrastive learning on paired sensing modalities a good path to a unified representation, as in Babel?
Should each modality have its own foundation model, or should many modalities share one?
Is there a natural center modality to build first, then align the others to it, as language came first and CLIP then aligned vision to it?
……
If you are interested in the problems above, or in other multi-modal sensing questions, please feel free to contact me with zhangxie289 at gmail dot com. I am open for any discussions and collaborations.
How we touch it
We touch this multi-modal sensing in the model, the data, the platform, and the community. Those four correspond to RF-CRATE, OctoNet, OctoSense, and the MuMoSe tutorial.
1Model
A transformer derived from mathematical theory in the complex domain. A full grounding in the physics is still open. The derivation is a bridge: it carries the learned model over to that physics, which enters the model as mathematics.
RF-CRATE
Towards White-Box Deep Wireless Sensing
Derived from the complex sparse rate reduction principle
Motivation and challenges. Deep wireless sensing works, and most of its models are black boxes: blocks chosen by hand, representations with no stated tie to the measurement. A radio recording is complex. Phase is delay, Doppler, and multipath. Dropping it, or splitting it into two real channels and hoping, leaves the physics outside the model. Labeled radio sets are also small beside vision pretraining, so a principled model still has to train.
Parsimony in the learned representation. A raw distribution is compressed, then sparsified into subspaces. Drawn on a radio scene, the same idea keeps the multipath components that belong to the gesture.Black-box models stack heuristic or domain blocks and leave an opaque representation. A white-box model starts from a first principle, iterates an optimization, and unrolls that iteration into the network.
Key insight. Wireless measurements are sparse and low-dimensional, which is the structure a white-box transformer already assumes in vision. Carrying that principle into radio means writing the objective in the complex domain, and deriving both attention and the residual block from it. Signal processing enters as the optimization being unrolled, including sparse recovery, rather than as a preprocess that the network never sees.
From complex sparse rate reduction to the model. Alternating minimization becomes one block: RF multi-head subspace attention, then a derived residual RF-MLP. The stack is a complex-valued white-box transformer.
Model. The architecture unrolls complex sparse rate reduction under CR-calculus, so phase stays inside the operator. The residual feed-forward block is derived by treating the metric as learnable. Subspace regularization then spreads the representation when labels are scarce. This is a step on the first three questions: a transformer adapted to complex radio, a physical objective with a learned fit, and a signal-processing method inside the layer. The fourth question, which objective a sensing foundation model should train on, stays open. The thermal counterpart, a physics-informed map from heat to a body-shaped cloud, is TAP3D.
Results
19.98%average gain, subspace regularization
3.39%classification gain, complex-valued
10.34%less regression error, complex-valued
5datasets, activity through respiration
Activity, gait, gesture, pose, and respiration. Competitive with strong black-box models. Mathematical interpretability is the claim. Reading the learned subspaces as paths, delay, and Doppler is the physical step still ahead.
2Data
Twelve modalities recorded together and tied to one pose, so fusion, alignment to another foundation model, and contrastive pairing across sensors can be compared on the same body.
OctoNet
A Large-Scale Multi-Modal Dataset for Human Activity Understanding Grounded in Motion-Captured 3D Pose Labels
Motivation and challenges. The questions above need recordings that were taken together. Existing multi-modal sets rarely pass seven modalities, and many radio sets are short on people and on activities worth recognizing. Without a shared body, there is no coordinate in which a fusion, a CLIP-style alignment, and a Babel-style pairing can be compared.
Twelve modalities, from RGB and depth through thermal, infrared array, millimeter-wave, UWB, Wi-Fi, acoustic, IMU, and heart rate, tied to one motion-captured 3D pose.
Key insight. High-fidelity 3D pose, from a professional motion-capture system, is a label every modality can share. It is one candidate for the center the fourth question asks about: a body coordinate, rather than language coming first. Activity class and skeleton then supervise the same recording, so a model can be compared across sensors, views, people, and rooms. Whether that pose should be the center, or whether language, vision, or a single sensor should be, is left open by the dataset on purpose.
Five nodes around a 4.1 m square, in three scenes. The same moment as RGB, depth, time-of-flight, thermal, infrared array, two radars, UWB, Wi-Fi CSI, acoustic, IMU, and the motion-capture skeleton.
What the dataset holds. More than 67.72 million synchronized frames, 41 participants, 62 activities, from structured routines to freestyle, human–environment, and healthcare tasks. Baselines cover activity recognition and 3D pose estimation, one modality at a time and early fusion. Fusion is the first strategy on the slide. The CLIP-style and contrastive strategies are questions the release is built to host.
Results
94.9%RGB, 10-class, Swin-T
98.3%IMU, 10-class
99.9%thermal + IRA + IMU, in domain
75.2%that fusion, cross-user
RGB with Swin-T is 93.1% on the full 62 classes. Thermal with DenseNet is 91.7% / 85.4% on the 10-class and 62-class tasks. The thermal–IRA–IMU fusion falls to 52.7% cross-scene. A shared embedding that is strong in one room is not yet a representation that travels.
3Platform
A shared loader, a shared set of signal operators, and a recorded experiment, so those comparisons are the same study with a different objective.
OctoSense
Building a Unified Ecosystem for Open-Source Wireless Sensing
Motivation and challenges. Vision can benchmark a model because the dataset, the loader, and the metric already exist. CSI, millimeter-wave, and UWB arrive in device-specific files, preprocessing is undocumented, and a missed split is enough to make a baseline unreproducible. Another dataset, or another repository, leaves that adaptation on the next reader.
A conventional study couples loading, transforms, and benchmarking into one-off scripts. OctoSense separates them: load by name, a shared transform, a model, and a benchmark specification.
Key insight. The repeated work sits in the meaning of the array: carrier, sampling rate, antenna layout, and the transform that turns that array into a model input. Attach the meaning once, and datasets, operators, and benchmarks can be swapped without rewriting the study.
One example, STFNet on Widar3.0. The same calls switch the dataset, including MM-Fi and OctoNet, chain signal-processing operators, and rerun the baseline from a specification file.
Model. Three components. A unified data abstraction writes physical metadata onto heterogeneous recordings at load time. Standardized signal operators are hardware-agnostic transforms. A benchmark engine stores the experiment in a specification file and reruns it. The stack stays compatible with PyTorch. Shipped datasets include Widar3.0, WiFi-Gait, SignFi, FallDar, CSI-Bench, HuPR, MM-Fi, XRF55, and OctoNet. Shipped models include STFNet, UniTS, RFNet, Swin Transformer, and ResNet.
Results
0.8 ptof a hand-written pipeline
>80%less boilerplate
~10 linesSTFNet on Widar3.0
9datasets already connected
Paired against hand-engineered implementations, accuracy stays within 0.8 percentage points. The same interface covers baseline inference, a custom model, a declarative pipeline, and a dataset that was not written for the platform.
4Community
The analogy and the two breaks, taught in one afternoon at UbiComp / ISWC 2026, together with the model, the data, and the platform.
MuMoSe
Towards Open-Sourced Datasets, Benchmarks, and Platforms for Multi-Modal Sensing
A three-hour tutorial for newcomers and active researchers. Module I covered the dataset landscape, with deep dives into OctoNet, RF-Behavior, and OctoSense. Module II set sensing models against vision–language models, then three paradigms: white-box models (RF-CRATE), physics-informed thermal-to-3D learning (TAP3D), and signal-processing–learning co-design (RF-LEGO). A notebook streamed, loaded, and visualized OctoNet and RF-Behavior. Slides, datasets, code, and the recordings stay on the tutorial site.
Opening. Motivation, scope, and format. Speaker: Prof. Chenshu Wu.
Module I · Datasets and benchmarks. OctoNet, hosting and access, Dr. Xie Zhang. RF-Behavior, gestures, activities, and affect, Dr. Si Zuo. OctoSense, Dr. Weiying Hou.
Module II · Models. RF-CRATE and TAP3D, Dr. Xie Zhang. RF-LEGO, Mr. Luca Jiang-Tao Yu.
Hands-on. Streaming download, modality loading, and visualization. Demo videos by Mr. Xuan Lian.
Closing. Open questions and discussion. Speaker: Prof. Chenshu Wu.
RF-Behavior is a five-stream radio dataset (radar, LoRa, RFID, IMU, and motion capture) from Aalto University, presented by Dr. Si Zuo. RF-LEGO, deep unrolling for radio, was presented by Mr. Luca Jiang-Tao Yu. Both appear here as neighboring work from the tutorial, under their own authors. Organized by Xie Zhang, Weiying Hou, and Chenshu Wu, and hosted by the HKU AIoT Lab.
References
Dongsheng Yuan* and Xie Zhang*, Weiying Hou, Sheng Lyu, Yuemin Yu, Luca Jiang-Tao Yu, Chengxiao Li, and Chenshu Wu. 2025. OctoNet: A Large-Scale Multi-Modal Dataset for Human Activity Understanding Grounded in Motion-Captured 3D Pose Labels. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. *Equal contribution.
Xie Zhang, Yina Wang, and Chenshu Wu. 2026. Towards White-Box Deep Wireless Sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 10, 3, Article 191 (September 2026). https://doi.org/10.1145/3831998
Weiying Hou, Xie Zhang, and Chenshu Wu. OctoSense: Building a Unified Ecosystem for Open-Source Wireless Sensing. In The ACM Conference on Embedded Networked Sensor Systems (SenSys 2027). Accepted.
Xie Zhang, Weiying Hou, and Chenshu Wu. 2026. Towards Open-Sourced Datasets, Benchmarks, and Platforms for Multi-Modal Sensing. Tutorial at the 2026 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the ACM International Symposium on Wearable Computers (UbiComp / ISWC 2026), October 11, 2026. https://aiot-lab.github.io/MuMoSe/
