Tkwer

MMEngine

Mon, 27 Oct 2025 00:00:00 +0000

Overview

MMEngine is designed as an efficient, next-generation multimodal data acquisition engine. Built with a blueprint node architecture, it provides an intuitive and user-friendly interaction experience for complex data processing workflows. The system enables seamless integration of multiple data sources with real-time processing and visualization capabilities.

Current Progress: We are collecting multimodal air-writing text data from over 100 volunteers using MMEngine. Our research aims to recognize free OOV (out-of-vocabulary) text in single-radar scenarios through cross-modal contrastive distillation. The ultimate goal is to enable air-writing interaction as seen in science fiction movies.

MMEngine blueprint node interface with real-time processing.

Core Features

Blueprint Node Design

Visual Programming: Drag-and-drop node-based workflow creation
Modular Architecture: Reusable processing blocks for different data types
Real-time Connection: Live data flow between nodes with instant feedback
User-Friendly Interface: Intuitive design accessible to both beginners and experts

Multimodal Data Support

Leap Motion Integration: ✅ Complete data acquisition and visualization
Radar Data Nodes: ✅ Complete mmWave radar integration
IMU Data Nodes: ✅ Complete IMU data acquisition and visualization
Extensible Framework: Support for additional sensor types and data sources
Synchronized Processing: Multi-sensor data fusion with timestamp alignment

Advanced Processing Capabilities

Deep Learning Nodes: Pre-built neural network modules for real-time inference
Signal Processing: JSON-configurable processing nodes for custom algorithms
Real-time Visualization: Live data streaming and interactive displays
Synchronized Annotation: Multi-modal data labeling and ground truth generation

Technical Architecture

Node System

Data Acquisition Nodes: Sensor input interfaces (Leap Motion, Radar, etc.)
Processing Nodes: Signal processing, filtering, and transformation
ML Inference Nodes: Real-time deep learning model execution
Visualization Nodes: Real-time plotting and data display
Export Nodes: Data saving and format conversion

Configuration System

JSON-Based: Flexible node parameter configuration
Hot-Reload: Runtime parameter updates without restart
Template Library: Pre-configured node templates for common tasks
Custom Nodes: User-defined processing blocks

Current Implementation Status

✅ Completed Features

Leap Motion Module: Full data acquisition and visualization pipeline
Blueprint Editor: Visual node connection and workflow design
Real-time Processing: Low-latency data streaming architecture
Basic Visualization: Multi-dimensional data plotting capabilities

🚧 In Development

Advanced ML Nodes: Extended deep learning model support Annotation Tools: Enhanced labeling and ground truth features Performance Optimization: GPU acceleration and memory management

🔮 Planned Features

Multi-Sensor Fusion: Advanced sensor data synchronization
Cloud Integration: Remote processing and data storage
Plugin System: Third-party node development framework
Collaborative Tools: Multi-user annotation and workflow sharing

Key Advantages

Ease of Use

No Coding Required: Visual programming eliminates complex scripting
Rapid Prototyping: Quick workflow creation and testing
Real-time Feedback: Instant visualization of processing results
Flexible Configuration: Easy parameter adjustment through GUI

Performance & Scalability

Efficient Processing: Optimized data pipelines for real-time operation
Modular Design: Scalable architecture for complex workflows
Resource Management: Intelligent memory and CPU utilization
Cross-Platform: Compatible with Windows, Linux, and macOS

MMEngine represents the future of multimodal data processing, making advanced sensor fusion and machine learning accessible through an intuitive visual interface.

RadarEngine

Wed, 27 Aug 2025 00:00:00 +0000

Overview

RadarEngine is a C/C++ radar data acquisition and visualization software built with the ImGui framework. This tool is designed to dramatically simplify radar operation by integrating all essential functions into a single, user-friendly interface. Users can configure all radar parameters, update firmware, collect data, and visualize results in real-time without complex setup procedures.

RadarEngine ImGui interface with real-time visualization.

Core Features

Integrated Radar Management

Firmware Update: Built-in firmware flashing and update capabilities
Parameter Configuration: All radar settings configurable through GUI
Real-time Control: Instant parameter adjustment with immediate feedback
Device Detection: Automatic radar module recognition and connection

Data Acquisition & Processing

Raw Data Collection: Direct radar signal capture and storage
Signal Processing: Built-in FFT, filtering, and preprocessing algorithms
Real-time Processing: Live signal analysis with minimal latency
Data Export: Multiple format support for further analysis

Visualization & Monitoring

Real-time Display: Instant visualization of processed radar data
Multiple Views: Range-Doppler maps, spectrograms, and time-domain plots
Interactive Interface: ImGui-based responsive controls and displays
Customizable Layout: Flexible window arrangement for different use cases

Technical Details

Development Stack

Language: C/C++ for high performance
GUI Framework: ImGui for immediate mode interface
Graphics: OpenGL for real-time rendering
Platform: Cross-platform compatibility (Windows/Linux)

Current Status

This project is currently in beta/development stage. Core functionalities are implemented and working, with ongoing improvements in:

User interface refinement
Additional signal processing algorithms
Enhanced visualization options
Performance optimization

Key Advantages

Simplified Operation

One-Stop Solution: All radar operations in a single application
No Complex Setup: Eliminates traditional multi-step radar configuration
Intuitive Interface: Easy-to-use GUI for both beginners and experts
Real-time Feedback: Immediate visualization of parameter changes

Performance & Efficiency

Low Latency: C/C++ implementation for optimal performance
Memory Efficient: Optimized for continuous operation
Real-time Processing: Live data analysis without delays
Responsive UI: ImGui ensures smooth user interaction

This software aims to bridge the gap between complex radar hardware and user-friendly operation, making radar technology more accessible for research, development, and practical applications.

Domain-Generalized mmWave Gesture Recognition via Multi-View Learning

Mon, 07 Apr 2025 00:00:00 +0000

Create your slides in Markdown - click the Slides button to check out the example.

Supplementary notes can be added here, including code, math, and images.

mmScribe: Streaming End-to-End Aerial Handwriting Text Translation via mmWave Radar

Mon, 07 Apr 2025 00:00:00 +0000

Create your slides in Markdown - click the Slides button to check out the example.

Supplementary notes can be added here, including code, math, and images.

mmScribe

Fri, 27 Sep 2024 00:00:00 +0000

Overview

mmScribe is an innovative Aerial Handwriting system that enables contactless human-computer interaction through millimeter-wave radar technology. The system accurately captures user handwriting and converts them into text input, providing a novel approach to human-computer interaction.

mmScribe Real-time Aerial Handwriting System

Key Features

🎯 Streaming Aerial Handwriting Recognition - Real-time gesture-to-text conversion
📱 Cross-platform Compatibility - Supports Android, Windows, and Raspberry Pi
⚡ Real-time Response - Low latency for seamless interaction
🔒 Privacy-Preserving - No camera required, protecting user privacy
🛠️ Easy Integration - Simple integration with existing systems
📊 Comprehensive Tools - Complete data analysis and processing toolkit

Platform Support

mmScribe supports multiple platforms through our runtime system:

Platform	Status	Description
Android	✅ Released	Full APK available for download
Windows	✅ Source Code	Complete source code and libraries
Raspberry Pi	✅ Source Code	Optimized for embedded systems

Hardware Requirements

ESP32-BGT60TR13 Radar Module
- 58-63GHz mmWave Radar
- USB/UART Interface
- 5V Power Supply

Quick Installation

Android

# Download and install APK
wget https://github.com/Tkwer/mmScribe/releases/latest/download/mmScribe.apk

Windows/Raspberry Pi

# Clone the repository
git clone https://github.com/Tkwer/mmScribe.git
cd mmScribe

# Install dependencies
pip install -r requirements.txt

Dataset

We provide a comprehensive dataset for aerial handwriting recognition using millimeter-wave radar:

🧑‍🤝‍🧑 12 participants (6 males, 6 females)
📝 15,488 total samples across all participants
📊 Rich feature set including micro-Doppler and range-time data
🎯 Ground truth data from Leap Motion controller

Dataset Structure

dataset/
├── datas1/    # Reserved dataset
├── datas2/    # Participant 001 (1212 samples)
├── datas3/    # Participant 002 (1202 samples)
...
└── datas14/   # Participant 013 (1192 samples)

Quick Start

Prerequisites

Python 3.8 or higher
CUDA-compatible GPU (optional, for faster processing)
Compatible radar hardware (ESP32-BGT60TR13)

Basic Usage

Select Dataset - Choose from our comprehensive dataset
Run Training:
```
python main.py
```
Deploy - Use the trained model on your target platform

Technical Details

The system leverages millimeter-wave radar technology to capture fine-grained hand movements in 3D space. By analyzing micro-Doppler signatures and range-time data, mmScribe can accurately recognize air handwriting without requiring any physical contact or camera-based tracking.

Key Advantages

Privacy-First Design: No visual data collection
Works in Any Lighting: Independent of ambient light conditions
Low Power Consumption: Efficient radar-based sensing
Robust Performance: Resistant to environmental interference

Applications

📝 Contactless text input for smart devices
🏥 Sterile environment interaction (medical settings)
🎮 Gaming and entertainment interfaces
🏭 Industrial control systems
♿ Accessibility solutions for users with mobility challenges

Documentation

For detailed documentation, including:

Runtime setup guides
Dataset usage instructions
API references
Training tutorials

Please visit the GitHub Repository.

License

This project is licensed under the MIT License - see the LICENSE file for details.

⭐ If you find this project useful, please consider giving it a star on GitHub!

Joint position estimation for hand motion using MIMO FMCW mmWave radar

Sun, 01 Sep 2024 00:00:00 +0000

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

Create your slides in Markdown - click the Slides button to check out the example.

Supplementary notes can be added here, including code, math, and images.

HCI Gesture Control

Sat, 27 Jul 2024 00:00:00 +0000

Overview

This project presents a revolutionary gesture control system for smart TVs and home entertainment devices using mmWave radar technology. The system enables mid-to-long range gesture recognition without requiring users to hold any physical devices, providing a truly hands-free smart home experience.

Device-free gesture control system for smart TV.

Key Features

Device-Free Interaction

No Remote Required: Control your TV and smart devices through natural hand gestures
Mid-to-Long Range Detection: Effective gesture recognition from 1-5 meters distance
Multiple Gesture Support: Swipe, point, wave, and custom gesture patterns

Seamless Integration

HID Device Simulation: Acts as a standard Human Interface Device (HID)
Universal Compatibility: Works with any smart TV, streaming device, or smart home system
Plug-and-Play: No additional software installation required on target devices

Smart Home Ready

Multi-Device Control: Switch between controlling TV, sound system, lights, and other IoT devices
Context-Aware: Automatically adapts gesture mapping based on active device
Real-Time Response: Low-latency gesture recognition for smooth user experience

Technical Advantages

mmWave Radar Technology

Privacy-Friendly: No camera required, protecting user privacy
Lighting Independent: Works in complete darkness or bright light
Weather Resistant: Unaffected by ambient conditions
High Precision: Accurate gesture detection and classification

HID Integration

Standard Protocol: Uses USB HID protocol for universal compatibility
Custom Mapping: Configurable gesture-to-command mapping
Multi-Modal Output: Supports keyboard, mouse, and media control commands

Applications

Smart TV Control	Audio System	Lighting Control
Volume, channel, menu navigation	Play, pause, volume control	On/off, brightness, color

Use Cases

Living Room Entertainment: Control TV, streaming services, and audio systems
Smart Home Automation: Manage lights, curtains, and climate control
Accessibility: Hands-free control for users with mobility limitations
Hygiene-Critical Environments: Touch-free interaction in medical or food service settings

System Architecture

The system consists of:

mmWave Radar Sensor: Captures gesture data in real-time
AI Processing Unit: Classifies gestures using machine learning
HID Interface: Translates gestures to standard device commands
Configuration Software: Allows custom gesture mapping and device profiles

This innovative approach bridges the gap between natural human interaction and smart home technology, making device control more intuitive and accessible than ever before.

MMHTSR: In-air handwriting trajectory sensing and reconstruction based on mmWave radar

Fri, 01 Sep 2023 00:00:00 +0000

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

Create your slides in Markdown - click the Slides button to check out the example.

Supplementary notes can be added here, including code, math, and images.

MMHTSR

Sun, 27 Aug 2023 00:00:00 +0000

Overview

MMHTSR is a framework based on Texas Instruments’ low-cost millimeter-wave radar for aerial trajectory sensing and reconstruction. We will be presenting our research work on this system. Additionally, we have developed a real-time system capable of online reconstruction and accurate recognition of in-air handwritten trajectory characters and gestures, as shown in the animated image below.

Real-time System.

Dataset vs Visualization

Dataset

At the same time, we have made our dataset publicly available, comprising over 10,000 data samples totaling approximately 130GB of raw radar signal data. This dataset encompasses 30 different types of in-air handwritten characters and gestures.

26 uppercase English alphabet symbols and 4 types of gesture actions vs dataset folder.

Dataset link: https://pan.baidu.com/s/1zwzfdnttbouxvKiKfAV6pg?pwd=zjvx code：zjvx

Visualization

In traditional network training and signal processing, writing and debugging code can be a complex and time-consuming task. However, through our platform, you can accomplish these tasks using an intuitive graphical interface, without the need for an in-depth understanding of programming details. This makes it easy even for researchers to engage in network training and signal processing, saving a significant amount of time and effort. Moreover, our platform offers rich visualization capabilities to help you comprehend and analyze your data as well as your model’s performance. These visualization tools provide you with deeper insights, assisting you in making more informed decisions.

Visualization platform.

Application


games	menu	input method

RadarSensing

Sun, 27 Aug 2023 00:00:00 +0000

Overview

RadarSensing is a comprehensive upper computer software platform designed for mmWave radar applications. The system integrates real-time data acquisition, advanced signal processing, time-frequency visualization, vital signs monitoring, and intelligent motion recognition into a unified interface, providing researchers and developers with powerful tools for radar-based sensing applications.

Core Features

Real-Time Data Acquisition

Multi-Radar Support: Compatible with TI AWR series, IWR series, and other mmWave radar platforms
High-Speed Streaming: Real-time data capture at up to 1000+ frames per second
Configurable Parameters: Adjustable chirp configuration, sampling rate, and frame structure
Data Recording: Automatic data logging with timestamp and metadata

Time-Frequency Signal Visualization

Range-Doppler Maps: Real-time heatmap visualization of range and velocity information
Range-Angle Processing: 2D/3D spatial mapping with beamforming algorithms
Spectrogram Analysis: Time-frequency domain representation for signal analysis
Customizable Display: Multiple visualization modes with adjustable color schemes and scaling

Vital Signs Monitoring

Heart Rate Detection: Non-contact cardiac rhythm monitoring with high accuracy
Respiratory Rate: Real-time breathing pattern analysis and rate calculation
Heart Rate Variability (HRV): Advanced cardiac health assessment metrics
Multi-Target Tracking: Simultaneous monitoring of multiple subjects
Medical-Grade Accuracy: Validated against clinical standards for healthcare applications

Motion Recognition & Classification

Gesture Recognition: Real-time hand gesture classification with machine learning
Activity Detection: Human activity recognition (walking, sitting, falling, etc.)
Gait Analysis: Detailed biomechanical movement assessment
Custom Training: User-defined gesture and motion pattern learning
Multi-Class Classification: Support for 50+ predefined motion categories

Technical Architecture

Signal Processing Pipeline

Raw Data Preprocessing: Noise reduction, calibration, and filtering
FFT Processing: Range and Doppler FFT with windowing functions
CFAR Detection: Constant False Alarm Rate target detection
Tracking Algorithms: Kalman filtering for multi-target tracking
Feature Extraction: Advanced signal features for classification

Machine Learning Integration

Deep Learning Models: CNN/RNN architectures for pattern recognition
Real-Time Inference: Optimized models for low-latency processing
Transfer Learning: Pre-trained models adaptable to specific applications
Model Training Tools: Built-in dataset management and training utilities

Hardware Integration

USB/Ethernet Interface: Multiple connection options for radar modules
GPIO Control: External trigger and synchronization support
Multi-Threading: Parallel processing for real-time performance
Memory Management: Efficient buffer handling for continuous operation

Applications

Healthcare Monitoring	Smart Home	Security & Surveillance
Patient monitoring	Occupancy detection	Intrusion detection

Use Cases

Medical Monitoring: Non-contact patient vital signs in hospitals and clinics
Elderly Care: Fall detection and health monitoring for assisted living
Smart Buildings: Occupancy sensing and energy management
Automotive: In-cabin monitoring and driver state assessment
Research & Development: Academic research and algorithm prototyping

Key Advantages

User-Friendly Interface

Intuitive GUI: Modern interface with drag-and-drop configuration
Real-Time Feedback: Instant visualization of processing results
Customizable Layouts: Flexible workspace arrangement for different applications
Export Capabilities: Data export in multiple formats (CSV, MAT, HDF5)

Performance Optimization

GPU Acceleration: CUDA support for intensive signal processing
Multi-Core Processing: Parallel algorithms utilizing all CPU cores
Memory Efficiency: Optimized memory usage for long-term operation
Low Latency: Sub-millisecond processing delays for real-time applications

Extensibility

Plugin Architecture: Modular design for custom algorithm integration
API Support: RESTful API for external system integration
Scripting Interface: Python/MATLAB scripting for automation
Open Source Components: Extensible with community contributions

This comprehensive platform bridges the gap between raw radar data and practical applications, making advanced radar sensing accessible to researchers, developers, and industry professionals.

RadarStream

Tue, 15 Mar 2022 00:00:00 +0000

Overview

RadarStream is a comprehensive real-time raw data acquisition, processing, and visualization system designed for Texas Instruments’ MIMO mmWave radar series. The system features a multi-threaded architecture with C-wrapped data acquisition modules to overcome Python’s GIL limitations, enabling true real-time, frame-loss-free radar data processing.

RadarStream Real-time System Interface

Key Features ✨

Real-time Multi-threaded Data Acquisition 🧵

Multi-threaded Architecture: Separate threads for data acquisition and processing
C-Wrapped Acquisition Module 🚀: Overcomes Python’s Global Interpreter Lock (GIL) for true multi-core processing
Frame-Loss-Free Operation: Near real-time data capture and handling
TI MIMO mmWave Radar Support: Compatible with IWR series radar sensors

Multi-dimensional Feature Extraction 📊

The system extracts comprehensive radar features for advanced signal processing:

RTI - Range-Time Information
DTI - Doppler-Time Information
RDI - Range-Doppler Information
RAI - Range-Azimuth Information
REI - Range-Elevation Information

Interactive Visualization Interface 🖥️

PyQt5-based GUI: Modern, intuitive user interface
Real-time Plotting: PyQtGraph for high-performance visualization
Multi-view Display: Simultaneous visualization of multiple feature dimensions
Configuration Management: Easy radar parameter configuration
Data Recording: Capture training data for machine learning models

Hardware Support

Tested Radar Platforms

The system has been validated with the following TI mmWave radar platforms:

✅ IWR6843ISK - 60 GHz Industrial Radar Sensor
✅ IWR6843ISK-OBS - Out-of-Box Solution variant
✅ IWR1843ISK - 77 GHz Automotive Radar Sensor

Required Hardware

Component	Description
TI mmWave Radar	IWR6843ISK, IWR6843ISK-OBS, or IWR1843ISK
DCA1000 EVM	Essential for raw data capture (ADC data streaming)
Power Supply	5V 3A DC power adapter
Ethernet Cable	For DCA1000 data connection
Micro USB Cable	For radar CLI interface
PC	Windows OS recommended

Connection Steps

Power Connection: Connect 5V 3A DC power to the radar board
Ethernet Connection: Connect DCA1000 EVM to PC via Ethernet cable
USB Connection: Connect radar board to PC via micro USB for CLI
Network Configuration: Configure IPv4 settings for DCA1000 (similar to mmWaveStudio setup)

Platform Comparison

Platform	Performance	Recommendation
Windows PC	✅ Excellent - Full frame rate, no data loss	⭐ Recommended
Raspberry Pi 4B	⚠️ Limited - Low frame rate, prone to data loss	Not recommended for real-time

Software Architecture

Technology Stack

Python 3.6+ - Main programming language
PyQt5 - GUI framework
PyQtGraph - High-performance plotting
NumPy - Numerical computing
PyTorch - Machine learning integration
C Extensions - Performance-critical data acquisition

Project Structure

RadarStream/
├── config/              # Radar configuration files
├── gesture_icons/       # Gesture visualization icons
├── libs/                # Radar communication libraries
├── STL3D/              # 3D printed mount files
├── iwr6843_tlv/        # TLV protocol implementation
├── dsp/                # Digital signal processing modules
├── main.py             # Application entry point
├── real_time_process.py # Real-time processing engine
├── radar_config.py     # Configuration utilities
└── UI_interface.py     # PyQt5 user interface

Multi-threaded Architecture

Main Thread (GUI)
    ↓
Data Acquisition Thread (C-wrapped)
    ↓
Processing Thread (Python)
    ↓
Visualization Thread (PyQtGraph)

Key Design Decisions:

✅ C-wrapped acquisition overcomes GIL limitations
✅ Separate threads prevent GUI blocking
✅ Lock-free queues for inter-thread communication
✅ Efficient memory management for continuous operation

Getting Started

Software Requirements

Install the required Python dependencies:

pip install pyqt5 pyqtgraph numpy torch matplotlib pyserial

Dependency List:

Python 3.6+
PyQt5 - GUI framework
PyQtGraph - Real-time plotting
NumPy - Numerical computing
PyTorch - Machine learning support
Matplotlib - Additional visualization
PySerial - Serial communication

Firmware Requirements

The radar firmware must be selected from the TI mmWave Industrial Toolbox:

Firmware Path: mmwave_industrial_toolbox_4_10_1/labs/Out_Of_Box_Demo/prebuilt_binaries/

Note: Version 4.10.1 is not strictly required - other versions of the mmWave Industrial Toolbox are also compatible.

Installation & Setup

1. Clone the Repository

git clone https://github.com/Tkwer/RadarStream.git
cd RadarStream

2. Hardware Connection

Connect TI mmWave radar sensor to PC via micro USB
Connect DCA1000 EVM to PC via Ethernet cable
Connect 5V 3A DC power supply to radar board
Configure network IPv4 settings for DCA1000

3. Network Configuration

Configure your PC’s Ethernet adapter with static IP settings (similar to mmWaveStudio setup):

IP Address: 192.168.33.30
Subnet Mask: 255.255.255.0
Default Gateway: 192.168.33.1

Usage

1. Launch the Application

python main.py

2. Configure the Radar

Select COM Port: Choose the appropriate COM port for the radar CLI interface
Load Configuration: Select a radar configuration file from the config/ directory
Send Config: Click “Send Config” to initialize the radar with selected parameters

3. Start Data Acquisition

Click “Start” to begin real-time data acquisition and visualization
Use the interface controls to:
- Visualize radar data in real-time across multiple feature dimensions
- Adjust visualization parameters
- Capture training data for machine learning models
- Record data sessions for offline analysis

4. Gesture Recognition (Optional)

Load a pre-trained gesture recognition model
Enable real-time gesture classification
View recognized gestures with visual feedback

Applications

RadarStream enables a wide range of radar-based applications:

Human-Computer Interaction 🖐️

Gesture Recognition: Contactless gesture control for smart devices
Air Writing: Handwriting recognition in 3D space
Sign Language Recognition: Accessibility applications

Motion Analysis 🏃

Activity Recognition: Classify human activities (walking, sitting, falling)
Gait Analysis: Biomechanical movement assessment
Sports Analytics: Performance monitoring and analysis

Smart Home & IoT 🏠

Occupancy Detection: Presence sensing for energy management
Vital Signs Monitoring: Non-contact heart rate and respiration
Fall Detection: Elderly care and safety monitoring

Research & Development 🔬

Algorithm Prototyping: Test new signal processing algorithms
Dataset Collection: Gather training data for machine learning
Radar Education: Teaching tool for radar signal processing

Technical Highlights

Performance Optimization

C-Wrapped Data Acquisition

The critical data acquisition module is implemented in C and wrapped for Python, providing:

True Multi-core Processing: Bypasses Python’s GIL limitation
Near-Zero Latency: Minimal delay between radar and processing
Frame-Loss-Free: Reliable data capture even at high frame rates
Memory Efficient: Optimized buffer management

Real-time Processing Pipeline

Radar Hardware → DCA1000 → Ethernet → C Acquisition Module
                                              ↓
                                    Lock-free Queue
                                              ↓
                            Python Processing Thread
                                              ↓
                                    Feature Extraction
                                              ↓
                            PyQtGraph Visualization

Digital Signal Processing

The dsp/ module provides comprehensive radar signal processing:

Range FFT: Extract range information from ADC data
Doppler FFT: Compute velocity information
Angle Estimation: MUSIC/Bartlett beamforming for angle-of-arrival
CFAR Detection: Constant False Alarm Rate target detection
Clutter Removal: Static clutter filtering
Calibration: Phase and amplitude calibration

Acknowledgment: DSP module references OpenRadar by PreSenseRadar.

Acknowledgements

This project builds upon and references:

real-time-radar by AndyYu0010 - Real-time processing architecture
OpenRadar by PreSenseRadar - DSP module implementation

Special thanks to the open-source radar community for their contributions and support.

Future Development

Planned Improvements 🚀

Validate More RF Boards: Test compatibility with additional TI radar platforms (AWR series, etc.)
Migrate to PySide6: Update from PyQt5 to PySide6 for better licensing and features
Flexible API: Make the libs/ folder API more modular and extensible
Enhanced ML Integration: Add more pre-trained models and training utilities
Cross-platform Support: Improve Linux and macOS compatibility
Performance Profiling: Add built-in performance monitoring tools

Community Contributions

If you encounter any issues or have suggestions for improvements, please:

🐛 Report Bugs: Submit issues on GitHub
💡 Feature Requests: Propose new features or enhancements
🔧 Pull Requests: Contribute code improvements
📖 Documentation: Help improve documentation and tutorials

RadarStream provides a powerful, flexible platform for real-time mmWave radar data acquisition and processing, making advanced radar sensing accessible for research, development, and practical applications.

mmWave Radar Near-Field Imaging System

Tue, 15 Jun 2021 00:00:00 +0000

Overview

This project presents a DIY millimeter-wave radar near-field imaging system built from scratch using a custom 2D linear rail platform. By integrating GRBL motion control with radar data acquisition and imaging algorithms into a single compact program, the system achieves vendor-independent operation and rapid prototyping capabilities.

Honest Assessment: While the imaging quality is admittedly subpar due to open-loop motor control and rapid scanning causing step loss, this project served as a valuable learning experience in radar imaging principles and mechatronic system integration.

System Architecture

Hardware Platform

2D Linear Rail System

DIY 2D Linear Rail Imaging Platform

The mechanical platform consists of:

X-Y Linear Rails: Custom-built 2D scanning mechanism
Stepper Motors: NEMA 17 stepper motors for both axes
GRBL Controller: GRBL-based motion control board
mmWave Radar: Mounted on the moving platform for scanning
Frame Structure: Aluminum extrusion frame for rigidity

Motion Control System

GRBL-Based Control

Open-Loop Stepper Control: Cost-effective but prone to step loss
G-Code Commands: Standard CNC control protocol
USB Serial Interface: Direct PC communication
Configurable Speed: Adjustable scanning speed (trade-off: speed vs. accuracy)

Key Limitation: The use of open-loop stepper motors without position feedback means:

⚠️ Rapid scanning can cause step loss
⚠️ No error correction for missed steps
⚠️ Position accuracy degrades over time
⚠️ Vibration and acceleration affect precision

Software Integration

All-in-One Program

One of the key advantages of this DIY approach is vendor independence. Unlike commercial systems requiring proprietary software, this system integrates everything into a single lightweight program.

Benefits:

✅ No dependency on vendor-specific software
✅ Easy to modify and experiment
✅ Lightweight and fast
✅ Complete control over scanning patterns
✅ Integrated data processing pipeline

Imaging Algorithm

Near-Field SAR Processing

Data Collection: Radar samples at each grid point
Range Profile Extraction: FFT processing of raw radar data
Spatial Sampling: 2D grid scanning pattern
Back-Projection Algorithm: Coherent summation for image formation
Image Reconstruction: 2D/3D visualization of targets

Implemented Algorithms:

Range-Doppler processing
Back-projection imaging
Frequency-domain focusing
Basic clutter suppression

Experimental Results

Imaging Performance

Near-Field Imaging Results

Honest Assessment

Image Quality: ⭐⭐☆☆☆ (Poor to Fair)

The imaging results are admittedly subpar due to several factors:

Root Causes of Poor Quality:

Open-Loop Control 🔴
- No position feedback
- Accumulated positioning errors
- Step loss during rapid movements
Speed vs. Accuracy Trade-off 🔴
- Prioritized fast scanning (to reduce total scan time)
- High acceleration causes mechanical vibration
- Insufficient settling time at each position
Mechanical Limitations 🟡
- Frame rigidity not optimal
- Backlash in linear rail system
- Motor vibration during movement
Limited Calibration 🟡
- Basic calibration only
- No real-time position verification
- Environmental factors not compensated

What Worked

Despite the limitations, the system successfully demonstrated:

✅ Basic near-field imaging capability
✅ Automated 2D scanning
✅ Real-time data acquisition and processing
✅ Vendor-independent operation
✅ Rapid prototyping and experimentation

Lessons Learned

This project provided valuable insights into radar imaging systems:

Technical Insights

Position Accuracy is Critical 🎯
- Imaging quality is directly tied to positioning precision
- Open-loop control is insufficient for high-quality imaging
- Sub-millimeter accuracy needed for good results
Speed-Accuracy Trade-off ⚖️
- Faster scanning reduces total time but degrades quality
- Need to balance throughput with positioning accuracy
- Acceleration profiles matter significantly
System Integration Complexity 🔧
- Synchronizing motion and data acquisition is challenging
- Timing jitter affects image quality
- Need robust error handling
DIY Advantages 💡
- Complete control over system parameters
- Easy to experiment with different algorithms
- Low cost for learning and prototyping
- No vendor lock-in

Future Improvements

Next-Generation Platform (Planned) 🚀

I plan to build an improved desktop-scale imaging platform with:

Mechanical Upgrades

CoreXY Structure 🔄
- Better speed and acceleration
- Reduced moving mass
- Improved positioning accuracy
- Smoother motion profiles

Motion Control Upgrades

Closed-Loop Stepper Motors 🔁
- Real-time position feedback
- Automatic error correction
- Stall detection and recovery
- Precise position verification

Radar Upgrades

Custom DIY Mini Radar Board 📡
- Compact form factor
- Optimized for near-field imaging
- Better integration with platform
- Lower cost and higher flexibility

Software Enhancements

Advanced imaging algorithms (RMA, Omega-K)
Real-time autofocus
Motion compensation
Enhanced visualization

Expected Improvements

With these upgrades, the next version should achieve:

📈 10x better positioning accuracy (closed-loop control)
📈 5x faster scanning (CoreXY kinematics)
📈 Significantly improved image quality (better hardware + algorithms)
📈 More reliable operation (error detection and correction)

Stay Tuned! 🎬

The next-generation platform is in the planning stages. Follow this space for updates on the improved desktop mmWave imaging system!

Applications

Despite current limitations, near-field radar imaging has promising applications:

🔍 Non-Destructive Testing: Detecting defects in materials
📦 Security Screening: Concealed object detection
🏥 Medical Imaging: Complementary to other modalities
🔬 Research & Education: Learning radar imaging principles
🛠️ Prototyping: Testing imaging algorithms

Technical Specifications

Current System

Component	Specification
Scanning Area	300mm × 300mm
Position Resolution	~1mm (theoretical, degraded in practice)
Scanning Speed	50-100 mm/s
Radar Frequency	77 GHz (mmWave)
Range Resolution	~5 cm
Total Scan Time	5-10 minutes (full area)
Control Interface	USB Serial (GRBL)
Software	Custom Python/C++ program

Conclusion: This project represents an honest exploration of DIY radar imaging. While the results fell short of commercial systems, the learning experience and system integration knowledge gained were invaluable. The next iteration will address the identified shortcomings with better hardware and control strategies.

Radar-Camera Fusion

Thu, 27 Aug 2020 00:00:00 +0000

Overview

Radar-Camera Fusion is an early-stage research project from my master’s studies, exploring multi-sensor fusion techniques for enhanced target detection and tracking. This project implements a late fusion strategy that combines the complementary strengths of mmWave radar and camera sensors to achieve robust object localization and tracking in various scenarios.

Note: This was one of my initial research explorations during my master’s program. While I may not recall every implementation detail, the core methodology and system architecture remain documented here.

System Architecture

The system implements a late fusion strategy where radar and camera data are processed independently before being fused at the decision level. This approach leverages the strengths of both sensors:

Camera: Provides rich visual information for object classification
mmWave Radar: Offers accurate range, velocity, and robust performance in adverse conditions

Fusion Pipeline

Camera Stream               Range/Azimuth Extraction
     ↓                                ↓
  ROI Crop                  Point Cloud Processing
     ↓                                ↓
YOLO Detection               Radar Point Cloud
     ↓                                ↓
DeepSORT Tracking                      ↓
     ↓                                ↓
Monocular Ranging                      ↓
     ↓                                ↓
     └────→ Data Fusion ←────┘
              ↓
      Fused Target Info

Core Components

1. Camera Processing Pipeline

ROI Extraction

Image Preprocessing: Frame capture and region of interest cropping
Adaptive ROI: Dynamic region selection based on scene context
Resolution Optimization: Balanced processing speed and detection accuracy

YOLO Object Detection

Real-time Detection: YOLOv5 for fast and accurate object detection
Multi-Class Recognition: Supports pedestrians, vehicles, and other objects
Bounding Box Output: Precise object localization in image coordinates

DeepSORT Tracking

Multi-Object Tracking: Maintains consistent IDs across frames
Appearance Features: Deep learning-based appearance descriptor
Kalman Filtering: Smooth trajectory prediction and association
ID Management: Handles occlusions and re-identification

Monocular Distance Estimation

Pinhole Camera Model: Geometric-based distance calculation
Calibration: Camera intrinsic parameter calibration
Height-Based Ranging: Estimates distance using object height and camera parameters
Azimuth Calculation: Computes horizontal angle from image coordinates

2. Radar Processing Pipeline

Point Cloud Generation

mmWave Radar: Processes raw radar data to extract point clouds
Range-Azimuth-Elevation: 3D spatial information for each detection
Velocity Information: Doppler-based velocity measurement
SNR Filtering: Signal quality-based point filtering

Target Extraction

Clustering: Groups radar points into distinct targets
Centroid Calculation: Computes target center position
Velocity Estimation: Extracts radial velocity for each target

3. Late Fusion Strategy

Spatial Alignment

Coordinate Transformation: Converts camera and radar coordinates to common reference frame
Calibration Matrix: Extrinsic calibration between camera and radar
Temporal Synchronization: Aligns timestamps between sensors

Data Association

Position Matching: Associates camera detections with radar points based on spatial proximity
Gating Threshold: Defines acceptable matching distance
Confidence Weighting: Combines detection confidences from both sensors

Fused Output

Enhanced Localization: Improved position accuracy by combining both sensors
Velocity Information: Radar-provided velocity enriches camera detections
Robust Detection: Maintains tracking even when one sensor fails

Technical Highlights

Advantages of Late Fusion

✅ Modularity: Independent processing allows easy sensor replacement or upgrade
✅ Robustness: System continues functioning if one sensor fails
✅ Flexibility: Can adjust fusion weights based on environmental conditions
✅ Interpretability: Clear understanding of each sensor’s contribution

Challenges Addressed

Calibration Complexity: Precise spatial and temporal alignment between sensors
Coordinate Transformation: Accurate mapping between different sensor coordinate systems
Association Ambiguity: Resolving which radar points correspond to which camera detections
Computational Efficiency: Real-time processing of dual sensor streams

Applications

The radar-camera fusion system is particularly well-suited for scenarios requiring both visual recognition and precise ranging:

Autonomous Driving

Pedestrian Detection: Enhanced detection accuracy in various lighting conditions
Vehicle Tracking: Robust tracking with velocity information
Obstacle Avoidance: Reliable distance measurement for path planning

Intelligent Surveillance

Perimeter Security: Combined visual identification and range verification
Intrusion Detection: Multi-modal confirmation reduces false alarms
Crowd Monitoring: Track multiple targets with unique identities

Smart Transportation

Traffic Flow Analysis: Vehicle counting and speed measurement
Parking Management: Occupancy detection with vehicle classification
Intersection Monitoring: Multi-target tracking at complex scenarios

Robotics

Navigation: Obstacle detection and avoidance
Human-Robot Interaction: Safe distance maintenance and gesture recognition
Object Manipulation: Precise localization for grasping tasks

System Demonstration

Radar-Camera Fusion System in Action

The demonstration shows the system’s ability to:

Detect and classify objects using camera-based YOLO
Track multiple targets with DeepSORT
Fuse camera-derived positions with radar point clouds
Provide accurate range and velocity information

Technical Specifications

Camera Module

Resolution: 1920×1080 (Full HD)
Frame Rate: 30 FPS
Field of View: 60° horizontal
Detection Network: YOLOv5
Tracking Algorithm: DeepSORT

Radar Module

Frequency: 77 GHz mmWave
Range Resolution: 0.1-0.2 m
Velocity Range: ±50 m/s
Angular Resolution: 15° (azimuth)
Update Rate: 10-20 Hz

Fusion Performance

Latency: < 100 ms (end-to-end)
Position Accuracy: ±0.3 m (fused)
Tracking Range: 0.5-50 m
Max Targets: 10+ simultaneous objects

Lessons Learned

This early research project provided valuable insights into multi-sensor fusion:

✅ Complementary Strengths: Camera excels at classification, radar at ranging
✅ Calibration is Critical: Accurate sensor alignment is essential for fusion
✅ Late Fusion Trade-offs: Simpler implementation but may miss low-level correlations
✅ Real-time Challenges: Synchronization and computational efficiency are key

Future Directions

While this project explored late fusion, modern approaches might consider:

Early Fusion: Fusing raw sensor data for richer feature extraction
Deep Learning Fusion: End-to-end neural networks for joint processing
Adaptive Fusion: Dynamic weighting based on environmental conditions
Additional Sensors: Incorporating LiDAR or thermal cameras

Slides

Tue, 05 Feb 2019 00:00:00 +0000

Create slides in Markdown with Wowchemy

Wowchemy | Documentation

Features

Efficiently write slides in Markdown
3-in-1: Create, Present, and Publish your slides
Supports speaker notes
Mobile friendly slides

Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export

Code Highlighting

Inline code: variable

Code block:

porridge = "blueberry"
if porridge == "blueberry":
    print("Eating...")

Math

In-line math: $x + y = z$

Block math:

$$ f\left( x \right) = ;\frac{{2\left( {x + 4} \right)\left( {x - 4} \right)}}{{\left( {x + 4} \right)\left( {x + 1} \right)}} $$

Fragments

Make content appear incrementally

{{% fragment %}} One {{% /fragment %}}
{{% fragment %}} **Two** {{% /fragment %}}
{{% fragment %}} Three {{% /fragment %}}

Press Space to play!

One Two Three

A fragment can accept two optional parameters:

class: use a custom style (requires definition in custom CSS)
weight: sets the order in which a fragment appears

Speaker Notes

Add speaker notes to your presentation

{{% speaker_note %}}

- Only the speaker can read these notes
- Press `S` key to view
  {{% /speaker_note %}}

Press the S key to view the speaker notes!

Themes

black: Black background, white text, blue links (default)
white: White background, black text, blue links
league: Gray background, white text, blue links
beige: Beige background, dark text, brown links
sky: Blue background, thin dark text, blue links

night: Black background, thick white text, orange links
serif: Cappuccino background, gray text, brown links
simple: White background, black text, blue links
solarized: Cream-colored background, dark green text, blue links

Custom Slide

Customize the slide style and background

{{< slide background-image="/media/boards.jpg" >}}
{{< slide background-color="#0000FF" >}}
{{< slide class="my-style" >}}

Custom CSS Example

Let’s make headers navy colored.

Create assets/css/reveal_custom.css with:

.reveal section h1,
.reveal section h2,
.reveal section h3 {
  color: navy;
}

Questions?

Ask

Documentation

An example conference paper

Mon, 01 Jul 2013 00:00:00 +0000

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

Create your slides in Markdown - click the Slides button to check out the example.

Supplementary notes can be added here, including code, math, and images.

Mon, 01 Jan 0001 00:00:00 +0000