Narya: Multi-Sensor Tactical Perimeter Defense

Overview

Narya is a multi-sensor tactical perimeter defense system built for a national security hackathon.

It fuses data from four sensor modalities — LiDAR, thermal, RF, and camera — to detect, classify, and assess threats at a perimeter in real time. Rather than relying on any single sensor, the system combines 3D geometry from an Ouster LiDAR, heat signatures from a FLIR Boson 640, RF emissions from a custom radio module, and visual person detection from YOLO running on a webcam. Each modality fills in where the others fall short.

The full stack runs on an NVIDIA Jetson at the edge. A fine-tuned Qwen2.5 model — trained locally via LoRA on an H100/H200 cluster — handles final threat classification and produces structured output: scene descriptions, per-entity threat levels, recommended actions, and Palantir-ready entity payloads.

---

Demo Video: Watch the demo

GitHub Repo: View the repo

---

What I worked on

• Designed the sensor fusion architecture connecting LiDAR, thermal, RF, and camera streams into a single aggregated data pipeline running on the Jetson edge device.
• Built the FastAPI/aiohttp backend that runs on the Jetson, aggregates all four sensor streams, and exposes them over WebSocket to both the desktop GUI and the web dashboard.
• Developed the PySide6 desktop operator GUI with thermal and LiDAR panels side-by-side, a 3D point cloud viewer, an RF window, and sensor calibration controls.
• Built the React/Vite web dashboard for mission briefing and per-entity object inspection.
• Implemented the PointPillars LiDAR object detection pipeline on the Jetson for real-time 3D point cloud inference.
• Built the ML pipeline that generates synthetic multi-modal scene packs, uses Gemini as a teacher analyst to produce labeled tactical assessments, and fine-tunes Qwen2.5 via LoRA on those labels.
• Integrated RF scanning and drone candidate detection via a custom radio module.
• Wired the model output to Palantir-compatible entity payloads for downstream situational awareness.

Outcome

• Delivered a working end-to-end demo: sensor data flows from hardware into the Jetson, the edge AI classifies the threat, and structured payloads push to Palantir.
• The fine-tuned Qwen2.5 model runs fully locally on the edge — no cloud dependency in the inference loop.
• Demonstrated a complete ISR workflow: detect a person, drone, or vehicle at the perimeter → fuse multi-modal evidence → classify and recommend an action → surface in Palantir for operator situational awareness.
• The multi-modal fusion approach is the core differentiator: a single sensor can be spoofed or miss edge cases; combining LiDAR geometry, thermal heat signatures, RF emissions, and visual detection makes the system significantly more robust.