--- license: mit license_link: LICENSE library_name: openvino pipeline_tag: object-detection tags: - openvino - yolo26 - detr - heatmap - speed - traffic - tracking - dlstreamer language: - en --- # Heatmap Generation | Property | Value | |---|---| | **Category** | Object Detection + Speed Heatmap Aggregation | | **Base Model** | [YOLO26](https://docs.ultralytics.com/models/yolo26/) (Ultralytics, default) or [DETR](https://huggingface.co/facebook/detr-resnet-50) (Facebook, alternative) | | **Source Framework** | PyTorch (Ultralytics, Hugging Face Transformers) | | **Supported Precisions** | FP32, FP16, INT8 (mixed-precision) | | **Inference Engine** | OpenVINO | | **Hardware** | CPU, GPU, NPU | | **Detected Class(es)** | All 80 COCO classes (heatmap colored by object speed) | --- ## Overview Heatmap Generation is a Metro Analytics use case that detects objects across video frames and colors each region of the scene by how fast traffic moves through it. It is built on [YOLO26](https://docs.ultralytics.com/models/yolo26/), a state-of-the-art real-time object detector, quantized to INT8 for efficient inference on Intel hardware. Each detection's per-frame displacement is used as a speed estimate, deposited over the object's footprint and averaged per location with Gaussian smoothing into a color-coded overlay. The overlay uses the following color scheme: - **Red** -- fast-moving traffic. - **Yellow / green** -- medium speed. - **Blue** -- slow-moving or stationary traffic. Typical Metro deployments include: - **Traffic Speed Mapping** -- highlight fast corridors and slow/congested lanes. - **Congestion Detection** -- surface persistently slow (blue) areas for safety planning. - **Pedestrian Flow Analysis** -- compare fast throughways against lingering areas. - **Incident Spotting** -- flag unusually fast or stalled movement. Available variants: `yolo26n`, `yolo26s`, `yolo26m`, `yolo26l`. Smaller variants (`yolo26n`, `yolo26s`) are recommended for high-FPS edge deployment; larger variants improve recall for small or distant objects. ### Model Selection [DETR](https://huggingface.co/facebook/detr-resnet-50) (DEtection TRansformer) is available as an alternative to YOLO26. DETR is a transformer-based, NMS-free detector trained on COCO and released under the Apache-2.0 license. The export script embeds DETR normalization, box decoding, and the class remap into the OpenVINO IR, so the DETR model produces the same output layout and class ids as YOLO26. The same OpenVINO and DLStreamer samples run either model; select one with the `MODEL` constant at the top of each sample. | Model | Variants | Input Size | Output | License | |---|---|---|---|---| | YOLO26 (default) | `yolo26n`, `yolo26s`, `yolo26m`, `yolo26l` | 640 x 640 | `[1, 300, 6]` | AGPL-3.0 | | DETR | `detr-resnet-50` | 800 x 800 | `[1, 100, 6]` | Apache-2.0 | Choose YOLO26 for maximum throughput on edge devices. Choose DETR when a permissively licensed transformer detector is preferred; it is heavier and runs at a lower frame rate. --- ## Prerequisites See the [released runtime setup](../README.md#released-runtimes) and [device support matrix](../DEVICE_SUPPORT_MATRIX.md) before choosing a runtime. The previous validation used the RC1 development stack. - Python 3.11+ Create and activate a Python virtual environment before running the scripts: ```bash python3 -m venv .venv --system-site-packages source .venv/bin/activate ``` > **Note:** The `--system-site-packages` flag is required so the virtual > environment can access the system-installed OpenVINO and DLStreamer Python > packages. --- ## Getting Started ### Download and Quantize Model Run the provided script to download, export to OpenVINO IR, and optionally quantize: ```bash wget https://huggingface.co/Intel/heatmap-generation/resolve/main/export_and_quantize.sh chmod +x export_and_quantize.sh ./export_and_quantize.sh ``` This exports the default **yolo26n** model in **FP16** precision. #### Optional: Select a Different Model, Variant, or Precision ```bash ./export_and_quantize.sh yolo26n FP32 # full-precision ./export_and_quantize.sh yolo26n INT8 # quantized ./export_and_quantize.sh yolo26s # larger variant, default FP16 ./export_and_quantize.sh detr-resnet-50 # DETR alternative, default FP16 ./export_and_quantize.sh detr-resnet-50 INT8 # DETR quantized ``` The first argument selects the model: any YOLO26 variant (`yolo26n`, `yolo26s`, `yolo26m`, `yolo26l`) or DETR (`detr-resnet-50`). The script performs the following steps: 1. Installs dependencies (`openvino`, `ultralytics` for YOLO26 or `transformers`, `torch` for DETR; adds `nncf` for INT8). 2. Downloads a sample test image (`test.jpg`) and a sample test video (`test_video.mp4`). 3. Downloads the PyTorch weights and exports to OpenVINO IR. 4. *(INT8 only)* Quantizes the model using NNCF post-training quantization. Output files (replace `yolo26n` with the selected model, for example `detr-resnet-50`): - `yolo26n_openvino_model/` -- FP32 or FP16 OpenVINO IR model directory. - `yolo26n_heatmap_int8.xml` / `.bin` -- INT8 quantized model *(only when `INT8` is selected)*. #### Precision / Device Compatibility | Precision | CPU | GPU | NPU | |---|---|---|---| | FP32 | Yes | Yes | No | | FP16 | Yes | Yes | Yes | | INT8 | Yes | Yes | Yes | ### OpenVINO Sample The sample below runs YOLO26 inference on a video, estimates each object's speed from its per-frame displacement, and writes a speed-colored heatmap overlay (red = fast, blue = slow) to `output_openvino.mp4`. It also saves the final speed heatmap as `heatmap.jpg`. Change the `device` string to run on CPU, GPU, or NPU. The full sample is in [openvino_sample.py](openvino_sample.py); run it with `python3 openvino_sample.py` or copy the code below. ```python # SPDX-License-Identifier: MIT # Copyright (C) Intel Corporation # # Heatmap Generation: OpenVINO Runtime sample. # See README.md for prerequisites and inputs. Run: python3 openvino_sample.py import cv2 import numpy as np import openvino as ov CONF_THRESHOLD = 0.4 # Select the model: "yolo26n" (default) or "detr-resnet-50". MODEL = "yolo26n" INPUT_SIZE = 800 if MODEL.startswith("detr") else 640 HEATMAP_ALPHA = 0.55 MATCH_DIST = 80.0 # max px between frames to treat detections as the same object MAX_SPEED = 20.0 # px/frame that maps to full red def render_speed_heatmap(frame, speed_sum, count, alpha): """Color traffic by average speed: blue = slow, yellow = medium, red = fast. Only regions where vehicles were seen are tinted, so empty background keeps its original color.""" avg = np.zeros_like(speed_sum) seen = count > 0 avg[seen] = speed_sum[seen] / count[seen] avg = cv2.GaussianBlur(avg, (0, 0), sigmaX=15) presence = cv2.GaussianBlur(seen.astype(np.float32), (0, 0), sigmaX=15) norm = np.clip(avg / MAX_SPEED, 0, 1) # 0 = slow (blue), 1 = fast (red) color = cv2.applyColorMap((norm * 255).astype(np.uint8), cv2.COLORMAP_JET) weight = (np.clip(presence, 0, 1) * alpha)[..., np.newaxis] overlay = frame.astype(np.float32) * (1 - weight) + color.astype(np.float32) * weight return overlay.astype(np.uint8), color core = ov.Core() model = core.read_model(f"{MODEL}_openvino_model/{MODEL}.xml") # Change device to "GPU" or "NPU" to run on integrated GPU or NPU. compiled = core.compile_model(model, "CPU") cap = cv2.VideoCapture("test_video.mp4") fps = cap.get(cv2.CAP_PROP_FPS) or 30.0 width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) writer = cv2.VideoWriter( "output_openvino.mp4", cv2.VideoWriter_fourcc(*"mp4v"), fps, (width, height)) speed_sum = np.zeros((height, width), dtype=np.float32) count = np.zeros((height, width), dtype=np.float32) prev_centroids = [] heatmap_color = None frame_idx = 0 total_dets = 0 while True: ok, frame = cap.read() if not ok: break frame_idx += 1 h0, w0 = frame.shape[:2] sx, sy = w0 / INPUT_SIZE, h0 / INPUT_SIZE blob = cv2.resize(frame, (INPUT_SIZE, INPUT_SIZE)) blob = cv2.cvtColor(blob, cv2.COLOR_BGR2RGB).astype(np.float32) / 255.0 blob = blob.transpose(2, 0, 1)[np.newaxis, ...] output = compiled([blob])[compiled.output(0)][0] dets = output[output[:, 4] >= CONF_THRESHOLD] total_dets += len(dets) cur_centroids = [] for det in dets: x1, y1 = int(det[0] * sx), int(det[1] * sy) x2, y2 = int(det[2] * sx), int(det[3] * sy) x1, x2 = max(0, x1), min(width, x2) y1, y2 = max(0, y1), min(height, y2) cx, cy = (x1 + x2) / 2, (y1 + y2) / 2 cur_centroids.append((cx, cy)) # Speed = displacement from the nearest detection in the previous frame. speed = 0.0 if prev_centroids: d = min(np.hypot(cx - px, cy - py) for px, py in prev_centroids) if d <= MATCH_DIST: speed = d speed_sum[y1:y2, x1:x2] += speed count[y1:y2, x1:x2] += 1.0 prev_centroids = cur_centroids overlay, heatmap_color = render_speed_heatmap(frame, speed_sum, count, HEATMAP_ALPHA) cv2.putText(overlay, f"Detections: {len(dets)}", (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (255, 255, 255), 2) writer.write(overlay) cap.release() writer.release() if heatmap_color is not None: cv2.imwrite("heatmap.jpg", heatmap_color) print("Saved: heatmap.jpg") print(f"Processed {frame_idx} frames, {total_dets} total detections", flush=True) ``` **Device targets:** - `"CPU"` -- default, works on all Intel platforms. - `"GPU"` -- Intel integrated or discrete GPU. - `"NPU"` -- Intel NPU (validate with `benchmark_app -d NPU`). #### Expected Output ![OpenVINO expected output](expected_output_openvino.gif) ### DLStreamer Sample The pipeline below runs the FP16 YOLO26 detector via `gvadetect`. A buffer probe estimates each object's speed from its per-frame displacement and overlays a speed-colored heatmap (red = fast, blue = slow) on each frame before encoding to `output_dlstreamer.mp4`. > **Notes on running this sample:** > > - Use the FP16 IR (`yolo26n_openvino_model/yolo26n.xml` or `detr-resnet-50_openvino_model/detr-resnet-50.xml`). Class names are > read automatically from the model's embedded `metadata.yaml` by > DLStreamer 2026.0+ -- no external `labels-file` is required. > - Export `PYTHONPATH` so the DLStreamer Python module is importable: > > ```bash > source /opt/intel/openvino_2026/setupvars.sh > source /opt/intel/dlstreamer/scripts/setup_dls_env.sh > export PYTHONPATH=/opt/intel/dlstreamer/python:\ > /opt/intel/dlstreamer/gstreamer/lib/python3/dist-packages:${PYTHONPATH:-} > ``` The full sample is in [dlstreamer_sample.py](dlstreamer_sample.py); run it with `python3 dlstreamer_sample.py` or copy the code below. ```python # SPDX-License-Identifier: MIT # Copyright (C) Intel Corporation # # Heatmap Generation: DLStreamer pipeline sample. # See README.md for prerequisites and inputs. Run: python3 dlstreamer_sample.py import gi gi.require_version("Gst", "1.0") gi.require_version("GstAnalytics", "1.0") from gi.repository import Gst, GLib, GstAnalytics import numpy as np Gst.init([]) # Import cv2 after Gst.init to avoid GStreamer re-initialization conflicts. import cv2 INPUT_VIDEO = "test_video.mp4" HEATMAP_ALPHA = 0.55 MATCH_DIST = 80.0 # max px between frames to treat detections as the same object MAX_SPEED = 20.0 # px/frame that maps to full red def render_speed_heatmap(frame, speed_sum, count, alpha): """Color traffic by average speed: blue = slow, yellow = medium, red = fast. Only regions where vehicles were seen are tinted, so empty background keeps its original color.""" avg = np.zeros_like(speed_sum) seen = count > 0 avg[seen] = speed_sum[seen] / count[seen] avg = cv2.GaussianBlur(avg, (0, 0), sigmaX=15) presence = cv2.GaussianBlur(seen.astype(np.float32), (0, 0), sigmaX=15) norm = np.clip(avg / MAX_SPEED, 0, 1) # 0 = slow (blue), 1 = fast (red) color = cv2.applyColorMap((norm * 255).astype(np.uint8), cv2.COLORMAP_JET) weight = (np.clip(presence, 0, 1) * alpha)[..., np.newaxis] overlay = frame.astype(np.float32) * (1 - weight) + color.astype(np.float32) * weight return overlay.astype(np.uint8) # Select the model: "yolo26n" (default) or "detr-resnet-50". MODEL = "yolo26n" # For CPU: change device=GPU to device=CPU. # For NPU: change device=GPU to device=NPU (batch-size=1, nireq=4 recommended). pipeline_str = ( f"filesrc location={INPUT_VIDEO} ! decodebin3 ! " "videoconvert ! video/x-raw,format=BGR ! " f"gvadetect model={MODEL}_openvino_model/{MODEL}.xml " "device=GPU " "threshold=0.4 ! queue ! " "appsink name=sink emit-signals=false sync=false" ) pipeline = Gst.parse_launch(pipeline_str) sink = pipeline.get_by_name("sink") pipeline.set_state(Gst.State.PLAYING) speed_sum = None count = None prev_centroids = [] writer = None frame_idx = 0 total_dets = 0 while True: sample = sink.emit("pull-sample") if sample is None: break buf = sample.get_buffer() caps = sample.get_caps().get_structure(0) width = caps.get_value("width") height = caps.get_value("height") if speed_sum is None: speed_sum = np.zeros((height, width), dtype=np.float32) count = np.zeros((height, width), dtype=np.float32) ok, mapinfo = buf.map(Gst.MapFlags.READ) if not ok: continue frame = np.ndarray((height, width, 3), dtype=np.uint8, buffer=mapinfo.data).copy() buf.unmap(mapinfo) frame_idx += 1 rmeta = GstAnalytics.buffer_get_analytics_relation_meta(buf) cur_centroids = [] det_count = 0 if rmeta is not None: idx = 1 while True: ok_od, od = rmeta.get_od_mtd(idx) if not ok_od: break _, x, y, w, h, _ = od.get_location() x1, y1 = max(0, int(x)), max(0, int(y)) x2, y2 = min(width, int(x + w)), min(height, int(y + h)) cx, cy = (x1 + x2) / 2, (y1 + y2) / 2 cur_centroids.append((cx, cy)) # Speed = displacement from the nearest detection last frame. speed = 0.0 if prev_centroids: d = min(np.hypot(cx - px, cy - py) for px, py in prev_centroids) if d <= MATCH_DIST: speed = d speed_sum[y1:y2, x1:x2] += speed count[y1:y2, x1:x2] += 1.0 det_count += 1 idx += 1 prev_centroids = cur_centroids total_dets += det_count overlay = render_speed_heatmap(frame, speed_sum, count, HEATMAP_ALPHA) if writer is None: writer = cv2.VideoWriter( "output_dlstreamer.mp4", cv2.VideoWriter_fourcc(*"mp4v"), 30.0, (width, height)) writer.write(overlay) print(f"Frame {frame_idx}: detections={det_count}", flush=True) pipeline.set_state(Gst.State.NULL) if writer: writer.release() print(f"Processed {frame_idx} frames, {total_dets} total detections", flush=True) ``` **Device targets:** - `device=GPU` -- default in the sample code. - `device=CPU` -- change `device=GPU` to `device=CPU`. - `device=NPU` -- change `device=GPU` to `device=NPU`; use `batch-size=1` and `nireq=4` for best NPU utilization. #### Expected Output ![DLStreamer expected output](expected_output_dlstreamer.gif) --- ## License Licensed under the MIT License. See [LICENSE](LICENSE) for details. ## References - [YOLO26 Documentation](https://docs.ultralytics.com/models/yolo26/) - [DETR Model Card (facebook/detr-resnet-50)](https://huggingface.co/facebook/detr-resnet-50) - [End-to-End Object Detection with Transformers (DETR paper)](https://arxiv.org/abs/2005.12872) - [Ultralytics Heatmap Guide](https://docs.ultralytics.com/guides/heatmaps/) - [OpenCV Color Maps](https://docs.opencv.org/4.x/d3/d50/group__imgproc__colormap.html) - [OpenVINO Documentation](https://docs.openvino.ai/) - [Intel DLStreamer](https://docs.openedgeplatform.intel.com/dev/edge-ai-libraries/dlstreamer/index.html)