scene_service.ingest.perception_concept_graphs

ConceptGraphs perception and persistent object-map integration.

Each tick runs the following pipeline:

• YOLO-World detects open-vocabulary objects.

• MobileSAM creates bounding-box-prompted masks.

• OpenCLIP produces one visual feature per detection.

• Depth, masks, camera intrinsics, and pose produce map-frame point clouds.

• Spatial and visual similarities associate detections with stored objects.

• Matched detections update objects; unmatched detections create new objects.

This replaces the earlier class-and-distance-only association path, which could not use visual features and frequently over-segmented objects.

The persistent MapObjectList is the source of truth. Every tick we project it back into scene’s ObjectRegistry so the existing web UI / MCP API keep working without changes.

Periodic cleanup runs denoise_objects + merge_overlap_objects from concept-graphs to GC duplicates that escape the per-tick merge.

Classes

ConceptGraphsDetector(*, rgb_fetcher_msg, ...)

Per-frame detector that runs the canonical concept-graphs merge pipeline.

class scene_service.ingest.perception_concept_graphs.ConceptGraphsDetector(*, rgb_fetcher_msg: Callable[[], Any | None], depth_fetcher_msg: Callable[[], Any | None], camera_info_fetcher: Callable[[], _CamIntrinsics | None], chassis_pose_fn: Callable[[], tuple[float, float, float, float] | None], on_detections: Callable[[list[Detection]], Awaitable[None]], registry: ObjectRegistry, world_frame_fn: Callable[[], str] | None = None, period_s: float = 0.6, confidence_threshold: float = 0.3, camera_height_m: float = 1.1, max_detections: int = 30, yolo_weights_path: str | None = None, sam_weights_path: str | None = None, clip_model_name: str | None = None, clip_pretrained: str | None = None, cfg_overrides: dict | None = None, hub: Any = None, camera_frame: str = 'camera_optical_frame', base_frame: str | None = None)

Bases: object

Per-frame detector that runs the canonical concept-graphs merge pipeline. Owns a persistent MapObjectList and projects it into scene’s ObjectRegistry once per tick so the web UI/MCP API stay consistent.

on_detections is kept for backwards compat but no longer used — we write to the registry directly via _project_to_registry. The caller still passes a registry via this side channel.

embed_text(texts: list[str]) → list[list[float]] | None

Encode texts with the already-loaded open_clip text encoder, L2-normalized to match the per-object image features. Returns one 512-d vector per input, or None when CLIP isn’t loaded (models unavailable / detector not started) so the caller can fall back.

Reuses the model loaded in start() — no second model instance. Runs synchronously under torch.no_grad on self._device.

export_3d_snapshot(max_points_per_obj: int = 256) → dict

latest_frame_bundle()

Return (rgb_bgr, K, T_cam_map) for projecting map-frame points into the current camera image, or None when any piece is unavailable.

Consumed by the scene-graph builder’s image-grounded relation pass. Reads the latest RGB frame, intrinsics, and camera→map transform WITHOUT holding _inference_lock — same rationale as export_3d_snapshot: the lock is held by the worker tick for ~100 ms of YOLO/SAM and the asyncio-loop caller must not block on it. A one-tick mismatch between frame and transform is immaterial (poses move slowly relative to the 30 s rebuild cadence).

async start() → None

async stop() → None