Object detection¶

Detect objects tile by tile and store the boxes as one ROI table.

Not every model produces a segmentation mask. A detector — a YOLO network, a spot finder — reports bounding boxes, and the natural home for those in an OME-Zarr container is a ROI table. The ObjectDetectionIterator runs a detector one tile at a time and owns everything around it: world coordinates, cross-tile deduplication, and one table at the end. Here the "model" is a Laplacian-of-Gaussian spot finder from skimage, so the whole tutorial runs without any ML dependency.

Step 1: write the detector function¶

The detector sees one tile's pixels and answers with a list of Roi boxes in the tile's own pixel coordinates (space="pixel"); the iterator anchors them into world coordinates. Any extra field — here the peak intensity as a confidence — rides along into the final table.

import math

import numpy as np
from skimage.feature import blob_log

from ngio import Roi


def find_nuclei(patch: np.ndarray) -> list[Roi]:
    """A spot detector: Laplacian-of-Gaussian blobs, as bounding boxes.

    The boxes are `Roi` objects in the patch's own pixel coordinates
    (`space="pixel"`); the iterator anchors them into world coordinates.
    Every extra field — here the peak intensity as a confidence — rides
    along into the table.
    """
    blobs = blob_log(patch, min_sigma=2, max_sigma=6, threshold=0.05)
    boxes = []
    for y, x, sigma in blobs:
        radius = sigma * math.sqrt(2)
        x_min, y_min = max(0.0, x - radius), max(0.0, y - radius)
        boxes.append(
            Roi.from_values(
                slices={
                    "x": (x_min, x + radius - x_min),
                    "y": (y_min, y + radius - y_min),
                },
                name=None,
                space="pixel",
                confidence=float(patch[int(y), int(x)]),
            )
        )
    return boxes

Step 2: create the OME-Zarr image¶

import skimage

from ngio import create_ome_zarr_from_array

data = skimage.data.human_mitosis().astype("float32") / 255.0
ome_zarr = create_ome_zarr_from_array(
    store="./data/human_mitosis_detection.zarr",
    array=data,
    pixelsize=0.1,  # Just a guess
    consolidation_mode="auto",
    overwrite=True,
)
image = ome_zarr.get_image()
print(ome_zarr)

OmeZarrContainer(levels=5)

Step 3: detect, tile by tile¶

The image is tiled with by_grid plus a with_halo read margin — Step 4 shows why; the full mechanics are in the iterators guide.

from ngio.iterators import GreedyNms, ObjectDetectionIterator, ThreadedMapper

iterator = (
    ObjectDetectionIterator(image)
    .with_nms(GreedyNms(iou_threshold=0.4))
    .by_grid(size_x=128, size_y=128)
    .with_halo(x=16, y=16)
)

detections = iterator.detect(find_nuclei, mapper=ThreadedMapper("auto"))
assert detections is not None  # a serial run always returns the table
ome_zarr.add_table("nuclei_detections", detections, overwrite=True)
print(f"Detected {len(detections.rois())} nuclei")

Detected 364 nuclei

Sanity check: read the table back¶

table = ome_zarr.get_table("nuclei_detections")
print(table_html(table.dataframe.head()))

FieldIndex	x_micrometer	y_micrometer	len_x_micrometer	len_y_micrometer	len_z_micrometer	len_t_second	confidence	label
1	12.39	8.29	0.82	0.82	1.00	1.00	0.62	1
2	12.59	4.79	0.82	0.82	1.00	1.00	0.63	2
3	11.83	12.93	0.94	0.94	1.00	1.00	0.37	3
4	12.50	1.80	1.19	1.19	1.00	1.00	0.32	4
5	8.03	2.73	0.94	0.94	1.00	1.00	0.31	5

The boxes come back as ordinary Roi objects in world coordinates, so drawing them is one to_pixel per box:

from matplotlib.patches import Rectangle

fig, ax = plt.subplots(figsize=(7, 7))
ax.imshow(image.get_as_numpy(), cmap="gray")
for roi in table.rois():
    box = roi.to_pixel(pixel_size=image.pixel_size)
    ax.add_patch(
        Rectangle(
            (box["x"].start, box["y"].start),
            box["x"].length,
            box["y"].length,
            fill=False,
            edgecolor="#f4a63a",
            linewidth=0.8,
        )
    )
ax.axis("off")
print(figure_html(fig, alt="Every detected nucleus outlined on the full image."))

Step 4: watch NMS work¶

The halo solves the boundary problem — a nucleus cut by a tile edge is still seen whole by at least one neighbour — at the price that both neighbours report it. Non-maximum suppression settles that: of two boxes overlapping at or above iou_threshold, only the higher-scoring one survives. You can watch it happen — the raw, pre-NMS view is a loop over the same haloed tiles, anchoring each tile's boxes into world coordinates, which is exactly what detect does before suppressing.

# Iterate the same (haloed) tiles yourself — the view `detect` sees before
# suppression.
raw_boxes = []
for patch, tile in iterator.iter_as_numpy():
    for box in find_nuclei(patch):
        raw_boxes.append(tile.anchor(box, pixel_size=image.pixel_size))

print(f"raw boxes: {len(raw_boxes)}, after NMS: {len(detections.rois())}")

raw boxes: 492, after NMS: 364

Most duplicates coincide exactly — both tiles saw the nucleus whole, reported the same box, and either can survive. The interesting ones sit near a seam, where one tile saw the nucleus clipped at its halo's reach: the boxes disagree, and NMS keeps the one the score ranks higher. Zooming onto a pair of tile boundaries shows both kinds:

def draw_boxes(ax, boxes, title):
    ax.imshow(image.get_as_numpy(), cmap="gray")
    for roi in boxes:
        box = roi.to_pixel(pixel_size=image.pixel_size)
        ax.add_patch(
            Rectangle(
                (box["x"].start, box["y"].start),
                box["x"].length,
                box["y"].length,
                fill=False,
                edgecolor="#f4a63a",
                linewidth=1.0,
            )
        )
    for edge in (256, 384):  # the tile boundaries crossing this zoom
        ax.axvline(edge - 0.5, color="white", ls="--", lw=0.8)
        ax.axhline(edge - 0.5, color="white", ls="--", lw=0.8)
    ax.set_xlim(191.5, 319.5)
    ax.set_ylim(447.5, 319.5)  # imshow's y axis grows downwards
    ax.set_title(title)
    ax.axis("off")


fig, (ax_raw, ax_kept) = plt.subplots(1, 2, figsize=(8.6, 4.4))
draw_boxes(ax_raw, raw_boxes, f"raw: {len(raw_boxes)} boxes")
draw_boxes(ax_kept, detections.rois(), f"after NMS: {len(detections.rois())} boxes")
print(
    figure_html(
        fig,
        alt="A zoom onto two tile boundaries: before suppression several "
        "nuclei near the seams carry two overlapping boxes, one from each "
        "neighbouring tile; after NMS each carries exactly one.",
    )
)

Anchor a local box yourself¶

The coordinate bookkeeping is one call you can also use in a custom flow: Roi.anchor turns a region's ROI plus a patch-local pixel box into the absolute world ROI —

tile_roi = iterator.rois[0]
box = Roi.from_values(slices={"x": (12, 30), "y": (4, 25)}, name=None, space="pixel")
abs_roi = tile_roi.anchor(box, pixel_size=image.pixel_size)
print(abs_roi)

name=None slices=(x: 1.2000000000000002->4.2, y: 0.4->2.9, t: 0.0->1.0, z: 0.0->1.0) label=None space='world'

The space fields are what keep this honest: anchor refuses a world-space box (already absolute — anchoring it would double the offset) and a pixel-space region, so the classic silent frame mix-up raises instead. Axes the box does not pin inherit the region's extent.

Next steps¶

Iterators — the detection contract in full.
Feature extraction — the read-only sibling: measurements into a feature table.
Table specifications — how ROI tables are stored.