Skip to content

Instance Segmentation vTool#

The Instance Segmentation vTool classifies objects at a pixel level in your images, i.e., it detects objects and associates each pixel with an object class, based on the model you have selected.

The Instance Segmentation vTool can classify objects in images it receives via the Image input pin. It outputs the results via the InferenceImage, Regions, Classes, BBoxes_px, and Scores output pins.

A bounding box is drawn around each object detected. In contrast to the Semantic Segmentation vTool, instance segmentation allows you to segment individual instances of the same class.

If you use the ROI Creator vTool to specify a region of interest before segmentation, the Instance Segmentation vTool accepts the ROI data via its Roi input pin. In that case, segmentation is only performed on the region of interest. As a side effect, this results in faster processing.

Instance Segmentation vTool

How It Works#

Assume you want to perform instance segmentation on an image to individually identify and segment each person, distinguishing them not only from the background but also from each other.

This is your input image showing six children and a mountain landscape in the background:

Input Image Showing Six Children and Mountain Landscape in Background

You configure the vTool by loading a model that has been trained to perform instance segmentation by identifying and separating individual instances of objects in an image, in this case people.

Bundle for Instance Segmentation Selected

Next, you can select the classes you want to detect and segment. Some models may be tailored exactly for your use case. In that case, all desired classes are already selected.

Select Classes Dialog

After configuring the vTool, you run the recipe that initiates the inference process. The vTool utilizes the model to analyze the image and segment the objects present. When the analysis has finished, the segmentation result is returned. It includes segmented regions, bounding boxes, and labels for each distinct instance detected in the image.

This is the resulting image in the pin data view:

Instance Segmentation with Results Highlighted

Six people have been identified and segmented as persons. The segmented regions are color-coded to distinguish between the individual instances.

Detailed Output

  • Regions: Six regions have been detected. Each object instance in the image is highlighted as a distinct region. These regions are separated from each other and the background, allowing for clear identification of each object.
  • Class: person
    The class label person shows that the model identified the object in the bounding box as a person.
  • Bounding Box: The bounding box is a rectangle drawn around the object detected. For each bounding box, the center pixel, width, height, and rotation are output.
  • Score: The score values, e.g., 0.915855, indicate how confident the model is in its detection, e.g., 91.58 %. Higher scores, closer to 1.0, suggest a higher level of confidence that the detected object is indeed a person.

Configuring the vTool#

To configure the Instance Segmentation vTool:

Instance Segmentation vTool Settings

  1. In the Recipe Management pane in the vTool Settings area, click Open Settings or double-click the vTool.
    The Instance Segmentation dialog opens.
  2. In the Server Selection area, select the server you want to connect to.
    The server must be running to be able to connect to it.
    • Option 1: Enter the IP address and port of your Triton Inference Server manually.
    • Option 2: Select the IP address / host name and port of your server in the drop-down list.
  3. Click Connect to establish a connection.
    The vTool then loads the available models.
  4. In the Model Selection area, select the bundle that contains the desired model in the drop-down list.
    After you have selected a bundle, the information below the drop-down list is populated with the bundle details allowing you to check that you have selected the correct bundle.
    Model Selection Area
  5. If the desired bundle isn't listed, click Deploy New Model in the drop-down list.
    Deploy New Model
    This opens the pylon AI Agent that allows you to deploy more models.
    After you have deployed a new model, you have to click the Refresh button.

  6. In the Detection Settings area, click Select Classes to select the classes you want to classify.
    A dialog with the available classes opens. By default, all classes are selected. You can select as many classes as you like.

    Info

    The parameters available in this area depend on the model selected. Some parameters are only available after selecting a certain model (e.g., NMS IOU Threshold). Default values are provided by the model.

  7. Specify the other options in the detection settings:

    1. Number of objects: You can specify a maximum number of objects to detect or select Unlimited.
      The limit applies to each class individually. If more objects are present in the image than specified here, the matches with the highest score will be detected.
    2. Maximum/Minimum height: Specify appropriate values if you want to filter bounding boxes by height. Because the height of the bounding box corresponds to the height of the object, this allows you to only detect objects of the desired size.
    3. Maximum/Minimum width: Specify appropriate values if you want to filter bounding boxes by width. Because the height of the bounding box corresponds to the width of the object, this allows you to only detect objects of the desired size.
    4. Confidence threshold: Confidence is a coefficient for neural networks which indicates how representative objects in the images are for the classes selected. Higher values mean higher confidence and a reduction of false positives.
    5. NMS IOU Threshold: Some bounding boxes may overlap. With this parameter you can control whether you want to merge the bounding boxes or not. Lower values mean less tolerance, and most bounding boxes will be merged.

  8. Click OK.

You can view the result of the segmentation in a pin data view. Here, you can select which outputs to display.

Inputs#

Image#

Accepts images directly from a Camera vTool or from a vTool that outputs images, e.g., the Image Format Converter vTool.

  • Data type: Image
  • Image format: 8-bit to 16-bit color images

Roi#

Accepts a region of interest from the ROI Creator vTool or any other vTool that outputs rectangles. If multiple rectangles are input, the first one is used as a single region of interest.

  • Data type: RectangleF, RectangleF Array

Outputs#

To use the vTool with Starter vTools, the data types of the output pins can be switched from multiple-value data to single-value data. To do this, use the Data Type Selector parameter in the Features - All pane.

InferenceImage#

Returns an image with bounding boxes drawn around the objects detected for every input image to visualize the results.

  • Data type: Image

Regions#

Returns a region for each class label.

  • Data type: Region Array

Classes#

Returns an n-by-1 vector of predicted class labels. n is the number of classes detected in the image.

  • Data type: String Array

BBoxes_px#

Returns a bounding box for every object detected. Each bounding box is defined by the coordinates of its center pixel, its width, and its height (x_center, y_center, width, height). The bounding box output can be used as input for the Roi input pin of other vTools.

  • Data type: RectangleF Array

Scores#

Returns the confidence scores for each predicted class.

  • Data type: Float Array

Payload#

Returns additional information regarding the performance of the Inference Server for evaluation. This output pin only becomes available if you enable it in the Features - All pane in the Debug Settings parameter group.

  • Data type: String

Typical Predecessors#

Typical Successors#