Third-Party Drivers, Libraries, and Samples#

This page provides information about third-party libraries and sample code for working with Basler Stereo mini cameras.

NumPy#

NumPy is used in the Python samples to process the raw component data returned by pypylon.

pip install numpy

OpenCV#

OpenCV can be used to process the intensity image retrieved from a Stereo mini camera. The intensity component provides an RGBA8 image, which can be converted to a format suitable for OpenCV processing.

pip install opencv-python

Example: Convert intensity component to an OpenCV image

import cv2
import numpy as np
from pypylon import pylon

# ... camera setup (see Python Programmer's Guide) ...

data_container = grab_result.GetDataContainer()
for i in range(data_container.DataComponentCount):
    component = data_container.GetDataComponentByIndex(i)
    try:
        if component.ComponentType == pylon.ComponentType_Intensity:
            raw = component.GetData()
            h, w = component.Height, component.Width
            # Stereo mini intensity is RGBA8
            rgba = np.frombuffer(raw, dtype=np.uint8).reshape(h, w, 4)
            bgr = cv2.cvtColor(rgba, cv2.COLOR_RGBA2BGR)
            cv2.imshow("Intensity", bgr)
            cv2.waitKey(1)
    finally:
        component.Release()

For detailed information, see OpenCV.

Open3D#

Open3D is used in the ShowPointCloud sample to visualize the 3D point cloud from the Stereo mini camera. The range component in Coord3D_ABC32f format provides direct XYZ coordinates.

pip install open3d

Example: Build an Open3D point cloud from range and intensity data

import numpy as np
import open3d as o3d
from pypylon import pylon

# ... camera setup with ComponentSelector Range, PixelFormat Coord3D_ABC32f ...

data_container = grab_result.GetDataContainer()
xyz = None
colors = None

for i in range(data_container.DataComponentCount):
    component = data_container.GetDataComponentByIndex(i)
    try:
        if component.ComponentType == pylon.ComponentType_Range:
            h, w = component.Height, component.Width
            raw = component.GetData()
            pts = np.frombuffer(raw, dtype=np.float32).reshape(h * w, 3)
            valid = np.isfinite(pts).all(axis=1)
            xyz = pts[valid]
        elif component.ComponentType == pylon.ComponentType_Intensity:
            h, w = component.Height, component.Width
            raw = component.GetData()
            rgba = np.frombuffer(raw, dtype=np.uint8).reshape(h * w, 4)
            colors = rgba[..., :3].astype(np.float64) / 255.0
    finally:
        component.Release()

if xyz is not None and colors is not None:
    pcd = o3d.geometry.PointCloud()
    pcd.points = o3d.utility.Vector3dVector(xyz)
    pcd.colors = o3d.utility.Vector3dVector(colors[:len(xyz)])
    o3d.visualization.draw_geometries([pcd])

For detailed information, see Open3D.

Point Cloud Library (PCL)#

PCL can be used for advanced 3D point cloud processing. The range data from the Stereo mini camera in Coord3D_ABC32f format can be fed into PCL data structures.

LinuxWindows

sudo apt-get install libpcl-dev

Download from PCL website.

For detailed information, see the C++ Programmer's Guide.

ROS 2#

Stereo mini cameras can be integrated into ROS 2 by using pypylon as a bridge and publishing sensor_msgs/Image topics.

Required packages:

pip install pypylon numpy

sudo apt install ros-humble-rclpy ros-humble-sensor-msgs

In this approach, image data is read from the Stereo mini data components (Intensity, Range) and published to ROS topics, for example:

/stereo_mini/intensity_rgba8 (encoding rgba8)
/stereo_mini/range_c16_mm (encoding mono16)

Minimal bridge example:

import numpy as np
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from pypylon import pylon


class StereoMiniBridge(Node):
    def __init__(self):
        super().__init__("stereo_mini_bridge")
        self.pub_intensity = self.create_publisher(Image, "/stereo_mini/intensity_rgba8", 10)

        di = pylon.DeviceInfo()
        di.SetDeviceClass("BaslerGTC/Basler/TLModel_keep")
        self.camera = pylon.InstantCamera(pylon.TlFactory.GetInstance().CreateFirstDevice(di))
        self.camera.Open()

        self.camera.ComponentSelector.Value = "Intensity"
        self.camera.ComponentEnable.Value = True
        try:
            self.camera.PixelFormat.Value = "RGBA8"
        except Exception:
            self.camera.PixelFormat.Value = "RGBA8packed"

        self.camera.StartGrabbing()
        self.timer = self.create_timer(0.03, self.tick)

    def tick(self):
        grab = self.camera.RetrieveResult(3000, pylon.TimeoutHandling_ThrowException)
        try:
            if not grab.GrabSucceeded():
                return
            dc = grab.GetDataContainer()
            for i in range(dc.DataComponentCount):
                c = dc.GetDataComponentByIndex(i)
                try:
                    if c.ComponentType == pylon.ComponentType_Intensity:
                        h, w = c.Height, c.Width
                        rgba = np.frombuffer(c.GetData(), dtype=np.uint8).reshape(h, w, 4)
                        msg = Image()
                        msg.header.stamp = self.get_clock().now().to_msg()
                        msg.header.frame_id = "stereo_mini"
                        msg.height = h
                        msg.width = w
                        msg.encoding = "rgba8"
                        msg.is_bigendian = 0
                        msg.step = w * 4
                        msg.data = rgba.tobytes()
                        self.pub_intensity.publish(msg)
                finally:
                    c.Release()
        finally:
            grab.Release()


def main():
    rclpy.init()
    node = StereoMiniBridge()
    rclpy.spin(node)


if __name__ == "__main__":
    main()