Hardware Applet: From Idea to Application

In this Application Note you will learn how to design and create an image processing application (=applet) in VisualApplets and how to use it on a frame grabber. You will also learn how to use this applet during runtime using microDisplay X and Framegrabber API and how to configure parameters of your applet in this software. You find an introduction how to design, build and use an applet during runtime in microDisplay X at Getting Started. In this Application Note you will learn the above mentioned phases of applet development and usage with the example of a Sobel edge filter application.

Workflow Description

The workflow is as follows:

  1. Designing an applet in VisualApplets.

  2. Simulating the applet in VisualApplets and performing a Design Rules Check (DRC).

  3. Building the applet.

  4. For mE5 platforms: Flashing the applet.

  5. Using the applet on the hardware via microDisplay X or Framegrabber API.

Designing an Applet in VisualApplets

Starting a New Project

The design phase of an applet takes place in VisualApplets. To open VisualApplets, click on the file VisualApplets.exe in the bin folder in the VisualApplets installation directory, or the VisualApplets program icon in the Windows Start menu or on the icon on your desktop.

  1. At first, the VisualApplets main window opens up with no project loaded.

    VisualApplets Main Window

    Figure 169. VisualApplets Main Window


  2. Click on File+New ( Ctrl+N ) or use the New icon from the File icon bar. A New Project window opens up which allows you to specify project name, target hardware platform, and target runtime. You can always change these settings later on.

  3. To follow the example here, use the following settings:

    Project Name: Sobel_Filter

    Hardware Platform: microEnable 5 marathon VCL

    Target Runtime: Win 64 (Windows/AMD64)

  4. Confirm your settings by clicking OK.

    New Project window

    Figure 170. New Project window


VisualApplets now starts a new project and you see a blank design window in the center of the program window. In the Project Info tab on the right, information regarding the current project, such as project name, target hardware, target platform etc. is displayed.

Operators and Links

In VisualApplets, image processing operations are represented by operators. You find all these operators in the Operator Library on the right side of the VisualApplets design window. You can very easily place operators into the design window using drag-and-drop.

An instance of an operator in the design is called a module. Operators can have input and output ports. Operators in a design (i.e., modules) can be connected using these ports. Connections between modules are called links which are represented in the design window by arrows. These modules and links represent the image- (or signal-) processing pipeline. Hence, the order of operations is determined by the order of modules.

The operators and links have properties, which describe the settings relevant for the current image processing pipeline, like image dimensions. To see and edit these properties, double-click the operators. The properties of the operators are explained in detail in the operator documentation under Help.

Operator Documentation in VisualApplets

Figure 171. Operator Documentation in VisualApplets


Implementation of the Design

This example implements a Sobel edge filter algorithm.

In the VisualApplets design sample library, there are “ready-to-use" samples for edge filtering and further image processing applications. You find these design examples under Examples\Processing in your VisualApplets installation directory with the corresponding documentation, see Processing Examples. You can use these example designs as base for your own application.

For the implementation of this sample Sobel filter design, locate the operators in the Operator Library, drag them into the design and connect them with links as described in the Getting Started section.

You can rename the modules later on for better overview in the design and for better usage during runtime.

Whenever useful, structure the design using HierarchicalBoxes for a better design overview.

And don't forget to save the design from time to time.

Design Components

The sample design in this application note consists of the following components as shown in the following figure:

Example Design Implementation Sobel_Filter.va

Figure 172. Example Design Implementation Sobel_Filter.va


  • A Camera Link grayscale camera interface

  • An image buffer

  • The Sobel edge filter module with edge filtering in x and y direction (based on the example Sobel_Multi_Gradient.va under Examples\Processing\Filter\EdgeDetection\Sobel_Multi_Gradient in the VisualApplets installation directory)

  • An adjustable binarization threshold and a scale operator for better visualization of the binarized image on the display during runtime

  • The DMA to PC.

You can use the CoefficientBuffer in the design as simulation source for images instead of using the images acquired by a camera. Via the operator SourceSelector you can choose the image input source for the processing pipeline in VisualApplets and during runtime.

Parameter Settings

To parameterize the operators and links, you can edit the parameters of each element by double-clicking the element.

Link Parameters

Links have properties, which define the properties of the processed images such as maximum image dimensions, the image protocol, the bit width, and a parameter which is related to the maximum possible bandwidth: the parallelism (i.e. numbers of pixel transferred at one design clock cycle). If these links are connected to specific operators (see corresponding operator documentation), e.g. the camera interface operators, you can edit and adjust these properties according to your purpose.

The higher the maximum image dimensions and the parallelism you choose, the higher the FPGA resource consumption is. Therefore, Basler recommends to set these values only as high as necessary in order to save FPGA resources. The link properties can only be changed during design phase in VisualApplets.

Link Properties

Figure 173. Link Properties


Operator Properties

There are two types of operator properties: static and dynamic. You can change static parameters only during design phase in VisualApplets, whereas you can change dynamic parameters during design phase and during runtime. See the screenshot below for examples of static and dynamic parameters for the operator IS_GreaterThan.

Static and Dynamic Operator Parameters

Figure 174. Static and Dynamic Operator Parameters


In this sample design, set the following parameters:

Operator IS_GreaterThan:

Set the parameter Number to 600.

This identifies the grayscale of the edges. If you want to find out the grayscale of your edges, set a simulation probe before the IS_GreaterThan operator, run a simulation, and zoom into the pixels that contain edges and see which grayscale they have. This is a dynamical paramter, which you can alter during runtime.

Operator SCALE:

Set the parameter ScaleFactor to 255.

This augments the visibility of the edges. This is a dynamical parameter. However, to save FPGA resources, you can set this parameter to Static.

Preparation for Parameter Access during Runtime

During runtime you can change the dynamic parameters. A common use case is that you have many operators in a design and you need to change the same parameters e.g. the image dimensions for many different operators in the design.

The library Parameters provides operators which enable accessing different module parameters in a design by controlling only one parameter. Additionally, these operators also enable that access parameters are displayed on a specific hierarchy level. These options make parameter configuration easier during runtime. Whether the library Parameters is available for you depends on the license you have purchased. For questions regarding your license, contact the Basler Sales Department. The example design Sobel_Filter.va (see Figure 172, 'Example Design Implementation Sobel_Filter.va') uses translate and reference parameters to easily access image width and height, the image source selector, the threshold value and the scale factor on the hierarchy level Parameters. For this, create an empty HierarchicalBox, name it Parameters and set the DisplayHierarchy parameter of the corresponding translate and reference parameters to Parameters. You find a detailed documentation of the Parameters library in the corresponding documentation, see Library Parameters.

Finalizing the Design

Simulation Sources and Probes

During design phase the simulation sources and probes are very helpful. Using the simulation sources you can load test images and check your processed image on every link in the processing pipeline within a few moments. Exception: Signals can’t be simulated. Furthermore, the simulation performs a Design Rule Check Level 1 for the formal correctness of the implementation. The simulation result is equivalent to the result during runtime. You find the simulation sources and probes in the icon menu or under Analysis in the text menu in VisualApplets. For a detailed description of the usage of the simulation sources and probes, see the detailed documentation under 'Simulation'.

In Figure 172, 'Example Design Implementation Sobel_Filter.va' you can see a test image loaded to a simulation source in the example design Sobel_Filter.va. After threshold and before DMA you can see the result of the current processing step.

Design Rule Check (DRC)

After you have finalized the implementation, it is recommended to perform a Design Rule Check Level 1 and 2. To perform a Design Rule Check Level 1 and 2, select Analysis+Design Rules Check Level 1 and 2 or use the icon Design Rules Check Level 1 and 2 from the icon bar.

Design Rule Check Level 1 reports formal errors in the implementation and their exact location in the design. You must correct these errors before the design can be translated to a hardware applet.

Design Rule Check Level 2 additionally gives information about the estimated FPGA resources used in the design. With this information you can check, whether enough FPGA resources are available for your image processing implementation. If the resources exceed 100% of one of the FPGA resources, you need to redesign your implementation to save resources.

In Figure 175, 'Design Rule Check 1 and 2 for the Example Design Sobel_Filter.va' the result of Design Rule Check 1 and 2 is shown: Everything is designed correctly and about 36% of the available Lookup Tables, 22% FlipFlops, 15 % Block RAM and 2 % of the available embedded ALUs are used. You can see a detailed overview on how many FPGA resources each element in the design consumes (see Figure 176, 'FPGA Resource Estimation') under Analysis+FPGA Resource Estimation

Design Rule Check 1 and 2 for the Example Design Sobel_Filter.va

Figure 175. Design Rule Check 1 and 2 for the Example Design Sobel_Filter.va


FPGA Resource Estimation

Figure 176. FPGA Resource Estimation


Building the Applet in VisualApplets

Precondition

The hardware applet build process can only be performed if the XILINX tools are properly installed. Under https://docs.baslerweb.com/visualapplets/installing-visualapplets#which-xilinx-toolchain-and-version-for-which-frame-grabber-platform you can find a detailed overview, which Xilinx versions is suitable for your frame grabber platform. In this sample design Sobel_Filter.va the microEnable 5 marathon VCL platform is selected. Thus, Xilinx ISE version 14.7 or Vivado versions between 2016.1 and 2020.1 are suitable.

Editing the Build Settings

Now you have finished the example design Sobel_Filter.va and want to translate the design into a hardware applet. For this translation process, called build you need to select the correct Xilinx build settings. Open Settings+Build Settings, and select the batch file settings64.bat of your corresponding Xilinx version (recommended: Xilinx Vivado 2018.2) from your file system and confirm with OK. You find a detailed description of editing the build settings under 'Build Settings'.

Build Hardware Applet Dialog

Figure 177. Build Hardware Applet Dialog


VisualApplets now uses the Xilinx tools to translate the application into the FPGA bitstream, i.e., the program or applet. The duration of this process depends on the complexity of the design. The build of highly complex designs might take several hours. For this example implementation Sobel_Filter.va, the build time is about 15 minutes. After successful build, the applet is fully generated. The name of the applet (*.hap file) is the same as the name of the design file (*.va): Sobel_Filter.hap

Running the Applet on Hardware

Precondition

To run an applet in hardware on a frame grabber, a programmable (V Series) frame grabber (hardware) needs to be installed on the system. Furthermore, the Framegrabber SDK software needs to be installed on the PC. For this example applet for the microEnable 5 marathon VCL, the latest possible Framegrabber SDK version is 5.7.

Flashing

If, like in this example, you use a frame grabber of the microEnable 5 series (marathon, ironman or a LightBridge), you need to flash your frame grabber with the new applet using microDiagnostics.

If you are going to use the new applet on a microEnable IV frame grabber or a frame grabber of the latest CXP 12 series, just skip this section.

To flash the frame grabber, perform the following steps:

  1. Start the tool microDiagnostics under bin in the Framegrabber SDK installation directory.

  2. Select your frame grabber.

  3. Select Firmware in the left side of the dialog.

  4. Assign your hardware applet to one of the displayed partitions (0 to 7): Go to the directory, where the created hardware applet is located and select the file.

    Firmware Partitions Displayed in microDiagnostics

    Figure 178. Firmware Partitions Displayed in microDiagnostics


  5. Click Flash Now

  6. Wait until the new firmware is completely installed. When flashing is completed, you get a message in microDiagnostics.

  7. Follow the instructions in the message.

Testing and Loading the Applet in microDisplay X

To test your applet Sobel_Filter.hap and to set some first parameters,use the program microDisplay X. Following steps are necessary to load an applet:

  1. Save the applet (here: Sobel_Filter.hap) under Hardware Applets\<your frame grabber> (here: mE5-MA-VCL) in the Framegrabber SDK installation directory.

  2. Start microDisplay X from the bin folder in the Framegrabber SDK installation directory.

  3. In microDisplay X, select the frame grabber you want to use under Acquisition Devices. Immediately, all applets available for the selected frame grabber are displayed.

  4. Select the *.hap file you want to use (here: Sobel_Filter.hap) with double-click.

Parameter Settings and Acquisition

After loading the applet, you can see the parameter tree and the image acquisition window in microDisplay X. Below, a screenshot of the applet Sobel_Filter.hap loaded in microDisplay X is shown.

Parameter Tree and Image Acquisition Window in microDisplay X

Figure 179. Parameter Tree and Image Acquisition Window in microDisplay X


On the left side, the parameter tree is displayed. These parameters are equivalent to the operator names during design phase in VisualApplets. It is important to use a good naming of the operators, in order to find the correct parameters you want to adjust for acquisition. Dynamic parameters are adjustable during runtime. There are two kinds of dynamic parameters: Those parameters, which are only adjustable before start of acquisition (e.g. DMA dimensions) and those, which can be adjusted even during image acquisition (e.g. threshold values of operator IS_GreaterThan).

In this example, you perform the image acquisition using an image from a simulator source and set the binarization threshold to value 150. If you have integrated the parameters of the Parameters library during design phase in VisualApplets, you can set all relevant dynamic parameters under the hierarchy level Parameters in the parameter tree. Otherwise, you need to step though the different hierarchy levels to find the correct parameters we want to set.

In this example, go to the hierarchy level Parameters and unfold it:

  1. Right mouse-click the Parameter Simulation Image. As a result, you can select the example image you want to use for image acquisition.

  2. Load this image by clicking Load Simulation Image.

  3. Set the parameter ImageSource selector to Simulator and the Binarization Threshold to value 150.

  4. Set the image dimensions to 1024x1024 pixels.

After you have set these parameters (or all parameters relevant for your applet), you can start the acquisition. Here you can select between continuous grabbing, grabbing of a sequence or a single frame. To stop the continuous grabbing, click on the Stop button. For the case of a multi process design, you can start and stop the the acquisition of the single processes separately.

For a detailed description of the applet configuration and acquisition with microDisplay X, see https://docs.baslerweb.com/frame-grabbers/files/en/Tools/mDisplayX/Content/3_Runtime/Tools/microDisplayX/Intro/WhatIsmDisplay.htm.

Starting the Applet in Your Own SDK

Instead of performing the applet configuration and acquisition in mircoDisplay X you can alternatively integrate them in your own SDK environment. In this section you find an introduction to the basic functionalities of the relevant SDK componets for image acquisition and configuration.

As base of a C++ SDK you can use the VisualApplets internal SDK code generator, which you find the icon menu or under Build+Generate SDK Example in VisualApplets. This function automatically generates the corresponding CMake File and the VisualStudio project files, which you can use as base for your SDK code.

Generated SDK Project Files

Figure 180. Generated SDK Project Files


The generated C++ code in the project file vasFgProject.cpp contains the following code components:

  1. Frame Grabber: Initialization

    Definition of number of acquisition cycles, definition of board index, call of function Fg_Init("Sobel_Filter.hap", boardIndex) to start the applet.

  2. Allocation of memory for buffers for acquisition.

    function: Fg_AllocMemEx(fg, ..., ...)

  3. Frame Grabber: Get* and Set* parameters for Process0

    functions: Fg_setParameterWithType(fg, ..., ..., ...) and Fg_getParameterWithType(fg, ..., ..., ...)

    Here all parameters contained in the design are listed. Basler recommends to delete all parameters, which are not necessary for adjustment during runtime, for better overview.

  4. Create the display(s)

    function CreateDisplay()

  5. Start acquisition at applet and camera for each present port:

    function: Fg_AcquireEx(fg, 0, nrOfCycles, ...,...);

  6. Grab images:

    function: Fg_getLastPicNumberBlockingEx(fg, ..., ..., ..., ...);

  7. Stop acquisition

    Function Fg_stopAcquireEx(fg, .., .., ...)

  8. Close the display

    function CloseDisplay(...);

  9. Release the memory for buffer(s)

    function Fg_FreeMemEx(fg, ..);

  10. Frame Grabber Uninitialization

    function Fg_FreeGrabber(fg);

A detailed description and explanation of the SDK functions is available at https://docs.baslerweb.com/frame-grabbers/files/en/documents/SDK/functions/fgrab__prototyp_8h.html

A detailed introduction into a SDK development is available at https://docs.baslerweb.com/frame-grabbers/files/en/documents/SDK/SDK.html.

You can integrate the generated sample SDK code into a larger SDK environment for the further software processing of the frame grabber (pre-)processed images.