ximea
Typically, the cameras for machine vision, industrial and scientific applications transfer RAW data and these need a complex image processing pipeline to convert to RGB in realtime.
This is a computationally heavy task, especially for high bandwidth cameras.
An example of such a camera is 65 Mpix model that is able to stream over 70 Frames per second through PCI Express Gen3 interface.
Computation of data from this model could be done on Intel/AMD CPUs, but this solution is difficult to accelerate further, especially with multicamera systems.
To overcome this bottleneck, it is possible to implement simplified algorithms on CPU, but most of the high quality algorithms are slow even with multicore CPUs, the slowest being: demosaicing, denoising, color grading, undistortion, resize, rotation to an arbitrary angle, compression, etc.
An optional solution is to utilize the Fastvideo SDK which is working on NVIDIA GPU, thus the full image processing pipeline is done on the graphics processing unit unburdening the CPU-based features.
Based on the Fastvideo SDK engine is the Fast CinemaDNG Processor software for Windows, which provides high performance and high quality RAW image processing.
In fact, the offered image quality is comparable to the results of RAW processing at Raw Therapee, Adobe Camera Raw and Lightroom Photo Editor software, but significantly faster.
To check the GPU-based performance for a specific camera on Fast CinemaDNG Processor software, it is now possible to test transforming RAW images to DNG format with a new open source PGM2DNG converter, which could be downloaded from Github.
Below is a description of a sample project with details about the whole process.
To show how to implement a software with GPU image processing pipeline the following would be required:
Source codes and links to supplementary libraries for gpu-camera-sample project you can find on Github.
Sample application to create a software capturing RAW images from XIMEA cameras can be found in XIMEA SDK and this code can be incorporated into final solution.
It is possible to utilize the default camera parameters to focus on GPU-based image processing, but also add any GUI to control camera parameters as well.
There are many more image processing functions in the Fastvideo SDK to implement.
This pipeline can be further modified as soon as other source codes become available. Roadmap.
It is also feasible to build a multi-camera solution based on this software.
The simplest way would be to run several processes (one per camera) at the same time to test it.
A more sophisticated approach would be to create a single image loader that will collect frames from different cameras for further processing on GPU.
From the benchmarks on NVIDIA GeForce RTX 2080ti it can be identified that GPU-based RAW image processing is very fast with total performance reaching up to 4 Gpix/s or more on multiple GPUs.
There is also an opportunity to utilize different compression options on GPU at the end of the pipeline:
JPEG (MJPEG), JPEG2000 (MJ2K), H.264 and H.265 encoders on GPU.
Please note that H.264/H.265 are implemented via hardware-based NVIDIA NVENC encoder and that compression could be done in parallel with CUDA code.
The software could also work with RAW images in PGM format which are stored on external SSD.
This is a good method for software evaluation and testing without a camera.
In case of initial tests for the mentioned 65 Mpix camera (9433 x 7000 resolution) with bayer Gpixel GMAX3265 image sensor and RAW frames in PGM format:
The pipeline included:
Data acquisition, dark frame, FFC, linearization, BPC, white balance, debayer HQLI, gamma sRGB, 16/8-bit transform, JPEG compression (subsampling 4:2:0, quality 90), viewport texture copy, monitor output at 30 fps.
Remarkable performance for JPEG encoding on GPU is confirmed by: 65 MPix color image being compressed within 3.3 ms on NVIDIA GeForce RTX 2080ti.
The user can test different NVIDIA GPUs to choose the best hardware in terms of price and performance for a particular task.