Case study: Volumetric capturing with more than 100 multiple camera setup

What are the fields of application?

Volumetric capture can be used in various research and development activities:
 

• Live tele-presence in Augmented VR or Mixed/Augmented Reality settings
• Performance Capture
• Free Viewpoint Video (FVV)
• Immersive Applications (i.e., events and/or gaming)
• Motion tracking and analysis
• Creation of Virtual characters from real actors to be used in movies and games

What is the challenge?

A customer of ours had the idea of creating an innovative system enabling the user to record a moving full body representation in 3D that enables a realistic, authentic, and particularly natural 3D image of moving persons in virtual worlds. This information can be used to create animations for virtual characters in the gaming industry, high-end CG movies, and motion analysis for sports. It captures live characters with multiple cameras at the same time and creates naturally moving dynamic 3D models, which can be observed from arbitrary viewpoints in the virtual scene.
While the customer was handling the software and analysis part, XIMEA was tasked to create the hardware platform and infrastructure that allows for recording all happenings in a 5 x 5 meters room from all perspectives. The entire room was essentially turned into a volumetric capture studio.
The customer had several requirements that needed to be fulfilled  including the coverage of every inch of the room from many perspectives. The application also required  synchronization between all cameras and a very low latency to support near-real-time processing. All synchronized videos had to be recorded to disk, at the desired resolution of 12 Mpix per camera running at 60 FPS.
All cameras had to be placed in the volumetric capture studio, while servers were in another room in the same building. The existing infrastructure was equipped with 12-fiber standard MTP connections between the rooms. Furthermore, we had the requirement to use the house-internal optical fiber inter-room cabling system using patch panels. Also, the cable lengths were determined to be 25 meters in the capture studio, 2 meters in the server room, and ~25 meters between the rooms, for a total of 52 meters.

How can it be done?

Our R&D team in Marianka, Slovakia and our sales team in Münster, Germany came together to discuss the requirements with our customer and find a solution in order to provide a perfect fit for our client’s demands.
To achieve low latency we decided to avoid controllers, data conversions, and protocols with overhead, such as USB. Therefore, we opted for PCI Express (peripheral component interconnect express) – a widely used bus interconnect interface, mainly used in server platforms. It is increasingly used as a storage interconnect solution as well, with the addition of NVMe storage devices into the PCIe ecosystem. PCIe currently supports up to 8 GT/s of throughput per PCIe Lane, with a roadmap up to 16 GT/s. Typical storage implementations utilize x2, x4, x8, and x16 lane width interconnect configurations from the host root complex, directly to, or through PCIe switches, to endpoint PCIe storage devices. Some of the  considerable advantages of PCI Express include the higher bandwidth and raw speed. It is a serial technology, and It offers lower delay or latency as it provides direct connectivity to chipsets. In essence, it is the high-speed serial computer expansion bus standard used by any modern PC.
PCI Express also provided the advantage that it can easily be extended to the outside of a PC. This only requires PCIe adapters to convert PCIe slots into copper or optical cables, which we had already available from previous projects.
Together with the customer we identified the need for 120 cameras total to be placed on 5 sides, 4 walls and on the ceiling. The customer opted for 60 color cameras and 60 monochrome cameras. All camera positions were pre-determined by the scientists. PCI Express technology also enables bundling multiple cameras into a PCIe switch (“camera cluster”), whose data can be offloaded using a single optical cable. The switch positions were defined as follows:

As the cameras would be more than 50 cm away (a limit for ribbon cables) from the switch, we agreed for FireFly ECUE flat copper cables between cameras and switch, which enable cable lengths of 3 meters. We determined the need for 20 switches to cover 120 cameras, i.e., 6 cameras per switch. Each server was able to handle and input 2x8 lanes of PCI Express, from 2 switches. Because of its modularity, the setup was easily extendable (see figure 2):

What is the Solution?

Each such setup with 1 server, 2 switches and 12 cameras is capable of processing a sustained data rate of 120 GB/s, or 12 x 10 GB/s per camera.
The XIMEA PCIe Switches and PCIe Adapter Cards make use of iPass - a connector standard for external PCI Express cabling. However, as we had the requirement to use the existing cabling infrastructure between the rooms, we created new cables for the conversion between iPass and MTP on both ends in the server room as well as in the studio. In this way, the customer was able to use a regular optical network patch panel.

Each iPass connector with PCIe Gen 3x8 requires 24 fibers. As the internal house network only provided 12 fibers per cable, we created cables that converted iPass into 2x12 fiber MTP so that the customer was able to use the existing patch panel.
The servers were selected by the customer (DELL) and then configured and tested in the XIMEA Lab. Each server ran 2 switches with a total of 12 cameras. We also ran tests with 8 cameras per switch.

Afterwards, the customer decided to use Dell PowerEdge Express Flash NVMe “Mixed Use” PCIe SSD on DELL Servers to enable Hot Swap and Dell Support - a high-performance storage device designed for solutions requiring high IOPs, ultra-low latency and enterprise-class storage reliability and serviceability.
XIMEA switches also cope with another hard requirement of our customer. They enable easy synchronization between all cameras and switches. On board DIP switches allow the rearrangement of multiple triggering configurations such that a single switch can accept a trigger input and distribute it to all the others (See Figure 4). The customer provided this single  triggering signal to one switch, which was then propagated throughout the whole system for a tight synchronization between cameras.

XIMEA offers a growing set of generic switches as adapters within its PCIe ecosystem. Those components allow customers to perform quick evaluations and prototyping of various setups for their applications. This test would then lead to custom developments and modifications. In this particular case, the existing 8 Port Switch already covered all of the required functionalities and could therefore be used directly.

Conclusion

The basic idea of volumetric video came up several decades ago, when it was especially important in sports to take pictures from different angles. If several cameras are already aimed at an athlete, why not try to get a 3-dimensional image from these recordings? The idea was then implemented for the first time in photogrammetry, as used by Google Street View. An object is photographed from several sides and then displayed in 3D. 
So, volumetric videos are created by using numerous cameras, which together represent a precise and realistic image from different angles. Both in film and in games, the recordings offer an incredibly detailed image quality. The possible uses for volumetric videos are limitless and can be implemented in every 3D environment.
Happily, we ended up assembling a volumetric system, which in turn allows us and our customer to progress to the next stage.