X-ray computed tomography (CT) and laminography are a unique tools for systematic examination of materials and biological samples in various research areas. CT employs the penetrating nature of X-rays to generate attenuation maps of the imaged object. In conventional CT systems, the object is placed on a rotating sample stage between a source emitting a beam of X-rays and a detector measuring the two-dimensional radiographic intensity image of the incident X-rays. As the object rotates with respect to the source-detector assembly, namely the gantry, radiographs are acquired for a set of typically equally spaced rotation positions of the object as it performs 360° revolution with respect to the fixed gantry. A tomographic reconstruction algorithm is then applied to the set of acquired radiographs to generate a volumetric X-ray attenuation map of the scanned object. The attenuation map consists of a three-dimensional array of voxels, each assigned a gray value corresponding to the calculated X-ray attenuation for the volumetric space occupied by the voxel. Laminography is a special kind of X-ray tomography used for flat samples. When a flat sample like for example a printed circuit board is examined in a CT, the X-ray absorption in the sample will increase drastically when the long edge of the sample is nearly parallel to the direction of the X-rays. In such cases it is advantageous to tilt the sample, so the normal to its large plane is not parallel to the incident beam and then rotate the sample around this vector perpendicular to the large plane of the sample.
Recent progress of X-ray flux density at synchrotron facilities combined with fast pixel detectors enables scanning times of less than a second. Combined with optical flow analysis, this has enabled new approaches for 4D (space+time) in situ, operando and even in vivo studies, e.g., of the structure evolution of materials and devices during technological processes, or of biological processes in cells, tissues and full small organisms.
Due to the complexity of the experimental setup, automation and software-controlled experiments with X-ray tomography are challenging. For a successful automated scan, the imaging and sample apparatus must be aligned and positioned properly, the sample be stabilized and the measurement setup controlled during the imaging process. Moreover, intelligent control of the imaging process with respect to the unpredictable spatio-temporal localization of the region of interest (ROI) and its evolution is particularly difficult to achieve because it requires information about the process under study. In the worst case the ROI may be completely missed or leave the FOV during the experiment. This problem is especially true if the position of the sample in the 4D space are not known a priori or are changing during the scan. Although two-dimensional (2D) radiographs already contain sufficient information for fast online feedback in many applications, in certain cases control decisions must be based on quality metrics that can only be derived from 3D or 4D image reconstruction.
Furthermore, the resolution and duration of in-vivo experiments with living objects is currently limited by radiation damage. The radiation dose may be reduced by taking fewer projections to reconstruct a single tomogram or by reducing the exposure time for each projection. With traditional reconstruction techniques, both approaches lead to various artifacts in the final images. Motion of the objects during the experiment also affects the reconstruction quality. The compressed sensing theory has demonstrated the feasibility to recover signals from the under sampled data. The incorporation of a sparsity prior can compensate incompleteness of measurements and the images can be reconstructed from fewer number of measurements using non-linear optimization scheme. However, such optimization techniques are very computationally demanding.

In a collaboration with the KIT Imaging Cluster, we develop a new generation control system to enable a new type of feedback driven experiments and to allow tracking of fast processes with high spatial and temporal resolutions. The developed control system is currently operated at two imaging beamlines of the KIT Imaging Cluster. While focusing on synchrotron tomography and laminography, our technologies can be easily adapted to different types of detectors generating image streams with bandwidth up to 8 GB/s and enable online monitoring, real-time data-processing, and data-driven control. The software platform is open-source and all software components are freely available: https://github.com/ufo-kit.

Our UFO (Ultra-Fast tOmography) platform covers the complete data path from the X-ray detector to the long-term data archive. The system includes hardware and software components and proposes a design guideline for the computing infrastructure. Primary design goals were low administrative effort, cost efficient off the shelf components, and the ability to distribute equipment in the synchrotron facility. Important components of the framework are networking, camera framework, computing environment and control system.
The data flow is organized as sketched in Fig. 2. First, the data is pre-processed using a single GPU directly at the camera station. At this stage the possible feedback based on 2D projections is returned to the beamline instrumentation. Then, after online data reduction, the imaging data is passed to the distributed UFO control system for 3D reconstruction and data analysis. Information for a second feedback loop based on reconstructed 3D images is extracted here. Further, the raw data is streamed to the high-speed streaming storage.

To build the DAQ network we use four distinct types of nodes: camera station, master server, computational nodes, and storage nodes. The architecture is shown in Fig. 3. An Infiniband fabric is used to connect all parts of the platform. Camera stations are located in the beamline hatch, but other components are outside and connected to the camera station using an optical Infiniband link.

Multiple software components were developed to meet the requirements of the UFO control system. An overview of the software components used to realize the UFO control system is given in Fig. 4. The central element of the system is the parallel UFO image processing framework. The Concert control system enables image-based control loops by integrating the UFO framework and fast beam-line devices. The interface to the cameras is provided by the Unified Camera Access library (libuca). Besides commercial cameras, we have designed a range of in-house streaming cameras serviced by the Alps driver framework. To get full advantage of the high-speed Infiniband network, the KIT Infiniband Remote Communication library KIRO was developed. It supports the native Infiniband RDMA protocol. The library can reach data transfer rates of up to 4.7 GB/s and latencies in the order of 20 us for the Infiniband FDR fabric.

The results of a successful experiment are first streamed to the high-speed storage partition using the FastWriter and, then, archived in the KIT Large Scale Data Facility (LSDF). To allow a quick preview of the archived 3D datasets, the web application Web Analysis of Volumes (WAVe) has been developed.  Many recent mobile devices are equipped with powerful GPU adapters and are capable to show the visualization of 3D volumes in WAVe.

UFO image processing framework

The central element of the system is the pipelined UFO image processing framework. It models the image processing workflow by a structured graph where each node represents an operator on the input data. By design, the UFO framework contains multiple levels of parallelism: fine-grained massive parallelism, pipelining and concurrent execution of branches within the graph. The implemented library of image processing algorithms is implemented in OpenCL and carefully optimized to recent architectures from AMD and NVIDIA.

The UFO scheduler is programmed to facilitate multi-GPU and clustered setups. The input for the scheduler is a graph of tasks and the description of the available hardware, like number of GPUs, CPUs and remote compute nodes and how they can be accessed. The scheduler tries to find the best mapping of the tasks to the available hardware. However, in most cases the hardware would be not utilized because there are far more compute units than tasks available. To accommodate for this situation, the scheduler expands the task graph by examining the graph’s structure. Each sequence consisting of tasks that potentially run on GPUs is duplicated and inserted into the task graph. At execution time each of these sequences will then be mapped to the same GPU in order to reduce inter-GPU memory transfers.
The same idea is employed to support clustering: Sequences of long-running computation are identified, encapsulated and a description sent to the remote node. In the original graph, a place holder node sends data to the remote compute node and receives resulting data from, thus re-injecting the data into the original graph.  At each computation node, an UFO daemon is started and listening for requests from the main server. When the UFO framework wants to transmit data to a daemon for processing, it sends a request to the daemon, asking it to prepare an empty buffer with enough size to store the data it wants to send. As soon as the daemon has acknowledged this request, the actual data that resides in the buffer is collected and sent to the daemon, which puts it back into the newly created, empty buffer. From this point on, the computing on the daemon continues the same way as in a normal, local UFO chain. And after the computing is done, the same thing happens again, only in the opposite direction.

"Concert" control system

The high-level control system “Concert” is used to manage beam-line devices and control the course of the experiment. It is based on the UFO framework and enables image-based control loops by integrating high-speed compute nodes and fast beam line devices. The control loops are implemented by easy-to-write Python scripts using components provided by “Concert” and the UFO framework. “Concert” defines interfaces for multiple standard components used at imaging beam-lines including high-speed detectors, rotational and linear motors, sample changers, beam monochromators, etc. To realize a common access scheme for different high-speed cameras, the Unified Camera Abstraction Library (libuca) is provided.  libuca works with local and remote cameras and is able to achieve transfer rates of 4750 frames per second if 8-bit 1-megapixel camera and FDR Infiniband interconnects are used. To communicate with standard devices a Tango wrapper is used. Easy to use components are provided to work with Aerotech and Compact RIO-based motors and a variety of other devices.  “Concert” greatly simplifies standard operations like alignment of rotation axis, focusing of the camera, finding and centering the beam. Finally “Concert” integrates with the data managements system. Metadata is automatically retrieved from the instruments, might be extended by the user and is saved with the measured data.

Real-time Reconstruction

For real-time tomographic reconstruction, we provide two methods: Filtered Back Projection (FBP) and Direct Fourier Inversion (DFI). Compared to the mostly used FBP algorithm, DFI is more sensitive to noise, but it is slightly faster. Both algorithms are adapted to GPU hardware and are optimized. Furthermore, we provide reduced-precision reconstruction modes. This reduction in image quality is often acceptable for real-time applications. It will speed-up execution while the produced results still are sufficient to make reliable control decisions. As shown on Fig. 6, the provided software outperforms the state-of-the-art algorithms in the ASTRA toolkit significantly. The optimized algorithms achieve a throughput of 1 GB/s on a single GPU adapter and up to 8 GB/s per GPU server. This exceeds the data rates of currently existing scientific camera interfaces. Beside real-time reconstruction, the developed algorithms can be also applied to accelerate otherwise slow low-dose reconstruction which is normally performed offline. Further details about developed algorithms are published in S. Chilingaryan et al. “Balancing load of GPU subsystems to accelerate image reconstruction in parallel beam tomography”, In Proc. of SBAC-PAD, Lyon 2018. Available at https://ieeexplore.ieee.org/document/8645862

Experiments and Projects

• UFO
• ROOF
• ASTOR / NOVA

For Students

We regularly offer a variety of student topics related to computed tomography. We expect students to have verses in image processing and good Python programming skills. Some topics also require a good understanding of GPU architectures and prior experience with parallel programming techniques. Please check the current vacancies or contact us if you have good programming skills, are able to work independently and are interested in high performance computing and tomography. The range of topics is broad and can include related tasks:

• General image processing: noise reduction, artifact compensation, etc.
• Development of advanced algorithms for tomographic reconstruction
• Parallelisation and optimization of image processing software
• optical flow, image segmentation and visualization