Standard digital color images of a scene of interest consist of three components – red, green and blue – which contain the intensity level for each of the pixels in three different groups of wavelengths.
This concept has been extended in the hyperspectral and multispectral imaging sensing modalities, where the data to be acquired consists of a three-dimensional datacube that has two spatial dimensions xx and yy and one spectral dimension λλ.
In simple terms, a datacube is a 3-D function f(x,y,λ)f(x,y,λ) that can be represented as a stacking of intensities of the scene at different wavelengths. An example datacube is shown in Figure 1.
Each of its entries is called a voxel. We also define a pixel's spectral signature as the stacking of its voxels in the spectral dimension f(x,y)={f(x,y,λ)}λf(x,y)={f(x,y,λ)}λ.
The spectral signature of a pixel can give a wealth of information about the corresponding point in the scene that is not captured by its color. For example, using spectral signatures, it is possible to identify the type of material observed (for example, vegetation vs. ground vs. water), or its chemical composition.
Datacubes are high-dimensional, since the standard number of pixels present in a digitized image is multiplied by the number of spectral bands desired. However, considerable structure is present in the observed data. The spatial structure common in natural images is also observed in hyperspectral imaging, while each pixel's spectral signature is usually smooth.
Compressive sensing (CS) architectures for hyperspectral imaging perform lower-dimensional projections that multiplex in the spatial domain, the spectral domain, or both. Below, we detail three example architectures, as well as three possible models to sparsify hyperspectral datacubes.
The single pixel camera uses a single photodetector to record random projections of the light emanated from the image, with the different random projections being captured in sequence. A single pixel hyperspectral camera requires a light modulating element that is reflective across the wavelengths of interest, as well as a sensor that can record the desired spectral bands separately [3]. A block diagram is shown in Figure 2.
The single sensor consists of a single spectrometer that spans the necessary wavelength range, which replaces the photodiode. The spectrometer records the intensity of the light reflected by the modulator in each wavelength. The same digital micromirror device (DMD) provides reflectivity for wavelengths from near infrared to near ultraviolet. Thus, by converting the datacube into a vector sorted by spectral band, the matrix that operates on the data to obtain the CS measurements is represented as
Φ
=
Φ
x
,
y
0
⋯
0
0
Φ
x
,
y
⋯
0
⋮
⋮
⋱
⋮
0
0
⋯
Φ
x
,
y
.
Φ
=
Φ
x
,
y
0
⋯
0
0
Φ
x
,
y
⋯
0
⋮
⋮
⋱
⋮
0
0
⋯
Φ
x
,
y
.
(1)This architecture performs multiplexing only in the spatial domain, i.e. dimensions xx and yy, since there is no mixing of the different spectral bands along the dimension λλ.
The dual disperser coded aperture snapshot spectral imager (DD-CASSI), shown in Figure 3, is an architecture that combines separate multiplexing in the spatial and spectral domain, which is then sensed by a wide-wavelength sensor/pixel array, thus flattening the spectral dimension [2].
First, a dispersive element separates the different spectral bands, which still overlap in the spatial domain. In simple terms, this element shears the datacube, with each spectral slice being displaced from the previous by a constant amount in the same spatial dimension. The resulting datacube is then masked using the coded aperture, whose effect is to "punch holes" in the sheared datacube by blocking certain pixels of light. Subsequently, a second dispersive element acts on the masked, sheared datacube; however, this element shears in the opposite direction, effectively inverting the shearing of the first dispersive element. The resulting datacube is upright, but features "sheared" holes of datacube voxels that have been masked out.
The resulting modified datacube is then received by a sensor array, which flattens the spectral dimension by measuring the sum of all the wavelengths received; the received light field resembles the target image, allowing for optical adjustments such as focusing. In this way, the measurements consist of full sampling in the spatial xx and yy dimensions, with an aggregation effect in the spectral λλ dimension.
The single disperser coded aperture snapshot spectral imager (SD-CASSI), shown in Figure 4, is a simplification of the DD-CASSI architecture in which the first dispersive element is removed [4]. Thus, the light field received at the sensors does not resemble the target image. Furthermore, since the shearing is not reversed, the area occupied by the sheared datacube is larger than that of the original datacube, requiring a slightly larger number of pixels for the capture.
This sparsity structure assumes that the spectral signature for all pixels in a neighborhood is close to constant; that is, that the datacube is piecewise constant with smooth borders in the spatial dimensions. The complexity of an image is then given by the number of spatial dyadic squares with constant spectral signature necessary to accurately approximate the datacube; see Figure 5. A reconstruction algorithm then searches for the signal of lowest complexity (i.e., with the fewest dyadic squares) that generates compressive measurements close to those observed [2].
This sparsity structure operates on each spectral band separately and assumes the same type of sparsity structure for each band [4]. The sparsity basis is drawn from those commonly used in images, such as wavelets, curvelets, or the discrete cosine basis. Since each basis operates only on a band, the resulting sparsity basis for the datacube can be represented as a block-diagonal matrix:
Ψ
=
Ψ
x
,
y
0
⋯
0
0
Ψ
x
,
y
⋯
0
⋮
⋮
⋱
⋮
0
0
⋯
Ψ
x
,
y
.
Ψ
=
Ψ
x
,
y
0
⋯
0
0
Ψ
x
,
y
⋯
0
⋮
⋮
⋱
⋮
0
0
⋯
Ψ
x
,
y
.
(2)This sparsity structure employs separate sparsity bases for the spatial dimensions and the spectral dimension, and builds a sparsity basis for the datacube using the Kronecker product of these two [1]:
Ψ
=
Ψ
λ
⊗
Ψ
x
,
y
=
Ψ
λ
[
1
,
1
]
Ψ
x
,
y
Ψ
λ
[
1
,
2
]
Ψ
x
,
y
⋯
Ψ
λ
[
2
,
1
]
Ψ
x
,
y
Ψ
λ
[
2
,
2
]
Ψ
x
,
y
⋯
⋮
⋮
⋱
.
Ψ
=
Ψ
λ
⊗
Ψ
x
,
y
=
Ψ
λ
[
1
,
1
]
Ψ
x
,
y
Ψ
λ
[
1
,
2
]
Ψ
x
,
y
⋯
Ψ
λ
[
2
,
1
]
Ψ
x
,
y
Ψ
λ
[
2
,
2
]
Ψ
x
,
y
⋯
⋮
⋮
⋱
.
(3)In this manner, the datacube sparsity bases simultaneously enforces both spatial and spectral structure, potentially achieving a sparsity level lower than the sums of the spatial sparsities for the separate spectral slices, depending on the level of structure between them and how well can this structure be captured through sparsity.
Compressive sensing will make the largest impact in applications with very large, high dimensional datasets that exhibit considerable amounts of structure. Hyperspectral imaging is a leading example of such applications; the sensor architectures and data structure models surveyed in this module show initial promising work in this new direction, enabling new ways of simultaneously sensing and compressing such data. For standard sensing architectures, the data structures surveyed also enable new transform coding-based compression schemes.
-
Duarte, M. and Baraniuk, R. (2009). Kronecker Compressive Sensing. [Preprint].
-
Gehm, M. and John, R. and Brady, D. and Willett, R. and Schultz, T. (2007). Single-shot compressive spectral imaging with a dual disperser architecture. Optics Express, 15(21), 14013–14027.
-
Kelly, K. and Takhar, D. and Sun, T. and Laska, J. and Duarte, M. and Baraniuk, R. (2007, Mar.). Compressed Sensing for Multispectral and Confocal Microscopy. In Proc. American Physical Society Meeting. Denver, CO
-
Wagadarikar, A. and John, R. and Willett, R. and Brady, D. (2008). Single disperser design for coded aperture snapshot spectral imaging. Appl. Optics, 47(10), B44–51.
