ChunkedArray< N, T > Class Template Referenceabstract

Interface and base class for chunked arrays. More...

#include <vigra/multi_array_chunked.hxx>

Inheritance diagram for ChunkedArray< N, T >:

Public Member Functions
std::string	backend () const
	Return the class that implements this ChunkedArray.
iterator	begin ()
	Create a scan-order iterator for the entire chunked array.
const_iterator	begin () const
	Create a read-only scan-order iterator for the entire chunked array.
template<unsigned int M>
MultiArrayView< N-1, T, ChunkedArrayTag >	bind (difference_type_1 index) const
	Create a lower dimensional view to the chunked array. More...
MultiArrayView< N-1, T, ChunkedArrayTag >	bindAt (MultiArrayIndex dim, MultiArrayIndex index) const
	Create a lower dimensional view to the chunked array. More...
MultiArrayView< N-1, T, ChunkedArrayTag >	bindInner (difference_type_1 index) const
	Create a lower dimensional view to the chunked array. More...
template<int M, class Index >
MultiArrayView< N-M, T, ChunkedArrayTag >	bindInner (const TinyVector< Index, M > &d) const
	Create a lower dimensional view to the chunked array. More...
MultiArrayView< N-1, T, ChunkedArrayTag >	bindOuter (difference_type_1 index) const
	Create a lower dimensional view to the chunked array. More...
template<int M, class Index >
MultiArrayView< N-M, T, ChunkedArrayTag >	bindOuter (const TinyVector< Index, M > &d) const
	Create a lower dimensional view to the chunked array. More...
std::size_t	cacheMaxSize () const
	Get the number of chunks the cache will hold. More...
int	cacheSize () const
	Number of chunks currently fitting into the cache.
const_iterator	cbegin () const
	Create a read-only scan-order iterator for the entire chunked array.
const_iterator	cend () const
	Create the end iterator for read-only scan-order iteration over the entire chunked array.
template<class U , class Stride >
void	checkoutSubarray (shape_type const &start, MultiArrayView< N, U, Stride > &subarray) const
	Copy an ROI of the chunked array into an ordinary MultiArrayView. More...
chunk_iterator	chunk_begin (shape_type const &start, shape_type const &stop)
	Create an iterator over all chunks intersected by the given ROI.
chunk_const_iterator	chunk_begin (shape_type const &start, shape_type const &stop) const
	Create a read-only iterator over all chunks intersected by the given ROI.
chunk_const_iterator	chunk_cbegin (shape_type const &start, shape_type const &stop) const
	Create a read-only iterator over all chunks intersected by the given ROI.
chunk_const_iterator	chunk_cend (shape_type const &start, shape_type const &stop) const
	Create the end iterator for read-only iteration over all chunks intersected by the given ROI.
chunk_iterator	chunk_end (shape_type const &start, shape_type const &stop)
	Create the end iterator for iteration over all chunks intersected by the given ROI.
chunk_const_iterator	chunk_end (shape_type const &start, shape_type const &stop) const
	Create the end iterator for read-only iteration over all chunks intersected by the given ROI.
virtual shape_type	chunkArrayShape () const
	Number of chunks along each coordinate direction.
shape_type	chunkShape (shape_type const &chunk_index) const
	Find the shape of the chunk indexed by 'chunk_index'. More...
shape_type const &	chunkShape () const
	Return the global chunk shape. More...
shape_type	chunkStart (shape_type const &global_start) const
	Find the chunk that contains array element 'global_start'.
shape_type	chunkStop (shape_type global_stop) const
	Find the chunk that is beyond array element 'global_stop'. More...
template<class U , class Stride >
void	commitSubarray (shape_type const &start, MultiArrayView< N, U, Stride > const &subarray)
	Copy an ordinary MultiArrayView into an ROI of the chunked array. More...
const_view_type	const_subarray (shape_type const &start, shape_type const &stop) const
	Create a read-only view to the specified ROI. More...
std::size_t	dataBytes () const
	Bytes of main memory occupied by the array's data. More...
std::size_t	dataBytesPerChunk () const
	Number of data bytes in an uncompressed chunk.
iterator	end ()
	Create the end iterator for scan-order iteration over the entire chunked array.
const_iterator	end () const
	Create the end iterator for read-only scan-order iteration over the entire chunked array.
value_type	getItem (shape_type const &point) const
	Read the array element at index 'point'. More...
bool	isInside (shape_type const &p) const
	Check if the given point is in the array domain.
template<class U , class C1 >
bool	operator!= (MultiArrayView< N, U, C1 > const &rhs) const
	Check if two arrays differ in at least one element.
template<class U , class C1 >
bool	operator== (MultiArrayView< N, U, C1 > const &rhs) const
	Check if two arrays are elementwise equal.
std::size_t	overheadBytes () const
	Bytes of main memory needed to manage the chunked storage.
virtual std::size_t	overheadBytesPerChunk () const =0
	Bytes of main memory needed to manage a single chunk.
void	releaseChunks (shape_type const &start, shape_type const &stop, bool destroy=false)
void	setCacheMaxSize (std::size_t c)
	Set the number of chunks the cache will hold. More...
void	setItem (shape_type const &point, value_type const &v)
	Write the array element at index 'point'. More...
shape_type const &	shape () const
	Return the shape in this array.
MultiArrayIndex	size () const
	Return the number of elements in this array.
view_type	subarray (shape_type const &start, shape_type const &stop)
	Create a view to the specified ROI. More...
const_view_type	subarray (shape_type const &start, shape_type const &stop) const
	Create a read-only view to the specified ROI. More...

Detailed Description

template<unsigned int N, class T>
class vigra::ChunkedArray< N, T >

Interface and base class for chunked arrays.

Very big data arrays (possibly bigger than the available RAM) can only be processed in smaller pieces. To support quick access to these pieces, it is advantegeous to store big arrays in chunks, i.e. as a collection of small rectagular subarrays. The class ChunkedArray encapsulates storage and handling of these chunks and provides various APIs to easily access the data.

#include <vigra/multi_array_chunked.hxx>
Namespace: vigra

Template Parameters

N	the array dimension
T	the type of the array elements

(these are the same as in MultiArrayView). The actual way of chunk storage is determined by the derived class the program uses:

• ChunkedArrayFull: Provides the chunked array API for a standard MultiArray (i.e. there is only one chunk for the entire array).

• ChunkedArrayLazy: All chunks reside in memory, but are only allocated upon first access.

• ChunkedArrayCompressed: Like ChunkedArrayLazy, but temporarily unused chunks are compressed in memory to save space.

• ChunkedArrayTmpFile: Chunks are stored in a memory-mapped file. Temporarily unused chunks are written to the hard-drive and deleted from memory.

• ChunkedArrayHDF5: Chunks are stored in a HDF5 dataset by means of HDF5's native chunked storage capabilities. Temporarily unused chunks are written to the hard-drive in compressed form and deleted from memory.

You must use these derived classes to construct a chunked array because ChunkedArray itself is an abstract class.

Chunks can be in one of the following states:

• uninitialized: Chunks are only initialized (i.e. allocated) upon the first write access. If an uninitialized chunk is accessed in a read-only manner, the system returns a pseudo-chunk whose elements have a user-provided fill value.

• asleep: The chunk is currently unused and has been compressed and/or swapped out to the hard drive.

• inactive: The chunk is currently unused, but still resides in memory.

• active: The chunk resides in memory and is currently in use.

• locked: Chunks are briefly in this state during transitions between the other states (e.g. while loading and/or decompression is in progress).

• failed: An unexpected error occured, e.g. the system is out of memory or a write to the hard drive failed.

In-memory chunks (active and inactive) are placed in a cache. If a chunk transitions from the 'asleep' to the 'active' state, it is added to the cache, and an 'inactive' chunk is removed and sent 'asleep'. If there is no 'inactive' chunk in the cache, the cache size is temporarily increased. All state transitions are thread-safe.

In order to optimize performance, the user should adjust the cache size (via setCacheMaxSize() or ChunkedArrayOptions) so that it can hold all chunks that are frequently needed (e.g. all chunks forming a row of the full array).

Another performance critical parameter is the chunk shape. While the system uses sensible defaults (512² for 2D arrays, 64³ for 3D, 64x64x16x4 for 4D, and 64x64x16x4x4 for 5D), the shape may need to be adjusted via the array's constructor to match the access patterns of the algorithms to be used. For speed reasons, chunk shapes must be powers of 2.

The data in the array can be accessed in several ways. The simplest is via calls to checkoutSubarray() and commitSubarray(): These functions copy an arbitrary subregion of a chunked array (possibly straddling many chunks) into a standard MultiArrayView for processing, and write results back into the chunked array:

ChunkedArray<3, float> & chunked_array = ...;

Shape3 roi_start(1000, 500, 500);
MultiArray<3, float> work_array(Shape3(100, 100, 100));

// copy data from region (1000,500,500)...(1100,600,600)
chunked_array.checkoutSubarray(roi_start, work_array);

... // work phase: process data in work_array as usual

// write results back into chunked_array
chunked_array.commitSubarray(roi_start, work_array);

The required chunks in chunked_array will only be active while the checkout and commit calls are executing. During the work phase, other threads can use the chunked array's cache to checkout or commit different subregions.

Alternatively, one can work directly on the chunk storage. This is most easily achieved by means of chunk iterators:

ChunkedArray<3, float> & chunked_array = ...;

// define the ROI to be processed
Shape3 roi_start(100, 200, 300), roi_end(1000, 2000, 600);

// get a pair of chunk iterators ( = iterators over chunks)
auto chunk = chunked_array.chunk_begin(roi_start, roi_end),
     end   = chunked_array.chunk_end(roi_start, roi_end);

// iterate over the chunks in the ROI
for(; chunk != end; ++chunk)
{
    // get a view to the current chunk's data
    // Note: The view actually refers to the intersection of the
    //       current chunk with the ROI. Thus, chunks which are
    //       partially outside the ROI are appropriately trimmed.
    MultiArrayView<3, float> chunk_view = *chunk;

    ... // work phase: process data in chunk_view as usual
}

No memory is duplicated in this approach, and only the current chunk needs to be active, so that a small chunk cache is sufficient. The iteration over chunks can be distributed over several threads that process the array data in parallel. The programmer must make sure that write operations to individual elements are synchronized between threads. This is usually achieved by ensuring that the threads are responsible for non-overlapping regions of the output array.

An even simpler method is direct element access via indexing. However, the chunked array has no control over the access order in this case, so it must potentially activate the present chunk upon each access. This is rather expensive and should only be used for debugging:

ChunkedArray<3, float> & chunked_array = ...;

Shape3 index(100, 200, 300);
// access data at coordinate 'index'
chunked_array.setItem(index, chunked_array.getItem(index) + 2.0);

Two additional APIs provide access in a way compatible with an ordinary MultiArrayView. These APIs should be used in functions that are supposed to work unchanged on both ordinary and chunked arrays. The first possibility is the chunked scan-order iterator:

ChunkedArray<3, float> & chunked_array = ...;

// get a pair of scan-order iterators ( = iterators over elements)
auto iter = chunked_array.begin(),
     end  = chunked_array.end();

// iterate over all array elements
for(; iter != end; ++iter)
{
    // access current element
     iter = *iter + 2.0;
}

A new chunk must potentially be activated whenever the iterator crosses a chunk boundary. Since the overhead of the activation operation can be amortized over many within-chunk steps, the iteration (excluding the workload within the loop) takes only twice as long as the iteration over an unstrided array using an ordinary StridedScanOrderIterator.

The final possibility is the creation of a MultiArrayView that accesses an arbitrary ROI directly:

ChunkedArray<3, float> & chunked_array = ...;

// define the ROI to be processed
Shape3 roi_start(100, 200, 300), roi_end(1000, 2000, 600);

// create view for ROI
MultiArrayView<3, float, ChunkedArrayTag> view =
                chunked_array.subarray(roi_start, roi_stop);

... // work phase: process view like any ordinary MultiArrayView

Similarly, a lower-dimensional view can be created with one of the bind functions. This approach has the advantage that 'view' can be passed to any function which is implemented in terms of MultiArrayViews. However, there are two disadvantages: First, data access in the view requires two steps (first find the chunk, then find the appropriate element in the chunk), which causes the chunked view to be slower than an ordinary MultiArrayView. Second, all chunks intersected by the view must remain active throughout the view's lifetime, which may require a big chunk cache and thus keeps many chunks in memory.

Member Function Documentation

std::size_t dataBytes

(

)

const

Bytes of main memory occupied by the array's data.

Compressed chunks are only counted with their compressed size. Chunks swapped out to the hard drive are not counted.

shape_type chunkStop

(

shape_type

global_stop

)

const

Find the chunk that is beyond array element 'global_stop'.

Specifically, this computes

chunkStart(global_stop - shape_type(1)) + shape_type(1)

shape_type chunkShape

(

shape_type const &

chunk_index

)

const

Find the shape of the chunk indexed by 'chunk_index'.

This may differ from the global chunk shape because chunks at the right/lower border of the array may be smaller than usual.

shape_type const& chunkShape

(

)

const

Return the global chunk shape.

This is the shape of all chunks that are completely contained in the array's domain.

void releaseChunks	(	shape_type const &	start,
		shape_type const &	stop,
		bool	destroy = false
	)

Sends all chunks asleep which are completely inside the given ROI. If destroy == true and the backend supports destruction (currently: ChunkedArrayLazy and ChunkedArrayCompressed), chunks will be deleted entirely. The chunk's contents after releaseChunks() are undefined. Currently, chunks retain their values when sent asleep, and assume the array's fill_value when deleted, but applications should not rely on this behavior.

void checkoutSubarray	(	shape_type const &	start,
		MultiArrayView< N, U, Stride > &	subarray
	)		const

Copy an ROI of the chunked array into an ordinary MultiArrayView.

The ROI's lower bound is given by 'start', its upper bound (in 'beyond' sense) is 'start + subarray.shape()'. Chunks in the ROI are only activated while the read is in progress.

void commitSubarray	(	shape_type const &	start,
		MultiArrayView< N, U, Stride > const &	subarray
	)

Copy an ordinary MultiArrayView into an ROI of the chunked array.

The ROI's lower bound is given by 'start', its upper bound (in 'beyond' sense) is 'start + subarray.shape()'. Chunks in the ROI are only activated while the write is in progress.

view_type subarray	(	shape_type const &	start,
		shape_type const &	stop
	)

Create a view to the specified ROI.

The view can be used like an ordinary MultiArrayView, but is a but slower. All chunks intersecting the view remain active throughout the view's lifetime.

const_view_type subarray	(	shape_type const &	start,
		shape_type const &	stop
	)		const

Create a read-only view to the specified ROI.

The view can be used like an ordinary MultiArrayView, but is a but slower. All chunks intersecting the view remain active throughout the view's lifetime.

const_view_type const_subarray	(	shape_type const &	start,
		shape_type const &	stop
	)		const

Create a read-only view to the specified ROI.

The view can be used like an ordinary MultiArrayView, but is a but slower. All chunks intersecting the view remain active throughout the view's lifetime.

value_type getItem

(

shape_type const &

point

)

const

Read the array element at index 'point'.

Since the corresponding chunk must potentially be activated first, this function may be slow and should mainly be used in debugging.

void setItem	(	shape_type const &	point,
		value_type const &	v
	)

Write the array element at index 'point'.

Since the corresponding chunk must potentially be activated first, this function may be slow and should mainly be used in debugging.

MultiArrayView<N-1, T, ChunkedArrayTag> bindAt	(	MultiArrayIndex	dim,
		MultiArrayIndex	index
	)		const

Create a lower dimensional view to the chunked array.

Dimension 'dim' is bound at 'index', all other dimensions remain unchanged. All chunks intersecting the view remain active throughout the view's lifetime.

MultiArrayView<N-1, T, ChunkedArrayTag> bind

(

difference_type_1

index

)

const

Create a lower dimensional view to the chunked array.

Dimension 'M' (given as a template parameter) is bound at 'index', all other dimensions remain unchanged. All chunks intersecting the view remain active throughout the view's lifetime.

MultiArrayView<N-1, T, ChunkedArrayTag> bindOuter

(

difference_type_1

index

)

const

Create a lower dimensional view to the chunked array.

Dimension 'N-1' is bound at 'index', all other dimensions remain unchanged. All chunks intersecting the view remain active throughout the view's lifetime.

MultiArrayView<N-M, T, ChunkedArrayTag> bindOuter

(

const TinyVector< Index, M > &

d

)

const

Create a lower dimensional view to the chunked array.

The M rightmost dimensions are bound to the indices given in 'd'. All chunks intersecting the view remain active throughout the view's lifetime.

MultiArrayView<N-1, T, ChunkedArrayTag> bindInner

(

difference_type_1

index

)

const

Create a lower dimensional view to the chunked array.

Dimension '0' is bound at 'index', all other dimensions remain unchanged. All chunks intersecting the view remain active throughout the view's lifetime.

MultiArrayView<N-M, T, ChunkedArrayTag> bindInner

(

const TinyVector< Index, M > &

d

)

const

Create a lower dimensional view to the chunked array.

The M leftmost dimensions are bound to the indices given in 'd'. All chunks intersecting the view remain active throughout the view's lifetime.

std::size_t cacheMaxSize

(

)

const

Get the number of chunks the cache will hold.

If there are any inactive chunks in the cache, these will be sent asleep until the max cahce size is reached. The max cache size may be temporarily overridden when more chunks need to be active simultaneously.

void setCacheMaxSize

(

std::size_t

c

)

Set the number of chunks the cache will hold.

This should be big enough to hold all chunks that are frequently needed and must therefore be adopted to the application's access pattern.

---

The documentation for this class was generated from the following file:

• vigra/multi_array_chunked.hxx