Skip to content Skip to navigation

Connexions

You are here: Home » Content » Portable Large File Access By Memory Mapped I/O in C++ Using Boost

Navigation

Recently Viewed

This feature requires Javascript to be enabled.

Portable Large File Access By Memory Mapped I/O in C++ Using Boost

Module by: Prakash Manandhar. E-mail the author

User rating (How does the rating system work?)
Ratings

Ratings allow you to judge the quality of modules. If other users have ranked the module then its average rating is displayed below. Ratings are calculated on a scale from one star (Poor) to five stars (Excellent).

How to rate a module

Hover over the star that corresponds to the rating you wish to assign. Click on the star to add your rating. Your rating should be based on the quality of the content. You must have an account and be logged in to rate content.

:
(0 ratings)

Summary: This is a short tutorial describing how to read large files (for example larger than 4GB) using memory mapped i/o in the Boost C++ API.

Reading large files in C++ can be tricky and platform dependent. Most modern operating systems support large file access in some form or another, and support for large file access is getting better. However, it can be tricky at times to write portable C++ code to read large files. Boost is a portable C++ source library which I have used to write portable code to read/write large files.

boost.iostreams has a mapped_file_source class that can be used to map files in the file system to arrays in memory, in read-only mode (there is another sink class for write only access). Although this supports pretty large files (1 GB or more), very large files can be problematic. Part of the interface for the class is shown below:

namespace boost { namespace iostreams {

class mapped_file_source {
public:
    ...
    explicit mapped_file_source( const std::string& path,
                                 size_type length = max_length,
                                 boost::intmax_t offset = 0 );
    void open( const std::string& path,
               size_type length = max_length,
               boost::intmax_t offset = 0 );
    bool is_open() const;
    void close();
    ...
};

} } // End namespace boost::io

We can either use the constructor or the open method to create the mapped source. If we create a map that is too large by setting the length parameter, to for example 4GB, we will get an access denied runtime exception. This tells us that we should try a smaller length value. Let us call this the page size (this is set to 1GB in the example below by using: #define MMAP_SIZE 1073741824). Then we can use a simple page shift process while we are serially reading the file. Random access can also be programmed but it might be slightly complicated. An example implementation for serial access is given below:

#define MMAP_SIZE 1073741824

unsigned long int page_start = 0;
unsigned long int file_pointer = 0;
unsigned long int page = 0;

bio::mapped_file_source m_file(RF_DATA_FILE, MMAP_SIZE, 0);

// reads specified number of bytes from the file
void read_bytes (void * buffer, unsigned long int num_bytes)
{
    assert (num_bytes < MMAP_SIZE);
    const unsigned long int end_pointer = file_pointer + num_bytes;
    if (end_pointer >= MMAP_SIZE) // repage
        {
            m_file.close ();
            m_file.open (RF_DATA_FILE, MMAP_SIZE, file_pointer);
            page_start += file_pointer;
            file_pointer = 0;
        }
        memcpy(buffer, m_file.data() + file_pointer, num_bytes);
        file_pointer += num_bytes;
}

Content actions

Give Feedback:

E-mail the module author | Rate module ( How does the rating system work?)

Rating system

Ratings

Ratings allow you to judge the quality of modules. If other users have ranked the module then its average rating is displayed below. Ratings are calculated on a scale from one star (Poor) to five stars (Excellent).

How to rate a module

Hover over the star that corresponds to the rating you wish to assign. Click on the star to add your rating. Your rating should be based on the quality of the content. You must have an account and be logged in to rate content.

(0 ratings)

Download:

Module PDF

Add module to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections directly in Connexions. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need a Connexions account to use 'My Favorites'.

| A lens (?)

Definition of a lens

Lenses

A lens is a custom view of Connexions content. You can think of it as a fancy kind of list that will let you see Connexions through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to Connexions materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual Connexions member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks