sella::file::GZCSVFileReader Class Reference

#include <GZCSVFileReader.h>

List of all members.

Public Member Functions
	STANDARD_TYPEDEF (sella::file::GZCSVFileReader)
	GZCSVFileReader (const std::string &file, char delimiter=DefaultDelimiter, char seperator=DefaultSeperator) THROWS(IOException)
virtual	~GZCSVFileReader () NO_EXCEPTIONS
void	read (std::vector< std::vector< std::string >> &data) THROWS(IOException)
void	close (bool unlink=false) NO_EXCEPTIONS
Static Public Attributes
static const char	DefaultDelimiter = 0x1e
	The default delimiter used between the fields of a record; the ASCII record seperator character.
static const char	DefaultSeperator = 0x1d
	The default seperator used between the records of a file; the ASCII group seperator character.
Protected Member Functions
bool	read (std::vector< std::string > &data, std::string prepend) THROWS(IOException)
bool	read (std::vector< std::string > &data) THROWS(IOException)

---

Detailed Description

A GZIP compressed CSV reader.

Definition at line 31 of file GZCSVFileReader.h.

---

Constructor & Destructor Documentation

GZCSVFileReader::GZCSVFileReader	(	const std::string &	file,
		char	delimiter = DefaultDelimiter,
		char	seperator = DefaultSeperator
	)

The base constructor for a file reader.

Parameters:

[in]

file

The name of the database file to read.

Exceptions:

IOException

if the file cannot be opened for reading.

Definition at line 22 of file GZCSVFileReader.cpp.

                                                                                                           :
        filename(file),

        delimiter(delimiter),
        seperator(seperator),

        descriptor(::open(file.c_str(), O_RDONLY | O_LARGEFILE))
{
        /* Ensure the file was opened for reading, and zero all buffers and state machines. */
        if (this->descriptor <= 0) {
                THROW(IOException, "failed to open file '%s' for reading", this->filename.c_str());
        }
        memset(&(this->stream), 0, sizeof(this->stream));
        memset(&(this->buffer), 0, sizeof(this->buffer));

        /* Initialize for the reading of GZIP compressed files. */
        if (::inflateInit2(&(this->stream), 16 + MAX_WBITS) != Z_OK) {
		::close(this->descriptor);

                THROW(IOException, "failed to initilize zlib instance for reading file '%s'", this->filename.c_str());
        }
        this->stream.avail_out = sizeof(this->buffer[1]);
}

GZCSVFileReader::~GZCSVFileReader

(

)

[virtual]

The class destructor; this destructor closes the file opened by this instance if it was not explicitly closed by the class user.

Definition at line 50 of file GZCSVFileReader.cpp.

References close().

                                                {
        if (this->descriptor != 0) {
                ::inflateEnd(&(this->stream));

                this->close();
        }
}

---

Member Function Documentation

void GZCSVFileReader::close

(

bool

unlink = false

)

Close the file opened by this instance. Failure to explicitly call this function will not cause a file descriptor leak if the class destructor is called: the destructor will call close() automatically if necessary.

Parameters:

[in]

unlink

When set to true, the file will be unlinked on close.

Definition at line 104 of file GZCSVFileReader.cpp.

Referenced by sella::file::GZCSVPathReader::read(), and ~GZCSVFileReader().

                                                     {
	::close(this->descriptor);

        if (unlink == true) {
                ::unlink(this->filename.c_str());
        }
        this->filename.clear();
        this->descriptor = 0;
}

bool GZCSVFileReader::read

(

std::vector< std::string > &

data

)

[protected]

Read the next record from the file and return all fields.

Parameters:

[out]

data

The vector in which to store all fields from the returned record.

Exceptions:

IOException

if unabled to read the next record for any reason.

Returns:

true if the next record was returned, or false if no more records area available in the file.

Definition at line 89 of file GZCSVFileReader.cpp.

                                                                          {
        /* Read field entries from the memory buffer, and as necessary fall back to reading from the disk. */
        while (this->read(data, std::string()) == false) {
                /* EMPTY LOOP */
        }
        return data.empty() ? false : true;
}

bool GZCSVFileReader::read	(	std::vector< std::string > &	data,
		std::string	prepend
	)			[protected]

Read the next field from the file. If required, this function will obtain additional data from the file and/or decompress additional data from the compressed data buffer.

Parameters:

[out]	data	The vector in which to store all fields from the returned record; if not empty when called, the function will clear the vector before writing the next set of fields to it.
[in]	prepend	A text value to prepend the the next field value; this is used for recursive calls on long or broken up (due to buffer size) fields.

Exceptions:

IOException

if unabled to read the next field for any reason.

Returns:

true if the end of the record was reached, else returns false.

Definition at line 131 of file GZCSVFileReader.cpp.

                                                                                             {
        uint8_t * head = this->stream.next_out - (sizeof(this->buffer[1]) - this->stream.avail_out);

        /* Determine if there is decopmressed data available. */
        if (this->stream.avail_out == sizeof(this->buffer[1])) {
                /* There is no decopressed data available, so determine if there is compressed data available in memory. */
                if (this->stream.avail_in == 0) {
                        /* There is no compressed data available, so read data from file into the compressed data buffer. */
                        if ((this->stream.avail_in = ::read(this->descriptor, this->buffer[0], sizeof(this->buffer[0]))) < 0) {
                                THROW(IOException, "failed to read from file '%s'", this->filename.c_str());
                        } else if (this->stream.avail_in == 0) {
                                return true; /* The end of the file was reached, and no more data could be read. */
                        }
                        this->stream.next_in = this->buffer[0];
                }

                /* Decompress data from the compressed buffer into the uncompressed buffer. */
                this->stream.avail_out = sizeof(this->buffer[1]);
                this->stream.next_out = this->buffer[1];

                switch (::inflate(&(this->stream), Z_SYNC_FLUSH)) {
                        case Z_STREAM_ERROR:
                                THROW(IOException, "zlib decompression state was destroyed");

                        case Z_NEED_DICT:
                                THROW(IOException, "zlib decompression failed due to missing dictionary");

                        case Z_DATA_ERROR:
                                THROW(IOException, "zlib decompression failed due to bad data in stream");

                        case Z_MEM_ERROR:
                                THROW(IOException, "zlib decompression failed due to a memory error");
                }
                head = this->stream.next_out - (sizeof(this->buffer[1]) - this->stream.avail_out);
        }

        /* Adjust the limits of the decompressed memory buffer. */
        for (uint8_t * i = head; i != this->stream.next_out; i++) {
                if ((*i) == this->seperator) { /* The end of the record has been reached; store the last field and return success. */
                        data.push_back(prepend.append(std::string(reinterpret_cast<char *>(head), i - head)));
                        this->stream.avail_out += (i - head + 1);

                        return true; /* The record request has been fullfilled. */
                } else if (*(i) == this->delimiter) { /* A record delimiter has been identified. */
                        data.push_back(prepend.append(std::string(reinterpret_cast<char *>(head), i - head)));
                        this->stream.avail_out += (i - head + 1);

                        return false; /* The record request has not yet been fullfilled. */
                }
        }
        this->stream.avail_out = sizeof(this->buffer[1]);

        /* Recursively call this function with a prepend value supplied to account for a field which is too long for the buffers. */
        return this->read(data, std::string(reinterpret_cast<char *>(head), this->stream.next_out - head));
}

void GZCSVFileReader::read

(

std::vector< std::vector< std::string >> &

data

)

Read the all fields from the file.

Parameters:

[out]

data

The vector in which to store all data from the file.

Exceptions:

IOException

if unabled to read the file for any reason.

Definition at line 65 of file GZCSVFileReader.cpp.

Referenced by sella::file::GZCSVPathReader::read().

                                                                                     {
        do {
                if ((data.empty() == false) && data.back().empty()) {
                        data.pop_back();
                }
                data.push_back(std::vector<std::string>());
        } while (this->read(data.back()) == true);

        if ((data.empty() == false) && data.back().empty()) {
                data.pop_back();
        }
}

---

The documentation for this class was generated from the following files:

• /builddir/build/BUILD/libutil++-1.4.0/src/sella/file/GZCSVFileReader.h
• /builddir/build/BUILD/libutil++-1.4.0/src/sella/file/GZCSVFileReader.cpp