Version 1.0.5 of 10 December 2007
Copyright © 1996-2007 Julian Seward
This program, bzip2
, the
associated library libbzip2
, and
all documentation, are copyright © 1996-2007 Julian Seward.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
The origin of this software must not be misrepresented; you must not claim that you wrote the original software. If you use this software in a product, an acknowledgment in the product documentation would be appreciated but is not required.
Altered source versions must be plainly marked as such, and must not be misrepresented as being the original software.
The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission.

PATENTS: To the best of my knowledge,
bzip2
and
libbzip2
do not use any patented
algorithms. However, I do not have the resources to carry
out a patent search. Therefore I cannot give any guarantee of
the above statement.
Table of Contents
libbzip2
bzip2
compresses files
using the Burrows-Wheeler block-sorting text compression
algorithm, and Huffman coding. Compression is generally
considerably better than that achieved by more conventional
LZ77/LZ78-based compressors, and approaches the performance of
the PPM family of statistical compressors.
bzip2
is built on top of
libbzip2
, a flexible library for
handling compressed data in the
bzip2
format. This manual
describes both how to use the program and how to work with the
library interface. Most of the manual is devoted to this
library, not the program, which is good news if your interest is
only in the program.
How to use bzip2 describes how to use
bzip2
; this is the only part
you need to read if you just want to know how to operate the
program.
Programming with libbzip2 describes the programming interfaces in detail, and
Miscellanea records some miscellaneous notes which I thought ought to be recorded somewhere.
Table of Contents
This chapter contains a copy of the
bzip2
man page, and nothing
else.
bzip2
,
bunzip2
- a block-sorting file
compressor, v1.0.4
bzcat
-
decompresses files to stdout
bzip2recover
-
recovers data from damaged bzip2 files
bzip2
[
-cdfkqstvzVL123456789 ] [ filenames ... ]
bunzip2
[
-fkvsVL ] [ filenames ... ]
bzcat
[ -s ] [
filenames ... ]
bzip2recover
filename
bzip2
compresses files
using the Burrows-Wheeler block sorting text compression
algorithm, and Huffman coding. Compression is generally
considerably better than that achieved by more conventional
LZ77/LZ78-based compressors, and approaches the performance of
the PPM family of statistical compressors.
The command-line options are deliberately very similar to
those of GNU gzip
, but they are
not identical.
bzip2
expects a list of
file names to accompany the command-line flags. Each file is
replaced by a compressed version of itself, with the name
original_name.bz2
. Each
compressed file has the same modification date, permissions, and,
when possible, ownership as the corresponding original, so that
these properties can be correctly restored at decompression time.
File name handling is naive in the sense that there is no
mechanism for preserving original file names, permissions,
ownerships or dates in filesystems which lack these concepts, or
have serious file name length restrictions, such as
MS-DOS.
bzip2
and
bunzip2
will by default not
overwrite existing files. If you want this to happen, specify
the -f
flag.
If no file names are specified,
bzip2
compresses from standard
input to standard output. In this case,
bzip2
will decline to write
compressed output to a terminal, as this would be entirely
incomprehensible and therefore pointless.
bunzip2
(or
bzip2 -d
) decompresses all
specified files. Files which were not created by
bzip2
will be detected and
ignored, and a warning issued.
bzip2
attempts to guess the
filename for the decompressed file from that of the compressed
file as follows:
filename.bz2
becomes
filename
filename.bz
becomes
filename
filename.tbz2
becomes
filename.tar
filename.tbz
becomes
filename.tar
anyothername
becomes
anyothername.out
If the file does not end in one of the recognised endings,
.bz2
,
.bz
,
.tbz2
or
.tbz
,
bzip2
complains that it cannot
guess the name of the original file, and uses the original name
with .out
appended.
As with compression, supplying no filenames causes decompression from standard input to standard output.
bunzip2
will correctly
decompress a file which is the concatenation of two or more
compressed files. The result is the concatenation of the
corresponding uncompressed files. Integrity testing
(-t
) of concatenated compressed
files is also supported.
You can also compress or decompress files to the standard
output by giving the -c
flag.
Multiple files may be compressed and decompressed like this. The
resulting outputs are fed sequentially to stdout. Compression of
multiple files in this manner generates a stream containing
multiple compressed file representations. Such a stream can be
decompressed correctly only by
bzip2
version 0.9.0 or later.
Earlier versions of bzip2
will
stop after decompressing the first file in the stream.
bzcat
(or
bzip2 -dc
) decompresses all
specified files to the standard output.
bzip2
will read arguments
from the environment variables
BZIP2
and
BZIP
, in that order, and will
process them before any arguments read from the command line.
This gives a convenient way to supply default arguments.
Compression is always performed, even if the compressed file is slightly larger than the original. Files of less than about one hundred bytes tend to get larger, since the compression mechanism has a constant overhead in the region of 50 bytes. Random data (including the output of most file compressors) is coded at about 8.05 bits per byte, giving an expansion of around 0.5%.
As a self-check for your protection,
bzip2
uses 32-bit CRCs to make
sure that the decompressed version of a file is identical to the
original. This guards against corruption of the compressed data,
and against undetected bugs in
bzip2
(hopefully very unlikely).
The chances of data corruption going undetected is microscopic,
about one chance in four billion for each file processed. Be
aware, though, that the check occurs upon decompression, so it
can only tell you that something is wrong. It can't help you
recover the original uncompressed data. You can use
bzip2recover
to try to recover
data from damaged files.
Return values: 0 for a normal exit, 1 for environmental
problems (file not found, invalid flags, I/O errors, etc.), 2
to indicate a corrupt compressed file, 3 for an internal
consistency error (eg, bug) which caused
bzip2
to panic.
-c --stdout
Compress or decompress to standard output.
-d --decompress
Force decompression.
bzip2
,
bunzip2
and
bzcat
are really the same
program, and the decision about what actions to take is done on
the basis of which name is used. This flag overrides that
mechanism, and forces bzip2 to decompress.
-z --compress
The complement to
-d
: forces compression,
regardless of the invokation name.
-t --test
Check integrity of the specified file(s), but don't decompress them. This really performs a trial decompression and throws away the result.
-f --force
Force overwrite of output files. Normally,
bzip2
will not overwrite
existing output files. Also forces
bzip2
to break hard links to
files, which it otherwise wouldn't do.
bzip2
normally declines
to decompress files which don't have the correct magic header
bytes. If forced (-f
),
however, it will pass such files through unmodified. This is
how GNU gzip
behaves.
-k --keep
Keep (don't delete) input files during compression or decompression.
-s --small
Reduce memory usage, for compression, decompression and testing. Files are decompressed and tested using a modified algorithm which only requires 2.5 bytes per block byte. This means any file can be decompressed in 2300k of memory, albeit at about half the normal speed.
During compression, -s
selects a block size of 200k, which limits memory use to around
the same figure, at the expense of your compression ratio. In
short, if your machine is low on memory (8 megabytes or less),
use -s
for everything. See
MEMORY MANAGEMENT below.
-q --quiet
Suppress non-essential warning messages. Messages pertaining to I/O errors and other critical events will not be suppressed.
-v --verbose
Verbose mode -- show the compression ratio for
each file processed. Further
-v
's increase the verbosity
level, spewing out lots of information which is primarily of
interest for diagnostic purposes.
-L --license -V --version
Display the software version, license terms and conditions.
-1
(or
--fast
) to
-9
(or
-best
)Set the block size to 100 k, 200 k ... 900 k
when compressing. Has no effect when decompressing. See MEMORY MANAGEMENT below. The
--fast
and
--best
aliases are primarily
for GNU gzip
compatibility.
In particular, --fast
doesn't
make things significantly faster. And
--best
merely selects the
default behaviour.
--
Treats all subsequent arguments as file names,
even if they start with a dash. This is so you can handle
files with names beginning with a dash, for example:
bzip2 --
-myfilename
.
--repetitive-fast
, --repetitive-best
These flags are redundant in versions 0.9.5 and above. They provided some coarse control over the behaviour of the sorting algorithm in earlier versions, which was sometimes useful. 0.9.5 and above have an improved algorithm which renders these flags irrelevant.
bzip2
compresses large
files in blocks. The block size affects both the compression
ratio achieved, and the amount of memory needed for compression
and decompression. The flags -1
through -9
specify the block
size to be 100,000 bytes through 900,000 bytes (the default)
respectively. At decompression time, the block size used for
compression is read from the header of the compressed file, and
bunzip2
then allocates itself
just enough memory to decompress the file. Since block sizes are
stored in compressed files, it follows that the flags
-1
to
-9
are irrelevant to and so
ignored during decompression.
Compression and decompression requirements, in bytes, can be estimated as:
Compression: 400k + ( 8 x block size ) Decompression: 100k + ( 4 x block size ), or 100k + ( 2.5 x block size )
Larger block sizes give rapidly diminishing marginal
returns. Most of the compression comes from the first two or
three hundred k of block size, a fact worth bearing in mind when
using bzip2
on small machines.
It is also important to appreciate that the decompression memory
requirement is set at compression time by the choice of block
size.
For files compressed with the default 900k block size,
bunzip2
will require about 3700
kbytes to decompress. To support decompression of any file on a
4 megabyte machine, bunzip2
has
an option to decompress using approximately half this amount of
memory, about 2300 kbytes. Decompression speed is also halved,
so you should use this option only where necessary. The relevant
flag is -s
.
In general, try and use the largest block size memory constraints allow, since that maximises the compression achieved. Compression and decompression speed are virtually unaffected by block size.
Another significant point applies to files which fit in a
single block -- that means most files you'd encounter using a
large block size. The amount of real memory touched is
proportional to the size of the file, since the file is smaller
than a block. For example, compressing a file 20,000 bytes long
with the flag -9
will cause the
compressor to allocate around 7600k of memory, but only touch
400k + 20000 * 8 = 560 kbytes of it. Similarly, the decompressor
will allocate 3700k but only touch 100k + 20000 * 4 = 180
kbytes.
Here is a table which summarises the maximum memory usage for different block sizes. Also recorded is the total compressed size for 14 files of the Calgary Text Compression Corpus totalling 3,141,622 bytes. This column gives some feel for how compression varies with block size. These figures tend to understate the advantage of larger block sizes for larger files, since the Corpus is dominated by smaller files.
Compress Decompress Decompress Corpus Flag usage usage -s usage Size -1 1200k 500k 350k 914704 -2 2000k 900k 600k 877703 -3 2800k 1300k 850k 860338 -4 3600k 1700k 1100k 846899 -5 4400k 2100k 1350k 845160 -6 5200k 2500k 1600k 838626 -7 6100k 2900k 1850k 834096 -8 6800k 3300k 2100k 828642 -9 7600k 3700k 2350k 828642
bzip2
compresses files in
blocks, usually 900kbytes long. Each block is handled
independently. If a media or transmission error causes a
multi-block .bz2
file to become
damaged, it may be possible to recover data from the undamaged
blocks in the file.
The compressed representation of each block is delimited by a 48-bit pattern, which makes it possible to find the block boundaries with reasonable certainty. Each block also carries its own 32-bit CRC, so damaged blocks can be distinguished from undamaged ones.
bzip2recover
is a simple
program whose purpose is to search for blocks in
.bz2
files, and write each block
out into its own .bz2
file. You
can then use bzip2 -t
to test
the integrity of the resulting files, and decompress those which
are undamaged.
bzip2recover
takes a
single argument, the name of the damaged file, and writes a
number of files rec0001file.bz2
,
rec0002file.bz2
, etc, containing
the extracted blocks. The output filenames are designed so that
the use of wildcards in subsequent processing -- for example,
bzip2 -dc rec*file.bz2 >
recovered_data
-- lists the files in the correct
order.
bzip2recover
should be of
most use dealing with large .bz2
files, as these will contain many blocks. It is clearly futile
to use it on damaged single-block files, since a damaged block
cannot be recovered. If you wish to minimise any potential data
loss through media or transmission errors, you might consider
compressing with a smaller block size.
The sorting phase of compression gathers together similar
strings in the file. Because of this, files containing very long
runs of repeated symbols, like "aabaabaabaab ..." (repeated
several hundred times) may compress more slowly than normal.
Versions 0.9.5 and above fare much better than previous versions
in this respect. The ratio between worst-case and average-case
compression time is in the region of 10:1. For previous
versions, this figure was more like 100:1. You can use the
-vvvv
option to monitor progress
in great detail, if you want.
Decompression speed is unaffected by these phenomena.
bzip2
usually allocates
several megabytes of memory to operate in, and then charges all
over it in a fairly random fashion. This means that performance,
both for compressing and decompressing, is largely determined by
the speed at which your machine can service cache misses.
Because of this, small changes to the code to reduce the miss
rate have been observed to give disproportionately large
performance improvements. I imagine
bzip2
will perform best on
machines with very large caches.
I/O error messages are not as helpful as they could be.
bzip2
tries hard to detect I/O
errors and exit cleanly, but the details of what the problem is
sometimes seem rather misleading.
This manual page pertains to version 1.0.5 of
bzip2
. Compressed data created by
this version is entirely forwards and backwards compatible with the
previous public releases, versions 0.1pl2, 0.9.0 and 0.9.5, 1.0.0,
1.0.1, 1.0.2 and 1.0.3, but with the following exception: 0.9.0 and
above can correctly decompress multiple concatenated compressed files.
0.1pl2 cannot do this; it will stop after decompressing just the first
file in the stream.
bzip2recover
versions
prior to 1.0.2 used 32-bit integers to represent bit positions in
compressed files, so it could not handle compressed files more
than 512 megabytes long. Versions 1.0.2 and above use 64-bit ints
on some platforms which support them (GNU supported targets, and
Windows). To establish whether or not
bzip2recover
was built with such
a limitation, run it without arguments. In any event you can
build yourself an unlimited version if you can recompile it with
MaybeUInt64
set to be an
unsigned 64-bit integer.
Julian Seward,
<EMAIL_ADDRESS>The ideas embodied in
bzip2
are due to (at least) the
following people: Michael Burrows and David Wheeler (for the
block sorting transformation), David Wheeler (again, for the
Huffman coder), Peter Fenwick (for the structured coding model in
the original bzip
, and many
refinements), and Alistair Moffat, Radford Neal and Ian Witten
(for the arithmetic coder in the original
bzip
). I am much indebted for
their help, support and advice. See the manual in the source
distribution for pointers to sources of documentation. Christian
von Roques encouraged me to look for faster sorting algorithms,
so as to speed up compression. Bela Lubkin encouraged me to
improve the worst-case compression performance.
Donna Robinson XMLised the documentation.
Many people sent
patches, helped with portability problems, lent machines, gave
advice and were generally helpful.
Table of Contents
This chapter describes the programming interface to
libbzip2
.
For general background information, particularly about memory use and performance aspects, you'd be well advised to read How to use bzip2 as well.
libbzip2
is a flexible
library for compressing and decompressing data in the
bzip2
data format. Although
packaged as a single entity, it helps to regard the library as
three separate parts: the low level interface, and the high level
interface, and some utility functions.
The structure of
libbzip2
's interfaces is similar
to that of Jean-loup Gailly's and Mark Adler's excellent
zlib
library.
All externally visible symbols have names beginning
BZ2_
. This is new in version
1.0. The intention is to minimise pollution of the namespaces of
library clients.
To use any part of the library, you need to
#include <bzlib.h>
into your sources.
This interface provides services for compressing and
decompressing data in memory. There's no provision for dealing
with files, streams or any other I/O mechanisms, just straight
memory-to-memory work. In fact, this part of the library can be
compiled without inclusion of
stdio.h
, which may be helpful
for embedded applications.
The low-level part of the library has no global variables and is therefore thread-safe.
Six routines make up the low level interface:
BZ2_bzCompressInit
,
BZ2_bzCompress
, and
BZ2_bzCompressEnd
for
compression, and a corresponding trio
BZ2_bzDecompressInit
,
BZ2_bzDecompress
and
BZ2_bzDecompressEnd
for
decompression. The *Init
functions allocate memory for compression/decompression and do
other initialisations, whilst the
*End
functions close down
operations and release memory.
The real work is done by
BZ2_bzCompress
and
BZ2_bzDecompress
. These
compress and decompress data from a user-supplied input buffer to
a user-supplied output buffer. These buffers can be any size;
arbitrary quantities of data are handled by making repeated calls
to these functions. This is a flexible mechanism allowing a
consumer-pull style of activity, or producer-push, or a mixture
of both.
This interface provides some handy wrappers around the
low-level interface to facilitate reading and writing
bzip2
format files
(.bz2
files). The routines
provide hooks to facilitate reading files in which the
bzip2
data stream is embedded
within some larger-scale file structure, or where there are
multiple bzip2
data streams
concatenated end-to-end.
For reading files,
BZ2_bzReadOpen
,
BZ2_bzRead
,
BZ2_bzReadClose
and
BZ2_bzReadGetUnused
are
supplied. For writing files,
BZ2_bzWriteOpen
,
BZ2_bzWrite
and
BZ2_bzWriteFinish
are
available.
As with the low-level library, no global variables are used
so the library is per se thread-safe. However, if I/O errors
occur whilst reading or writing the underlying compressed files,
you may have to consult errno
to
determine the cause of the error. In that case, you'd need a C
library which correctly supports
errno
in a multithreaded
environment.
To make the library a little simpler and more portable,
BZ2_bzReadOpen
and
BZ2_bzWriteOpen
require you to
pass them file handles (FILE*
s)
which have previously been opened for reading or writing
respectively. That avoids portability problems associated with
file operations and file attributes, whilst not being much of an
imposition on the programmer.
For very simple needs,
BZ2_bzBuffToBuffCompress
and
BZ2_bzBuffToBuffDecompress
are
provided. These compress data in memory from one buffer to
another buffer in a single function call. You should assess
whether these functions fulfill your memory-to-memory
compression/decompression requirements before investing effort in
understanding the more general but more complex low-level
interface.
Yoshioka Tsuneo
<EMAIL_ADDRESS>) has
contributed some functions to give better
zlib
compatibility. These
functions are BZ2_bzopen
,
BZ2_bzread
,
BZ2_bzwrite
,
BZ2_bzflush
,
BZ2_bzclose
,
BZ2_bzerror
and
BZ2_bzlibVersion
. You may find
these functions more convenient for simple file reading and
writing, than those in the high-level interface. These functions
are not (yet) officially part of the library, and are minimally
documented here. If they break, you get to keep all the pieces.
I hope to document them properly when time permits.
Yoshioka also contributed modifications to allow the library to be built as a Windows DLL.
The library is designed to recover cleanly in all situations, including the worst-case situation of decompressing random data. I'm not 100% sure that it can always do this, so you might want to add a signal handler to catch segmentation violations during decompression if you are feeling especially paranoid. I would be interested in hearing more about the robustness of the library to corrupted compressed data.
Version 1.0.3 more robust in this respect than any previous version. Investigations with Valgrind (a tool for detecting problems with memory management) indicate that, at least for the few files I tested, all single-bit errors in the decompressed data are caught properly, with no segmentation faults, no uses of uninitialised data, no out of range reads or writes, and no infinite looping in the decompressor. So it's certainly pretty robust, although I wouldn't claim it to be totally bombproof.
The file bzlib.h
contains
all definitions needed to use the library. In particular, you
should definitely not include
bzlib_private.h
.
In bzlib.h
, the various
return values are defined. The following list is not intended as
an exhaustive description of the circumstances in which a given
value may be returned -- those descriptions are given later.
Rather, it is intended to convey the rough meaning of each return
value. The first five actions are normal and not intended to
denote an error situation.
BZ_OK
The requested action was completed successfully.
BZ_RUN_OK, BZ_FLUSH_OK,
BZ_FINISH_OK
In
BZ2_bzCompress
, the requested
flush/finish/nothing-special action was completed
successfully.
BZ_STREAM_END
Compression of data was completed, or the logical stream end was detected during decompression.
The following return values indicate an error of some kind.
BZ_CONFIG_ERROR
Indicates that the library has been improperly
compiled on your platform -- a major configuration error.
Specifically, it means that
sizeof(char)
,
sizeof(short)
and
sizeof(int)
are not 1, 2 and
4 respectively, as they should be. Note that the library
should still work properly on 64-bit platforms which follow
the LP64 programming model -- that is, where
sizeof(long)
and
sizeof(void*)
are 8. Under
LP64, sizeof(int)
is still 4,
so libbzip2
, which doesn't
use the long
type, is
OK.
BZ_SEQUENCE_ERROR
When using the library, it is important to call
the functions in the correct sequence and with data structures
(buffers etc) in the correct states.
libbzip2
checks as much as it
can to ensure this is happening, and returns
BZ_SEQUENCE_ERROR
if not.
Code which complies precisely with the function semantics, as
detailed below, should never receive this value; such an event
denotes buggy code which you should
investigate.
BZ_PARAM_ERROR
Returned when a parameter to a function call is
out of range or otherwise manifestly incorrect. As with
BZ_SEQUENCE_ERROR
, this
denotes a bug in the client code. The distinction between
BZ_PARAM_ERROR
and
BZ_SEQUENCE_ERROR
is a bit
hazy, but still worth making.
BZ_MEM_ERROR
Returned when a request to allocate memory
failed. Note that the quantity of memory needed to decompress
a stream cannot be determined until the stream's header has
been read. So
BZ2_bzDecompress
and
BZ2_bzRead
may return
BZ_MEM_ERROR
even though some
of the compressed data has been read. The same is not true
for compression; once
BZ2_bzCompressInit
or
BZ2_bzWriteOpen
have
successfully completed,
BZ_MEM_ERROR
cannot
occur.
BZ_DATA_ERROR
Returned when a data integrity error is detected during decompression. Most importantly, this means when stored and computed CRCs for the data do not match. This value is also returned upon detection of any other anomaly in the compressed data.
BZ_DATA_ERROR_MAGIC
As a special case of
BZ_DATA_ERROR
, it is
sometimes useful to know when the compressed stream does not
start with the correct magic bytes ('B' 'Z'
'h'
).
BZ_IO_ERROR
Returned by
BZ2_bzRead
and
BZ2_bzWrite
when there is an
error reading or writing in the compressed file, and by
BZ2_bzReadOpen
and
BZ2_bzWriteOpen
for attempts
to use a file for which the error indicator (viz,
ferror(f)
) is set. On
receipt of BZ_IO_ERROR
, the
caller should consult errno
and/or perror
to acquire
operating-system specific information about the
problem.
BZ_UNEXPECTED_EOF
Returned by
BZ2_bzRead
when the
compressed file finishes before the logical end of stream is
detected.
BZ_OUTBUFF_FULL
Returned by
BZ2_bzBuffToBuffCompress
and
BZ2_bzBuffToBuffDecompress
to
indicate that the output data will not fit into the output
buffer provided.
typedef struct { char *next_in; unsigned int avail_in; unsigned int total_in_lo32; unsigned int total_in_hi32; char *next_out; unsigned int avail_out; unsigned int total_out_lo32; unsigned int total_out_hi32; void *state; void *(*bzalloc)(void *,int,int); void (*bzfree)(void *,void *); void *opaque; } bz_stream; int BZ2_bzCompressInit ( bz_stream *strm, int blockSize100k, int verbosity, int workFactor );
Prepares for compression. The
bz_stream
structure holds all
data pertaining to the compression activity. A
bz_stream
structure should be
allocated and initialised prior to the call. The fields of
bz_stream
comprise the entirety
of the user-visible data. state
is a pointer to the private data structures required for
compression.
Custom memory allocators are supported, via fields
bzalloc
,
bzfree
, and
opaque
. The value
opaque
is passed to as the first
argument to all calls to bzalloc
and bzfree
, but is otherwise
ignored by the library. The call bzalloc (
opaque, n, m )
is expected to return a pointer
p
to n *
m
bytes of memory, and bzfree (
opaque, p )
should free that memory.
If you don't want to use a custom memory allocator, set
bzalloc
,
bzfree
and
opaque
to
NULL
, and the library will then
use the standard malloc
/
free
routines.
Before calling
BZ2_bzCompressInit
, fields
bzalloc
,
bzfree
and
opaque
should be filled
appropriately, as just described. Upon return, the internal
state will have been allocated and initialised, and
total_in_lo32
,
total_in_hi32
,
total_out_lo32
and
total_out_hi32
will have been
set to zero. These four fields are used by the library to inform
the caller of the total amount of data passed into and out of the
library, respectively. You should not try to change them. As of
version 1.0, 64-bit counts are maintained, even on 32-bit
platforms, using the _hi32
fields to store the upper 32 bits of the count. So, for example,
the total amount of data in is (total_in_hi32
<< 32) + total_in_lo32
.
Parameter blockSize100k
specifies the block size to be used for compression. It should
be a value between 1 and 9 inclusive, and the actual block size
used is 100000 x this figure. 9 gives the best compression but
takes most memory.
Parameter verbosity
should
be set to a number between 0 and 4 inclusive. 0 is silent, and
greater numbers give increasingly verbose monitoring/debugging
output. If the library has been compiled with
-DBZ_NO_STDIO
, no such output
will appear for any verbosity setting.
Parameter workFactor
controls how the compression phase behaves when presented with
worst case, highly repetitive, input data. If compression runs
into difficulties caused by repetitive data, the library switches
from the standard sorting algorithm to a fallback algorithm. The
fallback is slower than the standard algorithm by perhaps a
factor of three, but always behaves reasonably, no matter how bad
the input.
Lower values of workFactor
reduce the amount of effort the standard algorithm will expend
before resorting to the fallback. You should set this parameter
carefully; too low, and many inputs will be handled by the
fallback algorithm and so compress rather slowly, too high, and
your average-to-worst case compression times can become very
large. The default value of 30 gives reasonable behaviour over a
wide range of circumstances.
Allowable values range from 0 to 250 inclusive. 0 is a special case, equivalent to using the default value of 30.
Note that the compressed output generated is the same regardless of whether or not the fallback algorithm is used.
Be aware also that this parameter may disappear entirely in future versions of the library. In principle it should be possible to devise a good way to automatically choose which algorithm to use. Such a mechanism would render the parameter obsolete.
Possible return values:
BZ_CONFIG_ERROR if the library has been mis-compiled BZ_PARAM_ERROR if strm is NULL or blockSize < 1 or blockSize > 9 or verbosity < 0 or verbosity > 4 or workFactor < 0 or workFactor > 250 BZ_MEM_ERROR if not enough memory is available BZ_OK otherwise
Allowable next actions:
BZ2_bzCompress if BZ_OK is returned no specific action needed in case of error
int BZ2_bzCompress ( bz_stream *strm, int action );
Provides more input and/or output buffer space for the
library. The caller maintains input and output buffers, and
calls BZ2_bzCompress
to transfer
data between them.
Before each call to
BZ2_bzCompress
,
next_in
should point at the data
to be compressed, and avail_in
should indicate how many bytes the library may read.
BZ2_bzCompress
updates
next_in
,
avail_in
and
total_in
to reflect the number
of bytes it has read.
Similarly, next_out
should
point to a buffer in which the compressed data is to be placed,
with avail_out
indicating how
much output space is available.
BZ2_bzCompress
updates
next_out
,
avail_out
and
total_out
to reflect the number
of bytes output.
You may provide and remove as little or as much data as you
like on each call of
BZ2_bzCompress
. In the limit,
it is acceptable to supply and remove data one byte at a time,
although this would be terribly inefficient. You should always
ensure that at least one byte of output space is available at
each call.
A second purpose of
BZ2_bzCompress
is to request a
change of mode of the compressed stream.
Conceptually, a compressed stream can be in one of four
states: IDLE, RUNNING, FLUSHING and FINISHING. Before
initialisation
(BZ2_bzCompressInit
) and after
termination (BZ2_bzCompressEnd
),
a stream is regarded as IDLE.
Upon initialisation
(BZ2_bzCompressInit
), the stream
is placed in the RUNNING state. Subsequent calls to
BZ2_bzCompress
should pass
BZ_RUN
as the requested action;
other actions are illegal and will result in
BZ_SEQUENCE_ERROR
.
At some point, the calling program will have provided all
the input data it wants to. It will then want to finish up -- in
effect, asking the library to process any data it might have
buffered internally. In this state,
BZ2_bzCompress
will no longer
attempt to read data from
next_in
, but it will want to
write data to next_out
. Because
the output buffer supplied by the user can be arbitrarily small,
the finishing-up operation cannot necessarily be done with a
single call of
BZ2_bzCompress
.
Instead, the calling program passes
BZ_FINISH
as an action to
BZ2_bzCompress
. This changes
the stream's state to FINISHING. Any remaining input (ie,
next_in[0 .. avail_in-1]
) is
compressed and transferred to the output buffer. To do this,
BZ2_bzCompress
must be called
repeatedly until all the output has been consumed. At that
point, BZ2_bzCompress
returns
BZ_STREAM_END
, and the stream's
state is set back to IDLE.
BZ2_bzCompressEnd
should then be
called.
Just to make sure the calling program does not cheat, the
library makes a note of avail_in
at the time of the first call to
BZ2_bzCompress
which has
BZ_FINISH
as an action (ie, at
the time the program has announced its intention to not supply
any more input). By comparing this value with that of
avail_in
over subsequent calls
to BZ2_bzCompress
, the library
can detect any attempts to slip in more data to compress. Any
calls for which this is detected will return
BZ_SEQUENCE_ERROR
. This
indicates a programming mistake which should be corrected.
Instead of asking to finish, the calling program may ask
BZ2_bzCompress
to take all the
remaining input, compress it and terminate the current
(Burrows-Wheeler) compression block. This could be useful for
error control purposes. The mechanism is analogous to that for
finishing: call BZ2_bzCompress
with an action of BZ_FLUSH
,
remove output data, and persist with the
BZ_FLUSH
action until the value
BZ_RUN
is returned. As with
finishing, BZ2_bzCompress
detects any attempt to provide more input data once the flush has
begun.
Once the flush is complete, the stream returns to the normal RUNNING state.
This all sounds pretty complex, but isn't really. Here's a
table which shows which actions are allowable in each state, what
action will be taken, what the next state is, and what the
non-error return values are. Note that you can't explicitly ask
what state the stream is in, but nor do you need to -- it can be
inferred from the values returned by
BZ2_bzCompress
.
IDLE/any Illegal. IDLE state only exists after BZ2_bzCompressEnd or before BZ2_bzCompressInit. Return value = BZ_SEQUENCE_ERROR RUNNING/BZ_RUN Compress from next_in to next_out as much as possible. Next state = RUNNING Return value = BZ_RUN_OK RUNNING/BZ_FLUSH Remember current value of next_in. Compress from next_in to next_out as much as possible, but do not accept any more input. Next state = FLUSHING Return value = BZ_FLUSH_OK RUNNING/BZ_FINISH Remember current value of next_in. Compress from next_in to next_out as much as possible, but do not accept any more input. Next state = FINISHING Return value = BZ_FINISH_OK FLUSHING/BZ_FLUSH Compress from next_in to next_out as much as possible, but do not accept any more input. If all the existing input has been used up and all compressed output has been removed Next state = RUNNING; Return value = BZ_RUN_OK else Next state = FLUSHING; Return value = BZ_FLUSH_OK FLUSHING/other Illegal. Return value = BZ_SEQUENCE_ERROR FINISHING/BZ_FINISH Compress from next_in to next_out as much as possible, but to not accept any more input. If all the existing input has been used up and all compressed output has been removed Next state = IDLE; Return value = BZ_STREAM_END else Next state = FINISHING; Return value = BZ_FINISH_OK FINISHING/other Illegal. Return value = BZ_SEQUENCE_ERROR
That still looks complicated? Well, fair enough. The usual sequence of calls for compressing a load of data is:
Get started with
BZ2_bzCompressInit
.
Shovel data in and shlurp out its compressed form
using zero or more calls of
BZ2_bzCompress
with action =
BZ_RUN
.
Finish up. Repeatedly call
BZ2_bzCompress
with action =
BZ_FINISH
, copying out the
compressed output, until
BZ_STREAM_END
is
returned.
Close up and go home. Call
BZ2_bzCompressEnd
.
If the data you want to compress fits into your input
buffer all at once, you can skip the calls of
BZ2_bzCompress ( ..., BZ_RUN )
and just do the BZ2_bzCompress ( ..., BZ_FINISH
)
calls.
All required memory is allocated by
BZ2_bzCompressInit
. The
compression library can accept any data at all (obviously). So
you shouldn't get any error return values from the
BZ2_bzCompress
calls. If you
do, they will be
BZ_SEQUENCE_ERROR
, and indicate
a bug in your programming.
Trivial other possible return values:
BZ_PARAM_ERROR if strm is NULL, or strm->s is NULL
int BZ2_bzCompressEnd ( bz_stream *strm );
Releases all memory associated with a compression stream.
Possible return values:
BZ_PARAM_ERROR if strm is NULL or strm->s is NULL BZ_OK otherwise
int BZ2_bzDecompressInit ( bz_stream *strm, int verbosity, int small );
Prepares for decompression. As with
BZ2_bzCompressInit
, a
bz_stream
record should be
allocated and initialised before the call. Fields
bzalloc
,
bzfree
and
opaque
should be set if a custom
memory allocator is required, or made
NULL
for the normal
malloc
/
free
routines. Upon return, the
internal state will have been initialised, and
total_in
and
total_out
will be zero.
For the meaning of parameter
verbosity
, see
BZ2_bzCompressInit
.
If small
is nonzero, the
library will use an alternative decompression algorithm which
uses less memory but at the cost of decompressing more slowly
(roughly speaking, half the speed, but the maximum memory
requirement drops to around 2300k). See How to use bzip2
for more information on memory management.
Note that the amount of memory needed to decompress a
stream cannot be determined until the stream's header has been
read, so even if
BZ2_bzDecompressInit
succeeds, a
subsequent BZ2_bzDecompress
could fail with
BZ_MEM_ERROR
.
Possible return values:
BZ_CONFIG_ERROR if the library has been mis-compiled BZ_PARAM_ERROR if ( small != 0 && small != 1 ) or (verbosity <; 0 || verbosity > 4) BZ_MEM_ERROR if insufficient memory is available
Allowable next actions:
BZ2_bzDecompress if BZ_OK was returned no specific action required in case of error
int BZ2_bzDecompress ( bz_stream *strm );
Provides more input and/out output buffer space for the
library. The caller maintains input and output buffers, and uses
BZ2_bzDecompress
to transfer
data between them.
Before each call to
BZ2_bzDecompress
,
next_in
should point at the
compressed data, and avail_in
should indicate how many bytes the library may read.
BZ2_bzDecompress
updates
next_in
,
avail_in
and
total_in
to reflect the number
of bytes it has read.
Similarly, next_out
should
point to a buffer in which the uncompressed output is to be
placed, with avail_out
indicating how much output space is available.
BZ2_bzCompress
updates
next_out
,
avail_out
and
total_out
to reflect the number
of bytes output.
You may provide and remove as little or as much data as you
like on each call of
BZ2_bzDecompress
. In the limit,
it is acceptable to supply and remove data one byte at a time,
although this would be terribly inefficient. You should always
ensure that at least one byte of output space is available at
each call.
Use of BZ2_bzDecompress
is
simpler than
BZ2_bzCompress
.
You should provide input and remove output as described
above, and repeatedly call
BZ2_bzDecompress
until
BZ_STREAM_END
is returned.
Appearance of BZ_STREAM_END
denotes that BZ2_bzDecompress
has detected the logical end of the compressed stream.
BZ2_bzDecompress
will not
produce BZ_STREAM_END
until all
output data has been placed into the output buffer, so once
BZ_STREAM_END
appears, you are
guaranteed to have available all the decompressed output, and
BZ2_bzDecompressEnd
can safely
be called.
If case of an error return value, you should call
BZ2_bzDecompressEnd
to clean up
and release memory.
Possible return values:
BZ_PARAM_ERROR if strm is NULL or strm->s is NULL or strm->avail_out < 1 BZ_DATA_ERROR if a data integrity error is detected in the compressed stream BZ_DATA_ERROR_MAGIC if the compressed stream doesn't begin with the right magic bytes BZ_MEM_ERROR if there wasn't enough memory available BZ_STREAM_END if the logical end of the data stream was detected and all output in has been consumed, eg s-->avail_out > 0 BZ_OK otherwise
Allowable next actions:
BZ2_bzDecompress if BZ_OK was returned BZ2_bzDecompressEnd otherwise
This interface provides functions for reading and writing
bzip2
format files. First, some
general points.
All of the functions take an
int*
first argument,
bzerror
. After each call,
bzerror
should be consulted
first to determine the outcome of the call. If
bzerror
is
BZ_OK
, the call completed
successfully, and only then should the return value of the
function (if any) be consulted. If
bzerror
is
BZ_IO_ERROR
, there was an
error reading/writing the underlying compressed file, and you
should then consult errno
/
perror
to determine the cause
of the difficulty. bzerror
may also be set to various other values; precise details are
given on a per-function basis below.
If bzerror
indicates
an error (ie, anything except
BZ_OK
and
BZ_STREAM_END
), you should
immediately call
BZ2_bzReadClose
(or
BZ2_bzWriteClose
, depending on
whether you are attempting to read or to write) to free up all
resources associated with the stream. Once an error has been
indicated, behaviour of all calls except
BZ2_bzReadClose
(BZ2_bzWriteClose
) is
undefined. The implication is that (1)
bzerror
should be checked
after each call, and (2) if
bzerror
indicates an error,
BZ2_bzReadClose
(BZ2_bzWriteClose
) should then
be called to clean up.
The FILE*
arguments
passed to BZ2_bzReadOpen
/
BZ2_bzWriteOpen
should be set
to binary mode. Most Unix systems will do this by default, but
other platforms, including Windows and Mac, will not. If you
omit this, you may encounter problems when moving code to new
platforms.
Memory allocation requests are handled by
malloc
/
free
. At present there is no
facility for user-defined memory allocators in the file I/O
functions (could easily be added, though).
typedef void BZFILE; BZFILE *BZ2_bzReadOpen( int *bzerror, FILE *f, int verbosity, int small, void *unused, int nUnused );
Prepare to read compressed data from file handle
f
.
f
should refer to a file which
has been opened for reading, and for which the error indicator
(ferror(f)
)is not set. If
small
is 1, the library will try
to decompress using less memory, at the expense of speed.
For reasons explained below,
BZ2_bzRead
will decompress the
nUnused
bytes starting at
unused
, before starting to read
from the file f
. At most
BZ_MAX_UNUSED
bytes may be
supplied like this. If this facility is not required, you should
pass NULL
and
0
for
unused
and
nUnused
respectively.
For the meaning of parameters
small
and
verbosity
, see
BZ2_bzDecompressInit
.
The amount of memory needed to decompress a file cannot be
determined until the file's header has been read. So it is
possible that BZ2_bzReadOpen
returns BZ_OK
but a subsequent
call of BZ2_bzRead
will return
BZ_MEM_ERROR
.
Possible assignments to
bzerror
:
BZ_CONFIG_ERROR if the library has been mis-compiled BZ_PARAM_ERROR if f is NULL or small is neither 0 nor 1 or ( unused == NULL && nUnused != 0 ) or ( unused != NULL && !(0 <= nUnused <= BZ_MAX_UNUSED) ) BZ_IO_ERROR if ferror(f) is nonzero BZ_MEM_ERROR if insufficient memory is available BZ_OK otherwise.
Possible return values:
Pointer to an abstract BZFILE if bzerror is BZ_OK NULL otherwise
Allowable next actions:
BZ2_bzRead if bzerror is BZ_OK BZ2_bzClose otherwise
int BZ2_bzRead ( int *bzerror, BZFILE *b, void *buf, int len );
Reads up to len
(uncompressed) bytes from the compressed file
b
into the buffer
buf
. If the read was
successful, bzerror
is set to
BZ_OK
and the number of bytes
read is returned. If the logical end-of-stream was detected,
bzerror
will be set to
BZ_STREAM_END
, and the number of
bytes read is returned. All other
bzerror
values denote an
error.
BZ2_bzRead
will supply
len
bytes, unless the logical
stream end is detected or an error occurs. Because of this, it
is possible to detect the stream end by observing when the number
of bytes returned is less than the number requested.
Nevertheless, this is regarded as inadvisable; you should instead
check bzerror
after every call
and watch out for
BZ_STREAM_END
.
Internally, BZ2_bzRead
copies data from the compressed file in chunks of size
BZ_MAX_UNUSED
bytes before
decompressing it. If the file contains more bytes than strictly
needed to reach the logical end-of-stream,
BZ2_bzRead
will almost certainly
read some of the trailing data before signalling
BZ_SEQUENCE_END
. To collect the
read but unused data once
BZ_SEQUENCE_END
has appeared,
call BZ2_bzReadGetUnused
immediately before
BZ2_bzReadClose
.
Possible assignments to
bzerror
:
BZ_PARAM_ERROR if b is NULL or buf is NULL or len < 0 BZ_SEQUENCE_ERROR if b was opened with BZ2_bzWriteOpen BZ_IO_ERROR if there is an error reading from the compressed file BZ_UNEXPECTED_EOF if the compressed file ended before the logical end-of-stream was detected BZ_DATA_ERROR if a data integrity error was detected in the compressed stream BZ_DATA_ERROR_MAGIC if the stream does not begin with the requisite header bytes (ie, is not a bzip2 data file). This is really a special case of BZ_DATA_ERROR. BZ_MEM_ERROR if insufficient memory was available BZ_STREAM_END if the logical end of stream was detected. BZ_OK otherwise.
Possible return values:
number of bytes read if bzerror is BZ_OK or BZ_STREAM_END undefined otherwise
Allowable next actions:
collect data from buf, then BZ2_bzRead or BZ2_bzReadClose if bzerror is BZ_OK collect data from buf, then BZ2_bzReadClose or BZ2_bzReadGetUnused if bzerror is BZ_SEQUENCE_END BZ2_bzReadClose otherwise
void BZ2_bzReadGetUnused( int* bzerror, BZFILE *b, void** unused, int* nUnused );
Returns data which was read from the compressed file but
was not needed to get to the logical end-of-stream.
*unused
is set to the address of
the data, and *nUnused
to the
number of bytes. *nUnused
will
be set to a value between 0
and
BZ_MAX_UNUSED
inclusive.
This function may only be called once
BZ2_bzRead
has signalled
BZ_STREAM_END
but before
BZ2_bzReadClose
.
Possible assignments to
bzerror
:
BZ_PARAM_ERROR if b is NULL or unused is NULL or nUnused is NULL BZ_SEQUENCE_ERROR if BZ_STREAM_END has not been signalled or if b was opened with BZ2_bzWriteOpen BZ_OK otherwise
Allowable next actions:
BZ2_bzReadClose
void BZ2_bzReadClose ( int *bzerror, BZFILE *b );
Releases all memory pertaining to the compressed file
b
.
BZ2_bzReadClose
does not call
fclose
on the underlying file
handle, so you should do that yourself if appropriate.
BZ2_bzReadClose
should be called
to clean up after all error situations.
Possible assignments to
bzerror
:
BZ_SEQUENCE_ERROR if b was opened with BZ2_bzOpenWrite BZ_OK otherwise
Allowable next actions:
none
BZFILE *BZ2_bzWriteOpen( int *bzerror, FILE *f, int blockSize100k, int verbosity, int workFactor );
Prepare to write compressed data to file handle
f
.
f
should refer to a file which
has been opened for writing, and for which the error indicator
(ferror(f)
)is not set.
For the meaning of parameters
blockSize100k
,
verbosity
and
workFactor
, see
BZ2_bzCompressInit
.
All required memory is allocated at this stage, so if the
call completes successfully,
BZ_MEM_ERROR
cannot be signalled
by a subsequent call to
BZ2_bzWrite
.
Possible assignments to
bzerror
:
BZ_CONFIG_ERROR if the library has been mis-compiled BZ_PARAM_ERROR if f is NULL or blockSize100k < 1 or blockSize100k > 9 BZ_IO_ERROR if ferror(f) is nonzero BZ_MEM_ERROR if insufficient memory is available BZ_OK otherwise
Possible return values:
Pointer to an abstract BZFILE if bzerror is BZ_OK NULL otherwise
Allowable next actions:
BZ2_bzWrite if bzerror is BZ_OK (you could go directly to BZ2_bzWriteClose, but this would be pretty pointless) BZ2_bzWriteClose otherwise
void BZ2_bzWrite ( int *bzerror, BZFILE *b, void *buf, int len );
Absorbs len
bytes from the
buffer buf
, eventually to be
compressed and written to the file.
Possible assignments to
bzerror
:
BZ_PARAM_ERROR if b is NULL or buf is NULL or len < 0 BZ_SEQUENCE_ERROR if b was opened with BZ2_bzReadOpen BZ_IO_ERROR if there is an error writing the compressed file. BZ_OK otherwise
void BZ2_bzWriteClose( int *bzerror, BZFILE* f, int abandon, unsigned int* nbytes_in, unsigned int* nbytes_out ); void BZ2_bzWriteClose64( int *bzerror, BZFILE* f, int abandon, unsigned int* nbytes_in_lo32, unsigned int* nbytes_in_hi32, unsigned int* nbytes_out_lo32, unsigned int* nbytes_out_hi32 );
Compresses and flushes to the compressed file all data so
far supplied by BZ2_bzWrite
.
The logical end-of-stream markers are also written, so subsequent
calls to BZ2_bzWrite
are
illegal. All memory associated with the compressed file
b
is released.
fflush
is called on the
compressed file, but it is not
fclose
'd.
If BZ2_bzWriteClose
is
called to clean up after an error, the only action is to release
the memory. The library records the error codes issued by
previous calls, so this situation will be detected automatically.
There is no attempt to complete the compression operation, nor to
fflush
the compressed file. You
can force this behaviour to happen even in the case of no error,
by passing a nonzero value to
abandon
.
If nbytes_in
is non-null,
*nbytes_in
will be set to be the
total volume of uncompressed data handled. Similarly,
nbytes_out
will be set to the
total volume of compressed data written. For compatibility with
older versions of the library,
BZ2_bzWriteClose
only yields the
lower 32 bits of these counts. Use
BZ2_bzWriteClose64
if you want
the full 64 bit counts. These two functions are otherwise
absolutely identical.
Possible assignments to
bzerror
:
BZ_SEQUENCE_ERROR if b was opened with BZ2_bzReadOpen BZ_IO_ERROR if there is an error writing the compressed file BZ_OK otherwise
The high-level library facilitates use of
bzip2
data streams which form
some part of a surrounding, larger data stream.
For writing, the library takes an open file handle,
writes compressed data to it,
fflush
es it but does not
fclose
it. The calling
application can write its own data before and after the
compressed data stream, using that same file handle.
Reading is more complex, and the facilities are not as
general as they could be since generality is hard to reconcile
with efficiency. BZ2_bzRead
reads from the compressed file in blocks of size
BZ_MAX_UNUSED
bytes, and in
doing so probably will overshoot the logical end of compressed
stream. To recover this data once decompression has ended,
call BZ2_bzReadGetUnused
after
the last call of BZ2_bzRead
(the one returning
BZ_STREAM_END
) but before
calling
BZ2_bzReadClose
.
This mechanism makes it easy to decompress multiple
bzip2
streams placed end-to-end.
As the end of one stream, when
BZ2_bzRead
returns
BZ_STREAM_END
, call
BZ2_bzReadGetUnused
to collect
the unused data (copy it into your own buffer somewhere). That
data forms the start of the next compressed stream. To start
uncompressing that next stream, call
BZ2_bzReadOpen
again, feeding in
the unused data via the unused
/
nUnused
parameters. Keep doing
this until BZ_STREAM_END
return
coincides with the physical end of file
(feof(f)
). In this situation
BZ2_bzReadGetUnused
will of
course return no data.
This should give some feel for how the high-level interface can be used. If you require extra flexibility, you'll have to bite the bullet and get to grips with the low-level interface.
Here's how you'd write data to a compressed file:
FILE* f; BZFILE* b; int nBuf; char buf[ /* whatever size you like */ ]; int bzerror; int nWritten; f = fopen ( "myfile.bz2", "w" ); if ( !f ) { /* handle error */ } b = BZ2_bzWriteOpen( &bzerror, f, 9 ); if (bzerror != BZ_OK) { BZ2_bzWriteClose ( b ); /* handle error */ } while ( /* condition */ ) { /* get data to write into buf, and set nBuf appropriately */ nWritten = BZ2_bzWrite ( &bzerror, b, buf, nBuf ); if (bzerror == BZ_IO_ERROR) { BZ2_bzWriteClose ( &bzerror, b ); /* handle error */ } } BZ2_bzWriteClose( &bzerror, b ); if (bzerror == BZ_IO_ERROR) { /* handle error */ }
And to read from a compressed file:
FILE* f; BZFILE* b; int nBuf; char buf[ /* whatever size you like */ ]; int bzerror; int nWritten; f = fopen ( "myfile.bz2", "r" ); if ( !f ) { /* handle error */ } b = BZ2_bzReadOpen ( &bzerror, f, 0, NULL, 0 ); if ( bzerror != BZ_OK ) { BZ2_bzReadClose ( &bzerror, b ); /* handle error */ } bzerror = BZ_OK; while ( bzerror == BZ_OK && /* arbitrary other conditions */) { nBuf = BZ2_bzRead ( &bzerror, b, buf, /* size of buf */ ); if ( bzerror == BZ_OK ) { /* do something with buf[0 .. nBuf-1] */ } } if ( bzerror != BZ_STREAM_END ) { BZ2_bzReadClose ( &bzerror, b ); /* handle error */ } else { BZ2_bzReadClose ( &bzerror, b ); }
int BZ2_bzBuffToBuffCompress( char* dest, unsigned int* destLen, char* source, unsigned int sourceLen, int blockSize100k, int verbosity, int workFactor );
Attempts to compress the data in source[0
.. sourceLen-1]
into the destination buffer,
dest[0 .. *destLen-1]
. If the
destination buffer is big enough,
*destLen
is set to the size of
the compressed data, and BZ_OK
is returned. If the compressed data won't fit,
*destLen
is unchanged, and
BZ_OUTBUFF_FULL
is
returned.
Compression in this manner is a one-shot event, done with a
single call to this function. The resulting compressed data is a
complete bzip2
format data
stream. There is no mechanism for making additional calls to
provide extra input data. If you want that kind of mechanism,
use the low-level interface.
For the meaning of parameters
blockSize100k
,
verbosity
and
workFactor
, see
BZ2_bzCompressInit
.
To guarantee that the compressed data will fit in its buffer, allocate an output buffer of size 1% larger than the uncompressed data, plus six hundred extra bytes.
BZ2_bzBuffToBuffDecompress
will not write data at or beyond
dest[*destLen]
, even in case of
buffer overflow.
Possible return values:
BZ_CONFIG_ERROR if the library has been mis-compiled BZ_PARAM_ERROR if dest is NULL or destLen is NULL or blockSize100k < 1 or blockSize100k > 9 or verbosity < 0 or verbosity > 4 or workFactor < 0 or workFactor > 250 BZ_MEM_ERROR if insufficient memory is available BZ_OUTBUFF_FULL if the size of the compressed data exceeds *destLen BZ_OK otherwise
int BZ2_bzBuffToBuffDecompress( char* dest, unsigned int* destLen, char* source, unsigned int sourceLen, int small, int verbosity );
Attempts to decompress the data in source[0
.. sourceLen-1]
into the destination buffer,
dest[0 .. *destLen-1]
. If the
destination buffer is big enough,
*destLen
is set to the size of
the uncompressed data, and BZ_OK
is returned. If the compressed data won't fit,
*destLen
is unchanged, and
BZ_OUTBUFF_FULL
is
returned.
source
is assumed to hold
a complete bzip2
format data
stream.
BZ2_bzBuffToBuffDecompress
tries
to decompress the entirety of the stream into the output
buffer.
For the meaning of parameters
small
and
verbosity
, see
BZ2_bzDecompressInit
.
Because the compression ratio of the compressed data cannot be known in advance, there is no easy way to guarantee that the output buffer will be big enough. You may of course make arrangements in your code to record the size of the uncompressed data, but such a mechanism is beyond the scope of this library.
BZ2_bzBuffToBuffDecompress
will not write data at or beyond
dest[*destLen]
, even in case of
buffer overflow.
Possible return values:
BZ_CONFIG_ERROR if the library has been mis-compiled BZ_PARAM_ERROR if dest is NULL or destLen is NULL or small != 0 && small != 1 or verbosity < 0 or verbosity > 4 BZ_MEM_ERROR if insufficient memory is available BZ_OUTBUFF_FULL if the size of the compressed data exceeds *destLen BZ_DATA_ERROR if a data integrity error was detected in the compressed data BZ_DATA_ERROR_MAGIC if the compressed data doesn't begin with the right magic bytes BZ_UNEXPECTED_EOF if the compressed data ends unexpectedly BZ_OK otherwise
Yoshioka Tsuneo has contributed some functions to give
better zlib
compatibility.
These functions are BZ2_bzopen
,
BZ2_bzread
,
BZ2_bzwrite
,
BZ2_bzflush
,
BZ2_bzclose
,
BZ2_bzerror
and
BZ2_bzlibVersion
. These
functions are not (yet) officially part of the library. If they
break, you get to keep all the pieces. Nevertheless, I think
they work ok.
typedef void BZFILE; const char * BZ2_bzlibVersion ( void );
Returns a string indicating the library version.
BZFILE * BZ2_bzopen ( const char *path, const char *mode ); BZFILE * BZ2_bzdopen ( int fd, const char *mode );
Opens a .bz2
file for
reading or writing, using either its name or a pre-existing file
descriptor. Analogous to fopen
and fdopen
.
int BZ2_bzread ( BZFILE* b, void* buf, int len ); int BZ2_bzwrite ( BZFILE* b, void* buf, int len );
Reads/writes data from/to a previously opened
BZFILE
. Analogous to
fread
and
fwrite
.
int BZ2_bzflush ( BZFILE* b ); void BZ2_bzclose ( BZFILE* b );
Flushes/closes a BZFILE
.
BZ2_bzflush
doesn't actually do
anything. Analogous to fflush
and fclose
.
const char * BZ2_bzerror ( BZFILE *b, int *errnum )
Returns a string describing the more recent error status of
b
, and also sets
*errnum
to its numerical
value.
In a deeply embedded application, you might want to use
just the memory-to-memory functions. You can do this
conveniently by compiling the library with preprocessor symbol
BZ_NO_STDIO
defined. Doing this
gives you a library containing only the following eight
functions:
BZ2_bzCompressInit
,
BZ2_bzCompress
,
BZ2_bzCompressEnd
BZ2_bzDecompressInit
,
BZ2_bzDecompress
,
BZ2_bzDecompressEnd
BZ2_bzBuffToBuffCompress
,
BZ2_bzBuffToBuffDecompress
When compiled like this, all functions will ignore
verbosity
settings.
libbzip2
contains a number
of internal assertion checks which should, needless to say, never
be activated. Nevertheless, if an assertion should fail,
behaviour depends on whether or not the library was compiled with
BZ_NO_STDIO
set.
For a normal compile, an assertion failure yields the message:
bzip2/libbzip2: internal error number N.
This is a bug in bzip2/libbzip2, 1.0.5 of 10 December 2007. Please report it to me at<EMAIL_ADDRESS>If this happened when you were using some program which uses libbzip2 as a component, you should also report this bug to the author(s) of that program. Please make an effort to report this bug; timely and accurate bug reports eventually lead to higher quality software. Thanks. Julian Seward, 10 December 2007.
where N
is some error code
number. If N == 1007
, it also
prints some extra text advising the reader that unreliable memory
is often associated with internal error 1007. (This is a
frequently-observed-phenomenon with versions 1.0.0/1.0.1).
exit(3)
is then
called.
For a stdio
-free library,
assertion failures result in a call to a function declared
as:
extern void bz_internal_error ( int errcode );
The relevant code is passed as a parameter. You should supply such a function.
In either case, once an assertion failure has occurred, any
bz_stream
records involved can
be regarded as invalid. You should not attempt to resume normal
operation with them.
You may, of course, change critical error handling to suit your needs. As I said above, critical errors indicate bugs in the library and should not occur. All "normal" error situations are indicated via error return codes from functions, and can be recovered from.
Everything related to Windows has been contributed by
Yoshioka Tsuneo
<EMAIL_ADDRESS>), so
you should send your queries to him (but perhaps Cc: me,
<EMAIL_ADDRESS>).
My vague understanding of what to do is: using Visual C++
5.0, open the project file
libbz2.dsp
, and build. That's
all.
If you can't open the project file for some reason, make a
new one, naming these files:
blocksort.c
,
bzlib.c
,
compress.c
,
crctable.c
,
decompress.c
,
huffman.c
,
randtable.c
and
libbz2.def
. You will also need
to name the header files bzlib.h
and bzlib_private.h
.
If you don't use VC++, you may need to define the
proprocessor symbol
_WIN32
.
Finally, dlltest.c
is a
sample program using the DLL. It has a project file,
dlltest.dsp
.
If you just want a makefile for Visual C, have a look at
makefile.msc
.
Be aware that if you compile
bzip2
itself on Win32, you must
set BZ_UNIX
to 0 and
BZ_LCCWIN32
to 1, in the file
bzip2.c
, before compiling.
Otherwise the resulting binary won't work correctly.
I haven't tried any of this stuff myself, but it all looks plausible.
Table of Contents
These are just some random thoughts of mine. Your mileage may vary.
bzip2-1.0.X
,
0.9.5
and
0.9.0
use exactly the same file
format as the original version,
bzip2-0.1
. This decision was
made in the interests of stability. Creating yet another
incompatible compressed file format would create further
confusion and disruption for users.
Nevertheless, this is not a painless decision. Development
work since the release of
bzip2-0.1
in August 1997 has
shown complexities in the file format which slow down
decompression and, in retrospect, are unnecessary. These
are:
The run-length encoder, which is the first of the compression transformations, is entirely irrelevant. The original purpose was to protect the sorting algorithm from the very worst case input: a string of repeated symbols. But algorithm steps Q6a and Q6b in the original Burrows-Wheeler technical report (SRC-124) show how repeats can be handled without difficulty in block sorting.
The randomisation mechanism doesn't really need to be there. Udi Manber and Gene Myers published a suffix array construction algorithm a few years back, which can be employed to sort any block, no matter how repetitive, in O(N log N) time. Subsequent work by Kunihiko Sadakane has produced a derivative O(N (log N)^2) algorithm which usually outperforms the Manber-Myers algorithm.
I could have changed to Sadakane's algorithm, but I find
it to be slower than bzip2
's
existing algorithm for most inputs, and the randomisation
mechanism protects adequately against bad cases. I didn't
think it was a good tradeoff to make. Partly this is due to
the fact that I was not flooded with email complaints about
bzip2-0.1
's performance on
repetitive data, so perhaps it isn't a problem for real
inputs.
Probably the best long-term solution, and the one I have incorporated into 0.9.5 and above, is to use the existing sorting algorithm initially, and fall back to a O(N (log N)^2) algorithm if the standard algorithm gets into difficulties.
The compressed file format was never designed to be
handled by a library, and I have had to jump though some hoops
to produce an efficient implementation of decompression. It's
a bit hairy. Try passing
decompress.c
through the C
preprocessor and you'll see what I mean. Much of this
complexity could have been avoided if the compressed size of
each block of data was recorded in the data stream.
An Adler-32 checksum, rather than a CRC32 checksum, would be faster to compute.
It would be fair to say that the
bzip2
format was frozen before I
properly and fully understood the performance consequences of
doing so.
Improvements which I was able to incorporate into 0.9.0, despite using the same file format, are:
Single array implementation of the inverse BWT. This significantly speeds up decompression, presumably because it reduces the number of cache misses.
Faster inverse MTF transform for large MTF values. The new implementation is based on the notion of sliding blocks of values.
bzip2-0.9.0
now reads
and writes files with fread
and fwrite
; version 0.1 used
putc
and
getc
. Duh! Well, you live
and learn.
Further ahead, it would be nice to be able to do random access into files. This will require some careful design of compressed file formats.
After some consideration, I have decided not to use GNU
autoconf
to configure 0.9.5 or
1.0.
autoconf
, admirable and
wonderful though it is, mainly assists with portability problems
between Unix-like platforms. But
bzip2
doesn't have much in the
way of portability problems on Unix; most of the difficulties
appear when porting to the Mac, or to Microsoft's operating
systems. autoconf
doesn't help
in those cases, and brings in a whole load of new
complexity.
Most people should be able to compile the library and program under Unix straight out-of-the-box, so to speak, especially if you have a version of GNU C available.
There are a couple of
__inline__
directives in the
code. GNU C (gcc
) should be
able to handle them. If you're not using GNU C, your C compiler
shouldn't see them at all. If your compiler does, for some
reason, see them and doesn't like them, just
#define
__inline__
to be
/* */
. One easy way to do this
is to compile with the flag
-D__inline__=
, which should be
understood by most Unix compilers.
If you still have difficulties, try compiling with the
macro BZ_STRICT_ANSI
defined.
This should enable you to build the library in a strictly ANSI
compliant environment. Building the program itself like this is
dangerous and not supported, since you remove
bzip2
's checks against
compressing directories, symbolic links, devices, and other
not-really-a-file entities. This could cause filesystem
corruption!
One other thing: if you create a
bzip2
binary for public distribution,
please consider linking it statically (gcc
-static
). This avoids all sorts of library-version
issues that others may encounter later on.
If you build bzip2
on
Win32, you must set BZ_UNIX
to 0
and BZ_LCCWIN32
to 1, in the
file bzip2.c
, before compiling.
Otherwise the resulting binary won't work correctly.
I tried pretty hard to make sure
bzip2
is bug free, both by
design and by testing. Hopefully you'll never need to read this
section for real.
Nevertheless, if bzip2
dies
with a segmentation fault, a bus error or an internal assertion
failure, it will ask you to email me a bug report. Experience from
years of feedback of bzip2 users indicates that almost all these
problems can be traced to either compiler bugs or hardware
problems.
Recompile the program with no optimisation, and
see if it works. And/or try a different compiler. I heard all
sorts of stories about various flavours of GNU C (and other
compilers) generating bad code for
bzip2
, and I've run across two
such examples myself.
2.7.X versions of GNU C are known to generate bad code
from time to time, at high optimisation levels. If you get
problems, try using the flags
-O2
-fomit-frame-pointer
-fno-strength-reduce
. You
should specifically not use
-funroll-loops
.
You may notice that the Makefile runs six tests as part of the build process. If the program passes all of these, it's a pretty good (but not 100%) indication that the compiler has done its job correctly.
If bzip2
crashes randomly, and the crashes are not repeatable, you may
have a flaky memory subsystem.
bzip2
really hammers your
memory hierarchy, and if it's a bit marginal, you may get these
problems. Ditto if your disk or I/O subsystem is slowly
failing. Yup, this really does happen.
Try using a different machine of the same type, and see if you can repeat the problem.
This isn't really a bug, but ... If
bzip2
tells you your file is
corrupted on decompression, and you obtained the file via FTP,
there is a possibility that you forgot to tell FTP to do a
binary mode transfer. That absolutely will cause the file to
be non-decompressible. You'll have to transfer it
again.
If you've incorporated
libbzip2
into your own program
and are getting problems, please, please, please, check that the
parameters you are passing in calls to the library, are correct,
and in accordance with what the documentation says is allowable.
I have tried to make the library robust against such problems,
but I'm sure I haven't succeeded.
Finally, if the above comments don't help, you'll have to send me a bug report. Now, it's just amazing how many people will send me a bug report saying something like:
bzip2 crashed with segmentation fault on my machine
and absolutely nothing else. Needless to say, a such a report is totally, utterly, completely and comprehensively 100% useless; a waste of your time, my time, and net bandwidth. With no details at all, there's no way I can possibly begin to figure out what the problem is.
The rules of the game are: facts, facts, facts. Don't omit them because "oh, they won't be relevant". At the bare minimum:
Machine type. Operating system version. Exact version of bzip2 (do bzip2 -V). Exact version of the compiler used. Flags passed to the compiler.
However, the most important single thing that will help me is the file that you were trying to compress or decompress at the time the problem happened. Without that, my ability to do anything more than speculate about the cause, is limited.
bzip2
is a resource hog.
It soaks up large amounts of CPU cycles and memory. Also, it
gives very large latencies. In the worst case, you can feed many
megabytes of uncompressed data into the library before getting
any compressed output, so this probably rules out applications
requiring interactive behaviour.
These aren't faults of my implementation, I hope, but more an intrinsic property of the Burrows-Wheeler transform (unfortunately). Maybe this isn't what you want.
If you want a compressor and/or library which is faster,
uses less memory but gets pretty good compression, and has
minimal latency, consider Jean-loup Gailly's and Mark Adler's
work, zlib-1.2.1
and
gzip-1.2.4
. Look for them at
http://www.zlib.org and
http://www.gzip.org
respectively.
For something faster and lighter still, you might try Markus F
X J Oberhumer's LZO
real-time
compression/decompression library, at
http://www.oberhumer.com/opensource.
bzip2
is not research
work, in the sense that it doesn't present any new ideas.
Rather, it's an engineering exercise based on existing
ideas.
Four documents describe essentially all the ideas behind
bzip2
:
Michael Burrows and D. J. Wheeler:
"A block-sorting lossless data compression algorithm"
10th May 1994.
Digital SRC Research Report 124.
ftp://ftp.digital.com/pub/DEC/SRC/research-reports/SRC-124.ps.gz
If you have trouble finding it, try searching at the
New Zealand Digital Library, http://www.nzdl.org.
Daniel S. Hirschberg and Debra A. LeLewer
"Efficient Decoding of Prefix Codes"
Communications of the ACM, April 1990, Vol 33, Number 4.
You might be able to get an electronic copy of this
from the ACM Digital Library.
David J. Wheeler
Program bred3.c and accompanying document bred3.ps.
This contains the idea behind the multi-table Huffman coding scheme.
ftp://ftp.cl.cam.ac.uk/users/djw3/
Jon L. Bentley and Robert Sedgewick
"Fast Algorithms for Sorting and Searching Strings"
Available from Sedgewick's web page,
www.cs.princeton.edu/~rs
The following paper gives valuable additional insights into the algorithm, but is not immediately the basis of any code used in bzip2.
Peter Fenwick:
Block Sorting Text Compression
Proceedings of the 19th Australasian Computer Science Conference,
Melbourne, Australia. Jan 31 - Feb 2, 1996.
ftp://ftp.cs.auckland.ac.nz/pub/peter-f/ACSC96paper.ps
Kunihiko Sadakane's sorting algorithm, mentioned above, is available from:
http://naomi.is.s.u-tokyo.ac.jp/~sada/papers/Sada98b.ps.gz
The Manber-Myers suffix array construction algorithm is described in a paper available from:
http://www.cs.arizona.edu/people/gene/PAPERS/suffix.ps
Finally, the following papers document some investigations I made into the performance of sorting and decompression algorithms:
Julian Seward
On the Performance of BWT Sorting Algorithms
Proceedings of the IEEE Data Compression Conference 2000
Snowbird, Utah. 28-30 March 2000.
Julian Seward
Space-time Tradeoffs in the Inverse B-W Transform
Proceedings of the IEEE Data Compression Conference 2001
Snowbird, Utah. 27-29 March 2001.


Data Operations
TOC
Logic Operations
- Example: 23, 11
Shift Operations
- Example: 15, -15
Questions
- Let X and Y be 39 and 127 respectively. Both are 8-bit unsigned numbers.
- Apply 1-bit logical/arithmetic left/right shift on X and Y. Show the process in binary format.
- Apply NOT, AND, NAND, OR, NOR, XOR, and XNOR on X and Y. Show the process in binary format.
- Show the results of problem 1. Conditions are the same, except that both numbers are now signed.
Arithmetic Operations
- Integer: 2’s complement
- Float: Sign and magnitude
- 1’s complement
Questions
- Show the following processes in binary format:
- 8-bit integer:
- 102 + 23
- -23 + 3
- -163 - 134
- IEEE 754:
- 34.75 + 23.125
- -34.75 + 23.125
- 34.75 - 23.125
- -34.75 - 23.125
- 8-bit integer:


Gets a read preference that forces reads to the primary or a secondary with one of the given sets of tags.
Gets a read preference that forces reads to the primary or a secondary with one of the given sets of tags. The driver will look for a secondary with each tag set in the given list, stopping after one is found, or the primary if none are available.
the list of tag sets to limit the list of secondaries to
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
ReadPreference which reads nearest node respective of tags.
1.2
Requires MongoDB 3.4 or greater
Gets a read preference that forces reads to the primary or a secondary with one of the given sets of tags.
Gets a read preference that forces reads to the primary or a secondary with one of the given sets of tags. The driver will look for a secondary with each tag set in the given list, stopping after one is found, or the primary if none are available.
the list of tag sets to limit the list of secondaries to
ReadPreference which reads nearest node respective of tags.
Gets a read preference that forces reads to the primary or a secondary with the given set of tags.
Gets a read preference that forces reads to the primary or a secondary with the given set of tags.
the set of tags to limit the list of secondaries to
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
ReadPreference which reads nearest node respective of tags.
1.2
Requires MongoDB 3.4 or greater
Gets a read preference that forces reads to the primary or a secondary with the given set of tags.
Gets a read preference that forces reads to the primary or a secondary with the given set of tags.
the set of tags to limit the list of secondaries to
ReadPreference which reads nearest node respective of tags.
Gets a read preference that forces reads to a primary or a secondary.
Gets a read preference that forces reads to a primary or a secondary.
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
ReadPreference which reads nearest
1.2
Requires MongoDB 3.4 or greater
Gets a read preference that forces reads to a primary or a secondary.
Gets a read preference that forces reads to a primary or a secondary.
ReadPreference which reads nearest
Gets a read preference that forces read to the primary.
Gets a read preference that forces read to the primary.
ReadPreference which reads from primary only
Gets a read preference that forces reads to the primary if available, otherwise to a secondary with one of the given sets of tags.
Gets a read preference that forces reads to the primary if available, otherwise to a secondary with one of the given sets of tags. The driver will look for a secondary with each tag set in the given list, stopping after one is found, or failing if no secondary can be found that matches any of the tag sets in the list.
the list of tag sets to limit the list of secondaries to
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
ReadPreference which reads primary if available, otherwise a secondary respective of tags.
1.2
Requires MongoDB 3.4 or greater
Gets a read preference that forces reads to the primary if available, otherwise to a secondary with one of the given sets of tags.
Gets a read preference that forces reads to the primary if available, otherwise to a secondary with one of the given sets of tags. The driver will look for a secondary with each tag set in the given list, stopping after one is found, or failing if no secondary can be found that matches any of the tag sets in the list.
the list of tag sets to limit the list of secondaries to
ReadPreference which reads primary if available, otherwise a secondary respective of tags.
Gets a read preference that forces reads to the primary if available, otherwise to a secondary with the given set of tags.
Gets a read preference that forces reads to the primary if available, otherwise to a secondary with the given set of tags.
the set of tags to limit the list of secondaries to.
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
ReadPreference which reads primary if available, otherwise a secondary respective of tags.
1.2
Requires MongoDB 3.4 or greater
Gets a read preference that forces reads to the primary if available, otherwise to a secondary with the given set of tags.
Gets a read preference that forces reads to the primary if available, otherwise to a secondary with the given set of tags.
the set of tags to limit the list of secondaries to.
ReadPreference which reads primary if available, otherwise a secondary respective of tags.
Gets a read preference that forces reads to the primary if available, otherwise to a secondary.
Gets a read preference that forces reads to the primary if available, otherwise to a secondary.
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
ReadPreference which reads primary if available.
1.2
Requires MongoDB 3.4 or greater
Gets a read preference that forces reads to the primary if available, otherwise to a secondary.
Gets a read preference that forces reads to the primary if available, otherwise to a secondary.
ReadPreference which reads primary if available.
Gets a read preference that forces reads to a secondary with one of the given sets of tags.
Gets a read preference that forces reads to a secondary with one of the given sets of tags. The driver will look for a secondary with each tag set in the given list, stopping after one is found, or failing if no secondary can be found that matches any of the tag sets in the list.
the list of tag sets to limit the list of secondaries to
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
ReadPreference which reads secondary respective of tags.
1.2
Requires MongoDB 3.4 or greater
Gets a read preference that forces reads to a secondary with one of the given sets of tags.
Gets a read preference that forces reads to a secondary with one of the given sets of tags. The driver will look for a secondary with each tag set in the given list, stopping after one is found, or failing if no secondary can be found that matches any of the tag sets in the list.
the list of tag sets to limit the list of secondaries to
ReadPreference which reads secondary respective of tags.
Gets a read preference that forces reads to a secondary with the given set of tags.
Gets a read preference that forces reads to a secondary with the given set of tags.
the set of tags to limit the list of secondaries to
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
ReadPreference which reads secondary respective of tags.
1.2
Requires MongoDB 3.4 or greater
Gets a read preference that forces reads to a secondary with the given set of tags.
Gets a read preference that forces reads to a secondary with the given set of tags.
the set of tags to limit the list of secondaries to
ReadPreference which reads secondary respective of tags.
Gets a read preference that forces reads to a secondary.
Gets a read preference that forces reads to a secondary.
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
ReadPreference which reads secondary.
1.2
Requires MongoDB 3.4 or greater
Gets a read preference that forces reads to a secondary.
Gets a read preference that forces reads to a secondary.
ReadPreference which reads secondary.
Gets a read preference that forces reads to a secondary with one of the given sets of tags.
Gets a read preference that forces reads to a secondary with one of the given sets of tags. The driver will look for a secondary with each tag set in the given list, stopping after one is found, or the primary if none are available.
the list of tag sets to limit the list of secondaries to
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
ReadPreference which reads secondary if available respective of tags, otherwise from primary irrespective of tags.
1.2
Requires MongoDB 3.4 or greater
Gets a read preference that forces reads to a secondary with one of the given sets of tags.
Gets a read preference that forces reads to a secondary with one of the given sets of tags. The driver will look for a secondary with each tag set in the given list, stopping after one is found, or the primary if none are available.
the list of tag sets to limit the list of secondaries to
ReadPreference which reads secondary if available respective of tags, otherwise from primary irrespective of tags.
Gets a read preference that forces reads to a secondary with the given set of tags, or the primary is none are available.
Gets a read preference that forces reads to a secondary with the given set of tags, or the primary is none are available.
the set of tags to limit the list of secondaries to
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
ReadPreference which reads secondary if available respective of tags, otherwise from primary irrespective of tags.
1.2
Requires MongoDB 3.4 or greater
Gets a read preference that forces reads to a secondary with the given set of tags, or the primary is none are available.
Gets a read preference that forces reads to a secondary with the given set of tags, or the primary is none are available.
the set of tags to limit the list of secondaries to
ReadPreference which reads secondary if available respective of tags, otherwise from primary irrespective of tags.
Gets a read preference that forces reads to a secondary if one is available, otherwise to the primary.
Gets a read preference that forces reads to a secondary if one is available, otherwise to the primary.
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
ReadPreference which reads secondary if available, otherwise from primary.
1.2
Requires MongoDB 3.4 or greater
Gets a read preference that forces reads to a secondary if one is available, otherwise to the primary.
Gets a read preference that forces reads to a secondary if one is available, otherwise to the primary.
ReadPreference which reads secondary if available, otherwise from primary.
Creates a taggable read preference from the given read preference name and list of tag sets.
Creates a taggable read preference from the given read preference name and list of tag sets.
the name of the read preference
the list of tag sets
the max allowable staleness of secondaries. A zero duration indicates the absence of a maximum.
the taggable read preference
1.2
Requires MongoDB 3.4 or greater
Creates a taggable read preference from the given read preference name and list of tag sets.
Creates a taggable read preference from the given read preference name and list of tag sets.
the name of the read preference
the list of tag sets
the taggable read preference
Creates a read preference from the given read preference name.
Creates a read preference from the given read preference name.
the name of the read preference
the read preference
The preferred replica set members to which a query or command can be sent.
1.0


If you wish to delete the Mongo database that stores all the entries you've created, press the button to the left. This will destroy the database and you will not have any entries, proceed with caution!
{% endblock %}
{% include "banners/footer.html" %}


How To Response JSON in PHP Using jQuery Ajax
Basic JSON Response
Add New Employee
Email
First Name
Last Name
Address
Submit
List of Employees
Edit Employee
×
Email
First Name
Last Name
Address
Update
Close


Shadertoy is a famous website hosting amazing shader experiments. People often ask how they can use those shaders with Three.js.
It's important to recognize it's called ShaderTOY for a reason. In general shadertoy shaders are not about best practices. Rather they are a fun challenge similar to say dwitter (write code in 140 characters) or js13kGames (make a game in 13k or less).
In the case of Shadertoy the puzzle is, write a function that for a given pixel location outputs a color that draws something interesting. It's a fun challenge and many of the result are amazing. But, it is not best practice.
Compare this amazing shadertoy shader that draws an entire city
Fullscreen on my GPU it runs at about 5 frames a second. Contrast that to a game like Cities: Skylines
This game runs 30-60 frames a second on the same machine because it uses more traditional techniques, drawing buildings made from triangles with textures on them, etc...
Still, let's go over using a Shadertoy shader with three.js.
This is the default shadertoy shader if you pick "New" on shadertoy.com, at least as of January 2019.
// By iq: https://www.shadertoy.com/user/iq // license: Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. void mainImage( out vec4 fragColor, in vec2 fragCoord ) { // Normalized pixel coordinates (from 0 to 1) vec2 uv = fragCoord/iResolution.xy; // Time varying pixel color vec3 col = 0.5 + 0.5*cos(iTime+uv.xyx+vec3(0,2,4)); // Output to screen fragColor = vec4(col,1.0); }
One thing important to understand about shaders is they are written in a
language called GLSL (Graphics Library Shading Language) designed for 3D math
which includes special types. Above we see vec4
, vec2
, vec3
as 3 such
special types. A vec2
has 2 values, a vec3
3, a vec4
4 values. They can be
addressed in a bunch of ways. The most common ways are with x
, y
, z
, and
w
as in
vec4 v1 = vec4(1.0, 2.0, 3.0, 4.0); float v2 = v1.x + v1.y; // adds 1.0 + 2.0
Unlike JavaScript, GLSL is more like C/C++ where variables have to have their
type declared so instead of var v = 1.2;
it's float v = 1.2;
declaring v
to be a floating point number.
Explaining GLSL in detail is more than we can do in this article. For a quick overview see this article and maybe follow that up with this series.
It should be noted that, at least as of January 2019, shadertoy.com only concerns itself with fragment shaders. A fragment shader's responsibility is, given a pixel location output a color for that pixel.
Looking at the function above we can see the shader has an out
parameter
called fragColor
. out
stands for output
. It's a parameter the function is
expected to provide a value for. We need to set this to some color.
It also has an in
(for input) parameter called fragCoord
. This is the pixel
coordinate that is about to be drawn. We can use that coordinate to decide on a
color. If the canvas we're drawing to is 400x300 pixels then the function will
be called 400x300 times or 120,000 times. Each time fragCoord
will be a
different pixel coordinate.
There are 2 more variables being used that are not defined in the code. One is
iResolution
. This is set to the resolution of the canvas. If the canvas is
400x300 then iResolution
would be 400,300 so as the pixel coordinates change
that makes uv
go from 0.0 to 1.0 across and up the texture. Working with
normalized values often makes things easier and so the majority of shadertoy
shaders start with something like this.
The other undefined variable in the shader is iTime
. This is the time since
the page loaded in seconds.
In shader jargon these global variables are called uniform variables. They are called uniform because they don't change, they stay uniform from one iteration of the shader to the next. It's important to note all of them are specific to shadertoy. They not official GLSL variables. They are variables the makers of shadertoy made up.
The Shadertoy docs define several more. For now let's write something that handles the two being used in the shader above.
The first thing to do is let's make a single plane that fills the canvas. If you haven't read it yet we did this in the article on backgrounds so let's grab that example but remove the cubes. It's pretty short so here's the entire thing
function main() { const canvas = document.querySelector('#c'); const renderer = new THREE.WebGLRenderer({antialias: true, canvas}); renderer.autoClearColor = false; const camera = new THREE.OrthographicCamera( -1, // left 1, // right 1, // top -1, // bottom -1, // near, 1, // far ); const scene = new THREE.Scene(); const plane = new THREE.PlaneGeometry(2, 2); const material = new THREE.MeshBasicMaterial({ color: 'red', }); scene.add(new THREE.Mesh(plane, material)); function resizeRendererToDisplaySize(renderer) { const canvas = renderer.domElement; const width = canvas.clientWidth; const height = canvas.clientHeight; const needResize = canvas.width !== width || canvas.height !== height; if (needResize) { renderer.setSize(width, height, false); } return needResize; } function render() { resizeRendererToDisplaySize(renderer); renderer.render(scene, camera); requestAnimationFrame(render); } requestAnimationFrame(render); } main();
As explained in the backgrounds article an
OrthographicCamera
with these parameters and a 2 unit plane will fill the
canvas. For now all we'll get is a red canvas as our plane is using a red
MeshBasicMaterial
.
Now that we have something working let's add the shadertoy shader.
const fragmentShader = ` #include <common> uniform vec3 iResolution; uniform float iTime; // By iq: https://www.shadertoy.com/user/iq // license: Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. void mainImage( out vec4 fragColor, in vec2 fragCoord ) { // Normalized pixel coordinates (from 0 to 1) vec2 uv = fragCoord/iResolution.xy; // Time varying pixel color vec3 col = 0.5 + 0.5*cos(iTime+uv.xyx+vec3(0,2,4)); // Output to screen fragColor = vec4(col,1.0); } void main() { mainImage(gl_FragColor, gl_FragCoord.xy); } `;
Above we declared the 2 uniform variables we talked about. Then we inserted the
shader GLSL code from shadertoy. Finally we called mainImage
passing it
gl_FragColor
and gl_FragCoord.xy
. gl_FragColor
is an official WebGL
global variable the shader is responsible for setting to whatever color it wants
the current pixel to be. gl_FragCoord
is another official WebGL global
variable that tells us the coordinate of the pixel we're currently choosing a
color for.
We then need to setup three.js uniforms so we can supply values to the shader.
const uniforms = { iTime: { value: 0 }, iResolution: { value: new THREE.Vector3() }, };
Each uniform in THREE.js has value
parameter. That value has to match the type
of the uniform.
Then we pass both the fragment shader and uniforms to a ShaderMaterial
.
-const material = new THREE.MeshBasicMaterial({ - color: 'red', -}); +const material = new THREE.ShaderMaterial({ + fragmentShader, + uniforms, +});
and before rendering we need to set the values of the uniforms
-function render() { +function render(time) { + time *= 0.001; // convert to seconds resizeRendererToDisplaySize(renderer); + const canvas = renderer.domElement; + uniforms.iResolution.value.set(canvas.width, canvas.height, 1); + uniforms.iTime.value = time; renderer.render(scene, camera); requestAnimationFrame(render); }
Note: I have no idea why
iResolution
is avec3
and what's in the 3rd value is not documented on shadertoy.com. It's not used above so just setting it to 1 for now. ¯\_(ツ)_/¯
This matches what we see on Shadertoy for a new shader, at least as of January 2019 😉. What's the shader above doing?
uv
goes from 0 to 1. cos(uv.xyx)
gives us 3 cosine values as a vec3
. One for uv.x
, another for uv.y
and another for uv.x
again.cos(iTime+uv.xyx)
makes them animate.vec3(0,2,4)
as in cos(iTime+uv.xyx+vec3(0,2,4))
offsets the cosine wavescos
goes from -1 to 1 so the 0.5 * 0.5 + cos(...)
converts from -1 <-> 1 to 0.0 <-> 1.0A minor change will make it easier to see the cosine waves. Right now uv
only
goes from 0 to 1. A cosine repeats at 2π so let's make it go from 0 to 40 by
multiplying by 40.0. That should make it repeat about 6.3 times.
-vec3 col = 0.5 + 0.5*cos(iTime+uv.xyx+vec3(0,2,4)); +vec3 col = 0.5 + 0.5*cos(iTime+uv.xyx*40.0+vec3(0,2,4));
Counting below I see about 6.3 repeats. We can see the blue between the red
since it's offset by 4 via the +vec3(0,2,4)
. Without that the blue and red
would overlap perfectly making purple.
Knowing how simple the inputs are and then seeing results like a city canal, a forest, a snail, a mushroom make the challenge all that much more impressive. Hopefully they also make it clear why it's not generally the right approach vs the more traditional ways of making scenes from triangles. The fact that so much math has to be put into computing the color of every pixel means those examples run very slow.
Some shadertoy shaders take textures as inputs like this one.
// By Daedelus: https://www.shadertoy.com/user/Daedelus // license: Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. #define TIMESCALE 0.25 #define TILES 8 #define COLOR 0.7, 1.6, 2.8 void mainImage( out vec4 fragColor, in vec2 fragCoord ) { vec2 uv = fragCoord.xy / iResolution.xy; uv.x *= iResolution.x / iResolution.y; vec4 noise = texture2D(iChannel0, floor(uv * float(TILES)) / float(TILES)); float p = 1.0 - mod(noise.r + noise.g + noise.b + iTime * float(TIMESCALE), 1.0); p = min(max(p * 3.0 - 1.8, 0.1), 2.0); vec2 r = mod(uv * float(TILES), 1.0); r = vec2(pow(r.x - 0.5, 2.0), pow(r.y - 0.5, 2.0)); p *= 1.0 - pow(min(1.0, 12.0 * dot(r, r)), 2.0); fragColor = vec4(COLOR, 1.0) * p; }
Passing a texture into a shader is similar to passing one into a normal material but we need to set up the texture on the uniforms.
First we'll add the uniform for the texture to the shader. They're referred to
as sampler2D
in GLSL.
const fragmentShader = ` #include <common> uniform vec3 iResolution; uniform float iTime; +uniform sampler2D iChannel0; ...
Then we can load a texture like we covered here and assign the uniform's value.
+const loader = new THREE.TextureLoader(); +const texture = loader.load('resources/images/bayer.png'); +texture.minFilter = THREE.NearestFilter; +texture.magFilter = THREE.NearestFilter; +texture.wrapS = THREE.RepeatWrapping; +texture.wrapT = THREE.RepeatWrapping; const uniforms = { iTime: { value: 0 }, iResolution: { value: new THREE.Vector3() }, + iChannel0: { value: texture }, };
So far we've been using Shadertoy shaders as they are used on
Shadertoy.com, namely drawing to cover the canvas.
There's no reason we need to limit it to just that use case though. The
important part to remember is the functions people write on shadertoy generally
just take a fragCoord
input and a iResolution
. fragCoord
does not have to
come from pixel coordinates, we could use something else like texture
coordinates instead and could then use them kind of like other textures. This
technique of using a function to generate textures is often called a
procedural texture.
Let's change the shader above to do this. The simplest thing to do might be to
take the texture coordinates that three.js normally supplies, multiply them by
iResolution
and pass that in for fragCoords
.
To do that we add in a varying. A varying is a value passed from the vertex
shader to the fragment shader that gets interpolated (or varied) between
vertices. To use it in our fragment shader we declare it. Three.js refers to its
texture coordinates as uv
with the v
in front meaning varying.
... +varying vec2 vUv; void main() { - mainImage(gl_FragColor, gl_FragCoord.xy); + mainImage(gl_FragColor, vUv * iResolution.xy); }
Then we need to also provide our own vertex shader. Here is a fairly common
minimal three.js vertex shader. Three.js declares and will provide values for
uv
, projectionMatrix
, modelViewMatrix
, and position
.
const vertexShader = ` varying vec2 vUv; void main() { vUv = uv; gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 ); } `;
We need to pass the vertex shader to the ShaderMaterial
const material = new THREE.ShaderMaterial({ vertexShader, fragmentShader, uniforms, });
We can set the iResolution
uniform value at init time since it will no longer change.
const uniforms = { iTime: { value: 0 }, - iResolution: { value: new THREE.Vector3() }, + iResolution: { value: new THREE.Vector3(1, 1, 1) }, iChannel0: { value: texture }, };
and we no longer need to set it at render time
-const canvas = renderer.domElement; -uniforms.iResolution.value.set(canvas.width, canvas.height, 1); uniforms.iTime.value = time;
Otherwise I copied back in the original camera and code that sets up 3 rotating cubes from the article on responsiveness. The result:
I hope this at least gets you started on how to use a shadertoy shader with three.js. Again, it's important to remember that most shadertoy shaders are an interesting challenge (draw everything with a single function) rather than the recommended way to actually display things in a performant way. Still, they are amazing, impressive, beautiful, and you can learn a ton by seeing how they work.


Contact Us--------
Home
Wildlife
Activities
Purchases
About Us
Contact Us
Do you wish to get in touch with us?
📞 | Talk to us on :
General :
+94 112712751
Fax : +94 112734542
📧 | Email us at :
<EMAIL_ADDRESS>Full name :
E-mail address :
Phone number :
Country :
Sri Lanka
India
Other
Message :