Add documentation for WARC data format
@@ -10,6 +10,7 @@ The data formats currently available are:
* :ref:`FILESYSTEM_HTML, FILESYSTEM_JSON, FILESYSTEM_CBOR <dataformat-filesystem>`
* :ref:`FILES <dataformat-files>`
+* :ref:`WARC <dataformat-warc>`
* :ref:`ELATICSEARCH <dataformat-elasticsearch>`
@@ -50,7 +51,7 @@ To fix this you can configure the file format to use a fixed size hash of the UR
FILES
-----
-Raw content and metadata is stored in rolling compressed files of fixed size (256MB).
+Raw content and metadata are stored in rolling compressed files of fixed size (256MB).
Each file is a JSON lines file (each line contains one JSON object) compressed using the DEFLATE algorithm.
Each JSON object has the following fields:
@@ -61,6 +62,32 @@ Each JSON object has the following fields:
* ``response_headers`` - An array containing the HTTP response headers
* ``fetch_time`` - A integer containing the time when the page was fetched (epoch)
+.. _dataformat-warc:
+
+-----
+WARC
+-----
+
+Raw content and metadata are stored in `WARC <https://en.wikipedia.org/wiki/Web_ARChive>`_ files.
+WARC is the standard format used by The Web Archive and other public web datasets
+such as "Common Crawl" and "ClueWeb".
+See http://commoncrawl.org/2014/04/navigating-the-warc-file-format/ for more details on the WARC format.
+
+Every WARC file generated by ACHE contains one `warcinfo` entry and one
+`response` entry for each downloaded page.
+By default, the files are compressed using GZIP format and have an approximate
+size of 250MB (usually slightly larger).
+The default settings can be changed using the following entries in ``ache.yml`` file:
+
+.. code:: yaml
+
+ target_storage.data_format.type: WARC # enable WARC file format
+ target_storage.data_format.warc.compress: true # enable GZIP compression
+ target_storage.data_format.warc.max_file_size: 262144000 # maximum file size in bytes
+
+Finally, ACHE also stores additional metadata as non-standard extension WARC
+headers prefixed by ``ACHE-*`` (e.g., ``ACHE-IsRelevant``, ``ACHE-Relevance``).
+
.. _dataformat-elasticsearch:
-------------
