Pass in in-depth website crawl tutorial
-In-Depth Crawl
-##############
+Running a In-Depth Website Crawl
+################################
-An in-depth crawl in ACHE will allow the user to crawl a list of websites and the links present in them completely. The crawler will stop once these links have been crawled.
+When you already know what sites need to be crawled, you can run a in-depth website crawl using ACHE. An in-depth crawl will allow the user to crawl a list of seeds websites and all the links found in them completely. The crawler will stop once these links have been crawled.
-The process for running an in-depth crawl is simpler than for a focused crawl. An in-depth crawl doesn't require a model, it just needs a list of websites to crawl along with a configuration file to act as the appropriate settings for ACHE.
+The process for running an in-depth crawl is simpler than for a focused crawl. An in-depth crawl doesn't require a model, it just needs a list of websites to crawl along with a configuration file with the appropriate settings for ACHE.
-The following steps explain how to run such a crawl on ACHE.
+The following steps explain how to run such a crawl using ACHE.
-
-Steps
------
#. Prepare Seeds
- A seeds file should contain URLs of all websites that need to be crawled. A sample seeds file can be seen at ``ache/config/sample.seeds``.
-
-#. Create Configuration File
- The configuration file will contain the settings and rules the crawler will adhere to. A sample of the config file is provided in ``ache/config/config_focused_crawl`` containing the following important parameters:
+ A seeds file should contain URLs of all websites that need to be crawled. A sample seeds file can be seen at `config/sample.seeds <https://github.com/ViDA-NYU/ache/blob/master/config/sample.seeds>`_.
- * ``target_storage.data_format.type`` - Which format to save the retrieved pages in. More detail :ref:`here <dataformat-filesystem>`.
+#. Create an ``ache.yml`` configuration file
+ The configuration file will contain the settings and rules the crawler will adhere to. A sample of the config file for running a in-depth website crawl is provided in `config/config_website_crawl/ache.yml <https://github.com/ViDA-NYU/ache/blob/master/config/config_website_crawl/ache.yml>`_ containing the following important parameters:
- * ``link_storage.max_pages_per_domain`` - The number of maximum pages to download per domain. Useful in case of traps/redirects.
+ * ``link_storage.link_strategy.use_scope`` - Whether the crawler should crawl the websites provided as seeds only. This needs to be set to ``true`` in a in-depth website crawl.
- * ``link_storage.download_sitemap_xml`` - Whether to use the sitemap for the website (if available).
+ * ``link_storage.download_sitemap_xml`` - Whether to use the Sitemap protocol for discovery of links (if available). It helps to discover URLs quicker.
- * ``link_storage.scheduler.host_min_access_interval`` - Time delay between sending subsequent requests to the same host.
+ * ``link_storage.scheduler.host_min_access_interval`` - Configures the minimum time interval (in milliseconds) to wait between subsequent requests to the same host to avoid overloading servers. If you are crawling your own web site, you can descrease this value to speed-up the crawl.
#. Run ACHE
- To run ACHE, in the terminal::
+ Finally, when you have created the config file *ache.yml*, and the *seeds file*, you can run ACHE in the terminal::
ache startCrawl -c <Path of Config Folder> -o <Path of Output Folder> -s <Path of Seeds File>
- ACHE will continue crawling until interrupted using ``CTRL+C`` or until the seeds list has been exhausted.
\ No newline at end of file
+ ACHE will continue crawling until interrupted using ``CTRL+C`` or until the queue of all links found during the crawl has been exhausted.
