diff options
| author | peter <peter> | 2005-01-18 17:46:31 +0000 |
|---|---|---|
| committer | peter <peter> | 2005-01-18 17:46:31 +0000 |
| commit | 1bac56fd0ba482dbcb8a71f128de1849b4abe837 (patch) | |
| tree | 567d63160e9c0a97b1563f6dbf544c6d13ba52aa /www/crawl/DESCR | |
| parent | acbc14f07618a44ab7becf4c6c3f4649010b1e01 (diff) | |
| download | pkgsrc-1bac56fd0ba482dbcb8a71f128de1849b4abe837.tar.gz | |
Initial import of crawl-0.4 into the NetBSD Packages Collection.
The crawl utility starts a depth-first traversal of the web at the specified
URLs. It stores all JPEG images that match the configured constraints.
Crawl is fairly fast and allows for graceful termination. After terminating
crawl, it is possible to restart it at exactly the same spot where it was
terminated. Crawl keeps a persistent database that allows multiple crawls
without revisiting sites.
The main features of crawl are:
* Saves encountered images or other media types
* Media selection based on regular expressions and size contraints
* Resume previous crawl after graceful termination
* Persistent database of visited URLs
* Very small and efficient code
* Asynchronous DNS lookups
* Supports robots.txt
Diffstat (limited to 'www/crawl/DESCR')
| -rw-r--r-- | www/crawl/DESCR | 16 |
1 files changed, 16 insertions, 0 deletions
diff --git a/www/crawl/DESCR b/www/crawl/DESCR new file mode 100644 index 00000000000..854815c3ab0 --- /dev/null +++ b/www/crawl/DESCR @@ -0,0 +1,16 @@ +The crawl utility starts a depth-first traversal of the web at the specified +URLs. It stores all JPEG images that match the configured constraints. +Crawl is fairly fast and allows for graceful termination. After terminating +crawl, it is possible to restart it at exactly the same spot where it was +terminated. Crawl keeps a persistent database that allows multiple crawls +without revisiting sites. + +The main features of crawl are: + + * Saves encountered images or other media types + * Media selection based on regular expressions and size contraints + * Resume previous crawl after graceful termination + * Persistent database of visited URLs + * Very small and efficient code + * Asynchronous DNS lookups + * Supports robots.txt |