summaryrefslogtreecommitdiff
path: root/www/crawl
AgeCommit message (Collapse)AuthorFilesLines
2006-04-22Removed the superfluous "quotes" and 'quotes' from variables that don'trillig1-2/+2
need them, for example RESTRICTED and SUBST_MESSAGE.*.
2005-12-29Remove USE_PKGINSTALL from pkgsrc now that mk/install/pkginstall.mkjlam1-2/+1
automatically detects whether we want the pkginstall machinery to be used by the package Makefile.
2005-12-05Fixed pkglint warnings. The warnings are mostly quoting issues, forrillig1-2/+2
example MAKE_ENV+=FOO=${BAR} is changed to MAKE_ENV+=FOO=${BAR:Q}. Some other changes are outlined in http://mail-index.netbsd.org/tech-pkg/2005/12/02/0034.html
2005-09-16"Oops." BUILDLINK_DEPMETHOD.libevent was defaulting to "build", which istv1-1/+2
no longer correct since update to libevent 1.x; it now uses libtool and generates a shlib. Remove the offending bl3 line, and bump all dependents' PKGREVISIONs, since the binary pkg changes for any OS that doesn't have a sufficient builtin libevent version (or the package has requested a non-builtin version).
2005-04-11Remove USE_BUILDLINK3 and NO_BUILDLINK; these are no longer used.tv1-2/+1
2005-02-24Add RMD160 checksums.wiz1-1/+2
2005-01-18Initial import of crawl-0.4 into the NetBSD Packages Collection.peter7-0/+104
The crawl utility starts a depth-first traversal of the web at the specified URLs. It stores all JPEG images that match the configured constraints. Crawl is fairly fast and allows for graceful termination. After terminating crawl, it is possible to restart it at exactly the same spot where it was terminated. Crawl keeps a persistent database that allows multiple crawls without revisiting sites. The main features of crawl are: * Saves encountered images or other media types * Media selection based on regular expressions and size contraints * Resume previous crawl after graceful termination * Persistent database of visited URLs * Very small and efficient code * Asynchronous DNS lookups * Supports robots.txt