Update to 0.8.0

0.8.0:
	* New fuzzy check logic:
		- use shingles algorithm for fuzzy matching
		- use blake2 instead of md5 for larger output space
		- combine fuzzy and strict matching
		- allow to organize private storages by means of keys
		- preserve compatibility with previous versions
	* New fuzzy storage:
		- use sqlite instead of own memory based hash tables
		- rework commands interface
		- add conversion from the old format
		- add fuzzy match by shignles
		- support old rspamd versions
	* Add lemmatizing for words used in fuzzy hashes that allows to improve match
	quality by using of the first forms of all words
	* Rework language detection
	* Fix several critical bugs, memory leaks and deadlocks:
		- memory leak in HTML nodes parsing
		- deadlock in logger code
		- deadlock in signals processing
		- crashes in fuzzy_storage
		- crashes in tokenizers if the input was empty
	* Import new libucl with several bugfixes and improvements
	* Support listening on ipv6 addresses only
	* Fix macro expansion in SPF module
	* Several bugfixes in DKIM module
	* Add load headers support for mime parts to the lua API
	* Add documentation for:
		- workers in general
		- fuzzy_storage worker
		- fuzzy_check plugin
		- mimepart and textpart lua API modules

0.7.6:
	* Apply boundary fix for dkim simple canonization
	* Fix ping command
	* Return nil if header was not found in lua_task
	* Fix hang in upstreams revive logic
	* Decode entitles when normalizing HTML parts
	* Fix logic of finding URLs in HTML parts
	* Do not include \0 into length of text when performing conversion to utf8
	* Fix raw vs parsed reperesentations
		Raw parts are now:
			- decoded b64/qp, but *NOT* converted to utf-8
		Processed parts are now:
			- converted to UTF-8
			- normalized if needed (e.g. HTML tags are stripped)
	* Rework DKIM canonization to line based
	* Fix fuzzy hashes addding
	* Use more specific hash function for fuzzy
	* Fix leaking of iconv descriptors
	* Fix PTR resolving in lua resolver
	* Rework spf module.
		- Copy data to memory pool as cached record might be destroyed causing
		  freed memory being passed to the protocol output (use after free)
		- Allow SPF_NEUTRAL policy to be handled separately
		- Add R_SPF_NEUTRAL to the default config
	* Rework `register_symbols` function
	* Allow to disable components of hfilter

0.7.5:
	* Fix owner when creating folder /run/rspamd (by @sfirmery)
	* Fix IP validity checks
	* Decode URLs obtained from HTML tags
	* Fix crash with unweighted upstreams
	* Stop processing headers in parts
	* Set sockaddr.sa_family properly when connectig to upstreams
	* Fix reload issues in surbl and fuzzy_check (reported by @citrin)
	* Fix timeouts in redirector
	* Improve lua errors reporting
	* Fix lua closures processing in libucl
	* Rework calling of lua functions from regexp module
	* Choose raw regexp for raw headers
	* Rework conversion to utf since glib one is broken
	* Ignore SGML style tags in html
	* Fix old bug with non-capturing https urls
	* Fix memory corruption on fuzzy reload (reported by @citrin)
	* Fix percents display in rspamc
	* Fix buffer update for DKIM
	* Do not validate utf for raw headers

0.7.4:
	* Fix build under *BSD
	* Detect HAN unicode script
	* Implement language detection heuristic for text parts
	* Fix time output in history
	* Improve piechart coloring
	* Fix \r\n conversion in DKIM module (reported by @citrin)
	* Try to detect systems with no IPv6 support
	* Fix multiple/single values in use settings (reported by @citrin)
	*  Rework IP addresses in upstreams:
		- Select ipv4/unix addresses if they exist and use ipv6 for ipv6 only
		  upstreams (since the support of ipv6 is poor in many OSes and
		  environments)
		- Free IP list on upstream destruction
		- Add test cases for addresses selection
		- Allow adding of free form IP addresses to upstreams
    * Fix endiannes in lua_radix search (reported by @citrin)
    * Soft shutdown should also set wanna_die flag (reported by @citrin)
    * Stop use-after-free in event loop termination
    * Fix processing of very short messages in DKIM (reported by @citrin)
    * Detect systems without shared mutexes
    * Fix issues with PTR and MX elements in SPF parser (reported by @citrin)

0.7.3:
	* New upstreams code:
		- simplify upstreams API;
		- unify strings parsing in upstreams definition;
		- add configuration options for the upstreams;
		- for failed upstreams re-resolve their addresses;
		- use all resolved addresses for an upstream (round-robin);
		- implement stable hashing and use it by default for upstreams;
		- add unit test for upstreams module.
	* Rework signals processing in all rspamd workers:
		- signals are now processed in the event loop;
		- implement the most common signal handlers for all workers;
		- add callbacks for workers specific signal handlers
	*  Fix critical issue with fuzzy storage:
		Fuzzy stroage could not save any hashes on termination due to bugged
		signals handling
    * Fix roll history IP storage
    * Rework ipv4/ipv6 handling in parsing addresses:
		- turn off support of IPV6_V6ONLY socket option;
		- create ipv6 socket prior to ipv4 one to handle systems with v6/v4
		sockets enabled (Linux)
    * Remove CBL as it's wholly included in Spamhaus XBL (by @fatalbanana)
    * Remove nszones.com fake RBL (by @citrin)
    * Fix upstreams interaction for fuzzy_check
    * Verify spf PTR records (reported by @citrin)
    * Fix spf MX records parsing
    * Add compatibility for old libevent (by @yellowman)
    * Sync bugfixes from libucl

0.7.2:
	* Convert all maps to the compressed radix trie
	* Allow IPv6 addresses in IP maps
	* Remove dynamic items support from symbols cache
	* Allow hex encoded output of strings
	* Fix bug with control connections count
	* Process fuzzy weight correctly (reported by @fatalbanana)
	* Remove extra reference retain of http connection on error
	* Remove deprecated options from the default config
	* Add `one_shot` attr to metric's symbols
	* Doc: add documentation for metrics
	* Add Upstart job to debian packaging (by @CameronNemo)
	* Config: improve SURBL symbols descriptions (by @citrin)
	* Config: reflect SURBL changes (by @citrin):
    	- Outblaze removed, malware moved to separate list:
    	http://www.surbl.org/news/internal/MW-malware-sublist-added-to-multi
    * Fix C modules initialization on restart
    * Treat single IP as a single IP in radix lists (reported by @citrin)
    * Do not touch file and core limits if not asked explicitly (reported by @citrin)
    * Improve logging for fuzzy errors
    * Block SIGPIPE for HTTP writing
    * Doc: update manual pages
    * Fix HTTP connection termination
    * Reduce default number of parallel requests to 8
    * Sync with libucl include features

0.7.1:
	* Fix typo in stat output.
	* Fix issues with includes crossing with the system includes
	* Restore testing framework
	* Add radix trie test suite
	* Implement new path-compressed radix trie.
	    - The performance benefit over the old algorithm is about 1.5 times.
		- Memory usage is significantly lower as well.
	    - Now radix trie can accept any IPv4/IPv6 values
	* Various improvements to the memory pools code
	* Fix writing reply to a client when no filters are defined
	* Write base32 encoded fuzzy
	* Fix 'soft reject' action
	* Fix rspamd reload and modules reconfiguration
	* Fix subject rewriting for the default subject
	* Fix states for processing task and pre-filters
	* Fix issues with connection closing
	* Fix crashes in rdns
	* Fix ratelimit pre-filter
	* Update exim patch.
		- Update to the recent exim version
		- Strip extra leading src/ from the patch
		- Remove sendfile since it was broken
		- Fix rspamd spam report for exim
	* Improve documentation

0.7.0:
	* Use HTTP protocol for all operatiosn
	* Webui worker is now removed and controller works as webui
	* Allow to serve static files via controller by option `static_dir`
	* Rspamd interface is now a part of rspamd
	* Rspamc client has been rewritten to use HTTP and non-blocking mode
	allowing to start multiple operations simultaneously (see `-n` option)
	* Lua API was completely reworked to satisfy modern standards of LUA:
		* Module `lua-message` was removed
		* Reduced number of superglobals registered by rspamd
		* Many functions has been redesigned
		* Symbols registration is now more convenient
	* Users settings has been rewritten as lua plugin
	* Reworked headers system as gmime's based one misses many headers and is
	very slow to get headers values
	* Reorganized code and removed embedded jannsson by using UCL for all json
	parsing
	* Migrated to `librdns` for DNS resolving that improves concurrency for
	DNS requests significantly
	* Fixed tonns of bugs in MIME processing
	* Improved metrcis and default symbol's weights
	* Added new RBL's
	* Fixed a number of issues in the modules
	* Removed several memory leaks found
	* Fix unicode processing
	* Fix fuzzy checking for unicode parts
	* Significantly improve documentation and especially LUA API docs
	* See migration notes at https://rspamd.com/doc/migration.html

5 files changed, 72 insertions, 19 deletions

diff --git a/mail/rspamd/Makefile b/mail/rspamd/Makefile
index d625e7f3ddb..e05bcb58763 100644
--- a/mail/rspamd/Makefile
+++ b/mail/rspamd/Makefile
@@ -1,9 +1,9 @@
-# $NetBSD: Makefile,v 1.8 2014/10/19 22:27:46 alnsn Exp $
+# $NetBSD: Makefile,v 1.9 2015/01/17 17:34:07 wiedi Exp $
 
-DISTNAME=	rspamd-0.6.9
-PKGREVISION=	2
+DISTNAME=	rspamd-0.8.0
 CATEGORIES=	mail
-MASTER_SITES=	https://rspamd.com/downloads/
+MASTER_SITES=	http://www.rspamd.com/downloads/
+EXTRACT_SUFX=	.tar.xz
 
 MAINTAINER=	wiedi@frubar.net
 HOMEPAGE=	https://www.rspamd.com/
@@ -12,7 +12,8 @@ LICENSE=	2-clause-bsd
 
 USE_CMAKE=	yes
 USE_TOOLS+=	perl pkg-config
-
+USE_LANGUAGES=	c99
+CPPFLAGS.SunOS+=	-D__EXTENSIONS__
 LDFLAGS+=	${COMPILER_RPATH_FLAG}${PREFIX}/lib/rspamd
 
 PKG_SYSCONFSUBDIR=	rspamd
@@ -49,5 +50,6 @@ post-extract:
 .include "../../devel/libevent/buildlink3.mk"
 .include "../../devel/glib2/buildlink3.mk"
 .include "../../mail/gmime/buildlink3.mk"
+.include "../../databases/sqlite3/buildlink3.mk"
 .include "../../lang/lua/buildlink3.mk"
 .include "../../mk/bsd.pkg.mk"
diff --git a/mail/rspamd/PLIST b/mail/rspamd/PLIST
index a53a9aa0a07..62f9a8d1671 100644
--- a/mail/rspamd/PLIST
+++ b/mail/rspamd/PLIST
@@ -1,28 +1,21 @@
-@comment $NetBSD: PLIST,v 1.1 2014/02/07 14:39:19 wiedi Exp $
+@comment $NetBSD: PLIST,v 1.2 2015/01/17 17:34:07 wiedi Exp $
 bin/rspamc
 bin/rspamc-${PKGVERSION}
 bin/rspamd
 bin/${PKGNAME}
-include/rspamd/librspamdclient.h
 lib/rspamd/librspamd-cdb.so
 lib/rspamd/librspamd-cdb.so.${PKGVERSION}
 lib/rspamd/librspamd-http-parser.so
 lib/rspamd/librspamd-http-parser.so.${PKGVERSION}
-lib/rspamd/librspamd-json.so
-lib/rspamd/librspamd-json.so.${PKGVERSION}
 lib/rspamd/librspamd-lua.so
 lib/rspamd/librspamd-lua.so.${PKGVERSION}
 lib/rspamd/librspamd-mime.so
 lib/rspamd/librspamd-mime.so.${PKGVERSION}
 lib/rspamd/librspamd-server.so
 lib/rspamd/librspamd-server.so.${PKGVERSION}
-lib/rspamd/librspamd-ucl.so
-lib/rspamd/librspamd-ucl.so.${PKGVERSION}
 lib/rspamd/librspamd-util.so
 lib/rspamd/librspamd-util.so.${PKGVERSION}
 lib/rspamd/librspamdclient.so
-lib/rspamd/librspamdclient.so.${PKGVERSION}
-lib/rspamd/librspamdclient_static.a
 man/man1/rspamc.1
 man/man8/rspamd.8
 share/examples/rspamd/2tld.inc
@@ -51,5 +44,15 @@ share/rspamd/lua/once_received.lua
 share/rspamd/lua/phishing.lua
 share/rspamd/lua/ratelimit.lua
 share/rspamd/lua/rbl.lua
+share/rspamd/lua/settings.lua
 share/rspamd/lua/trie.lua
-share/rspamd/lua/whitelist.lua
+share/rspamd/www/README.md
+share/rspamd/www/css/rspamd.css
+share/rspamd/www/img/asc.png
+share/rspamd/www/img/desc.png
+share/rspamd/www/img/spinner.gif
+share/rspamd/www/img/spinner.png
+share/rspamd/www/index.html
+share/rspamd/www/js/jquery.paginatetable.js
+share/rspamd/www/js/rspamd.js
+share/rspamd/www/plugins.txt
diff --git a/mail/rspamd/distinfo b/mail/rspamd/distinfo
index d434f802aab..04ae1c54256 100644
--- a/mail/rspamd/distinfo
+++ b/mail/rspamd/distinfo
@@ -1,6 +1,7 @@
-$NetBSD: distinfo,v 1.2 2014/05/14 16:07:06 wiedi Exp $
+$NetBSD: distinfo,v 1.3 2015/01/17 17:34:07 wiedi Exp $
 
-SHA1 (rspamd-0.6.9.tar.gz) = aed555b278509c99020624f38d2a77c38a3fa0f6
-RMD160 (rspamd-0.6.9.tar.gz) = 233d75efd3af370e3b504d9f13f7f282482a0cbe
-Size (rspamd-0.6.9.tar.gz) = 1088703 bytes
-SHA1 (patch-CMakeLists.txt) = e6d86ceea7351276937c8e6bc25d1497f3ad06da
+SHA1 (rspamd-0.8.0.tar.xz) = b3d704c0cf3ed92271f3f4f640f6406c7ec267b2
+RMD160 (rspamd-0.8.0.tar.xz) = 02b4889a3e5d3d89357be50a05bf4a6b15e2b738
+Size (rspamd-0.8.0.tar.xz) = 754636 bytes
+SHA1 (patch-src_client_rspamc.c) = cd4001655198b93fed898a01b5adbc206a25c604
+SHA1 (patch-src_lua_lua__task.c) = 88dfa4f7ee98aad5d91b4aac4a40324ea1339598
diff --git a/mail/rspamd/patches/patch-src_client_rspamc.c b/mail/rspamd/patches/patch-src_client_rspamc.c
new file mode 100644
index 00000000000..6515a555b3f
--- /dev/null
+++ b/mail/rspamd/patches/patch-src_client_rspamc.c
@@ -0,0 +1,32 @@
+$NetBSD: patch-src_client_rspamc.c,v 1.1 2015/01/17 17:34:07 wiedi Exp $
+the struct dirent d_type is not available on SunOS
+
+--- src/client/rspamc.c.orig	2014-12-22 17:57:39.000000000 +0000
++++ src/client/rspamc.c
+@@ -853,6 +853,9 @@ rspamc_process_dir (struct event_base *e
+ 	DIR *d;
+ 	gint cur_req = 0;
+ 	struct dirent *ent;
++#if defined(__sun)
++	struct stat sb;
++#endif
+ 	FILE *in;
+ 	char filebuf[PATH_MAX];
+ 
+@@ -860,9 +863,14 @@ rspamc_process_dir (struct event_base *e
+ 
+ 	if (d != NULL) {
+ 		while ((ent = readdir (d))) {
++			rspamd_snprintf (filebuf, sizeof (filebuf), "%s%c%s",
++					name, G_DIR_SEPARATOR, ent->d_name);
++#if defined(__sun)
++			if (stat (filebuf, &sb)) continue;
++			if (S_ISREG (sb.st_mode)) {
++#else
+ 			if (ent->d_type == DT_REG || ent->d_type == DT_UNKNOWN) {
+-				rspamd_snprintf (filebuf, sizeof (filebuf), "%s%c%s",
+-						name, G_DIR_SEPARATOR, ent->d_name);
++#endif
+ 				if (access (filebuf, R_OK) != -1) {
+ 					in = fopen (filebuf, "r");
+ 					if (in == NULL) {
diff --git a/mail/rspamd/patches/patch-src_lua_lua__task.c b/mail/rspamd/patches/patch-src_lua_lua__task.c
new file mode 100644
index 00000000000..a2ab23644eb
--- /dev/null
+++ b/mail/rspamd/patches/patch-src_lua_lua__task.c
@@ -0,0 +1,15 @@
+$NetBSD: patch-src_lua_lua__task.c,v 1.1 2015/01/17 17:34:07 wiedi Exp $
+
+tm_gmtoff is not a POSIX field
+--- src/lua/lua_task.c.orig	2015-01-02 11:07:18.000000000 +0000
++++ src/lua/lua_task.c
+@@ -1765,7 +1765,9 @@ lua_task_get_date (lua_State *L)
+ 
+ 				tt = tim;
+ 				localtime_r (&tt, &t);
++#if !defined(__sun)
+ 				t.tm_gmtoff = 0;
++#endif
+ 				t.tm_isdst = 0;
+ 				tim = mktime (&t);
+ 			}