john: Update to 1.9.0

pkgsrc changes:
 - Document all the patches
 - Honors user's CFLAGS and don't remove -Wall from CFLAGS in patch-aa: they are
   usually pretty useful
 - Unset OPT_{NORMAL,INLINE} optimizations via MAKE_FLAGS to minimize patch-aa
 - Remove not needed NO_CONFIGURE
 - Use pre-configure as stage for SUBST (now that NO_CONFIGURE is removed)

Changes:
The following changes have been made between John 1.8.0 and 1.9.0:

* Increased the interleaving for bcrypt on x86-64 from 2x to 3x for a major
speedup on CPUs without SMT.  Unfortunately, this sometimes results in a minor
performance regression when running multiple threads on CPUs with SMT.
* Recognize the $2b$ bcrypt prefix.
* In the generic crypt(3) format, detect descrypt with valid vs. invalid salts
as separate id's for our heuristics on supported hash types.
* Introduced a number of optimizations for faster handling of large password
hash files, including loading, cracking, and "--show".  Some of these use more
memory than before, yet in a more efficient manner.
* Benchmark using all-different candidate passwords of length 7 by default.
* Dropped undocumented special handling of "Mc" in 'c' and 'C' rule commands.
* Dropped undocumented limitation of the 'M' and 'Q' rule commands where they
would sometimes memorize/check only up to the current hash type's length limit
yet this optimization wouldn't necessarily be transparent (e.g., if a later
command would extract a substring from above the hash type's length limit and
bring it to within the limit).
* Implemented special-case handling of repeated rule commands '$', '^', '[',
']', '{', and '}', as well as faster handling of the 'D' command.
* When built with "--fork" support, disallow session names with all-digit
suffixes since these clash with those produced by "--fork".
* Forward SIGTERM to --fork'ed children.
* Set stdout to line buffered (rather than potentially fully buffered), except
for "--stdout", "--show", and auxiliary programs such as "unshadow".
* On Windows, restore normal processing of Ctrl-C in case our parent (such as
Johnny the GUI) had disabled it.
* Added linux-x86*-avx512 and linux-x86*-avx2 make targets, which use
respectively AVX-512 and AVX2 for bitslice DES.
* Added linux-mic make target for Intel MIC (first generation Xeon Phi, aka
Knights Corner), which uses its 512-bit SIMD intrinsics for bitslice DES.
(For second generation Xeon Phi, aka Knights Landing, use linux-x86-64-avx512.)
* Added linux-arm64le, linux-arm32le-neon, and linux-arm32le make targets.
(The first two of these make use of ASIMD or NEON for bitslice DES.)
* Added linux-sparc64 make target.
* Made a minor optimization to MMX and SSE2 assembly code for LM hash.
* Dropped Ultrix and SCO support.
* Don't probe for alternate config file names (like john.ini when on Unix).
* "DokuWiki" external mode sample has been added to the default john.conf.
* Fixed operator precedence in the external mode compiler to be the same as C.
* Fixed an out of bounds write bug in the external mode virtual machine.
* Fixed a bug introduced in version 1.7.4 in the wordlist rules engine, where
some sequences of rule commands could overflow a word buffer.
* Fixed a bug where unaligned access SSE/AVX instructions would unnecessarily
be generated by GCC 4.6+ in the bitslice DES code in non-OpenMP builds.
* Fixed a bug where "Warning: no OpenMP support for this hash type" could be
printed in "--stdout" mode.
* Made assorted other bugfixes, portability and documentation enhancements.

6 files changed, 71 insertions, 43 deletions

diff --git a/security/john/Makefile b/security/john/Makefile
index 35247bd4a1d..f9659ee8d36 100644
--- a/security/john/Makefile
+++ b/security/john/Makefile
@@ -1,9 +1,9 @@
-# $NetBSD: Makefile,v 1.28 2020/01/26 17:32:03 rillig Exp $
+# $NetBSD: Makefile,v 1.29 2020/08/08 02:09:01 leot Exp $
 #
 
-DISTNAME=	john-1.8.0
+DISTNAME=	john-1.9.0
 CATEGORIES=	security textproc
-MASTER_SITES=	http://www.openwall.com/john/j/
+MASTER_SITES=	http://www.openwall.com/john/k/
 EXTRACT_SUFX=	.tar.xz
 
 MAINTAINER=	pkgsrc-users@NetBSD.org
@@ -11,7 +11,6 @@ HOMEPAGE=	https://www.openwall.com/john/
 COMMENT=	Unix Password Cracker
 LICENSE=	gnu-gpl-v2
 
-NO_CONFIGURE=	yes
 BUILD_DIRS=	src
 TEST_TARGET=	check
 
@@ -68,12 +67,17 @@ BUILD_TARGET=		generic
 BUILD_TARGET=		generic
 .endif
 
+# Avoid possible per-platform optimization
+MAKE_FLAGS+=	OPT_NORMAL=
+MAKE_FLAGS+=	OPT_INLINE=
+
 INSTALLATION_DIRS=	bin share/john
 
-SUBST_CLASSES+=		john
-SUBST_STAGE.john=	pre-build
-SUBST_FILES.john=	src/params.h
-SUBST_VARS.john=	PREFIX
+SUBST_CLASSES+=			fix-paths
+SUBST_STAGE.fix-paths=		pre-configure
+SUBST_MESSAGE.fix-paths=	Fixing hardcoded paths.
+SUBST_FILES.fix-paths=		src/params.h
+SUBST_VARS.fix-paths=		PREFIX
 
 do-install:
 	# programs and scripts
@@ -86,7 +90,8 @@ do-install:
 	${LN} -sf john ${DESTDIR}${PREFIX}/bin/${i}
 .endfor
 	# data files
-.for i in ascii.chr digits.chr lm_ascii.chr password.lst
+.for i in alnum.chr alpha.chr ascii.chr digits.chr lm_ascii.chr lower.chr \
+    lowernum.chr lowerspace.chr upper.chr uppernum.chr password.lst
 	${INSTALL_DATA} ${WRKSRC}/run/${i} ${DESTDIR}${PREFIX}/share/john
 .endfor
 
diff --git a/security/john/PLIST b/security/john/PLIST
index bcfba2150b7..3e5f86c1c4c 100644
--- a/security/john/PLIST
+++ b/security/john/PLIST
@@ -1,11 +1,18 @@
-@comment $NetBSD: PLIST,v 1.4 2014/09/14 13:20:08 shattered Exp $
+@comment $NetBSD: PLIST,v 1.5 2020/08/08 02:09:01 leot Exp $
 bin/john
 bin/mailer
 bin/unafs
 bin/unique
 bin/unshadow
+share/john/alnum.chr
+share/john/alpha.chr
 share/john/ascii.chr
 share/john/digits.chr
 share/john/john.conf
 share/john/lm_ascii.chr
+share/john/lower.chr
+share/john/lowernum.chr
+share/john/lowerspace.chr
 share/john/password.lst
+share/john/upper.chr
+share/john/uppernum.chr
diff --git a/security/john/distinfo b/security/john/distinfo
index 619b91ba532..6e3dd9b3d09 100644
--- a/security/john/distinfo
+++ b/security/john/distinfo
@@ -1,8 +1,9 @@
-$NetBSD: distinfo,v 1.15 2015/11/04 01:17:47 agc Exp $
+$NetBSD: distinfo,v 1.16 2020/08/08 02:09:01 leot Exp $
 
-SHA1 (john-1.8.0.tar.xz) = 423901b9b281c26656234ee31b362f1c0c2b680c
-RMD160 (john-1.8.0.tar.xz) = fda9971cd78bdf2b5154586efde2504c9a5974a4
-SHA512 (john-1.8.0.tar.xz) = e2820ecabd7ea4c7404c3d40f064994d0f5746b093c859a58fbf2588492ebcef411d9f3088c8e8ee726284ead969e26fdae834cdae3eb2d32408fc79ed906543
-Size (john-1.8.0.tar.xz) = 4468704 bytes
-SHA1 (patch-aa) = 15ba3d70444435c6705f189b32af109e1f211ff4
-SHA1 (patch-ab) = 4f8c9e1bba14f64b651c6d08337f004a3e278e44
+SHA1 (john-1.9.0.tar.xz) = 5e15e41adb96768cb1ea05af1b3128059df842df
+RMD160 (john-1.9.0.tar.xz) = 16114db2fe5004ce4044a2d56b46ef209b941dc4
+SHA512 (john-1.9.0.tar.xz) = 26e9a245e9f050344ae8c3320e1e24dad6bc2b73e99cc041a50c2d124b2c898c93d048727451ab8a2ba1c8aa4d8d462a8c0c9fea7de39f3479a0c170a5e9ad60
+Size (john-1.9.0.tar.xz) = 8944932 bytes
+SHA1 (patch-aa) = 8bc111b75bb848f7373f339c77075886e0c54098
+SHA1 (patch-ab) = 7cce81d7692b556ee35b90912192ae48c5b98cc7
+SHA1 (patch-src_logger.c) = e97548fe857ee58da21cab2d763155360b70a964
diff --git a/security/john/patches/patch-aa b/security/john/patches/patch-aa
index 808010f2da5..4809fe9ed2d 100644
--- a/security/john/patches/patch-aa
+++ b/security/john/patches/patch-aa
@@ -1,23 +1,20 @@
-$NetBSD: patch-aa,v 1.9 2014/09/14 13:20:08 shattered Exp $
+$NetBSD: patch-aa,v 1.10 2020/08/08 02:09:01 leot Exp $
 
---- src/Makefile.orig	2013-05-29 23:21:25.000000000 +0000
+Honors user's {C,LD}FLAGS and do not strip.
+
+--- src/Makefile.orig	2019-04-10 16:53:34.000000000 +0000
 +++ src/Makefile
-@@ -28,14 +28,13 @@ OMPFLAGS =
- #OMPFLAGS = -fopenmp -D_FORTIFY_SOURCE=0
+@@ -30,11 +30,11 @@ OMPFLAGS =
+ #OMPFLAGS = -openmp
  # Sun Studio with OpenMP (set the OMP_NUM_THREADS env var at runtime)
  #OMPFLAGS = -xopenmp
--CFLAGS = -c -Wall -Wdeclaration-after-statement -O2 -fomit-frame-pointer $(OMPFLAGS)
-+CFLAGS = -c $(OMPFLAGS)
+-CFLAGS = -c -Wall -O2 -fomit-frame-pointer $(OMPFLAGS)
++CFLAGS += -c -Wall $(OMPFLAGS)
  # CFLAGS for use on the main john.c file only
  CFLAGS_MAIN = $(CFLAGS)
  ASFLAGS = -c $(OMPFLAGS)
 -LDFLAGS = -s $(OMPFLAGS)
--OPT_NORMAL = -funroll-loops
--# Remove the "-Os" if you're using an ancient version of gcc
--OPT_INLINE = -Os -funroll-loops -finline-functions
-+LDFLAGS += -s $(OMPFLAGS)
-+OPT_NORMAL =
-+OPT_INLINE =
- 
- JOHN_OBJS = \
- 	DES_fmt.o DES_std.o DES_bs.o DES_bs_b.o \
++LDFLAGS += $(OMPFLAGS)
+ OPT_NORMAL = -funroll-loops
+ # Remove the "-Os" if you're using an ancient version of gcc
+ OPT_INLINE = -Os -funroll-loops -finline-functions
diff --git a/security/john/patches/patch-ab b/security/john/patches/patch-ab
index 440db43d2a7..0f4d3a886ba 100644
--- a/security/john/patches/patch-ab
+++ b/security/john/patches/patch-ab
@@ -1,8 +1,10 @@
-$NetBSD: patch-ab,v 1.2 2010/12/13 10:32:35 wiz Exp $
+$NetBSD: patch-ab,v 1.3 2020/08/08 02:09:01 leot Exp $
 
---- src/params.h.orig	2010-06-14 02:38:55.000000000 +0000
+Use @PREFIX@ instead of hardcoding /usr.
+
+--- src/params.h.orig	2019-04-12 01:00:35.000000000 +0000
 +++ src/params.h
-@@ -51,15 +51,15 @@
+@@ -75,15 +75,15 @@
   * notes above.
   */
  #ifndef JOHN_SYSTEMWIDE
@@ -21,18 +23,19 @@ $NetBSD: patch-ab,v 1.2 2010/12/13 10:32:35 wiz Exp $
  #endif
  #define JOHN_PRIVATE_HOME		"~/.john"
  #endif
-@@ -103,8 +103,8 @@
- /*
+@@ -136,9 +136,9 @@
   * File names.
   */
+ #ifdef __DJGPP__
+-#define CFG_FULL_NAME			"$JOHN/john.ini"
++#define CFG_FULL_NAME			"@PREFIX@/share/john/john.ini"
+ #else
 -#define CFG_FULL_NAME			"$JOHN/john.conf"
--#define CFG_ALT_NAME			"$JOHN/john.ini"
-+#define CFG_FULL_NAME			"@PREFIX@/share/john/john.conf"
-+#define CFG_ALT_NAME			"@PREFIX@/share/john/john.ini"
++#define CFG_FULL_NAME			"@PREFIX@/share/john/john.ini"
+ #endif
  #if JOHN_SYSTEMWIDE
- #define CFG_PRIVATE_FULL_NAME		JOHN_PRIVATE_HOME "/john.conf"
- #define CFG_PRIVATE_ALT_NAME		JOHN_PRIVATE_HOME "/john.ini"
-@@ -118,7 +118,7 @@
+ #ifdef __DJGPP__
+@@ -156,7 +156,7 @@
  #endif
  #define LOG_SUFFIX			".log"
  #define RECOVERY_SUFFIX			".rec"
diff --git a/security/john/patches/patch-src_logger.c b/security/john/patches/patch-src_logger.c
new file mode 100644
index 00000000000..72b74a89a13
--- /dev/null
+++ b/security/john/patches/patch-src_logger.c
@@ -0,0 +1,15 @@
+$NetBSD: patch-src_logger.c,v 1.1 2020/08/08 02:09:01 leot Exp $
+
+Properly define _XOPEN_SOURCE.
+
+--- src/logger.c.orig	2015-05-31 06:38:14.000000000 +0000
++++ src/logger.c
+@@ -8,7 +8,7 @@
+  * There's ABSOLUTELY NO WARRANTY, express or implied.
+  */
+ 
+-#define _XOPEN_SOURCE /* for fileno(3) and fsync(2) */
++#define _XOPEN_SOURCE 500 /* for fileno(3) and fsync(2) */
+ 
+ #define NEED_OS_FLOCK
+ #include "os.h"