v1.0

Add an ARM PMULL implementation of CRC-32.  This is based on a patch by
Jun He <jun.he@linaro.org> as well as the x86 PCLMUL implementation.

.gitignore

@@ -10,7 +10,10 @@
 /.prog-cflags
 /programs/config.h
 /benchmark
+/checksum
 /gzip
 /gunzip
+/run_tests.log
+/test_checksums
 tags
 cscope*

COPYING.LIB

@@ -1,121 +0,0 @@
-Creative Commons Legal Code
-
-CC0 1.0 Universal
-
-    CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
-    LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
-    ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
-    INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
-    REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
-    PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
-    THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
-    HEREUNDER.
-
-Statement of Purpose
-
-The laws of most jurisdictions throughout the world automatically confer
-exclusive Copyright and Related Rights (defined below) upon the creator
-and subsequent owner(s) (each and all, an "owner") of an original work of
-authorship and/or a database (each, a "Work").
-
-Certain owners wish to permanently relinquish those rights to a Work for
-the purpose of contributing to a commons of creative, cultural and
-scientific works ("Commons") that the public can reliably and without fear
-of later claims of infringement build upon, modify, incorporate in other
-works, reuse and redistribute as freely as possible in any form whatsoever
-and for any purposes, including without limitation commercial purposes.
-These owners may contribute to the Commons to promote the ideal of a free
-culture and the further production of creative, cultural and scientific
-works, or to gain reputation or greater distribution for their Work in
-part through the use and efforts of others.
-
-For these and/or other purposes and motivations, and without any
-expectation of additional consideration or compensation, the person
-associating CC0 with a Work (the "Affirmer"), to the extent that he or she
-is an owner of Copyright and Related Rights in the Work, voluntarily
-elects to apply CC0 to the Work and publicly distribute the Work under its
-terms, with knowledge of his or her Copyright and Related Rights in the
-Work and the meaning and intended legal effect of CC0 on those rights.
-
-1. Copyright and Related Rights. A Work made available under CC0 may be
-protected by copyright and related or neighboring rights ("Copyright and
-Related Rights"). Copyright and Related Rights include, but are not
-limited to, the following:
-
-  i. the right to reproduce, adapt, distribute, perform, display,
-     communicate, and translate a Work;
- ii. moral rights retained by the original author(s) and/or performer(s);
-iii. publicity and privacy rights pertaining to a person's image or
-     likeness depicted in a Work;
- iv. rights protecting against unfair competition in regards to a Work,
-     subject to the limitations in paragraph 4(a), below;
-  v. rights protecting the extraction, dissemination, use and reuse of data
-     in a Work;
- vi. database rights (such as those arising under Directive 96/9/EC of the
-     European Parliament and of the Council of 11 March 1996 on the legal
-     protection of databases, and under any national implementation
-     thereof, including any amended or successor version of such
-     directive); and
-vii. other similar, equivalent or corresponding rights throughout the
-     world based on applicable law or treaty, and any national
-     implementations thereof.
-
-2. Waiver. To the greatest extent permitted by, but not in contravention
-of, applicable law, Affirmer hereby overtly, fully, permanently,
-irrevocably and unconditionally waives, abandons, and surrenders all of
-Affirmer's Copyright and Related Rights and associated claims and causes
-of action, whether now known or unknown (including existing as well as
-future claims and causes of action), in the Work (i) in all territories
-worldwide, (ii) for the maximum duration provided by applicable law or
-treaty (including future time extensions), (iii) in any current or future
-medium and for any number of copies, and (iv) for any purpose whatsoever,
-including without limitation commercial, advertising or promotional
-purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
-member of the public at large and to the detriment of Affirmer's heirs and
-successors, fully intending that such Waiver shall not be subject to
-revocation, rescission, cancellation, termination, or any other legal or
-equitable action to disrupt the quiet enjoyment of the Work by the public
-as contemplated by Affirmer's express Statement of Purpose.
-
-3. Public License Fallback. Should any part of the Waiver for any reason
-be judged legally invalid or ineffective under applicable law, then the
-Waiver shall be preserved to the maximum extent permitted taking into
-account Affirmer's express Statement of Purpose. In addition, to the
-extent the Waiver is so judged Affirmer hereby grants to each affected
-person a royalty-free, non transferable, non sublicensable, non exclusive,
-irrevocable and unconditional license to exercise Affirmer's Copyright and
-Related Rights in the Work (i) in all territories worldwide, (ii) for the
-maximum duration provided by applicable law or treaty (including future
-time extensions), (iii) in any current or future medium and for any number
-of copies, and (iv) for any purpose whatsoever, including without
-limitation commercial, advertising or promotional purposes (the
-"License"). The License shall be deemed effective as of the date CC0 was
-applied by Affirmer to the Work. Should any part of the License for any
-reason be judged legally invalid or ineffective under applicable law, such
-partial invalidity or ineffectiveness shall not invalidate the remainder
-of the License, and in such case Affirmer hereby affirms that he or she
-will not (i) exercise any of his or her remaining Copyright and Related
-Rights in the Work or (ii) assert any associated claims and causes of
-action with respect to the Work, in either case contrary to Affirmer's
-express Statement of Purpose.
-
-4. Limitations and Disclaimers.
-
- a. No trademark or patent rights held by Affirmer are waived, abandoned,
-    surrendered, licensed or otherwise affected by this document.
- b. Affirmer offers the Work as-is and makes no representations or
-    warranties of any kind concerning the Work, express, implied,
-    statutory or otherwise, including without limitation warranties of
-    title, merchantability, fitness for a particular purpose, non
-    infringement, or the absence of latent or other defects, accuracy, or
-    the present or absence of errors, whether or not discoverable, all to
-    the greatest extent permissible under applicable law.
- c. Affirmer disclaims responsibility for clearing rights of other persons
-    that may apply to the Work or any use thereof, including without
-    limitation any person's Copyright and Related Rights in the Work.
-    Further, Affirmer disclaims responsibility for obtaining any necessary
-    consents, permissions or other rights required for any use of the
-    Work.
- d. Affirmer understands and acknowledges that Creative Commons is not a
-    party to this document and has no duty or obligation with respect to
-    this CC0 or use of the Work.

Makefile

@@ -33,23 +33,29 @@ override CFLAGS :=							\
 
 STATIC_LIB_SUFFIX := .a
 SHARED_LIB_SUFFIX := .so
+SHARED_LIB_CFLAGS := -fPIC
 PROG_SUFFIX       :=
 PROG_CFLAGS       :=
-PIC_REQUIRED      := 1
 HARD_LINKS        := 1
 
 # Compiling for Windows with MinGW?
-ifneq ($(findstring -mingw,$(CC)),)
-    ifeq ($(AR),ar)
-        AR := $(patsubst %-gcc,%-ar,$(CC))
-    endif
+ifneq ($(findstring -mingw,$(shell $(CC) -dumpmachine 2>/dev/null)),)
     STATIC_LIB_SUFFIX := .lib
     SHARED_LIB_SUFFIX := .dll
+    SHARED_LIB_CFLAGS :=
     PROG_SUFFIX       := .exe
     PROG_CFLAGS       := -static -municode
-    PIC_REQUIRED      :=
     HARD_LINKS        :=
     override CFLAGS   := $(CFLAGS) $(call cc-option,-Wno-pedantic-ms-format)
+
+    # If AR was not already overridden, then derive it from $(CC).
+    # Note that CC may take different forms, e.g. "cc", "gcc",
+    # "x86_64-w64-mingw32-gcc", or "x86_64-w64-mingw32-gcc-6.3.1".
+    # On Windows it may also have a .exe extension.
+    ifeq ($(AR),ar)
+        AR := $(shell echo $(CC) | \
+                sed -E 's/g?cc(-?[0-9]+(\.[0-9]+)*)?(\.exe)?$$/ar\3/')
+    endif
 endif
 
 ##############################################################################
@@ -70,7 +76,7 @@ endif
 ##############################################################################
 
 COMMON_HEADERS := $(wildcard common/*.h)
-ALL_TARGETS :=
+DEFAULT_TARGETS :=
 
 #### Library
 
@@ -79,9 +85,10 @@ SHARED_LIB := libdeflate$(SHARED_LIB_SUFFIX)
 
 LIB_CFLAGS += $(CFLAGS) -fvisibility=hidden -D_ANSI_SOURCE
 
-LIB_HEADERS := $(wildcard lib/*.h)
+LIB_HEADERS := $(wildcard lib/*.h) $(wildcard lib/*/*.h)
 
-LIB_SRC := lib/aligned_malloc.c lib/deflate_decompress.c lib/x86_cpu_features.c
+LIB_SRC := lib/aligned_malloc.c lib/deflate_decompress.c \
+	   $(wildcard lib/*/cpu_features.c)
 
 DECOMPRESSION_ONLY :=
 ifndef DECOMPRESSION_ONLY
@@ -104,33 +111,28 @@ ifndef DISABLE_GZIP
     endif
 endif
 
-LIB_OBJ := $(LIB_SRC:.c=.o)
-LIB_PIC_OBJ := $(LIB_SRC:.c=.pic.o)
-ifdef PIC_REQUIRED
-    SHLIB_OBJ := $(LIB_PIC_OBJ)
-else
-    SHLIB_OBJ := $(LIB_OBJ)
-endif
+STATIC_LIB_OBJ := $(LIB_SRC:.c=.o)
+SHARED_LIB_OBJ := $(LIB_SRC:.c=.shlib.o)
 
-# Compile position dependent library object files
-$(LIB_OBJ): %.o: %.c $(LIB_HEADERS) $(COMMON_HEADERS) .lib-cflags
+# Compile static library object files
+$(STATIC_LIB_OBJ): %.o: %.c $(LIB_HEADERS) $(COMMON_HEADERS) .lib-cflags
 	$(QUIET_CC) $(CC) -o $@ -c $(LIB_CFLAGS) $<
 
-# Compile position independent library object files
-$(LIB_PIC_OBJ): %.pic.o: %.c $(LIB_HEADERS) $(COMMON_HEADERS) .lib-cflags
-	$(QUIET_CC) $(CC) -o $@ -c $(LIB_CFLAGS) -fPIC $<
-
-# Link shared library
-$(SHARED_LIB):$(SHLIB_OBJ)
-	$(QUIET_CCLD) $(CC) -o $@ $(LDFLAGS) $(LIB_CFLAGS) -shared $+
-
-ALL_TARGETS += $(SHARED_LIB)
+# Compile shared library object files
+$(SHARED_LIB_OBJ): %.shlib.o: %.c $(LIB_HEADERS) $(COMMON_HEADERS) .lib-cflags
+	$(QUIET_CC) $(CC) -o $@ -c $(LIB_CFLAGS) $(SHARED_LIB_CFLAGS) -DLIBDEFLATE_DLL $<
 
 # Create static library
-$(STATIC_LIB):$(LIB_OBJ)
+$(STATIC_LIB):$(STATIC_LIB_OBJ)
 	$(QUIET_AR) $(AR) cr $@ $+
 
-ALL_TARGETS += $(STATIC_LIB)
+DEFAULT_TARGETS += $(STATIC_LIB)
+
+# Create shared library
+$(SHARED_LIB):$(SHARED_LIB_OBJ)
+	$(QUIET_CCLD) $(CC) -o $@ $(LDFLAGS) $(LIB_CFLAGS) -shared $+
+
+DEFAULT_TARGETS += $(SHARED_LIB)
 
 # Rebuild if CC or LIB_CFLAGS changed
 .lib-cflags: FORCE
@@ -144,39 +146,44 @@ ALL_TARGETS += $(STATIC_LIB)
 
 #### Programs
 
-PROG_CFLAGS += $(CFLAGS)		\
-	       -D_DEFAULT_SOURCE	\
-	       -D_FILE_OFFSET_BITS=64	\
+PROG_CFLAGS += $(CFLAGS)		 \
+	       -D_POSIX_C_SOURCE=200809L \
+	       -D_FILE_OFFSET_BITS=64	 \
 	       -DHAVE_CONFIG_H
 
 PROG_COMMON_HEADERS := programs/prog_util.h programs/config.h
-PROG_COMMON_SRC := programs/prog_util.c programs/tgetopt.c
-PROG_SPECIFIC_SRC := programs/gzip.c programs/benchmark.c
+PROG_COMMON_SRC     := programs/prog_util.c programs/tgetopt.c
+NONTEST_PROGRAM_SRC := programs/gzip.c
+TEST_PROGRAM_SRC    := programs/benchmark.c programs/test_checksums.c \
+			programs/checksum.c
 
-PROG_COMMON_OBJ := $(PROG_COMMON_SRC:.c=.o)
-PROG_SPECIFIC_OBJ := $(PROG_SPECIFIC_SRC:.c=.o)
-PROG_OBJ := $(PROG_COMMON_OBJ) $(PROG_SPECIFIC_OBJ)
+NONTEST_PROGRAMS := $(NONTEST_PROGRAM_SRC:programs/%.c=%$(PROG_SUFFIX))
+DEFAULT_TARGETS  += $(NONTEST_PROGRAMS)
+TEST_PROGRAMS    := $(TEST_PROGRAM_SRC:programs/%.c=%$(PROG_SUFFIX))
+
+PROG_COMMON_OBJ     := $(PROG_COMMON_SRC:%.c=%.o)
+NONTEST_PROGRAM_OBJ := $(NONTEST_PROGRAM_SRC:%.c=%.o)
+TEST_PROGRAM_OBJ    := $(TEST_PROGRAM_SRC:%.c=%.o)
+PROG_OBJ := $(PROG_COMMON_OBJ) $(NONTEST_PROGRAM_OBJ) $(TEST_PROGRAM_OBJ)
 
 # Generate autodetected configuration header
 programs/config.h:programs/detect.sh .prog-cflags
-	$(QUIET_GEN) CC=$(CC) $< > $@
+	$(QUIET_GEN) CC="$(CC)" CFLAGS="$(PROG_CFLAGS)" $< > $@
 
 # Compile program object files
 $(PROG_OBJ): %.o: %.c $(PROG_COMMON_HEADERS) $(COMMON_HEADERS) .prog-cflags
 	$(QUIET_CC) $(CC) -o $@ -c $(PROG_CFLAGS) $<
 
-# Link benchmark program
-benchmark$(PROG_SUFFIX):programs/benchmark.o $(PROG_COMMON_OBJ) $(STATIC_LIB)
-	$(QUIET_CCLD) $(CC) -o $@ $(LDFLAGS) $(PROG_CFLAGS) $+ -lz
+# Link the programs.
+#
+# Note: the test programs are not compiled by default.  One reason is that the
+# test programs must be linked with zlib for doing comparisons.
 
-# 'benchmark' is not compiled by default because of the zlib link requirement.
-#ALL_TARGETS += benchmark$(PROG_SUFFIX)
-
-# Link gzip program
-gzip$(PROG_SUFFIX):programs/gzip.o $(PROG_COMMON_OBJ) $(STATIC_LIB)
+$(NONTEST_PROGRAMS): %$(PROG_SUFFIX): programs/%.o $(PROG_COMMON_OBJ) $(STATIC_LIB)
 	$(QUIET_CCLD) $(CC) -o $@ $(LDFLAGS) $(PROG_CFLAGS) $+
 
-ALL_TARGETS += gzip$(PROG_SUFFIX)
+$(TEST_PROGRAMS): %$(PROG_SUFFIX): programs/%.o $(PROG_COMMON_OBJ) $(STATIC_LIB)
+	$(QUIET_CCLD) $(CC) -o $@ $(LDFLAGS) $(PROG_CFLAGS) $+ -lz
 
 ifdef HARD_LINKS
 # Hard link gunzip to gzip
@@ -188,7 +195,7 @@ gunzip$(PROG_SUFFIX):gzip$(PROG_SUFFIX)
 	$(QUIET_CP) cp -f $< $@
 endif
 
-ALL_TARGETS += gunzip$(PROG_SUFFIX)
+DEFAULT_TARGETS += gunzip$(PROG_SUFFIX)
 
 # Rebuild if CC or PROG_CFLAGS changed
 .prog-cflags: FORCE
@@ -200,27 +207,32 @@ ALL_TARGETS += gunzip$(PROG_SUFFIX)
 
 ##############################################################################
 
-all:$(ALL_TARGETS)
+all:$(DEFAULT_TARGETS)
+
+test_programs:$(TEST_PROGRAMS)
 
 help:
 	@echo "Available targets:"
 	@echo "------------------"
-	@for target in $(ALL_TARGETS) benchmark$(PROG_SUFFIX); do \
+	@for target in $(DEFAULT_TARGETS) $(TEST_PROGRAMS); do \
 		echo -e "$$target";		\
 	done
 
 clean:
 	rm -f *.a *.dll *.exe *.exp *.so \
-		lib/*.o lib/*.obj programs/*.o programs/*.obj \
-		benchmark gzip gunzip programs/config.h \
+		lib/*.o lib/*/*.o \
+		lib/*.obj lib/*/*.obj \
+		lib/*.dllobj lib/*/*.dllobj \
+		programs/*.o programs/*.obj \
+		$(DEFAULT_TARGETS) $(TEST_PROGRAMS) programs/config.h \
 		libdeflate.lib libdeflatestatic.lib \
 		.lib-cflags .prog-cflags
 
 realclean: clean
-	rm -f tags cscope*
+	rm -f tags cscope* run_tests.log
 
 FORCE:
 
-.PHONY: all help clean realclean
+.PHONY: all test_programs help clean realclean
 
 .DEFAULT_GOAL = all

Makefile.msc

@@ -5,45 +5,53 @@
 #	nmake /f Makefile.msc
 #
 
+.SUFFIXES: .c .obj .dllobj
+
 CC = cl
 LD = link
 AR = lib
 CFLAGS = /MD /O2 -I. -Icommon
 LDFLAGS =
 
-STATICLIB = libdeflatestatic.lib
-SHAREDLIB = libdeflate.dll
-IMPLIB    = libdeflate.lib
+STATIC_LIB = libdeflatestatic.lib
+SHARED_LIB = libdeflate.dll
+IMPORT_LIB = libdeflate.lib
 
-LIB_OBJ = lib/aligned_malloc.obj	\
+STATIC_LIB_OBJ =			\
+	  lib/aligned_malloc.obj	\
 	  lib/adler32.obj		\
 	  lib/crc32.obj			\
 	  lib/deflate_compress.obj	\
 	  lib/deflate_decompress.obj	\
 	  lib/gzip_compress.obj		\
 	  lib/gzip_decompress.obj	\
-	  lib/x86_cpu_features.obj	\
+	  lib/x86/cpu_features.obj	\
 	  lib/zlib_compress.obj		\
 	  lib/zlib_decompress.obj
 
+SHARED_LIB_OBJ = $(STATIC_LIB_OBJ:.obj=.dllobj)
+
 PROG_COMMON_OBJ = programs/prog_util.obj \
 		  programs/tgetopt.obj \
-		  $(STATICLIB)
+		  $(STATIC_LIB)
 
 PROG_CFLAGS = $(CFLAGS) -Iprograms
 
-all: $(STATICLIB) $(SHAREDLIB) $(IMPLIB) gzip.exe gunzip.exe
+all: $(STATIC_LIB) $(SHARED_LIB) $(IMPORT_LIB) gzip.exe gunzip.exe
 
 .c.obj:
 	$(CC) -c /Fo$@ $(CFLAGS) $**
 
-$(STATICLIB): $(LIB_OBJ)
-	$(AR) $(ARFLAGS) -out:$@ $(LIB_OBJ)
+.c.dllobj:
+	$(CC) -c /Fo$@ $(CFLAGS) /DLIBDEFLATE_DLL $**
 
-$(SHAREDLIB): $(LIB_OBJ)
-	$(LD) $(LDFLAGS) -out:$@ -dll -implib:$(IMPLIB) $(LIB_OBJ)
+$(STATIC_LIB): $(STATIC_LIB_OBJ)
+	$(AR) $(ARFLAGS) -out:$@ $(STATIC_LIB_OBJ)
 
-$(IMPLIB): $(SHAREDLIB)
+$(SHARED_LIB): $(SHARED_LIB_OBJ)
+	$(LD) $(LDFLAGS) -out:$@ -dll -implib:$(IMPORT_LIB) $(SHARED_LIB_OBJ)
+
+$(IMPORT_LIB): $(SHARED_LIB)
 
 gzip.exe:programs/gzip.obj $(PROG_COMMON_OBJ)
 	$(LD) $(LDFLAGS) -out:$@ $**
@@ -53,4 +61,5 @@ gunzip.exe:gzip.exe
 
 clean:
 	-del *.dll *.exe *.exp libdeflate.lib libdeflatestatic.lib gzip.lib \
-		lib\*.obj programs\*.obj 2>nul
+		lib\*.obj lib\*\*.obj lib\*.dllobj lib\*\*.dllobj \
+		programs\*.obj 2>nul

NEWS

@@ -1,3 +1,68 @@
+Version 1.0:
+	Added support for multi-member gzip files.
+
+	Moved architecture-specific code into subdirectories.  If you
+	aren't using the provided Makefile to build libdeflate, you now
+	need to compile lib/*.c and lib/*/*.c instead of just lib/*.c.
+
+	Added an ARM PMULL implementation of CRC-32, which speeds up
+	gzip compression and decompression on 32-bit and 64-bit ARM
+	processors that have the Cryptography Extensions.
+
+	Improved detection of CPU features, resulting in accelerated
+	functions being used in more cases.  This includes:
+
+	   - Detect CPU features on 32-bit x86, not just 64-bit as was
+	     done previously.
+
+	   - Detect CPU features on ARM, both 32 and 64-bit.
+	     (Limited to Linux only currently.)
+
+Version 0.8:
+	Build fixes for certain platforms and compilers.
+
+	libdeflate now produces the same output on all CPU architectures.
+
+	Improved documentation for building libdeflate on Windows.
+
+Version 0.7:
+	Fixed a very rare bug that caused data to be compressed incorrectly.
+	The bug affected compression levels 7 and below since libdeflate v0.2.
+	Although there have been no user reports of the bug, and I believe it
+	would have been highly unlikely to encounter on realistic data, it could
+	occur on data specially crafted to reproduce it.
+
+	Fixed a compilation error when building with clang 3.7.
+
+Version 0.6:
+	Various improvements to the gzip program's behavior.
+
+	Faster CRC-32 on AVX-capable processors.
+
+	Other minor changes.
+
+Version 0.5:
+	The CRC-32 checksum algorithm has been optimized with carryless
+	multiplication instructions for x86_64 (PCLMUL).  This speeds up gzip
+	compression and decompression.
+
+	Build fixes for certain platforms and compilers.
+
+	Added more test programs and scripts.
+
+	libdeflate is now entirely MIT-licensed.
+
+Version 0.4:
+	The Adler-32 checksum algorithm has been optimized with vector
+	instructions for x86_64 (SSE2 and AVX2) and ARM (NEON).  This speeds up
+	zlib compression and decompression.
+
+	To avoid naming collisions, functions and definitions in libdeflate's
+	API have been renamed to be prefixed with "libdeflate_" or
+	"LIBDEFLATE_".  Programs using the old API will need to be updated.
+
+	Various bug fixes and other improvements.
+
 Version 0.3:
 	Some bug fixes and other minor changes.
 

README.md

@@ -18,7 +18,7 @@ libdeflate itself is a library, but the following command-line programs which
 use this library are also provided:
 
 * gzip (or gunzip), a program which mostly behaves like the standard equivalent,
-  except that it does not yet support reading from standard input and does not
+  except that it does not yet have good streaming support and therefore does not
   yet support very large files
 * benchmark, a program for benchmarking in-memory compression and decompression
 
@@ -37,20 +37,100 @@ the Makefile for details.
 
 ## For Windows
 
-MinGW (GCC) is the recommended compiler to use when building binaries for
-Windows.  MinGW can be used on either Windows or Linux.  Use a command like:
+Prebuilt Windows binaries can be downloaded from
+https://github.com/ebiggers/libdeflate/releases.  But if you need to build the
+binaries yourself, MinGW (gcc) is the recommended compiler to use.  If you're
+performing the build *on* Windows (as opposed to cross-compiling for Windows on
+Linux, for example), you'll need to follow the directions in **one** of the two
+sections below to set up a minimal UNIX-compatible environment using either
+Cygwin or MSYS2, then do the build.  (Other MinGW distributions may not work, as
+they often omit basic UNIX tools such as `sh`.)
 
-    $ make CC=x86_64-w64-mingw32-gcc
+Alternatively, libdeflate may be built using the Visual Studio toolchain by
+running `nmake /f Makefile.msc`.  However, while this is supported in the sense
+that it will produce working binaries, it is not recommended because the
+binaries built with MinGW will be significantly faster.
 
-Windows binaries prebuilt with MinGW may also be downloaded from
-https://github.com/ebiggers/libdeflate/releases.
-
-Alternatively, a separate Makefile, `Makefile.msc`, is provided for the tools
-that come with Visual Studio, for those who strongly prefer that toolchain.
-
-As usual, 64-bit binaries are faster than 32-bit binaries and should be
+Also note that 64-bit binaries are faster than 32-bit binaries and should be
 preferred whenever possible.
 
+### Using Cygwin
+
+Run the Cygwin installer, available from https://cygwin.com/setup-x86_64.exe.
+When you get to the package selection screen, choose the following additional
+packages from category "Devel":
+
+- git
+- make
+- mingw64-i686-binutils
+- mingw64-i686-gcc-g++
+- mingw64-x86_64-binutils
+- mingw64-x86_64-gcc-g++
+
+(You may skip the mingw64-i686 packages if you don't need to build 32-bit
+binaries.)
+
+After the installation finishes, open a Cygwin terminal.  Then download
+libdeflate's source code (if you haven't already) and `cd` into its directory:
+
+    git clone https://github.com/ebiggers/libdeflate
+    cd libdeflate
+
+(Note that it's not required to use `git`; an alternative is to extract a .zip
+or .tar.gz archive of the source code downloaded from the releases page.
+Also, in case you need to find it in the file browser, note that your home
+directory in Cygwin is usually located at `C:\cygwin64\home\<your username>`.)
+
+Then, to build 64-bit binaries:
+
+    make CC=x86_64-w64-mingw32-gcc
+
+or to build 32-bit binaries:
+
+    make CC=i686-w64-mingw32-gcc
+
+### Using MSYS2
+
+Run the MSYS2 installer, available from http://www.msys2.org/.  After
+installing, open an MSYS2 shell and run:
+
+    pacman -Syu
+
+Say `y`, then when it's finished, close the shell window and open a new one.
+Then run the same command again:
+
+    pacman -Syu
+
+Then, install the packages needed to build libdeflate:
+
+    pacman -S git \
+              make \
+              mingw-w64-i686-binutils \
+              mingw-w64-i686-gcc \
+              mingw-w64-x86_64-binutils \
+              mingw-w64-x86_64-gcc
+
+(You may skip the mingw-w64-i686 packages if you don't need to build 32-bit
+binaries.)
+
+Then download libdeflate's source code (if you haven't already):
+
+    git clone https://github.com/ebiggers/libdeflate
+
+(Note that it's not required to use `git`; an alternative is to extract a .zip
+or .tar.gz archive of the source code downloaded from the releases page.
+Also, in case you need to find it in the file browser, note that your home
+directory in MSYS2 is usually located at `C:\msys64\home\<your username>`.)
+
+Then, to build 64-bit binaries, open "MSYS2 MinGW 64-bit" from the Start menu
+and run the following commands:
+
+    cd libdeflate
+    make clean
+    make
+
+Or to build 32-bit binaries, do the same but use "MSYS2 MinGW 32-bit" instead.
+
 # API
 
 libdeflate has a simple API that is not zlib-compatible.  You can create
@@ -150,10 +230,15 @@ compression format.
 
 # License
 
-The library portion of libdeflate has been released into the public domain.
-There is NO WARRANTY, to the extent permitted by law.  See the CC0 Public Domain
-Dedication in the COPYING.LIB file for details.
+libdeflate is [MIT-licensed](COPYING).
 
-The non-library portions of the project, such as the gzip program, may be
-modified and/or redistributed under the terms of the MIT license.  There is NO
-WARRANTY, to the extent permitted by law.  See the COPYING file for details.
+Additional notes (informational only):
+
+- I am not aware of any patents covering libdeflate.
+
+- Old versions of libdeflate were public domain; I only started copyrighting
+  changes in newer versions.  Portions of the source code that have not been
+  changed since being released in a public domain version can theoretically
+  still be used as public domain if you want to.  But for practical purposes, it
+  probably would be easier to just take the MIT license option, which is nearly
+  the same anyway.

common/common_defs.h

@@ -76,7 +76,7 @@ typedef size_t machine_word_t;
 /*                         Optional compiler features                         */
 /* ========================================================================== */
 
-/* LIBEXPORT - annotate a function that is part of the library API */
+/* LIBEXPORT - export a function from a shared library */
 #ifndef LIBEXPORT
 #  define LIBEXPORT
 #endif
@@ -121,10 +121,33 @@ typedef size_t machine_word_t;
 #  define COMPILER_SUPPORTS_TARGET_FUNCTION_ATTRIBUTE 0
 #endif
 
-/* Does the compiler support __attribute__((target("bmi2")))? */
+/* Which targets are supported with the 'target' function attribute? */
 #ifndef COMPILER_SUPPORTS_BMI2_TARGET
 #  define COMPILER_SUPPORTS_BMI2_TARGET 0
 #endif
+#ifndef COMPILER_SUPPORTS_AVX_TARGET
+#  define COMPILER_SUPPORTS_AVX_TARGET 0
+#endif
+
+/*
+ * Which targets are supported with the 'target' function attribute and have
+ * intrinsics that work within 'target'-ed functions?
+ */
+#ifndef COMPILER_SUPPORTS_SSE2_TARGET_INTRINSICS
+#  define COMPILER_SUPPORTS_SSE2_TARGET_INTRINSICS 0
+#endif
+#ifndef COMPILER_SUPPORTS_PCLMUL_TARGET_INTRINSICS
+#  define COMPILER_SUPPORTS_PCLMUL_TARGET_INTRINSICS 0
+#endif
+#ifndef COMPILER_SUPPORTS_AVX2_TARGET_INTRINSICS
+#  define COMPILER_SUPPORTS_AVX2_TARGET_INTRINSICS 0
+#endif
+#ifndef COMPILER_SUPPORTS_NEON_TARGET_INTRINSICS
+#  define COMPILER_SUPPORTS_NEON_TARGET_INTRINSICS 0
+#endif
+#ifndef COMPILER_SUPPORTS_PMULL_TARGET_INTRINSICS
+#  define COMPILER_SUPPORTS_PMULL_TARGET_INTRINSICS 0
+#endif
 
 /* _aligned_attribute(n) - declare that the annotated variable, or variables of
  * the annotated type, are to be aligned on n-byte boundaries */
@@ -140,6 +163,7 @@ typedef size_t machine_word_t;
 #define MAX(a, b)		((a) >= (b) ? (a) : (b))
 #define DIV_ROUND_UP(n, d)	(((n) + (d) - 1) / (d))
 #define STATIC_ASSERT(expr)	((void)sizeof(char[1 - 2 * !(expr)]))
+#define ALIGN(n, a)		(((n) + (a) - 1) & ~((a) - 1))
 
 /* ========================================================================== */
 /*                           Endianness handling                              */

common/compiler_gcc.h

@@ -1,12 +1,33 @@
 /*
- * compiler_gcc.h - definitions for the GNU C Compiler.  Currently this also
- * handles clang and the Intel C Compiler.
+ * compiler_gcc.h - definitions for the GNU C Compiler.  This also handles clang
+ * and the Intel C Compiler (icc).
+ *
+ * TODO: icc is not well tested, so some things are currently disabled even
+ * though they maybe can be enabled on some icc versions.
  */
 
-#define GCC_PREREQ(major, minor)					\
-	(!defined(__clang__) && !defined(__INTEL_COMPILER) &&		\
-	 (__GNUC__ > (major) ||						\
-	  (__GNUC__ == (major) && __GNUC_MINOR__ >= (minor))))
+#if !defined(__clang__) && !defined(__INTEL_COMPILER)
+#  define GCC_PREREQ(major, minor)		\
+	(__GNUC__ > (major) ||			\
+	 (__GNUC__ == (major) && __GNUC_MINOR__ >= (minor)))
+#else
+#  define GCC_PREREQ(major, minor)	0
+#endif
+
+/* Note: only check the clang version when absolutely necessary!
+ * "Vendors" such as Apple can use different version numbers. */
+#ifdef __clang__
+#  ifdef __apple_build_version__
+#    define CLANG_PREREQ(major, minor, apple_version)	\
+	(__apple_build_version__ >= (apple_version))
+#  else
+#    define CLANG_PREREQ(major, minor, apple_version)	\
+	(__clang_major__ > (major) ||			\
+	 (__clang_major__ == (major) && __clang_minor__ >= (minor)))
+#  endif
+#else
+#  define CLANG_PREREQ(major, minor, apple_version)	0
+#endif
 
 #ifndef __has_attribute
 #  define __has_attribute(attribute)	0
@@ -33,12 +54,57 @@
 #define prefetchw(addr)		__builtin_prefetch((addr), 1)
 #define _aligned_attribute(n)	__attribute__((aligned(n)))
 
-#define COMPILER_SUPPORTS_TARGET_FUNCTION_ATTRIBUTE		\
+#define COMPILER_SUPPORTS_TARGET_FUNCTION_ATTRIBUTE	\
 	(GCC_PREREQ(4, 4) || __has_attribute(target))
 
-#define COMPILER_SUPPORTS_BMI2_TARGET				\
-	(COMPILER_SUPPORTS_TARGET_FUNCTION_ATTRIBUTE &&		\
-	 (GCC_PREREQ(4, 7) || __has_builtin(__builtin_ia32_pdep_di)))
+#if COMPILER_SUPPORTS_TARGET_FUNCTION_ATTRIBUTE
+
+#  if defined(__i386__) || defined(__x86_64__)
+
+#    define COMPILER_SUPPORTS_PCLMUL_TARGET	\
+	(GCC_PREREQ(4, 4) || __has_builtin(__builtin_ia32_pclmulqdq128))
+
+#    define COMPILER_SUPPORTS_AVX_TARGET	\
+	(GCC_PREREQ(4, 6) || __has_builtin(__builtin_ia32_maxps256))
+
+#    define COMPILER_SUPPORTS_BMI2_TARGET	\
+	(GCC_PREREQ(4, 7) || __has_builtin(__builtin_ia32_pdep_di))
+
+#    define COMPILER_SUPPORTS_AVX2_TARGET	\
+	(GCC_PREREQ(4, 7) || __has_builtin(__builtin_ia32_pmaddwd256))
+
+	/*
+	 * Prior to gcc 4.9 (r200349) and clang 3.8 (r239883), x86 intrinsics
+	 * not available in the main target could not be used in 'target'
+	 * attribute functions.  Unfortunately clang has no feature test macro
+	 * for this so we have to check its version.
+	 */
+#    if GCC_PREREQ(4, 9) || CLANG_PREREQ(3, 8, 7030000)
+#      define COMPILER_SUPPORTS_SSE2_TARGET_INTRINSICS	1
+#      define COMPILER_SUPPORTS_PCLMUL_TARGET_INTRINSICS	\
+		COMPILER_SUPPORTS_PCLMUL_TARGET
+#      define COMPILER_SUPPORTS_AVX2_TARGET_INTRINSICS	\
+		COMPILER_SUPPORTS_AVX2_TARGET
+#    endif
+#  elif defined(__arm__) || defined(__aarch64__)
+	/*
+	 * Prior to gcc 6.1 (r230411 for arm, r226563 for aarch64), NEON
+	 * and crypto intrinsics not available in the main target could not be
+	 * used in 'target' attribute functions.
+	 *
+	 * clang as of 5.0.1 still doesn't allow it.  But, it does seem to allow
+	 * the pmull intrinsics if only __ARM_NEON is enabled.
+	 */
+#    define COMPILER_SUPPORTS_NEON_TARGET_INTRINSICS	GCC_PREREQ(6, 1)
+#    ifdef __ARM_NEON
+#      define COMPILER_SUPPORTS_PMULL_TARGET_INTRINSICS	\
+		(GCC_PREREQ(6, 1) || __has_builtin(__builtin_neon_vmull_p64))
+#    else
+#      define COMPILER_SUPPORTS_PMULL_TARGET_INTRINSICS	\
+		(GCC_PREREQ(6, 1))
+#    endif
+#  endif
+#endif /* COMPILER_SUPPORTS_TARGET_FUNCTION_ATTRIBUTE */
 
 /* Newer gcc supports __BYTE_ORDER__.  Older gcc doesn't. */
 #ifdef __BYTE_ORDER__

common/compiler_msc.h

@@ -2,8 +2,6 @@
  * compiler_msc.h - definitions for the Microsoft C Compiler
  */
 
-#define BUILDING_LIBDEFLATE
-
 #define LIBEXPORT	__declspec(dllexport)
 
 /*

lib/adler32.c

@@ -1,26 +1,38 @@
 /*
  * adler32.c - Adler-32 checksum algorithm
  *
- * Written in 2014-2015 by Eric Biggers <ebiggers3@gmail.com>
+ * Copyright 2016 Eric Biggers
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  */
 
-#include "adler32.h"
+#include "lib_common.h"
+#include "libdeflate.h"
 
-/*
- * The Adler-32 divisor, or "base", value.
- */
+/* The Adler-32 divisor, or "base", value. */
 #define DIVISOR 65521
 
 /*
- * MAX_BYTES_PER_CHUNK is the most bytes that can be processed without the
+ * MAX_CHUNK_SIZE is the most bytes that can be processed without the
  * possibility of s2 overflowing when it is represented as an unsigned 32-bit
  * integer.  This value was computed using the following Python script:
  *
@@ -40,58 +52,43 @@
  * has value 0xFF and that s1 and s2 started with the highest possible values
  * modulo the divisor.
  */
-#define MAX_BYTES_PER_CHUNK	5552
+#define MAX_CHUNK_SIZE	5552
 
-/* Number of bytes to process per loop iteration  */
-#define UNROLL_FACTOR	4
+typedef u32 (*adler32_func_t)(u32, const u8 *, size_t);
 
-u32
-adler32(const void *buffer, size_t size)
+/* Include architecture-specific implementations if available */
+#undef DEFAULT_IMPL
+#undef DISPATCH
+#if defined(__arm__) || defined(__aarch64__)
+#  include "arm/adler32_impl.h"
+#elif defined(__i386__) || defined(__x86_64__)
+#  include "x86/adler32_impl.h"
+#endif
+
+/* Define a generic implementation if needed */
+#ifndef DEFAULT_IMPL
+#define DEFAULT_IMPL adler32_generic
+static u32 adler32_generic(u32 adler, const u8 *p, size_t size)
 {
-	u32 s1 = 1;
-	u32 s2 = 0;
-	const u8 *p = buffer;
-	const u8 *end = p + size;
+	u32 s1 = adler & 0xFFFF;
+	u32 s2 = adler >> 16;
+	const u8 * const end = p + size;
+
 	while (p != end) {
-		size_t chunk_size = MIN(end - p, MAX_BYTES_PER_CHUNK);
+		size_t chunk_size = MIN(end - p, MAX_CHUNK_SIZE);
 		const u8 *chunk_end = p + chunk_size;
+		size_t num_unrolled_iterations = chunk_size / 4;
 
-	#if UNROLL_FACTOR > 1
-		size_t num_unrolled_iterations = chunk_size / UNROLL_FACTOR;
 		while (num_unrolled_iterations--) {
 			s1 += *p++;
 			s2 += s1;
-		#if UNROLL_FACTOR >= 2
 			s1 += *p++;
 			s2 += s1;
-		#endif
-		#if UNROLL_FACTOR >= 3
 			s1 += *p++;
 			s2 += s1;
-		#endif
-		#if UNROLL_FACTOR >= 4
 			s1 += *p++;
 			s2 += s1;
-		#endif
-		#if UNROLL_FACTOR >= 5
-			s1 += *p++;
-			s2 += s1;
-		#endif
-		#if UNROLL_FACTOR >= 6
-			s1 += *p++;
-			s2 += s1;
-		#endif
-		#if UNROLL_FACTOR >= 7
-			s1 += *p++;
-			s2 += s1;
-		#endif
-		#if UNROLL_FACTOR >= 8
-			s1 += *p++;
-			s2 += s1;
-		#endif
-			STATIC_ASSERT(UNROLL_FACTOR <= 8);
 		}
-	#endif /* UNROLL_FACTOR > 1 */
 		while (p != chunk_end) {
 			s1 += *p++;
 			s2 += s1;
@@ -99,5 +96,35 @@ adler32(const void *buffer, size_t size)
 		s1 %= DIVISOR;
 		s2 %= DIVISOR;
 	}
+
 	return (s2 << 16) | s1;
 }
+#endif /* !DEFAULT_IMPL */
+
+#ifdef DISPATCH
+static u32 dispatch(u32, const u8 *, size_t);
+
+static volatile adler32_func_t adler32_impl = dispatch;
+
+/* Choose the fastest implementation at runtime */
+static u32 dispatch(u32 adler, const u8 *buffer, size_t size)
+{
+	adler32_func_t f = arch_select_adler32_func();
+
+	if (f == NULL)
+		f = DEFAULT_IMPL;
+
+	adler32_impl = f;
+	return adler32_impl(adler, buffer, size);
+}
+#else
+#  define adler32_impl DEFAULT_IMPL /* only one implementation, use it */
+#endif
+
+LIBDEFLATEAPI u32
+libdeflate_adler32(u32 adler, const void *buffer, size_t size)
+{
+	if (buffer == NULL) /* return initial value */
+		return 1;
+	return adler32_impl(adler, buffer, size);
+}

lib/adler32.h

@@ -1,12 +0,0 @@
-/*
- * adler32.h - Adler-32 checksum algorithm
- */
-
-#ifndef _LIB_ADLER32_H
-#define _LIB_ADLER32_H
-
-#include "common_defs.h"
-
-extern u32 adler32(const void *buffer, size_t size);
-
-#endif /* _LIB_ADLER32_H */

lib/adler32_vec_template.h

@@ -0,0 +1,124 @@
+/*
+ * adler32_vec_template.h - template for vectorized Adler-32 implementations
+ *
+ * Copyright 2016 Eric Biggers
+ *
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+/*
+ * This file contains a template for vectorized Adler-32 implementations.
+ *
+ * The inner loop between reductions modulo 65521 of an unvectorized Adler-32
+ * implementation looks something like this:
+ *
+ *	do {
+ *		s1 += *p;
+ *		s2 += s1;
+ *	} while (++p != chunk_end);
+ *
+ * For vectorized calculation of s1, we only need to sum the input bytes.  They
+ * can be accumulated into multiple counters which are eventually summed
+ * together.
+ *
+ * For vectorized calculation of s2, the basic idea is that for each iteration
+ * that processes N bytes, we can perform the following vectorizable
+ * calculation:
+ *
+ *	s2 += N*byte_1 + (N-1)*byte_2 + (N-2)*byte_3 + ... + 1*byte_N
+ *
+ * Or, equivalently, we can sum the byte_1...byte_N for each iteration into N
+ * separate counters, then do the multiplications by N...1 just once at the end
+ * rather than once per iteration.
+ *
+ * Also, we must account for how previous bytes will affect s2 by doing the
+ * following at beginning of each iteration:
+ *
+ *	s2 += s1 * N
+ *
+ * Furthermore, like s1, "s2" can actually be multiple counters which are
+ * eventually summed together.
+ */
+
+static u32 ATTRIBUTES
+FUNCNAME(u32 adler, const u8 *p, size_t size)
+{
+	u32 s1 = adler & 0xFFFF;
+	u32 s2 = adler >> 16;
+	const u8 * const end = p + size;
+	const u8 *vend;
+	const size_t max_chunk_size =
+		MIN(MAX_CHUNK_SIZE, IMPL_MAX_CHUNK_SIZE) -
+		(MIN(MAX_CHUNK_SIZE, IMPL_MAX_CHUNK_SIZE) %
+		 IMPL_SEGMENT_SIZE);
+
+	/* Process a byte at a time until the needed alignment is reached */
+	if (p != end && (uintptr_t)p % IMPL_ALIGNMENT) {
+		do {
+			s1 += *p++;
+			s2 += s1;
+		} while (p != end && (uintptr_t)p % IMPL_ALIGNMENT);
+		s1 %= DIVISOR;
+		s2 %= DIVISOR;
+	}
+
+	/*
+	 * Process "chunks" of bytes using vector instructions.  Chunk sizes are
+	 * limited to MAX_CHUNK_SIZE, which guarantees that s1 and s2 never
+	 * overflow before being reduced modulo DIVISOR.  For vector processing,
+	 * chunk sizes are also made evenly divisible by IMPL_SEGMENT_SIZE and
+	 * may be further limited to IMPL_MAX_CHUNK_SIZE.
+	 */
+	STATIC_ASSERT(IMPL_SEGMENT_SIZE % IMPL_ALIGNMENT == 0);
+	vend = end - ((size_t)(end - p) % IMPL_SEGMENT_SIZE);
+	while (p != vend) {
+		size_t chunk_size = MIN((size_t)(vend - p), max_chunk_size);
+
+		s2 += s1 * chunk_size;
+
+		FUNCNAME_CHUNK((const void *)p, (const void *)(p + chunk_size),
+			       &s1, &s2);
+
+		p += chunk_size;
+		s1 %= DIVISOR;
+		s2 %= DIVISOR;
+	}
+
+	/* Process any remaining bytes */
+	if (p != end) {
+		do {
+			s1 += *p++;
+			s2 += s1;
+		} while (p != end);
+		s1 %= DIVISOR;
+		s2 %= DIVISOR;
+	}
+
+	return (s2 << 16) | s1;
+}
+
+#undef FUNCNAME
+#undef FUNCNAME_CHUNK
+#undef ATTRIBUTES
+#undef IMPL_ALIGNMENT
+#undef IMPL_SEGMENT_SIZE
+#undef IMPL_MAX_CHUNK_SIZE

lib/aligned_malloc.c

@@ -1,15 +1,30 @@
 /*
  * aligned_malloc.c - aligned memory allocation
  *
- * Written in 2014-2015 by Eric Biggers <ebiggers3@gmail.com>
+ * Originally public domain; changes after 2016-09-07 are copyrighted.
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Copyright 2016 Eric Biggers
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  */
 
 /*
@@ -25,15 +40,11 @@
 void *
 aligned_malloc(size_t alignment, size_t size)
 {
-	const uintptr_t mask = alignment - 1;
-	char *ptr = NULL;
-	char *raw_ptr;
-
-	raw_ptr = malloc(mask + sizeof(size_t) + size);
-	if (raw_ptr) {
-		ptr = (char *)raw_ptr + sizeof(size_t);
-		ptr = (void *)(((uintptr_t)ptr + mask) & ~mask);
-		*((size_t *)ptr - 1) = ptr - raw_ptr;
+	void *ptr = malloc(sizeof(void *) + alignment - 1 + size);
+	if (ptr) {
+		void *orig_ptr = ptr;
+		ptr = (void *)ALIGN((uintptr_t)ptr + sizeof(void *), alignment);
+		((void **)ptr)[-1] = orig_ptr;
 	}
 	return ptr;
 }
@@ -42,5 +53,5 @@ void
 aligned_free(void *ptr)
 {
 	if (ptr)
-		free((char *)ptr - *((size_t *)ptr - 1));
+		free(((void **)ptr)[-1]);
 }

lib/aligned_malloc.h

@@ -2,12 +2,12 @@
  * aligned_malloc.c - aligned memory allocation
  */
 
-#ifndef _LIB_ALIGNED_MALLOC_H
-#define _LIB_ALIGNED_MALLOC_H
+#ifndef LIB_ALIGNED_MALLOC_H
+#define LIB_ALIGNED_MALLOC_H
 
-#include "common_defs.h"
+#include "lib_common.h"
 
 extern void *aligned_malloc(size_t alignment, size_t size);
 extern void aligned_free(void *ptr);
 
-#endif /* _LIB_ALIGNED_MALLOC_H */
+#endif /* LIB_ALIGNED_MALLOC_H */

lib/arm/adler32_impl.h

@@ -0,0 +1,120 @@
+/*
+ * arm/adler32_impl.h - ARM implementations of Adler-32 checksum algorithm
+ *
+ * Copyright 2016 Eric Biggers
+ *
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include "cpu_features.h"
+
+/* NEON implementation */
+#undef DISPATCH_NEON
+#if !defined(DEFAULT_IMPL) &&	\
+	(defined(__ARM_NEON) || (ARM_CPU_FEATURES_ENABLED &&	\
+				 COMPILER_SUPPORTS_NEON_TARGET_INTRINSICS))
+#  define FUNCNAME		adler32_neon
+#  define FUNCNAME_CHUNK	adler32_neon_chunk
+#  define IMPL_ALIGNMENT	16
+#  define IMPL_SEGMENT_SIZE	32
+/* Prevent unsigned overflow of the 16-bit precision byte counters */
+#  define IMPL_MAX_CHUNK_SIZE	(32 * (0xFFFF / 0xFF))
+#  ifdef __ARM_NEON
+#    define ATTRIBUTES
+#    define DEFAULT_IMPL	adler32_neon
+#  else
+#    ifdef __arm__
+#      define ATTRIBUTES	__attribute__((target("fpu=neon")))
+#    else
+#      define ATTRIBUTES	__attribute__((target("+simd")))
+#    endif
+#    define DISPATCH		1
+#    define DISPATCH_NEON	1
+#  endif
+#  include <arm_neon.h>
+static forceinline ATTRIBUTES void
+adler32_neon_chunk(const uint8x16_t *p, const uint8x16_t * const end,
+		   u32 *s1, u32 *s2)
+{
+	uint32x4_t v_s1 = (uint32x4_t) { 0, 0, 0, 0 };
+	uint32x4_t v_s2 = (uint32x4_t) { 0, 0, 0, 0 };
+	uint16x8_t v_byte_sums_a = (uint16x8_t) { 0, 0, 0, 0, 0, 0, 0, 0 };
+	uint16x8_t v_byte_sums_b = (uint16x8_t) { 0, 0, 0, 0, 0, 0, 0, 0 };
+	uint16x8_t v_byte_sums_c = (uint16x8_t) { 0, 0, 0, 0, 0, 0, 0, 0 };
+	uint16x8_t v_byte_sums_d = (uint16x8_t) { 0, 0, 0, 0, 0, 0, 0, 0 };
+
+	do {
+		const uint8x16_t bytes1 = *p++;
+		const uint8x16_t bytes2 = *p++;
+		uint16x8_t tmp;
+
+		v_s2 += v_s1;
+
+		/* Vector Pairwise Add Long (u8 => u16) */
+		tmp = vpaddlq_u8(bytes1);
+
+		/* Vector Pairwise Add and Accumulate Long (u8 => u16) */
+		tmp = vpadalq_u8(tmp, bytes2);
+
+		/* Vector Pairwise Add and Accumulate Long (u16 => u32) */
+		v_s1 = vpadalq_u16(v_s1, tmp);
+
+		/* Vector Add Wide (u8 => u16) */
+		v_byte_sums_a = vaddw_u8(v_byte_sums_a, vget_low_u8(bytes1));
+		v_byte_sums_b = vaddw_u8(v_byte_sums_b, vget_high_u8(bytes1));
+		v_byte_sums_c = vaddw_u8(v_byte_sums_c, vget_low_u8(bytes2));
+		v_byte_sums_d = vaddw_u8(v_byte_sums_d, vget_high_u8(bytes2));
+
+	} while (p != end);
+
+	/* Vector Shift Left (u32) */
+	v_s2 = vqshlq_n_u32(v_s2, 5);
+
+	/* Vector Multiply Accumulate Long (u16 => u32) */
+	v_s2 = vmlal_u16(v_s2, vget_low_u16(v_byte_sums_a),  (uint16x4_t) { 32, 31, 30, 29 });
+	v_s2 = vmlal_u16(v_s2, vget_high_u16(v_byte_sums_a), (uint16x4_t) { 28, 27, 26, 25 });
+	v_s2 = vmlal_u16(v_s2, vget_low_u16(v_byte_sums_b),  (uint16x4_t) { 24, 23, 22, 21 });
+	v_s2 = vmlal_u16(v_s2, vget_high_u16(v_byte_sums_b), (uint16x4_t) { 20, 19, 18, 17 });
+	v_s2 = vmlal_u16(v_s2, vget_low_u16(v_byte_sums_c),  (uint16x4_t) { 16, 15, 14, 13 });
+	v_s2 = vmlal_u16(v_s2, vget_high_u16(v_byte_sums_c), (uint16x4_t) { 12, 11, 10,  9 });
+	v_s2 = vmlal_u16(v_s2, vget_low_u16 (v_byte_sums_d), (uint16x4_t) {  8,  7,  6,  5 });
+	v_s2 = vmlal_u16(v_s2, vget_high_u16(v_byte_sums_d), (uint16x4_t) {  4,  3,  2,  1 });
+
+	*s1 += v_s1[0] + v_s1[1] + v_s1[2] + v_s1[3];
+	*s2 += v_s2[0] + v_s2[1] + v_s2[2] + v_s2[3];
+}
+#  include "../adler32_vec_template.h"
+#endif /* NEON implementation */
+
+#ifdef DISPATCH
+static inline adler32_func_t
+arch_select_adler32_func(void)
+{
+	u32 features = get_cpu_features();
+
+#ifdef DISPATCH_NEON
+	if (features & ARM_CPU_FEATURE_NEON)
+		return adler32_neon;
+#endif
+	return NULL;
+}
+#endif /* DISPATCH */

lib/arm/cpu_features.c

@@ -0,0 +1,119 @@
+/*
+ * arm/cpu_features.c - feature detection for ARM processors
+ *
+ * Copyright 2018 Eric Biggers
+ *
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+/*
+ * ARM processors don't have a standard way for unprivileged programs to detect
+ * processor features.  But, on Linux we can read the AT_HWCAP and AT_HWCAP2
+ * values from /proc/self/auxv.
+ *
+ * Ideally we'd use the C library function getauxval(), but it's not guaranteed
+ * to be available: it was only added to glibc in 2.16, and in Android it was
+ * added to API level 18 for ARM and level 21 for AArch64.
+ */
+
+#include "cpu_features.h"
+
+#if ARM_CPU_FEATURES_ENABLED
+
+#include <errno.h>
+#include <fcntl.h>
+#include <string.h>
+#include <unistd.h>
+
+#define AT_HWCAP	16
+#define AT_HWCAP2	26
+
+volatile u32 _cpu_features = 0;
+
+static void scan_auxv(unsigned long *hwcap, unsigned long *hwcap2)
+{
+	int fd;
+	unsigned long auxbuf[32];
+	int filled = 0;
+	int i;
+
+	fd = open("/proc/self/auxv", O_RDONLY);
+	if (fd < 0)
+		return;
+
+	for (;;) {
+		do {
+			int ret = read(fd, &((char *)auxbuf)[filled],
+				       sizeof(auxbuf) - filled);
+			if (ret <= 0) {
+				if (ret < 0 && errno == EINTR)
+					continue;
+				goto out;
+			}
+			filled += ret;
+		} while (filled < 2 * sizeof(long));
+
+		i = 0;
+		do {
+			unsigned long type = auxbuf[i];
+			unsigned long value = auxbuf[i + 1];
+
+			if (type == AT_HWCAP)
+				*hwcap = value;
+			else if (type == AT_HWCAP2)
+				*hwcap2 = value;
+			i += 2;
+			filled -= 2 * sizeof(long);
+		} while (filled >= 2 * sizeof(long));
+
+		memmove(auxbuf, &auxbuf[i], filled);
+	}
+out:
+	close(fd);
+}
+
+void setup_cpu_features(void)
+{
+	u32 features = 0;
+	unsigned long hwcap = 0;
+	unsigned long hwcap2 = 0;
+
+	scan_auxv(&hwcap, &hwcap2);
+
+#ifdef __arm__
+	STATIC_ASSERT(sizeof(long) == 4);
+	if (hwcap & (1 << 12))	/* HWCAP_NEON */
+		features |= ARM_CPU_FEATURE_NEON;
+	if (hwcap2 & (1 << 1))	/* HWCAP2_PMULL */
+		features |= ARM_CPU_FEATURE_PMULL;
+#else
+	STATIC_ASSERT(sizeof(long) == 8);
+	if (hwcap & (1 << 1))	/* HWCAP_ASIMD */
+		features |= ARM_CPU_FEATURE_NEON;
+	if (hwcap & (1 << 4))	/* HWCAP_PMULL */
+		features |= ARM_CPU_FEATURE_PMULL;
+#endif
+
+	_cpu_features = features | ARM_CPU_FEATURES_KNOWN;
+}
+
+#endif /* ARM_CPU_FEATURES_ENABLED */

lib/arm/cpu_features.h

@@ -0,0 +1,37 @@
+/*
+ * arm/cpu_features.h - feature detection for ARM processors
+ */
+
+#ifndef LIB_ARM_CPU_FEATURES_H
+#define LIB_ARM_CPU_FEATURES_H
+
+#include "../lib_common.h"
+
+#if (defined(__arm__) || defined(__aarch64__)) && \
+	defined(__linux__) && COMPILER_SUPPORTS_TARGET_FUNCTION_ATTRIBUTE
+#  define ARM_CPU_FEATURES_ENABLED 1
+#else
+#  define ARM_CPU_FEATURES_ENABLED 0
+#endif
+
+#if ARM_CPU_FEATURES_ENABLED
+
+#define ARM_CPU_FEATURE_NEON		0x00000001
+#define ARM_CPU_FEATURE_PMULL		0x00000002
+
+#define ARM_CPU_FEATURES_KNOWN		0x80000000
+
+extern volatile u32 _cpu_features;
+
+extern void setup_cpu_features(void);
+
+static inline u32 get_cpu_features(void)
+{
+	if (_cpu_features == 0)
+		setup_cpu_features();
+	return _cpu_features;
+}
+
+#endif /* ARM_CPU_FEATURES_ENABLED */
+
+#endif /* LIB_ARM_CPU_FEATURES_H */

lib/arm/crc32_impl.h

@@ -0,0 +1,166 @@
+/*
+ * arm/crc32_impl.h
+ *
+ * Copyright 2017 Jun He <jun.he@linaro.org>
+ * Copyright 2018 Eric Biggers
+ *
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include "cpu_features.h"
+
+/*
+ * CRC-32 folding with ARM Crypto extension-PMULL
+ *
+ * This works the same way as the x86 PCLMUL version.
+ * See x86/crc32_pclmul_template.h for an explanation.
+ */
+#undef DISPATCH_PMULL
+#if (defined(__ARM_FEATURE_CRYPTO) ||	\
+     (ARM_CPU_FEATURES_ENABLED &&	\
+      COMPILER_SUPPORTS_PMULL_TARGET_INTRINSICS)) && \
+      /* not yet tested on big endian, probably needs changes to work there */ \
+    (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) && \
+      /* clang as of v5.0.1 doesn't allow pmull intrinsics in 32-bit mode, even
+       * when compiling with -mfpu=crypto-neon-fp-armv8 */ \
+    !(defined(__clang__) && defined(__arm__))
+#  define FUNCNAME		crc32_pmull
+#  define FUNCNAME_ALIGNED	crc32_pmull_aligned
+#  ifdef __ARM_FEATURE_CRYPTO
+#    define ATTRIBUTES
+#    define DEFAULT_IMPL	crc32_pmull
+#  else
+#    ifdef __arm__
+#      define ATTRIBUTES	__attribute__((target("fpu=crypto-neon-fp-armv8")))
+#    else
+#      ifdef __clang__
+#        define ATTRIBUTES	__attribute__((target("crypto")))
+#      else
+#        define ATTRIBUTES	__attribute__((target("+crypto")))
+#      endif
+#    endif
+#    define DISPATCH		1
+#    define DISPATCH_PMULL	1
+#  endif
+
+#include <arm_neon.h>
+
+static forceinline ATTRIBUTES uint8x16_t
+clmul_00(uint8x16_t a, uint8x16_t b)
+{
+	return (uint8x16_t)vmull_p64((poly64_t)vget_low_u8(a),
+				     (poly64_t)vget_low_u8(b));
+}
+
+static forceinline ATTRIBUTES uint8x16_t
+clmul_10(uint8x16_t a, uint8x16_t b)
+{
+	return (uint8x16_t)vmull_p64((poly64_t)vget_low_u8(a),
+				     (poly64_t)vget_high_u8(b));
+}
+
+static forceinline ATTRIBUTES uint8x16_t
+clmul_11(uint8x16_t a, uint8x16_t b)
+{
+	return (uint8x16_t)vmull_high_p64((poly64x2_t)a, (poly64x2_t)b);
+}
+
+static forceinline ATTRIBUTES uint8x16_t
+fold_128b(uint8x16_t dst, uint8x16_t src, uint8x16_t multipliers)
+{
+	return dst ^ clmul_00(src, multipliers) ^ clmul_11(src, multipliers);
+}
+
+static forceinline ATTRIBUTES u32
+crc32_pmull_aligned(u32 remainder, const uint8x16_t *p, size_t nr_segs)
+{
+	/* Constants precomputed by gen_crc32_multipliers.c.  Do not edit! */
+	const uint8x16_t multipliers_4 =
+		(uint8x16_t)(uint64x2_t){ 0x8F352D95, 0x1D9513D7 };
+	const uint8x16_t multipliers_1 =
+		(uint8x16_t)(uint64x2_t){ 0xAE689191, 0xCCAA009E };
+	const uint8x16_t final_multiplier =
+		(uint8x16_t)(uint64x2_t){ 0xB8BC6765 };
+	const uint8x16_t mask32 = (uint8x16_t)(uint32x4_t){ 0xFFFFFFFF };
+	const uint8x16_t barrett_reduction_constants =
+			(uint8x16_t)(uint64x2_t){ 0x00000001F7011641,
+						  0x00000001DB710641 };
+	const uint8x16_t zeroes = (uint8x16_t){ 0 };
+
+	const uint8x16_t * const end = p + nr_segs;
+	const uint8x16_t * const end512 = p + (nr_segs & ~3);
+	uint8x16_t x0, x1, x2, x3;
+
+	x0 = *p++ ^ (uint8x16_t)(uint32x4_t){ remainder };
+	if (nr_segs >= 4) {
+		x1 = *p++;
+		x2 = *p++;
+		x3 = *p++;
+
+		/* Fold 512 bits at a time */
+		while (p != end512) {
+			x0 = fold_128b(*p++, x0, multipliers_4);
+			x1 = fold_128b(*p++, x1, multipliers_4);
+			x2 = fold_128b(*p++, x2, multipliers_4);
+			x3 = fold_128b(*p++, x3, multipliers_4);
+		}
+
+		/* Fold 512 bits => 128 bits */
+		x1 = fold_128b(x1, x0, multipliers_1);
+		x2 = fold_128b(x2, x1, multipliers_1);
+		x0 = fold_128b(x3, x2, multipliers_1);
+	}
+
+	/* Fold 128 bits at a time */
+	while (p != end)
+		x0 = fold_128b(*p++, x0, multipliers_1);
+
+	/* Fold 128 => 96 bits, implicitly appending 32 zeroes */
+	x0 = vextq_u8(x0, zeroes, 8) ^ clmul_10(x0, multipliers_1);
+
+	/* Fold 96 => 64 bits */
+	x0 = vextq_u8(x0, zeroes, 4) ^ clmul_00(x0 & mask32, final_multiplier);
+
+	/* Reduce 64 => 32 bits using Barrett reduction */
+	x1 = x0;
+	x0 = clmul_00(x0 & mask32, barrett_reduction_constants);
+	x0 = clmul_10(x0 & mask32, barrett_reduction_constants);
+	return vgetq_lane_u32((uint32x4_t)(x0 ^ x1), 1);
+}
+#define IMPL_ALIGNMENT		16
+#define IMPL_SEGMENT_SIZE	16
+#include "../crc32_vec_template.h"
+#endif /* PMULL implementation */
+
+#ifdef DISPATCH
+static inline crc32_func_t
+arch_select_crc32_func(void)
+{
+	u32 features = get_cpu_features();
+
+#ifdef DISPATCH_PMULL
+	if (features & ARM_CPU_FEATURE_PMULL)
+		return crc32_pmull;
+#endif
+	return NULL;
+}
+#endif /* DISPATCH */

lib/arm/matchfinder_impl.h

@@ -0,0 +1,93 @@
+/*
+ * arm/matchfinder_impl.h - ARM implementations of matchfinder functions
+ *
+ * Copyright 2016 Eric Biggers
+ *
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#ifdef __ARM_NEON
+#  if MATCHFINDER_ALIGNMENT < 16
+#    undef MATCHFINDER_ALIGNMENT
+#    define MATCHFINDER_ALIGNMENT 16
+#  endif
+#  include <arm_neon.h>
+static forceinline bool
+matchfinder_init_neon(mf_pos_t *data, size_t size)
+{
+	int16x8_t v, *p;
+	size_t n;
+
+	if (size % (sizeof(int16x8_t) * 4) != 0)
+		return false;
+
+	STATIC_ASSERT(sizeof(mf_pos_t) == 2);
+	v = (int16x8_t) {
+		MATCHFINDER_INITVAL, MATCHFINDER_INITVAL, MATCHFINDER_INITVAL,
+		MATCHFINDER_INITVAL, MATCHFINDER_INITVAL, MATCHFINDER_INITVAL,
+		MATCHFINDER_INITVAL, MATCHFINDER_INITVAL,
+	};
+	p = (int16x8_t *)data;
+	n = size / (sizeof(int16x8_t) * 4);
+	do {
+		p[0] = v;
+		p[1] = v;
+		p[2] = v;
+		p[3] = v;
+		p += 4;
+	} while (--n);
+	return true;
+}
+#undef arch_matchfinder_init
+#define arch_matchfinder_init matchfinder_init_neon
+
+static forceinline bool
+matchfinder_rebase_neon(mf_pos_t *data, size_t size)
+{
+	int16x8_t v, *p;
+	size_t n;
+
+	if (size % (sizeof(int16x8_t) * 4) != 0)
+		return false;
+
+	STATIC_ASSERT(sizeof(mf_pos_t) == 2);
+	v = (int16x8_t) {
+		(u16)-MATCHFINDER_WINDOW_SIZE, (u16)-MATCHFINDER_WINDOW_SIZE,
+		(u16)-MATCHFINDER_WINDOW_SIZE, (u16)-MATCHFINDER_WINDOW_SIZE,
+		(u16)-MATCHFINDER_WINDOW_SIZE, (u16)-MATCHFINDER_WINDOW_SIZE,
+		(u16)-MATCHFINDER_WINDOW_SIZE, (u16)-MATCHFINDER_WINDOW_SIZE,
+	};
+	p = (int16x8_t *)data;
+	n = size / (sizeof(int16x8_t) * 4);
+	do {
+		p[0] = vqaddq_s16(p[0], v);
+		p[1] = vqaddq_s16(p[1], v);
+		p[2] = vqaddq_s16(p[2], v);
+		p[3] = vqaddq_s16(p[3], v);
+		p += 4;
+	} while (--n);
+	return true;
+}
+#undef arch_matchfinder_rebase
+#define arch_matchfinder_rebase matchfinder_rebase_neon
+
+#endif /* __ARM_NEON */

lib/bt_matchfinder.h

@@ -1,15 +1,30 @@
 /*
  * bt_matchfinder.h - Lempel-Ziv matchfinding with a hash table of binary trees
  *
- * Written in 2014-2016 by Eric Biggers <ebiggers3@gmail.com>
+ * Originally public domain; changes after 2016-09-07 are copyrighted.
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Copyright 2016 Eric Biggers
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  *
  * ----------------------------------------------------------------------------
  *
@@ -138,8 +153,7 @@ bt_matchfinder_advance_one_byte(struct bt_matchfinder * const restrict mf,
 	const u8 *in_next = in_base + cur_pos;
 	u32 depth_remaining = max_search_depth;
 	const s32 cutoff = cur_pos - MATCHFINDER_WINDOW_SIZE;
-	u32 next_seq4;
-	u32 next_seq3;
+	u32 next_hashseq;
 	u32 hash3;
 	u32 hash4;
 	s32 cur_node;
@@ -155,14 +169,13 @@ bt_matchfinder_advance_one_byte(struct bt_matchfinder * const restrict mf,
 	STATIC_ASSERT(BT_MATCHFINDER_HASH3_WAYS >= 1 &&
 		      BT_MATCHFINDER_HASH3_WAYS <= 2);
 
-	next_seq4 = load_u32_unaligned(in_next + 1);
-	next_seq3 = loaded_u32_to_u24(next_seq4);
+	next_hashseq = get_unaligned_le32(in_next + 1);
 
 	hash3 = next_hashes[0];
 	hash4 = next_hashes[1];
 
-	next_hashes[0] = lz_hash(next_seq3, BT_MATCHFINDER_HASH3_ORDER);
-	next_hashes[1] = lz_hash(next_seq4, BT_MATCHFINDER_HASH4_ORDER);
+	next_hashes[0] = lz_hash(next_hashseq & 0xFFFFFF, BT_MATCHFINDER_HASH3_ORDER);
+	next_hashes[1] = lz_hash(next_hashseq, BT_MATCHFINDER_HASH4_ORDER);
 	prefetchw(&mf->hash3_tab[next_hashes[0]]);
 	prefetchw(&mf->hash4_tab[next_hashes[1]]);
 
@@ -211,8 +224,7 @@ bt_matchfinder_advance_one_byte(struct bt_matchfinder * const restrict mf,
 		matchptr = &in_base[cur_node];
 
 		if (matchptr[len] == in_next[len]) {
-			len = lz_extend(in_next, matchptr, len + 1,
-					(record_matches ? max_len : nice_len));
+			len = lz_extend(in_next, matchptr, len + 1, max_len);
 			if (!record_matches || len > best_len) {
 				if (record_matches) {
 					best_len = len;
@@ -325,7 +337,6 @@ static forceinline void
 bt_matchfinder_skip_position(struct bt_matchfinder *mf,
 			     const u8 *in_base,
 			     ptrdiff_t cur_pos,
-			     u32 max_len,
 			     u32 nice_len,
 			     u32 max_search_depth,
 			     u32 next_hashes[2])
@@ -334,7 +345,7 @@ bt_matchfinder_skip_position(struct bt_matchfinder *mf,
 	bt_matchfinder_advance_one_byte(mf,
 					in_base,
 					cur_pos,
-					max_len,
+					nice_len,
 					nice_len,
 					max_search_depth,
 					next_hashes,

lib/crc32.c

@@ -1,22 +1,35 @@
 /*
  * crc32.c - CRC-32 checksum algorithm for the gzip format
  *
- * Written in 2014-2015 by Eric Biggers <ebiggers3@gmail.com>
+ * Copyright 2016 Eric Biggers
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  */
 
 /*
  * High-level description of CRC
  * =============================
  *
- * Consider a bit sequence 'bits[1...len]'.  Interpet 'bits' as the "message"
+ * Consider a bit sequence 'bits[1...len]'.  Interpret 'bits' as the "message"
  * polynomial M(x) with coefficients in GF(2) (the field of integers modulo 2),
  * where the coefficient of 'x^i' is 'bits[len - i]'.  Then, compute:
  *
@@ -24,7 +37,7 @@
  *
  * where G(x) is a selected "generator" polynomial of degree 'n'.  The remainder
  * R(x) is a polynomial of max degree 'n - 1'.  The CRC of 'bits' is R(x)
- * interpeted as a bitstring of length 'n'.
+ * interpreted as a bitstring of length 'n'.
  *
  * CRC used in gzip
  * ================
@@ -57,16 +70,16 @@
  * subtraction can be implemented as bitwise exclusive OR (since we are working
  * in GF(2)).  Here is an unoptimized implementation:
  *
- *	static u32 crc32_gzip(const u8 *buffer, size_t nbytes)
+ *	static u32 crc32_gzip(const u8 *buffer, size_t size)
  *	{
  *		u32 remainder = 0;
  *		const u32 divisor = 0xEDB88320;
  *
- *		for (size_t i = 0; i < nbytes * 8 + 32; i++) {
+ *		for (size_t i = 0; i < size * 8 + 32; i++) {
  *			int bit;
  *			u32 multiple;
  *
- *			if (i < nbytes * 8)
+ *			if (i < size * 8)
  *				bit = (buffer[i / 8] >> (i % 8)) & 1;
  *			else
  *				bit = 0; // one of the 32 appended 0 bits
@@ -101,12 +114,12 @@
  * of 'multiple' until 32 bits later, we need not actually add each message bit
  * until that point:
  *
- *	static u32 crc32_gzip(const u8 *buffer, size_t nbytes)
+ *	static u32 crc32_gzip(const u8 *buffer, size_t size)
  *	{
  *		u32 remainder = ~0;
  *		const u32 divisor = 0xEDB88320;
  *
- *		for (size_t i = 0; i < nbytes * 8; i++) {
+ *		for (size_t i = 0; i < size * 8; i++) {
  *			int bit;
  *			u32 multiple;
  *
@@ -151,39 +164,67 @@
  *
  * In crc32_slice8(), this method is extended to 8 bytes at a time.  The
  * intermediate remainder (which we never actually store explicitly) is 96 bits.
+ *
+ * On CPUs that support fast carryless multiplication, CRCs can be computed even
+ * more quickly via "folding".  See e.g. the x86 PCLMUL implementation.
  */
 
-#define CRC32_SLICE8
+#include "lib_common.h"
+#include "libdeflate.h"
 
-#include "crc32.h"
+typedef u32 (*crc32_func_t)(u32, const u8 *, size_t);
+
+/* Include architecture-specific implementations if available */
+#undef CRC32_SLICE1
+#undef CRC32_SLICE4
+#undef CRC32_SLICE8
+#undef DEFAULT_IMPL
+#undef DISPATCH
+#if defined(__arm__) || defined(__aarch64__)
+#  include "arm/crc32_impl.h"
+#elif defined(__i386__) || defined(__x86_64__)
+#  include "x86/crc32_impl.h"
+#endif
+
+/*
+ * Define a generic implementation (crc32_slice8()) if needed.  crc32_slice1()
+ * may also be needed as a fallback for architecture-specific implementations.
+ */
+
+#ifndef DEFAULT_IMPL
+#  define CRC32_SLICE8	1
+#  define DEFAULT_IMPL	crc32_slice8
+#endif
+
+#if defined(CRC32_SLICE1) || defined(CRC32_SLICE4) || defined(CRC32_SLICE8)
 #include "crc32_table.h"
-
 static forceinline u32
 crc32_update_byte(u32 remainder, u8 next_byte)
 {
 	return (remainder >> 8) ^ crc32_table[(u8)remainder ^ next_byte];
 }
+#endif
 
 #ifdef CRC32_SLICE1
 static u32
-crc32_slice1(u32 remainder, const u8 *buffer, size_t nbytes)
+crc32_slice1(u32 remainder, const u8 *buffer, size_t size)
 {
 	size_t i;
 
 	STATIC_ASSERT(ARRAY_LEN(crc32_table) >= 0x100);
 
-	for (i = 0; i < nbytes; i++)
+	for (i = 0; i < size; i++)
 		remainder = crc32_update_byte(remainder, buffer[i]);
 	return remainder;
 }
-#endif
+#endif /* CRC32_SLICE1 */
 
 #ifdef CRC32_SLICE4
 static u32
-crc32_slice4(u32 remainder, const u8 *buffer, size_t nbytes)
+crc32_slice4(u32 remainder, const u8 *buffer, size_t size)
 {
 	const u8 *p = buffer;
-	const u8 *end = buffer + nbytes;
+	const u8 *end = buffer + size;
 	const u8 *end32;
 
 	STATIC_ASSERT(ARRAY_LEN(crc32_table) >= 0x400);
@@ -206,14 +247,14 @@ crc32_slice4(u32 remainder, const u8 *buffer, size_t nbytes)
 
 	return remainder;
 }
-#endif
+#endif /* CRC32_SLICE4 */
 
 #ifdef CRC32_SLICE8
 static u32
-crc32_slice8(u32 remainder, const u8 *buffer, size_t nbytes)
+crc32_slice8(u32 remainder, const u8 *buffer, size_t size)
 {
 	const u8 *p = buffer;
-	const u8 *end = buffer + nbytes;
+	const u8 *end = buffer + size;
 	const u8 *end64;
 
 	STATIC_ASSERT(ARRAY_LEN(crc32_table) >= 0x800);
@@ -241,20 +282,32 @@ crc32_slice8(u32 remainder, const u8 *buffer, size_t nbytes)
 
 	return remainder;
 }
+#endif /* CRC32_SLICE8 */
+
+#ifdef DISPATCH
+static u32 dispatch(u32, const u8 *, size_t);
+
+static volatile crc32_func_t crc32_impl = dispatch;
+
+/* Choose the fastest implementation at runtime */
+static u32 dispatch(u32 remainder, const u8 *buffer, size_t size)
+{
+	crc32_func_t f = arch_select_crc32_func();
+
+	if (f == NULL)
+		f = DEFAULT_IMPL;
+
+	crc32_impl = f;
+	return crc32_impl(remainder, buffer, size);
+}
+#else
+#  define crc32_impl DEFAULT_IMPL /* only one implementation, use it */
 #endif
 
-u32
-crc32_gzip(const void *buffer, size_t nbytes)
+LIBDEFLATEAPI u32
+libdeflate_crc32(u32 remainder, const void *buffer, size_t size)
 {
-	u32 remainder = ~0;
-#if defined(CRC32_SLICE1)
-	remainder = crc32_slice1(remainder, buffer, nbytes);
-#elif defined(CRC32_SLICE4)
-	remainder = crc32_slice4(remainder, buffer, nbytes);
-#elif defined(CRC32_SLICE8)
-	remainder = crc32_slice8(remainder, buffer, nbytes);
-#else
-#  error "don't know which CRC-32 implementation to use!"
-#endif
-	return ~remainder;
+	if (buffer == NULL) /* return initial value */
+		return 0;
+	return ~crc32_impl(~remainder, buffer, size);
 }

lib/crc32.h

@@ -1,12 +0,0 @@
-/*
- * crc32.h - CRC-32 checksum algorithm for the gzip format
- */
-
-#ifndef _LIB_CRC32_H
-#define _LIB_CRC32_H
-
-#include "common_defs.h"
-
-extern u32 crc32_gzip(const void *buffer, size_t size);
-
-#endif /* _LIB_CRC32_H */

lib/crc32_vec_template.h

@@ -0,0 +1,61 @@
+/*
+ * crc32_vec_template.h - template for vectorized CRC-32 implementations
+ *
+ * Copyright 2016 Eric Biggers
+ *
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#define CRC32_SLICE1	1
+static u32 crc32_slice1(u32, const u8 *, size_t);
+
+/*
+ * Template for vectorized CRC-32 implementations.
+ *
+ * Note: on unaligned ends of the buffer, we fall back to crc32_slice1() instead
+ * of crc32_slice8() because only a few bytes need to be processed, so a smaller
+ * table is preferable.
+ */
+static u32 ATTRIBUTES
+FUNCNAME(u32 remainder, const u8 *p, size_t size)
+{
+	if ((uintptr_t)p % IMPL_ALIGNMENT) {
+		size_t n = MIN(size, -(uintptr_t)p % IMPL_ALIGNMENT);
+
+		remainder = crc32_slice1(remainder, p, n);
+		p += n;
+		size -= n;
+	}
+	if (size >= IMPL_SEGMENT_SIZE) {
+		remainder = FUNCNAME_ALIGNED(remainder, (const void *)p,
+					     size / IMPL_SEGMENT_SIZE);
+		p += size - (size % IMPL_SEGMENT_SIZE);
+		size %= IMPL_SEGMENT_SIZE;
+	}
+	return crc32_slice1(remainder, p, size);
+}
+
+#undef FUNCNAME
+#undef FUNCNAME_ALIGNED
+#undef ATTRIBUTES
+#undef IMPL_ALIGNMENT
+#undef IMPL_SEGMENT_SIZE

lib/decompress_template.h

@@ -1,15 +1,28 @@
 /*
- * decompress_impl.h
+ * decompress_template.h
  *
- * Written in 2014-2016 by Eric Biggers <ebiggers3@gmail.com>
+ * Copyright 2016 Eric Biggers
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  */
 
 /*
@@ -18,11 +31,11 @@
  * target instruction sets.
  */
 
-static enum decompress_result ATTRIBUTES
-FUNCNAME(struct deflate_decompressor * restrict d,
+static enum libdeflate_result ATTRIBUTES
+FUNCNAME(struct libdeflate_decompressor * restrict d,
 	 const void * restrict in, size_t in_nbytes,
 	 void * restrict out, size_t out_nbytes_avail,
-	 size_t *actual_out_nbytes_ret)
+	 size_t *actual_in_nbytes_ret, size_t *actual_out_nbytes_ret)
 {
 	u8 *out_next = out;
 	u8 * const out_end = out_next + out_nbytes_avail;
@@ -192,7 +205,7 @@ next_block:
 
 		SAFETY_CHECK(len == (u16)~nlen);
 		if (unlikely(len > out_end - out_next))
-			return DECOMPRESS_INSUFFICIENT_SPACE;
+			return LIBDEFLATE_INSUFFICIENT_SPACE;
 		SAFETY_CHECK(len <= in_end - in_next);
 
 		memcpy(out_next, in_next, len);
@@ -253,7 +266,7 @@ next_block:
 		if (entry & HUFFDEC_LITERAL) {
 			/* Literal  */
 			if (unlikely(out_next == out_end))
-				return DECOMPRESS_INSUFFICIENT_SPACE;
+				return LIBDEFLATE_INSUFFICIENT_SPACE;
 			*out_next++ = (u8)(entry >> HUFFDEC_RESULT_SHIFT);
 			continue;
 		}
@@ -276,7 +289,7 @@ next_block:
 		STATIC_ASSERT(HUFFDEC_END_OF_BLOCK_LENGTH == 0);
 		if (unlikely((size_t)length - 1 >= out_end - out_next)) {
 			if (unlikely(length != HUFFDEC_END_OF_BLOCK_LENGTH))
-				return DECOMPRESS_INSUFFICIENT_SPACE;
+				return LIBDEFLATE_INSUFFICIENT_SPACE;
 			goto block_done;
 		}
 
@@ -379,11 +392,22 @@ block_done:
 
 	/* That was the last block.  */
 
+	/* Discard any readahead bits and check for excessive overread */
+	ALIGN_INPUT();
+
+	/* Optionally return the actual number of bytes read */
+	if (actual_in_nbytes_ret)
+		*actual_in_nbytes_ret = in_next - (u8 *)in;
+
+	/* Optionally return the actual number of bytes written */
 	if (actual_out_nbytes_ret) {
 		*actual_out_nbytes_ret = out_next - (u8 *)out;
 	} else {
 		if (out_next != out_end)
-			return DECOMPRESS_SHORT_OUTPUT;
+			return LIBDEFLATE_SHORT_OUTPUT;
 	}
-	return DECOMPRESS_SUCCESS;
+	return LIBDEFLATE_SUCCESS;
 }
+
+#undef FUNCNAME
+#undef ATTRIBUTES

lib/deflate_compress.c

@@ -1,15 +1,30 @@
 /*
  * deflate_compress.c - a compressor for DEFLATE
  *
- * Written in 2014-2016 by Eric Biggers <ebiggers3@gmail.com>
+ * Originally public domain; changes after 2016-09-07 are copyrighted.
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Copyright 2016 Eric Biggers
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  */
 
 #include <stdlib.h>
@@ -51,12 +66,24 @@
 #endif
 
 /*
- * The minimum and maximum block lengths, in bytes of source data, which the
- * parsing algorithms may choose.  Caveat: due to implementation details, the
- * actual maximum will be slightly higher than the number defined below.
+ * The compressor always chooses a block of at least MIN_BLOCK_LENGTH bytes,
+ * except if the last block has to be shorter.
  */
 #define MIN_BLOCK_LENGTH	10000
-#define MAX_BLOCK_LENGTH	300000
+
+/*
+ * The compressor attempts to end blocks after SOFT_MAX_BLOCK_LENGTH bytes, but
+ * the final length might be slightly longer due to matches extending beyond
+ * this limit.
+ */
+#define SOFT_MAX_BLOCK_LENGTH	300000
+
+/*
+ * The number of observed matches or literals that represents sufficient data to
+ * decide whether the current block should be terminated or not.
+ */
+#define NUM_OBSERVATIONS_PER_BLOCK_CHECK       512
+
 
 #if SUPPORT_NEAR_OPTIMAL_PARSING
 /* Constants specific to the near-optimal parsing algorithm */
@@ -77,7 +104,7 @@
  * However, fallback behavior (immediately terminating the block) on cache
  * overflow is still required.
  */
-#  define CACHE_LENGTH      (MAX_BLOCK_LENGTH * 5)
+#  define CACHE_LENGTH      (SOFT_MAX_BLOCK_LENGTH * 5)
 
 #endif /* SUPPORT_NEAR_OPTIMAL_PARSING */
 
@@ -85,7 +112,7 @@
  * These are the compressor-side limits on the codeword lengths for each Huffman
  * code.  To make outputting bits slightly faster, some of these limits are
  * lower than the limits defined by the DEFLATE format.  This does not
- * significantly affect the compression ratio, at least for the block sizes we
+ * significantly affect the compression ratio, at least for the block lengths we
  * use.
  */
 #define MAX_LITLEN_CODEWORD_LEN		14
@@ -219,10 +246,19 @@ struct deflate_costs {
  */
 struct deflate_sequence {
 
-	/* The number of literals in the run.  This may be 0.  The literals are
-	 * not stored explicitly in this structure; instead, they are read
-	 * directly from the uncompressed data.  */
-	u16 litrunlen;
+	/* Bits 0..22: the number of literals in this run.  This may be 0 and
+	 * can be at most about SOFT_MAX_BLOCK_LENGTH.  The literals are not
+	 * stored explicitly in this structure; instead, they are read directly
+	 * from the uncompressed data.
+	 *
+	 * Bits 23..31: the length of the match which follows the literals, or 0
+	 * if this literal run was the last in the block, so there is no match
+	 * which follows it.  */
+	u32 litrunlen_and_length;
+
+	/* If 'length' doesn't indicate end-of-block, then this is the offset of
+	 * the match which follows the literals.  */
+	u16 offset;
 
 	/* If 'length' doesn't indicate end-of-block, then this is the offset
 	 * symbol of the match which follows the literals.  */
@@ -231,15 +267,6 @@ struct deflate_sequence {
 	/* If 'length' doesn't indicate end-of-block, then this is the length
 	 * slot of the match which follows the literals.  */
 	u8 length_slot;
-
-	/* The length of the match which follows the literals, or 0 if this this
-	 * sequence's literal run was the last literal run in the block, so
-	 * there is no match that follows it.  */
-	u16 length;
-
-	/* If 'length' doesn't indicate end-of-block, then this is the offset of
-	 * the match which follows the literals.  */
-	u16 offset;
 };
 
 #if SUPPORT_NEAR_OPTIMAL_PARSING
@@ -297,10 +324,10 @@ struct block_split_stats {
 };
 
 /* The main DEFLATE compressor structure  */
-struct deflate_compressor {
+struct libdeflate_compressor {
 
 	/* Pointer to the compress() implementation chosen at allocation time */
-	size_t (*impl)(struct deflate_compressor *,
+	size_t (*impl)(struct libdeflate_compressor *,
 		       const u8 *, size_t, u8 *, size_t);
 
 	/* Frequency counters for the current block  */
@@ -365,7 +392,7 @@ struct deflate_compressor {
 			 * that can ever be chosen for a single block, plus one
 			 * for the special entry at the end.  */
 			struct deflate_sequence sequences[
-				DIV_ROUND_UP(MAX_BLOCK_LENGTH,
+				DIV_ROUND_UP(SOFT_MAX_BLOCK_LENGTH,
 					     DEFLATE_MIN_MATCH_LEN) + 1];
 		} g; /* (g)reedy */
 
@@ -411,11 +438,12 @@ struct deflate_compressor {
 			 * This array must be large enough to accommodate the
 			 * worst-case number of nodes, which occurs if we find a
 			 * match of length DEFLATE_MAX_MATCH_LEN at position
-			 * MAX_BLOCK_LENGTH - 1, producing a block of length
-			 * MAX_BLOCK_LENGTH - 1 + DEFLATE_MAX_MATCH_LEN.  Add
-			 * one for the end-of-block node.
+			 * SOFT_MAX_BLOCK_LENGTH - 1, producing a block of
+			 * length SOFT_MAX_BLOCK_LENGTH - 1 +
+			 * DEFLATE_MAX_MATCH_LEN.  Add one for the end-of-block
+			 * node.
 			 */
-			struct deflate_optimum_node optimum_nodes[MAX_BLOCK_LENGTH - 1 +
+			struct deflate_optimum_node optimum_nodes[SOFT_MAX_BLOCK_LENGTH - 1 +
 								  DEFLATE_MAX_MATCH_LEN + 1];
 
 			/* The current cost model being used.  */
@@ -463,18 +491,28 @@ struct deflate_output_bitstream {
 	u8 *end;
 };
 
-#define MIN_OUTPUT_SIZE	(UNALIGNED_ACCESS_IS_FAST ? sizeof(bitbuf_t) : 1)
+/*
+ * OUTPUT_END_PADDING is the size, in bytes, of the extra space that must be
+ * present following os->end, in order to not overrun the buffer when generating
+ * output.  When UNALIGNED_ACCESS_IS_FAST, we need at least sizeof(bitbuf_t)
+ * bytes for put_unaligned_leword().  Otherwise we need only 1 byte.  However,
+ * to make the compression algorithm produce the same result on all CPU
+ * architectures (which is sometimes desirable), we have to unconditionally use
+ * the maximum for any CPU, which is sizeof(bitbuf_t) == 8.
+ */
+#define OUTPUT_END_PADDING	8
 
 /* Initialize the output bitstream.  'size' is assumed to be at least
- * MIN_OUTPUT_SIZE.  */
+ * OUTPUT_END_PADDING.  */
 static void
-deflate_init_output(struct deflate_output_bitstream *os, void *buffer, size_t size)
+deflate_init_output(struct deflate_output_bitstream *os,
+		    void *buffer, size_t size)
 {
 	os->bitbuf = 0;
 	os->bitcount = 0;
 	os->begin = buffer;
 	os->next = os->begin;
-	os->end = os->begin + size - MIN_OUTPUT_SIZE;
+	os->end = os->begin + size - OUTPUT_END_PADDING;
 }
 
 /* Add some bits to the bitbuffer variable of the output bitstream.  The caller
@@ -572,9 +610,15 @@ heapify_array(u32 A[], unsigned length)
 		heapify_subtree(A, length, subtree_idx);
 }
 
-/* Sort the array 'A', which contains 'length' unsigned 32-bit integers.  */
+/*
+ * Sort the array 'A', which contains 'length' unsigned 32-bit integers.
+ *
+ * Note: name this function heap_sort() instead of heapsort() to avoid colliding
+ * with heapsort() from stdlib.h on BSD-derived systems --- though this isn't
+ * necessary when compiling with -D_ANSI_SOURCE, which is the better solution.
+ */
 static void
-heapsort(u32 A[], unsigned length)
+heap_sort(u32 A[], unsigned length)
 {
 	A--; /* Use 1-based indices  */
 
@@ -682,8 +726,8 @@ sort_symbols(unsigned num_syms, const u32 freqs[restrict],
 	}
 
 	/* Sort the symbols counted in the last counter.  */
-	heapsort(symout + counters[num_counters - 2],
-		 counters[num_counters - 1] - counters[num_counters - 2]);
+	heap_sort(symout + counters[num_counters - 2],
+		  counters[num_counters - 1] - counters[num_counters - 2]);
 
 	return num_used_syms;
 }
@@ -1113,7 +1157,7 @@ make_canonical_huffman_code(unsigned num_syms, unsigned max_codeword_len,
  * This must be called when starting a new DEFLATE block.
  */
 static void
-deflate_reset_symbol_frequencies(struct deflate_compressor *c)
+deflate_reset_symbol_frequencies(struct libdeflate_compressor *c)
 {
 	memset(&c->freqs, 0, sizeof(c->freqs));
 }
@@ -1185,7 +1229,7 @@ deflate_make_huffman_codes(const struct deflate_freqs *freqs,
 
 /* Initialize c->static_codes.  */
 static void
-deflate_init_static_codes(struct deflate_compressor *c)
+deflate_init_static_codes(struct libdeflate_compressor *c)
 {
 	unsigned i;
 
@@ -1206,7 +1250,7 @@ deflate_init_static_codes(struct deflate_compressor *c)
 
 /* Return the offset slot for the specified match offset.  */
 static forceinline unsigned
-deflate_get_offset_slot(struct deflate_compressor *c, unsigned offset)
+deflate_get_offset_slot(struct libdeflate_compressor *c, unsigned offset)
 {
 #if USE_FULL_OFFSET_SLOT_FAST
 	return c->offset_slot_fast[offset];
@@ -1316,7 +1360,7 @@ deflate_compute_precode_items(const u8 lens[restrict],
 
 /* Precompute the information needed to output Huffman codes. */
 static void
-deflate_precompute_huffman_header(struct deflate_compressor *c)
+deflate_precompute_huffman_header(struct libdeflate_compressor *c)
 {
 	/* Compute how many litlen and offset symbols are needed. */
 
@@ -1379,7 +1423,7 @@ deflate_precompute_huffman_header(struct deflate_compressor *c)
 
 /* Output the Huffman codes. */
 static void
-deflate_write_huffman_header(struct deflate_compressor *c,
+deflate_write_huffman_header(struct libdeflate_compressor *c,
 			     struct deflate_output_bitstream *os)
 {
 	unsigned i;
@@ -1424,8 +1468,8 @@ deflate_write_sequences(struct deflate_output_bitstream * restrict os,
 	const struct deflate_sequence *seq = sequences;
 
 	for (;;) {
-		unsigned litrunlen = seq->litrunlen;
-		unsigned length;
+		u32 litrunlen = seq->litrunlen_and_length & 0x7FFFFF;
+		unsigned length = seq->litrunlen_and_length >> 23;
 		unsigned length_slot;
 		unsigned litlen_symbol;
 		unsigned offset_symbol;
@@ -1492,9 +1536,6 @@ deflate_write_sequences(struct deflate_output_bitstream * restrict os,
 		#endif
 		}
 
-
-		length = seq->length;
-
 		if (length == 0)
 			return;
 
@@ -1551,7 +1592,7 @@ deflate_write_sequences(struct deflate_output_bitstream * restrict os,
 static void
 deflate_write_item_list(struct deflate_output_bitstream *os,
 			const struct deflate_codes *codes,
-			struct deflate_compressor *c,
+			struct libdeflate_compressor *c,
 			u32 block_length)
 {
 	struct deflate_optimum_node *cur_node = &c->p.n.optimum_nodes[0];
@@ -1657,7 +1698,7 @@ deflate_write_uncompressed_blocks(struct deflate_output_bitstream *os,
  * uncompressed), then output it.
  */
 static void
-deflate_flush_block(struct deflate_compressor * restrict c,
+deflate_flush_block(struct libdeflate_compressor * restrict c,
 		    struct deflate_output_bitstream * restrict os,
 		    const u8 * restrict block_begin, u32 block_length,
 		    bool is_final_block, bool use_item_list)
@@ -1766,19 +1807,18 @@ deflate_flush_block(struct deflate_compressor * restrict c,
 }
 
 static forceinline void
-deflate_choose_literal(struct deflate_compressor *c, unsigned literal,
-		       unsigned *litrunlen_p)
+deflate_choose_literal(struct libdeflate_compressor *c, unsigned literal,
+		       u32 *litrunlen_p)
 {
 	c->freqs.litlen[literal]++;
 	++*litrunlen_p;
 }
 
 static forceinline void
-deflate_choose_match(struct deflate_compressor *c,
+deflate_choose_match(struct libdeflate_compressor *c,
 		     unsigned length, unsigned offset,
-		     unsigned *litrunlen_p, struct deflate_sequence **next_seq_p)
+		     u32 *litrunlen_p, struct deflate_sequence **next_seq_p)
 {
-	unsigned litrunlen = *litrunlen_p;
 	struct deflate_sequence *seq = *next_seq_p;
 	unsigned length_slot = deflate_length_slot[length];
 	unsigned offset_slot = deflate_get_offset_slot(c, offset);
@@ -1786,8 +1826,7 @@ deflate_choose_match(struct deflate_compressor *c,
 	c->freqs.litlen[257 + length_slot]++;
 	c->freqs.offset[offset_slot]++;
 
-	seq->litrunlen = litrunlen;
-	seq->length = length;
+	seq->litrunlen_and_length = ((u32)length << 23) | *litrunlen_p;
 	seq->offset = offset;
 	seq->length_slot = length_slot;
 	seq->offset_symbol = offset_slot;
@@ -1797,10 +1836,9 @@ deflate_choose_match(struct deflate_compressor *c,
 }
 
 static forceinline void
-deflate_finish_sequence(struct deflate_sequence *seq, unsigned litrunlen)
+deflate_finish_sequence(struct deflate_sequence *seq, u32 litrunlen)
 {
-	seq->litrunlen = litrunlen;
-	seq->length = 0;
+	seq->litrunlen_and_length = litrunlen; /* length = 0 */
 }
 
 /******************************************************************************/
@@ -1829,10 +1867,10 @@ deflate_finish_sequence(struct deflate_sequence *seq, unsigned litrunlen)
  * For determining whether the frequency distributions are "different enough" to
  * start a new block, the simply heuristic of splitting when the sum of absolute
  * differences exceeds a constant seems to be good enough.  We also add a number
- * proportional to the block size so that the algorithm is more likely to end
- * large blocks than small blocks.  This reflects the general expectation that
- * it will become increasingly beneficial to start a new block as the current
- * blocks grows larger.
+ * proportional to the block length so that the algorithm is more likely to end
+ * long blocks than short blocks.  This reflects the general expectation that it
+ * will become increasingly beneficial to start a new block as the current
+ * block grows longer.
  *
  * Finally, for an approximation, it is not strictly necessary that the exact
  * symbols being used are considered.  With "near-optimal parsing", for example,
@@ -1874,7 +1912,7 @@ observe_match(struct block_split_stats *stats, unsigned length)
 }
 
 static bool
-do_end_block_check(struct block_split_stats *stats, u32 block_size)
+do_end_block_check(struct block_split_stats *stats, u32 block_length)
 {
 	int i;
 
@@ -1893,8 +1931,8 @@ do_end_block_check(struct block_split_stats *stats, u32 block_size)
 		}
 
 		/* Ready to end the block? */
-		if (total_delta + (block_size >> 12) * stats->num_observations >=
-		    200 * stats->num_observations)
+		if (total_delta + (block_length / 4096) * stats->num_observations >=
+		    NUM_OBSERVATIONS_PER_BLOCK_CHECK * 200 / 512 * stats->num_observations)
 			return true;
 	}
 
@@ -1912,9 +1950,9 @@ should_end_block(struct block_split_stats *stats,
 		 const u8 *in_block_begin, const u8 *in_next, const u8 *in_end)
 {
 	/* Ready to check block split statistics? */
-	if (stats->num_new_observations < 512 ||
+	if (stats->num_new_observations < NUM_OBSERVATIONS_PER_BLOCK_CHECK ||
 	    in_next - in_block_begin < MIN_BLOCK_LENGTH ||
-	    in_end - in_next < 16384)
+	    in_end - in_next < MIN_BLOCK_LENGTH)
 		return false;
 
 	return do_end_block_check(stats, in_next - in_block_begin);
@@ -1926,7 +1964,7 @@ should_end_block(struct block_split_stats *stats,
  * This is the "greedy" DEFLATE compressor. It always chooses the longest match.
  */
 static size_t
-deflate_compress_greedy(struct deflate_compressor * restrict c,
+deflate_compress_greedy(struct libdeflate_compressor * restrict c,
 			const u8 * restrict in, size_t in_nbytes,
 			u8 * restrict out, size_t out_nbytes_avail)
 {
@@ -1945,7 +1983,8 @@ deflate_compress_greedy(struct deflate_compressor * restrict c,
 		/* Starting a new DEFLATE block.  */
 
 		const u8 * const in_block_begin = in_next;
-		const u8 * const in_max_block_end = in_next + MIN(in_end - in_next, MAX_BLOCK_LENGTH);
+		const u8 * const in_max_block_end =
+			in_next + MIN(in_end - in_next, SOFT_MAX_BLOCK_LENGTH);
 		u32 litrunlen = 0;
 		struct deflate_sequence *next_seq = c->p.g.sequences;
 
@@ -2010,7 +2049,7 @@ deflate_compress_greedy(struct deflate_compressor * restrict c,
  * literal and continues to the next position.  If no, it outputs the match.
  */
 static size_t
-deflate_compress_lazy(struct deflate_compressor * restrict c,
+deflate_compress_lazy(struct libdeflate_compressor * restrict c,
 		      const u8 * restrict in, size_t in_nbytes,
 		      u8 * restrict out, size_t out_nbytes_avail)
 {
@@ -2029,7 +2068,8 @@ deflate_compress_lazy(struct deflate_compressor * restrict c,
 		/* Starting a new DEFLATE block.  */
 
 		const u8 * const in_block_begin = in_next;
-		const u8 * const in_max_block_end = in_next + MIN(in_end - in_next, MAX_BLOCK_LENGTH);
+		const u8 * const in_max_block_end =
+			in_next + MIN(in_end - in_next, SOFT_MAX_BLOCK_LENGTH);
 		u32 litrunlen = 0;
 		struct deflate_sequence *next_seq = c->p.g.sequences;
 
@@ -2155,13 +2195,13 @@ deflate_compress_lazy(struct deflate_compressor * restrict c,
 /*
  * Follow the minimum-cost path in the graph of possible match/literal choices
  * for the current block and compute the frequencies of the Huffman symbols that
- * are needed to output those matches and literals.
+ * would be needed to output those matches and literals.
  */
 static void
-deflate_tally_item_list(struct deflate_compressor *c,
-			struct deflate_optimum_node *end_node)
+deflate_tally_item_list(struct libdeflate_compressor *c, u32 block_length)
 {
 	struct deflate_optimum_node *cur_node = &c->p.n.optimum_nodes[0];
+	struct deflate_optimum_node *end_node = &c->p.n.optimum_nodes[block_length];
 	do {
 		unsigned length = cur_node->item & OPTIMUM_LEN_MASK;
 		unsigned offset = cur_node->item >> OPTIMUM_OFFSET_SHIFT;
@@ -2180,7 +2220,8 @@ deflate_tally_item_list(struct deflate_compressor *c,
 
 /* Set the current cost model from the codeword lengths specified in @lens.  */
 static void
-deflate_set_costs(struct deflate_compressor *c, const struct deflate_lens *lens)
+deflate_set_costs_from_codes(struct libdeflate_compressor *c,
+			     const struct deflate_lens *lens)
 {
 	unsigned i;
 
@@ -2232,17 +2273,17 @@ deflate_default_offset_slot_cost(unsigned offset_slot)
 }
 
 /*
- * Set default Huffman symbol costs for the first optimization pass.
+ * Set default symbol costs for the first block's first optimization pass.
  *
- * It works well to assume that each Huffman symbol is equally probable.  This
- * results in each symbol being assigned a cost of (-log2(1.0/num_syms) * (1 <<
+ * It works well to assume that each symbol is equally probable.  This results
+ * in each symbol being assigned a cost of (-log2(1.0/num_syms) * (1 <<
  * COST_SHIFT)) where 'num_syms' is the number of symbols in the corresponding
  * alphabet.  However, we intentionally bias the parse towards matches rather
  * than literals by using a slightly lower default cost for length symbols than
  * for literals.  This often improves the compression ratio slightly.
  */
 static void
-deflate_set_default_costs(struct deflate_compressor *c)
+deflate_set_default_costs(struct libdeflate_compressor *c)
 {
 	unsigned i;
 
@@ -2276,7 +2317,7 @@ deflate_adjust_cost(u32 *cost_p, u32 default_cost)
  * defaults, but don't simply set them to the defaults.
  */
 static void
-deflate_adjust_costs(struct deflate_compressor *c)
+deflate_adjust_costs(struct libdeflate_compressor *c)
 {
 	unsigned i;
 
@@ -2297,120 +2338,131 @@ deflate_adjust_costs(struct deflate_compressor *c)
 				    deflate_default_offset_slot_cost(i));
 }
 
+/*
+ * Find the minimum-cost path through the graph of possible match/literal
+ * choices for this block.
+ *
+ * We find the minimum cost path from 'c->p.n.optimum_nodes[0]', which
+ * represents the node at the beginning of the block, to
+ * 'c->p.n.optimum_nodes[block_length]', which represents the node at the end of
+ * the block.  Edge costs are evaluated using the cost model 'c->p.n.costs'.
+ *
+ * The algorithm works backwards, starting at the end node and proceeding
+ * backwards one node at a time.  At each node, the minimum cost to reach the
+ * end node is computed and the match/literal choice that begins that path is
+ * saved.
+ */
 static void
-deflate_optimize_and_write_block(struct deflate_compressor *c,
-				 struct deflate_output_bitstream *os,
-				 const u8 * const block_begin,
-				 const u32 block_length,
-				 const struct lz_match * const end_cache_ptr,
-				 const bool is_final_block)
+deflate_find_min_cost_path(struct libdeflate_compressor *c,
+			   const u32 block_length,
+			   const struct lz_match *cache_ptr)
+{
+	struct deflate_optimum_node *end_node = &c->p.n.optimum_nodes[block_length];
+	struct deflate_optimum_node *cur_node = end_node;
+
+	cur_node->cost_to_end = 0;
+	do {
+		unsigned num_matches;
+		unsigned literal;
+		u32 best_cost_to_end;
+
+		cur_node--;
+		cache_ptr--;
+
+		num_matches = cache_ptr->length;
+		literal = cache_ptr->offset;
+
+		/* It's always possible to choose a literal.  */
+		best_cost_to_end = c->p.n.costs.literal[literal] +
+				   (cur_node + 1)->cost_to_end;
+		cur_node->item = ((u32)literal << OPTIMUM_OFFSET_SHIFT) | 1;
+
+		/* Also consider matches if there are any.  */
+		if (num_matches) {
+			const struct lz_match *match;
+			unsigned len;
+			unsigned offset;
+			unsigned offset_slot;
+			u32 offset_cost;
+			u32 cost_to_end;
+
+			/*
+			 * Consider each length from the minimum
+			 * (DEFLATE_MIN_MATCH_LEN) to the length of the longest
+			 * match found at this position.  For each length, we
+			 * consider only the smallest offset for which that
+			 * length is available.  Although this is not guaranteed
+			 * to be optimal due to the possibility of a larger
+			 * offset costing less than a smaller offset to code,
+			 * this is a very useful heuristic.
+			 */
+			match = cache_ptr - num_matches;
+			len = DEFLATE_MIN_MATCH_LEN;
+			do {
+				offset = match->offset;
+				offset_slot = deflate_get_offset_slot(c, offset);
+				offset_cost = c->p.n.costs.offset_slot[offset_slot];
+				do {
+					cost_to_end = offset_cost +
+						      c->p.n.costs.length[len] +
+						      (cur_node + len)->cost_to_end;
+					if (cost_to_end < best_cost_to_end) {
+						best_cost_to_end = cost_to_end;
+						cur_node->item = ((u32)offset << OPTIMUM_OFFSET_SHIFT) | len;
+					}
+				} while (++len <= match->length);
+			} while (++match != cache_ptr);
+			cache_ptr -= num_matches;
+		}
+		cur_node->cost_to_end = best_cost_to_end;
+	} while (cur_node != &c->p.n.optimum_nodes[0]);
+}
+
+/*
+ * Choose the literal/match sequence to use for the current block.  The basic
+ * algorithm finds a minimum-cost path through the block's graph of
+ * literal/match choices, given a cost model.  However, the cost of each symbol
+ * is unknown until the Huffman codes have been built, but at the same time the
+ * Huffman codes depend on the frequencies of chosen symbols.  Consequently,
+ * multiple passes must be used to try to approximate an optimal solution.  The
+ * first pass uses default costs, mixed with the costs from the previous block
+ * if any.  Later passes use the Huffman codeword lengths from the previous pass
+ * as the costs.
+ */
+static void
+deflate_optimize_block(struct libdeflate_compressor *c, u32 block_length,
+		       const struct lz_match *cache_ptr, bool is_first_block)
 {
-	struct deflate_optimum_node * const end_node =
-		&c->p.n.optimum_nodes[block_length];
 	unsigned num_passes_remaining = c->p.n.num_optim_passes;
 	u32 i;
 
-	/* Force the block to really end at 'end_node', even if some matches
-	 * extend beyond it.  */
+	/* Force the block to really end at the desired length, even if some
+	 * matches extend beyond it. */
 	for (i = block_length; i <= MIN(block_length - 1 + DEFLATE_MAX_MATCH_LEN,
 					ARRAY_LEN(c->p.n.optimum_nodes) - 1); i++)
 		c->p.n.optimum_nodes[i].cost_to_end = 0x80000000;
 
-	do {
-		/*
-		 * Beginning a new optimization pass and finding a new
-		 * minimum-cost path through the graph of possible match/literal
-		 * choices for this block.
-		 *
-		 * We find the minimum cost path from 'c->optimum_nodes[0]',
-		 * which represents the node at the beginning of the block, to
-		 * 'end_node', which represents the node at the end of the
-		 * block.  Edge costs are evaluated using the cost model
-		 * 'c->costs'.
-		 *
-		 * The algorithm works backward, starting at 'end_node' and
-		 * proceeding backwards one position at a time.  At each
-		 * position, the minimum cost to reach 'end_node' is computed
-		 * and the match/literal choice is saved.
-		 */
-		struct deflate_optimum_node *cur_node = end_node;
-		const struct lz_match *cache_ptr = end_cache_ptr;
+	/* Set the initial costs. */
+	if (is_first_block)
+		deflate_set_default_costs(c);
+	else
+		deflate_adjust_costs(c);
 
-		cur_node->cost_to_end = 0;
-		do {
-			unsigned num_matches;
-			unsigned literal;
-			u32 best_cost_to_end;
-			u32 best_item;
+	for (;;) {
+		/* Find the minimum cost path for this pass. */
+		deflate_find_min_cost_path(c, block_length, cache_ptr);
 
-			cur_node--;
-			cache_ptr--;
+		/* Compute frequencies of the chosen symbols. */
+		deflate_reset_symbol_frequencies(c);
+		deflate_tally_item_list(c, block_length);
 
-			num_matches = cache_ptr->length;
-			literal = cache_ptr->offset;
+		if (--num_passes_remaining == 0)
+			break;
 
-			/* It's always possible to choose a literal.  */
-			best_cost_to_end = c->p.n.costs.literal[literal] +
-					   (cur_node + 1)->cost_to_end;
-			best_item = ((u32)literal << OPTIMUM_OFFSET_SHIFT) | 1;
-
-			/* Also consider matches if there are any.  */
-			if (num_matches) {
-				const struct lz_match *match;
-				unsigned len;
-				unsigned offset;
-				unsigned offset_slot;
-				u32 offset_cost;
-				u32 cost_to_end;
-
-				/*
-				 * Consider each length from the minimum
-				 * (DEFLATE_MIN_MATCH_LEN) to the length of the
-				 * longest match found at this position.  For
-				 * each length, we consider only the smallest
-				 * offset for which that length is available.
-				 * Although this is not guaranteed to be optimal
-				 * due to the possibility of a larger offset
-				 * costing less than a smaller offset to code,
-				 * this is a very useful heuristic.
-				 */
-				match = cache_ptr - num_matches;
-				len = DEFLATE_MIN_MATCH_LEN;
-				do {
-					offset = match->offset;
-					offset_slot = deflate_get_offset_slot(c, offset);
-					offset_cost = c->p.n.costs.offset_slot[offset_slot];
-					do {
-						cost_to_end = offset_cost +
-							      c->p.n.costs.length[len] +
-							      (cur_node + len)->cost_to_end;
-						if (cost_to_end < best_cost_to_end) {
-							best_cost_to_end = cost_to_end;
-							best_item = ((u32)offset << OPTIMUM_OFFSET_SHIFT) | len;
-						}
-					} while (++len <= match->length);
-				} while (++match != cache_ptr);
-				cache_ptr -= num_matches;
-			}
-			cur_node->cost_to_end = best_cost_to_end;
-			cur_node->item = best_item;
-		} while (cur_node != &c->p.n.optimum_nodes[0]);
-
-		/* Tally Huffman symbol frequencies.  */
-		deflate_tally_item_list(c, end_node);
-
-		/* If this wasn't the last pass, update the cost model.  */
-		if (num_passes_remaining > 1) {
-			deflate_make_huffman_codes(&c->freqs, &c->codes);
-			deflate_set_costs(c, &c->codes.lens);
-			deflate_reset_symbol_frequencies(c);
-		}
-	} while (--num_passes_remaining);
-
-	/* All optimization passes are done.  Output a block using the
-	 * minimum-cost path computed on the last optimization pass.  */
-	deflate_flush_block(c, os, block_begin, block_length,
-			    is_final_block, true);
+		/* At least one optimization pass remains; update the costs. */
+		deflate_make_huffman_codes(&c->freqs, &c->codes);
+		deflate_set_costs_from_codes(c, &c->codes.lens);
+	}
 }
 
 /*
@@ -2427,7 +2479,7 @@ deflate_optimize_and_write_block(struct deflate_compressor *c,
  *   (so iterative optimization must be used)
  */
 static size_t
-deflate_compress_near_optimal(struct deflate_compressor * restrict c,
+deflate_compress_near_optimal(struct libdeflate_compressor * restrict c,
 			      const u8 * restrict in, size_t in_nbytes,
 			      u8 * restrict out, size_t out_nbytes_avail)
 {
@@ -2448,17 +2500,17 @@ deflate_compress_near_optimal(struct deflate_compressor * restrict c,
 
 		struct lz_match *cache_ptr = c->p.n.match_cache;
 		const u8 * const in_block_begin = in_next;
-		const u8 * const in_max_block_end = in_next + MIN(in_end - in_next, MAX_BLOCK_LENGTH);
+		const u8 * const in_max_block_end =
+			in_next + MIN(in_end - in_next, SOFT_MAX_BLOCK_LENGTH);
 		const u8 *next_observation = in_next;
 
 		init_block_split_stats(&c->split_stats);
-		deflate_reset_symbol_frequencies(c);
 
 		/*
 		 * Find matches until we decide to end the block.  We end the
 		 * block if any of the following is true:
 		 *
-		 * (1) Maximum block size has been reached
+		 * (1) Maximum block length has been reached
 		 * (2) Match catch may overflow.
 		 * (3) Block split heuristic says to split now.
 		 */
@@ -2556,7 +2608,6 @@ deflate_compress_near_optimal(struct deflate_compressor * restrict c,
 						bt_matchfinder_skip_position(&c->p.n.bt_mf,
 									     in_cur_base,
 									     in_next - in_cur_base,
-									     max_len,
 									     nice_len,
 									     c->max_search_depth,
 									     next_hashes);
@@ -2571,16 +2622,12 @@ deflate_compress_near_optimal(struct deflate_compressor * restrict c,
 			 cache_ptr < &c->p.n.match_cache[CACHE_LENGTH] &&
 			 !should_end_block(&c->split_stats, in_block_begin, in_next, in_end));
 
-		/* All the matches for this block have been cached.  Now compute
-		 * a near-optimal sequence of literals and matches, and output
-		 * the block.  */
-		if (in_block_begin == in)
-			deflate_set_default_costs(c);
-		else
-			deflate_adjust_costs(c);
-		deflate_optimize_and_write_block(c, &os, in_block_begin,
-						 in_next - in_block_begin,
-						 cache_ptr, in_next == in_end);
+		/* All the matches for this block have been cached.  Now choose
+		 * the sequence of items to output and flush the block.  */
+		deflate_optimize_block(c, in_next - in_block_begin, cache_ptr,
+				       in_block_begin == in);
+		deflate_flush_block(c, &os, in_block_begin, in_next - in_block_begin,
+				    in_next == in_end, true);
 	} while (in_next != in_end);
 
 	return deflate_flush_output(&os);
@@ -2590,7 +2637,7 @@ deflate_compress_near_optimal(struct deflate_compressor * restrict c,
 
 /* Initialize c->offset_slot_fast.  */
 static void
-deflate_init_offset_slot_fast(struct deflate_compressor *c)
+deflate_init_offset_slot_fast(struct libdeflate_compressor *c)
 {
 	unsigned offset_slot;
 	unsigned offset;
@@ -2622,25 +2669,23 @@ deflate_init_offset_slot_fast(struct deflate_compressor *c)
 	}
 }
 
-LIBEXPORT struct deflate_compressor *
-deflate_alloc_compressor(unsigned int compression_level)
+LIBDEFLATEAPI struct libdeflate_compressor *
+libdeflate_alloc_compressor(int compression_level)
 {
-	struct deflate_compressor *c;
+	struct libdeflate_compressor *c;
 	size_t size;
 
 #if SUPPORT_NEAR_OPTIMAL_PARSING
 	if (compression_level >= 8)
-		size = offsetof(struct deflate_compressor, p) + sizeof(c->p.n);
+		size = offsetof(struct libdeflate_compressor, p) + sizeof(c->p.n);
 	else
 #endif
-		size = offsetof(struct deflate_compressor, p) + sizeof(c->p.g);
+		size = offsetof(struct libdeflate_compressor, p) + sizeof(c->p.g);
 
 	c = aligned_malloc(MATCHFINDER_ALIGNMENT, size);
 	if (!c)
 		return NULL;
 
-	c->compression_level = compression_level;
-
 	switch (compression_level) {
 	case 1:
 		c->impl = deflate_compress_greedy;
@@ -2725,24 +2770,28 @@ deflate_alloc_compressor(unsigned int compression_level)
 		return NULL;
 	}
 
+	c->compression_level = compression_level;
+
 	deflate_init_offset_slot_fast(c);
 	deflate_init_static_codes(c);
 
 	return c;
 }
 
-LIBEXPORT size_t
-deflate_compress(struct deflate_compressor *c,
-		 const void *in, size_t in_nbytes,
-		 void *out, size_t out_nbytes_avail)
+LIBDEFLATEAPI size_t
+libdeflate_deflate_compress(struct libdeflate_compressor *c,
+			    const void *in, size_t in_nbytes,
+			    void *out, size_t out_nbytes_avail)
 {
-	if (unlikely(out_nbytes_avail < MIN_OUTPUT_SIZE))
+	if (unlikely(out_nbytes_avail < OUTPUT_END_PADDING))
 		return 0;
 
 	/* For extremely small inputs just use a single uncompressed block. */
 	if (unlikely(in_nbytes < 16)) {
 		struct deflate_output_bitstream os;
 		deflate_init_output(&os, out, out_nbytes_avail);
+		if (in_nbytes == 0)
+			in = &os; /* Avoid passing NULL to memcpy() */
 		deflate_write_uncompressed_block(&os, in, in_nbytes, true);
 		return deflate_flush_output(&os);
 	}
@@ -2750,20 +2799,21 @@ deflate_compress(struct deflate_compressor *c,
 	return (*c->impl)(c, in, in_nbytes, out, out_nbytes_avail);
 }
 
-LIBEXPORT void
-deflate_free_compressor(struct deflate_compressor *c)
+LIBDEFLATEAPI void
+libdeflate_free_compressor(struct libdeflate_compressor *c)
 {
 	aligned_free(c);
 }
 
 unsigned int
-deflate_get_compression_level(struct deflate_compressor *c)
+deflate_get_compression_level(struct libdeflate_compressor *c)
 {
 	return c->compression_level;
 }
 
-LIBEXPORT size_t
-deflate_compress_bound(struct deflate_compressor *c, size_t in_nbytes)
+LIBDEFLATEAPI size_t
+libdeflate_deflate_compress_bound(struct libdeflate_compressor *c,
+				  size_t in_nbytes)
 {
 	/*
 	 * The worst case is all uncompressed blocks where one block has length
@@ -2772,5 +2822,5 @@ deflate_compress_bound(struct deflate_compressor *c, size_t in_nbytes)
 	 * and alignment to a byte boundary; 2 for LEN; and 2 for NLEN.
 	 */
 	size_t max_num_blocks = MAX(DIV_ROUND_UP(in_nbytes, MIN_BLOCK_LENGTH), 1);
-	return (5 * max_num_blocks) + in_nbytes + 1 + MIN_OUTPUT_SIZE;
+	return (5 * max_num_blocks) + in_nbytes + 1 + OUTPUT_END_PADDING;
 }

lib/deflate_compress.h

@@ -1,11 +1,14 @@
-#ifndef _LIB_DEFLATE_COMPRESS_H
-#define _LIB_DEFLATE_COMPRESS_H
+#ifndef LIB_DEFLATE_COMPRESS_H
+#define LIB_DEFLATE_COMPRESS_H
+
+#include "lib_common.h"
 
 /* DEFLATE compression is private to deflate_compress.c, but we do need to be
  * able to query the compression level for zlib and gzip header generation.  */
 
-struct deflate_compressor;
+struct libdeflate_compressor;
 
-extern unsigned int deflate_get_compression_level(struct deflate_compressor *c);
+extern unsigned int
+deflate_get_compression_level(struct libdeflate_compressor *c);
 
-#endif /* _LIB_DEFLATE_COMPRESS_H */
+#endif /* LIB_DEFLATE_COMPRESS_H */

lib/deflate_constants.h

@@ -2,8 +2,8 @@
  * deflate_constants.h - constants for the DEFLATE compression format
  */
 
-#ifndef _LIB_DEFLATE_CONSTANTS_H
-#define _LIB_DEFLATE_CONSTANTS_H
+#ifndef LIB_DEFLATE_CONSTANTS_H
+#define LIB_DEFLATE_CONSTANTS_H
 
 /* Valid block types  */
 #define DEFLATE_BLOCKTYPE_UNCOMPRESSED		0
@@ -63,4 +63,4 @@
 	(DEFLATE_MAX_LITLEN_CODEWORD_LEN + DEFLATE_MAX_EXTRA_LENGTH_BITS + \
 	DEFLATE_MAX_OFFSET_CODEWORD_LEN + DEFLATE_MAX_EXTRA_OFFSET_BITS)
 
-#endif /* _LIB_DEFLATE_CONSTANTS_H */
+#endif /* LIB_DEFLATE_CONSTANTS_H */

lib/deflate_decompress.c

@@ -1,15 +1,28 @@
 /*
  * deflate_decompress.c - a decompressor for DEFLATE
  *
- * Written in 2014-2016 by Eric Biggers <ebiggers3@gmail.com>
+ * Copyright 2016 Eric Biggers
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  *
  * ---------------------------------------------------------------------------
  *
@@ -36,13 +49,12 @@
 
 #include "deflate_constants.h"
 #include "unaligned.h"
-#include "x86_cpu_features.h"
 
 #include "libdeflate.h"
 
 /*
  * If the expression passed to SAFETY_CHECK() evaluates to false, then the
- * decompression routine immediately returns DECOMPRESS_BAD_DATA, indicating the
+ * decompression routine immediately returns LIBDEFLATE_BAD_DATA, indicating the
  * compressed data is invalid.
  *
  * Theoretically, these checks could be disabled for specialized applications
@@ -52,7 +64,7 @@
 #  pragma message("UNSAFE DECOMPRESSION IS ENABLED. THIS MUST ONLY BE USED IF THE DECOMPRESSOR INPUT WILL ALWAYS BE TRUSTED!")
 #  define SAFETY_CHECK(expr)	(void)(expr)
 #else
-#  define SAFETY_CHECK(expr)	if (unlikely(!(expr))) return DECOMPRESS_BAD_DATA
+#  define SAFETY_CHECK(expr)	if (unlikely(!(expr))) return LIBDEFLATE_BAD_DATA
 #endif
 
 /*
@@ -98,7 +110,7 @@ typedef u8 len_t;
  * decompression state, but rather only some arrays that are too large to
  * comfortably allocate on the stack.
  */
-struct deflate_decompressor {
+struct libdeflate_decompressor {
 
 	/*
 	 * The arrays aren't all needed at the same time.  'precode_lens' and
@@ -189,17 +201,20 @@ typedef machine_word_t bitbuf_t;
 /*
  * Fill the bitbuffer variable, reading one byte at a time.
  *
- * Note: if we would overrun the input buffer, we just don't read anything,
- * leaving the bits as 0 but marking them as filled.  This makes the
- * implementation simpler because this removes the need to distinguish between
- * "real" overruns and overruns that occur because of our own lookahead during
- * Huffman decoding.  The disadvantage is that a "real" overrun can go
- * undetected, and deflate_decompress() may return a success status rather than
- * the expected failure status if one occurs.  However, this is irrelevant
- * because even if this specific case were to be handled "correctly", one could
- * easily come up with a different case where the compressed data would be
- * corrupted in such a way that fully retains its validity.  Users should run a
- * checksum against the uncompressed data if they wish to detect corruptions.
+ * If we would overread the input buffer, we just don't read anything, leaving
+ * the bits zeroed but marking them filled.  This simplifies the decompressor
+ * because it removes the need to distinguish between real overreads and
+ * overreads that occur only because of the decompressor's own lookahead.
+ *
+ * The disadvantage is that real overreads are not detected immediately.
+ * However, this is safe because the decompressor is still guaranteed to make
+ * forward progress when presented never-ending 0 bits.  In an existing block
+ * output will be getting generated, whereas new blocks can only be uncompressed
+ * (since the type code for uncompressed blocks is 0), for which we check for
+ * previous overread.  But even if we didn't check, uncompressed blocks would
+ * fail to validate because LEN would not equal ~NLEN.  So the decompressor will
+ * eventually either detect that the output buffer is full, or detect invalid
+ * input, or finish the final block.
  */
 #define FILL_BITS_BYTEWISE()					\
 do {								\
@@ -261,17 +276,19 @@ if (!HAVE_BITS(n)) {						\
 #define POP_BITS(n) (tmp32 = BITS(n), REMOVE_BITS(n), tmp32)
 
 /*
- * Align the input to the next byte boundary, discarding any remaining bits in
- * the current byte.
+ * Verify that the input buffer hasn't been overread, then align the input to
+ * the next byte boundary, discarding any remaining bits in the current byte.
  *
- * Note that if the bitbuffer variable currently contains more than 8 bits, then
+ * Note that if the bitbuffer variable currently contains more than 7 bits, then
  * we must rewind 'in_next', effectively putting those bits back.  Only the bits
  * in what would be the "current" byte if we were reading one byte at a time can
  * be actually discarded.
  */
 #define ALIGN_INPUT()							\
 do {									\
-	in_next -= (bitsleft >> 3) - MIN(overrun_count, bitsleft >> 3);	\
+	SAFETY_CHECK(overrun_count <= (bitsleft >> 3));			\
+	in_next -= (bitsleft >> 3) - overrun_count;			\
+	overrun_count = 0;						\
 	bitbuf = 0;							\
 	bitsleft = 0;							\
 } while(0)
@@ -712,7 +729,7 @@ build_decode_table(u32 decode_table[],
 
 /* Build the decode table for the precode.  */
 static bool
-build_precode_decode_table(struct deflate_decompressor *d)
+build_precode_decode_table(struct libdeflate_decompressor *d)
 {
 	/* When you change TABLEBITS, you must change ENOUGH, and vice versa! */
 	STATIC_ASSERT(PRECODE_TABLEBITS == 7 && PRECODE_ENOUGH == 128);
@@ -728,7 +745,7 @@ build_precode_decode_table(struct deflate_decompressor *d)
 
 /* Build the decode table for the literal/length code.  */
 static bool
-build_litlen_decode_table(struct deflate_decompressor *d,
+build_litlen_decode_table(struct libdeflate_decompressor *d,
 			  unsigned num_litlen_syms, unsigned num_offset_syms)
 {
 	/* When you change TABLEBITS, you must change ENOUGH, and vice versa! */
@@ -745,7 +762,7 @@ build_litlen_decode_table(struct deflate_decompressor *d,
 
 /* Build the decode table for the offset code.  */
 static bool
-build_offset_decode_table(struct deflate_decompressor *d,
+build_offset_decode_table(struct libdeflate_decompressor *d,
 			  unsigned num_litlen_syms, unsigned num_offset_syms)
 {
 	/* When you change TABLEBITS, you must change ENOUGH, and vice versa! */
@@ -784,90 +801,93 @@ copy_word_unaligned(const void *src, void *dst)
  *                         Main decompression routine
  *****************************************************************************/
 
-#define FUNCNAME deflate_decompress_default
-#define ATTRIBUTES
-#include "decompress_impl.h"
-#undef FUNCNAME
-#undef ATTRIBUTES
+typedef enum libdeflate_result (*decompress_func_t)
+	(struct libdeflate_decompressor * restrict d,
+	 const void * restrict in, size_t in_nbytes,
+	 void * restrict out, size_t out_nbytes_avail,
+	 size_t *actual_in_nbytes_ret, size_t *actual_out_nbytes_ret);
 
-#if X86_CPU_FEATURES_ENABLED && \
-	COMPILER_SUPPORTS_BMI2_TARGET && !defined(__BMI2__)
-#  define FUNCNAME deflate_decompress_bmi2
-#  define ATTRIBUTES __attribute__((target("bmi2")))
-#  include "decompress_impl.h"
-#  undef FUNCNAME
-#  undef ATTRIBUTES
-#  define DISPATCH_ENABLED 1
-#else
-#  define DISPATCH_ENABLED 0
+#undef DEFAULT_IMPL
+#undef DISPATCH
+#if defined(__i386__) || defined(__x86_64__)
+#  include "x86/decompress_impl.h"
 #endif
 
-#if DISPATCH_ENABLED
+#ifndef DEFAULT_IMPL
+#  define FUNCNAME deflate_decompress_default
+#  define ATTRIBUTES
+#  include "decompress_template.h"
+#  define DEFAULT_IMPL deflate_decompress_default
+#endif
 
-static enum decompress_result
-dispatch(struct deflate_decompressor * restrict d,
+#ifdef DISPATCH
+static enum libdeflate_result
+dispatch(struct libdeflate_decompressor * restrict d,
 	 const void * restrict in, size_t in_nbytes,
 	 void * restrict out, size_t out_nbytes_avail,
-	 size_t *actual_out_nbytes_ret);
+	 size_t *actual_in_nbytes_ret, size_t *actual_out_nbytes_ret);
 
-typedef enum decompress_result (*decompress_func_t)
-	(struct deflate_decompressor * restrict d,
+static volatile decompress_func_t decompress_impl = dispatch;
+
+/* Choose the fastest implementation at runtime */
+static enum libdeflate_result
+dispatch(struct libdeflate_decompressor * restrict d,
 	 const void * restrict in, size_t in_nbytes,
 	 void * restrict out, size_t out_nbytes_avail,
-	 size_t *actual_out_nbytes_ret);
-
-static decompress_func_t decompress_impl = dispatch;
-
-static enum decompress_result
-dispatch(struct deflate_decompressor * restrict d,
-	 const void * restrict in, size_t in_nbytes,
-	 void * restrict out, size_t out_nbytes_avail,
-	 size_t *actual_out_nbytes_ret)
+	 size_t *actual_in_nbytes_ret, size_t *actual_out_nbytes_ret)
 {
-	decompress_func_t f = deflate_decompress_default;
-#if X86_CPU_FEATURES_ENABLED
-	if (x86_have_cpu_feature(X86_CPU_FEATURE_BMI2))
-		f = deflate_decompress_bmi2;
-#endif
+	decompress_func_t f = arch_select_decompress_func();
+
+	if (f == NULL)
+		f = DEFAULT_IMPL;
+
 	decompress_impl = f;
 	return (*f)(d, in, in_nbytes, out, out_nbytes_avail,
-		    actual_out_nbytes_ret);
+		    actual_in_nbytes_ret, actual_out_nbytes_ret);
 }
-#endif /* DISPATCH_ENABLED */
+#else
+#  define decompress_impl DEFAULT_IMPL /* only one implementation, use it */
+#endif
 
 
 /*
  * This is the main DEFLATE decompression routine.  See libdeflate.h for the
  * documentation.
  *
- * Note that the real code is in decompress_impl.h.  The part here just handles
- * calling the appropriate implementation depending on the CPU features at
- * runtime.
+ * Note that the real code is in decompress_template.h.  The part here just
+ * handles calling the appropriate implementation depending on the CPU features
+ * at runtime.
  */
-LIBEXPORT enum decompress_result
-deflate_decompress(struct deflate_decompressor * restrict d,
-		   const void * restrict in, size_t in_nbytes,
-		   void * restrict out, size_t out_nbytes_avail,
-		   size_t *actual_out_nbytes_ret)
+LIBDEFLATEAPI enum libdeflate_result
+libdeflate_deflate_decompress_ex(struct libdeflate_decompressor * restrict d,
+				 const void * restrict in, size_t in_nbytes,
+				 void * restrict out, size_t out_nbytes_avail,
+				 size_t *actual_in_nbytes_ret,
+				 size_t *actual_out_nbytes_ret)
 {
-#if DISPATCH_ENABLED
-	return (*decompress_impl)(d, in, in_nbytes, out, out_nbytes_avail,
-				  actual_out_nbytes_ret);
-#else
-	return deflate_decompress_default(d, in, in_nbytes, out,
-					  out_nbytes_avail,
-					  actual_out_nbytes_ret);
-#endif
+	return decompress_impl(d, in, in_nbytes, out, out_nbytes_avail,
+			       actual_in_nbytes_ret, actual_out_nbytes_ret);
 }
 
-LIBEXPORT struct deflate_decompressor *
-deflate_alloc_decompressor(void)
+LIBDEFLATEAPI enum libdeflate_result
+libdeflate_deflate_decompress(struct libdeflate_decompressor * restrict d,
+			      const void * restrict in, size_t in_nbytes,
+			      void * restrict out, size_t out_nbytes_avail,
+			      size_t *actual_out_nbytes_ret)
 {
-	return malloc(sizeof(struct deflate_decompressor));
+	return libdeflate_deflate_decompress_ex(d, in, in_nbytes,
+						out, out_nbytes_avail,
+						NULL, actual_out_nbytes_ret);
 }
 
-LIBEXPORT void
-deflate_free_decompressor(struct deflate_decompressor *d)
+LIBDEFLATEAPI struct libdeflate_decompressor *
+libdeflate_alloc_decompressor(void)
+{
+	return malloc(sizeof(struct libdeflate_decompressor));
+}
+
+LIBDEFLATEAPI void
+libdeflate_free_decompressor(struct libdeflate_decompressor *d)
 {
 	free(d);
 }

lib/gzip_compress.c

@@ -1,27 +1,42 @@
 /*
  * gzip_compress.c - compress with a gzip wrapper
  *
- * Written in 2014-2016 by Eric Biggers <ebiggers3@gmail.com>
+ * Originally public domain; changes after 2016-09-07 are copyrighted.
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Copyright 2016 Eric Biggers
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  */
 
-#include "crc32.h"
 #include "deflate_compress.h"
 #include "gzip_constants.h"
 #include "unaligned.h"
 
 #include "libdeflate.h"
 
-LIBEXPORT size_t
-gzip_compress(struct deflate_compressor *c, const void *in, size_t in_size,
-	      void *out, size_t out_nbytes_avail)
+LIBDEFLATEAPI size_t
+libdeflate_gzip_compress(struct libdeflate_compressor *c,
+			 const void *in, size_t in_size,
+			 void *out, size_t out_nbytes_avail)
 {
 	u8 *out_next = out;
 	unsigned compression_level;
@@ -54,14 +69,14 @@ gzip_compress(struct deflate_compressor *c, const void *in, size_t in_size,
 	*out_next++ = GZIP_OS_UNKNOWN;	/* OS  */
 
 	/* Compressed data  */
-	deflate_size = deflate_compress(c, in, in_size, out_next,
+	deflate_size = libdeflate_deflate_compress(c, in, in_size, out_next,
 					out_nbytes_avail - GZIP_MIN_OVERHEAD);
 	if (deflate_size == 0)
 		return 0;
 	out_next += deflate_size;
 
 	/* CRC32 */
-	put_unaligned_le32(crc32_gzip(in, in_size), out_next);
+	put_unaligned_le32(libdeflate_crc32(0, in, in_size), out_next);
 	out_next += 4;
 
 	/* ISIZE */
@@ -71,8 +86,10 @@ gzip_compress(struct deflate_compressor *c, const void *in, size_t in_size,
 	return out_next - (u8 *)out;
 }
 
-LIBEXPORT size_t
-gzip_compress_bound(struct deflate_compressor *c, size_t in_nbytes)
+LIBDEFLATEAPI size_t
+libdeflate_gzip_compress_bound(struct libdeflate_compressor *c,
+			       size_t in_nbytes)
 {
-	return GZIP_MIN_OVERHEAD + deflate_compress_bound(c, in_nbytes);
+	return GZIP_MIN_OVERHEAD +
+	       libdeflate_deflate_compress_bound(c, in_nbytes);
 }

lib/gzip_constants.h

@@ -2,8 +2,8 @@
  * gzip_constants.h - constants for the gzip wrapper format
  */
 
-#ifndef _LIB_GZIP_CONSTANTS_H
-#define _LIB_GZIP_CONSTANTS_H
+#ifndef LIB_GZIP_CONSTANTS_H
+#define LIB_GZIP_CONSTANTS_H
 
 #define GZIP_MIN_HEADER_SIZE	10
 #define GZIP_FOOTER_SIZE	8
@@ -42,4 +42,4 @@
 #define GZIP_OS_RISCOS		13
 #define GZIP_OS_UNKNOWN		255
 
-#endif /* _LIB_GZIP_CONSTANTS_H */
+#endif /* LIB_GZIP_CONSTANTS_H */

lib/gzip_decompress.c

@@ -1,47 +1,63 @@
 /*
  * gzip_decompress.c - decompress with a gzip wrapper
  *
- * Written in 2014-2016 by Eric Biggers <ebiggers3@gmail.com>
+ * Originally public domain; changes after 2016-09-07 are copyrighted.
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Copyright 2016 Eric Biggers
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  */
 
-#include "crc32.h"
 #include "gzip_constants.h"
 #include "unaligned.h"
 
 #include "libdeflate.h"
 
-LIBEXPORT enum decompress_result
-gzip_decompress(struct deflate_decompressor *d,
-		const void *in, size_t in_nbytes,
-		void *out, size_t out_nbytes_avail,
-		size_t *actual_out_nbytes_ret)
+LIBDEFLATEAPI enum libdeflate_result
+libdeflate_gzip_decompress_ex(struct libdeflate_decompressor *d,
+			      const void *in, size_t in_nbytes,
+			      void *out, size_t out_nbytes_avail,
+			      size_t *actual_in_nbytes_ret,
+			      size_t *actual_out_nbytes_ret)
 {
 	const u8 *in_next = in;
 	const u8 * const in_end = in_next + in_nbytes;
 	u8 flg;
+	size_t actual_in_nbytes;
 	size_t actual_out_nbytes;
-	enum decompress_result result;
+	enum libdeflate_result result;
 
 	if (in_nbytes < GZIP_MIN_OVERHEAD)
-		return DECOMPRESS_BAD_DATA;
+		return LIBDEFLATE_BAD_DATA;
 
 	/* ID1 */
 	if (*in_next++ != GZIP_ID1)
-		return DECOMPRESS_BAD_DATA;
+		return LIBDEFLATE_BAD_DATA;
 	/* ID2 */
 	if (*in_next++ != GZIP_ID2)
-		return DECOMPRESS_BAD_DATA;
+		return LIBDEFLATE_BAD_DATA;
 	/* CM */
 	if (*in_next++ != GZIP_CM_DEFLATE)
-		return DECOMPRESS_BAD_DATA;
+		return LIBDEFLATE_BAD_DATA;
 	flg = *in_next++;
 	/* MTIME */
 	in_next += 4;
@@ -51,7 +67,7 @@ gzip_decompress(struct deflate_decompressor *d,
 	in_next += 1;
 
 	if (flg & GZIP_FRESERVED)
-		return DECOMPRESS_BAD_DATA;
+		return LIBDEFLATE_BAD_DATA;
 
 	/* Extra field */
 	if (flg & GZIP_FEXTRA) {
@@ -59,7 +75,7 @@ gzip_decompress(struct deflate_decompressor *d,
 		in_next += 2;
 
 		if (in_end - in_next < (u32)xlen + GZIP_FOOTER_SIZE)
-			return DECOMPRESS_BAD_DATA;
+			return LIBDEFLATE_BAD_DATA;
 
 		in_next += xlen;
 	}
@@ -69,7 +85,7 @@ gzip_decompress(struct deflate_decompressor *d,
 		while (*in_next++ != 0 && in_next != in_end)
 			;
 		if (in_end - in_next < GZIP_FOOTER_SIZE)
-			return DECOMPRESS_BAD_DATA;
+			return LIBDEFLATE_BAD_DATA;
 	}
 
 	/* File comment (zero terminated) */
@@ -77,22 +93,23 @@ gzip_decompress(struct deflate_decompressor *d,
 		while (*in_next++ != 0 && in_next != in_end)
 			;
 		if (in_end - in_next < GZIP_FOOTER_SIZE)
-			return DECOMPRESS_BAD_DATA;
+			return LIBDEFLATE_BAD_DATA;
 	}
 
 	/* CRC16 for gzip header */
 	if (flg & GZIP_FHCRC) {
 		in_next += 2;
 		if (in_end - in_next < GZIP_FOOTER_SIZE)
-			return DECOMPRESS_BAD_DATA;
+			return LIBDEFLATE_BAD_DATA;
 	}
 
 	/* Compressed data  */
-	result = deflate_decompress(d, in_next,
-				    in_end - GZIP_FOOTER_SIZE - in_next,
-				    out, out_nbytes_avail,
-				    actual_out_nbytes_ret);
-	if (result != DECOMPRESS_SUCCESS)
+	result = libdeflate_deflate_decompress_ex(d, in_next,
+					in_end - GZIP_FOOTER_SIZE - in_next,
+					out, out_nbytes_avail,
+					&actual_in_nbytes,
+					actual_out_nbytes_ret);
+	if (result != LIBDEFLATE_SUCCESS)
 		return result;
 
 	if (actual_out_nbytes_ret)
@@ -100,16 +117,32 @@ gzip_decompress(struct deflate_decompressor *d,
 	else
 		actual_out_nbytes = out_nbytes_avail;
 
-	in_next = in_end - GZIP_FOOTER_SIZE;
+	in_next += actual_in_nbytes;
 
 	/* CRC32 */
-	if (crc32_gzip(out, actual_out_nbytes) != get_unaligned_le32(in_next))
-		return DECOMPRESS_BAD_DATA;
+	if (libdeflate_crc32(0, out, actual_out_nbytes) !=
+	    get_unaligned_le32(in_next))
+		return LIBDEFLATE_BAD_DATA;
 	in_next += 4;
 
 	/* ISIZE */
 	if ((u32)actual_out_nbytes != get_unaligned_le32(in_next))
-		return DECOMPRESS_BAD_DATA;
+		return LIBDEFLATE_BAD_DATA;
+	in_next += 4;
 
-	return DECOMPRESS_SUCCESS;
+	if (actual_in_nbytes_ret)
+		*actual_in_nbytes_ret = in_next - (u8 *)in;
+
+	return LIBDEFLATE_SUCCESS;
+}
+
+LIBDEFLATEAPI enum libdeflate_result
+libdeflate_gzip_decompress(struct libdeflate_decompressor *d,
+			   const void *in, size_t in_nbytes,
+			   void *out, size_t out_nbytes_avail,
+			   size_t *actual_out_nbytes_ret)
+{
+	return libdeflate_gzip_decompress_ex(d, in, in_nbytes,
+					     out, out_nbytes_avail,
+					     NULL, actual_out_nbytes_ret);
 }

lib/hc_matchfinder.h

@@ -1,15 +1,30 @@
 /*
  * hc_matchfinder.h - Lempel-Ziv matchfinding with a hash table of linked lists
  *
- * Written in 2014-2016 by Eric Biggers <ebiggers3@gmail.com>
+ * Originally public domain; changes after 2016-09-07 are copyrighted.
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Copyright 2016 Eric Biggers
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  *
  * ---------------------------------------------------------------------------
  *
@@ -179,7 +194,7 @@ hc_matchfinder_longest_match(struct hc_matchfinder * const restrict mf,
 	const u8 *best_matchptr = in_next;
 	mf_pos_t cur_node3, cur_node4;
 	u32 hash3, hash4;
-	u32 next_seq3, next_seq4;
+	u32 next_hashseq;
 	u32 seq4;
 	const u8 *matchptr;
 	u32 len;
@@ -217,10 +232,9 @@ hc_matchfinder_longest_match(struct hc_matchfinder * const restrict mf,
 	mf->next_tab[cur_pos] = cur_node4;
 
 	/* Compute the next hash codes.  */
-	next_seq4 = load_u32_unaligned(in_next + 1);
-	next_seq3 = loaded_u32_to_u24(next_seq4);
-	next_hashes[0] = lz_hash(next_seq3, HC_MATCHFINDER_HASH3_ORDER);
-	next_hashes[1] = lz_hash(next_seq4, HC_MATCHFINDER_HASH4_ORDER);
+	next_hashseq = get_unaligned_le32(in_next + 1);
+	next_hashes[0] = lz_hash(next_hashseq & 0xFFFFFF, HC_MATCHFINDER_HASH3_ORDER);
+	next_hashes[1] = lz_hash(next_hashseq, HC_MATCHFINDER_HASH4_ORDER);
 	prefetchw(&mf->hash3_tab[next_hashes[0]]);
 	prefetchw(&mf->hash4_tab[next_hashes[1]]);
 
@@ -355,7 +369,7 @@ hc_matchfinder_skip_positions(struct hc_matchfinder * const restrict mf,
 {
 	u32 cur_pos;
 	u32 hash3, hash4;
-	u32 next_seq3, next_seq4;
+	u32 next_hashseq;
 	u32 remaining = count;
 
 	if (unlikely(count + 5 > in_end - in_next))
@@ -374,10 +388,9 @@ hc_matchfinder_skip_positions(struct hc_matchfinder * const restrict mf,
 		mf->next_tab[cur_pos] = mf->hash4_tab[hash4];
 		mf->hash4_tab[hash4] = cur_pos;
 
-		next_seq4 = load_u32_unaligned(++in_next);
-		next_seq3 = loaded_u32_to_u24(next_seq4);
-		hash3 = lz_hash(next_seq3, HC_MATCHFINDER_HASH3_ORDER);
-		hash4 = lz_hash(next_seq4, HC_MATCHFINDER_HASH4_ORDER);
+		next_hashseq = get_unaligned_le32(++in_next);
+		hash3 = lz_hash(next_hashseq & 0xFFFFFF, HC_MATCHFINDER_HASH3_ORDER);
+		hash4 = lz_hash(next_hashseq, HC_MATCHFINDER_HASH4_ORDER);
 		cur_pos++;
 	} while (--remaining);
 

lib/lib_common.h

@@ -0,0 +1,35 @@
+/*
+ * lib_common.h - internal header included by all library code
+ */
+
+#ifndef LIB_LIB_COMMON_H
+#define LIB_LIB_COMMON_H
+
+#ifdef LIBDEFLATE_H
+#  error "lib_common.h must always be included before libdeflate.h"
+   /* because BUILDING_LIBDEFLATE must be set first */
+#endif
+
+#define BUILDING_LIBDEFLATE
+
+#include "common_defs.h"
+
+/*
+ * Prefix with "_libdeflate_" all global symbols which are not part of the API.
+ * This avoids exposing overly generic names when libdeflate is built as a
+ * static library.
+ *
+ * Note that the chosen prefix is not really important and can be changed
+ * without breaking library users.  It was just chosen so that the resulting
+ * symbol names are unlikely to conflict with those from any other software.
+ * Also note that this fixup has no useful effect when libdeflate is built as a
+ * shared library, since these symbols are not exported.
+ */
+#define SYM_FIXUP(sym)			_libdeflate_##sym
+#define aligned_malloc			SYM_FIXUP(aligned_malloc)
+#define aligned_free			SYM_FIXUP(aligned_free)
+#define deflate_get_compression_level	SYM_FIXUP(deflate_get_compression_level)
+#define _cpu_features			SYM_FIXUP(_cpu_features)
+#define setup_cpu_features		SYM_FIXUP(setup_cpu_features)
+
+#endif /* LIB_LIB_COMMON_H */

lib/matchfinder_avx2.h

@@ -1,53 +0,0 @@
-/*
- * matchfinder_avx2.h - matchfinding routines optimized for Intel AVX2 (Advanced
- * Vector Extensions)
- */
-
-#include <immintrin.h>
-
-static forceinline bool
-matchfinder_init_avx2(mf_pos_t *data, size_t size)
-{
-	__m256i v, *p;
-	size_t n;
-
-	if (size % sizeof(__m256i) * 4)
-		return false;
-
-	STATIC_ASSERT(sizeof(mf_pos_t) == 2);
-	v = _mm256_set1_epi16(MATCHFINDER_INITVAL);
-	p = (__m256i *)data;
-	n = size / (sizeof(__m256i) * 4);
-	do {
-		p[0] = v;
-		p[1] = v;
-		p[2] = v;
-		p[3] = v;
-		p += 4;
-	} while (--n);
-	return true;
-}
-
-static forceinline bool
-matchfinder_rebase_avx2(mf_pos_t *data, size_t size)
-{
-	__m256i v, *p;
-	size_t n;
-
-	if ((size % sizeof(__m256i) * 4 != 0))
-		return false;
-
-	STATIC_ASSERT(sizeof(mf_pos_t) == 2);
-	v = _mm256_set1_epi16((u16)-MATCHFINDER_WINDOW_SIZE);
-	p = (__m256i *)data;
-	n = size / (sizeof(__m256i) * 4);
-	do {
-		/* PADDSW: Add Packed Signed Integers With Signed Saturation  */
-		p[0] = _mm256_adds_epi16(p[0], v);
-		p[1] = _mm256_adds_epi16(p[1], v);
-		p[2] = _mm256_adds_epi16(p[2], v);
-		p[3] = _mm256_adds_epi16(p[3], v);
-		p += 4;
-	} while (--n);
-	return true;
-}

lib/matchfinder_common.h

@@ -2,10 +2,10 @@
  * matchfinder_common.h - common code for Lempel-Ziv matchfinding
  */
 
-#ifndef _LIB_MATCHFINDER_COMMON_H
-#define _LIB_MATCHFINDER_COMMON_H
+#ifndef LIB_MATCHFINDER_COMMON_H
+#define LIB_MATCHFINDER_COMMON_H
 
-#include "common_defs.h"
+#include "lib_common.h"
 #include "unaligned.h"
 
 #ifndef MATCHFINDER_WINDOW_ORDER
@@ -20,27 +20,14 @@ typedef s16 mf_pos_t;
 
 #define MATCHFINDER_ALIGNMENT 8
 
-#ifdef __AVX2__
-#  include "matchfinder_avx2.h"
-#  if MATCHFINDER_ALIGNMENT < 32
-#    undef MATCHFINDER_ALIGNMENT
-#    define MATCHFINDER_ALIGNMENT 32
-#  endif
-#endif
+#define arch_matchfinder_init(data, size)	false
+#define arch_matchfinder_rebase(data, size)	false
 
-#ifdef __SSE2__
-#  include "matchfinder_sse2.h"
-#  if MATCHFINDER_ALIGNMENT < 16
-#    undef MATCHFINDER_ALIGNMENT
-#    define MATCHFINDER_ALIGNMENT 16
-#  endif
-#endif
-
-#ifdef __ARM_NEON__
-#  include "matchfinder_neon.h"
-#  if MATCHFINDER_ALIGNMENT < 16
-#    undef MATCHFINDER_ALIGNMENT
-#    define MATCHFINDER_ALIGNMENT 16
+#ifdef _aligned_attribute
+#  if defined(__arm__) || defined(__aarch64__)
+#    include "arm/matchfinder_impl.h"
+#  elif defined(__i386__) || defined(__x86_64__)
+#    include "x86/matchfinder_impl.h"
 #  endif
 #endif
 
@@ -56,20 +43,8 @@ matchfinder_init(mf_pos_t *data, size_t num_entries)
 {
 	size_t i;
 
-#if defined(__AVX2__) && defined(_aligned_attribute)
-	if (matchfinder_init_avx2(data, num_entries * sizeof(data[0])))
+	if (arch_matchfinder_init(data, num_entries * sizeof(data[0])))
 		return;
-#endif
-
-#if defined(__SSE2__) && defined(_aligned_attribute)
-	if (matchfinder_init_sse2(data, num_entries * sizeof(data[0])))
-		return;
-#endif
-
-#if defined(__ARM_NEON__) && defined(_aligned_attribute)
-	if (matchfinder_init_neon(data, num_entries * sizeof(data[0])))
-		return;
-#endif
 
 	for (i = 0; i < num_entries; i++)
 		data[i] = MATCHFINDER_INITVAL;
@@ -101,20 +76,8 @@ matchfinder_rebase(mf_pos_t *data, size_t num_entries)
 {
 	size_t i;
 
-#if defined(__AVX2__) && defined(_aligned_attribute)
-	if (matchfinder_rebase_avx2(data, num_entries * sizeof(data[0])))
+	if (arch_matchfinder_rebase(data, num_entries * sizeof(data[0])))
 		return;
-#endif
-
-#if defined(__SSE2__) && defined(_aligned_attribute)
-	if (matchfinder_rebase_sse2(data, num_entries * sizeof(data[0])))
-		return;
-#endif
-
-#if defined(__ARM_NEON__) && defined(_aligned_attribute)
-	if (matchfinder_rebase_neon(data, num_entries * sizeof(data[0])))
-		return;
-#endif
 
 	if (MATCHFINDER_WINDOW_SIZE == 32768) {
 		/* Branchless version for 32768 byte windows.  If the value was
@@ -202,4 +165,4 @@ word_differs:
 	return len;
 }
 
-#endif /* _LIB_MATCHFINDER_COMMON_H */
+#endif /* LIB_MATCHFINDER_COMMON_H */

lib/matchfinder_neon.h

@@ -1,61 +0,0 @@
-/*
- * matchfinder_neon.h - matchfinding routines optimized for ARM NEON (Advanced
- * SIMD) instructions
- */
-
-#include <arm_neon.h>
-
-static forceinline bool
-matchfinder_init_neon(mf_pos_t *data, size_t size)
-{
-	int16x8_t v, *p;
-	size_t n;
-
-	if (size % sizeof(int16x8_t) * 4)
-		return false;
-
-	STATIC_ASSERT(sizeof(mf_pos_t) == 2);
-	v = (int16x8_t) {
-		MATCHFINDER_INITVAL, MATCHFINDER_INITVAL, MATCHFINDER_INITVAL,
-		MATCHFINDER_INITVAL, MATCHFINDER_INITVAL, MATCHFINDER_INITVAL,
-		MATCHFINDER_INITVAL, MATCHFINDER_INITVAL,
-	};
-	p = (int16x8_t *)data;
-	n = size / (sizeof(int16x8_t) * 4);
-	do {
-		p[0] = v;
-		p[1] = v;
-		p[2] = v;
-		p[3] = v;
-		p += 4;
-	} while (--n);
-	return true;
-}
-
-static forceinline bool
-matchfinder_rebase_neon(mf_pos_t *data, size_t size)
-{
-	int16x8_t v, *p;
-	size_t n;
-
-	if ((size % sizeof(int16x8_t) * 4 != 0))
-		return false;
-
-	STATIC_ASSERT(sizeof(mf_pos_t) == 2);
-	v = (int16x8_t) {
-		(u16)-MATCHFINDER_WINDOW_SIZE, (u16)-MATCHFINDER_WINDOW_SIZE,
-		(u16)-MATCHFINDER_WINDOW_SIZE, (u16)-MATCHFINDER_WINDOW_SIZE,
-		(u16)-MATCHFINDER_WINDOW_SIZE, (u16)-MATCHFINDER_WINDOW_SIZE,
-		(u16)-MATCHFINDER_WINDOW_SIZE, (u16)-MATCHFINDER_WINDOW_SIZE,
-	};
-	p = (int16x8_t *)data;
-	n = size / (sizeof(int16x8_t) * 4);
-	do {
-		p[0] = vqaddq_s16(p[0], v);
-		p[1] = vqaddq_s16(p[1], v);
-		p[2] = vqaddq_s16(p[2], v);
-		p[3] = vqaddq_s16(p[3], v);
-		p += 4;
-	} while (--n);
-	return true;
-}

lib/matchfinder_sse2.h

@@ -1,53 +0,0 @@
-/*
- * matchfinder_sse2.h - matchfinding routines optimized for Intel SSE2
- * (Streaming SIMD Extensions).
- */
-
-#include <emmintrin.h>
-
-static forceinline bool
-matchfinder_init_sse2(mf_pos_t *data, size_t size)
-{
-	__m128i v, *p;
-	size_t n;
-
-	if (size % sizeof(__m128i) * 4)
-		return false;
-
-	STATIC_ASSERT(sizeof(mf_pos_t) == 2);
-	v = _mm_set1_epi16(MATCHFINDER_INITVAL);
-	p = (__m128i *)data;
-	n = size / (sizeof(__m128i) * 4);
-	do {
-		p[0] = v;
-		p[1] = v;
-		p[2] = v;
-		p[3] = v;
-		p += 4;
-	} while (--n);
-	return true;
-}
-
-static forceinline bool
-matchfinder_rebase_sse2(mf_pos_t *data, size_t size)
-{
-	__m128i v, *p;
-	size_t n;
-
-	if ((size % sizeof(__m128i) * 4 != 0))
-		return false;
-
-	STATIC_ASSERT(sizeof(mf_pos_t) == 2);
-	v = _mm_set1_epi16((u16)-MATCHFINDER_WINDOW_SIZE);
-	p = (__m128i *)data;
-	n = size / (sizeof(__m128i) * 4);
-	do {
-		/* PADDSW: Add Packed Signed Integers With Signed Saturation  */
-		p[0] = _mm_adds_epi16(p[0], v);
-		p[1] = _mm_adds_epi16(p[1], v);
-		p[2] = _mm_adds_epi16(p[2], v);
-		p[3] = _mm_adds_epi16(p[3], v);
-		p += 4;
-	} while (--n);
-	return true;
-}

lib/unaligned.h

@@ -5,7 +5,7 @@
 #ifndef LIB_UNALIGNED_H
 #define LIB_UNALIGNED_H
 
-#include "common_defs.h"
+#include "lib_common.h"
 
 /*
  * Naming note:

lib/x86/adler32_impl.h

@@ -0,0 +1,198 @@
+/*
+ * x86/adler32_impl.h - x86 implementations of Adler-32 checksum algorithm
+ *
+ * Copyright 2016 Eric Biggers
+ *
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include "cpu_features.h"
+
+/* AVX2 implementation */
+#undef DISPATCH_AVX2
+#if !defined(DEFAULT_IMPL) &&	\
+	(defined(__AVX2__) || (X86_CPU_FEATURES_ENABLED &&	\
+			       COMPILER_SUPPORTS_AVX2_TARGET_INTRINSICS))
+#  define FUNCNAME		adler32_avx2
+#  define FUNCNAME_CHUNK	adler32_avx2_chunk
+#  define IMPL_ALIGNMENT	32
+#  define IMPL_SEGMENT_SIZE	32
+#  define IMPL_MAX_CHUNK_SIZE	MAX_CHUNK_SIZE
+#  ifdef __AVX2__
+#    define ATTRIBUTES
+#    define DEFAULT_IMPL	adler32_avx2
+#  else
+#    define ATTRIBUTES		__attribute__((target("avx2")))
+#    define DISPATCH		1
+#    define DISPATCH_AVX2	1
+#  endif
+#  include <immintrin.h>
+static forceinline ATTRIBUTES void
+adler32_avx2_chunk(const __m256i *p, const __m256i *const end, u32 *s1, u32 *s2)
+{
+	const __m256i zeroes = _mm256_setzero_si256();
+	const __v32qi multipliers = (__v32qi) { 32, 31, 30, 29, 28, 27, 26, 25,
+						24, 23, 22, 21, 20, 19, 18, 17,
+						16, 15, 14, 13, 12, 11, 10, 9,
+						8,  7,  6,  5,  4,  3,  2,  1 };
+	const __v16hi ones = (__v16hi)_mm256_set1_epi16(1);
+	__v8si v_s1 = (__v8si)zeroes;
+	__v8si v_s1_sums = (__v8si)zeroes;
+	__v8si v_s2 = (__v8si)zeroes;
+
+	do {
+		__m256i bytes = *p++;
+		__v16hi sums = (__v16hi)_mm256_maddubs_epi16(
+						bytes, (__m256i)multipliers);
+		v_s1_sums += v_s1;
+		v_s1 += (__v8si)_mm256_sad_epu8(bytes, zeroes);
+		v_s2 += (__v8si)_mm256_madd_epi16((__m256i)sums, (__m256i)ones);
+	} while (p != end);
+
+	v_s1 = (__v8si)_mm256_hadd_epi32((__m256i)v_s1, zeroes);
+	v_s1 = (__v8si)_mm256_hadd_epi32((__m256i)v_s1, zeroes);
+	*s1 += (u32)v_s1[0] + (u32)v_s1[4];
+
+	v_s2 += (__v8si)_mm256_slli_epi32((__m256i)v_s1_sums, 5);
+	v_s2 = (__v8si)_mm256_hadd_epi32((__m256i)v_s2, zeroes);
+	v_s2 = (__v8si)_mm256_hadd_epi32((__m256i)v_s2, zeroes);
+	*s2 += (u32)v_s2[0] + (u32)v_s2[4];
+}
+#  include "../adler32_vec_template.h"
+#endif /* AVX2 implementation */
+
+/* SSE2 implementation */
+#undef DISPATCH_SSE2
+#if !defined(DEFAULT_IMPL) &&	\
+	(defined(__SSE2__) || (X86_CPU_FEATURES_ENABLED &&	\
+			       COMPILER_SUPPORTS_SSE2_TARGET_INTRINSICS))
+#  define FUNCNAME		adler32_sse2
+#  define FUNCNAME_CHUNK	adler32_sse2_chunk
+#  define IMPL_ALIGNMENT	16
+#  define IMPL_SEGMENT_SIZE	32
+/*
+ * The 16-bit precision byte counters must not be allowed to undergo *signed*
+ * overflow, otherwise the signed multiplications at the end (_mm_madd_epi16)
+ * would behave incorrectly.
+ */
+#  define IMPL_MAX_CHUNK_SIZE	(32 * (0x7FFF / 0xFF))
+#  ifdef __SSE2__
+#    define ATTRIBUTES
+#    define DEFAULT_IMPL	adler32_sse2
+#  else
+#    define ATTRIBUTES		__attribute__((target("sse2")))
+#    define DISPATCH		1
+#    define DISPATCH_SSE2	1
+#  endif
+#  include <emmintrin.h>
+static forceinline ATTRIBUTES void
+adler32_sse2_chunk(const __m128i *p, const __m128i *const end, u32 *s1, u32 *s2)
+{
+	const __m128i zeroes = _mm_setzero_si128();
+
+	/* s1 counters: 32-bit, sum of bytes */
+	__v4si v_s1 = (__v4si)zeroes;
+
+	/* s2 counters: 32-bit, sum of s1 values */
+	__v4si v_s2 = (__v4si)zeroes;
+
+	/*
+	 * Thirty-two 16-bit counters for byte sums.  Each accumulates the bytes
+	 * that eventually need to be multiplied by a number 32...1 for addition
+	 * into s2.
+	 */
+	__v8hi v_byte_sums_a = (__v8hi)zeroes;
+	__v8hi v_byte_sums_b = (__v8hi)zeroes;
+	__v8hi v_byte_sums_c = (__v8hi)zeroes;
+	__v8hi v_byte_sums_d = (__v8hi)zeroes;
+
+	do {
+		/* Load the next 32 bytes */
+		const __m128i bytes1 = *p++;
+		const __m128i bytes2 = *p++;
+
+		/*
+		 * Accumulate the previous s1 counters into the s2 counters.
+		 * Logically, this really should be v_s2 += v_s1 * 32, but we
+		 * can do the multiplication (or left shift) later.
+		 */
+		v_s2 += v_s1;
+
+		/*
+		 * s1 update: use "Packed Sum of Absolute Differences" to add
+		 * the bytes horizontally with 8 bytes per sum.  Then add the
+		 * sums to the s1 counters.
+		 */
+		v_s1 += (__v4si)_mm_sad_epu8(bytes1, zeroes);
+		v_s1 += (__v4si)_mm_sad_epu8(bytes2, zeroes);
+
+		/*
+		 * Also accumulate the bytes into 32 separate counters that have
+		 * 16-bit precision.
+		 */
+		v_byte_sums_a += (__v8hi)_mm_unpacklo_epi8(bytes1, zeroes);
+		v_byte_sums_b += (__v8hi)_mm_unpackhi_epi8(bytes1, zeroes);
+		v_byte_sums_c += (__v8hi)_mm_unpacklo_epi8(bytes2, zeroes);
+		v_byte_sums_d += (__v8hi)_mm_unpackhi_epi8(bytes2, zeroes);
+
+	} while (p != end);
+
+	/* Finish calculating the s2 counters */
+	v_s2 = (__v4si)_mm_slli_epi32((__m128i)v_s2, 5);
+	v_s2 += (__v4si)_mm_madd_epi16((__m128i)v_byte_sums_a,
+				       (__m128i)(__v8hi){ 32, 31, 30, 29, 28, 27, 26, 25 });
+	v_s2 += (__v4si)_mm_madd_epi16((__m128i)v_byte_sums_b,
+				       (__m128i)(__v8hi){ 24, 23, 22, 21, 20, 19, 18, 17 });
+	v_s2 += (__v4si)_mm_madd_epi16((__m128i)v_byte_sums_c,
+				       (__m128i)(__v8hi){ 16, 15, 14, 13, 12, 11, 10, 9 });
+	v_s2 += (__v4si)_mm_madd_epi16((__m128i)v_byte_sums_d,
+				       (__m128i)(__v8hi){ 8,  7,  6,  5,  4,  3,  2,  1 });
+
+	/* Now accumulate what we computed into the real s1 and s2 */
+	v_s1 += (__v4si)_mm_shuffle_epi32((__m128i)v_s1, 0x31);
+	v_s1 += (__v4si)_mm_shuffle_epi32((__m128i)v_s1, 0x02);
+	*s1 += _mm_cvtsi128_si32((__m128i)v_s1);
+
+	v_s2 += (__v4si)_mm_shuffle_epi32((__m128i)v_s2, 0x31);
+	v_s2 += (__v4si)_mm_shuffle_epi32((__m128i)v_s2, 0x02);
+	*s2 += _mm_cvtsi128_si32((__m128i)v_s2);
+}
+#  include "../adler32_vec_template.h"
+#endif /* SSE2 implementation */
+
+#ifdef DISPATCH
+static inline adler32_func_t
+arch_select_adler32_func(void)
+{
+	u32 features = get_cpu_features();
+
+#ifdef DISPATCH_AVX2
+	if (features & X86_CPU_FEATURE_AVX2)
+		return adler32_avx2;
+#endif
+#ifdef DISPATCH_SSE2
+	if (features & X86_CPU_FEATURE_SSE2)
+		return adler32_sse2;
+#endif
+	return NULL;
+}
+#endif /* DISPATCH */

lib/x86/cpu_features.c

@@ -1,28 +1,35 @@
 /*
- * x86_cpu_features.c - feature detection for x86 processors
+ * x86/cpu_features.c - feature detection for x86 processors
  *
- * Written in 2015 by Eric Biggers <ebiggers3@gmail.com>
+ * Copyright 2016 Eric Biggers
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  */
 
-#include "x86_cpu_features.h"
+#include "cpu_features.h"
 
 #if X86_CPU_FEATURES_ENABLED
 
-#define DEBUG 0
-
-#if DEBUG
-#  include <stdio.h>
-#endif
-
-u32 _x86_cpu_features = 0;
+volatile u32 _cpu_features = 0;
 
 /* With old GCC versions we have to manually save and restore the x86_32 PIC
  * register (ebx).  See: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=47602  */
@@ -58,9 +65,8 @@ read_xcr(u32 index)
 
 #define IS_SET(reg, bit) ((reg) & ((u32)1 << (bit)))
 
-/* Initialize _x86_cpu_features with bits for interesting processor features. */
-void
-x86_setup_cpu_features(void)
+/* Initialize _cpu_features with bits for interesting processor features. */
+void setup_cpu_features(void)
 {
 	u32 features = 0;
 	u32 dummy1, dummy2, dummy3, dummy4;
@@ -76,23 +82,11 @@ x86_setup_cpu_features(void)
 	/* Standard feature flags  */
 	cpuid(1, 0, &dummy1, &dummy2, &features_2, &features_1);
 
-	if (IS_SET(features_1, 25))
-		features |= X86_CPU_FEATURE_SSE;
-
 	if (IS_SET(features_1, 26))
 		features |= X86_CPU_FEATURE_SSE2;
 
-	if (IS_SET(features_2, 0))
-		features |= X86_CPU_FEATURE_SSE3;
-
-	if (IS_SET(features_2, 9))
-		features |= X86_CPU_FEATURE_SSSE3;
-
-	if (IS_SET(features_2, 19))
-		features |= X86_CPU_FEATURE_SSE4_1;
-
-	if (IS_SET(features_2, 20))
-		features |= X86_CPU_FEATURE_SSE4_2;
+	if (IS_SET(features_2, 1))
+		features |= X86_CPU_FEATURE_PCLMULQDQ;
 
 	if (IS_SET(features_2, 27)) /* OSXSAVE set?  */
 		if ((read_xcr(0) & 0x6) == 0x6)
@@ -107,9 +101,6 @@ x86_setup_cpu_features(void)
 	/* Extended feature flags  */
 	cpuid(7, 0, &dummy1, &features_3, &features_4, &dummy4);
 
-	if (IS_SET(features_3, 3))
-		features |= X86_CPU_FEATURE_BMI;
-
 	if (os_saves_ymm_regs && IS_SET(features_3, 5))
 		features |= X86_CPU_FEATURE_AVX2;
 
@@ -117,33 +108,7 @@ x86_setup_cpu_features(void)
 		features |= X86_CPU_FEATURE_BMI2;
 
 out:
-
-#if DEBUG
-	printf("Detected x86 CPU features: ");
-	if (features & X86_CPU_FEATURE_SSE)
-		printf("SSE ");
-	if (features & X86_CPU_FEATURE_SSE2)
-		printf("SSE2 ");
-	if (features & X86_CPU_FEATURE_SSE3)
-		printf("SSE3 ");
-	if (features & X86_CPU_FEATURE_SSSE3)
-		printf("SSSE3 ");
-	if (features & X86_CPU_FEATURE_SSE4_1)
-		printf("SSE4.1 ");
-	if (features & X86_CPU_FEATURE_SSE4_2)
-		printf("SSE4.2 ");
-	if (features & X86_CPU_FEATURE_BMI)
-		printf("BMI ");
-	if (features & X86_CPU_FEATURE_AVX)
-		printf("AVX ");
-	if (features & X86_CPU_FEATURE_BMI2)
-		printf("BMI2 ");
-	if (features & X86_CPU_FEATURE_AVX2)
-		printf("AVX2 ");
-	printf("\n");
-#endif /* DEBUG */
-
-	_x86_cpu_features = features | X86_CPU_FEATURES_KNOWN;
+	_cpu_features = features | X86_CPU_FEATURES_KNOWN;
 }
 
 #endif /* X86_CPU_FEATURES_ENABLED */

lib/x86/cpu_features.h

@@ -0,0 +1,40 @@
+/*
+ * x86/cpu_features.h - feature detection for x86 processors
+ */
+
+#ifndef LIB_X86_CPU_FEATURES_H
+#define LIB_X86_CPU_FEATURES_H
+
+#include "../lib_common.h"
+
+#if (defined(__i386__) || defined(__x86_64__)) && \
+	COMPILER_SUPPORTS_TARGET_FUNCTION_ATTRIBUTE
+#  define X86_CPU_FEATURES_ENABLED 1
+#else
+#  define X86_CPU_FEATURES_ENABLED 0
+#endif
+
+#if X86_CPU_FEATURES_ENABLED
+
+#define X86_CPU_FEATURE_SSE2		0x00000001
+#define X86_CPU_FEATURE_PCLMULQDQ	0x00000002
+#define X86_CPU_FEATURE_AVX		0x00000004
+#define X86_CPU_FEATURE_AVX2		0x00000008
+#define X86_CPU_FEATURE_BMI2		0x00000010
+
+#define X86_CPU_FEATURES_KNOWN		0x80000000
+
+extern volatile u32 _cpu_features;
+
+extern void setup_cpu_features(void);
+
+static inline u32 get_cpu_features(void)
+{
+	if (_cpu_features == 0)
+		setup_cpu_features();
+	return _cpu_features;
+}
+
+#endif /* X86_CPU_FEATURES_ENABLED */
+
+#endif /* LIB_X86_CPU_FEATURES_H */

lib/x86/crc32_impl.h

@@ -0,0 +1,87 @@
+/*
+ * x86/crc32_impl.h - x86 implementations of CRC-32 checksum algorithm
+ *
+ * Copyright 2016 Eric Biggers
+ *
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include "cpu_features.h"
+
+/*
+ * Include the PCLMUL/AVX implementation?  Although our PCLMUL-optimized CRC-32
+ * function doesn't use any AVX intrinsics specifically, it can benefit a lot
+ * from being compiled for an AVX target: on Skylake, ~16700 MB/s vs. ~10100
+ * MB/s.  I expect this is related to the PCLMULQDQ instructions being assembled
+ * in the newer three-operand form rather than the older two-operand form.
+ *
+ * Note: this is only needed if __AVX__ is *not* defined, since otherwise the
+ * "regular" PCLMUL implementation would already be AVX enabled.
+ */
+#undef DISPATCH_PCLMUL_AVX
+#if !defined(DEFAULT_IMPL) && !defined(__AVX__) &&	\
+	X86_CPU_FEATURES_ENABLED && COMPILER_SUPPORTS_AVX_TARGET &&	\
+	(defined(__PCLMUL__) || COMPILER_SUPPORTS_PCLMUL_TARGET_INTRINSICS)
+#  define FUNCNAME		crc32_pclmul_avx
+#  define FUNCNAME_ALIGNED	crc32_pclmul_avx_aligned
+#  define ATTRIBUTES		__attribute__((target("pclmul,avx")))
+#  define DISPATCH		1
+#  define DISPATCH_PCLMUL_AVX	1
+#  include "crc32_pclmul_template.h"
+#endif
+
+/* PCLMUL implementation */
+#undef DISPATCH_PCLMUL
+#if !defined(DEFAULT_IMPL) &&	\
+	(defined(__PCLMUL__) || (X86_CPU_FEATURES_ENABLED &&	\
+				 COMPILER_SUPPORTS_PCLMUL_TARGET_INTRINSICS))
+#  define FUNCNAME		crc32_pclmul
+#  define FUNCNAME_ALIGNED	crc32_pclmul_aligned
+#  ifdef __PCLMUL__
+#    define ATTRIBUTES
+#    define DEFAULT_IMPL	crc32_pclmul
+#  else
+#    define ATTRIBUTES		__attribute__((target("pclmul")))
+#    define DISPATCH		1
+#    define DISPATCH_PCLMUL	1
+#  endif
+#  include "crc32_pclmul_template.h"
+#endif
+
+#ifdef DISPATCH
+static inline crc32_func_t
+arch_select_crc32_func(void)
+{
+	u32 features = get_cpu_features();
+
+#ifdef DISPATCH_PCLMUL_AVX
+	if ((features & X86_CPU_FEATURE_PCLMULQDQ) &&
+	    (features & X86_CPU_FEATURE_AVX))
+		return crc32_pclmul_avx;
+#endif
+#ifdef DISPATCH_PCLMUL
+	if (features & X86_CPU_FEATURE_PCLMULQDQ)
+		return crc32_pclmul;
+#endif
+	return NULL;
+}
+#endif /* DISPATCH */

lib/x86/crc32_pclmul_template.h

@@ -0,0 +1,262 @@
+/*
+ * x86/crc32_pclmul_template.h
+ *
+ * Copyright 2016 Eric Biggers
+ *
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include <wmmintrin.h>
+
+/*
+ * CRC-32 folding with PCLMULQDQ.
+ *
+ * The basic idea is to repeatedly "fold" each 512 bits into the next 512 bits,
+ * producing an abbreviated message which is congruent the original message
+ * modulo the generator polynomial G(x).
+ *
+ * Folding each 512 bits is implemented as eight 64-bit folds, each of which
+ * uses one carryless multiplication instruction.  It's expected that CPUs may
+ * be able to execute some of these multiplications in parallel.
+ *
+ * Explanation of "folding": let A(x) be 64 bits from the message, and let B(x)
+ * be 95 bits from a constant distance D later in the message.  The relevant
+ * portion of the message can be written as:
+ *
+ *	M(x) = A(x)*x^D + B(x)
+ *
+ * ... where + and * represent addition and multiplication, respectively, of
+ * polynomials over GF(2).  Note that when implemented on a computer, these
+ * operations are equivalent to XOR and carryless multiplication, respectively.
+ *
+ * For the purpose of CRC calculation, only the remainder modulo the generator
+ * polynomial G(x) matters:
+ *
+ *	M(x) mod G(x) = (A(x)*x^D + B(x)) mod G(x)
+ *
+ * Since the modulo operation can be applied anywhere in a sequence of additions
+ * and multiplications without affecting the result, this is equivalent to:
+ *
+ *	M(x) mod G(x) = (A(x)*(x^D mod G(x)) + B(x)) mod G(x)
+ *
+ * For any D, 'x^D mod G(x)' will be a polynomial with maximum degree 31, i.e.
+ * a 32-bit quantity.  So 'A(x) * (x^D mod G(x))' is equivalent to a carryless
+ * multiplication of a 64-bit quantity by a 32-bit quantity, producing a 95-bit
+ * product.  Then, adding (XOR-ing) the product to B(x) produces a polynomial
+ * with the same length as B(x) but with the same remainder as 'A(x)*x^D +
+ * B(x)'.  This is the basic fold operation with 64 bits.
+ *
+ * Note that the carryless multiplication instruction PCLMULQDQ actually takes
+ * two 64-bit inputs and produces a 127-bit product in the low-order bits of a
+ * 128-bit XMM register.  This works fine, but care must be taken to account for
+ * "bit endianness".  With the CRC version implemented here, bits are always
+ * ordered such that the lowest-order bit represents the coefficient of highest
+ * power of x and the highest-order bit represents the coefficient of the lowest
+ * power of x.  This is backwards from the more intuitive order.  Still,
+ * carryless multiplication works essentially the same either way.  It just must
+ * be accounted for that when we XOR the 95-bit product in the low-order 95 bits
+ * of a 128-bit XMM register into 128-bits of later data held in another XMM
+ * register, we'll really be XOR-ing the product into the mathematically higher
+ * degree end of those later bits, not the lower degree end as may be expected.
+ *
+ * So given that caveat and the fact that we process 512 bits per iteration, the
+ * 'D' values we need for the two 64-bit halves of each 128 bits of data are:
+ *
+ *	D = (512 + 95) - 64	 for the higher-degree half of each 128 bits,
+ *				 i.e. the lower order bits in the XMM register
+ *
+ *	D = (512 + 95) - 128	 for the lower-degree half of each 128 bits,
+ *				 i.e. the higher order bits in the XMM register
+ *
+ * The required 'x^D mod G(x)' values were precomputed.
+ *
+ * When <= 512 bits remain in the message, we finish up by folding across
+ * smaller distances.  This works similarly; the distance D is just different,
+ * so different constant multipliers must be used.  Finally, once the remaining
+ * message is just 64 bits, it is is reduced to the CRC-32 using Barrett
+ * reduction (explained later).
+ *
+ * For more information see the original paper from Intel:
+ *	"Fast CRC Computation for Generic Polynomials Using PCLMULQDQ Instruction"
+ *	December 2009
+ *	http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
+ */
+static u32 ATTRIBUTES
+FUNCNAME_ALIGNED(u32 remainder, const __m128i *p, size_t nr_segs)
+{
+	/* Constants precomputed by gen_crc32_multipliers.c.  Do not edit! */
+	const __v2di multipliers_4 = (__v2di){ 0x8F352D95, 0x1D9513D7 };
+	const __v2di multipliers_2 = (__v2di){ 0xF1DA05AA, 0x81256527 };
+	const __v2di multipliers_1 = (__v2di){ 0xAE689191, 0xCCAA009E };
+	const __v2di final_multiplier = (__v2di){ 0xB8BC6765 };
+	const __m128i mask32 = (__m128i)(__v4si){ 0xFFFFFFFF };
+	const __v2di barrett_reduction_constants =
+			(__v2di){ 0x00000001F7011641, 0x00000001DB710641 };
+
+	const __m128i * const end = p + nr_segs;
+	const __m128i * const end512 = p + (nr_segs & ~3);
+	__m128i x0, x1, x2, x3;
+
+	/*
+	 * Account for the current 'remainder', i.e. the CRC of the part of the
+	 * message already processed.  Explanation: rewrite the message
+	 * polynomial M(x) in terms of the first part A(x), the second part
+	 * B(x), and the length of the second part in bits |B(x)| >= 32:
+	 *
+	 *	M(x) = A(x)*x^|B(x)| + B(x)
+	 *
+	 * Then the CRC of M(x) is:
+	 *
+	 *	CRC(M(x)) = CRC(A(x)*x^|B(x)| + B(x))
+	 *	          = CRC(A(x)*x^32*x^(|B(x)| - 32) + B(x))
+	 *	          = CRC(CRC(A(x))*x^(|B(x)| - 32) + B(x))
+	 *
+	 * Note: all arithmetic is modulo G(x), the generator polynomial; that's
+	 * why A(x)*x^32 can be replaced with CRC(A(x)) = A(x)*x^32 mod G(x).
+	 *
+	 * So the CRC of the full message is the CRC of the second part of the
+	 * message where the first 32 bits of the second part of the message
+	 * have been XOR'ed with the CRC of the first part of the message.
+	 */
+	x0 = *p++;
+	x0 ^= (__m128i)(__v4si){ remainder };
+
+	if (p > end512) /* only 128, 256, or 384 bits of input? */
+		goto _128_bits_at_a_time;
+	x1 = *p++;
+	x2 = *p++;
+	x3 = *p++;
+
+	/* Fold 512 bits at a time */
+	for (; p != end512; p += 4) {
+		__m128i y0, y1, y2, y3;
+
+		y0 = p[0];
+		y1 = p[1];
+		y2 = p[2];
+		y3 = p[3];
+
+		/*
+		 * Note: the immediate constant for PCLMULQDQ specifies which
+		 * 64-bit halves of the 128-bit vectors to multiply:
+		 *
+		 * 0x00 means low halves (higher degree polynomial terms for us)
+		 * 0x11 means high halves (lower degree polynomial terms for us)
+		 */
+		y0 ^= _mm_clmulepi64_si128(x0, multipliers_4, 0x00);
+		y1 ^= _mm_clmulepi64_si128(x1, multipliers_4, 0x00);
+		y2 ^= _mm_clmulepi64_si128(x2, multipliers_4, 0x00);
+		y3 ^= _mm_clmulepi64_si128(x3, multipliers_4, 0x00);
+		y0 ^= _mm_clmulepi64_si128(x0, multipliers_4, 0x11);
+		y1 ^= _mm_clmulepi64_si128(x1, multipliers_4, 0x11);
+		y2 ^= _mm_clmulepi64_si128(x2, multipliers_4, 0x11);
+		y3 ^= _mm_clmulepi64_si128(x3, multipliers_4, 0x11);
+
+		x0 = y0;
+		x1 = y1;
+		x2 = y2;
+		x3 = y3;
+	}
+
+	/* Fold 512 bits => 128 bits */
+	x2 ^= _mm_clmulepi64_si128(x0, multipliers_2, 0x00);
+	x3 ^= _mm_clmulepi64_si128(x1, multipliers_2, 0x00);
+	x2 ^= _mm_clmulepi64_si128(x0, multipliers_2, 0x11);
+	x3 ^= _mm_clmulepi64_si128(x1, multipliers_2, 0x11);
+	x3 ^= _mm_clmulepi64_si128(x2, multipliers_1, 0x00);
+	x3 ^= _mm_clmulepi64_si128(x2, multipliers_1, 0x11);
+	x0 = x3;
+
+_128_bits_at_a_time:
+	while (p != end) {
+		/* Fold 128 bits into next 128 bits */
+		x1 = *p++;
+		x1 ^= _mm_clmulepi64_si128(x0, multipliers_1, 0x00);
+		x1 ^= _mm_clmulepi64_si128(x0, multipliers_1, 0x11);
+		x0 = x1;
+	}
+
+	/* Now there are just 128 bits left, stored in 'x0'. */
+
+	/*
+	 * Fold 128 => 96 bits.  This also implicitly appends 32 zero bits,
+	 * which is equivalent to multiplying by x^32.  This is needed because
+	 * the CRC is defined as M(x)*x^32 mod G(x), not just M(x) mod G(x).
+	 */
+	x0 = _mm_srli_si128(x0, 8) ^
+	     _mm_clmulepi64_si128(x0, multipliers_1, 0x10);
+
+	/* Fold 96 => 64 bits */
+	x0 = _mm_srli_si128(x0, 4) ^
+	     _mm_clmulepi64_si128(x0 & mask32, final_multiplier, 0x00);
+
+        /*
+	 * Finally, reduce 64 => 32 bits using Barrett reduction.
+	 *
+	 * Let M(x) = A(x)*x^32 + B(x) be the remaining message.  The goal is to
+	 * compute R(x) = M(x) mod G(x).  Since degree(B(x)) < degree(G(x)):
+	 *
+	 *	R(x) = (A(x)*x^32 + B(x)) mod G(x)
+	 *	     = (A(x)*x^32) mod G(x) + B(x)
+	 *
+	 * Then, by the Division Algorithm there exists a unique q(x) such that:
+	 *
+	 *	A(x)*x^32 mod G(x) = A(x)*x^32 - q(x)*G(x)
+	 *
+	 * Since the left-hand side is of maximum degree 31, the right-hand side
+	 * must be too.  This implies that we can apply 'mod x^32' to the
+	 * right-hand side without changing its value:
+	 *
+	 *	(A(x)*x^32 - q(x)*G(x)) mod x^32 = q(x)*G(x) mod x^32
+	 *
+	 * Note that '+' is equivalent to '-' in polynomials over GF(2).
+	 *
+	 * We also know that:
+	 *
+	 *	              / A(x)*x^32 \
+	 *	q(x) = floor (  ---------  )
+	 *	              \    G(x)   /
+	 *
+	 * To compute this efficiently, we can multiply the top and bottom by
+	 * x^32 and move the division by G(x) to the top:
+	 *
+	 *	              / A(x) * floor(x^64 / G(x)) \
+	 *	q(x) = floor (  -------------------------  )
+	 *	              \           x^32            /
+	 *
+	 * Note that floor(x^64 / G(x)) is a constant.
+	 *
+	 * So finally we have:
+	 *
+	 *	                          / A(x) * floor(x^64 / G(x)) \
+	 *	R(x) = B(x) + G(x)*floor (  -------------------------  )
+	 *	                          \           x^32            /
+	 */
+	x1 = x0;
+	x0 = _mm_clmulepi64_si128(x0 & mask32, barrett_reduction_constants, 0x00);
+	x0 = _mm_clmulepi64_si128(x0 & mask32, barrett_reduction_constants, 0x10);
+	return _mm_cvtsi128_si32(_mm_srli_si128(x0 ^ x1, 4));
+}
+
+#define IMPL_ALIGNMENT		16
+#define IMPL_SEGMENT_SIZE	16
+#include "../crc32_vec_template.h"

lib/x86/decompress_impl.h

@@ -0,0 +1,26 @@
+#include "cpu_features.h"
+
+/* Include the BMI2-optimized version? */
+#undef DISPATCH_BMI2
+#if !defined(__BMI2__) && X86_CPU_FEATURES_ENABLED && \
+	COMPILER_SUPPORTS_BMI2_TARGET
+#  define FUNCNAME	deflate_decompress_bmi2
+#  define ATTRIBUTES	__attribute__((target("bmi2")))
+#  define DISPATCH	1
+#  define DISPATCH_BMI2	1
+#  include "../decompress_template.h"
+#endif
+
+#ifdef DISPATCH
+static inline decompress_func_t
+arch_select_decompress_func(void)
+{
+	u32 features = get_cpu_features();
+
+#ifdef DISPATCH_BMI2
+	if (features & X86_CPU_FEATURE_BMI2)
+		return deflate_decompress_bmi2;
+#endif
+	return NULL;
+}
+#endif /* DISPATCH */

lib/x86/matchfinder_impl.h

@@ -0,0 +1,164 @@
+/*
+ * x86/matchfinder_impl.h - x86 implementations of matchfinder functions
+ *
+ * Copyright 2016 Eric Biggers
+ *
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#ifdef __AVX2__
+#  if MATCHFINDER_ALIGNMENT < 32
+#    undef MATCHFINDER_ALIGNMENT
+#    define MATCHFINDER_ALIGNMENT 32
+#  endif
+#  include <immintrin.h>
+static forceinline bool
+matchfinder_init_avx2(mf_pos_t *data, size_t size)
+{
+	__m256i v, *p;
+	size_t n;
+
+	if (size % (sizeof(__m256i) * 4) != 0)
+		return false;
+
+	STATIC_ASSERT(sizeof(mf_pos_t) == 2);
+	v = _mm256_set1_epi16(MATCHFINDER_INITVAL);
+	p = (__m256i *)data;
+	n = size / (sizeof(__m256i) * 4);
+	do {
+		p[0] = v;
+		p[1] = v;
+		p[2] = v;
+		p[3] = v;
+		p += 4;
+	} while (--n);
+	return true;
+}
+
+static forceinline bool
+matchfinder_rebase_avx2(mf_pos_t *data, size_t size)
+{
+	__m256i v, *p;
+	size_t n;
+
+	if (size % (sizeof(__m256i) * 4) != 0)
+		return false;
+
+	STATIC_ASSERT(sizeof(mf_pos_t) == 2);
+	v = _mm256_set1_epi16((u16)-MATCHFINDER_WINDOW_SIZE);
+	p = (__m256i *)data;
+	n = size / (sizeof(__m256i) * 4);
+	do {
+		/* PADDSW: Add Packed Signed Integers With Signed Saturation  */
+		p[0] = _mm256_adds_epi16(p[0], v);
+		p[1] = _mm256_adds_epi16(p[1], v);
+		p[2] = _mm256_adds_epi16(p[2], v);
+		p[3] = _mm256_adds_epi16(p[3], v);
+		p += 4;
+	} while (--n);
+	return true;
+}
+#endif /* __AVX2__ */
+
+#ifdef __SSE2__
+#  if MATCHFINDER_ALIGNMENT < 16
+#    undef MATCHFINDER_ALIGNMENT
+#    define MATCHFINDER_ALIGNMENT 16
+#  endif
+#  include <emmintrin.h>
+static forceinline bool
+matchfinder_init_sse2(mf_pos_t *data, size_t size)
+{
+	__m128i v, *p;
+	size_t n;
+
+	if (size % (sizeof(__m128i) * 4) != 0)
+		return false;
+
+	STATIC_ASSERT(sizeof(mf_pos_t) == 2);
+	v = _mm_set1_epi16(MATCHFINDER_INITVAL);
+	p = (__m128i *)data;
+	n = size / (sizeof(__m128i) * 4);
+	do {
+		p[0] = v;
+		p[1] = v;
+		p[2] = v;
+		p[3] = v;
+		p += 4;
+	} while (--n);
+	return true;
+}
+
+static forceinline bool
+matchfinder_rebase_sse2(mf_pos_t *data, size_t size)
+{
+	__m128i v, *p;
+	size_t n;
+
+	if (size % (sizeof(__m128i) * 4) != 0)
+		return false;
+
+	STATIC_ASSERT(sizeof(mf_pos_t) == 2);
+	v = _mm_set1_epi16((u16)-MATCHFINDER_WINDOW_SIZE);
+	p = (__m128i *)data;
+	n = size / (sizeof(__m128i) * 4);
+	do {
+		/* PADDSW: Add Packed Signed Integers With Signed Saturation  */
+		p[0] = _mm_adds_epi16(p[0], v);
+		p[1] = _mm_adds_epi16(p[1], v);
+		p[2] = _mm_adds_epi16(p[2], v);
+		p[3] = _mm_adds_epi16(p[3], v);
+		p += 4;
+	} while (--n);
+	return true;
+}
+#endif /* __SSE2__ */
+
+#undef arch_matchfinder_init
+static forceinline bool
+arch_matchfinder_init(mf_pos_t *data, size_t size)
+{
+#ifdef __AVX2__
+	if (matchfinder_init_avx2(data, size))
+		return true;
+#endif
+#ifdef __SSE2__
+	if (matchfinder_init_sse2(data, size))
+		return true;
+#endif
+	return false;
+}
+
+#undef arch_matchfinder_rebase
+static forceinline bool
+arch_matchfinder_rebase(mf_pos_t *data, size_t size)
+{
+#ifdef __AVX2__
+	if (matchfinder_rebase_avx2(data, size))
+		return true;
+#endif
+#ifdef __SSE2__
+	if (matchfinder_rebase_sse2(data, size))
+		return true;
+#endif
+	return false;
+}

lib/x86_cpu_features.h

@@ -1,47 +0,0 @@
-/*
- * x86_cpu_features.h - feature detection for x86 processors
- */
-
-#ifndef _LIB_X86_CPU_FEATURES_H
-#define _LIB_X86_CPU_FEATURES_H
-
-#include "common_defs.h"
-
-#if defined(__x86_64__) && COMPILER_SUPPORTS_TARGET_FUNCTION_ATTRIBUTE
-#  define X86_CPU_FEATURES_ENABLED 1
-#else
-#  define X86_CPU_FEATURES_ENABLED 0
-#endif
-
-#if X86_CPU_FEATURES_ENABLED
-
-#define X86_CPU_FEATURE_SSE		0x00000001
-#define X86_CPU_FEATURE_SSE2		0x00000002
-#define X86_CPU_FEATURE_SSE3		0x00000004
-#define X86_CPU_FEATURE_SSSE3		0x00000008
-#define X86_CPU_FEATURE_SSE4_1		0x00000010
-#define X86_CPU_FEATURE_SSE4_2		0x00000020
-#define X86_CPU_FEATURE_AVX		0x00000040
-#define X86_CPU_FEATURE_BMI		0x00000080
-#define X86_CPU_FEATURE_AVX2		0x00000100
-#define X86_CPU_FEATURE_BMI2		0x00000200
-
-#define X86_CPU_FEATURES_KNOWN		0x80000000
-
-extern u32 _x86_cpu_features;
-
-extern void
-x86_setup_cpu_features(void);
-
-/* Does the processor have the specified feature?  */
-static inline bool
-x86_have_cpu_feature(u32 feature)
-{
-	if (_x86_cpu_features == 0)
-		x86_setup_cpu_features();
-	return _x86_cpu_features & feature;
-}
-
-#endif /* X86_CPU_FEATURES_ENABLED */
-
-#endif /* _LIB_X86_CPU_FEATURES_H */

lib/zlib_compress.c

@@ -1,27 +1,42 @@
 /*
  * zlib_compress.c - compress with a zlib wrapper
  *
- * Written in 2014-2016 by Eric Biggers <ebiggers3@gmail.com>
+ * Originally public domain; changes after 2016-09-07 are copyrighted.
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Copyright 2016 Eric Biggers
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  */
 
-#include "adler32.h"
 #include "deflate_compress.h"
 #include "unaligned.h"
 #include "zlib_constants.h"
 
 #include "libdeflate.h"
 
-LIBEXPORT size_t
-zlib_compress(struct deflate_compressor *c, const void *in, size_t in_size,
-	      void *out, size_t out_nbytes_avail)
+LIBDEFLATEAPI size_t
+libdeflate_zlib_compress(struct libdeflate_compressor *c,
+			 const void *in, size_t in_size,
+			 void *out, size_t out_nbytes_avail)
 {
 	u8 *out_next = out;
 	u16 hdr;
@@ -50,21 +65,23 @@ zlib_compress(struct deflate_compressor *c, const void *in, size_t in_size,
 	out_next += 2;
 
 	/* Compressed data  */
-	deflate_size = deflate_compress(c, in, in_size, out_next,
+	deflate_size = libdeflate_deflate_compress(c, in, in_size, out_next,
 					out_nbytes_avail - ZLIB_MIN_OVERHEAD);
 	if (deflate_size == 0)
 		return 0;
 	out_next += deflate_size;
 
 	/* ADLER32  */
-	put_unaligned_be32(adler32(in, in_size), out_next);
+	put_unaligned_be32(libdeflate_adler32(1, in, in_size), out_next);
 	out_next += 4;
 
 	return out_next - (u8 *)out;
 }
 
-LIBEXPORT size_t
-zlib_compress_bound(struct deflate_compressor *c, size_t in_nbytes)
+LIBDEFLATEAPI size_t
+libdeflate_zlib_compress_bound(struct libdeflate_compressor *c,
+			       size_t in_nbytes)
 {
-	return ZLIB_MIN_OVERHEAD + deflate_compress_bound(c, in_nbytes);
+	return ZLIB_MIN_OVERHEAD +
+	       libdeflate_deflate_compress_bound(c, in_nbytes);
 }

lib/zlib_constants.h

@@ -2,8 +2,8 @@
  * zlib_constants.h - constants for the zlib wrapper format
  */
 
-#ifndef _LIB_ZLIB_CONSTANTS_H
-#define _LIB_ZLIB_CONSTANTS_H
+#ifndef LIB_ZLIB_CONSTANTS_H
+#define LIB_ZLIB_CONSTANTS_H
 
 #define ZLIB_MIN_HEADER_SIZE	2
 #define ZLIB_FOOTER_SIZE	4
@@ -18,4 +18,4 @@
 #define ZLIB_DEFAULT_COMPRESSION	2
 #define ZLIB_SLOWEST_COMPRESSION	3
 
-#endif /* _LIB_ZLIB_CONSTANTS_H */
+#endif /* LIB_ZLIB_CONSTANTS_H */

lib/zlib_decompress.c

@@ -1,37 +1,51 @@
 /*
  * zlib_decompress.c - decompress with a zlib wrapper
  *
- * Written in 2014-2016 by Eric Biggers <ebiggers3@gmail.com>
+ * Originally public domain; changes after 2016-09-07 are copyrighted.
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Copyright 2016 Eric Biggers
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  */
 
-#include "adler32.h"
 #include "unaligned.h"
 #include "zlib_constants.h"
 
 #include "libdeflate.h"
 
-LIBEXPORT enum decompress_result
-zlib_decompress(struct deflate_decompressor *d,
-		const void *in, size_t in_nbytes,
-		void *out, size_t out_nbytes_avail,
-		size_t *actual_out_nbytes_ret)
+LIBDEFLATEAPI enum libdeflate_result
+libdeflate_zlib_decompress(struct libdeflate_decompressor *d,
+			   const void *in, size_t in_nbytes,
+			   void *out, size_t out_nbytes_avail,
+			   size_t *actual_out_nbytes_ret)
 {
 	const u8 *in_next = in;
 	const u8 * const in_end = in_next + in_nbytes;
 	u16 hdr;
 	size_t actual_out_nbytes;
-	enum decompress_result result;
+	enum libdeflate_result result;
 
 	if (in_nbytes < ZLIB_MIN_OVERHEAD)
-		return DECOMPRESS_BAD_DATA;
+		return LIBDEFLATE_BAD_DATA;
 
 	/* 2 byte header: CMF and FLG  */
 	hdr = get_unaligned_be16(in_next);
@@ -39,26 +53,26 @@ zlib_decompress(struct deflate_decompressor *d,
 
 	/* FCHECK */
 	if ((hdr % 31) != 0)
-		return DECOMPRESS_BAD_DATA;
+		return LIBDEFLATE_BAD_DATA;
 
 	/* CM */
 	if (((hdr >> 8) & 0xF) != ZLIB_CM_DEFLATE)
-		return DECOMPRESS_BAD_DATA;
+		return LIBDEFLATE_BAD_DATA;
 
 	/* CINFO */
 	if ((hdr >> 12) > ZLIB_CINFO_32K_WINDOW)
-		return DECOMPRESS_BAD_DATA;
+		return LIBDEFLATE_BAD_DATA;
 
 	/* FDICT */
 	if ((hdr >> 5) & 1)
-		return DECOMPRESS_BAD_DATA;
+		return LIBDEFLATE_BAD_DATA;
 
 	/* Compressed data  */
-	result = deflate_decompress(d, in_next,
-				    in_end - ZLIB_FOOTER_SIZE - in_next,
-				    out, out_nbytes_avail,
-				    actual_out_nbytes_ret);
-	if (result != DECOMPRESS_SUCCESS)
+	result = libdeflate_deflate_decompress(d, in_next,
+					in_end - ZLIB_FOOTER_SIZE - in_next,
+					out, out_nbytes_avail,
+					actual_out_nbytes_ret);
+	if (result != LIBDEFLATE_SUCCESS)
 		return result;
 
 	if (actual_out_nbytes_ret)
@@ -69,8 +83,9 @@ zlib_decompress(struct deflate_decompressor *d,
 	in_next = in_end - ZLIB_FOOTER_SIZE;
 
 	/* ADLER32  */
-	if (adler32(out, actual_out_nbytes) != get_unaligned_be32(in_next))
-		return DECOMPRESS_BAD_DATA;
+	if (libdeflate_adler32(1, out, actual_out_nbytes) !=
+	    get_unaligned_be32(in_next))
+		return LIBDEFLATE_BAD_DATA;
 
-	return DECOMPRESS_SUCCESS;
+	return LIBDEFLATE_SUCCESS;
 }

libdeflate.h

@@ -9,18 +9,21 @@
 extern "C" {
 #endif
 
-#define LIBDEFLATE_VERSION_MAJOR	0
-#define LIBDEFLATE_VERSION_MINOR	3
-#define LIBDEFLATE_VERSION_STRING	"0.3"
+#define LIBDEFLATE_VERSION_MAJOR	1
+#define LIBDEFLATE_VERSION_MINOR	0
+#define LIBDEFLATE_VERSION_STRING	"1.0"
 
 #include <stddef.h>
+#include <stdint.h>
 
-/* Microsoft C / Visual Studio garbage.  If you want to link to the DLL version
- * of libdeflate, then #define LIBDEFLATE_DLL.  */
-#ifdef _MSC_VER
+/*
+ * On Windows, if you want to link to the DLL version of libdeflate, then
+ * #define LIBDEFLATE_DLL.
+ */
+#ifdef LIBDEFLATE_DLL
 #  ifdef BUILDING_LIBDEFLATE
-#    define LIBDEFLATEAPI __declspec(dllexport)
-#  elif defined(LIBDEFLATE_DLL)
+#    define LIBDEFLATEAPI LIBEXPORT
+#  elif defined(_WIN32) || defined(__CYGWIN__)
 #    define LIBDEFLATEAPI __declspec(dllimport)
 #  endif
 #endif
@@ -32,46 +35,50 @@ extern "C" {
 /*                             Compression                                    */
 /* ========================================================================== */
 
-struct deflate_compressor;
+struct libdeflate_compressor;
 
 /*
- * deflate_alloc_compressor() allocates a new DEFLATE compressor.
- * 'compression_level' is the compression level on a zlib-like scale but with a
- * higher maximum value (1 = fastest, 6 = medium/default, 9 = slow, 12 =
- * slowest).  The return value is a pointer to the new DEFLATE compressor, or
- * NULL if out of memory.
+ * libdeflate_alloc_compressor() allocates a new compressor that supports
+ * DEFLATE, zlib, and gzip compression.  'compression_level' is the compression
+ * level on a zlib-like scale but with a higher maximum value (1 = fastest, 6 =
+ * medium/default, 9 = slow, 12 = slowest).  The return value is a pointer to
+ * the new compressor, or NULL if out of memory.
  *
  * Note: for compression, the sliding window size is defined at compilation time
  * to 32768, the largest size permissible in the DEFLATE format.  It cannot be
  * changed at runtime.
+ *
+ * A single compressor is not safe to use by multiple threads concurrently.
+ * However, different threads may use different compressors concurrently.
  */
-LIBDEFLATEAPI struct deflate_compressor *
-deflate_alloc_compressor(unsigned int compression_level);
+LIBDEFLATEAPI struct libdeflate_compressor *
+libdeflate_alloc_compressor(int compression_level);
 
 /*
- * deflate_compress() performs DEFLATE compression on a buffer of data.  The
- * function attempts to compress 'in_nbytes' bytes of data located at 'in' and
- * write the results to 'out', which has space for 'out_nbytes_avail' bytes.
- * The return value is the compressed size in bytes, or 0 if the data could not
- * be compressed to 'out_nbytes_avail' bytes or fewer.
+ * libdeflate_deflate_compress() performs raw DEFLATE compression on a buffer of
+ * data.  The function attempts to compress 'in_nbytes' bytes of data located at
+ * 'in' and write the results to 'out', which has space for 'out_nbytes_avail'
+ * bytes.  The return value is the compressed size in bytes, or 0 if the data
+ * could not be compressed to 'out_nbytes_avail' bytes or fewer.
  */
 LIBDEFLATEAPI size_t
-deflate_compress(struct deflate_compressor *compressor,
-		 const void *in, size_t in_nbytes,
-		 void *out, size_t out_nbytes_avail);
+libdeflate_deflate_compress(struct libdeflate_compressor *compressor,
+			    const void *in, size_t in_nbytes,
+			    void *out, size_t out_nbytes_avail);
 
 /*
- * deflate_compress_bound() returns a worst-case upper bound on the number of
- * bytes of compressed data that may be produced by compressing any buffer of
- * length less than or equal to 'in_nbytes' using deflate_compress() with the
- * specified compressor.  Mathematically, this bound will necessarily be a
- * number greater than or equal to 'in_nbytes'.  It may be an overestimate of
- * the true upper bound.  The return value is guaranteed to be the same for all
- * invocations with the same compressor and same 'in_nbytes'.
+ * libdeflate_deflate_compress_bound() returns a worst-case upper bound on the
+ * number of bytes of compressed data that may be produced by compressing any
+ * buffer of length less than or equal to 'in_nbytes' using
+ * libdeflate_deflate_compress() with the specified compressor.  Mathematically,
+ * this bound will necessarily be a number greater than or equal to 'in_nbytes'.
+ * It may be an overestimate of the true upper bound.  The return value is
+ * guaranteed to be the same for all invocations with the same compressor and
+ * same 'in_nbytes'.
  *
  * As a special case, 'compressor' may be NULL.  This causes the bound to be
- * taken across *any* deflate_compressor that could ever be allocated with this
- * build of the library, with any options.
+ * taken across *any* libdeflate_compressor that could ever be allocated with
+ * this build of the library, with any options.
  *
  * Note that this function is not necessary in many applications.  With
  * block-based compression, it is usually preferable to separately store the
@@ -79,154 +86,220 @@ deflate_compress(struct deflate_compressor *compressor,
  * to less than their original size uncompressed.  In that scenario, there is no
  * need to know the worst-case compressed size, since the maximum number of
  * bytes of compressed data that may be used would always be one less than the
- * input length.  You can just pass a buffer of that size to deflate_compress()
- * and store the data uncompressed if deflate_compress() returns 0, indicating
- * that the compressed data did not fit into the provided output buffer.
+ * input length.  You can just pass a buffer of that size to
+ * libdeflate_deflate_compress() and store the data uncompressed if
+ * libdeflate_deflate_compress() returns 0, indicating that the compressed data
+ * did not fit into the provided output buffer.
  */
 LIBDEFLATEAPI size_t
-deflate_compress_bound(struct deflate_compressor *compressor, size_t in_nbytes);
+libdeflate_deflate_compress_bound(struct libdeflate_compressor *compressor,
+				  size_t in_nbytes);
 
 /*
- * Like deflate_compress(), but stores the data in the zlib wrapper format.
+ * Like libdeflate_deflate_compress(), but stores the data in the zlib wrapper
+ * format.
  */
 LIBDEFLATEAPI size_t
-zlib_compress(struct deflate_compressor *compressor,
-	      const void *in, size_t in_nbytes,
-	      void *out, size_t out_nbytes_avail);
+libdeflate_zlib_compress(struct libdeflate_compressor *compressor,
+			 const void *in, size_t in_nbytes,
+			 void *out, size_t out_nbytes_avail);
 
 /*
- * Like deflate_compress_bound(), but assumes the data will be compressed with
- * zlib_compress() rather than with deflate_compress().
+ * Like libdeflate_deflate_compress_bound(), but assumes the data will be
+ * compressed with libdeflate_zlib_compress() rather than with
+ * libdeflate_deflate_compress().
  */
 LIBDEFLATEAPI size_t
-zlib_compress_bound(struct deflate_compressor *compressor, size_t in_nbytes);
+libdeflate_zlib_compress_bound(struct libdeflate_compressor *compressor,
+			       size_t in_nbytes);
 
 /*
- * Like deflate_compress(), but stores the data in the gzip wrapper format.
+ * Like libdeflate_deflate_compress(), but stores the data in the gzip wrapper
+ * format.
  */
 LIBDEFLATEAPI size_t
-gzip_compress(struct deflate_compressor *compressor,
-	      const void *in, size_t in_nbytes,
-	      void *out, size_t out_nbytes_avail);
+libdeflate_gzip_compress(struct libdeflate_compressor *compressor,
+			 const void *in, size_t in_nbytes,
+			 void *out, size_t out_nbytes_avail);
 
 /*
- * Like deflate_compress_bound(), but assumes the data will be compressed with
- * gzip_compress() rather than with deflate_compress().
+ * Like libdeflate_deflate_compress_bound(), but assumes the data will be
+ * compressed with libdeflate_gzip_compress() rather than with
+ * libdeflate_deflate_compress().
  */
 LIBDEFLATEAPI size_t
-gzip_compress_bound(struct deflate_compressor *compressor, size_t in_nbytes);
+libdeflate_gzip_compress_bound(struct libdeflate_compressor *compressor,
+			       size_t in_nbytes);
 
 /*
- * deflate_free_compressor() frees a DEFLATE compressor that was allocated with
- * deflate_alloc_compressor().  If a NULL pointer is passed in, no action is
+ * libdeflate_free_compressor() frees a compressor that was allocated with
+ * libdeflate_alloc_compressor().  If a NULL pointer is passed in, no action is
  * taken.
  */
 LIBDEFLATEAPI void
-deflate_free_compressor(struct deflate_compressor *compressor);
+libdeflate_free_compressor(struct libdeflate_compressor *compressor);
 
 /* ========================================================================== */
 /*                             Decompression                                  */
 /* ========================================================================== */
 
-struct deflate_decompressor;
+struct libdeflate_decompressor;
 
 /*
- * deflate_alloc_decompressor() allocates a new DEFLATE decompressor.  The
- * return value is a pointer to the new DEFLATE decompressor, or NULL if out of
- * memory.
+ * libdeflate_alloc_decompressor() allocates a new decompressor that can be used
+ * for DEFLATE, zlib, and gzip decompression.  The return value is a pointer to
+ * the new decompressor, or NULL if out of memory.
  *
  * This function takes no parameters, and the returned decompressor is valid for
  * decompressing data that was compressed at any compression level and with any
- * sliding window size.  It can also be used for any wrapper format (raw
- * DEFLATE, zlib, or gzip); however, the appropriate decompression function must
- * be called.
+ * sliding window size.
+ *
+ * A single decompressor is not safe to use by multiple threads concurrently.
+ * However, different threads may use different decompressors concurrently.
  */
-LIBDEFLATEAPI struct deflate_decompressor *
-deflate_alloc_decompressor(void);
+LIBDEFLATEAPI struct libdeflate_decompressor *
+libdeflate_alloc_decompressor(void);
 
 /*
- * Result of a call to deflate_decompress(), zlib_decompress(), or
- * gzip_decompress().
+ * Result of a call to libdeflate_deflate_decompress(),
+ * libdeflate_zlib_decompress(), or libdeflate_gzip_decompress().
  */
-enum decompress_result {
+enum libdeflate_result {
 	/* Decompression was successful.  */
-	DECOMPRESS_SUCCESS = 0,
+	LIBDEFLATE_SUCCESS = 0,
 
 	/* Decompressed failed because the compressed data was invalid, corrupt,
 	 * or otherwise unsupported.  */
-	DECOMPRESS_BAD_DATA = 1,
+	LIBDEFLATE_BAD_DATA = 1,
 
 	/* A NULL 'actual_out_nbytes_ret' was provided, but the data would have
 	 * decompressed to fewer than 'out_nbytes_avail' bytes.  */
-	DECOMPRESS_SHORT_OUTPUT = 2,
+	LIBDEFLATE_SHORT_OUTPUT = 2,
 
 	/* The data would have decompressed to more than 'out_nbytes_avail'
 	 * bytes.  */
-	DECOMPRESS_INSUFFICIENT_SPACE = 3,
+	LIBDEFLATE_INSUFFICIENT_SPACE = 3,
 };
 
 /*
- * deflate_decompress() decompresses 'in_nbytes' bytes of DEFLATE-compressed
- * data at 'in' and writes the uncompressed data to 'out', which is a buffer of
- * at least 'out_nbytes_avail' bytes.  If decompression was successful, then 0
- * (DECOMPRESS_SUCCESS) is returned; otherwise, a nonzero result code such as
- * DECOMPRESS_BAD_DATA is returned.  If a nonzero result code is returned, then
- * the contents of the output buffer are undefined.
+ * libdeflate_deflate_decompress() decompresses the DEFLATE-compressed stream
+ * from the buffer 'in' with compressed size up to 'in_nbytes' bytes.  The
+ * uncompressed data is written to 'out', a buffer with size 'out_nbytes_avail'
+ * bytes.  If decompression succeeds, then 0 (LIBDEFLATE_SUCCESS) is returned.
+ * Otherwise, a nonzero result code such as LIBDEFLATE_BAD_DATA is returned.  If
+ * a nonzero result code is returned, then the contents of the output buffer are
+ * undefined.
  *
- * deflate_decompress() can be used in cases where the actual uncompressed size
- * is known (recommended) or unknown (not recommended):
+ * Decompression stops at the end of the DEFLATE stream (as indicated by the
+ * BFINAL flag), even if it is actually shorter than 'in_nbytes' bytes.
+ *
+ * libdeflate_deflate_decompress() can be used in cases where the actual
+ * uncompressed size is known (recommended) or unknown (not recommended):
  *
  *   - If the actual uncompressed size is known, then pass the actual
  *     uncompressed size as 'out_nbytes_avail' and pass NULL for
- *     'actual_out_nbytes_ret'.  This makes deflate_decompress() fail with
- *     DECOMPRESS_SHORT_OUTPUT if the data decompressed to fewer than the
+ *     'actual_out_nbytes_ret'.  This makes libdeflate_deflate_decompress() fail
+ *     with LIBDEFLATE_SHORT_OUTPUT if the data decompressed to fewer than the
  *     specified number of bytes.
  *
  *   - If the actual uncompressed size is unknown, then provide a non-NULL
  *     'actual_out_nbytes_ret' and provide a buffer with some size
  *     'out_nbytes_avail' that you think is large enough to hold all the
  *     uncompressed data.  In this case, if the data decompresses to less than
- *     or equal to 'out_nbytes_avail' bytes, then deflate_decompress() will
- *     write the actual uncompressed size to *actual_out_nbytes_ret and return 0
- *     (DECOMPRESS_SUCCESS).  Otherwise, it will return
- *     DECOMPRESS_INSUFFICIENT_SPACE if the provided buffer was not large enough
- *     but no other problems were encountered, or another nonzero result code if
- *     decompression failed for another reason.
+ *     or equal to 'out_nbytes_avail' bytes, then
+ *     libdeflate_deflate_decompress() will write the actual uncompressed size
+ *     to *actual_out_nbytes_ret and return 0 (LIBDEFLATE_SUCCESS).  Otherwise,
+ *     it will return LIBDEFLATE_INSUFFICIENT_SPACE if the provided buffer was
+ *     not large enough but no other problems were encountered, or another
+ *     nonzero result code if decompression failed for another reason.
  */
-LIBDEFLATEAPI enum decompress_result
-deflate_decompress(struct deflate_decompressor *decompressor,
-		   const void *in, size_t in_nbytes,
-		   void *out, size_t out_nbytes_avail,
-		   size_t *actual_out_nbytes_ret);
+LIBDEFLATEAPI enum libdeflate_result
+libdeflate_deflate_decompress(struct libdeflate_decompressor *decompressor,
+			      const void *in, size_t in_nbytes,
+			      void *out, size_t out_nbytes_avail,
+			      size_t *actual_out_nbytes_ret);
 
 /*
- * Like deflate_decompress(), but assumes the zlib wrapper format instead of raw
- * DEFLATE.
+ * Like libdeflate_deflate_decompress(), but adds the 'actual_in_nbytes_ret'
+ * argument.  If decompression succeeds and 'actual_in_nbytes_ret' is not NULL,
+ * then the actual compressed size of the DEFLATE stream (aligned to the next
+ * byte boundary) is written to *actual_in_nbytes_ret.
  */
-LIBDEFLATEAPI enum decompress_result
-zlib_decompress(struct deflate_decompressor *decompressor,
-		const void *in, size_t in_nbytes,
-		void *out, size_t out_nbytes_avail,
-		size_t *actual_out_nbytes_ret);
+LIBDEFLATEAPI enum libdeflate_result
+libdeflate_deflate_decompress_ex(struct libdeflate_decompressor *decompressor,
+				 const void *in, size_t in_nbytes,
+				 void *out, size_t out_nbytes_avail,
+				 size_t *actual_in_nbytes_ret,
+				 size_t *actual_out_nbytes_ret);
 
 /*
- * Like deflate_decompress(), but assumes the gzip wrapper format instead of raw
- * DEFLATE.
+ * Like libdeflate_deflate_decompress(), but assumes the zlib wrapper format
+ * instead of raw DEFLATE.
  */
-LIBDEFLATEAPI enum decompress_result
-gzip_decompress(struct deflate_decompressor *decompressor,
-		const void *in, size_t in_nbytes,
-		void *out, size_t out_nbytes_avail,
-		size_t *actual_out_nbytes_ret);
+LIBDEFLATEAPI enum libdeflate_result
+libdeflate_zlib_decompress(struct libdeflate_decompressor *decompressor,
+			   const void *in, size_t in_nbytes,
+			   void *out, size_t out_nbytes_avail,
+			   size_t *actual_out_nbytes_ret);
 
 /*
- * deflate_free_decompressor() frees a DEFLATE decompressor that was allocated
- * with deflate_alloc_decompressor().  If a NULL pointer is passed in, no action
+ * Like libdeflate_deflate_decompress(), but assumes the gzip wrapper format
+ * instead of raw DEFLATE.
+ *
+ * If multiple gzip-compressed members are concatenated, then only the first
+ * will be decompressed.  Use libdeflate_gzip_decompress_ex() if you need
+ * multi-member support.
+ */
+LIBDEFLATEAPI enum libdeflate_result
+libdeflate_gzip_decompress(struct libdeflate_decompressor *decompressor,
+			   const void *in, size_t in_nbytes,
+			   void *out, size_t out_nbytes_avail,
+			   size_t *actual_out_nbytes_ret);
+
+/*
+ * Like libdeflate_gzip_decompress(), but adds the 'actual_in_nbytes_ret'
+ * argument.  If 'actual_in_nbytes_ret' is not NULL and the decompression
+ * succeeds (indicating that the first gzip-compressed member in the input
+ * buffer was decompressed), then the actual number of input bytes consumed is
+ * written to *actual_in_nbytes_ret.
+ */
+LIBDEFLATEAPI enum libdeflate_result
+libdeflate_gzip_decompress_ex(struct libdeflate_decompressor *decompressor,
+			      const void *in, size_t in_nbytes,
+			      void *out, size_t out_nbytes_avail,
+			      size_t *actual_in_nbytes_ret,
+			      size_t *actual_out_nbytes_ret);
+
+/*
+ * libdeflate_free_decompressor() frees a decompressor that was allocated with
+ * libdeflate_alloc_decompressor().  If a NULL pointer is passed in, no action
  * is taken.
  */
 LIBDEFLATEAPI void
-deflate_free_decompressor(struct deflate_decompressor *decompressor);
+libdeflate_free_decompressor(struct libdeflate_decompressor *decompressor);
 
+/* ========================================================================== */
+/*                                Checksums                                   */
+/* ========================================================================== */
+
+/*
+ * libdeflate_adler32() updates a running Adler-32 checksum with 'len' bytes of
+ * data and returns the updated checksum.  When starting a new checksum, the
+ * required initial value for 'adler' is 1.  This value is also returned when
+ * 'buffer' is specified as NULL.
+ */
+LIBDEFLATEAPI uint32_t
+libdeflate_adler32(uint32_t adler32, const void *buffer, size_t len);
+
+
+/*
+ * libdeflate_crc32() updates a running CRC-32 checksum with 'len' bytes of data
+ * and returns the updated checksum.  When starting a new checksum, the required
+ * initial value for 'crc' is 0.  This value is also returned when 'buffer' is
+ * specified as NULL.
+ */
+LIBDEFLATEAPI uint32_t
+libdeflate_crc32(uint32_t crc, const void *buffer, size_t len);
 
 #ifdef __cplusplus
 }

programs/benchmark.c

@@ -29,41 +29,7 @@
 
 #include "prog_util.h"
 
-static const tchar *const optstring = T("1::2::3::4::5::6::7::8::9::ghs:VYZz");
-
-static void
-show_usage(FILE *fp)
-{
-	fprintf(fp,
-"Usage: %"TS" [-LVL] [-ghVYZz] [-s SIZE] [FILE]...\n"
-"Benchmark DEFLATE compression and decompression on the specified FILEs.\n"
-"\n"
-"Options:\n"
-"  -1        fastest (worst) compression\n"
-"  -6        medium compression (default)\n"
-"  -12       slowest (best) compression\n"
-"  -g        use gzip wrapper\n"
-"  -h        print this help\n"
-"  -s SIZE   chunk size\n"
-"  -V        show version and legal information\n"
-"  -Y        compress with libz, not libdeflate\n"
-"  -Z        decompress with libz, not libdeflate\n"
-"  -z        use zlib wrapper\n",
-	program_invocation_name);
-}
-
-static void
-show_version(void)
-{
-	printf(
-"libdeflate compression benchmark program v" LIBDEFLATE_VERSION_STRING "\n"
-"Copyright 2016 Eric Biggers\n"
-"\n"
-"This program is free software which may be modified and/or redistributed\n"
-"under the terms of the MIT license.  There is NO WARRANTY, to the extent\n"
-"permitted by law.  See the COPYING file for details.\n"
-	);
-}
+static const tchar *const optstring = T("1::2::3::4::5::6::7::8::9::C:D:ghs:VYZz");
 
 enum wrapper {
 	NO_WRAPPER,
@@ -72,17 +38,108 @@ enum wrapper {
 };
 
 struct compressor {
+	int level;
+	enum wrapper wrapper;
+	const struct engine *engine;
 	void *private;
-	size_t (*compress)(void *, const void *, size_t, void *, size_t);
-	void (*free_private)(void *);
 };
 
 struct decompressor {
+	enum wrapper wrapper;
+	const struct engine *engine;
 	void *private;
-	bool (*decompress)(void *, const void *, size_t, void *, size_t);
-	void (*free_private)(void *);
 };
 
+struct engine {
+	const tchar *name;
+
+	bool (*init_compressor)(struct compressor *);
+	size_t (*compress)(struct compressor *, const void *, size_t,
+			   void *, size_t);
+	void (*destroy_compressor)(struct compressor *);
+
+	bool (*init_decompressor)(struct decompressor *);
+	bool (*decompress)(struct decompressor *, const void *, size_t,
+			   void *, size_t);
+	void (*destroy_decompressor)(struct decompressor *);
+};
+
+/******************************************************************************/
+
+static bool
+libdeflate_engine_init_compressor(struct compressor *c)
+{
+	c->private = alloc_compressor(c->level);
+	return c->private != NULL;
+}
+
+static size_t
+libdeflate_engine_compress(struct compressor *c, const void *in,
+			   size_t in_nbytes, void *out, size_t out_nbytes_avail)
+{
+	switch (c->wrapper) {
+	case ZLIB_WRAPPER:
+		return libdeflate_zlib_compress(c->private, in, in_nbytes,
+						out, out_nbytes_avail);
+	case GZIP_WRAPPER:
+		return libdeflate_gzip_compress(c->private, in, in_nbytes,
+						out, out_nbytes_avail);
+	default:
+		return libdeflate_deflate_compress(c->private, in, in_nbytes,
+						   out, out_nbytes_avail);
+	}
+}
+
+static void
+libdeflate_engine_destroy_compressor(struct compressor *c)
+{
+	libdeflate_free_compressor(c->private);
+}
+
+static bool
+libdeflate_engine_init_decompressor(struct decompressor *d)
+{
+	d->private = alloc_decompressor();
+	return d->private != NULL;
+}
+
+static bool
+libdeflate_engine_decompress(struct decompressor *d, const void *in,
+			     size_t in_nbytes, void *out, size_t out_nbytes)
+{
+	switch (d->wrapper) {
+	case ZLIB_WRAPPER:
+		return !libdeflate_zlib_decompress(d->private, in, in_nbytes,
+						   out, out_nbytes, NULL);
+	case GZIP_WRAPPER:
+		return !libdeflate_gzip_decompress(d->private, in, in_nbytes,
+						   out, out_nbytes, NULL);
+	default:
+		return !libdeflate_deflate_decompress(d->private, in, in_nbytes,
+						      out, out_nbytes, NULL);
+	}
+}
+
+static void
+libdeflate_engine_destroy_decompressor(struct decompressor *d)
+{
+	libdeflate_free_decompressor(d->private);
+}
+
+static const struct engine libdeflate_engine = {
+	.name			= T("libdeflate"),
+
+	.init_compressor	= libdeflate_engine_init_compressor,
+	.compress		= libdeflate_engine_compress,
+	.destroy_compressor	= libdeflate_engine_destroy_compressor,
+
+	.init_decompressor	= libdeflate_engine_init_decompressor,
+	.decompress		= libdeflate_engine_decompress,
+	.destroy_decompressor	= libdeflate_engine_destroy_decompressor,
+};
+
+/******************************************************************************/
+
 static int
 get_libz_window_bits(enum wrapper wrapper)
 {
@@ -97,11 +154,43 @@ get_libz_window_bits(enum wrapper wrapper)
 	}
 }
 
-static size_t
-libz_compress(void *private, const void *in, size_t in_nbytes,
-	      void *out, size_t out_nbytes_avail)
+static bool
+libz_engine_init_compressor(struct compressor *c)
 {
-	z_stream *z = private;
+	z_stream *z;
+
+	if (c->level > 9) {
+		msg("libz only supports up to compression level 9");
+		return false;
+	}
+
+	z = xmalloc(sizeof(*z));
+	if (z == NULL)
+		return false;
+
+	z->next_in = NULL;
+	z->avail_in = 0;
+	z->zalloc = NULL;
+	z->zfree = NULL;
+	z->opaque = NULL;
+	if (deflateInit2(z, c->level, Z_DEFLATED,
+			 get_libz_window_bits(c->wrapper),
+			 8, Z_DEFAULT_STRATEGY) != Z_OK)
+	{
+		msg("unable to initialize deflater");
+		free(z);
+		return false;
+	}
+
+	c->private = z;
+	return true;
+}
+
+static size_t
+libz_engine_compress(struct compressor *c, const void *in, size_t in_nbytes,
+		     void *out, size_t out_nbytes_avail)
+{
+	z_stream *z = c->private;
 
 	deflateReset(z);
 
@@ -116,219 +205,198 @@ libz_compress(void *private, const void *in, size_t in_nbytes,
 	return out_nbytes_avail - z->avail_out;
 }
 
-static bool
-libz_decompress(void *private, const void *in, size_t in_nbytes,
-		void *out, size_t out_nbytes_avail)
+static void
+libz_engine_destroy_compressor(struct compressor *c)
 {
-	z_stream *z = private;
+	z_stream *z = c->private;
+
+	deflateEnd(z);
+	free(z);
+}
+
+static bool
+libz_engine_init_decompressor(struct decompressor *d)
+{
+	z_stream *z;
+
+	z = xmalloc(sizeof(*z));
+	if (z == NULL)
+		return false;
+
+	z->next_in = NULL;
+	z->avail_in = 0;
+	z->zalloc = NULL;
+	z->zfree = NULL;
+	z->opaque = NULL;
+	if (inflateInit2(z, get_libz_window_bits(d->wrapper)) != Z_OK) {
+		msg("unable to initialize inflater");
+		free(z);
+		return false;
+	}
+
+	d->private = z;
+	return true;
+}
+
+static bool
+libz_engine_decompress(struct decompressor *d, const void *in, size_t in_nbytes,
+		       void *out, size_t out_nbytes)
+{
+	z_stream *z = d->private;
 
 	inflateReset(z);
 
 	z->next_in = (void *)in;
 	z->avail_in = in_nbytes;
 	z->next_out = out;
-	z->avail_out = out_nbytes_avail;
+	z->avail_out = out_nbytes;
 
-	return (inflate(z, Z_FINISH) == Z_STREAM_END && z->avail_out == 0);
+	return inflate(z, Z_FINISH) == Z_STREAM_END && z->avail_out == 0;
 }
 
 static void
-libz_free_compressor(void *private)
+libz_engine_destroy_decompressor(struct decompressor *d)
 {
-	deflateEnd((z_stream *)private);
-	free(private);
+	z_stream *z = d->private;
+
+	inflateEnd(z);
+	free(z);
 }
 
-static void
-libz_free_decompressor(void *private)
+static const struct engine libz_engine = {
+	.name			= T("libz"),
+
+	.init_compressor	= libz_engine_init_compressor,
+	.compress		= libz_engine_compress,
+	.destroy_compressor	= libz_engine_destroy_compressor,
+
+	.init_decompressor	= libz_engine_init_decompressor,
+	.decompress		= libz_engine_decompress,
+	.destroy_decompressor	= libz_engine_destroy_decompressor,
+};
+
+/******************************************************************************/
+
+static const struct engine * const all_engines[] = {
+	&libdeflate_engine,
+	&libz_engine,
+};
+
+#define DEFAULT_ENGINE libdeflate_engine
+
+static const struct engine *
+name_to_engine(const tchar *name)
 {
-	inflateEnd((z_stream *)private);
-	free(private);
+	size_t i;
+
+	for (i = 0; i < ARRAY_LEN(all_engines); i++)
+		if (tstrcmp(all_engines[i]->name, name) == 0)
+			return all_engines[i];
+	return NULL;
 }
 
-static size_t
-libdeflate_deflate_compress(void *private, const void *in, size_t in_nbytes,
-			    void *out, size_t out_nbytes_avail)
-{
-	return deflate_compress(private, in, in_nbytes,
-				out, out_nbytes_avail);
-}
+/******************************************************************************/
 
 static bool
-libdeflate_deflate_decompress(void *private, const void *in, size_t in_nbytes,
-			      void *out, size_t out_nbytes_avail)
+compressor_init(struct compressor *c, int level, enum wrapper wrapper,
+		const struct engine *engine)
 {
-	return 0 == deflate_decompress(private, in, in_nbytes,
-				       out, out_nbytes_avail, NULL);
-}
-
-static size_t
-libdeflate_zlib_compress(void *private, const void *in, size_t in_nbytes,
-			 void *out, size_t out_nbytes_avail)
-{
-	return zlib_compress(private, in, in_nbytes, out, out_nbytes_avail);
-}
-
-static bool
-libdeflate_zlib_decompress(void *private, const void *in, size_t in_nbytes,
-			   void *out, size_t out_nbytes_avail)
-{
-	return 0 == zlib_decompress(private, in, in_nbytes,
-				    out, out_nbytes_avail, NULL);
-}
-
-static size_t
-libdeflate_gzip_compress(void *private, const void *in, size_t in_nbytes,
-			 void *out, size_t out_nbytes_avail)
-{
-	return gzip_compress(private, in, in_nbytes, out, out_nbytes_avail);
-}
-
-static bool
-libdeflate_gzip_decompress(void *private, const void *in, size_t in_nbytes,
-			   void *out, size_t out_nbytes_avail)
-{
-	return 0 == gzip_decompress(private, in, in_nbytes,
-				    out, out_nbytes_avail, NULL);
-}
-
-static void
-libdeflate_free_compressor(void *private)
-{
-	deflate_free_compressor(private);
-}
-
-static void
-libdeflate_free_decompressor(void *private)
-{
-	deflate_free_decompressor(private);
-}
-
-static int
-compressor_init(struct compressor *c, int level,
-		enum wrapper wrapper, bool use_libz)
-{
-	if (use_libz) {
-		z_stream *z;
-
-		if (level > 9) {
-			msg("libz only supports up to compression level 9");
-			return -1;
-		}
-
-		z = xmalloc(sizeof(z_stream));
-		if (z == NULL)
-			return -1;
-
-		c->private = z;
-
-		z->next_in = NULL;
-		z->avail_in = 0;
-		z->zalloc = NULL;
-		z->zfree = NULL;
-		z->opaque = NULL;
-		if (deflateInit2(z, level, Z_DEFLATED,
-				 get_libz_window_bits(wrapper),
-				 8, Z_DEFAULT_STRATEGY) != Z_OK)
-		{
-			msg("unable to initialize deflater");
-			free(z);
-			return -1;
-		}
-		c->compress = libz_compress;
-		c->free_private = libz_free_compressor;
-	} else {
-		c->private = alloc_compressor(level);
-		if (c->private == NULL)
-			return -1;
-		switch (wrapper) {
-		case ZLIB_WRAPPER:
-			c->compress = libdeflate_zlib_compress;
-			break;
-		case GZIP_WRAPPER:
-			c->compress = libdeflate_gzip_compress;
-			break;
-		default:
-			c->compress = libdeflate_deflate_compress;
-			break;
-		}
-		c->free_private = libdeflate_free_compressor;
-	}
-	return 0;
+	c->level = level;
+	c->wrapper = wrapper;
+	c->engine = engine;
+	return engine->init_compressor(c);
 }
 
 static size_t
 do_compress(struct compressor *c, const void *in, size_t in_nbytes,
 	    void *out, size_t out_nbytes_avail)
 {
-	return (*c->compress)(c->private, in, in_nbytes, out, out_nbytes_avail);
+	return c->engine->compress(c, in, in_nbytes, out, out_nbytes_avail);
 }
 
 static void
 compressor_destroy(struct compressor *c)
 {
-	(*c->free_private)(c->private);
+	c->engine->destroy_compressor(c);
 }
 
-static int
-decompressor_init(struct decompressor *d, enum wrapper wrapper, bool use_libz)
+static bool
+decompressor_init(struct decompressor *d, enum wrapper wrapper,
+		  const struct engine *engine)
 {
-	if (use_libz) {
-		z_stream *z;
-
-		z = xmalloc(sizeof(z_stream));
-		if (z == NULL)
-			return -1;
-
-		d->private = z;
-
-		z->next_in = NULL;
-		z->avail_in = 0;
-		z->zalloc = NULL;
-		z->zfree = NULL;
-		z->opaque = NULL;
-		if (inflateInit2(z, get_libz_window_bits(wrapper)) != Z_OK) {
-			msg("unable to initialize inflater");
-			free(z);
-			return -1;
-		}
-
-		d->decompress = libz_decompress;
-		d->free_private = libz_free_decompressor;
-	} else {
-		d->private = alloc_decompressor();
-		if (d->private == NULL)
-			return -1;
-		switch (wrapper) {
-		case ZLIB_WRAPPER:
-			d->decompress = libdeflate_zlib_decompress;
-			break;
-		case GZIP_WRAPPER:
-			d->decompress = libdeflate_gzip_decompress;
-			break;
-		default:
-			d->decompress = libdeflate_deflate_decompress;
-			break;
-		}
-		d->free_private = libdeflate_free_decompressor;
-	}
-	return 0;
+	d->wrapper = wrapper;
+	d->engine = engine;
+	return engine->init_decompressor(d);
 }
 
 static bool
 do_decompress(struct decompressor *d, const void *in, size_t in_nbytes,
-	      void *out, size_t out_nbytes_avail)
+	      void *out, size_t out_nbytes)
 {
-	return (*d->decompress)(d->private, in, in_nbytes,
-				out, out_nbytes_avail);
+	return d->engine->decompress(d, in, in_nbytes, out, out_nbytes);
 }
 
 static void
 decompressor_destroy(struct decompressor *d)
 {
-	(*d->free_private)(d->private);
+	d->engine->destroy_decompressor(d);
 }
 
+/******************************************************************************/
+
+static void
+show_available_engines(FILE *fp)
+{
+	size_t i;
+
+	fprintf(fp, "Available ENGINEs are: ");
+	for (i = 0; i < ARRAY_LEN(all_engines); i++) {
+		fprintf(fp, "%"TS, all_engines[i]->name);
+		if (i < ARRAY_LEN(all_engines) - 1)
+			fprintf(fp, ", ");
+	}
+	fprintf(fp, ".  Default is %"TS"\n", DEFAULT_ENGINE.name);
+}
+
+static void
+show_usage(FILE *fp)
+{
+	fprintf(fp,
+"Usage: %"TS" [-LVL] [-C ENGINE] [-D ENGINE] [-ghVz] [-s SIZE] [FILE]...\n"
+"Benchmark DEFLATE compression and decompression on the specified FILEs.\n"
+"\n"
+"Options:\n"
+"  -1        fastest (worst) compression\n"
+"  -6        medium compression (default)\n"
+"  -12       slowest (best) compression\n"
+"  -C ENGINE compression engine\n"
+"  -D ENGINE decompression engine\n"
+"  -g        use gzip wrapper\n"
+"  -h        print this help\n"
+"  -s SIZE   chunk size\n"
+"  -V        show version and legal information\n"
+"  -z        use zlib wrapper\n"
+"\n", program_invocation_name);
+
+	show_available_engines(fp);
+}
+
+static void
+show_version(void)
+{
+	printf(
+"libdeflate compression benchmark program v" LIBDEFLATE_VERSION_STRING "\n"
+"Copyright 2016 Eric Biggers\n"
+"\n"
+"This program is free software which may be modified and/or redistributed\n"
+"under the terms of the MIT license.  There is NO WARRANTY, to the extent\n"
+"permitted by law.  See the COPYING file for details.\n"
+	);
+}
+
+
+/******************************************************************************/
+
 static int
 do_benchmark(struct file_stream *in, void *original_buf, void *compressed_buf,
 	     void *decompressed_buf, u32 chunk_size,
@@ -345,33 +413,31 @@ do_benchmark(struct file_stream *in, void *original_buf, void *compressed_buf,
 		u32 original_size = ret;
 		u32 compressed_size;
 		u64 start_time;
-		bool result;
+		bool ok;
 
 		total_uncompressed_size += original_size;
 
 		/* Compress the chunk of data. */
-		start_time = current_time();
+		start_time = timer_ticks();
 		compressed_size = do_compress(compressor,
 					      original_buf,
 					      original_size,
 					      compressed_buf,
 					      original_size - 1);
-		total_compress_time += current_time() - start_time;
+		total_compress_time += timer_ticks() - start_time;
 
 		if (compressed_size) {
 			/* Successfully compressed the chunk of data. */
 
 			/* Decompress the data we just compressed and compare
 			 * the result with the original. */
-			start_time = current_time();
-			result = do_decompress(decompressor,
-					       compressed_buf,
-					       compressed_size,
-					       decompressed_buf,
-					       original_size);
-			total_decompress_time += current_time() - start_time;
+			start_time = timer_ticks();
+			ok = do_decompress(decompressor,
+					   compressed_buf, compressed_size,
+					   decompressed_buf, original_size);
+			total_decompress_time += timer_ticks() - start_time;
 
-			if (!result) {
+			if (!ok) {
 				msg("%"TS": failed to decompress data",
 				    in->name);
 				return -1;
@@ -412,11 +478,11 @@ do_benchmark(struct file_stream *in, void *original_buf, void *compressed_buf,
 	       (unsigned int)(total_compressed_size * 100000 /
 				total_uncompressed_size % 1000));
 	printf("\tCompression time: %"PRIu64" ms (%"PRIu64" MB/s)\n",
-	       total_compress_time / 1000000,
-	       1000 * total_uncompressed_size / total_compress_time);
+	       timer_ticks_to_ms(total_compress_time),
+	       timer_MB_per_s(total_uncompressed_size, total_compress_time));
 	printf("\tDecompression time: %"PRIu64" ms (%"PRIu64" MB/s)\n",
-	       total_decompress_time / 1000000,
-	       1000 * total_uncompressed_size / total_decompress_time);
+	       timer_ticks_to_ms(total_decompress_time),
+	       timer_MB_per_s(total_uncompressed_size, total_decompress_time));
 
 	return 0;
 }
@@ -427,8 +493,8 @@ tmain(int argc, tchar *argv[])
 	u32 chunk_size = 1048576;
 	int level = 6;
 	enum wrapper wrapper = NO_WRAPPER;
-	bool compress_with_libz = false;
-	bool decompress_with_libz = false;
+	const struct engine *compress_engine = &DEFAULT_ENGINE;
+	const struct engine *decompress_engine = &DEFAULT_ENGINE;
 	void *original_buf = NULL;
 	void *compressed_buf = NULL;
 	void *decompressed_buf = NULL;
@@ -454,7 +520,23 @@ tmain(int argc, tchar *argv[])
 		case '9':
 			level = parse_compression_level(opt_char, toptarg);
 			if (level == 0)
-				return -1;
+				return 1;
+			break;
+		case 'C':
+			compress_engine = name_to_engine(toptarg);
+			if (compress_engine == NULL) {
+				msg("invalid compression engine: \"%"TS"\"", toptarg);
+				show_available_engines(stderr);
+				return 1;
+			}
+			break;
+		case 'D':
+			decompress_engine = name_to_engine(toptarg);
+			if (decompress_engine == NULL) {
+				msg("invalid decompression engine: \"%"TS"\"", toptarg);
+				show_available_engines(stderr);
+				return 1;
+			}
 			break;
 		case 'g':
 			wrapper = GZIP_WRAPPER;
@@ -472,11 +554,11 @@ tmain(int argc, tchar *argv[])
 		case 'V':
 			show_version();
 			return 0;
-		case 'Y':
-			compress_with_libz = true;
+		case 'Y': /* deprecated, use '-C libz' instead */
+			compress_engine = &libz_engine;
 			break;
-		case 'Z':
-			decompress_with_libz = true;
+		case 'Z': /* deprecated, use '-D libz' instead */
+			decompress_engine = &libz_engine;
 			break;
 		case 'z':
 			wrapper = ZLIB_WRAPPER;
@@ -499,12 +581,10 @@ tmain(int argc, tchar *argv[])
 	    decompressed_buf == NULL)
 		goto out0;
 
-	ret = compressor_init(&compressor, level, wrapper, compress_with_libz);
-	if (ret)
+	if (!compressor_init(&compressor, level, wrapper, compress_engine))
 		goto out0;
 
-	ret = decompressor_init(&decompressor, wrapper, decompress_with_libz);
-	if (ret)
+	if (!decompressor_init(&decompressor, wrapper, decompress_engine))
 		goto out1;
 
 	if (argc == 0) {
@@ -522,15 +602,13 @@ tmain(int argc, tchar *argv[])
 	printf("\tWrapper: %s\n",
 	       wrapper == NO_WRAPPER ? "None" :
 	       wrapper == ZLIB_WRAPPER ? "zlib" : "gzip");
-	printf("\tCompression engine: %s\n",
-	       compress_with_libz ? "libz" : "libdeflate");
-	printf("\tDecompression engine: %s\n",
-	       decompress_with_libz ? "libz" : "libdeflate");
+	printf("\tCompression engine: %"TS"\n", compress_engine->name);
+	printf("\tDecompression engine: %"TS"\n", decompress_engine->name);
 
 	for (i = 0; i < argc; i++) {
 		struct file_stream in;
 
-		ret = xopen_for_read(argv[i], &in);
+		ret = xopen_for_read(argv[i], true, &in);
 		if (ret != 0)
 			goto out2;
 

programs/checksum.c

@@ -0,0 +1,197 @@
+/*
+ * checksum.c - Adler-32 and CRC-32 checksumming program
+ *
+ * Copyright 2016 Eric Biggers
+ *
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include <zlib.h>
+
+#include "prog_util.h"
+
+static const tchar *const optstring = T("Ahs:tZ");
+
+static void
+show_usage(FILE *fp)
+{
+	fprintf(fp,
+"Usage: %"TS" [-A] [-h] [-s SIZE] [-t] [-Z] [FILE]...\n"
+"Calculate Adler-32 or CRC-32 checksums of the specified FILEs.\n"
+"\n"
+"Options:\n"
+"  -A        use Adler-32 (default is CRC-32)\n"
+"  -h        print this help\n"
+"  -s SIZE   chunk size\n"
+"  -t        show checksum speed, excluding I/O\n"
+"  -Z        use zlib implementation instead of libdeflate\n",
+	program_invocation_name);
+}
+
+static u32
+zlib_adler32(u32 adler, const void *buf, size_t len)
+{
+	return adler32(adler, buf, len);
+}
+
+static u32
+zlib_crc32(u32 crc, const void *buf, size_t len)
+{
+	return crc32(crc, buf, len);
+}
+
+typedef u32 (*cksum_fn_t)(u32, const void *, size_t);
+
+static int
+checksum_stream(struct file_stream *in, cksum_fn_t cksum, u32 *sum,
+		void *buf, size_t bufsize, u64 *size_ret, u64 *elapsed_ret)
+{
+	u64 size = 0;
+	u64 elapsed = 0;
+
+	for (;;) {
+		ssize_t ret;
+		u64 start_time;
+
+		ret = xread(in, buf, bufsize);
+		if (ret < 0)
+			return ret;
+		if (ret == 0)
+			break;
+
+		size += ret;
+		start_time = timer_ticks();
+		*sum = cksum(*sum, buf, ret);
+		elapsed += timer_ticks() - start_time;
+	}
+
+	if (elapsed == 0)
+		elapsed = 1;
+	*size_ret = size;
+	*elapsed_ret = elapsed;
+	return 0;
+}
+
+int
+tmain(int argc, tchar *argv[])
+{
+	bool use_adler32 = false;
+	bool use_zlib_impl = false;
+	bool do_timing = false;
+	void *buf;
+	size_t bufsize = 131072;
+	tchar *default_file_list[] = { NULL };
+	cksum_fn_t cksum;
+	int opt_char;
+	int i;
+	int ret;
+
+	program_invocation_name = get_filename(argv[0]);
+
+	while ((opt_char = tgetopt(argc, argv, optstring)) != -1) {
+		switch (opt_char) {
+		case 'A':
+			use_adler32 = true;
+			break;
+		case 'h':
+			show_usage(stdout);
+			return 0;
+		case 's':
+			bufsize = tstrtoul(toptarg, NULL, 10);
+			if (bufsize == 0) {
+				msg("invalid chunk size: \"%"TS"\"", toptarg);
+				return 1;
+			}
+			break;
+		case 't':
+			do_timing = true;
+			break;
+		case 'Z':
+			use_zlib_impl = true;
+			break;
+		default:
+			show_usage(stderr);
+			return 1;
+		}
+	}
+
+	argc -= toptind;
+	argv += toptind;
+
+	if (use_adler32) {
+		if (use_zlib_impl)
+			cksum = zlib_adler32;
+		else
+			cksum = libdeflate_adler32;
+	} else {
+		if (use_zlib_impl)
+			cksum = zlib_crc32;
+		else
+			cksum = libdeflate_crc32;
+	}
+
+	buf = xmalloc(bufsize);
+	if (buf == NULL)
+		return 1;
+
+	if (argc == 0) {
+		argv = default_file_list;
+		argc = ARRAY_LEN(default_file_list);
+	} else {
+		for (i = 0; i < argc; i++)
+			if (argv[i][0] == '-' && argv[i][1] == '\0')
+				argv[i] = NULL;
+	}
+
+	for (i = 0; i < argc; i++) {
+		struct file_stream in;
+		u32 sum = cksum(0, NULL, 0);
+		u64 size = 0;
+		u64 elapsed = 0;
+
+		ret = xopen_for_read(argv[i], true, &in);
+		if (ret != 0)
+			goto out;
+
+		ret = checksum_stream(&in, cksum, &sum, buf, bufsize,
+				      &size, &elapsed);
+		if (ret == 0) {
+			if (do_timing) {
+				printf("%08"PRIx32"\t%"TS"\t"
+				       "%"PRIu64" ms\t%"PRIu64" MB/s\n",
+				       sum, in.name, timer_ticks_to_ms(elapsed),
+				       timer_MB_per_s(size, elapsed));
+			} else {
+				printf("%08"PRIx32"\t%"TS"\t\n", sum, in.name);
+			}
+		}
+
+		xclose(&in);
+
+		if (ret != 0)
+			goto out;
+	}
+	ret = 0;
+out:
+	free(buf);
+	return -ret;
+}

programs/detect.sh

@@ -5,28 +5,58 @@ if [ -z "$CC" ]; then
 fi
 
 echo "/* THIS FILE WAS AUTOMATICALLY GENERATED.  DO NOT EDIT. */"
-echo "#ifndef _CONFIG_H"
-echo "#define _CONFIG_H"
+echo "#ifndef CONFIG_H"
+echo "#define CONFIG_H"
 
 tmpfile="$(mktemp -t libdeflate_config.XXXXXXXX)"
 trap "rm -f \"$tmpfile\"" EXIT
 
+program_compiles() {
+	echo "$1" > "$tmpfile"
+	$CC $CFLAGS -x c "$tmpfile" -o /dev/null > /dev/null 2>&1
+}
+
 check_function() {
 	funcname="$1"
 	macro="HAVE_$(echo $funcname | tr a-z A-Z)"
-	echo "int main() { $funcname(); }" > "$tmpfile"
 	echo
 	echo "/* Is the $funcname() function available? */"
-	if $CC -x c $tmpfile -o /dev/null > /dev/null 2>&1; then
+	if program_compiles "int main() { $funcname(); }"; then
 		echo "#define $macro 1"
 	else
 		echo "/* $macro is not set */"
 	fi
 }
 
+have_stat_field() {
+	program_compiles "#include <sys/types.h>
+			  #include <sys/stat.h>
+			  int main() { struct stat st; st.$1; }"
+}
+
+check_stat_nanosecond_precision() {
+	echo
+	echo "/* Does stat() provide nanosecond-precision timestamps? */"
+	if have_stat_field st_atim; then
+		echo "#define HAVE_STAT_NANOSECOND_PRECISION 1"
+	elif have_stat_field st_atimespec; then
+		# Nonstandard field names used by OS X and older BSDs
+		echo "#define HAVE_STAT_NANOSECOND_PRECISION 1"
+		echo "#define st_atim st_atimespec"
+		echo "#define st_mtim st_mtimespec"
+		echo "#define st_ctim st_ctimespec"
+	else
+		echo "/* HAVE_STAT_NANOSECOND_PRECISION is not set */"
+	fi
+}
+
 check_function clock_gettime
 check_function futimens
 check_function futimes
+check_function posix_fadvise
+check_function posix_madvise
+
+check_stat_nanosecond_precision
 
 echo
-echo "#endif /* _CONFIG_H */"
+echo "#endif /* CONFIG_H */"

programs/gzip.c

@@ -27,6 +27,7 @@
 
 #include "prog_util.h"
 
+#include <errno.h>
 #include <sys/types.h>
 #include <sys/stat.h>
 #ifdef _WIN32
@@ -46,7 +47,7 @@ struct options {
 	const tchar *suffix;
 };
 
-static const tchar *const optstring = T("1::2::3::4::5::6::7::8::9::cdfhkS:V");
+static const tchar *const optstring = T("1::2::3::4::5::6::7::8::9::cdfhknS:V");
 
 static void
 show_usage(FILE *fp)
@@ -64,7 +65,7 @@ show_usage(FILE *fp)
 "  -f        overwrite existing output files\n"
 "  -h        print this help\n"
 "  -k        don't delete input files\n"
-"  -S SUF    use suffix .SUF instead of .gz\n"
+"  -S SUF    use suffix SUF instead of .gz\n"
 "  -V        show version and legal information\n",
 	program_invocation_name);
 }
@@ -88,9 +89,13 @@ is_gunzip(void)
 {
 	if (tstrxcmp(program_invocation_name, T("gunzip")) == 0)
 		return true;
+	if (tstrxcmp(program_invocation_name, T("libdeflate-gunzip")) == 0)
+		return true;
 #ifdef _WIN32
 	if (tstrxcmp(program_invocation_name, T("gunzip.exe")) == 0)
 		return true;
+	if (tstrxcmp(program_invocation_name, T("libdeflate-gunzip.exe")) == 0)
+		return true;
 #endif
 	return false;
 }
@@ -98,10 +103,15 @@ is_gunzip(void)
 static const tchar *
 get_suffix(const tchar *path, const tchar *suffix)
 {
-	const tchar *dot = tstrrchr(get_filename(path), '.');
+	size_t path_len = tstrlen(path);
+	size_t suffix_len = tstrlen(suffix);
+	const tchar *p;
 
-	if (dot != NULL && tstrxcmp(dot + 1, suffix) == 0)
-		return dot;
+	if (path_len <= suffix_len)
+		return NULL;
+	p = &path[path_len - suffix_len];
+	if (tstrxcmp(p, suffix) == 0)
+		return p;
 	return NULL;
 }
 
@@ -111,8 +121,23 @@ has_suffix(const tchar *path, const tchar *suffix)
 	return get_suffix(path, suffix) != NULL;
 }
 
+static tchar *
+append_suffix(const tchar *path, const tchar *suffix)
+{
+	size_t path_len = tstrlen(path);
+	size_t suffix_len = tstrlen(suffix);
+	tchar *suffixed_path;
+
+	suffixed_path = xmalloc((path_len + suffix_len + 1) * sizeof(tchar));
+	if (suffixed_path == NULL)
+		return NULL;
+	tmemcpy(suffixed_path, path, path_len);
+	tmemcpy(&suffixed_path[path_len], suffix, suffix_len + 1);
+	return suffixed_path;
+}
+
 static int
-do_compress(struct deflate_compressor *compressor,
+do_compress(struct libdeflate_compressor *compressor,
 	    struct file_stream *in, struct file_stream *out)
 {
 	const void *uncompressed_data = in->mmap_mem;
@@ -122,7 +147,8 @@ do_compress(struct deflate_compressor *compressor,
 	size_t max_compressed_size;
 	int ret;
 
-	max_compressed_size = gzip_compress_bound(compressor, uncompressed_size);
+	max_compressed_size = libdeflate_gzip_compress_bound(compressor,
+							     uncompressed_size);
 	compressed_data = xmalloc(max_compressed_size);
 	if (compressed_data == NULL) {
 		msg("%"TS": file is probably too large to be processed by this "
@@ -131,12 +157,13 @@ do_compress(struct deflate_compressor *compressor,
 		goto out;
 	}
 
-	actual_compressed_size =
-		gzip_compress(compressor, uncompressed_data, uncompressed_size,
-			      compressed_data, max_compressed_size);
-
+	actual_compressed_size = libdeflate_gzip_compress(compressor,
+							  uncompressed_data,
+							  uncompressed_size,
+							  compressed_data,
+							  max_compressed_size);
 	if (actual_compressed_size == 0) {
-		msg("Bug in gzip_compress_bound()!");
+		msg("Bug in libdeflate_gzip_compress_bound()!");
 		ret = -1;
 		goto out;
 	}
@@ -155,15 +182,17 @@ load_u32_gzip(const u8 *p)
 }
 
 static int
-do_decompress(struct deflate_decompressor *decompressor,
+do_decompress(struct libdeflate_decompressor *decompressor,
 	      struct file_stream *in, struct file_stream *out)
 {
 	const u8 *compressed_data = in->mmap_mem;
 	size_t compressed_size = in->mmap_size;
 	void *uncompressed_data = NULL;
 	size_t uncompressed_size;
-	enum decompress_result result;
-	int ret;
+	size_t actual_in_nbytes;
+	size_t actual_out_nbytes;
+	enum libdeflate_result result;
+	int ret = 0;
 
 	if (compressed_size < sizeof(u32)) {
 	       msg("%"TS": not in gzip format", in->name);
@@ -173,40 +202,70 @@ do_decompress(struct deflate_decompressor *decompressor,
 
 	uncompressed_size = load_u32_gzip(&compressed_data[compressed_size - 4]);
 
-	uncompressed_data = xmalloc(uncompressed_size);
-	if (uncompressed_data == NULL) {
-		msg("%"TS": file is probably too large to be processed by this "
-		    "program", in->name);
-		ret = -1;
-		goto out;
-	}
+	do {
+		if (uncompressed_data == NULL) {
+			uncompressed_data = xmalloc(uncompressed_size);
+			if (uncompressed_data == NULL) {
+				msg("%"TS": file is probably too large to be "
+				    "processed by this program", in->name);
+				ret = -1;
+				goto out;
+			}
+		}
 
-	result = gzip_decompress(decompressor, compressed_data, compressed_size,
-				 uncompressed_data, uncompressed_size, NULL);
+		result = libdeflate_gzip_decompress_ex(decompressor,
+						       compressed_data,
+						       compressed_size,
+						       uncompressed_data,
+						       uncompressed_size,
+						       &actual_in_nbytes,
+						       &actual_out_nbytes);
 
-	if (result == DECOMPRESS_INSUFFICIENT_SPACE) {
-		msg("%"TS": file corrupt or too large to be processed by this "
-		    "program", in->name);
-		ret = -1;
-		goto out;
-	}
+		if (result == LIBDEFLATE_INSUFFICIENT_SPACE) {
+			if (uncompressed_size * 2 <= uncompressed_size) {
+				msg("%"TS": file corrupt or too large to be "
+				    "processed by this program", in->name);
+				ret = -1;
+				goto out;
+			}
+			uncompressed_size *= 2;
+			free(uncompressed_data);
+			uncompressed_data = NULL;
+			continue;
+		}
 
-	if (result != DECOMPRESS_SUCCESS) {
-		msg("%"TS": file corrupt or not in gzip format", in->name);
-		ret = -1;
-		goto out;
-	}
+		if (result != LIBDEFLATE_SUCCESS) {
+			msg("%"TS": file corrupt or not in gzip format",
+			    in->name);
+			ret = -1;
+			goto out;
+		}
 
-	ret = full_write(out, uncompressed_data, uncompressed_size);
+		if (actual_in_nbytes == 0 ||
+		    actual_in_nbytes > compressed_size ||
+		    actual_out_nbytes > uncompressed_size) {
+			msg("Bug in libdeflate_gzip_decompress_ex()!");
+			ret = -1;
+			goto out;
+		}
+
+		ret = full_write(out, uncompressed_data, actual_out_nbytes);
+		if (ret != 0)
+			goto out;
+
+		compressed_data += actual_in_nbytes;
+		compressed_size -= actual_in_nbytes;
+
+	} while (compressed_size != 0);
 out:
 	free(uncompressed_data);
 	return ret;
 }
 
 static int
-stat_file(struct file_stream *in, struct stat *stbuf, bool allow_hard_links)
+stat_file(struct file_stream *in, stat_t *stbuf, bool allow_hard_links)
 {
-	if (fstat(in->fd, stbuf) != 0) {
+	if (tfstat(in->fd, stbuf) != 0) {
 		msg("%"TS": unable to stat file", in->name);
 		return -1;
 	}
@@ -228,7 +287,7 @@ stat_file(struct file_stream *in, struct stat *stbuf, bool allow_hard_links)
 }
 
 static void
-restore_mode(struct file_stream *out, const struct stat *stbuf)
+restore_mode(struct file_stream *out, const stat_t *stbuf)
 {
 #ifndef _WIN32
 	if (fchmod(out->fd, stbuf->st_mode) != 0)
@@ -237,7 +296,7 @@ restore_mode(struct file_stream *out, const struct stat *stbuf)
 }
 
 static void
-restore_owner_and_group(struct file_stream *out, const struct stat *stbuf)
+restore_owner_and_group(struct file_stream *out, const stat_t *stbuf)
 {
 #ifndef _WIN32
 	if (fchown(out->fd, stbuf->st_uid, stbuf->st_gid) != 0) {
@@ -249,33 +308,33 @@ restore_owner_and_group(struct file_stream *out, const struct stat *stbuf)
 
 static void
 restore_timestamps(struct file_stream *out, const tchar *newpath,
-		   const struct stat *stbuf)
+		   const stat_t *stbuf)
 {
 	int ret;
-#if defined(HAVE_FUTIMENS)
+#if defined(HAVE_FUTIMENS) && defined(HAVE_STAT_NANOSECOND_PRECISION)
 	struct timespec times[2] = {
 		stbuf->st_atim, stbuf->st_mtim,
 	};
 	ret = futimens(out->fd, times);
-#elif defined(HAVE_FUTIMES)
+#elif defined(HAVE_FUTIMES) && defined(HAVE_STAT_NANOSECOND_PRECISION)
 	struct timeval times[2] = {
 		{ stbuf->st_atim.tv_sec, stbuf->st_atim.tv_nsec / 1000, },
 		{ stbuf->st_mtim.tv_sec, stbuf->st_mtim.tv_nsec / 1000, },
 	};
 	ret = futimes(out->fd, times);
-#else /* HAVE_FUTIMES */
+#else
 	struct tutimbuf times = {
 		stbuf->st_atime, stbuf->st_mtime,
 	};
 	ret = tutime(newpath, &times);
-#endif /* !HAVE_FUTIMES */
+#endif
 	if (ret != 0)
 		msg_errno("%"TS": unable to preserve timestamps", out->name);
 }
 
 static void
 restore_metadata(struct file_stream *out, const tchar *newpath,
-		 const struct stat *stbuf)
+		 const stat_t *stbuf)
 {
 	restore_mode(out, stbuf);
 	restore_owner_and_group(out, stbuf);
@@ -283,32 +342,58 @@ restore_metadata(struct file_stream *out, const tchar *newpath,
 }
 
 static int
-decompress_file(struct deflate_decompressor *decompressor, const tchar *path,
+decompress_file(struct libdeflate_decompressor *decompressor, const tchar *path,
 		const struct options *options)
 {
+	tchar *oldpath = (tchar *)path;
 	tchar *newpath = NULL;
 	struct file_stream in;
 	struct file_stream out;
-	struct stat stbuf;
+	stat_t stbuf;
 	int ret;
 	int ret2;
 
-	if (path != NULL && !options->to_stdout) {
+	if (path != NULL) {
 		const tchar *suffix = get_suffix(path, options->suffix);
 		if (suffix == NULL) {
-			msg("\"%"TS"\" does not end with the .%"TS" suffix -- "
-			    "skipping", path, options->suffix);
-			ret = -2;
-			goto out;
+			/*
+			 * Input file is unsuffixed.  If the file doesn't exist,
+			 * then try it suffixed.  Otherwise, if we're not
+			 * writing to stdout, skip the file with warning status.
+			 * Otherwise, go ahead and try to open the file anyway
+			 * (which will very likely fail).
+			 */
+			if (tstat(path, &stbuf) != 0 && errno == ENOENT) {
+				oldpath = append_suffix(path, options->suffix);
+				if (oldpath == NULL)
+					return -1;
+				if (!options->to_stdout)
+					newpath = (tchar *)path;
+			} else if (!options->to_stdout) {
+				msg("\"%"TS"\" does not end with the %"TS" "
+				    "suffix -- skipping",
+				    path, options->suffix);
+				return -2;
+			}
+		} else if (!options->to_stdout) {
+			/*
+			 * Input file is suffixed, and we're not writing to
+			 * stdout.  Strip the suffix to get the path to the
+			 * output file.
+			 */
+			newpath = xmalloc((suffix - oldpath + 1) *
+					  sizeof(tchar));
+			if (newpath == NULL)
+				return -1;
+			tmemcpy(newpath, oldpath, suffix - oldpath);
+			newpath[suffix - oldpath] = '\0';
 		}
-		newpath = xmalloc((suffix - path + 1) * sizeof(tchar));
-		tmemcpy(newpath, path, suffix - path);
-		newpath[suffix - path] = '\0';
 	}
 
-	ret = xopen_for_read(path, &in);
+	ret = xopen_for_read(oldpath, options->force || options->to_stdout,
+			     &in);
 	if (ret != 0)
-		goto out_free_newpath;
+		goto out_free_paths;
 
 	if (!options->force && isatty(in.fd)) {
 		msg("Refusing to read compressed data from terminal.  "
@@ -317,7 +402,8 @@ decompress_file(struct deflate_decompressor *decompressor, const tchar *path,
 		goto out_close_in;
 	}
 
-	ret = stat_file(&in, &stbuf, options->force || newpath == NULL);
+	ret = stat_file(&in, &stbuf, options->force || options->keep ||
+			oldpath == NULL || newpath == NULL);
 	if (ret != 0)
 		goto out_close_in;
 
@@ -325,6 +411,7 @@ decompress_file(struct deflate_decompressor *decompressor, const tchar *path,
 	if (ret != 0)
 		goto out_close_in;
 
+	/* TODO: need a streaming-friendly solution */
 	ret = map_file_contents(&in, stbuf.st_size);
 	if (ret != 0)
 		goto out_close_out;
@@ -333,7 +420,7 @@ decompress_file(struct deflate_decompressor *decompressor, const tchar *path,
 	if (ret != 0)
 		goto out_close_out;
 
-	if (path != NULL && newpath != NULL)
+	if (oldpath != NULL && newpath != NULL)
 		restore_metadata(&out, newpath, &stbuf);
 	ret = 0;
 out_close_out:
@@ -344,49 +431,44 @@ out_close_out:
 		tunlink(newpath);
 out_close_in:
 	xclose(&in);
-	if (ret == 0 && path != NULL && newpath != NULL && !options->keep)
-		tunlink(path);
-out_free_newpath:
-	free(newpath);
-out:
+	if (ret == 0 && oldpath != NULL && newpath != NULL && !options->keep)
+		tunlink(oldpath);
+out_free_paths:
+	if (newpath != path)
+		free(newpath);
+	if (oldpath != path)
+		free(oldpath);
 	return ret;
 }
 
 static int
-compress_file(struct deflate_compressor *compressor, const tchar *path,
+compress_file(struct libdeflate_compressor *compressor, const tchar *path,
 	      const struct options *options)
 {
 	tchar *newpath = NULL;
 	struct file_stream in;
 	struct file_stream out;
-	struct stat stbuf;
+	stat_t stbuf;
 	int ret;
 	int ret2;
 
 	if (path != NULL && !options->to_stdout) {
-		size_t path_nchars, suffix_nchars;
-
 		if (!options->force && has_suffix(path, options->suffix)) {
-			msg("%"TS": already has .%"TS" suffix -- skipping",
+			msg("%"TS": already has %"TS" suffix -- skipping",
 			    path, options->suffix);
-			ret = -2;
-			goto out;
+			return 0;
 		}
-		path_nchars = tstrlen(path);
-		suffix_nchars = tstrlen(options->suffix);
-		newpath = xmalloc((path_nchars + 1 + suffix_nchars + 1) *
-					sizeof(tchar));
-		tmemcpy(newpath, path, path_nchars);
-		newpath[path_nchars] = '.';
-		tmemcpy(&newpath[path_nchars + 1], options->suffix,
-			suffix_nchars + 1);
+		newpath = append_suffix(path, options->suffix);
+		if (newpath == NULL)
+			return -1;
 	}
 
-	ret = xopen_for_read(path, &in);
+	ret = xopen_for_read(path, options->force || options->to_stdout, &in);
 	if (ret != 0)
 		goto out_free_newpath;
 
-	ret = stat_file(&in, &stbuf, options->force || newpath == NULL);
+	ret = stat_file(&in, &stbuf, options->force || options->keep ||
+			path == NULL || newpath == NULL);
 	if (ret != 0)
 		goto out_close_in;
 
@@ -401,8 +483,9 @@ compress_file(struct deflate_compressor *compressor, const tchar *path,
 		goto out_close_out;
 	}
 
+	/* TODO: need a streaming-friendly solution */
 	ret = map_file_contents(&in, stbuf.st_size);
-	if (ret)
+	if (ret != 0)
 		goto out_close_out;
 
 	ret = do_compress(compressor, &in, &out);
@@ -424,13 +507,13 @@ out_close_in:
 		tunlink(path);
 out_free_newpath:
 	free(newpath);
-out:
 	return ret;
 }
 
 int
 tmain(int argc, tchar *argv[])
 {
+	tchar *default_file_list[] = { NULL };
 	struct options options;
 	int opt_char;
 	int i;
@@ -443,7 +526,7 @@ tmain(int argc, tchar *argv[])
 	options.force = false;
 	options.keep = false;
 	options.compression_level = 6;
-	options.suffix = T("gz");
+	options.suffix = T(".gz");
 
 	while ((opt_char = tgetopt(argc, argv, optstring)) != -1) {
 		switch (opt_char) {
@@ -459,7 +542,7 @@ tmain(int argc, tchar *argv[])
 			options.compression_level =
 				parse_compression_level(opt_char, toptarg);
 			if (options.compression_level == 0)
-				return -1;
+				return 1;
 			break;
 		case 'c':
 			options.to_stdout = true;
@@ -476,8 +559,20 @@ tmain(int argc, tchar *argv[])
 		case 'k':
 			options.keep = true;
 			break;
+		case 'n':
+			/*
+			 * -n means don't save or restore the original filename
+			 *  in the gzip header.  Currently this implementation
+			 *  already behaves this way by default, so accept the
+			 *  option as a no-op.
+			 */
+			break;
 		case 'S':
 			options.suffix = toptarg;
+			if (options.suffix[0] == T('\0')) {
+				msg("invalid suffix");
+				return 1;
+			}
 			break;
 		case 'V':
 			show_version();
@@ -492,20 +587,17 @@ tmain(int argc, tchar *argv[])
 	argc -= toptind;
 
 	if (argc == 0) {
-		show_usage(stderr);
-		return 1;
-	}
-	for (i = 0; i < argc; i++) {
-		if (argv[i][0] == '-' && argv[i][1] == '\0') {
-			msg("This implementation of gzip does not yet "
-			    "support reading from standard input.");
-			return 1;
-		}
+		argv = default_file_list;
+		argc = ARRAY_LEN(default_file_list);
+	} else {
+		for (i = 0; i < argc; i++)
+			if (argv[i][0] == '-' && argv[i][1] == '\0')
+				argv[i] = NULL;
 	}
 
 	ret = 0;
 	if (options.decompress) {
-		struct deflate_decompressor *d;
+		struct libdeflate_decompressor *d;
 
 		d = alloc_decompressor();
 		if (d == NULL)
@@ -514,9 +606,9 @@ tmain(int argc, tchar *argv[])
 		for (i = 0; i < argc; i++)
 			ret |= -decompress_file(d, argv[i], &options);
 
-		deflate_free_decompressor(d);
+		libdeflate_free_decompressor(d);
 	} else {
-		struct deflate_compressor *c;
+		struct libdeflate_compressor *c;
 
 		c = alloc_compressor(options.compression_level);
 		if (c == NULL)
@@ -525,7 +617,7 @@ tmain(int argc, tchar *argv[])
 		for (i = 0; i < argc; i++)
 			ret |= -compress_file(c, argv[i], &options);
 
-		deflate_free_compressor(c);
+		libdeflate_free_compressor(c);
 	}
 
 	/*

programs/prog_util.c

@@ -39,6 +39,22 @@
 #  include <sys/time.h>
 #endif
 
+#ifndef O_BINARY
+#  define O_BINARY 0
+#endif
+#ifndef O_SEQUENTIAL
+#  define O_SEQUENTIAL 0
+#endif
+#ifndef O_NOFOLLOW
+#  define O_NOFOLLOW 0
+#endif
+#ifndef O_NONBLOCK
+#  define O_NONBLOCK 0
+#endif
+#ifndef O_NOCTTY
+#  define O_NOCTTY 0
+#endif
+
 /* The invocation name of the program (filename component only) */
 const tchar *program_invocation_name;
 
@@ -92,16 +108,16 @@ xmalloc(size_t size)
 }
 
 /*
- * Retrieve the current time in nanoseconds since a start time which is fixed
- * for the duration of program execution but is otherwise unspecified
+ * Return the number of timer ticks that have elapsed since some unspecified
+ * point fixed at the start of program execution
  */
 u64
-current_time(void)
+timer_ticks(void)
 {
 #ifdef _WIN32
-	FILETIME ft;
-	GetSystemTimeAsFileTime(&ft);
-	return 100 * (((u64)ft.dwHighDateTime << 32) | ft.dwLowDateTime);
+	LARGE_INTEGER count;
+	QueryPerformanceCounter(&count);
+	return count.QuadPart;
 #elif defined(HAVE_CLOCK_GETTIME)
 	struct timespec ts;
 	clock_gettime(CLOCK_MONOTONIC, &ts);
@@ -109,10 +125,43 @@ current_time(void)
 #else
 	struct timeval tv;
 	gettimeofday(&tv, NULL);
-	return (1000000000 * (u64)tv.tv_sec) + (1000 * (u64)tv.tv_usec);
+	return (1000000 * (u64)tv.tv_sec) + tv.tv_usec;
 #endif
 }
 
+/*
+ * Return the number of timer ticks per second
+ */
+static u64
+timer_frequency(void)
+{
+#ifdef _WIN32
+	LARGE_INTEGER freq;
+	QueryPerformanceFrequency(&freq);
+	return freq.QuadPart;
+#elif defined(HAVE_CLOCK_GETTIME)
+	return 1000000000;
+#else
+	return 1000000;
+#endif
+}
+
+/*
+ * Convert a number of elapsed timer ticks to milliseconds
+ */
+u64 timer_ticks_to_ms(u64 ticks)
+{
+	return ticks * 1000 / timer_frequency();
+}
+
+/*
+ * Convert a byte count and a number of elapsed timer ticks to MB/s
+ */
+u64 timer_MB_per_s(u64 bytes, u64 ticks)
+{
+	return bytes * timer_frequency() / ticks / 1000000;
+}
+
 /*
  * Retrieve a pointer to the filename component of the specified path.
  *
@@ -153,8 +202,9 @@ quote_path(const tchar *path)
 
 /* Open a file for reading, or set up standard input for reading */
 int
-xopen_for_read(const tchar *path, struct file_stream *strm)
+xopen_for_read(const tchar *path, bool symlink_ok, struct file_stream *strm)
 {
+	strm->mmap_token = NULL;
 	strm->mmap_mem = NULL;
 
 	if (path == NULL) {
@@ -173,13 +223,18 @@ xopen_for_read(const tchar *path, struct file_stream *strm)
 	if (strm->name == NULL)
 		return -1;
 
-	strm->fd = topen(path, O_RDONLY | O_BINARY | O_NOFOLLOW);
+	strm->fd = topen(path, O_RDONLY | O_BINARY | O_NONBLOCK | O_NOCTTY |
+			 (symlink_ok ? 0 : O_NOFOLLOW) | O_SEQUENTIAL);
 	if (strm->fd < 0) {
 		msg_errno("Can't open %"TS" for reading", strm->name);
 		free(strm->name);
 		return -1;
 	}
 
+#if defined(HAVE_POSIX_FADVISE) && (O_SEQUENTIAL == 0)
+	posix_fadvise(strm->fd, 0, 0, POSIX_FADV_SEQUENTIAL);
+#endif
+
 	return 0;
 }
 
@@ -187,6 +242,9 @@ xopen_for_read(const tchar *path, struct file_stream *strm)
 int
 xopen_for_write(const tchar *path, bool overwrite, struct file_stream *strm)
 {
+	int ret = -1;
+
+	strm->mmap_token = NULL;
 	strm->mmap_mem = NULL;
 
 	if (path == NULL) {
@@ -216,6 +274,7 @@ retry:
 			if (!isatty(STDERR_FILENO) || !isatty(STDIN_FILENO)) {
 				msg("%"TS" already exists; use -f to overwrite",
 				    strm->name);
+				ret = -2; /* warning only */
 				goto err;
 			}
 			fprintf(stderr, "%"TS": %"TS" already exists; "
@@ -237,13 +296,60 @@ retry:
 
 err:
 	free(strm->name);
-	return -1;
+	return ret;
+}
+
+/* Read the full contents of a file into memory */
+static int
+read_full_contents(struct file_stream *strm)
+{
+	size_t filled = 0;
+	size_t capacity = 4096;
+	char *buf;
+	int ret;
+
+	buf = xmalloc(capacity);
+	if (buf == NULL)
+		return -1;
+	do {
+		if (filled == capacity) {
+			char *newbuf;
+
+			if (capacity == SIZE_MAX)
+				goto oom;
+			capacity += MIN(SIZE_MAX - capacity, capacity);
+			newbuf = realloc(buf, capacity);
+			if (newbuf == NULL)
+				goto oom;
+			buf = newbuf;
+		}
+		ret = xread(strm, &buf[filled], capacity - filled);
+		if (ret < 0)
+			goto err;
+		filled += ret;
+	} while (ret != 0);
+
+	strm->mmap_mem = buf;
+	strm->mmap_size = filled;
+	return 0;
+
+err:
+	free(buf);
+	return ret;
+oom:
+	msg("Out of memory!  %"TS" is too large to be processed by "
+	    "this program as currently implemented.", strm->name);
+	ret = -1;
+	goto err;
 }
 
 /* Map the contents of a file into memory */
 int
 map_file_contents(struct file_stream *strm, u64 size)
 {
+	if (size == 0) /* mmap isn't supported on empty files */
+		return read_full_contents(strm);
+
 	if (size > SIZE_MAX) {
 		msg("%"TS" is too large to be processed by this program",
 		    strm->name);
@@ -254,8 +360,11 @@ map_file_contents(struct file_stream *strm, u64 size)
 				(HANDLE)(intptr_t)_get_osfhandle(strm->fd),
 				NULL, PAGE_READONLY, 0, 0, NULL);
 	if (strm->mmap_token == NULL) {
+		DWORD err = GetLastError();
+		if (err == ERROR_BAD_EXE_FORMAT) /* mmap unsupported */
+			return read_full_contents(strm);
 		msg("Unable create file mapping for %"TS": Windows error %u",
-		    strm->name, (unsigned int)GetLastError());
+		    strm->name, (unsigned int)err);
 		return -1;
 	}
 
@@ -267,10 +376,12 @@ map_file_contents(struct file_stream *strm, u64 size)
 		CloseHandle((HANDLE)strm->mmap_token);
 		return -1;
 	}
-#else
+#else /* _WIN32 */
 	strm->mmap_mem = mmap(NULL, size, PROT_READ, MAP_SHARED, strm->fd, 0);
 	if (strm->mmap_mem == MAP_FAILED) {
 		strm->mmap_mem = NULL;
+		if (errno == ENODEV) /* mmap isn't supported on this file */
+			return read_full_contents(strm);
 		if (errno == ENOMEM) {
 			msg("%"TS" is too large to be processed by this "
 			    "program", strm->name);
@@ -280,7 +391,13 @@ map_file_contents(struct file_stream *strm, u64 size)
 		}
 		return -1;
 	}
+
+#ifdef HAVE_POSIX_MADVISE
+	posix_madvise(strm->mmap_mem, size, POSIX_MADV_SEQUENTIAL);
 #endif
+	strm->mmap_token = strm; /* anything that's not NULL */
+
+#endif /* !_WIN32 */
 	strm->mmap_size = size;
 	return 0;
 }
@@ -300,6 +417,8 @@ xread(struct file_stream *strm, void *buf, size_t count)
 		if (res == 0)
 			break;
 		if (res < 0) {
+			if (errno == EAGAIN || errno == EINTR)
+				continue;
 			msg_errno("Error reading from %"TS, strm->name);
 			return -1;
 		}
@@ -309,39 +428,6 @@ xread(struct file_stream *strm, void *buf, size_t count)
 	return orig_count - count;
 }
 
-/* Skip over 'count' bytes from a file, returning 0 on success or -1 on error */
-int
-skip_bytes(struct file_stream *strm, size_t count)
-{
-	size_t bufsize;
-	char *buffer;
-	ssize_t ret;
-
-	if (count == 0)
-		return 0;
-
-	bufsize = MIN(count, 4096);
-	buffer = xmalloc(bufsize);
-	if (buffer == NULL)
-		return -1;
-	do {
-		size_t n = MIN(count, bufsize);
-		ret = xread(strm, buffer, n);
-		if (ret < 0)
-			goto out;
-		if (ret != n) {
-			msg("%"TS": unexpected end-of-file", strm->name);
-			ret = -1;
-			goto out;
-		}
-		count -= ret;
-	} while (count != 0);
-	ret = 0;
-out:
-	free(buffer);
-	return ret;
-}
-
 /* Write to a file, returning 0 if all bytes were written or -1 on error */
 int
 full_write(struct file_stream *strm, const void *buf, size_t count)
@@ -365,23 +451,27 @@ int
 xclose(struct file_stream *strm)
 {
 	int ret = 0;
-	if (strm->fd >= 0 && !strm->is_standard_stream) {
+
+	if (!strm->is_standard_stream) {
 		if (close(strm->fd) != 0) {
 			msg_errno("Error closing %"TS, strm->name);
 			ret = -1;
 		}
 		free(strm->name);
-
-		if (strm->mmap_mem != NULL) {
-#ifdef _WIN32
-			UnmapViewOfFile(strm->mmap_mem);
-			CloseHandle((HANDLE)strm->mmap_token);
-#else
-			munmap(strm->mmap_mem, strm->mmap_size);
-#endif
-			strm->mmap_mem = NULL;
-		}
 	}
+
+	if (strm->mmap_token != NULL) {
+#ifdef _WIN32
+		UnmapViewOfFile(strm->mmap_mem);
+		CloseHandle((HANDLE)strm->mmap_token);
+#else
+		munmap(strm->mmap_mem, strm->mmap_size);
+#endif
+		strm->mmap_token = NULL;
+	} else {
+		free(strm->mmap_mem);
+	}
+	strm->mmap_mem = NULL;
 	strm->fd = -1;
 	strm->name = NULL;
 	return ret;
@@ -413,12 +503,12 @@ parse_compression_level(tchar opt_char, const tchar *arg)
 }
 
 /* Allocate a new DEFLATE compressor */
-struct deflate_compressor *
+struct libdeflate_compressor *
 alloc_compressor(int level)
 {
-	struct deflate_compressor *c;
+	struct libdeflate_compressor *c;
 
-	c = deflate_alloc_compressor(level);
+	c = libdeflate_alloc_compressor(level);
 	if (c == NULL) {
 		msg_errno("Unable to allocate compressor with "
 			  "compression level %d", level);
@@ -427,12 +517,12 @@ alloc_compressor(int level)
 }
 
 /* Allocate a new DEFLATE decompressor */
-struct deflate_decompressor *
+struct libdeflate_decompressor *
 alloc_decompressor(void)
 {
-	struct deflate_decompressor *d;
+	struct libdeflate_decompressor *d;
 
-	d = deflate_alloc_decompressor();
+	d = libdeflate_alloc_decompressor();
 	if (d == NULL)
 		msg_errno("Unable to allocate decompressor");
 

programs/prog_util.h

@@ -74,21 +74,18 @@ extern int wmain(int argc, wchar_t **argv);
 #  define	tstrtoul	wcstoul
 #  define	tstrxcmp	wcsicmp
 #  define	tunlink		_wunlink
-#  define	tutimbuf	_utimbuf
-#  define	tutime		_wutime
+#  define	tutimbuf	__utimbuf64
+#  define	tutime		_wutime64
+#  define	tstat		_wstat64
+#  define	tfstat		_fstat64
+#  define	stat_t		struct _stat64
 #  ifdef _MSC_VER
-     /* Standard file descriptors?  What are those? */
 #    define	STDIN_FILENO	0
 #    define	STDOUT_FILENO	1
 #    define	STDERR_FILENO	2
-
-     /* Fix the broken stat-related definitions. */
-#    define	fstat		_fstat
-#    define	stat		_stat
 #    define	S_ISREG(m)      (((m) & S_IFMT) == S_IFREG)
 #    define	S_ISDIR(m)      (((m) & S_IFMT) == S_IFDIR)
 #  endif
-#  define	O_NOFOLLOW	0
 
 #else /* _WIN32 */
 
@@ -111,7 +108,9 @@ extern int wmain(int argc, wchar_t **argv);
 #  define	tunlink		unlink
 #  define	tutimbuf	utimbuf
 #  define	tutime		utime
-#  define	O_BINARY	0
+#  define	tstat		stat
+#  define	tfstat		fstat
+#  define	stat_t		struct stat
 
 #endif /* !_WIN32 */
 
@@ -122,7 +121,9 @@ extern void _printf(1, 2) msg_errno(const char *fmt, ...);
 
 extern void *xmalloc(size_t size);
 
-extern u64 current_time(void);
+extern u64 timer_ticks(void);
+extern u64 timer_ticks_to_ms(u64 ticks);
+extern u64 timer_MB_per_s(u64 bytes, u64 ticks);
 
 extern const tchar *get_filename(const tchar *path);
 
@@ -135,21 +136,21 @@ struct file_stream {
 	size_t mmap_size;
 };
 
-extern int xopen_for_read(const tchar *path, struct file_stream *strm);
+extern int xopen_for_read(const tchar *path, bool symlink_ok,
+			  struct file_stream *strm);
 extern int xopen_for_write(const tchar *path, bool force,
 			   struct file_stream *strm);
 extern int map_file_contents(struct file_stream *strm, u64 size);
 
 extern ssize_t xread(struct file_stream *strm, void *buf, size_t count);
-extern int skip_bytes(struct file_stream *strm, size_t count);
 extern int full_write(struct file_stream *strm, const void *buf, size_t count);
 
 extern int xclose(struct file_stream *strm);
 
 extern int parse_compression_level(tchar opt_char, const tchar *arg);
 
-extern struct deflate_compressor *alloc_compressor(int level);
-extern struct deflate_decompressor *alloc_decompressor(void);
+extern struct libdeflate_compressor *alloc_compressor(int level);
+extern struct libdeflate_decompressor *alloc_decompressor(void);
 
 /* tgetopt.c */
 

programs/test_checksums.c

@@ -0,0 +1,182 @@
+/*
+ * test_checksums.c
+ *
+ * Verify that libdeflate's Adler-32 and CRC-32 functions produce the same
+ * results as their zlib equivalents.
+ */
+
+#include <stdlib.h>
+#include <time.h>
+#include <zlib.h>
+
+#include "prog_util.h"
+
+static unsigned int rng_seed;
+
+static void
+assertion_failed(const char *file, int line)
+{
+	fprintf(stderr, "Assertion failed at %s:%d\n", file, line);
+	fprintf(stderr, "RNG seed was %u\n", rng_seed);
+	abort();
+}
+
+#define ASSERT(expr) if (!(expr)) assertion_failed(__FILE__, __LINE__);
+
+typedef u32 (*cksum_fn_t)(u32, const void *, size_t);
+
+static u32
+zlib_adler32(u32 adler, const void *buf, size_t len)
+{
+	return adler32(adler, buf, len);
+}
+
+static u32
+zlib_crc32(u32 crc, const void *buf, size_t len)
+{
+	return crc32(crc, buf, len);
+}
+
+static u32
+select_initial_crc(void)
+{
+	if (rand() & 1)
+		return 0;
+	return ((u32)rand() << 16) | rand();
+}
+
+static u32
+select_initial_adler(void)
+{
+	u32 lo, hi;
+
+	if (rand() & 1)
+		return 1;
+
+	lo = (rand() % 4 == 0 ? 65520 : rand() % 65521);
+	hi = (rand() % 4 == 0 ? 65520 : rand() % 65521);
+	return (hi << 16) | lo;
+}
+
+static void
+test_initial_values(cksum_fn_t cksum, u32 expected)
+{
+	ASSERT(cksum(0, NULL, 0) == expected);
+	if (cksum != zlib_adler32) /* broken */
+		ASSERT(cksum(0, NULL, 1) == expected);
+	ASSERT(cksum(0, NULL, 1234) == expected);
+	ASSERT(cksum(1234, NULL, 0) == expected);
+	ASSERT(cksum(1234, NULL, 1234) == expected);
+}
+
+static void
+test_multipart(const u8 *buffer, size_t size, const char *name,
+	       cksum_fn_t cksum, u32 v, u32 expected)
+{
+	size_t division = rand() % (size + 1);
+	v = cksum(v, buffer, division);
+	v = cksum(v, buffer + division, size - division);
+	if (v != expected) {
+		fprintf(stderr, "%s checksum failed multipart test\n", name);
+		ASSERT(0);
+	}
+}
+
+static void
+test_checksums(const void *buffer, size_t size, const char *name,
+	       cksum_fn_t cksum1, cksum_fn_t cksum2, u32 initial_value)
+{
+	u32 v1 = cksum1(initial_value, buffer, size);
+	u32 v2 = cksum2(initial_value, buffer, size);
+
+	if (v1 != v2) {
+		fprintf(stderr, "%s checksum mismatch\n", name);
+		fprintf(stderr, "initial_value=0x%08"PRIx32", buffer=%p, "
+			"size=%zu, buffer=", initial_value, buffer, size);
+		for (size_t i = 0; i < MIN(size, 256); i++)
+			fprintf(stderr, "%02x", ((const u8 *)buffer)[i]);
+		if (size > 256)
+			fprintf(stderr, "...");
+		fprintf(stderr, "\n");
+		ASSERT(0);
+	}
+
+	if ((rand() & 15) == 0) {
+		test_multipart(buffer, size, name, cksum1, initial_value, v1);
+		test_multipart(buffer, size, name, cksum2, initial_value, v1);
+	}
+}
+
+static void
+test_crc32(const void *buffer, size_t size, u32 initial_value)
+{
+	test_checksums(buffer, size, "CRC-32",
+		       libdeflate_crc32, zlib_crc32, initial_value);
+}
+
+static void
+test_adler32(const void *buffer, size_t size, u32 initial_value)
+{
+	test_checksums(buffer, size, "Adler-32",
+		       libdeflate_adler32, zlib_adler32, initial_value);
+}
+
+static void test_random_buffers(u8 *buffer, size_t limit, u32 num_iter)
+{
+	for (u32 i = 0; i < num_iter; i++) {
+		size_t start = rand() % limit;
+		size_t len = rand() % (limit - start);
+
+		for (size_t j = start; j < start + len; j++)
+			buffer[j] = rand();
+
+		test_adler32(&buffer[start], len, select_initial_adler());
+		test_crc32(&buffer[start], len, select_initial_crc());
+	}
+}
+
+int
+tmain(int argc, tchar *argv[])
+{
+	u8 *buffer = malloc(32768);
+
+	rng_seed = time(NULL);
+	srand(rng_seed);
+
+	test_initial_values(libdeflate_adler32, 1);
+	test_initial_values(zlib_adler32, 1);
+	test_initial_values(libdeflate_crc32, 0);
+	test_initial_values(zlib_crc32, 0);
+
+	/* Test different buffer sizes and alignments */
+	test_random_buffers(buffer, 256, 5000);
+	test_random_buffers(buffer, 1024, 500);
+	test_random_buffers(buffer, 32768, 50);
+
+	/*
+	 * Test Adler-32 overflow cases.  For example, given all 0xFF bytes and
+	 * the highest possible initial (s1, s2) of (65520, 65520), then s2 if
+	 * stored as a 32-bit unsigned integer will overflow if > 5552 bytes are
+	 * processed.  Implementations must make sure to reduce s2 modulo 65521
+	 * before that point.  Also, some implementations make use of 16-bit
+	 * counters which can overflow earlier.
+	 */
+	memset(buffer, 0xFF, 32768);
+	for (u32 i = 0; i < 20; i++) {
+		u32 initial_value;
+
+		if (i == 0)
+			initial_value = ((u32)65520 << 16) | 65520;
+		else
+			initial_value = select_initial_adler();
+
+		test_adler32(buffer, 5553, initial_value);
+		test_adler32(buffer, rand() % 32769, initial_value);
+		buffer[rand() % 32768] = 0xFE;
+	}
+
+	printf("Adler-32 and CRC-32 checksum tests passed!\n");
+
+	free(buffer);
+	return 0;
+}

tools/afl-fuzz/deflate_compress/fuzz.c

@@ -7,8 +7,8 @@
 
 int main(int argc, char **argv)
 {
-	struct deflate_decompressor *d;
-	struct deflate_compressor *c;
+	struct libdeflate_decompressor *d;
+	struct libdeflate_compressor *c;
 	int ret;
 	int fd = open(argv[1], O_RDONLY);
 	struct stat stbuf;
@@ -20,21 +20,21 @@ int main(int argc, char **argv)
 	ret = read(fd, in, sizeof in);
 	assert(ret == sizeof in);
 
-	c = deflate_alloc_compressor(6);
-	d = deflate_alloc_decompressor();
+	c = libdeflate_alloc_compressor(6);
+	d = libdeflate_alloc_decompressor();
 
 	char out[sizeof(in)];
 	char checkarray[sizeof(in)];
 
-	size_t csize = deflate_compress(c, in,sizeof in, out, sizeof out);
+	size_t csize = libdeflate_deflate_compress(c, in,sizeof in, out, sizeof out);
 	if (csize) {
-		enum decompress_result res;
-		res = deflate_decompress(d, out, csize, checkarray, sizeof in, NULL);
+		enum libdeflate_result res;
+		res = libdeflate_deflate_decompress(d, out, csize, checkarray, sizeof in, NULL);
 		assert(!res);
 		assert(!memcmp(in, checkarray, sizeof in));
 	}
 
-	deflate_free_compressor(c);
-	deflate_free_decompressor(d);
+	libdeflate_free_compressor(c);
+	libdeflate_free_decompressor(d);
 	return 0;
 }

tools/afl-fuzz/deflate_decompress/fuzz.c

@@ -6,7 +6,7 @@
 
 int main(int argc, char **argv)
 {
-	struct deflate_decompressor *d;
+	struct libdeflate_decompressor *d;
 	int ret;
 	int fd = open(argv[1], O_RDONLY);
 	struct stat stbuf;
@@ -20,9 +20,9 @@ int main(int argc, char **argv)
 
 	char out[sizeof(in) * 3];
 
-	d = deflate_alloc_decompressor();
+	d = libdeflate_alloc_decompressor();
 
-	deflate_decompress(d, in, sizeof in, out, sizeof out, NULL);
-	deflate_free_decompressor(d);
+	libdeflate_deflate_decompress(d, in, sizeof in, out, sizeof out, NULL);
+	libdeflate_free_decompressor(d);
 	return 0;
 }

tools/afl-fuzz/gzip_decompress/fuzz.c

@@ -6,7 +6,7 @@
 
 int main(int argc, char **argv)
 {
-	struct deflate_decompressor *d;
+	struct libdeflate_decompressor *d;
 	int ret;
 	int fd = open(argv[1], O_RDONLY);
 	struct stat stbuf;
@@ -20,9 +20,9 @@ int main(int argc, char **argv)
 
 	char out[sizeof(in) * 3];
 
-	d = deflate_alloc_decompressor();
+	d = libdeflate_alloc_decompressor();
 
-	gzip_decompress(d, in, sizeof in, out, sizeof out, NULL);
-	deflate_free_decompressor(d);
+	libdeflate_gzip_decompress(d, in, sizeof in, out, sizeof out, NULL);
+	libdeflate_free_decompressor(d);
 	return 0;
 }

tools/afl-fuzz/zlib_decompress/fuzz.c

@@ -6,7 +6,7 @@
 
 int main(int argc, char **argv)
 {
-	struct deflate_decompressor *d;
+	struct libdeflate_decompressor *d;
 	int ret;
 	int fd = open(argv[1], O_RDONLY);
 	struct stat stbuf;
@@ -20,9 +20,9 @@ int main(int argc, char **argv)
 
 	char out[sizeof(in) * 3];
 
-	d = deflate_alloc_decompressor();
+	d = libdeflate_alloc_decompressor();
 
-	zlib_decompress(d, in, sizeof in, out, sizeof out, NULL);
-	deflate_free_decompressor(d);
+	libdeflate_zlib_decompress(d, in, sizeof in, out, sizeof out, NULL);
+	libdeflate_free_decompressor(d);
 	return 0;
 }

tools/android_build.sh

@@ -0,0 +1,115 @@
+#!/bin/bash
+
+set -eu
+
+ARCH="arm32"
+COMPILER="gcc"
+NDKDIR="/opt/android-ndk"
+ENABLE_NEON=false
+ENABLE_CRYPTO=false
+
+usage() {
+	cat << EOF
+Usage: $0 [OPTION]... -- [BENCHMARK_PROGRAM_ARG]...
+Build the libdeflate test programs for Android
+
+  --arch=ARCH          Architecture: arm32|arm64 (default: $ARCH)
+  --compiler=COMPILER  Compiler: gcc|clang (default: $COMPILER)
+  --ndkdir=NDKDIR      Android NDK directory (default: $NDKDIR)
+  --enable-neon        Enable NEON instructions
+  --enable-crypto      Enable crypto extensions (implies NEON too)
+EOF
+}
+
+if ! options=$(getopt -o '' \
+	-l 'arch:,compiler:,ndkdir:,enable-neon,enable-crypto,help' -- "$@"); then
+	usage
+	exit 1
+fi
+
+eval set -- "$options"
+
+while [ $# -gt 0 ]; do
+	case "$1" in
+	--arch)
+		ARCH="$2"
+		shift
+		;;
+	--compiler)
+		COMPILER="$2"
+		shift
+		;;
+	--ndkdir)
+		NDKDIR="$2"
+		shift
+		;;
+	--enable-neon)
+		ENABLE_NEON=true
+		;;
+	--enable-crypto)
+		ENABLE_CRYPTO=true
+		;;
+	--help)
+		usage
+		exit 0
+		;;
+	--)
+		shift
+		break
+		;;
+	*)
+		echo 1>&2 "Unknown option \"$1\""
+		usage
+		exit 1
+	esac
+	shift
+done
+
+CFLAGS="-fPIC"
+
+case "$ARCH" in
+arm|arm32|aarch32)
+	GCC_TOOLCHAIN="arm-linux-androideabi-4.9"
+	CLANG_TARGET="armv7-none-linux-androideabi"
+	if $ENABLE_CRYPTO; then
+		CFLAGS+=" -march=armv7-a -mfloat-abi=softfp -mfpu=crypto-neon-fp-armv8"
+	elif $ENABLE_NEON; then
+		CFLAGS+=" -march=armv7-a -mfloat-abi=softfp -mfpu=neon"
+	else
+		CFLAGS+=" -march=armv6"
+	fi
+	CFLAGS+=" --sysroot=\"$NDKDIR/platforms/android-12/arch-arm\""
+	;;
+arm64|aarch64)
+	GCC_TOOLCHAIN="aarch64-linux-android-4.9"
+	CLANG_TARGET="aarch64-none-linux-android"
+	if $ENABLE_CRYPTO; then
+		CFLAGS+=" -march=armv8-a+crypto"
+	else
+		CFLAGS+=" -march=armv8-a"
+	fi
+	CFLAGS+=" --sysroot=\"$NDKDIR/platforms/android-21/arch-arm64\""
+	;;
+*)
+	echo 1>&2 "Unknown architecture: \"$ARCH\""
+	usage
+	exit 1
+esac
+
+case "$COMPILER" in
+gcc)
+	CC="\"$NDKDIR/toolchains/$GCC_TOOLCHAIN/prebuilt/linux-x86_64/bin/${GCC_TOOLCHAIN%-*}-gcc\""
+	;;
+clang)
+	CC="\"$NDKDIR/toolchains/llvm/prebuilt/linux-x86_64/bin/clang\""
+	CFLAGS+=" -target \"$CLANG_TARGET\""
+	CFLAGS+=" -gcc-toolchain \"$NDKDIR/toolchains/$GCC_TOOLCHAIN/prebuilt/linux-x86_64\""
+	;;
+*)
+	echo 1>&2 "Unknown compiler: \"$COMPILER\""
+	usage
+	exit 1
+esac
+
+make -j$(grep -c processor /proc/cpuinfo) test_programs \
+	CC="$CC" CFLAGS="$CFLAGS" LDFLAGS="-pie"

tools/arm_test.sh

@@ -1,13 +0,0 @@
-#!/bin/sh
-
-set -e
-
-NDKDIR=/opt/android-ndk
-
-make -j benchmark \
-	CC="$NDKDIR/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-gcc" \
-	CFLAGS="--sysroot=$NDKDIR/platforms/android-12/arch-arm -march=armv7-a -fPIC -pie -mfpu=neon -mfloat-abi=softfp"
-
-adb push benchmark /data/local/tmp
-adb push $HOME/data/testdata  /data/local/tmp
-adb shell /data/local/tmp/benchmark "$@" /data/local/tmp/testdata

tools/checksum_benchmarks.sh

@@ -0,0 +1,153 @@
+#!/bin/bash
+
+set -eu
+
+if [ $# -eq 0 ]; then
+	FILE="$HOME/data/silesia"
+	[ ! -e "$FILE" ] && FILE="$HOME/silesia"
+	[ ! -e "$FILE" ] && FILE="$HOME/data/testdata"
+	[ ! -e "$FILE" ] && FILE="$HOME/testdata"
+	echo "Using default FILE: $FILE"
+	echo
+elif [ $# -eq 1 ]; then
+	FILE="$1"
+else
+	echo "Usage: $0 [FILE]" 1>&2
+	exit 1
+fi
+
+ARCH="$(uname -m)"
+
+if [ -n "$(git status -s | egrep 'adler32_impl.h|crc32_impl.h')" ]; then
+	echo "This script will overwrite adler32_impl.h and crc32_impl.h," \
+		"which have uncommitted changes.  Refusing to run." 1>&2
+	exit 1
+fi
+
+have_cpu_feature() {
+	local feature="$1"
+	local tag
+	case $ARCH in
+	arm*|aarch*)
+		tag="Features"
+		;;
+	*)
+		tag="flags"
+		;;
+	esac
+	grep -q "^$tag"$'[ \t]'"*:.*\<$feature\>" /proc/cpuinfo
+}
+
+make_and_test() {
+	make "$@" checksum test_checksums > /dev/null
+	./test_checksums > /dev/null
+}
+
+__do_benchmark() {
+	local impl="$1" speed
+	shift
+	local flags="$CKSUM_FLAGS $*"
+
+	speed=$(./checksum $flags -t "$FILE" | \
+		grep -o '[0-9]\+ MB/s' | grep -o '[0-9]\+')
+	printf "%-45s%-10s\n" "$CKSUM_NAME ($impl)" "$speed"
+}
+
+do_benchmark() {
+	local impl="$1"
+
+	if [ "$impl" = zlib ]; then
+		__do_benchmark "$impl" "-Z"
+	else
+		make_and_test CFLAGS="$EXTRA_CFLAGS"
+		__do_benchmark "libdeflate, $impl"
+		if [ $ARCH = x86_64 ]; then
+			make_and_test CFLAGS="-m32 $EXTRA_CFLAGS"
+			__do_benchmark "libdeflate, $impl, 32-bit"
+		fi
+	fi
+}
+
+sort_by_speed() {
+	awk '{print $NF, $0}' | sort -nr | cut -f2- -d' '
+}
+
+disable_impl() {
+	local name="$1"
+	local extra_cflags="$2"
+
+	sed -i '/^\#ifdef DISPATCH_'"$name"'$/aif (0)' lib/*/{adler,crc}32_impl.h
+	EXTRA_CFLAGS+=" $extra_cflags"
+}
+
+restore_impls() {
+	git checkout -f lib/*/{adler,crc}32_impl.h
+}
+
+trap restore_impls EXIT
+
+cat << EOF
+Method                                       Speed (MB/s)
+------                                       ------------
+EOF
+
+# CRC-32
+CKSUM_NAME="CRC-32"
+CKSUM_FLAGS=""
+EXTRA_CFLAGS=""
+{
+case $ARCH in
+i386|x86_64)
+	if have_cpu_feature pclmulqdq && have_cpu_feature avx; then
+		do_benchmark "PCLMUL/AVX"
+		disable_impl "PCLMUL_AVX" "-mno-avx"
+	fi
+	if have_cpu_feature pclmulqdq; then
+		do_benchmark "PCLMUL"
+		disable_impl "PCLMUL" "-mno-pclmul"
+	fi
+	;;
+arm*|aarch*)
+	if have_cpu_feature pmull; then
+		do_benchmark "PMULL"
+		disable_impl "PMULL" ""
+	fi
+	;;
+esac
+do_benchmark "generic"
+do_benchmark "zlib"
+} | sort_by_speed
+
+# Adler-32
+CKSUM_NAME="Adler-32"
+CKSUM_FLAGS="-A"
+EXTRA_CFLAGS=""
+echo
+{
+case $ARCH in
+i386|x86_64)
+	if have_cpu_feature avx2; then
+		do_benchmark "AVX2"
+		disable_impl "AVX2" "-mno-avx2"
+	fi
+	if have_cpu_feature sse2; then
+		do_benchmark "SSE2"
+		disable_impl "SSE2" "-mno-sse2"
+	fi
+	;;
+arm*)
+	if have_cpu_feature neon; then
+		do_benchmark "NEON"
+		disable_impl "NEON" "-mfpu=vfpv3"
+	fi
+	;;
+aarch*)
+	if have_cpu_feature asimd; then
+		do_benchmark "NEON"
+		disable_impl "NEON" "-march=armv8-a+nosimd"
+	fi
+	;;
+esac
+do_benchmark "generic"
+do_benchmark "zlib"
+} | sort_by_speed

tools/exec_tests.sh

@@ -0,0 +1,30 @@
+#
+# Helper script used by run_tests.sh, not intended to be run directly
+#
+
+set -eu
+
+run_cmd() {
+	echo "$WRAPPER $@"
+	$WRAPPER "$@" > /dev/null
+}
+
+run_cmd ./test_checksums
+
+for format in '' '-g' '-z'; do
+	for ref_impl in '' '-Y' '-Z'; do
+		run_cmd ./benchmark $format $ref_impl $SMOKEDATA
+	done
+done
+for level in 1 3 7 9; do
+	for ref_impl in '' '-Y'; do
+		run_cmd ./benchmark -$level $ref_impl $SMOKEDATA
+	done
+done
+for level in 1 3 7 9 12; do
+	for ref_impl in '' '-Z'; do
+		run_cmd ./benchmark -$level $ref_impl $SMOKEDATA
+	done
+done
+
+echo "exec_tests finished successfully" # Needed for 'adb shell'

tools/gen_crc32_multipliers.c

@@ -0,0 +1,108 @@
+/*
+ * gen_crc32_multipliers.c
+ *
+ * Copyright 2016 Eric Biggers
+ *
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include <inttypes.h>
+#include <stdio.h>
+
+/* generator polynomial G(x) */
+#define CRCPOLY		0xEDB88320 /* G(x) without x^32 term */
+#define CRCPOLY_FULL	(((uint64_t)CRCPOLY << 1) | 1) /* G(x) */
+
+/* Compute x^D mod G(x) */
+static uint32_t
+compute_multiplier(int D)
+{
+	/* Start with x^0 mod G(x) */
+	uint32_t remainder = 0x80000000;
+
+	/* Each iteration, 'remainder' becomes x^i mod G(x) */
+	for (int i = 1; i <= D; i++)
+		remainder = (remainder >> 1) ^ ((remainder & 1) ? CRCPOLY : 0);
+
+	/* Now 'remainder' is x^D mod G(x) */
+	return remainder;
+}
+
+/* Compute floor(x^64 / G(x)) */
+static uint64_t
+compute_barrett_reduction_constant(void)
+{
+	uint64_t quotient = 0;
+	uint64_t dividend = 0x1;
+
+	for (int i = 0; i < 64 - 32 + 1; i++) {
+		if ((dividend >> i) & 1) {
+			quotient |= (uint64_t)1 << i;
+			dividend ^= CRCPOLY_FULL << i;
+		}
+	}
+
+	return quotient;
+}
+
+/*
+ * This program computes the constant multipliers needed for carryless
+ * multiplication accelerated CRC-32.  It assumes 128-bit vectors divided into
+ * two 64-bit halves which are multiplied separately with different 32-bit
+ * multipliers, producing two 95-bit products.  For a given number of 128-bit
+ * vectors per iteration, the program outputs a pair of multipliers, one for
+ * each 64-bit half.
+ *
+ * Careful: all polynomials are "bit-reversed", meaning that the low-order bits
+ * have the highest degree and the high-order bits have the lowest degree!
+ */
+int
+main(void)
+{
+	printf("\t/* Constants precomputed by gen_crc32_multipliers.c.  "
+	       "Do not edit! */\n");
+
+	/* High and low multipliers for each needed vector count */
+	for (int order = 2; order >= 0; order--) {
+		int vecs_per_iteration = 1 << order;
+		int right = (128 * vecs_per_iteration) + 95;
+		printf("\tconst __v2di multipliers_%d = (__v2di)"
+		       "{ 0x%08"PRIX32", 0x%08"PRIX32" };\n",
+		       vecs_per_iteration,
+		       compute_multiplier(right - 64) /* higher degree half */,
+		       compute_multiplier(right - 128) /* lower degree half */);
+	}
+
+	/* Multiplier for final 96 => 64 bit fold */
+	printf("\tconst __v2di final_multiplier = (__v2di){ 0x%08"PRIX32" };\n",
+	       compute_multiplier(63));
+
+	/* 32-bit mask */
+	printf("\tconst __m128i mask32 = (__m128i)(__v4si){ 0xFFFFFFFF };\n");
+
+	/* Constants for final 64 => 32 bit reduction */
+	printf("\tconst __v2di barrett_reduction_constants =\n"
+	       "\t\t\t(__v2di){ 0x%016"PRIX64", 0x%016"PRIX64" };\n",
+	       compute_barrett_reduction_constant(), CRCPOLY_FULL);
+
+	return 0;
+}

tools/gen_crc32_table.c

@@ -1,15 +1,30 @@
 /*
  * gen_crc32_table.c - a program for CRC-32 table generation
  *
- * Written in 2014-2015 by Eric Biggers <ebiggers3@gmail.com>
+ * Originally public domain; changes after 2016-09-07 are copyrighted.
  *
- * To the extent possible under law, the author(s) have dedicated all copyright
- * and related and neighboring rights to this software to the public domain
- * worldwide. This software is distributed without any warranty.
+ * Copyright 2016 Eric Biggers
  *
- * You should have received a copy of the CC0 Public Domain Dedication along
- * with this software. If not, see
- * <http://creativecommons.org/publicdomain/zero/1.0/>.
+ * Permission is hereby granted, free of charge, to any person
+ * obtaining a copy of this software and associated documentation
+ * files (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following
+ * conditions:
+ *
+ * The above copyright notice and this permission notice shall be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
  */
 
 #include <stdint.h>

tools/gzip_tests.sh

@@ -0,0 +1,429 @@
+#!/bin/bash
+#
+# Test script for libdeflate's gzip and gunzip programs.
+#
+# To run, you must set GZIP and GUNZIP in the environment to the absolute paths
+# to the gzip and gunzip programs to test.  All tests should pass regardless of
+# whether the GNU versions or the libdeflate versions, or a combination, of
+# these programs are used.
+#
+# The environmental variable SMOKEDATA must also be set to a file containing
+# test data.
+#
+
+set -eu -o pipefail
+
+export -n GZIP GUNZIP SMOKEDATA
+
+TMPDIR="$(mktemp -d)"
+CURRENT_TEST=
+
+cleanup() {
+	if [ -n "$CURRENT_TEST" ]; then
+		echo "TEST FAILED: \"$CURRENT_TEST\""
+	fi
+	rm -rf -- "$TMPDIR"
+}
+
+trap cleanup EXIT
+
+SMOKEDATA="$(realpath "$SMOKEDATA")"
+cd "$TMPDIR"
+
+begin_test() {
+	CURRENT_TEST="$1"
+	rm -rf -- "$TMPDIR"/*
+	cp "$SMOKEDATA" file
+}
+
+gzip() {
+	$GZIP "$@"
+}
+
+gunzip() {
+	$GUNZIP "$@"
+}
+
+assert_status() {
+	local expected_status="$1"
+	local expected_msg="$2"
+	shift 2
+	(
+		set +e
+		eval "$*" 2>&1 >/dev/null
+		local actual_status=$?
+		if [ $actual_status -ne $expected_status ]; then
+			echo 1>&2 "Command '$*' exited with status" \
+				"$actual_status but expected status" \
+				"$expected_status"
+			exit 1
+		fi
+		exit 0
+	) > command_output
+	if ! egrep -q "$expected_msg" command_output; then
+		echo 1>&2 "Expected output of command '$*' to match regex" \
+			"'$expected_msg'"
+		echo 1>&2 "Actual output was:"
+		echo 1>&2 "---------------------------------------------------"
+		cat 1>&2 command_output
+		echo 1>&2 "---------------------------------------------------"
+		return 1
+	fi
+}
+
+assert_error() {
+	assert_status 1 "$@"
+}
+
+assert_warning() {
+	assert_status 2 "$@"
+}
+
+assert_skipped() {
+	assert_warning '\<(ignored|skipping|unchanged)\>' "$@"
+}
+
+
+begin_test 'Basic compression and decompression works'
+cp file orig
+gzip file
+[ ! -e file -a -e file.gz ]
+gunzip file.gz
+[ -e file -a ! -e file.gz ]
+cmp file orig
+
+
+begin_test 'gzip -d is gunzip'
+cp file orig
+gzip file
+gzip -d file.gz
+cmp file orig
+
+
+begin_test '-k (keep original file) works'
+cp file orig
+gzip -k file
+cmp file orig
+rm file
+cp file.gz orig.gz
+gunzip -k file.gz
+cmp file.gz orig.gz
+
+
+begin_test '-c (write to stdout) works'
+cp file orig
+gzip -k file
+gzip -c file > 2.gz
+cmp file orig
+cmp file.gz 2.gz
+gunzip -c 2.gz > file
+cmp file.gz 2.gz
+cmp file orig
+
+
+begin_test 'Reading from stdin works'
+gzip < file > 1.gz
+gzip - < file > 2.gz
+cat file | gzip > 3.gz
+cat file | gzip - > 4.gz
+cmp file <(gunzip < 1.gz)
+cmp file <(gunzip - < 2.gz)
+cmp file <(cat 3.gz | gunzip)
+cmp file <(cat 4.gz | gunzip -)
+
+
+begin_test '-n option is accepted'
+gzip -n file
+gunzip -n file.gz
+
+
+begin_test 'can specify multiple options'
+gzip -fk1 file
+cmp <(gzip -c -1 file) file.gz
+gunzip -kfd file.gz
+
+
+begin_test 'Compression levels'
+if [ "$GZIP" = /bin/gzip ]; then
+	assert_error '\<invalid option\>' gzip -10
+	max_level=9
+else
+	for level in 13 99999 1a; do
+		assert_error '\<Invalid compression level\>' gzip -$level
+	done
+	max_level=12
+fi
+for level in `seq 1 $max_level`; do
+	gzip -c -$level file > file$level
+	cmp file <(gunzip -c file$level)
+done
+rm file command_output
+cmp <(ls -S) <(ls -v) # file,file{1..max_level} have decreasing size
+
+
+begin_test 'Overwriting output file requires -f'
+cp file orig
+echo -n > file.gz
+gzip -c file > 2.gz
+assert_warning 'already exists' gzip file </dev/null
+cmp file.gz /dev/null
+gzip -f file
+cmp 2.gz file.gz
+echo -n > file
+assert_warning 'already exists' gunzip file.gz </dev/null
+gunzip -f file.gz
+cmp file orig
+
+
+begin_test 'Nonexistent input file fails, even with -f'
+for prog in 'gzip' 'gzip -f' 'gunzip' 'gunzip -f'; do
+	assert_error 'No such file or directory' $prog NONEXISTENT
+done
+
+
+begin_test 'Compressing already-suffixed file requires -f or -c'
+gzip file
+gzip -c file.gz > c.gz
+gzip file.gz 2>&1 >/dev/null | grep -q 'already has .gz suffix'
+[ -e file.gz -a ! -e file.gz.gz ]
+gzip -f file.gz
+[ ! -e file.gz -a -e file.gz.gz ]
+cmp file.gz.gz c.gz
+
+
+begin_test 'Decompressing unsuffixed file only works with -c'
+gzip file && mv file.gz file
+assert_skipped gunzip file
+assert_skipped gunzip -f file
+gunzip -c file > orig
+mv file file.gz && gunzip file.gz && cmp file orig
+
+
+begin_test '... unless there is a corresponding suffixed file'
+cp file orig
+gzip file
+[ ! -e file -a -e file.gz ]
+gunzip -c file > tmp
+cmp tmp orig
+rm tmp
+ln -s NONEXISTENT file
+gunzip -c file > tmp
+cmp tmp orig
+rm tmp file
+gunzip file
+[ -e file -a ! -e file.gz ]
+cmp file orig
+
+
+begin_test 'Directory is skipped, even with -f'
+mkdir dir
+mkdir dir.gz
+for opt in '' '-f' '-c'; do
+	assert_skipped gzip $opt dir
+done
+#assert_skipped gzip dir.gz  # XXX: GNU gzip warns, libdeflate gzip no-ops
+for opt in '' '-f' '-c'; do
+	for name in dir dir.gz; do
+		assert_skipped gunzip $opt $name
+	done
+done
+
+
+begin_test '(gzip) symlink is rejected without -f or -c'
+ln -s file symlink1
+ln -s file symlink2
+assert_error 'Too many levels of symbolic links' gzip symlink1
+[ -e file -a -e symlink1 -a ! -e symlink1.gz ]
+gzip -f symlink1
+[ -e file -a ! -e symlink1 -a -e symlink1.gz ]
+gzip -c symlink2 > /dev/null
+
+
+begin_test '(gunzip) symlink is rejected without -f or -c'
+gzip file
+ln -s file.gz symlink1.gz
+ln -s file.gz symlink2.gz
+assert_error 'Too many levels of symbolic links' gunzip symlink1
+[ -e file.gz -a -e symlink1.gz -a ! -e symlink1 ]
+gunzip -f symlink1.gz
+[ -e file.gz -a ! -e symlink1.gz -a -e symlink1 ]
+gunzip -c symlink2.gz > /dev/null
+
+
+begin_test 'FIFO is skipped, even with -f'
+mkfifo foo
+mkfifo foo.gz
+assert_skipped gzip foo
+assert_skipped gzip -f foo
+#assert_skipped gzip -c foo # XXX: works with GNU gzip, not libdeflate's
+assert_skipped gunzip foo.gz
+assert_skipped gunzip -f foo.gz
+#assert_skipped gunzip -c foo.gz # XXX: works with GNU gzip, not libdeflate's
+
+
+begin_test '(gzip) overwriting symlink does not follow symlink'
+echo 1 > 1
+echo 2 > 2
+gzip 1
+ln -s 1.gz 2.gz
+gzip -f 2
+gunzip 1.gz
+cmp <(echo 1) 1
+
+
+begin_test '(gunzip) overwriting symlink does not follow symlink'
+echo 1 > 1
+echo 2 > 2
+gzip 2
+ln -s 1 2
+gunzip -f 2.gz
+cmp <(echo 1) 1
+cmp <(echo 2) 2
+
+
+begin_test '(gzip) hard linked file skipped without -f or -c'
+cp file orig
+ln file link
+[ $(stat -c %h file) -eq 2 ]
+assert_skipped gzip file
+gzip -c file > /dev/null
+[ $(stat -c %h file) -eq 2 ]
+gzip -f file
+[ $(stat -c %h link) -eq 1 ]
+[ $(stat -c %h file.gz) -eq 1 ]
+cmp link orig
+# XXX: GNU gzip skips hard linked files with -k, libdeflate's doesn't
+
+
+begin_test '(gunzip) hard linked file skipped without -f or -c'
+gzip file
+ln file.gz link.gz
+cp file.gz orig.gz
+[ $(stat -c %h file.gz) -eq 2 ]
+assert_skipped gunzip file.gz
+gunzip -c file.gz > /dev/null
+[ $(stat -c %h file.gz) -eq 2 ]
+gunzip -f file
+[ $(stat -c %h link.gz) -eq 1 ]
+[ $(stat -c %h file) -eq 1 ]
+cmp link.gz orig.gz
+
+
+begin_test 'Multiple files'
+cp file file2
+gzip file file2
+[ ! -e file -a ! -e file2 -a -e file.gz -a -e file2.gz ]
+gunzip file.gz file2.gz
+[ -e file -a -e file2 -a ! -e file.gz -a ! -e file2.gz ]
+
+
+begin_test 'Multiple files, continue on warning'
+mkdir 1
+cp file 2
+assert_skipped gzip 1 2
+[ ! -e 1.gz ]
+cmp file <(gunzip -c 2.gz)
+rmdir 1
+mkdir 1.gz
+assert_skipped gunzip 1.gz 2.gz
+[ ! -e 1 ]
+cmp 2 file
+
+
+begin_test 'Multiple files, continue on error'
+cp file 1
+cp file 2
+chmod -r 1
+assert_error 'Permission denied' gzip 1 2
+[ ! -e 1.gz ]
+cmp file <(gunzip -c 2.gz)
+rm -f 1
+cp 2.gz 1.gz
+chmod -r 1.gz
+assert_error 'Permission denied' gunzip 1.gz 2.gz
+[ ! -e 1 ]
+cmp 2 file
+
+
+begin_test 'Compressing empty file'
+echo -n > empty
+gzip empty
+gunzip empty.gz
+cmp /dev/null empty
+
+
+begin_test 'Decompressing malformed file'
+echo -n > foo.gz
+assert_error '\<(not in gzip format|unexpected end of file)\>' \
+	gunzip foo.gz
+echo 1 > foo.gz
+assert_error '\<not in gzip format\>' gunzip foo.gz
+echo abcdefgh > foo.gz
+assert_error '\<not in gzip format\>' gunzip foo.gz
+xxd -r > foo.gz <<-EOF
+00000000: 1f8b 0800 0000 0000 00ff 4b4c 4a4e 4924  ..........KLJNI$
+00000010: 1673 0100 6c5b a262 2e00 0000            .s..l[.b....
+EOF
+assert_error '\<(not in gzip format|crc error)\>' gunzip foo.gz
+
+
+for suf in .foo foo .blaaaaaaaaaaaaaaaargh; do
+	begin_test "Custom suffix: $suf"
+	gzip -S $suf file
+	[ ! -e file -a ! -e file.gz -a -e file$suf ]
+	assert_skipped gunzip file$suf
+	gunzip -S $suf file$suf
+	[ -e file -a ! -e file.gz -a ! -e file$suf ]
+done
+# DIFFERENCE: GNU gzip lower cases suffix, we don't
+
+
+begin_test 'Empty suffix is rejected'
+assert_error '\<invalid suffix\>' gzip -S '""' file
+assert_error '\<invalid suffix\>' gunzip -S '""' file
+
+
+begin_test 'Timestamps and mode are preserved'
+chmod 777 file
+orig_stat="$(stat -c '%a;%x;%y' file)"
+gzip file
+sleep 1
+gunzip file.gz
+[ "$(stat -c '%a;%x;%y' file)" = "$orig_stat" ]
+
+
+begin_test 'Decompressing multi-member gzip file'
+cat file file > orig
+gzip -c file > file.gz
+gzip -c file >> file.gz
+gunzip -f file.gz
+cmp file orig
+
+
+begin_test 'Decompressing multi-member gzip file (final member smaller)'
+echo 'hello world' > 2
+cat file 2 > orig
+gzip -c file > file.gz
+gzip -c 2 >> file.gz
+gunzip -f file.gz
+cmp file orig
+
+
+begin_test 'Help option'
+gzip -h 2>&1 | grep -q 'Usage'
+gunzip -h 2>&1 | grep -q 'Usage'
+
+
+begin_test 'Incorrect usage'
+for prog in gzip gunzip; do
+	for opt in '--invalid-option' '-0'; do
+		assert_error '\<(unrecognized|invalid) option\>' $prog $opt
+	done
+done
+
+
+begin_test 'Version information'
+gzip -V | grep -q Copyright
+gunzip -V | grep -q Copyright
+
+CURRENT_TEST=

tools/make-windows-releases

@@ -4,13 +4,14 @@ set -eu
 
 for arch in 'i686' 'x86_64'; do
 	make clean
-	make -j CC=${arch}-w64-mingw32-gcc CFLAGS="-Werror" all benchmark.exe
+	make -j CC=${arch}-w64-mingw32-gcc CFLAGS="-Werror" all \
+		benchmark.exe checksum.exe
 	dir=libdeflate-$(git describe --tags | tr -d v)-windows-${arch}-bin
 	rm -rf $dir ${dir}.zip
 	mkdir $dir
 	cp libdeflate.dll libdeflate.lib libdeflate.h *.exe $dir
 	${arch}-w64-mingw32-strip ${dir}/libdeflate.dll ${dir}/*.exe
-	for file in COPYING COPYING.LIB NEWS; do
+	for file in COPYING NEWS; do
 		sed < $file > ${dir}/${file}.txt -e 's/$/\r/g'
 	done
 	for file in README.md; do

tools/mips_build.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+set -eu
+
+TOOLCHAIN_DIR=$HOME/src/ddwrt-toolchains/toolchain-mips_34kc_gcc-5.1.0_musl-1.1.9
+
+make -j$(grep -c processor /proc/cpuinfo) all test_programs \
+	CC="$TOOLCHAIN_DIR/bin/mips-openwrt-linux-musl-gcc" \
+	CFLAGS="-DNEED_PRINTF -Werror"

tools/mips_test.sh

@@ -1,12 +0,0 @@
-#!/bin/sh
-
-set -e
-
-TOOLCHAIN_DIR=$HOME/src/ddwrt-toolchains/toolchain-mips_34kc_gcc-5.1.0_musl-1.1.9
-
-make -j benchmark \
-	CC="$TOOLCHAIN_DIR/bin/mips-openwrt-linux-musl-gcc" \
-	CFLAGS="-DNEED_PRINTF"
-
-scp benchmark $HOME/data/test root@dd-wrt:
-ssh root@dd-wrt ./benchmark "$@" test

tools/pgo_build.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+
+# Try gcc profile-guided optimizations
+
+set -eu
+
+MAKE="make -j$(grep -c processor /proc/cpuinfo)"
+DATAFILE="$HOME/data/silesia"
+
+$MAKE benchmark > /dev/null
+echo "====================="
+echo "Original performance:"
+echo "---------------------"
+./benchmark "$@" "$DATAFILE"
+
+$MAKE CFLAGS=-fprofile-generate LDFLAGS=-fprofile-generate benchmark > /dev/null
+./benchmark "$@" "$DATAFILE" > /dev/null
+$MAKE CFLAGS=-fprofile-use benchmark > /dev/null
+rm -f {lib,programs}/*.gcda
+echo "=========================="
+echo "PGO-optimized performance:"
+echo "--------------------------"
+./benchmark "$@" "$DATAFILE"

tools/run_tests.sh

@@ -0,0 +1,366 @@
+#!/bin/bash
+#
+# Test script for libdeflate
+#
+#	Usage: ./tools/run_tests.sh [TESTGROUP]... [-TESTGROUP]...
+#
+# By default all tests are run, but it is possible to explicitly include or
+# exclude specific test groups.
+#
+
+set -eu -o pipefail
+cd "$(dirname "$0")/.."
+
+TESTGROUPS=(all)
+
+set_test_groups() {
+	TESTGROUPS=("$@")
+	local have_exclusion=0
+	local have_all=0
+	for group in "${TESTGROUPS[@]}"; do
+		if [[ $group == -* ]]; then
+			have_exclusion=1
+		elif [[ $group == all ]]; then
+			have_all=1
+		fi
+	done
+	if (( have_exclusion && !have_all )); then
+		TESTGROUPS=(all "${TESTGROUPS[@]}")
+	fi
+}
+
+if [ $# -gt 0 ]; then
+	set_test_groups "$@"
+fi
+
+SMOKEDATA="${SMOKEDATA:=$HOME/data/smokedata}"
+if [ ! -e "$SMOKEDATA" ]; then
+	echo "SMOKEDATA (value: $SMOKEDATA) does not exist.  Set the" \
+	      "environmental variable SMOKEDATA to a file to use in" \
+	      "compression/decompression tests." 1>&2
+	exit 1
+fi
+
+NDKDIR="${NDKDIR:=/opt/android-ndk}"
+
+FILES=("$SMOKEDATA" ./tools/exec_tests.sh benchmark test_checksums)
+EXEC_TESTS_CMD="WRAPPER= SMOKEDATA=\"$(basename $SMOKEDATA)\" sh exec_tests.sh"
+NPROC=$(grep -c processor /proc/cpuinfo)
+VALGRIND="valgrind --quiet --error-exitcode=100 --leak-check=full --errors-for-leak-kinds=all"
+SANITIZE_CFLAGS="-fsanitize=undefined -fno-sanitize-recover=undefined,integer"
+
+TMPFILE="$(mktemp)"
+trap "rm -f \"$TMPFILE\"" EXIT
+
+###############################################################################
+
+rm -f run_tests.log
+exec >  >(tee -ia run_tests.log)
+exec 2> >(tee -ia run_tests.log >&2)
+
+TESTS_SKIPPED=0
+log_skip() {
+	log "[WARNING, TEST SKIPPED]: $@"
+	TESTS_SKIPPED=1
+}
+
+log() {
+	echo "[$(date)] $@"
+}
+
+run_cmd() {
+	log "$@"
+	"$@" > /dev/null
+}
+
+test_group_included() {
+	local included=0 group
+	for group in "${TESTGROUPS[@]}"; do
+		if [ "$group" = "$1" ]; then
+			included=1 # explicitly included
+			break
+		fi
+		if [ "$group" = "-$1" ]; then
+			included=0 # explicitly excluded
+			break
+		fi
+		if [ "$group" = "all" ]; then # implicitly included
+			included=1
+		fi
+	done
+	if (( included )); then
+		log "Starting test group: $1"
+	fi
+	(( included ))
+}
+
+have_valgrind() {
+	if ! type -P valgrind > /dev/null; then
+		log_skip "valgrind not found; can't run tests with valgrind"
+		return 1
+	fi
+}
+
+have_ubsan() {
+	if ! type -P clang > /dev/null; then
+		log_skip "clang not found; can't run tests with UBSAN"
+		return 1
+	fi
+}
+
+have_python() {
+	if ! type -P python3 > /dev/null; then
+		log_skip "Python not found"
+		return 1
+	fi
+}
+
+###############################################################################
+
+native_build_and_test() {
+	make "$@" -j$NPROC all test_programs > /dev/null
+	WRAPPER="$WRAPPER" SMOKEDATA="$SMOKEDATA" sh ./tools/exec_tests.sh \
+			> /dev/null
+}
+
+native_tests() {
+	test_group_included native || return 0
+	local compiler compilers_to_try=(gcc)
+	local cflags cflags_to_try=("")
+	shopt -s nullglob
+	compilers_to_try+=(/usr/bin/gcc-[0-9]*)
+	compilers_to_try+=(/usr/bin/clang-[0-9]*)
+	compilers_to_try+=(/opt/gcc*/bin/gcc)
+	compilers_to_try+=(/opt/clang*/bin/clang)
+	shopt -u nullglob
+
+	if [ "$(uname -m)" = "x86_64" ]; then
+		cflags_to_try+=("-march=native")
+		cflags_to_try+=("-m32")
+	fi
+	for compiler in ${compilers_to_try[@]}; do
+		for cflags in "${cflags_to_try[@]}"; do
+			if [ "$cflags" = "-m32" ] && \
+			   $compiler -v |& grep -q -- '--disable-multilib'
+			then
+				continue
+			fi
+			log "Running tests with CC=$compiler," \
+				"CFLAGS=$cflags"
+			WRAPPER= native_build_and_test \
+				CC=$compiler CFLAGS="$cflags -Werror"
+		done
+	done
+
+	if have_valgrind; then
+		log "Running tests with Valgrind"
+		WRAPPER="$VALGRIND" native_build_and_test
+	fi
+
+	if have_ubsan; then
+		log "Running tests with undefined behavior sanitizer"
+		WRAPPER= native_build_and_test CC=clang CFLAGS="$SANITIZE_CFLAGS"
+	fi
+}
+
+###############################################################################
+
+checksum_benchmarks() {
+	test_group_included checksum_benchmarks || return 0
+	./tools/checksum_benchmarks.sh
+}
+
+###############################################################################
+
+android_build_and_test() {
+	run_cmd ./tools/android_build.sh --ndkdir="$NDKDIR" "$@"
+	run_cmd adb push "${FILES[@]}" /data/local/tmp/
+
+	# Note: adb shell always returns 0, even if the shell command fails...
+	log "adb shell \"cd /data/local/tmp && $EXEC_TESTS_CMD\""
+	adb shell "cd /data/local/tmp && $EXEC_TESTS_CMD" > "$TMPFILE"
+	if ! grep -q "exec_tests finished successfully" "$TMPFILE"; then
+		log "Android test failure!  adb shell output:"
+		cat "$TMPFILE"
+		return 1
+	fi
+}
+
+android_tests() {
+	local compiler
+
+	test_group_included android || return 0
+	if [ ! -e $NDKDIR ]; then
+		log_skip "Android NDK was not found in NDKDIR=$NDKDIR!" \
+		         "If you want to run the Android tests, set the" \
+			 "environmental variable NDKDIR to the location of" \
+			 "your Android NDK installation"
+		return 0
+	fi
+
+	if ! type -P adb > /dev/null; then
+		log_skip "adb (android-tools) is not installed"
+		return 0
+	fi
+
+	if ! adb devices | grep -q 'device$'; then
+		log_skip "No Android device is currently attached"
+		return 0
+	fi
+
+	for compiler in gcc clang; do
+		for flags in "" "--enable-neon" "--enable-crypto"; do
+			for arch in arm32 arm64; do
+				android_build_and_test --arch=$arch \
+					--compiler=$compiler $flags
+			done
+		done
+	done
+}
+
+###############################################################################
+
+mips_tests() {
+	test_group_included mips || return 0
+	if [ "$(hostname)" != "zzz" ] || [ "$(uname -m)" != "x86_64" ]; then
+		log_skip "MIPS tests are not supported on this host"
+		return 0
+	fi
+	if ! ping -c 1 dd-wrt > /dev/null; then
+		log_skip "Can't run MIPS tests: dd-wrt system not available"
+		return 0
+	fi
+	run_cmd ./tools/mips_build.sh
+	run_cmd scp "${FILES[@]}" root@dd-wrt:
+	run_cmd ssh root@dd-wrt "$EXEC_TESTS_CMD"
+
+	log "Checking that compression on big endian CPU produces same output"
+	run_cmd scp gzip root@dd-wrt:
+	run_cmd ssh root@dd-wrt \
+		"rm -f big*.gz;
+		 ./gzip -c -6 $(basename $SMOKEDATA) > big6.gz;
+		 ./gzip -c -10 $(basename $SMOKEDATA) > big10.gz"
+	run_cmd scp root@dd-wrt:big*.gz .
+	make -j$NPROC gzip > /dev/null
+	./gzip -c -6 "$SMOKEDATA" > little6.gz
+	./gzip -c -10 "$SMOKEDATA" > little10.gz
+	if ! cmp big6.gz little6.gz || ! cmp big10.gz little10.gz; then
+		echo 1>&2 "Compressed data differed on big endian vs. little endian!"
+		return 1
+	fi
+	rm big*.gz little*.gz
+}
+
+###############################################################################
+
+windows_tests() {
+	local arch
+
+	test_group_included windows || return 0
+
+	# Windows: currently compiled but not run
+	for arch in i686 x86_64; do
+		local compiler=${arch}-w64-mingw32-gcc
+		if ! type -P $compiler > /dev/null; then
+			log_skip "$compiler not found"
+			continue
+		fi
+		run_cmd make CC=$compiler CFLAGS=-Werror -j$NPROC \
+			all test_programs
+	done
+}
+
+###############################################################################
+
+static_analysis_tests() {
+	test_group_included static_analysis || return 0
+	if ! type -P scan-build > /dev/null; then
+		log_skip "clang static analyzer (scan-build) not found"
+		return 0
+	fi
+	run_cmd scan-build --status-bugs make -j$NPROC all test_programs
+}
+
+###############################################################################
+
+gzip_tests() {
+	test_group_included gzip || return 0
+
+	local gzip gunzip
+	run_cmd make -j$NPROC gzip gunzip
+	for gzip in "$PWD/gzip" /bin/gzip; do
+		for gunzip in "$PWD/gunzip" /bin/gunzip; do
+			log "Running gzip program tests with GZIP=$gzip," \
+				"GUNZIP=$gunzip"
+			GZIP="$gzip" GUNZIP="$gunzip" SMOKEDATA="$SMOKEDATA" \
+				./tools/gzip_tests.sh
+		done
+	done
+
+	if have_valgrind; then
+		log "Running gzip program tests with Valgrind"
+		GZIP="$VALGRIND $PWD/gzip" GUNZIP="$VALGRIND $PWD/gunzip" \
+			SMOKEDATA="$SMOKEDATA" ./tools/gzip_tests.sh
+	fi
+
+	if have_ubsan; then
+		log "Running gzip program tests with undefined behavior sanitizer"
+		run_cmd make -j$NPROC CC=clang CFLAGS="$SANITIZE_CFLAGS" gzip gunzip
+		GZIP="$PWD/gzip" GUNZIP="$PWD/gunzip" \
+			SMOKEDATA="$SMOKEDATA" ./tools/gzip_tests.sh
+	fi
+}
+
+###############################################################################
+
+edge_case_tests() {
+	test_group_included edge_case || return 0
+
+	# Regression test for "deflate_compress: fix corruption with long
+	# literal run".  Try to compress a file longer than 65535 bytes where no
+	# 2-byte sequence (3 would be sufficient) is repeated <= 32768 bytes
+	# apart, and the distribution of bytes remains constant throughout, and
+	# yet not all bytes are used so the data is still slightly compressible.
+	# There will be no matches in this data, but the compressor should still
+	# output a compressed block, and this block should contain more than
+	# 65535 consecutive literals, which triggered the bug.
+	#
+	# Note: on random data, this situation is extremely unlikely if the
+	# compressor uses all matches it finds, since random data will on
+	# average have a 3-byte match every (256**3)/32768 = 512 bytes.
+	if have_python; then
+		python3 > "$TMPFILE" << EOF
+import sys
+for i in range(2):
+    for stride in range(1,251):
+        b = bytes(stride*multiple % 251 for multiple in range(251))
+        sys.stdout.buffer.write(b)
+EOF
+		run_cmd make -j$NPROC benchmark
+		run_cmd ./benchmark -3 "$TMPFILE"
+		run_cmd ./benchmark -6 "$TMPFILE"
+		run_cmd ./benchmark -12 "$TMPFILE"
+	fi
+}
+
+###############################################################################
+
+log "Starting libdeflate tests"
+log "	TESTGROUPS=(${TESTGROUPS[@]})"
+log "	SMOKEDATA=$SMOKEDATA"
+log "	NDKDIR=$NDKDIR"
+
+native_tests
+checksum_benchmarks
+android_tests
+mips_tests
+windows_tests
+static_analysis_tests
+gzip_tests
+edge_case_tests
+
+if (( TESTS_SKIPPED )); then
+	log "No tests failed, but some tests were skipped.  See above."
+else
+	log "All tests passed!"
+fi

tools/windows_build.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+set -eu
+
+make -j CC=i686-w64-mingw32-gcc all test_programs
+cp -vf *.exe /j/exe/
+make -j CC=x86_64-w64-mingw32-gcc all test_programs
+cp -vf *.exe /j/exe64/
+
+sudo $HOME/bin/sudo/restart-smbd

tools/windows_test.sh

@@ -1,10 +0,0 @@
-#!/bin/sh
-
-set -e
-
-make -j CC=i686-w64-mingw32-gcc
-cp -vf *.exe /j/exe/
-make -j CC=x86_64-w64-mingw32-gcc
-cp -vf *.exe /j/exe64/
-
-sudo $HOME/bin/sudo/restart-smbd