fixed incorrect assert()

Incredibly evaded detection so far, probably because -size_t is still perfectly defined size_t which seems good enough for gcc and clang, but Visual finds that rather fishy.
Merge pull request #2244 from hctim/dev
2020-07-15 12:43:11 -07:00 · 2020-07-15 12:40:17 -07:00 · 2020-07-14 09:03:02 -07:00 · 2020-07-13 14:26:13 -04:00 · 2020-07-13 01:20:11 +00:00 · 2020-07-11 02:16:52 -07:00
46 changed files with 1147 additions and 334 deletions
@@ -0,0 +1,184 @@
+name: generic-dev
+
+on:
+  pull_request:
+    branches: [ dev, master, actionsTest ]
+
+jobs:
+
+# Dev PR jobs that still have to be migrated from travis
+#
+# icc (need self-hosted)
+# versionTag
+# valgrindTest (keeps failing for some reason. need investigation)
+# staticAnalyze (need trusty so need self-hosted)
+# pcc-fuzz: (need trusty so need self-hosted)
+# min-decomp-macros (flakey)
+#
+# setting up self-hosted is pretty straightforward, but
+# I need admins permissions to the repo for that it looks like
+# So I'm tabling that for now
+#
+# The master branch exclusive jobs will be in a separate
+# workflow file (the osx tests and meson build that is)
+
+  benchmarking:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: make benchmarking
+      run: make benchmarking
+
+  test:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: make test
+      run: make test
+
+  gcc-6-7-libzstd:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: gcc-6 + gcc-7 + libzstdmt compilation
+      run: |
+        make gcc6install gcc7install
+        CC=gcc-6 CFLAGS=-Werror make -j all
+        make clean
+        CC=gcc-7 CFLAGS=-Werror make -j all
+        make clean
+        LDFLAGS=-Wl,--no-undefined make -C lib libzstd-mt
+        make -C tests zbufftest-dll
+
+  gcc-8-asan-ubsan-testzstd:
+    runs-on: ubuntu-16.04 # fails on 18.04
+    steps:
+    - uses: actions/checkout@v2
+    - name: gcc-8 + ASan + UBSan + Test Zstd
+      run: |
+        make gcc8install
+        CC=gcc-8 CFLAGS="-Werror" make -j all
+        make clean
+        CC=gcc-8 make -j uasan-test-zstd </dev/null
+
+  gcc-6-asan-ubsan-testzstd-32bit:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: gcc-6 + ASan + UBSan + Test Zstd, 32bit mode
+      run: |
+        make gcc6install libc6install
+        CC=gcc-6 CFLAGS="-Werror -m32" make -j all32
+        make clean
+        CC=gcc-6 make -j uasan-test-zstd32
+
+  clang-38-msan-testzstd:
+    runs-on: ubuntu-16.04 # fails on 18.04
+    steps:
+    - uses: actions/checkout@v2
+    - name: clang-3.8 + MSan + Test Zstd
+      run: |
+        # make clang38install (doesn't work)
+        sudo apt-add-repository "deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty-3.8 main"
+        sudo apt-get update
+        sudo apt-get install clang-3.8
+        CC=clang-3.8 make clean msan-test-zstd HAVE_ZLIB=0 HAVE_LZ4=0 HAVE_LZMA=0
+
+  cmake-build-and-test-check:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: cmake build and test check
+      run: make cmakebuild
+
+  gcc-8-asan-ubsan-fuzz:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: gcc-8 + ASan + UBSan + Fuzz Test
+      run: |
+        make gcc8install
+        CC=gcc-8 make clean uasan-fuzztest
+
+  gcc-6-asan-ubsan-fuzz32:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: gcc-6 + ASan + UBSan + Fuzz Test 32bit
+      run: |
+        make gcc6install libc6install
+        CC=gcc-6 CFLAGS="-O2 -m32" make uasan-fuzztest
+
+  clang-38-msan-fuzz:
+    runs-on: ubuntu-16.04 # fails on 18.04
+    steps:
+    - uses: actions/checkout@v2
+    - name: clang-3.8 + MSan + Fuzz Test
+      run: |
+        # make clang38install (doesn't work)
+        sudo apt-add-repository "deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty-3.8 main"
+        sudo apt-get update
+        sudo apt-get install clang-3.8
+        CC=clang-3.8 make clean msan-fuzztest
+
+  asan-ubsan-msan-regression:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: ASan + UBSan + MSan + Regression Test
+      run: |
+        make -j uasanregressiontest
+        make clean
+        make -j msanregressiontest
+
+  cpp-gnu90-c99-compatibility:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: C++, gnu90 and c99 compatibility
+      run: |
+        make cxxtest
+        make clean
+        make gnu90build
+        make clean
+        make c99build
+        make clean
+        make travis-install
+
+  mingw-cross-compilation:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: mingw cross-compilation
+      run: |
+        # sudo update-alternatives --set x86_64-w64-mingw32-g++ /usr/bin/x86_64-w64-mingw32-g++-posix; (doesn't work)
+        sudo apt-get install gcc-mingw-w64
+        CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ CFLAGS="-Werror -O1" make zstd
+
+  armbuild:
+    runs-on: ubuntu-16.04 # doesn't work on latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: ARM Build Test
+      run: |
+        make arminstall
+        make armbuild
+
+  armfuzz:
+    runs-on: ubuntu-16.04 # doesn't work on latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: Qemu ARM emulation + Fuzz Test
+      run: |
+        make arminstall
+        make armfuzz
+
+  bourne-shell:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: Bourne shell compatibility (shellcheck)
+      run: |
+        wget https://github.com/koalaman/shellcheck/releases/download/v0.7.1/shellcheck-v0.7.1.linux.x86_64.tar.xz
+        tar -xf shellcheck-v0.7.1.linux.x86_64.tar.xz
+        shellcheck-v0.7.1/shellcheck --shell=sh --severity=warning --exclude=SC2010 tests/playTests.sh
@@ -0,0 +1,68 @@
+name: generic-release
+
+on:
+  pull_request:
+    # This will eventually only be for pushes to master
+    # but for dogfooding purposes, I'm running it even 
+    # on dev pushes
+    branches: [ dev, master, actionsTest ]
+
+jobs:
+  # missing jobs
+  # 
+  # ppc64le + fuzz test
+  # Qemu PPC64 + Fuzz test
+  # Qemu aarch64 + Fuzz Test (on Xenial)
+  # versions comp   
+  # meson test
+
+  osx:
+    runs-on: macos-10.15
+    steps:
+    - uses: actions/checkout@v2
+    - name: OS-X
+      run: |
+        make test
+        # make -c lib all (need to fix. not working right now)
+  
+  zbuff:
+    runs-on: ubuntu-16.04
+    steps:
+    - uses: actions/checkout@v2
+    - name: zbuff test
+      run: |
+        make -C tests test-zbuff
+        
+  tsan:
+    runs-on: ubuntu-16.04
+    steps:
+    - uses: actions/checkout@v2
+    - name: thread sanitizer
+      run: |
+        sudo apt-add-repository "deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty-3.8 main"
+        sudo apt-get update
+        sudo apt-get install clang-3.8
+        CC=clang-3.8 make tsan-test-zstream
+        CC=clang-3.8 make tsan-fuzztest
+  zlib-wrapper:
+    runs-on: ubuntu-16.04
+    steps:
+    - uses: actions/checkout@v2
+    - name: zlib wrapper test
+      run: |
+        make gpp6install valgrindinstall
+        make -C zlibWrapper test
+        make -C zlibWrapper valgrindTest
+  
+  lz4-threadpool-partial-libs:
+    runs-on: ubuntu-16.04
+    steps:
+    - uses: actions/checkout@v2
+    - name: LZ4, thread pool, and partial libs testslib wrapper test
+      run: |
+        make lz4install
+        make -C tests test-lz4
+        make clean
+        make -C tests test-pool
+        make clean
+        bash tests/libzstd_partial_builds.sh
@@ -3,21 +3,28 @@ on: [pull_request]
 jobs:
  Fuzzing:
    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        sanitizer: [address, undefined, memory]
    steps:
-    - name: Build Fuzzers
+    - name: Build Fuzzers (${{ matrix.sanitizer }})
+      id: build
      uses: google/oss-fuzz/infra/cifuzz/actions/build_fuzzers@master
      with:
        oss-fuzz-project-name: 'zstd'
        dry-run: false
-    - name: Run Fuzzers
+        sanitizer: ${{ matrix.sanitizer }}
+    - name: Run Fuzzers (${{ matrix.sanitizer }})
      uses: google/oss-fuzz/infra/cifuzz/actions/run_fuzzers@master
      with:
        oss-fuzz-project-name: 'zstd'
        fuzz-seconds: 600
        dry-run: false
+        sanitizer: ${{ matrix.sanitizer }}
    - name: Upload Crash
      uses: actions/upload-artifact@v1
-      if: failure()
+      if: failure() && steps.build.outcome == 'success'
      with:
-        name: artifacts
+        name: ${{ matrix.sanitizer }}-artifacts
        path: ./out/artifacts
@@ -22,6 +22,7 @@ zstdmt

 # Test artefacts
 tmp*
+*.zst
 dictionary.
 dictionary
 NUL
@@ -32,6 +33,7 @@ bin/
 .buckd/
 buck-out/
 build-*
+*.gcda

 # Other files
 .directory
@@ -126,6 +126,20 @@ just `contrib/largeNbDicts` and nothing else, you can run:
 scan-build make -C contrib/largeNbDicts largeNbDicts
 ```

+### Pitfalls of static analysis
+`scan-build` is part of our regular CI suite. Other static analyzers are not.
+
+It can be useful to look at additional static analyzers once in a while (and we do), but it's not a good idea to multiply the nb of analyzers run continuously at each commit and PR. The reasons are :
+
+- Static analyzers are full of false positive. The signal to noise ratio is actually pretty low.
+- A good CI policy is "zero-warning tolerance". That means that all issues must be solved, including false positives. This quickly becomes a tedious workload.
+- Multiple static analyzers will feature multiple kind of false positives, sometimes applying to the same code but in different ways leading to :
+   + torteous code, trying to please multiple constraints, hurting readability and therefore maintenance. Sometimes, such complexity introduce other more subtle bugs, that are just out of scope of the analyzers.
+   + sometimes, these constraints are mutually exclusive : if one try to solve one, the other static analyzer will complain, they can't be both happy at the same time.
+- As if that was not enough, the list of false positives change with each version. It's hard enough to follow one static analyzer, but multiple ones with their own update agenda, this quickly becomes a massive velocity reducer.
+
+This is different from running a static analyzer once in a while, looking at the output, and __cherry picking__ a few warnings that seem helpful, either because they detected a genuine risk of bug, or because it helps expressing the code in a way which is more readable or more difficult to misuse. These kind of reports can be useful, and are accepted.
+
 ## Performance
 Performance is extremely important for zstd and we only merge pull requests whose performance
 landscape and corresponding trade-offs have been adequately analyzed, reproduced, and presented.
@@ -8,6 +8,9 @@
 # You may select, at your option, one of the above-listed licenses.
 # ################################################################

+# verbose mode (print commands) on V=1 or VERBOSE=1
+Q = $(if $(filter 1,$(V) $(VERBOSE)),,@)
+
 PRGDIR   = programs
 ZSTDDIR  = lib
 BUILDIR  = build
@@ -46,8 +49,8 @@ allmost: allzstd zlibwrapper
 # skip zwrapper, can't build that on alternate architectures without the proper zlib installed
 .PHONY: allzstd
 allzstd: lib-all
-	$(MAKE) -C $(PRGDIR) all
-	$(MAKE) -C $(TESTDIR) all
+	$(Q)$(MAKE) -C $(PRGDIR) all
+	$(Q)$(MAKE) -C $(TESTDIR) all

 .PHONY: all32
 all32:
@@ -55,18 +58,19 @@ all32:
 	$(MAKE) -C $(TESTDIR) all32

 .PHONY: lib lib-release libzstd.a
+lib-all : lib
 lib lib-release lib-all :
-	@$(MAKE) -C $(ZSTDDIR) $@
+	$(Q)$(MAKE) -C $(ZSTDDIR) $@

 .PHONY: zstd zstd-release
 zstd zstd-release:
-	@$(MAKE) -C $(PRGDIR) $@
-	cp $(PRGDIR)/zstd$(EXT) .
+	$(Q)$(MAKE) -C $(PRGDIR) $@
+	$(Q)cp $(PRGDIR)/zstd$(EXT) .

 .PHONY: zstdmt
 zstdmt:
-	@$(MAKE) -C $(PRGDIR) $@
-	cp $(PRGDIR)/zstd$(EXT) ./zstdmt$(EXT)
+	$(Q)$(MAKE) -C $(PRGDIR) $@
+	$(Q)cp $(PRGDIR)/zstd$(EXT) ./zstdmt$(EXT)

 .PHONY: zlibwrapper
 zlibwrapper: lib
@@ -79,12 +83,12 @@ test: MOREFLAGS += -g -DDEBUGLEVEL=$(DEBUGLEVEL) -Werror
 test:
 	MOREFLAGS="$(MOREFLAGS)" $(MAKE) -j -C $(PRGDIR) allVariants
 	$(MAKE) -C $(TESTDIR) $@
-	ZSTD=../../programs/zstd $(MAKE) -C doc/educational_decoder test
+	ZSTD=../../programs/zstd $(MAKE) -C doc/educational_decoder $@

 ## shortest: same as `make check`
 .PHONY: shortest
 shortest:
-	$(MAKE) -C $(TESTDIR) $@
+	$(Q)$(MAKE) -C $(TESTDIR) $@

 ## check: run basic tests for `zstd` cli
 .PHONY: check
@@ -127,17 +131,17 @@ cleanTabs:

 .PHONY: clean
 clean:
-	@$(MAKE) -C $(ZSTDDIR) $@ > $(VOID)
-	@$(MAKE) -C $(PRGDIR) $@ > $(VOID)
-	@$(MAKE) -C $(TESTDIR) $@ > $(VOID)
-	@$(MAKE) -C $(ZWRAPDIR) $@ > $(VOID)
-	@$(MAKE) -C examples/ $@ > $(VOID)
-	@$(MAKE) -C contrib/gen_html $@ > $(VOID)
-	@$(MAKE) -C contrib/pzstd $@ > $(VOID)
-	@$(MAKE) -C contrib/seekable_format/examples $@ > $(VOID)
-	@$(MAKE) -C contrib/largeNbDicts $@ > $(VOID)
-	@$(RM) zstd$(EXT) zstdmt$(EXT) tmp*
-	@$(RM) -r lz4
+	$(Q)$(MAKE) -C $(ZSTDDIR) $@ > $(VOID)
+	$(Q)$(MAKE) -C $(PRGDIR) $@ > $(VOID)
+	$(Q)$(MAKE) -C $(TESTDIR) $@ > $(VOID)
+	$(Q)$(MAKE) -C $(ZWRAPDIR) $@ > $(VOID)
+	$(Q)$(MAKE) -C examples/ $@ > $(VOID)
+	$(Q)$(MAKE) -C contrib/gen_html $@ > $(VOID)
+	$(Q)$(MAKE) -C contrib/pzstd $@ > $(VOID)
+	$(Q)$(MAKE) -C contrib/seekable_format/examples $@ > $(VOID)
+	$(Q)$(MAKE) -C contrib/largeNbDicts $@ > $(VOID)
+	$(Q)$(RM) zstd$(EXT) zstdmt$(EXT) tmp*
+	$(Q)$(RM) -r lz4
 	@echo Cleaning completed

 #------------------------------------------------------------------------------
@@ -161,7 +165,7 @@ EGREP = egrep $(EGREP_OPTIONS)
 ## list: Print all targets and their descriptions (if provided)
 .PHONY: list
 list:
-	@TARGETS=$$($(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null \
+	$(Q)TARGETS=$$($(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null \
 		| awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' \
 		| $(EGREP) -v  -e '^[^[:alnum:]]' | sort); \
 	{ \
@@ -176,13 +180,13 @@ list:

 .PHONY: install armtest usan asan uasan
 install:
-	@$(MAKE) -C $(ZSTDDIR) $@
-	@$(MAKE) -C $(PRGDIR) $@
+	$(Q)$(MAKE) -C $(ZSTDDIR) $@
+	$(Q)$(MAKE) -C $(PRGDIR) $@

 .PHONY: uninstall
 uninstall:
-	@$(MAKE) -C $(ZSTDDIR) $@
-	@$(MAKE) -C $(PRGDIR) $@
+	$(Q)$(MAKE) -C $(ZSTDDIR) $@
+	$(Q)$(MAKE) -C $(PRGDIR) $@

 .PHONY: travis-install
 travis-install:
@@ -176,6 +176,12 @@ Going into `build` directory, you will find additional possibilities:
 You can build the zstd binary via buck by executing: `buck build programs:zstd` from the root of the repo.
 The output binary will be in `buck-out/gen/programs/`.

+## Testing
+
+You can run quick local smoke tests by executing the `playTest.sh` script from the `src/tests` directory.
+Two env variables `$ZSTD_BIN` and `$DATAGEN_BIN` are needed for the test script to locate the zstd and datagen binary.
+For information on CI testing, please refer to TESTING.md
+
 ## Status

 Zstandard is currently deployed within Facebook. It is used continuously to compress large amounts of data in multiple formats and use cases.
@@ -60,7 +60,6 @@ macro(ADD_ZSTD_COMPILATION_FLAGS)
             CMAKE_CXX_FLAGS_MINSIZEREL CMAKE_CXX_FLAGS_RELWITHDEBINFO)
        if( ${flag_var} )
            separate_arguments(${flag_var})
-            list(REMOVE_DUPLICATES ${flag_var})
            string(REPLACE ";" " " ${flag_var} "${${flag_var}}")
        endif()
    endforeach ()
@@ -134,7 +134,7 @@ if (ZSTD_BUILD_STATIC)
            OUTPUT_NAME ${STATIC_LIBRARY_BASE_NAME})
 endif ()

-if (UNIX)
+if (UNIX OR MINGW)
    # pkg-config
    set(PREFIX "${CMAKE_INSTALL_PREFIX}")
    set(LIBDIR "${CMAKE_INSTALL_LIBDIR}")
@@ -152,9 +152,7 @@ endif ()
 # install target
 install(FILES
    "${LIBRARY_DIR}/zstd.h"
-    "${LIBRARY_DIR}/deprecated/zbuff.h"
    "${LIBRARY_DIR}/dictBuilder/zdict.h"
-    "${LIBRARY_DIR}/dictBuilder/cover.h"
    "${LIBRARY_DIR}/common/zstd_errors.h"
    DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}")

@@ -163,6 +161,7 @@ install(TARGETS ${library_targets}
    INCLUDES DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
    ARCHIVE DESTINATION "${CMAKE_INSTALL_LIBDIR}"
    LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}"
+    RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}"
    )

 # uninstall target
@@ -127,6 +127,5 @@ pkgconfig.generate(libzstd,
  url: 'http://www.zstd.net/')

 install_headers(join_paths(zstd_rootdir, 'lib/zstd.h'),
-  join_paths(zstd_rootdir, 'lib/deprecated/zbuff.h'),
  join_paths(zstd_rootdir, 'lib/dictBuilder/zdict.h'),
  join_paths(zstd_rootdir, 'lib/common/zstd_errors.h'))
@@ -13,10 +13,10 @@

 namespace pzstd {

-constexpr int ERROR = 1;
-constexpr int INFO = 2;
-constexpr int DEBUG = 3;
-constexpr int VERBOSE = 4;
+constexpr int kLogError = 1;
+constexpr int kLogInfo = 2;
+constexpr int kLogDebug = 3;
+constexpr int kLogVerbose = 4;

 class Logger {
  std::mutex mutex_;
@@ -94,12 +94,12 @@ static std::uint64_t handleOneInput(const Options &options,
    if (!options.decompress) {
      double ratio = static_cast<double>(bytesWritten) /
                     static_cast<double>(bytesRead + !bytesRead);
-      state.log(INFO, "%-20s :%6.2f%%   (%6" PRIu64 " => %6" PRIu64
+      state.log(kLogInfo, "%-20s :%6.2f%%   (%6" PRIu64 " => %6" PRIu64
                   " bytes, %s)\n",
                   inputFileName.c_str(), ratio * 100, bytesRead, bytesWritten,
                   outputFileName.c_str());
    } else {
-      state.log(INFO, "%-20s: %" PRIu64 " bytes \n",
+      state.log(kLogInfo, "%-20s: %" PRIu64 " bytes \n",
                   inputFileName.c_str(),bytesWritten);
    }
  }
@@ -139,12 +139,12 @@ static FILE *openOutputFile(const Options &options,
    auto outputFd = std::fopen(outputFile.c_str(), "rb");
    if (outputFd != nullptr) {
      std::fclose(outputFd);
-      if (!state.log.logsAt(INFO)) {
+      if (!state.log.logsAt(kLogInfo)) {
        state.errorHolder.setError("Output file exists");
        return nullptr;
      }
      state.log(
-          INFO,
+          kLogInfo,
          "pzstd: %s already exists; do you wish to overwrite (y/n) ? ",
          outputFile.c_str());
      int c = getchar();
@@ -170,7 +170,7 @@ int pzstdMain(const Options &options) {
    auto printErrorGuard = makeScopeGuard([&] {
      if (state.errorHolder.hasError()) {
        returnCode = 1;
-        state.log(ERROR, "pzstd: %s: %s.\n", input.c_str(),
+        state.log(kLogError, "pzstd: %s: %s.\n", input.c_str(),
                  state.errorHolder.getError().c_str());
      }
    });
@@ -388,7 +388,7 @@ std::uint64_t asyncCompressChunks(
  // Break the input up into chunks of size `step` and compress each chunk
  // independently.
  size_t step = calculateStep(size, numThreads, params);
-  state.log(DEBUG, "Chosen frame size: %zu\n", step);
+  state.log(kLogDebug, "Chosen frame size: %zu\n", step);
  auto status = FileStatus::Continue;
  while (status == FileStatus::Continue && !state.errorHolder.hasError()) {
    // Make a new input queue that we will put the chunk's input data into.
@@ -403,7 +403,7 @@ std::uint64_t asyncCompressChunks(
    });
    // Pass the output queue to the writer thread.
    chunks.push(std::move(out));
-    state.log(VERBOSE, "%s\n", "Starting a new frame");
+    state.log(kLogVerbose, "%s\n", "Starting a new frame");
    // Fill the input queue for the compression job we just started
    status = readData(*in, ZSTD_CStreamInSize(), step, fd, &bytesRead);
  }
@@ -540,14 +540,14 @@ std::uint64_t asyncDecompressFrames(
    if (frameSize == 0) {
      // We hit a non SkippableFrame ==> not compressed by pzstd or corrupted
      // Pass the rest of the source to this decompression task
-      state.log(VERBOSE, "%s\n",
+      state.log(kLogVerbose, "%s\n",
          "Input not in pzstd format, falling back to serial decompression");
      while (status == FileStatus::Continue && !state.errorHolder.hasError()) {
        status = readData(*in, chunkSize, chunkSize, fd, &totalBytesRead);
      }
      break;
    }
-    state.log(VERBOSE, "Decompressing a frame of size %zu", frameSize);
+    state.log(kLogVerbose, "Decompressing a frame of size %zu", frameSize);
    // Fill the input queue for the decompression job we just started
    status = readData(*in, chunkSize, frameSize, fd, &totalBytesRead);
  }
@@ -573,7 +573,7 @@ std::uint64_t writeFile(
    bool decompress) {
  auto& errorHolder = state.errorHolder;
  auto lineClearGuard = makeScopeGuard([&state] {
-    state.log.clear(INFO);
+    state.log.clear(kLogInfo);
  });
  std::uint64_t bytesWritten = 0;
  std::shared_ptr<BufferWorkQueue> out;
@@ -602,7 +602,7 @@ std::uint64_t writeFile(
        return bytesWritten;
      }
      bytesWritten += buffer.size();
-      state.log.update(INFO, "Written: %u MB   ",
+      state.log.update(kLogInfo, "Written: %u MB   ",
                static_cast<std::uint32_t>(bytesWritten >> 20));
    }
  }
@@ -41,7 +41,7 @@ class SharedState {
      auto parameters = options.determineParameters();
      cStreamPool.reset(new ResourcePool<ZSTD_CStream>{
          [this, parameters]() -> ZSTD_CStream* {
-            this->log(VERBOSE, "%s\n", "Creating new ZSTD_CStream");
+            this->log(kLogVerbose, "%s\n", "Creating new ZSTD_CStream");
            auto zcs = ZSTD_createCStream();
            if (zcs) {
              auto err = ZSTD_initCStream_advanced(
@@ -59,7 +59,7 @@ class SharedState {
    } else {
      dStreamPool.reset(new ResourcePool<ZSTD_DStream>{
          [this]() -> ZSTD_DStream* {
-            this->log(VERBOSE, "%s\n", "Creating new ZSTD_DStream");
+            this->log(kLogVerbose, "%s\n", "Creating new ZSTD_DStream");
            auto zds = ZSTD_createDStream();
            if (zds) {
              auto err = ZSTD_initDStream(zds);
@@ -1,4 +1,9 @@
+
+# build artifacts
 zstddeclib.c
 zstdenclib.c
 zstd.c
 zstd.h
+
+# test artifacts
+temp*
@@ -6,6 +6,56 @@ OUT_FILE="tempbin"
 # Optional temporary compiled WebAssembly
 OUT_WASM="temp.wasm"

+# Source files to compile using Emscripten.
+IN_FILES="examples/emscripten.c"
+
+# Emscripten build using emcc.
+emscripten_emcc_build() {
+  # Compile the the same example as above
+  CC_FLAGS="-Wall -Wextra -Werror -Os -g0 -flto"
+  emcc $CC_FLAGS -s WASM=1 -I. -o $OUT_WASM $IN_FILES
+  # Did compilation work?
+  if [ $? -ne 0 ]; then
+    echo "Compiling ${IN_FILES}: FAILED"
+    exit 1
+  fi
+  echo "Compiling ${IN_FILES}: PASSED"
+  rm -f $OUT_WASM
+}
+
+# Emscripten build using docker.
+emscripten_docker_build() {
+  docker container run --rm \
+    --volume $PWD:/code \
+    --workdir /code \
+    emscripten/emsdk:latest \
+    emcc $CC_FLAGS -s WASM=1 -I. -o $OUT_WASM $IN_FILES
+  # Did compilation work?
+  if [ $? -ne 0 ]; then
+      echo "Compiling ${IN_FILES} (using docker): FAILED"
+    exit 1
+  fi
+  echo "Compiling ${IN_FILES} (using docker): PASSED"
+  rm -f $OUT_WASM
+}
+
+# Try Emscripten build using emcc or docker.
+try_emscripten_build() {
+  which emcc > /dev/null
+  if [ $? -eq 0 ]; then
+    emscripten_emcc_build
+    return $?
+  fi
+
+  which docker > /dev/null
+  if [ $? -eq 0 ]; then
+    emscripten_docker_build
+    return $?
+  fi
+
+  echo "(Skipping Emscripten test)"
+}
+
 # Amalgamate the sources
 ./create_single_file_decoder.sh
 # Did combining work?
@@ -35,21 +85,7 @@ if [ $retVal -ne 0 ]; then
 fi
 echo "Running simple.c: PASSED"

-# Is Emscripten available?
-which emcc > /dev/null
-if [ $? -ne 0 ]; then
-  echo "(Skipping Emscripten test)"
-else
-  # Compile the Emscripten example
-  CC_FLAGS="-Wall -Wextra -Werror -Os -g0 -flto --llvm-lto 3 -lGL -DNDEBUG=1"
-  emcc $CC_FLAGS -s WASM=1 -o $OUT_WASM examples/emscripten.c
-  # Did compilation work?
-  if [ $? -ne 0 ]; then
-    echo "Compiling emscripten.c: FAILED"
-    exit 1
-  fi
-  echo "Compiling emscripten.c: PASSED"
-  rm -f $OUT_WASM
-fi
+# Try Emscripten build if emcc or docker command is available.
+try_emscripten_build

 exit 0
@@ -9,6 +9,56 @@ OUT_FILE="tempbin"
 # Optional temporary compiled WebAssembly
 OUT_WASM="temp.wasm"

+# Source files to compile using Emscripten.
+IN_FILES="zstd.c examples/roundtrip.c"
+
+# Emscripten build using emcc.
+emscripten_emcc_build() {
+  # Compile the the same example as above
+  CC_FLAGS="-Wall -Wextra -Werror -Os -g0 -flto"
+  emcc $CC_FLAGS -s WASM=1 -I. -o $OUT_WASM $IN_FILES
+  # Did compilation work?
+  if [ $? -ne 0 ]; then
+    echo "Compiling ${IN_FILES}: FAILED"
+    exit 1
+  fi
+  echo "Compiling ${IN_FILES}: PASSED"
+  rm -f $OUT_WASM
+}
+
+# Emscripten build using docker.
+emscripten_docker_build() {
+  docker container run --rm \
+    --volume $PWD:/code \
+    --workdir /code \
+    emscripten/emsdk:latest \
+    emcc $CC_FLAGS -s WASM=1 -I. -o $OUT_WASM $IN_FILES
+  # Did compilation work?
+  if [ $? -ne 0 ]; then
+      echo "Compiling ${IN_FILES} (using docker): FAILED"
+    exit 1
+  fi
+  echo "Compiling ${IN_FILES} (using docker): PASSED"
+  rm -f $OUT_WASM
+}
+
+# Try Emscripten build using emcc or docker.
+try_emscripten_build() {
+  which emcc > /dev/null
+  if [ $? -eq 0 ]; then
+    emscripten_emcc_build
+    return $?
+  fi
+
+  which docker > /dev/null
+  if [ $? -eq 0 ]; then
+    emscripten_docker_build
+    return $?
+  fi
+
+  echo "(Skipping Emscripten test)"
+}
+
 # Amalgamate the sources
 ./create_single_file_library.sh
 # Did combining work?
@@ -41,21 +91,7 @@ if [ $retVal -ne 0 ]; then
 fi
 echo "Running roundtrip.c: PASSED"

-# Is Emscripten available?
-which emcc > /dev/null
-if [ $? -ne 0 ]; then
-  echo "(Skipping Emscripten test)"
-else
-  # Compile the the same example as above
-  CC_FLAGS="-Wall -Wextra -Werror -Os -g0 -flto"
-  emcc $CC_FLAGS -s WASM=1 -I. -o $OUT_WASM zstd.c examples/roundtrip.c
-  # Did compilation work?
-  if [ $? -ne 0 ]; then
-    echo "Compiling emscripten.c: FAILED"
-    exit 1
-  fi
-  echo "Compiling emscripten.c: PASSED"
-  rm -f $OUT_WASM
-fi
+# Try Emscripten build if emcc or docker command is available.
+try_emscripten_build

 exit 0
@@ -39,9 +39,6 @@
 #undef  XXH_INLINE_ALL
 #define XXH_INLINE_ALL
 #define ZSTD_LEGACY_SUPPORT 0
-#define ZSTD_LIB_DICTBUILDER 0
-#define ZSTD_LIB_DEPRECATED 0
-#define ZSTD_NOBENCH
 #ifndef __EMSCRIPTEN__
 #define ZSTD_MULTITHREAD
 #endif
@@ -74,3 +71,8 @@
 #include "decompress/zstd_ddict.c"
 #include "decompress/zstd_decompress.c"
 #include "decompress/zstd_decompress_block.c"
+
+#include "dictBuilder/cover.c"
+#include "dictBuilder/divsufsort.c"
+#include "dictBuilder/fastcover.c"
+#include "dictBuilder/zdict.c"
@@ -37,9 +37,6 @@
 #undef  XXH_INLINE_ALL
 #define XXH_INLINE_ALL
 #define ZSTD_LEGACY_SUPPORT 0
-#define ZSTD_LIB_COMPRESSION 0
-#define ZSTD_LIB_DEPRECATED 0
-#define ZSTD_NOBENCH
 #define ZSTD_STRIP_ERROR_STRINGS

 #include "common/debug.c"
@@ -3,7 +3,7 @@ Zstandard Compression Format

 ### Notices

-Copyright (c) 2016-present Yann Collet, Facebook, Inc.
+Copyright (c) 2016-2020 Yann Collet, Facebook, Inc.

 Permission is granted to copy and distribute this document
 for any purpose and without charge,
@@ -16,7 +16,7 @@ Distribution of this document is unlimited.

 ### Version

-0.3.5 (13/11/19)
+0.3.6 (25/05/20)


 Introduction
@@ -291,21 +291,10 @@ Format is __little-endian__.
 It's allowed to represent a small ID (for example `13`)
 with a large 4-bytes dictionary ID, even if it is less efficient.

-_Reserved ranges :_
-Within private environments, any `Dictionary_ID` can be used.
-
-However, for frames and dictionaries distributed in public space,
-`Dictionary_ID` must be attributed carefully.
-Rules for public environment are not yet decided,
-but the following ranges are reserved for some future registrar :
- low range  : `<= 32767`
- high range : `>= (1 << 31)`
-
-Outside of these ranges, any value of `Dictionary_ID`
-which is both `>= 32768` and `< (1<<31)` can be used freely,
-even in public environment.
-
-
+A value of `0` has same meaning as no `Dictionary_ID`,
+in which case the frame may or may not need a dictionary to be decoded,
+and the ID of such a dictionary is not specified.
+The decoder must know this information by other means.

 #### `Frame_Content_Size`

@@ -389,7 +378,7 @@ __`Block_Size`__
 The upper 21 bits of `Block_Header` represent the `Block_Size`.

 When `Block_Type` is `Compressed_Block` or `Raw_Block`,
-`Block_Size` is the size of `Block_Content` (hence excluding `Block_Header`).  
+`Block_Size` is the size of `Block_Content` (hence excluding `Block_Header`).

 When `Block_Type` is `RLE_Block`, since `Block_Content`’s size is always 1,
 `Block_Size` represents the number of times this byte must be repeated.
@@ -1429,13 +1418,17 @@ __`Dictionary_ID`__ : 4 bytes, stored in __little-endian__ format.
              It's used by decoders to check if they use the correct dictionary.

 _Reserved ranges :_
-              If the frame is going to be distributed in a private environment,
-              any `Dictionary_ID` can be used.
-              However, for public distribution of compressed frames,
-              the following ranges are reserved and shall not be used :
+If the dictionary is going to be distributed in a public environment,
+the following ranges of `Dictionary_ID` are reserved for some future registrar
+and shall not be used :
+
+    - low range  : <= 32767
+    - high range : >= (2^31)
+
+Outside of these ranges, any value of `Dictionary_ID`
+which is both `>= 32768` and `< (1<<31)` can be used freely,
+even in public environment.

-              - low range  : <= 32767
-              - high range : >= (2^31)

 __`Entropy_Tables`__ : follow the same format as tables in [compressed blocks].
              See the relevant [FSE](#fse-table-description)
@@ -1455,7 +1448,7 @@ __`Content`__ : The rest of the dictionary is its content.
              As long as the amount of data decoded from this frame is less than or
              equal to `Window_Size`, sequence commands may specify offsets longer
              than the total length of decoded output so far to reference back to the
-              dictionary, even parts of the dictionary with offsets larger than `Window_Size`.  
+              dictionary, even parts of the dictionary with offsets larger than `Window_Size`.
              After the total output has surpassed `Window_Size` however,
              this is no longer allowed and the dictionary is no longer accessible.

@@ -1673,6 +1666,7 @@ or at least provide a meaningful error code explaining for which reason it canno

 Version changes
 ---------------
+- 0.3.6 : clarifications for Dictionary_ID
 - 0.3.5 : clarifications for Block_Maximum_Size
 - 0.3.4 : clarifications for FSE decoding table
 - 0.3.3 : clarifications for field Block_Size
@@ -39,6 +39,7 @@ static void compressFile_orDie(const char* fname, const char* outName, int cLeve
     */
    CHECK_ZSTD( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, cLevel) );
    CHECK_ZSTD( ZSTD_CCtx_setParameter(cctx, ZSTD_c_checksumFlag, 1) );
+    ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, 4);

    /* This loop read from the input file, compresses that entire chunk,
     * and writes all output produced to the output file.
@@ -49,6 +49,7 @@ ifeq ($HAVE_COLORNEVER, 1)
 GREP_OPTIONS += --color=never
 endif
 GREP = grep $(GREP_OPTIONS)
+SED_ERE_OPT ?= -E

 ZSTDCOMMON_FILES := $(sort $(wildcard common/*.c))
 ZSTDCOMP_FILES := $(sort $(wildcard compress/*.c))
@@ -220,13 +221,14 @@ libzstd : $(LIBZSTD)
 .PHONY: lib
 lib : libzstd.a libzstd

-.PHONY: lib-mt
+# note : do not define lib-mt or lib-release as .PHONY
+# make does not consider implicit pattern rule for .PHONY target
+
 %-mt : CPPFLAGS += -DZSTD_MULTITHREAD
 %-mt : LDFLAGS  += -pthread
 %-mt : %
 	@echo multi-threading build completed

-.PHONY: lib-release
 %-release : DEBUGFLAGS :=
 %-release : %
 	@echo release build completed
@@ -268,13 +270,13 @@ LIBDIR      ?= $(libdir)
 includedir  ?= $(PREFIX)/include
 INCLUDEDIR  ?= $(includedir)

-PCLIBDIR ?= $(shell echo "$(LIBDIR)" | sed -n -E -e "s@^$(EXEC_PREFIX)(/|$$)@@p")
-PCINCDIR ?= $(shell echo "$(INCLUDEDIR)" | sed -n -E -e "s@^$(PREFIX)(/|$$)@@p")
+PCLIBDIR ?= $(shell echo "$(LIBDIR)" | sed -n $(SED_ERE_OPT) -e "s@^$(EXEC_PREFIX)(/|$$)@@p")
+PCINCDIR ?= $(shell echo "$(INCLUDEDIR)" | sed -n $(SED_ERE_OPT) -e "s@^$(PREFIX)(/|$$)@@p")

 ifeq (,$(PCLIBDIR))
 # Additional prefix check is required, since the empty string is technically a
 # valid PCLIBDIR
-ifeq (,$(shell echo "$(LIBDIR)" | sed -n -E -e "\\@^$(EXEC_PREFIX)(/|$$)@ p"))
+ifeq (,$(shell echo "$(LIBDIR)" | sed -n $(SED_ERE_OPT) -e "\\@^$(EXEC_PREFIX)(/|$$)@ p"))
 $(error configured libdir ($(LIBDIR)) is outside of prefix ($(PREFIX)), can't generate pkg-config file)
 endif
 endif
@@ -282,7 +284,7 @@ endif
 ifeq (,$(PCINCDIR))
 # Additional prefix check is required, since the empty string is technically a
 # valid PCINCDIR
-ifeq (,$(shell echo "$(INCLUDEDIR)" | sed -n -E -e "\\@^$(PREFIX)(/|$$)@ p"))
+ifeq (,$(shell echo "$(INCLUDEDIR)" | sed -n $(SED_ERE_OPT) -e "\\@^$(PREFIX)(/|$$)@ p"))
 $(error configured includedir ($(INCLUDEDIR)) is outside of exec_prefix ($(EXEC_PREFIX)), can't generate pkg-config file)
 endif
 endif
@@ -306,7 +308,7 @@ INSTALL_DATA    ?= $(INSTALL) -m 644
 libzstd.pc:
 libzstd.pc: libzstd.pc.in
 	@echo creating pkgconfig
-	$(Q)@sed -E -e 's|@PREFIX@|$(PREFIX)|' \
+	$(Q)@sed $(SED_ERE_OPT) -e 's|@PREFIX@|$(PREFIX)|' \
          -e 's|@LIBDIR@|$(PCLIBDIR)|' \
          -e 's|@INCLUDEDIR@|$(PCINCDIR)|' \
          -e 's|@VERSION@|$(VERSION)|' \
@@ -336,7 +338,6 @@ install-includes:
 	$(Q)$(INSTALL) -d -m 755 $(DESTDIR)$(INCLUDEDIR)/
 	$(Q)$(INSTALL_DATA) zstd.h $(DESTDIR)$(INCLUDEDIR)
 	$(Q)$(INSTALL_DATA) common/zstd_errors.h $(DESTDIR)$(INCLUDEDIR)
-	$(Q)$(INSTALL_DATA) deprecated/zbuff.h $(DESTDIR)$(INCLUDEDIR)     # prototypes generate deprecation warnings
 	$(Q)$(INSTALL_DATA) dictBuilder/zdict.h $(DESTDIR)$(INCLUDEDIR)

 uninstall:
@@ -347,7 +348,6 @@ uninstall:
 	$(Q)$(RM) $(DESTDIR)$(PKGCONFIGDIR)/libzstd.pc
 	$(Q)$(RM) $(DESTDIR)$(INCLUDEDIR)/zstd.h
 	$(Q)$(RM) $(DESTDIR)$(INCLUDEDIR)/zstd_errors.h
-	$(Q)$(RM) $(DESTDIR)$(INCLUDEDIR)/zbuff.h   # Deprecated streaming functions
 	$(Q)$(RM) $(DESTDIR)$(INCLUDEDIR)/zdict.h
 	@echo zstd libraries successfully uninstalled

@@ -38,6 +38,17 @@

 #endif

+/**
+  On MSVC qsort requires that functions passed into it use the __cdecl calling conversion(CC). 
+  This explictly marks such functions as __cdecl so that the code will still compile 
+  if a CC other than __cdecl has been made the default.
+*/
+#if  defined(_MSC_VER)
+#  define WIN_CDECL __cdecl
+#else
+#  define WIN_CDECL 
+#endif
+
 /**
 * FORCE_INLINE_TEMPLATE is used to define C "templates", which take constant
 * parameters. They must be inlined for the compiler to eliminate the constant
@@ -79,7 +79,7 @@ extern "C" {
 #  include <assert.h>
 #else
 #  ifndef assert   /* assert may be already defined, due to prior #include <assert.h> */
-#    define assert(condition) ((void)0)   /* disable assert (default) */
+#    define assert(condition) ((void) (condition))   /* disable assert (default) */
 #  endif
 #endif

@@ -301,7 +301,7 @@ int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque)
 struct POOL_ctx_s {
    int dummy;
 };
-static POOL_ctx g_ctx;
+static POOL_ctx g_poolCtx;

 POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) {
    return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
@@ -311,11 +311,11 @@ POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customM
    (void)numThreads;
    (void)queueSize;
    (void)customMem;
-    return &g_ctx;
+    return &g_poolCtx;
 }

 void POOL_free(POOL_ctx* ctx) {
-    assert(!ctx || ctx == &g_ctx);
+    assert(!ctx || ctx == &g_poolCtx);
    (void)ctx;
 }

@@ -337,7 +337,7 @@ int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque) {

 size_t POOL_sizeof(POOL_ctx* ctx) {
    if (ctx==NULL) return 0;  /* supports sizeof NULL */
-    assert(ctx == &g_ctx);
+    assert(ctx == &g_poolCtx);
    return sizeof(*ctx);
 }

@@ -1123,45 +1123,61 @@ ZSTD_sizeof_matchState(const ZSTD_compressionParameters* const cParams,
    return tableSpace + optSpace;
 }

+static size_t ZSTD_estimateCCtxSize_usingCCtxParams_internal(
+        const ZSTD_compressionParameters* cParams,
+        const ldmParams_t* ldmParams,
+        const int isStatic,
+        const size_t buffInSize,
+        const size_t buffOutSize,
+        const U64 pledgedSrcSize)
+{
+    size_t const windowSize = MAX(1, (size_t)MIN(((U64)1 << cParams->windowLog), pledgedSrcSize));
+    size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, windowSize);
+    U32    const divider = (cParams->minMatch==3) ? 3 : 4;
+    size_t const maxNbSeq = blockSize / divider;
+    size_t const tokenSpace = ZSTD_cwksp_alloc_size(WILDCOPY_OVERLENGTH + blockSize)
+                            + ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(seqDef))
+                            + 3 * ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(BYTE));
+    size_t const entropySpace = ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE);
+    size_t const blockStateSpace = 2 * ZSTD_cwksp_alloc_size(sizeof(ZSTD_compressedBlockState_t));
+    size_t const matchStateSize = ZSTD_sizeof_matchState(cParams, /* forCCtx */ 1);
+
+    size_t const ldmSpace = ZSTD_ldm_getTableSize(*ldmParams);
+    size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(*ldmParams, blockSize);
+    size_t const ldmSeqSpace = ldmParams->enableLdm ?
+        ZSTD_cwksp_alloc_size(maxNbLdmSeq * sizeof(rawSeq)) : 0;
+
+
+    size_t const bufferSpace = ZSTD_cwksp_alloc_size(buffInSize)
+                             + ZSTD_cwksp_alloc_size(buffOutSize);
+
+    size_t const cctxSpace = isStatic ? ZSTD_cwksp_alloc_size(sizeof(ZSTD_CCtx)) : 0;
+
+    size_t const neededSpace =
+        cctxSpace +
+        entropySpace +
+        blockStateSpace +
+        ldmSpace +
+        ldmSeqSpace +
+        matchStateSize +
+        tokenSpace +
+        bufferSpace;
+
+    DEBUGLOG(5, "estimate workspace : %u", (U32)neededSpace);
+    return neededSpace;
+}
+
 size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params)
 {
-    RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
-    {   ZSTD_compressionParameters const cParams =
+    ZSTD_compressionParameters const cParams =
                ZSTD_getCParamsFromCCtxParams(params, ZSTD_CONTENTSIZE_UNKNOWN, 0);
-        size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << cParams.windowLog);
-        U32    const divider = (cParams.minMatch==3) ? 3 : 4;
-        size_t const maxNbSeq = blockSize / divider;
-        size_t const tokenSpace = ZSTD_cwksp_alloc_size(WILDCOPY_OVERLENGTH + blockSize)
-                                + ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(seqDef))
-                                + 3 * ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(BYTE));
-        size_t const entropySpace = ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE);
-        size_t const blockStateSpace = 2 * ZSTD_cwksp_alloc_size(sizeof(ZSTD_compressedBlockState_t));
-        size_t const matchStateSize = ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 1);

-        size_t const ldmSpace = ZSTD_ldm_getTableSize(params->ldmParams);
-        size_t const ldmSeqSpace = ZSTD_cwksp_alloc_size(ZSTD_ldm_getMaxNbSeq(params->ldmParams, blockSize) * sizeof(rawSeq));
-
-        /* estimateCCtxSize is for one-shot compression. So no buffers should
-         * be needed. However, we still allocate two 0-sized buffers, which can
-         * take space under ASAN. */
-        size_t const bufferSpace = ZSTD_cwksp_alloc_size(0)
-                                 + ZSTD_cwksp_alloc_size(0);
-
-        size_t const cctxSpace = ZSTD_cwksp_alloc_size(sizeof(ZSTD_CCtx));
-
-        size_t const neededSpace =
-            cctxSpace +
-            entropySpace +
-            blockStateSpace +
-            ldmSpace +
-            ldmSeqSpace +
-            matchStateSize +
-            tokenSpace +
-            bufferSpace;
-
-        DEBUGLOG(5, "estimate workspace : %u", (U32)neededSpace);
-        return neededSpace;
-    }
+    RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
+    /* estimateCCtxSize is for one-shot compression. So no buffers should
+     * be needed. However, we still allocate two 0-sized buffers, which can
+     * take space under ASAN. */
+    return ZSTD_estimateCCtxSize_usingCCtxParams_internal(
+        &cParams, &params->ldmParams, 1, 0, 0, ZSTD_CONTENTSIZE_UNKNOWN);
 }

 size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams)
@@ -1192,14 +1208,13 @@ size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params)
    RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
    {   ZSTD_compressionParameters const cParams =
                ZSTD_getCParamsFromCCtxParams(params, ZSTD_CONTENTSIZE_UNKNOWN, 0);
-        size_t const CCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(params);
        size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << cParams.windowLog);
        size_t const inBuffSize = ((size_t)1 << cParams.windowLog) + blockSize;
        size_t const outBuffSize = ZSTD_compressBound(blockSize) + 1;
-        size_t const streamingSize = ZSTD_cwksp_alloc_size(inBuffSize)
-                                   + ZSTD_cwksp_alloc_size(outBuffSize);

-        return CCtxSize + streamingSize;
+        return ZSTD_estimateCCtxSize_usingCCtxParams_internal(
+            &cParams, &params->ldmParams, 1, inBuffSize, outBuffSize,
+            ZSTD_CONTENTSIZE_UNKNOWN);
    }
 }

@@ -1441,45 +1456,28 @@ static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
        size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, windowSize);
        U32    const divider = (params.cParams.minMatch==3) ? 3 : 4;
        size_t const maxNbSeq = blockSize / divider;
-        size_t const tokenSpace = ZSTD_cwksp_alloc_size(WILDCOPY_OVERLENGTH + blockSize)
-                                + ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(seqDef))
-                                + 3 * ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(BYTE));
        size_t const buffOutSize = (zbuff==ZSTDb_buffered) ? ZSTD_compressBound(blockSize)+1 : 0;
        size_t const buffInSize = (zbuff==ZSTDb_buffered) ? windowSize + blockSize : 0;
-        size_t const matchStateSize = ZSTD_sizeof_matchState(&params.cParams, /* forCCtx */ 1);
        size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(params.ldmParams, blockSize);

-        ZSTD_indexResetPolicy_e needsIndexReset = zc->initialized ? ZSTDirp_continue : ZSTDirp_reset;
+        int const indexTooClose = ZSTD_indexTooCloseToMax(zc->blockState.matchState.window);
+        ZSTD_indexResetPolicy_e needsIndexReset =
+            (!indexTooClose && zc->initialized) ? ZSTDirp_continue : ZSTDirp_reset;

-        if (ZSTD_indexTooCloseToMax(zc->blockState.matchState.window)) {
-            needsIndexReset = ZSTDirp_reset;
-        }
+        size_t const neededSpace =
+            ZSTD_estimateCCtxSize_usingCCtxParams_internal(
+                &params.cParams, &params.ldmParams, zc->staticSize != 0,
+                buffInSize, buffOutSize, pledgedSrcSize);
+        FORWARD_IF_ERROR(neededSpace, "cctx size estimate failed!");

        if (!zc->staticSize) ZSTD_cwksp_bump_oversized_duration(ws, 0);

        /* Check if workspace is large enough, alloc a new one if needed */
-        {   size_t const cctxSpace = zc->staticSize ? ZSTD_cwksp_alloc_size(sizeof(ZSTD_CCtx)) : 0;
-            size_t const entropySpace = ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE);
-            size_t const blockStateSpace = 2 * ZSTD_cwksp_alloc_size(sizeof(ZSTD_compressedBlockState_t));
-            size_t const bufferSpace = ZSTD_cwksp_alloc_size(buffInSize) + ZSTD_cwksp_alloc_size(buffOutSize);
-            size_t const ldmSpace = ZSTD_ldm_getTableSize(params.ldmParams);
-            size_t const ldmSeqSpace = ZSTD_cwksp_alloc_size(maxNbLdmSeq * sizeof(rawSeq));
-
-            size_t const neededSpace =
-                cctxSpace +
-                entropySpace +
-                blockStateSpace +
-                ldmSpace +
-                ldmSeqSpace +
-                matchStateSize +
-                tokenSpace +
-                bufferSpace;
-
+        {
            int const workspaceTooSmall = ZSTD_cwksp_sizeof(ws) < neededSpace;
            int const workspaceWasteful = ZSTD_cwksp_check_wasteful(ws, neededSpace);

-            DEBUGLOG(4, "Need %zuKB workspace, including %zuKB for match state, and %zuKB for buffers",
-                        neededSpace>>10, matchStateSize>>10, bufferSpace>>10);
+            DEBUGLOG(4, "Need %zu B workspace", neededSpace);
            DEBUGLOG(4, "windowSize: %zu - blockSize: %zu", windowSize, blockSize);

            if (workspaceTooSmall || workspaceWasteful) {
@@ -1579,6 +1577,12 @@ static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
            zc->ldmState.loadedDictEnd = 0;
        }

+        /* Due to alignment, when reusing a workspace, we can actually consume
+         * up to 3 extra bytes for alignment. See the comments in zstd_cwksp.h
+         */
+        assert(ZSTD_cwksp_used(ws) >= neededSpace &&
+               ZSTD_cwksp_used(ws) <= neededSpace + 3);
+
        DEBUGLOG(3, "wksp: finished allocating, %zd bytes remain available", ZSTD_cwksp_available_space(ws));
        zc->initialized = 1;

@@ -1638,7 +1642,7 @@ ZSTD_resetCCtx_byAttachingCDict(ZSTD_CCtx* cctx,
        assert(windowLog != 0);
        /* Resize working context table params for input only, since the dict
         * has its own tables. */
-        /* pledgeSrcSize == 0 means 0! */
+        /* pledgedSrcSize == 0 means 0! */
        params.cParams = ZSTD_adjustCParams_internal(*cdict_cParams, pledgedSrcSize, 0);
        params.cParams.windowLog = windowLog;
        FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize,
@@ -2887,22 +2891,28 @@ static size_t ZSTD_loadDictionaryContent(ZSTD_matchState_t* ms,


 /* Dictionaries that assign zero probability to symbols that show up causes problems
-   when FSE encoding.  Refuse dictionaries that assign zero probability to symbols
-   that we may encounter during compression.
-   NOTE: This behavior is not standard and could be improved in the future. */
-static size_t ZSTD_checkDictNCount(short* normalizedCounter, unsigned dictMaxSymbolValue, unsigned maxSymbolValue) {
+ * when FSE encoding. Mark dictionaries with zero probability symbols as FSE_repeat_check
+ * and only dictionaries with 100% valid symbols can be assumed valid.
+ */
+static FSE_repeat ZSTD_dictNCountRepeat(short* normalizedCounter, unsigned dictMaxSymbolValue, unsigned maxSymbolValue)
+{
    U32 s;
-    RETURN_ERROR_IF(dictMaxSymbolValue < maxSymbolValue, dictionary_corrupted, "dict fse tables don't have all symbols");
-    for (s = 0; s <= maxSymbolValue; ++s) {
-        RETURN_ERROR_IF(normalizedCounter[s] == 0, dictionary_corrupted, "dict fse tables don't have all symbols");
+    if (dictMaxSymbolValue < maxSymbolValue) {
+        return FSE_repeat_check;
    }
-    return 0;
+    for (s = 0; s <= maxSymbolValue; ++s) {
+        if (normalizedCounter[s] == 0) {
+            return FSE_repeat_check;
+        }
+    }
+    return FSE_repeat_valid;
 }

 size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace,
-                         short* offcodeNCount, unsigned* offcodeMaxValue,
                         const void* const dict, size_t dictSize)
 {
+    short offcodeNCount[MaxOff+1];
+    unsigned offcodeMaxValue = MaxOff;
    const BYTE* dictPtr = (const BYTE*)dict;    /* skip magic num and dict ID */
    const BYTE* const dictEnd = dictPtr + dictSize;
    dictPtr += 8;
@@ -2924,16 +2934,16 @@ size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace,
    }

    {   unsigned offcodeLog;
-        size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, offcodeMaxValue, &offcodeLog, dictPtr, dictEnd-dictPtr);
+        size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, &offcodeMaxValue, &offcodeLog, dictPtr, dictEnd-dictPtr);
        RETURN_ERROR_IF(FSE_isError(offcodeHeaderSize), dictionary_corrupted, "");
        RETURN_ERROR_IF(offcodeLog > OffFSELog, dictionary_corrupted, "");
-        /* Defer checking offcodeMaxValue because we need to know the size of the dictionary content */
        /* fill all offset symbols to avoid garbage at end of table */
        RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp(
                bs->entropy.fse.offcodeCTable,
                offcodeNCount, MaxOff, offcodeLog,
                workspace, HUF_WORKSPACE_SIZE)),
            dictionary_corrupted, "");
+        /* Defer checking offcodeMaxValue because we need to know the size of the dictionary content */
        dictPtr += offcodeHeaderSize;
    }

@@ -2942,13 +2952,12 @@ size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace,
        size_t const matchlengthHeaderSize = FSE_readNCount(matchlengthNCount, &matchlengthMaxValue, &matchlengthLog, dictPtr, dictEnd-dictPtr);
        RETURN_ERROR_IF(FSE_isError(matchlengthHeaderSize), dictionary_corrupted, "");
        RETURN_ERROR_IF(matchlengthLog > MLFSELog, dictionary_corrupted, "");
-        /* Every match length code must have non-zero probability */
-        FORWARD_IF_ERROR( ZSTD_checkDictNCount(matchlengthNCount, matchlengthMaxValue, MaxML), "");
        RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp(
                bs->entropy.fse.matchlengthCTable,
                matchlengthNCount, matchlengthMaxValue, matchlengthLog,
                workspace, HUF_WORKSPACE_SIZE)),
            dictionary_corrupted, "");
+        bs->entropy.fse.matchlength_repeatMode = ZSTD_dictNCountRepeat(matchlengthNCount, matchlengthMaxValue, MaxML);
        dictPtr += matchlengthHeaderSize;
    }

@@ -2957,13 +2966,12 @@ size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace,
        size_t const litlengthHeaderSize = FSE_readNCount(litlengthNCount, &litlengthMaxValue, &litlengthLog, dictPtr, dictEnd-dictPtr);
        RETURN_ERROR_IF(FSE_isError(litlengthHeaderSize), dictionary_corrupted, "");
        RETURN_ERROR_IF(litlengthLog > LLFSELog, dictionary_corrupted, "");
-        /* Every literal length code must have non-zero probability */
-        FORWARD_IF_ERROR( ZSTD_checkDictNCount(litlengthNCount, litlengthMaxValue, MaxLL), "");
        RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp(
                bs->entropy.fse.litlengthCTable,
                litlengthNCount, litlengthMaxValue, litlengthLog,
                workspace, HUF_WORKSPACE_SIZE)),
            dictionary_corrupted, "");
+        bs->entropy.fse.litlength_repeatMode = ZSTD_dictNCountRepeat(litlengthNCount, litlengthMaxValue, MaxLL);
        dictPtr += litlengthHeaderSize;
    }

@@ -2973,6 +2981,22 @@ size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace,
    bs->rep[2] = MEM_readLE32(dictPtr+8);
    dictPtr += 12;

+    {   size_t const dictContentSize = (size_t)(dictEnd - dictPtr);
+        U32 offcodeMax = MaxOff;
+        if (dictContentSize <= ((U32)-1) - 128 KB) {
+            U32 const maxOffset = (U32)dictContentSize + 128 KB; /* The maximum offset that must be supported */
+            offcodeMax = ZSTD_highbit32(maxOffset); /* Calculate minimum offset code required to represent maxOffset */
+        }
+        /* All offset values <= dictContentSize + 128 KB must be representable for a valid table */
+        bs->entropy.fse.offcode_repeatMode = ZSTD_dictNCountRepeat(offcodeNCount, offcodeMaxValue, MIN(offcodeMax, MaxOff));
+
+        /* All repCodes must be <= dictContentSize and != 0 */
+        {   U32 u;
+            for (u=0; u<3; u++) {
+                RETURN_ERROR_IF(bs->rep[u] == 0, dictionary_corrupted, "");
+                RETURN_ERROR_IF(bs->rep[u] > dictContentSize, dictionary_corrupted, "");
+    }   }   }
+
    return dictPtr - (const BYTE*)dict;
 }

@@ -2995,8 +3019,6 @@ static size_t ZSTD_loadZstdDictionary(ZSTD_compressedBlockState_t* bs,
 {
    const BYTE* dictPtr = (const BYTE*)dict;
    const BYTE* const dictEnd = dictPtr + dictSize;
-    short offcodeNCount[MaxOff+1];
-    unsigned offcodeMaxValue = MaxOff;
    size_t dictID;
    size_t eSize;

@@ -3005,32 +3027,16 @@ static size_t ZSTD_loadZstdDictionary(ZSTD_compressedBlockState_t* bs,
    assert(MEM_readLE32(dictPtr) == ZSTD_MAGIC_DICTIONARY);

    dictID = params->fParams.noDictIDFlag ? 0 :  MEM_readLE32(dictPtr + 4 /* skip magic number */ );
-    eSize = ZSTD_loadCEntropy(bs, workspace, offcodeNCount, &offcodeMaxValue, dict, dictSize);
+    eSize = ZSTD_loadCEntropy(bs, workspace, dict, dictSize);
    FORWARD_IF_ERROR(eSize, "ZSTD_loadCEntropy failed");
    dictPtr += eSize;

-    {   size_t const dictContentSize = (size_t)(dictEnd - dictPtr);
-        U32 offcodeMax = MaxOff;
-        if (dictContentSize <= ((U32)-1) - 128 KB) {
-            U32 const maxOffset = (U32)dictContentSize + 128 KB; /* The maximum offset that must be supported */
-            offcodeMax = ZSTD_highbit32(maxOffset); /* Calculate minimum offset code required to represent maxOffset */
-        }
-        /* All offset values <= dictContentSize + 128 KB must be representable */
-        FORWARD_IF_ERROR(ZSTD_checkDictNCount(offcodeNCount, offcodeMaxValue, MIN(offcodeMax, MaxOff)), "");
-        /* All repCodes must be <= dictContentSize and != 0*/
-        {   U32 u;
-            for (u=0; u<3; u++) {
-                RETURN_ERROR_IF(bs->rep[u] == 0, dictionary_corrupted, "");
-                RETURN_ERROR_IF(bs->rep[u] > dictContentSize, dictionary_corrupted, "");
-        }   }
-
-        bs->entropy.fse.offcode_repeatMode = FSE_repeat_valid;
-        bs->entropy.fse.matchlength_repeatMode = FSE_repeat_valid;
-        bs->entropy.fse.litlength_repeatMode = FSE_repeat_valid;
+    {
+        size_t const dictContentSize = (size_t)(dictEnd - dictPtr);
        FORWARD_IF_ERROR(ZSTD_loadDictionaryContent(
            ms, NULL, ws, params, dictPtr, dictContentSize, dtlm), "");
-        return dictID;
    }
+    return dictID;
 }

 /** ZSTD_compress_insertDictionary() :
@@ -1045,7 +1045,6 @@ MEM_STATIC void ZSTD_debugTable(const U32* table, U32 max)
 * assumptions : magic number supposed already checked
 *               and dictSize >= 8 */
 size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace,
-                         short* offcodeNCount, unsigned* offcodeMaxValue,
                         const void* const dict, size_t dictSize);

 void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs);
@@ -603,7 +603,7 @@ static size_t ZSTD_estimateSubBlockSize_symbolType(symbolEncodingType_e type,
                        const BYTE* codeTable, unsigned maxCode,
                        size_t nbSeq, const FSE_CTable* fseCTable,
                        const U32* additionalBits,
-                        short const* defaultNorm, U32 defaultNormLog,
+                        short const* defaultNorm, U32 defaultNormLog, U32 defaultMax,
                        void* workspace, size_t wkspSize)
 {
    unsigned* const countWksp = (unsigned*)workspace;
@@ -615,7 +615,11 @@ static size_t ZSTD_estimateSubBlockSize_symbolType(symbolEncodingType_e type,

    HIST_countFast_wksp(countWksp, &max, codeTable, nbSeq, workspace, wkspSize);  /* can't fail */
    if (type == set_basic) {
-        cSymbolTypeSizeEstimateInBits = ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, countWksp, max);
+        /* We selected this encoding type, so it must be valid. */
+        assert(max <= defaultMax);
+        cSymbolTypeSizeEstimateInBits = max <= defaultMax
+                ? ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, countWksp, max)
+                : ERROR(GENERIC);
    } else if (type == set_rle) {
        cSymbolTypeSizeEstimateInBits = 0;
    } else if (type == set_compressed || type == set_repeat) {
@@ -643,15 +647,15 @@ static size_t ZSTD_estimateSubBlockSize_sequences(const BYTE* ofCodeTable,
    size_t cSeqSizeEstimate = 0;
    cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->ofType, ofCodeTable, MaxOff,
                                         nbSeq, fseTables->offcodeCTable, NULL,
-                                         OF_defaultNorm, OF_defaultNormLog,
+                                         OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
                                         workspace, wkspSize);
    cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->llType, llCodeTable, MaxLL,
                                         nbSeq, fseTables->litlengthCTable, LL_bits,
-                                         LL_defaultNorm, LL_defaultNormLog,
+                                         LL_defaultNorm, LL_defaultNormLog, MaxLL,
                                         workspace, wkspSize);
    cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->mlType, mlCodeTable, MaxML,
                                         nbSeq, fseTables->matchlengthCTable, ML_bits,
-                                         ML_defaultNorm, ML_defaultNormLog,
+                                         ML_defaultNorm, ML_defaultNormLog, MaxML,
                                         workspace, wkspSize);
    if (writeEntropy) cSeqSizeEstimate += fseMetadata->fseTablesSize;
    return cSeqSizeEstimate + sequencesSectionHeaderSize;
@@ -809,7 +813,7 @@ static size_t ZSTD_compressSubBlock_multi(const seqStore_t* seqStorePtr,
        if (sp < send) {
            seqDef const* seq;
            repcodes_t rep;
-            memcpy(&rep, prevCBlock->rep, sizeof(rep)); 
+            memcpy(&rep, prevCBlock->rep, sizeof(rep));
            for (seq = sstart; seq < sp; ++seq) {
                rep = ZSTD_updateRep(rep.rep, seq->offset - 1, ZSTD_getSequenceLength(seqStorePtr, seq).litLength == 0);
            }
@@ -483,6 +483,11 @@ MEM_STATIC size_t ZSTD_cwksp_sizeof(const ZSTD_cwksp* ws) {
    return (size_t)((BYTE*)ws->workspaceEnd - (BYTE*)ws->workspace);
 }

+MEM_STATIC size_t ZSTD_cwksp_used(const ZSTD_cwksp* ws) {
+    return (size_t)((BYTE*)ws->tableEnd - (BYTE*)ws->workspace)
+         + (size_t)((BYTE*)ws->workspaceEnd - (BYTE*)ws->allocStart);
+}
+
 MEM_STATIC int ZSTD_cwksp_reserve_failed(const ZSTD_cwksp* ws) {
    return ws->allocFailed;
 }
@@ -721,7 +721,7 @@ static size_t ZSTD_decompressMultiFrame(ZSTD_DCtx* dctx,
            decodedSize = ZSTD_decompressLegacy(dst, dstCapacity, src, frameSize, dict, dictSize);
            if (ZSTD_isError(decodedSize)) return decodedSize;

-            assert(decodedSize <=- dstCapacity);
+            assert(decodedSize <= dstCapacity);
            dst = (BYTE*)dst + decodedSize;
            dstCapacity -= decodedSize;

@@ -1524,7 +1524,7 @@ static void ZSTD_DCtx_updateOversizedDuration(ZSTD_DStream* zds, size_t const ne
 {
    if (ZSTD_DCtx_isOverflow(zds, neededInBuffSize, neededOutBuffSize))
        zds->oversizedDuration++;
-    else 
+    else
        zds->oversizedDuration = 0;
 }

@@ -1731,7 +1731,7 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB

                {   int const tooSmall = (zds->inBuffSize < neededInBuffSize) || (zds->outBuffSize < neededOutBuffSize);
                    int const tooLarge = ZSTD_DCtx_isOversizedTooLong(zds);
-                    
+
                    if (tooSmall || tooLarge) {
                        size_t const bufferSize = neededInBuffSize + neededOutBuffSize;
                        DEBUGLOG(4, "inBuff  : from %u to %u",
@@ -948,7 +948,7 @@ ZSTD_decodeSequence(seqState_t* seqState, const ZSTD_longOffset_e longOffsets, c
 }

 #ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
-static int ZSTD_dictionaryIsActive(ZSTD_DCtx const* dctx, BYTE const* prefixStart, BYTE const* oLitEnd)
+MEM_STATIC int ZSTD_dictionaryIsActive(ZSTD_DCtx const* dctx, BYTE const* prefixStart, BYTE const* oLitEnd)
 {
    size_t const windowSize = dctx->fParams.windowSize;
    /* No dictionary used. */
@@ -969,6 +969,7 @@ MEM_STATIC void ZSTD_assertValidSequence(
        seq_t const seq,
        BYTE const* prefixStart, BYTE const* virtualStart)
 {
+#if DEBUGLEVEL >= 1
    size_t const windowSize = dctx->fParams.windowSize;
    size_t const sequenceSize = seq.litLength + seq.matchLength;
    BYTE const* const oLitEnd = op + seq.litLength;
@@ -986,6 +987,9 @@ MEM_STATIC void ZSTD_assertValidSequence(
        /* Offset must be within our window. */
        assert(seq.offset <= windowSize);
    }
+#else
+    (void)dctx, (void)op, (void)oend, (void)seq, (void)prefixStart, (void)virtualStart;
+#endif
 }
 #endif

@@ -40,23 +40,33 @@
 *  Constants
 ***************************************/
 #define COVER_MAX_SAMPLES_SIZE (sizeof(size_t) == 8 ? ((unsigned)-1) : ((unsigned)1 GB))
-#define DEFAULT_SPLITPOINT 1.0
+#define COVER_DEFAULT_SPLITPOINT 1.0

 /*-*************************************
 *  Console display
 ***************************************/
+#ifndef LOCALDISPLAYLEVEL
 static int g_displayLevel = 2;
+#endif
+#undef  DISPLAY
 #define DISPLAY(...)                                                           \
  {                                                                            \
    fprintf(stderr, __VA_ARGS__);                                              \
    fflush(stderr);                                                            \
  }
+#undef  LOCALDISPLAYLEVEL
 #define LOCALDISPLAYLEVEL(displayLevel, l, ...)                                \
  if (displayLevel >= l) {                                                     \
    DISPLAY(__VA_ARGS__);                                                      \
  } /* 0 : no display;   1: errors;   2: default;  3: details;  4: debug */
+#undef  DISPLAYLEVEL
 #define DISPLAYLEVEL(l, ...) LOCALDISPLAYLEVEL(g_displayLevel, l, __VA_ARGS__)

+#ifndef LOCALDISPLAYUPDATE
+static const clock_t refreshRate = CLOCKS_PER_SEC * 15 / 100;
+static clock_t g_time = 0;
+#endif
+#undef  LOCALDISPLAYUPDATE
 #define LOCALDISPLAYUPDATE(displayLevel, l, ...)                               \
  if (displayLevel >= l) {                                                     \
    if ((clock() - g_time > refreshRate) || (displayLevel >= 4)) {             \
@@ -64,9 +74,8 @@ static int g_displayLevel = 2;
      DISPLAY(__VA_ARGS__);                                                    \
    }                                                                          \
  }
+#undef  DISPLAYUPDATE
 #define DISPLAYUPDATE(l, ...) LOCALDISPLAYUPDATE(g_displayLevel, l, __VA_ARGS__)
-static const clock_t refreshRate = CLOCKS_PER_SEC * 15 / 100;
-static clock_t g_time = 0;

 /*-*************************************
 * Hash table
@@ -120,9 +129,9 @@ static int COVER_map_init(COVER_map_t *map, U32 size) {
 /**
 * Internal hash function
 */
-static const U32 prime4bytes = 2654435761U;
+static const U32 COVER_prime4bytes = 2654435761U;
 static U32 COVER_map_hash(COVER_map_t *map, U32 key) {
-  return (key * prime4bytes) >> (32 - map->sizeLog);
+  return (key * COVER_prime4bytes) >> (32 - map->sizeLog);
 }

 /**
@@ -215,7 +224,7 @@ typedef struct {
 } COVER_ctx_t;

 /* We need a global context for qsort... */
-static COVER_ctx_t *g_ctx = NULL;
+static COVER_ctx_t *g_coverCtx = NULL;

 /*-*************************************
 *  Helper functions
@@ -258,11 +267,11 @@ static int COVER_cmp8(COVER_ctx_t *ctx, const void *lp, const void *rp) {

 /**
 * Same as COVER_cmp() except ties are broken by pointer value
- * NOTE: g_ctx must be set to call this function.  A global is required because
+ * NOTE: g_coverCtx must be set to call this function.  A global is required because
 * qsort doesn't take an opaque pointer.
 */
-static int COVER_strict_cmp(const void *lp, const void *rp) {
-  int result = COVER_cmp(g_ctx, lp, rp);
+static int WIN_CDECL COVER_strict_cmp(const void *lp, const void *rp) {
+  int result = COVER_cmp(g_coverCtx, lp, rp);
  if (result == 0) {
    result = lp < rp ? -1 : 1;
  }
@@ -271,8 +280,8 @@ static int COVER_strict_cmp(const void *lp, const void *rp) {
 /**
 * Faster version for d <= 8.
 */
-static int COVER_strict_cmp8(const void *lp, const void *rp) {
-  int result = COVER_cmp8(g_ctx, lp, rp);
+static int WIN_CDECL COVER_strict_cmp8(const void *lp, const void *rp) {
+  int result = COVER_cmp8(g_coverCtx, lp, rp);
  if (result == 0) {
    result = lp < rp ? -1 : 1;
  }
@@ -603,7 +612,7 @@ static size_t COVER_ctx_init(COVER_ctx_t *ctx, const void *samplesBuffer,
    /* qsort doesn't take an opaque pointer, so pass as a global.
     * On OpenBSD qsort() is not guaranteed to be stable, their mergesort() is.
     */
-    g_ctx = ctx;
+    g_coverCtx = ctx;
 #if defined(__OpenBSD__)
    mergesort(ctx->suffix, ctx->suffixSize, sizeof(U32),
          (ctx->d <= 8 ? &COVER_strict_cmp8 : &COVER_strict_cmp));
@@ -1106,7 +1115,7 @@ ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_cover(
  /* constants */
  const unsigned nbThreads = parameters->nbThreads;
  const double splitPoint =
-      parameters->splitPoint <= 0.0 ? DEFAULT_SPLITPOINT : parameters->splitPoint;
+      parameters->splitPoint <= 0.0 ? COVER_DEFAULT_SPLITPOINT : parameters->splitPoint;
  const unsigned kMinD = parameters->d == 0 ? 6 : parameters->d;
  const unsigned kMaxD = parameters->d == 0 ? 8 : parameters->d;
  const unsigned kMinK = parameters->k == 0 ? 50 : parameters->k;
@@ -21,6 +21,7 @@
 #include "../common/threading.h"
 #include "cover.h"
 #include "../common/zstd_internal.h" /* includes zstd.h */
+#include "../compress/zstd_compress_internal.h" /* ZSTD_hash*() */
 #ifndef ZDICT_STATIC_LINKING_ONLY
 #define ZDICT_STATIC_LINKING_ONLY
 #endif
@@ -33,7 +34,7 @@
 #define FASTCOVER_MAX_SAMPLES_SIZE (sizeof(size_t) == 8 ? ((unsigned)-1) : ((unsigned)1 GB))
 #define FASTCOVER_MAX_F 31
 #define FASTCOVER_MAX_ACCEL 10
-#define DEFAULT_SPLITPOINT 0.75
+#define FASTCOVER_DEFAULT_SPLITPOINT 0.75
 #define DEFAULT_F 20
 #define DEFAULT_ACCEL 1

@@ -41,18 +42,28 @@
 /*-*************************************
 *  Console display
 ***************************************/
+#ifndef LOCALDISPLAYLEVEL
 static int g_displayLevel = 2;
+#endif
+#undef  DISPLAY
 #define DISPLAY(...)                                                           \
  {                                                                            \
    fprintf(stderr, __VA_ARGS__);                                              \
    fflush(stderr);                                                            \
  }
+#undef  LOCALDISPLAYLEVEL
 #define LOCALDISPLAYLEVEL(displayLevel, l, ...)                                \
  if (displayLevel >= l) {                                                     \
    DISPLAY(__VA_ARGS__);                                                      \
  } /* 0 : no display;   1: errors;   2: default;  3: details;  4: debug */
+#undef  DISPLAYLEVEL
 #define DISPLAYLEVEL(l, ...) LOCALDISPLAYLEVEL(g_displayLevel, l, __VA_ARGS__)

+#ifndef LOCALDISPLAYUPDATE
+static const clock_t refreshRate = CLOCKS_PER_SEC * 15 / 100;
+static clock_t g_time = 0;
+#endif
+#undef  LOCALDISPLAYUPDATE
 #define LOCALDISPLAYUPDATE(displayLevel, l, ...)                               \
  if (displayLevel >= l) {                                                     \
    if ((clock() - g_time > refreshRate) || (displayLevel >= 4)) {             \
@@ -60,31 +71,21 @@ static int g_displayLevel = 2;
      DISPLAY(__VA_ARGS__);                                                    \
    }                                                                          \
  }
+#undef  DISPLAYUPDATE
 #define DISPLAYUPDATE(l, ...) LOCALDISPLAYUPDATE(g_displayLevel, l, __VA_ARGS__)
-static const clock_t refreshRate = CLOCKS_PER_SEC * 15 / 100;
-static clock_t g_time = 0;


 /*-*************************************
 * Hash Functions
 ***************************************/
-static const U64 prime6bytes = 227718039650203ULL;
-static size_t ZSTD_hash6(U64 u, U32 h) { return (size_t)(((u  << (64-48)) * prime6bytes) >> (64-h)) ; }
-static size_t ZSTD_hash6Ptr(const void* p, U32 h) { return ZSTD_hash6(MEM_readLE64(p), h); }
-
-static const U64 prime8bytes = 0xCF1BBCDCB7A56463ULL;
-static size_t ZSTD_hash8(U64 u, U32 h) { return (size_t)(((u) * prime8bytes) >> (64-h)) ; }
-static size_t ZSTD_hash8Ptr(const void* p, U32 h) { return ZSTD_hash8(MEM_readLE64(p), h); }
-
-
 /**
- * Hash the d-byte value pointed to by p and mod 2^f
+ * Hash the d-byte value pointed to by p and mod 2^f into the frequency vector
 */
-static size_t FASTCOVER_hashPtrToIndex(const void* p, U32 h, unsigned d) {
+static size_t FASTCOVER_hashPtrToIndex(const void* p, U32 f, unsigned d) {
  if (d == 6) {
-    return ZSTD_hash6Ptr(p, h) & ((1 << h) - 1);
+    return ZSTD_hash6Ptr(p, f);
  }
-  return ZSTD_hash8Ptr(p, h) & ((1 << h) - 1);
+  return ZSTD_hash8Ptr(p, f);
 }


@@ -617,7 +618,7 @@ ZDICT_optimizeTrainFromBuffer_fastCover(
    /* constants */
    const unsigned nbThreads = parameters->nbThreads;
    const double splitPoint =
-        parameters->splitPoint <= 0.0 ? DEFAULT_SPLITPOINT : parameters->splitPoint;
+        parameters->splitPoint <= 0.0 ? FASTCOVER_DEFAULT_SPLITPOINT : parameters->splitPoint;
    const unsigned kMinD = parameters->d == 0 ? 6 : parameters->d;
    const unsigned kMaxD = parameters->d == 0 ? 8 : parameters->d;
    const unsigned kMinK = parameters->k == 0 ? 50 : parameters->k;
@@ -69,7 +69,9 @@ static const U32 g_selectivity_default = 9;
 /*-*************************************
 *  Console display
 ***************************************/
+#undef  DISPLAY
 #define DISPLAY(...)         { fprintf(stderr, __VA_ARGS__); fflush( stderr ); }
+#undef  DISPLAYLEVEL
 #define DISPLAYLEVEL(l, ...) if (notificationLevel>=l) { DISPLAY(__VA_ARGS__); }    /* 0 : no display;   1: errors;   2: default;  3: details;  4: debug */

 static clock_t ZDICT_clockSpan(clock_t nPrevious) { return clock() - nPrevious; }
@@ -105,20 +107,17 @@ size_t ZDICT_getDictHeaderSize(const void* dictBuffer, size_t dictSize)
    size_t headerSize;
    if (dictSize <= 8 || MEM_readLE32(dictBuffer) != ZSTD_MAGIC_DICTIONARY) return ERROR(dictionary_corrupted);

-    {   unsigned offcodeMaxValue = MaxOff;
-        ZSTD_compressedBlockState_t* bs = (ZSTD_compressedBlockState_t*)malloc(sizeof(ZSTD_compressedBlockState_t));
+    {   ZSTD_compressedBlockState_t* bs = (ZSTD_compressedBlockState_t*)malloc(sizeof(ZSTD_compressedBlockState_t));
        U32* wksp = (U32*)malloc(HUF_WORKSPACE_SIZE);
-        short* offcodeNCount = (short*)malloc((MaxOff+1)*sizeof(short));
-        if (!bs || !wksp || !offcodeNCount) {
+        if (!bs || !wksp) {
            headerSize = ERROR(memory_allocation);
        } else {
            ZSTD_reset_compressedBlockState(bs);
-            headerSize = ZSTD_loadCEntropy(bs, wksp, offcodeNCount, &offcodeMaxValue, dictBuffer, dictSize);
+            headerSize = ZSTD_loadCEntropy(bs, wksp, dictBuffer, dictSize);
        }

        free(bs);
        free(wksp);
-        free(offcodeNCount);
    }

    return headerSize;
@@ -532,6 +531,7 @@ static size_t ZDICT_trainBuffer_legacy(dictItem* dictList, U32 dictListSize,
    clock_t displayClock = 0;
    clock_t const refreshRate = CLOCKS_PER_SEC * 3 / 10;

+#   undef  DISPLAYUPDATE
 #   define DISPLAYUPDATE(l, ...) if (notificationLevel>=l) { \
            if (ZDICT_clockSpan(displayClock) > refreshRate)  \
            { displayClock = clock(); DISPLAY(__VA_ARGS__); \
@@ -74,7 +74,7 @@ extern "C" {
 /*-*************************************
 *  Debug
 ***************************************/
-#include "debug.h"
+#include "../common/debug.h"
 #ifndef assert
 #  define assert(condition) ((void)0)
 #endif
@@ -365,16 +365,20 @@ typedef enum {
    ZSTD_c_dictIDFlag=202,   /* When applicable, dictionary's ID is written into frame header (default:1) */

    /* multi-threading parameters */
-    /* These parameters are only useful if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).
-     * They return an error otherwise. */
+    /* These parameters are only active if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).
+     * Otherwise, trying to set any other value than default (0) will be a no-op and return an error.
+     * In a situation where it's unknown if the linked library supports multi-threading or not,
+     * setting ZSTD_c_nbWorkers to any value >= 1 and consulting the return value provides a quick way to check this property.
+     */
    ZSTD_c_nbWorkers=400,    /* Select how many threads will be spawned to compress in parallel.
-                              * When nbWorkers >= 1, triggers asynchronous mode when used with ZSTD_compressStream*() :
+                              * When nbWorkers >= 1, triggers asynchronous mode when invoking ZSTD_compressStream*() :
                              * ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller,
-                              * while compression work is performed in parallel, within worker threads.
+                              * while compression is performed in parallel, within worker thread(s).
                              * (note : a strong exception to this rule is when first invocation of ZSTD_compressStream2() sets ZSTD_e_end :
                              *  in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call).
                              * More workers improve speed, but also increase memory usage.
-                              * Default value is `0`, aka "single-threaded mode" : no worker is spawned, compression is performed inside Caller's thread, all invocations are blocking */
+                              * Default value is `0`, aka "single-threaded mode" : no worker is spawned,
+                              * compression is performed inside Caller's thread, and all invocations are blocking */
    ZSTD_c_jobSize=401,      /* Size of a compression job. This value is enforced only when nbWorkers >= 1.
                              * Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads.
                              * 0 means default, which is dynamically determined based on compression parameters.
@@ -227,7 +227,7 @@ zstd-dll : $(ZSTD_CLI_OBJ)
 	$(CC) $(FLAGS) $^ -o $@$(EXT) $(LDFLAGS)


-## zstd-pgo: zstd executable optimized with pgo. `gcc` only.
+## zstd-pgo: zstd executable optimized with PGO.
 zstd-pgo :
 	$(MAKE) clean
 	$(MAKE) zstd MOREFLAGS=-fprofile-generate
@@ -624,7 +624,6 @@ FIO_openDstFile(FIO_prefs_t* const prefs,
    }
 }

-
 /*! FIO_createDictBuffer() :
 *  creates a buffer, pointed by `*bufferPtr`,
 *  loads `filename` content into it, up to DICTSIZE_MAX bytes.
@@ -669,15 +668,9 @@ static size_t FIO_createDictBuffer(void** bufferPtr, const char* fileName, FIO_p
 * Checks for and warns if there are any files that would have the same output path
 */
 int FIO_checkFilenameCollisions(const char** filenameTable, unsigned nbFiles) {
-    const char **filenameTableSorted, *c, *prevElem, *filename;
+    const char **filenameTableSorted, *prevElem, *filename;
    unsigned u;

-    #if defined(_MSC_VER) || defined(__MINGW32__) || defined (__MSVCRT__) /* windows support */
-    c = "\\";
-    #else
-    c = "/";
-    #endif
-
    filenameTableSorted = (const char**) malloc(sizeof(char*) * nbFiles);
    if (!filenameTableSorted) {
        DISPLAY("Unable to malloc new str array, not checking for name collisions\n");
@@ -685,7 +678,7 @@ int FIO_checkFilenameCollisions(const char** filenameTable, unsigned nbFiles) {
    }

    for (u = 0; u < nbFiles; ++u) {
-        filename = strrchr(filenameTable[u], c[0]);
+        filename = strrchr(filenameTable[u], PATH_SEP);
        if (filename == NULL) {
            filenameTableSorted[u] = filenameTable[u];
        } else {
@@ -771,9 +764,12 @@ static void FIO_adjustMemLimitForPatchFromMode(FIO_prefs_t* const prefs,
                                    unsigned long long const maxSrcFileSize)
 {
    unsigned long long maxSize = MAX(prefs->memLimit, MAX(dictSize, maxSrcFileSize));
+    unsigned const maxWindowSize = (1U << ZSTD_WINDOWLOG_MAX);
+    if (maxSize == UTIL_FILESIZE_UNKNOWN)
+        EXM_THROW(42, "Using --patch-from with stdin requires --stream-size");
    assert(maxSize != UTIL_FILESIZE_UNKNOWN);
-    if (maxSize > UINT_MAX)
-        EXM_THROW(42, "Can't handle files larger than %u GB\n", UINT_MAX/(1 GB) + 1);
+    if (maxSize > maxWindowSize)
+        EXM_THROW(42, "Can't handle files larger than %u GB\n", maxWindowSize/(1 GB));
    FIO_setMemLimit(prefs, (unsigned)maxSize);
 }

@@ -809,7 +805,7 @@ static void FIO_adjustParamsForPatchFromMode(FIO_prefs_t* const prefs,
    comprParams->windowLog = MIN(ZSTD_WINDOWLOG_MAX, fileWindowLog);
    if (fileWindowLog > ZSTD_cycleLog(cParams.hashLog, cParams.strategy)) {
        if (!prefs->ldmFlag)
-            DISPLAYLEVEL(1, "long mode automaticaly triggered\n");
+            DISPLAYLEVEL(1, "long mode automatically triggered\n");
        FIO_setLdmFlag(prefs, 1);
    }
    if (cParams.strategy >= ZSTD_btopt) {
@@ -838,8 +834,10 @@ static cRess_t FIO_createCResources(FIO_prefs_t* const prefs,

    /* need to update memLimit before calling createDictBuffer
     * because of memLimit check inside it */
-    if (prefs->patchFromMode)
-        FIO_adjustParamsForPatchFromMode(prefs, &comprParams, UTIL_getFileSize(dictFileName), maxSrcFileSize, cLevel);
+    if (prefs->patchFromMode) {
+        unsigned long long const ssSize = (unsigned long long)prefs->streamSrcSize;
+        FIO_adjustParamsForPatchFromMode(prefs, &comprParams, UTIL_getFileSize(dictFileName), ssSize > 0 ? ssSize : maxSrcFileSize, cLevel);
+    }
    ress.dstBuffer = malloc(ress.dstBufferSize);
    ress.dictBufferSize = FIO_createDictBuffer(&ress.dictBuffer, dictFileName, prefs);   /* works with dictFileName==NULL */
    if (!ress.srcBuffer || !ress.dstBuffer)
@@ -1505,7 +1503,7 @@ static int FIO_compressFilename_dstFile(FIO_prefs_t* const prefs,
        } else if ( strcmp(dstFileName, stdoutmark)
                 && strcmp(dstFileName, nulmark)
                 && transfer_permissions) {
-            DISPLAYLEVEL(6, "FIO_compressFilename_dstFile: transfering permissions into dst: %s \n", dstFileName);
+            DISPLAYLEVEL(6, "FIO_compressFilename_dstFile: transferring permissions into dst: %s \n", dstFileName);
            UTIL_setFileStat(dstFileName, &statbuf);
        } else {
            DISPLAYLEVEL(6, "FIO_compressFilename_dstFile: do not transfer permissions into dst: %s \n", dstFileName);
@@ -1594,6 +1592,7 @@ int FIO_compressFilename(FIO_prefs_t* const prefs, const char* dstFileName,
    cRess_t const ress = FIO_createCResources(prefs, dictFileName, UTIL_getFileSize(srcFileName), compressionLevel, comprParams);
    int const result = FIO_compressFilename_srcFile(prefs, ress, dstFileName, srcFileName, compressionLevel);

+#define DISPLAY_LEVEL_DEFAULT 2

    FIO_freeCResources(ress);
    return result;
@@ -1658,6 +1657,7 @@ static unsigned long long FIO_getLargestFileSize(const char** inFileNames, unsig
 */
 int FIO_compressMultipleFilenames(FIO_prefs_t* const prefs,
                                  const char** inFileNamesTable, unsigned nbFiles,
+                                  const char* outMirroredRootDirName,
                                  const char* outDirName,
                                  const char* outFileName, const char* suffix,
                                  const char* dictFileName, int compressionLevel,
@@ -1684,12 +1684,30 @@ int FIO_compressMultipleFilenames(FIO_prefs_t* const prefs,
            ress.dstFile = NULL;
        }
    } else {
-        unsigned u;
+        unsigned int u=0;
+        if (outMirroredRootDirName)
+            UTIL_mirrorSourceFilesDirectories(inFileNamesTable, nbFiles, outMirroredRootDirName);
+
        for (u=0; u<nbFiles; u++) {
            const char* const srcFileName = inFileNamesTable[u];
-            const char* const dstFileName = FIO_determineCompressedName(srcFileName, outDirName, suffix);  /* cannot fail */
+            const char* dstFileName = NULL;
+            if (outMirroredRootDirName) {
+                char* validMirroredDirName = UTIL_createMirroredDestDirName(srcFileName, outMirroredRootDirName);
+                if (validMirroredDirName) {
+                    dstFileName = FIO_determineCompressedName(srcFileName, validMirroredDirName, suffix);
+                    free(validMirroredDirName);
+                } else {
+                    DISPLAYLEVEL(2, "zstd: --output-dir-mirror cannot compress '%s' into '%s' \n", srcFileName, outMirroredRootDirName);
+                    error=1;
+                    continue;
+                }
+            } else {
+                dstFileName = FIO_determineCompressedName(srcFileName, outDirName, suffix);  /* cannot fail */
+            }
+
            error |= FIO_compressFilename_srcFile(prefs, ress, dstFileName, srcFileName, compressionLevel);
        }
+
        if (outDirName)
            FIO_checkFilenameCollisions(inFileNamesTable ,nbFiles);
    }
@@ -2531,10 +2549,10 @@ FIO_determineDstName(const char* srcFileName, const char* outDirName)
    /* note : dstFileNameBuffer memory is not going to be free */
 }

-
 int
 FIO_decompressMultipleFilenames(FIO_prefs_t* const prefs,
                                const char** srcNamesTable, unsigned nbFiles,
+                                const char* outMirroredRootDirName,
                                const char* outDirName, const char* outFileName,
                                const char* dictFileName)
 {
@@ -2553,12 +2571,25 @@ FIO_decompressMultipleFilenames(FIO_prefs_t* const prefs,
            EXM_THROW(72, "Write error : %s : cannot properly close output file",
                        strerror(errno));
    } else {
-        unsigned u;
+        unsigned int u = 0;
+        if (outMirroredRootDirName)
+            UTIL_mirrorSourceFilesDirectories(srcNamesTable, nbFiles, outMirroredRootDirName);
+
        for (u=0; u<nbFiles; u++) {   /* create dstFileName */
            const char* const srcFileName = srcNamesTable[u];
-            const char* const dstFileName = FIO_determineDstName(srcFileName, outDirName);
+            const char* dstFileName = NULL;
+            if (outMirroredRootDirName) {
+                char* validMirroredDirName = UTIL_createMirroredDestDirName(srcFileName, outMirroredRootDirName);
+                if (validMirroredDirName) {
+                    dstFileName = FIO_determineDstName(srcFileName, validMirroredDirName);
+                    free(validMirroredDirName);
+                } else {
+                    DISPLAYLEVEL(2, "zstd: --output-dir-mirror cannot decompress '%s' into '%s'\n", srcFileName, outMirroredRootDirName);
+                }
+            } else {
+                dstFileName = FIO_determineDstName(srcFileName, outDirName);
+            }
            if (dstFileName == NULL) { error=1; continue; }
-
            error |= FIO_decompressSrcFile(prefs, ress, dstFileName, srcFileName);
        }
        if (outDirName)
@@ -122,6 +122,7 @@ int FIO_listMultipleFiles(unsigned numFiles, const char** filenameTable, int dis
 * @return : nb of missing files */
 int FIO_compressMultipleFilenames(FIO_prefs_t* const prefs,
                                  const char** inFileNamesTable, unsigned nbFiles,
+                                  const char* outMirroredDirName,
                                  const char* outDirName,
                                  const char* outFileName, const char* suffix,
                                  const char* dictFileName, int compressionLevel,
@@ -131,6 +132,7 @@ int FIO_compressMultipleFilenames(FIO_prefs_t* const prefs,
 * @return : nb of missing or skipped files */
 int FIO_decompressMultipleFilenames(FIO_prefs_t* const prefs,
                                    const char** srcNamesTable, unsigned nbFiles,
+                                    const char* outMirroredDirName,
                                    const char* outDirName,
                                    const char* outFileName,
                                    const char* dictFileName);
@@ -45,7 +45,6 @@ extern "C" {
 #  include <string.h>       /* strerror, memcpy */
 #endif /* #ifdef _WIN32 */

-
 /*-****************************************
 *  Internal Macros
 ******************************************/
@@ -130,6 +129,18 @@ int UTIL_getFileStat(const char* infilename, stat_t *statbuf)
    return 1;
 }

+int UTIL_getDirectoryStat(const char* infilename, stat_t *statbuf)
+{
+#if defined(_MSC_VER)
+    int const r = _stat64(infilename, statbuf);
+    if (!r && (statbuf->st_mode & _S_IFDIR)) return 1;
+#else
+    int const r = stat(infilename, statbuf);
+    if (!r && S_ISDIR(statbuf->st_mode)) return 1;
+#endif
+    return 0;
+}
+
 /* like chmod, but avoid changing permission of /dev/null */
 int UTIL_chmod(char const* filename, mode_t permissions)
 {
@@ -178,14 +189,7 @@ int UTIL_setFileStat(const char *filename, stat_t *statbuf)
 int UTIL_isDirectory(const char* infilename)
 {
    stat_t statbuf;
-#if defined(_MSC_VER)
-    int const r = _stat64(infilename, &statbuf);
-    if (!r && (statbuf.st_mode & _S_IFDIR)) return 1;
-#else
-    int const r = stat(infilename, &statbuf);
-    if (!r && S_ISDIR(statbuf.st_mode)) return 1;
-#endif
-    return 0;
+    return UTIL_getDirectoryStat(infilename, &statbuf);
 }

 int UTIL_compareStr(const void *p1, const void *p2) {
@@ -633,6 +637,287 @@ const char* UTIL_getFileExtension(const char* infilename)
   return extension;
 }

+static int pathnameHas2Dots(const char *pathname)
+{
+    return NULL != strstr(pathname, "..");
+}
+
+static int isFileNameValidForMirroredOutput(const char *filename)
+{
+    return !pathnameHas2Dots(filename);
+}
+
+
+#define DIR_DEFAULT_MODE 0755
+static mode_t getDirMode(const char *dirName)
+{
+    stat_t st;
+    int ret = UTIL_getDirectoryStat(dirName, &st);
+    if (!ret) {
+        UTIL_DISPLAY("zstd: failed to get DIR stats %s: %s\n", dirName, strerror(errno));
+        return DIR_DEFAULT_MODE;
+    }
+    return st.st_mode;
+}
+
+static int makeDir(const char *dir, mode_t mode)
+{
+#if defined(_MSC_VER) || defined(__MINGW32__) || defined (__MSVCRT__)
+    int ret = _mkdir(dir);
+    (void) mode;
+#else
+    int ret = mkdir(dir, mode);
+#endif
+    if (ret != 0) {
+        if (errno == EEXIST)
+            return 0;
+        UTIL_DISPLAY("zstd: failed to create DIR %s: %s\n", dir, strerror(errno));
+    }
+    return ret;
+}
+
+/* this function requires a mutable input string */
+static void convertPathnameToDirName(char *pathname)
+{
+    size_t len = 0;
+    char* pos = NULL;
+    /* get dir name from pathname similar to 'dirname()' */
+    assert(pathname != NULL);
+
+    /* remove trailing '/' chars */
+    len = strlen(pathname);
+    assert(len > 0);
+    while (pathname[len] == PATH_SEP) {
+        pathname[len] = '\0';
+        len--;
+    }
+    if (len == 0) return;
+
+    /* if input is a single file, return '.' instead. i.e.
+     * "xyz/abc/file.txt" => "xyz/abc"
+       "./file.txt"       => "."
+       "file.txt"         => "."
+     */
+    pos = strrchr(pathname, PATH_SEP);
+    if (pos == NULL) {
+        pathname[0] = '.';
+        pathname[1] = '\0';
+    } else {
+        *pos = '\0';
+    }
+}
+
+/* pathname must be valid */
+static const char* trimLeadingRootChar(const char *pathname)
+{
+    assert(pathname != NULL);
+    if (pathname[0] == PATH_SEP)
+        return pathname + 1;
+    return pathname;
+}
+
+/* pathname must be valid */
+static const char* trimLeadingCurrentDirConst(const char *pathname)
+{
+    assert(pathname != NULL);
+    if ((pathname[0] == '.') && (pathname[1] == PATH_SEP))
+        return pathname + 2;
+    return pathname;
+}
+
+static char*
+trimLeadingCurrentDir(char *pathname)
+{
+    /* 'union charunion' can do const-cast without compiler warning */
+    union charunion {
+        char *chr;
+        const char* cchr;
+    } ptr;
+    ptr.cchr = trimLeadingCurrentDirConst(pathname);
+    return ptr.chr;
+}
+
+/* remove leading './' or '/' chars here */
+static const char * trimPath(const char *pathname)
+{
+    return trimLeadingRootChar(
+            trimLeadingCurrentDirConst(pathname));
+}
+
+static char* mallocAndJoin2Dir(const char *dir1, const char *dir2)
+{
+    const size_t dir1Size = strlen(dir1);
+    const size_t dir2Size = strlen(dir2);
+    char *outDirBuffer, *buffer, trailingChar;
+
+    assert(dir1 != NULL && dir2 != NULL);
+    outDirBuffer = (char *) malloc(dir1Size + dir2Size + 2);
+    CONTROL(outDirBuffer != NULL);
+
+    strncpy(outDirBuffer, dir1, dir1Size);
+    outDirBuffer[dir1Size] = '\0';
+
+    if (dir2[0] == '.')
+        return outDirBuffer;
+
+    buffer = outDirBuffer + dir1Size;
+    trailingChar = *(buffer - 1);
+    if (trailingChar != PATH_SEP) {
+        *buffer = PATH_SEP;
+        buffer++;
+    }
+    strncpy(buffer, dir2, dir2Size);
+    buffer[dir2Size] = '\0';
+
+    return outDirBuffer;
+}
+
+/* this function will return NULL if input srcFileName is not valid name for mirrored output path */
+char* UTIL_createMirroredDestDirName(const char* srcFileName, const char* outDirRootName)
+{
+    char* pathname = NULL;
+    if (!isFileNameValidForMirroredOutput(srcFileName))
+        return NULL;
+
+    pathname = mallocAndJoin2Dir(outDirRootName, trimPath(srcFileName));
+
+    convertPathnameToDirName(pathname);
+    return pathname;
+}
+
+static int
+mirrorSrcDir(char* srcDirName, const char* outDirName)
+{
+    mode_t srcMode;
+    int status = 0;
+    char* newDir = mallocAndJoin2Dir(outDirName, trimPath(srcDirName));
+    if (!newDir)
+        return -ENOMEM;
+
+    srcMode = getDirMode(srcDirName);
+    status = makeDir(newDir, srcMode);
+    free(newDir);
+    return status;
+}
+
+static int
+mirrorSrcDirRecursive(char* srcDirName, const char* outDirName)
+{
+    int status = 0;
+    char* pp = trimLeadingCurrentDir(srcDirName);
+    char* sp = NULL;
+
+    while ((sp = strchr(pp, PATH_SEP)) != NULL) {
+        if (sp != pp) {
+            *sp = '\0';
+            status = mirrorSrcDir(srcDirName, outDirName);
+            if (status != 0)
+                return status;
+            *sp = PATH_SEP;
+        }
+        pp = sp + 1;
+    }
+    status = mirrorSrcDir(srcDirName, outDirName);
+    return status;
+}
+
+static void
+makeMirroredDestDirsWithSameSrcDirMode(char** srcDirNames, unsigned nbFile, const char* outDirName)
+{
+    unsigned int i = 0;
+    for (i = 0; i < nbFile; i++)
+        mirrorSrcDirRecursive(srcDirNames[i], outDirName);
+}
+
+static int
+firstIsParentOrSameDirOfSecond(const char* firstDir, const char* secondDir)
+{
+    size_t firstDirLen  = strlen(firstDir),
+           secondDirLen = strlen(secondDir);
+    return firstDirLen <= secondDirLen &&
+           (secondDir[firstDirLen] == PATH_SEP || secondDir[firstDirLen] == '\0') &&
+           0 == strncmp(firstDir, secondDir, firstDirLen);
+}
+
+static int compareDir(const void* pathname1, const void* pathname2) {
+    /* sort it after remove the leading '/'  or './'*/
+    const char* s1 = trimPath(*(char * const *) pathname1);
+    const char* s2 = trimPath(*(char * const *) pathname2);
+    return strcmp(s1, s2);
+}
+
+static void
+makeUniqueMirroredDestDirs(char** srcDirNames, unsigned nbFile, const char* outDirName)
+{
+    unsigned int i = 0, uniqueDirNr = 0;
+    char** uniqueDirNames = NULL;
+
+    if (nbFile == 0)
+        return;
+
+    uniqueDirNames = (char** ) malloc(nbFile * sizeof (char *));
+    CONTROL(uniqueDirNames != NULL);
+
+    /* if dirs is "a/b/c" and "a/b/c/d", we only need call:
+     * we just need "a/b/c/d" */
+    qsort((void *)srcDirNames, nbFile, sizeof(char*), compareDir);
+
+    uniqueDirNr = 1;
+    uniqueDirNames[uniqueDirNr - 1] = srcDirNames[0];
+    for (i = 1; i < nbFile; i++) {
+        char* prevDirName = srcDirNames[i - 1];
+        char* currDirName = srcDirNames[i];
+
+        /* note: we alwasy compare trimmed path, i.e.:
+         * src dir of "./foo" and "/foo" will be both saved into:
+         * "outDirName/foo/" */
+        if (!firstIsParentOrSameDirOfSecond(trimPath(prevDirName),
+                                            trimPath(currDirName)))
+            uniqueDirNr++;
+
+        /* we need maintain original src dir name instead of trimmed
+         * dir, so we can retrive the original src dir's mode_t */
+        uniqueDirNames[uniqueDirNr - 1] = currDirName;
+    }
+
+    makeMirroredDestDirsWithSameSrcDirMode(uniqueDirNames, uniqueDirNr, outDirName);
+
+    free(uniqueDirNames);
+}
+
+static void
+makeMirroredDestDirs(char** srcFileNames, unsigned nbFile, const char* outDirName)
+{
+    unsigned int i = 0;
+    for (i = 0; i < nbFile; ++i)
+        convertPathnameToDirName(srcFileNames[i]);
+    makeUniqueMirroredDestDirs(srcFileNames, nbFile, outDirName);
+}
+
+void UTIL_mirrorSourceFilesDirectories(const char** inFileNames, unsigned int nbFile, const char* outDirName)
+{
+    unsigned int i = 0, validFilenamesNr = 0;
+    char** srcFileNames = (char **) malloc(nbFile * sizeof (char *));
+    CONTROL(srcFileNames != NULL);
+
+    /* check input filenames is valid */
+    for (i = 0; i < nbFile; ++i) {
+        if (isFileNameValidForMirroredOutput(inFileNames[i])) {
+            char* fname = STRDUP(inFileNames[i]);
+            CONTROL(fname != NULL);
+            srcFileNames[validFilenamesNr++] = fname;
+        }
+    }
+
+    if (validFilenamesNr > 0) {
+        makeDir(outDirName, DIR_DEFAULT_MODE);
+        makeMirroredDestDirs(srcFileNames, validFilenamesNr, outDirName);
+    }
+
+    for (i = 0; i < validFilenamesNr; i++)
+        free(srcFileNames[i]);
+    free(srcFileNames);
+}

 FileNamesTable*
 UTIL_createExpandedFNT(const char** inputNames, size_t nbIfns, int followLinks)
@@ -104,6 +104,14 @@ extern int g_utilDisplayLevel;
    typedef struct stat stat_t;
 #endif

+#if defined(_MSC_VER) || defined(__MINGW32__) || defined (__MSVCRT__) /* windows support */
+#define PATH_SEP '\\'
+#define STRDUP(s) _strdup(s)
+#else
+#define PATH_SEP '/'
+#include <libgen.h>
+#define STRDUP(s) strdup(s)
+#endif

 int UTIL_fileExist(const char* filename);
 int UTIL_isRegularFile(const char* infilename);
@@ -118,9 +126,13 @@ U64 UTIL_getFileSize(const char* infilename);
 U64 UTIL_getTotalFileSize(const char* const * fileNamesTable, unsigned nbFiles);
 int UTIL_getFileStat(const char* infilename, stat_t* statbuf);
 int UTIL_setFileStat(const char* filename, stat_t* statbuf);
+int UTIL_getDirectoryStat(const char* infilename, stat_t* statbuf);
 int UTIL_chmod(char const* filename, mode_t permissions);   /*< like chmod, but avoid changing permission of /dev/null */
 int UTIL_compareStr(const void *p1, const void *p2);
 const char* UTIL_getFileExtension(const char* infilename);
+void  UTIL_mirrorSourceFilesDirectories(const char** fileNamesTable, unsigned int nbFiles, const char *outDirName);
+char* UTIL_createMirroredDestDirName(const char* srcFileName, const char* outDirRootName);
+


 /*-****************************************
@@ -207,6 +219,7 @@ void UTIL_refFilename(FileNamesTable* fnt, const char* filename);
 #  define UTIL_HAS_CREATEFILELIST
 #elif defined(__linux__) || (PLATFORM_POSIX_VERSION >= 200112L)  /* opendir, readdir require POSIX.1-2001 */
 #  define UTIL_HAS_CREATEFILELIST
+#  define UTIL_HAS_MIRRORFILELIST
 #else
   /* do not define UTIL_HAS_CREATEFILELIST */
 #endif
@@ -123,7 +123,7 @@ Note: cannot use both this and \-D together Note: \fB\-\-long\fR mode will be au
 This is also used during compression when using with \-\-patch\-from=\. In this case, this parameter overrides that maximum size allowed for a dictionary\. (128 MB)\.
 .
 .IP "\(bu" 4
-\fB\-T#\fR, \fB\-\-threads=#\fR: Compress using \fB#\fR working threads (default: 1)\. If \fB#\fR is 0, attempt to detect and use the number of physical CPU cores\. In all cases, the nb of threads is capped to ZSTDMT_NBTHREADS_MAX==200\. This modifier does nothing if \fBzstd\fR is compiled without multithread support\.
+\fB\-T#\fR, \fB\-\-threads=#\fR: Compress using \fB#\fR working threads (default: 1)\. If \fB#\fR is 0, attempt to detect and use the number of physical CPU cores\. In all cases, the nb of threads is capped to ZSTDMT_NBWORKERS_MAX==200\. This modifier does nothing if \fBzstd\fR is compiled without multithread support\.
 .
 .IP "\(bu" 4
 \fB\-\-single\-thread\fR: Does not spawn a thread for compression, use a single thread for both I/O and compression\. In this mode, compression is serialized with I/O, which is slightly slower\. (This is different from \fB\-T1\fR, which spawns 1 compression thread in parallel of I/O)\. This mode is the only one available when multithread support is disabled\. Single\-thread mode features lower memory usage\. Final compressed result is slightly different from \fB\-T1\fR\.
@@ -129,12 +129,12 @@ the last one takes effect.

    Note: cannot use both this and -D together
    Note: `--long` mode will be automatically activated if chainLog < fileLog
-        (fileLog being the windowLog requried to cover the whole file). You 
+        (fileLog being the windowLog required to cover the whole file). You
        can also manually force it.
 	Node: for all levels, you can use --patch-from in --single-thread mode
 		to improve compression ratio at the cost of speed
-    Note: for level 19, you can get increased compression ratio at the cost 
-        of speed by specifying `--zstd=targetLength=` to be something large 
+    Note: for level 19, you can get increased compression ratio at the cost
+        of speed by specifying `--zstd=targetLength=` to be something large
        (i.e 4096), and by setting a large `--zstd=chainLog=`
 * `-M#`, `--memory=#`:
    Set a memory usage limit. By default, Zstandard uses 128 MB for decompression
@@ -147,7 +147,7 @@ the last one takes effect.
 * `-T#`, `--threads=#`:
    Compress using `#` working threads (default: 1).
    If `#` is 0, attempt to detect and use the number of physical CPU cores.
-    In all cases, the nb of threads is capped to ZSTDMT_NBTHREADS_MAX==200.
+    In all cases, the nb of threads is capped to ZSTDMT_NBWORKERS_MAX==200.
    This modifier does nothing if `zstd` is compiled without multithread support.
 * `--single-thread`:
    Does not spawn a thread for compression, use a single thread for both I/O and compression.
@@ -208,7 +208,7 @@ the last one takes effect.
    This setting overrides default and can force sparse mode over stdout.
 * `--[no-]content-size`:
    enable / disable whether or not the original size of the file is placed in
-    the header of the compressed file. The default option is 
+    the header of the compressed file. The default option is
    --content-size (meaning that the original size will be placed in the header).
 * `--rm`:
    remove source file(s) after successful compression or decompression
@@ -227,6 +227,15 @@ the last one takes effect.
    if multiple files, from different directories, end up having the same name.
    Collision resolution ensures first file with a given name will be present in `dir`,
    while in combination with `-f`, the last file will be present instead.
+* `--output-dir-mirror[=dir]`:
+    similar to `--output-dir-flat`, the output files are stored underneath target
+    `dir` directory, but this option will preserve input directories structure in output `dir`.
+
+    If the input directory has "..", the files in this directory will be ignored. If
+    the input directory is absolute directory (i.e. "/var/tmp/abc"), it will be
+    stored into the "output-dir/var/tmp/abc".
+    If there is multiple input files or directories, the collision resolution will be same as
+    `--output-dir-flat`.
 * `--format=FORMAT`:
    compress and decompress in other formats. If compiled with
    support, zstd can compress to or decompress from other compression algorithm
@@ -238,6 +247,7 @@ the last one takes effect.
    display version number and exit.
    Advanced : `-vV` also displays supported formats.
    `-vvV` also displays POSIX support.
+    `-q` will only display the version number, suitable for machine reading.
 * `-v`, `--verbose`:
    verbose mode
 * `--show-default-cparams`:
@@ -150,6 +150,10 @@ static void usage_advanced(const char* programName)
    DISPLAYOUT( "--output-dir-flat=DIR : all resulting files are stored into DIR \n");
 #endif

+#ifdef UTIL_HAS_MIRRORFILELIST
+    DISPLAYOUT( "--output-dir-mirror==DIR : all resulting files are stored into DIR in the original directory structure \n");
+#endif
+
    DISPLAYOUT( "--      : All arguments after \"--\" are treated as files \n");

 #ifndef ZSTD_NOCOMPRESS
@@ -544,6 +548,11 @@ static unsigned parseCompressionParameters(const char* stringPtr, ZSTD_compressi

 static void printVersion(void)
 {
+    if (g_displayLevel < DISPLAY_LEVEL_DEFAULT) {
+        DISPLAYOUT("%s\n", ZSTD_VERSION_STRING);
+        return;
+    }
+
    DISPLAYOUT(WELCOME_MESSAGE);
    if (g_displayLevel >= 3) {
    /* format support */
@@ -640,6 +649,7 @@ int main(int const argCount, const char* argv[])
        rsyncable = 0,
        nextArgumentIsOutFileName = 0,
        nextArgumentIsOutDirName = 0,
+        nextArgumentIsMirroredOutDirName = 0,
        nextArgumentIsMaxDict = 0,
        nextArgumentIsDictID = 0,
        nextArgumentsAreFiles = 0,
@@ -667,6 +677,7 @@ int main(int const argCount, const char* argv[])
    const char* programName = argv[0];
    const char* outFileName = NULL;
    const char* outDirName = NULL;
+    const char* outMirroredDirName = NULL;
    const char* dictFileName = NULL;
    const char* patchFromDictFileName = NULL;
    const char* suffix = ZSTD_EXTENSION;
@@ -763,6 +774,9 @@ int main(int const argCount, const char* argv[])
                if (!strcmp(argument, "--rm")) { FIO_setRemoveSrcFile(prefs, 1); continue; }
                if (!strcmp(argument, "--priority=rt")) { setRealTimePrio = 1; continue; }
                if (!strcmp(argument, "--output-dir-flat")) {nextArgumentIsOutDirName=1; lastCommand=1; continue; }
+#ifdef UTIL_HAS_MIRRORFILELIST
+                if (!strcmp(argument, "--output-dir-mirror")) {nextArgumentIsMirroredOutDirName=1; lastCommand=1; continue; }
+#endif
                if (!strcmp(argument, "--show-default-cparams")) { showDefaultCParams = 1; continue; }
                if (!strcmp(argument, "--content-size")) { contentSize = 1; continue; }
                if (!strcmp(argument, "--no-content-size")) { contentSize = 0; continue; }
@@ -833,6 +847,7 @@ int main(int const argCount, const char* argv[])
                if (longCommandWArg(&argument, "--target-compressed-block-size=")) { targetCBlockSize = readSizeTFromChar(&argument); continue; }
                if (longCommandWArg(&argument, "--size-hint=")) { srcSizeHint = readSizeTFromChar(&argument); continue; }
                if (longCommandWArg(&argument, "--output-dir-flat=")) { outDirName = argument; continue; }
+                if (longCommandWArg(&argument, "--output-dir-mirror=")) { outMirroredDirName = argument; continue; }
                if (longCommandWArg(&argument, "--patch-from=")) { patchFromDictFileName = argument; continue; }
                if (longCommandWArg(&argument, "--long")) {
                    unsigned ldmWindowLog = 0;
@@ -1061,6 +1076,13 @@ int main(int const argCount, const char* argv[])
            continue;
        }

+        if (nextArgumentIsMirroredOutDirName) {
+            nextArgumentIsMirroredOutDirName = 0;
+            lastCommand = 0;
+            outMirroredDirName = argument;
+            continue;
+        }
+
        /* none of the above : add filename to list */
        UTIL_refFilename(filenames, argument);
    }
@@ -1325,7 +1347,7 @@ int main(int const argCount, const char* argv[])
        if ((filenames->tableSize==1) && outFileName)
          operationResult = FIO_compressFilename(prefs, outFileName, filenames->fileNames[0], dictFileName, cLevel, compressionParams);
        else
-          operationResult = FIO_compressMultipleFilenames(prefs, filenames->fileNames, (unsigned)filenames->tableSize, outDirName, outFileName, suffix, dictFileName, cLevel, compressionParams);
+          operationResult = FIO_compressMultipleFilenames(prefs, filenames->fileNames, (unsigned)filenames->tableSize, outMirroredDirName, outDirName, outFileName, suffix, dictFileName, cLevel, compressionParams);
 #else
        (void)contentSize; (void)suffix; (void)adapt; (void)rsyncable; (void)ultra; (void)cLevel; (void)ldmFlag; (void)literalCompressionMode; (void)targetCBlockSize; (void)streamSrcSize; (void)srcSizeHint; (void)ZSTD_strategyMap; /* not used when ZSTD_NOCOMPRESS set */
        DISPLAY("Compression not supported \n");
@@ -1335,7 +1357,7 @@ int main(int const argCount, const char* argv[])
        if (filenames->tableSize == 1 && outFileName) {
            operationResult = FIO_decompressFilename(prefs, outFileName, filenames->fileNames[0], dictFileName);
        } else {
-            operationResult = FIO_decompressMultipleFilenames(prefs, filenames->fileNames, (unsigned)filenames->tableSize, outDirName, outFileName, dictFileName);
+            operationResult = FIO_decompressMultipleFilenames(prefs, filenames->fileNames, (unsigned)filenames->tableSize, outMirroredDirName, outDirName, outFileName, dictFileName);
        }
 #else
        DISPLAY("Decompression not supported \n");
@@ -0,0 +1 @@
+<?xml version="1.0" encoding="UTF-8" ?><VAST version="2.0"><Ad id="43C53990160C658B"><Wrapper><AdSystem version="1.0">ads60.vertamedia.com</AdSystem><VASTAdTagURI>http://ads60.vertamedia.com/ops/43C53990160C658B/46991</VASTAdTagURI><Impression><![CDATA[http://ads60.vertamedia.com/i/?x=1&adId=43C53990160C658B&aid=46991&cmpId=13765&fi=24929&advId=13177&pubId=28786&sid=0&width=300&height=250&domain=m.gazetaexpress.com]]></Impression><Creatives><Creative><Linear><TrackingEvents><Tracking event="start">http://ads60.vertamedia.com/ve/43C53990160C658B/53</Tracking><Tracking event="midpoint">http://ads60.vertamedia.com/ve/43C53990160C658B/54</Tracking><Tracking event="firstQuartile">http://ads60.vertamedia.com/ve/43C53990160C658B/55</Tracking><Tracking event="thirdQuartile">http://ads60.vertamedia.com/ve/43C53990160C658B/56</Tracking><Tracking event="complete">http://ads60.vertamedia.com/ve/43C53990160C658B/57</Tracking><Tracking event="skip">http://ads60.vertamedia.com/ve/43C53990160C658B/66</Tracking></TrackingEvents><VideoClicks><ClickTracking>http://ads60.vertamedia.com/ve/43C53990160C658B/71</ClickTracking></VideoClicks></Linear></Creative></Creatives></Wrapper></Ad></VAST>
@@ -8,22 +8,22 @@ die() {
 }

 datagen() {
-    "$DATAGEN_BIN" $@
+    "$DATAGEN_BIN" "$@"
 }

 zstd() {
    if [ -z "$EXEC_PREFIX" ]; then
-        "$ZSTD_BIN" $@
+        "$ZSTD_BIN" "$@"
    else
-        "$EXEC_PREFIX" "$ZSTD_BIN" $@
+        "$EXEC_PREFIX" "$ZSTD_BIN" "$@"
    fi
 }

 sudoZstd() {
    if [ -z "$EXEC_PREFIX" ]; then
-        sudo "$ZSTD_BIN" $@
+        sudo "$ZSTD_BIN" "$@"
    else
-        sudo "$EXEC_PREFIX" "$ZSTD_BIN" $@
+        sudo "$EXEC_PREFIX" "$ZSTD_BIN" "$@"
    fi
 }

@@ -78,6 +78,11 @@ println() {
    printf '%b\n' "${*}"
 }

+if [ -z "${size}" ]; then
+    size=
+else
+    size=${size}
+fi

 SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
 PRGDIR="$SCRIPT_DIR/../programs"
@@ -123,10 +128,25 @@ case "$UNAME" in
  SunOS) DIFF="gdiff" ;;
 esac

-println "\nStarting playTests.sh isWindows=$isWindows EXE_PREFIX='$EXE_PREFIX' ZSTD_BIN='$ZSTD_BIN' DATAGEN_BIN='$DATAGEN_BIN'"

-[ -n "$ZSTD_BIN" ] || die "\$ZSTD_BIN variable must be defined!"
-[ -n "$DATAGEN_BIN" ] || die "\$DATAGEN_BIN variable must be defined!"
+# check if ZSTD_BIN is defined. if not, use the default value
+if [ -z "${ZSTD_BIN}" ]; then
+  println "\nZSTD_BIN is not set. Using the default value..."
+  ZSTD_BIN="$PRGDIR/zstd"
+fi
+
+# check if DATAGEN_BIN is defined. if not, use the default value
+if [ -z "${DATAGEN_BIN}" ]; then
+  println "\nDATAGEN_BIN is not set. Using the default value..."
+  DATAGEN_BIN="$TESTDIR/datagen"
+fi
+
+ZSTD_BIN="$EXE_PREFIX$ZSTD_BIN"
+
+# assertions
+[ -n "$ZSTD_BIN" ] || die "zstd not found at $ZSTD_BIN! \n Please define ZSTD_BIN pointing to the zstd binary. You might also consider rebuilding zstd follwing the instructions in README.md"
+[ -n "$DATAGEN_BIN" ] || die "datagen not found at $DATAGEN_BIN! \n Please define DATAGEN_BIN pointing to the datagen binary. You might also consider rebuilding zstd tests following the instructions in README.md. "
+println "\nStarting playTests.sh isWindows=$isWindows EXE_PREFIX='$EXE_PREFIX' ZSTD_BIN='$ZSTD_BIN' DATAGEN_BIN='$DATAGEN_BIN'"

 if echo hello | zstd -v -T2 2>&1 > $INTOVOID | grep -q 'multi-threading is disabled'
 then
@@ -425,6 +445,31 @@ test -f tmpOutDirDecomp/tmp2
 test -f tmpOutDirDecomp/tmp1
 rm -rf tmp*

+if [ "$isWindows" = false ] ; then
+    println "\n===>  compress multiple files into an output directory and mirror input folder, --output-dir-mirror"
+    println "test --output-dir-mirror" > tmp1
+    mkdir -p tmpInputTestDir/we/must/go/deeper
+    println cool > tmpInputTestDir/we/must/go/deeper/tmp2
+    zstd tmp1 -r tmpInputTestDir --output-dir-mirror tmpOutDir
+    test -f tmpOutDir/tmp1.zst
+    test -f tmpOutDir/tmpInputTestDir/we/must/go/deeper/tmp2.zst
+
+    println "test: compress input dir will be ignored if it has '..'"
+    zstd  -r tmpInputTestDir/we/must/../must --output-dir-mirror non-exist && die "input cannot contain '..'"
+    test ! -d non-exist
+
+    println "test : decompress multiple files into an output directory, --output-dir-mirror"
+    zstd tmpOutDir -r -d --output-dir-mirror tmpOutDirDecomp
+    test -f tmpOutDirDecomp/tmpOutDir/tmp1
+    test -f tmpOutDirDecomp/tmpOutDir/tmpInputTestDir/we/must/go/deeper/tmp2
+
+    println "test: decompress input dir will be ignored if it has '..'"
+    zstd  -r tmpOutDir/tmpInputTestDir/we/must/../must --output-dir-mirror non-exist && die "input cannot contain '..'"
+    test ! -d non-exist
+
+    rm -rf tmp*
+fi
+

 println "test : compress multiple files reading them from a file, --filelist=FILE"
 println "Hello world!, file1" > tmp1
@@ -873,8 +918,9 @@ datagen | zstd -c | zstd -t

 println "\n===>  golden files tests "

-zstd -t -r "$TESTDIR/golden-compression"
+zstd -t -r "$TESTDIR/golden-decompression"
 zstd -c -r "$TESTDIR/golden-compression" | zstd -t
+zstd -D "$TESTDIR/golden-dictionaries/http-dict-missing-symbols" "$TESTDIR/golden-compression/http" -c | zstd -D "$TESTDIR/golden-dictionaries/http-dict-missing-symbols" -t


 println "\n===>  benchmark mode tests "
@@ -1280,7 +1326,14 @@ println "\n===> patch-from long mode trigger larger file test"

 datagen -g5000000 > tmp_dict
 datagen -g5000000 > tmp_patch
-zstd -15 --patch-from=tmp_dict tmp_patch 2>&1 | grep "long mode automaticaly triggered"
+zstd -15 --patch-from=tmp_dict tmp_patch 2>&1 | grep "long mode automatically triggered"
+rm -rf tmp*
+
+println "\n===> patch-from --stream-size test"
+datagen -g1000 -P50 > tmp_dict
+datagen -g1000 -P10 > tmp_patch
+cat tmp_patch | zstd -f --patch-from=tmp_dict -c -o tmp_patch_diff && die
+cat tmp_patch | zstd -f --patch-from=tmp_dict --stream-size=1000 -c -o tmp_patch_diff
 rm -rf tmp*

 println "\n===>   large files tests "
Author	SHA1	Message	Date
Yann Collet	2fb5881303	fixed incorrect assert() Incredibly evaded detection so far, probably because -size_t is still perfectly defined size_t which seems good enough for gcc and clang, but Visual finds that rather fishy.	2020-07-15 12:43:11 -07:00
Yann Collet	7d3d5f2a20	Merge pull request #2244 from hctim/dev Fix -Wunused-variable under FUZZING_BUILD_MODE...	2020-07-15 12:40:17 -07:00
Mitch Phillips	23b55d6b3e	Fix -Wunused-variable under FUZZING_BUILD_MODE... Fuzzing build modes (FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) doesn't necessarily imply that assert() is enabled, according to the manual. When the current do-nothing is expanded under -Wunused-variable (-Wall), it results in unused variables in some of the FUZZING_BUILD_MODE... blocks. This patch extends the do-nothing to avoid the unused variable.	2020-07-14 09:03:02 -07:00
Nick Terrell	bd21e4b264	Merge pull request #2242 from evverx/cifuzz-ubsan-msan cifuzz: turn on UBsan and MSan	2020-07-13 14:26:13 -04:00
Evgeny Vereshchagin	8e20f133e7	cifuzz: turn on UBsan and MSan	2020-07-13 01:20:11 +00:00
Yann Collet	16b353b207	minor doc clarification regarding MT parameters	2020-07-11 02:16:52 -07:00
Nick Terrell	bdf716bc64	Merge pull request #2239 from evverx/evverx-cifuzz cifuzz: don't upload artifacts when CIFuzz fails to build fuzz targets	2020-07-10 11:26:15 -04:00
Yann Collet	a3296dae38	Merge pull request #2219 from xxie24/output-dir-mirror Add output-dir-mirror option	2020-07-10 01:21:47 -07:00
Yann Collet	865ecf67ae	Merge pull request #2183 from tonytheodore/patch-1 Install pkg-config file on MINGW also	2020-07-10 01:20:59 -07:00
Yann Collet	78ebe4b5de	Merge pull request #2237 from marxin/gcda-to-gitignore Add GCC PGO data files to .gitignore.	2020-07-09 08:48:36 -07:00
Yann Collet	c0ee26ec15	Merge pull request #2236 from marxin/pgo-fix-comment Update comment about PGO build.	2020-07-09 08:48:04 -07:00
Yann Collet	2fbc438a5d	Merge pull request #2231 from marxin/use-threads-in-example Use threads in streaming_compression.c example.	2020-07-09 08:41:44 -07:00
Martin Liska	7276ab15bc	Add GCC PGO data files to .gitignore.	2020-07-09 09:32:54 +02:00
Martin Liska	5b9d6b992a	Update comment about PGO build.	2020-07-09 09:31:51 +02:00
Martin Liska	461348c09a	Use threads in streaming_compression.c example. Fixes #1584.	2020-07-09 09:20:06 +02:00
Evgeny Vereshchagin	3b9df5896f	cifuzz: don't upload artifacts when CIFuzz fails to build fuzz targets It should address https://github.com/google/oss-fuzz/issues/3982	2020-07-09 06:02:48 +00:00
Yann Collet	978cea5d56	Merge pull request #2230 from cwoffenden/minor-tidy Remove superfluous build-time macros (single-file libs)	2020-07-08 11:49:40 -07:00
Yann Collet	fedecb29d8	Merge pull request #2235 from bimbashrestha/decom-macros [actions] removing flakey min-decomp	2020-07-08 11:49:04 -07:00
Bimba Shrestha	2598af451f	removing min decomp	2020-07-08 11:11:45 -07:00
Carl Woffenden	9bfd371d9f	Remove superfluous build-time options	2020-07-08 11:39:17 +02:00
Yann Collet	2cdd33ae16	Merge pull request #2227 from yoshihitoh/single-file-dict-emscripten [contrib] Fix single-file compilation error on Emscripten build.	2020-07-07 08:51:20 -07:00
yoshihitoh	bc5fe33243	Change Emscripten docker image to the official one which is used for testing the single-file feature.	2020-07-07 19:10:12 +09:00
Yann Collet	f014c3cd40	Merge pull request #2228 from marxin/rename-ZSTDMT_NBTHREADS_MAX Rename ZSTDMT_NBTHREADS_MAX to ZSTDMT_NBWORKERS_MAX.	2020-07-06 16:06:49 -07:00
Martin Liska	4307e870a2	Rename ZSTDMT_NBTHREADS_MAX to ZSTDMT_NBWORKERS_MAX. Rename it in zstd man page.	2020-07-07 00:18:20 +02:00
yoshihitoh	a035654ab8	Enable docker based Emscripten build on single file library test if emcc is not available.	2020-07-05 22:25:20 +09:00
Nick Terrell	a2b6a8fbc6	Merge pull request #2220 from dkcasset/fix-lib-Makefile-older-sed-version Replace -E option with equivalent -r for older versions of sed	2020-06-29 17:19:24 -07:00
Nick Terrell	3690fe30f3	Merge pull request #2221 from BioDataAnalysis/bda_install_dll Install MSVC runtime shared libraries (DLL) with cmake	2020-06-29 17:14:50 -07:00
dkcasset	82e7e2b47e	Add variable for sed extended RE option (defaults to -E)	2020-06-29 13:44:23 -07:00
Mario Emmenlauer	37f36b9670	build/cmake/lib/CMakeLists.txt: Install shared library (dll) for Windows	2020-06-29 14:15:12 +02:00
yoshihitoh	c6548eac8e	Rename static vars to avoid redefinition error.	2020-06-29 10:51:50 +09:00
dkcasset	b0ed66ef92	Replace -E option with equivalent -r for older versions of sed	2020-06-26 10:43:28 -07:00
Xin Xie	9a8ccd4ba3	Add output-dir-mirror option	2020-06-24 22:12:11 -07:00
Nick Terrell	7afd5d85d3	Merge pull request #2218 from terrelln/assert-seq Fix unused variable warnings in fuzzing build mode without asserts	2020-06-22 17:41:18 -07:00
Nick Terrell	081691a3aa	Merge pull request #2217 from terrelln/cover-redundant [cover] Remove unnecessary mask and dedup hash functions	2020-06-22 17:41:13 -07:00
Nick Terrell	370933fa20	Merge pull request #2209 from Niadb/dev Explicitly use __cdecl for qsort, to avoid warning when default calling convention is not __cdecl	2020-06-22 17:41:03 -07:00
Nick Terrell	cce0edfdbe	Fix unused variable warnings in fuzzing build mode without asserts Fix unused vairable warnings when `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is defined but asserts are disabled. Fixes #2210.	2020-06-22 12:56:57 -07:00
Nick Terrell	2312b819af	[cover] Remove unnecessary mask and dedup hash functions * Remove the unnecessary mask, since `ZSTD_hash()` already ensures the output is mod 2^h. Dedup the hash functions and use `ZSTD_hash*()` directly.	2020-06-22 12:52:13 -07:00
Nick Terrell	78601d0806	Merge pull request #2212 from cwoffenden/single-file-dict Single-file libs now include dictBuilder	2020-06-22 12:46:54 -07:00
Nick Terrell	34c0d85916	Merge pull request #2214 from terrelln/superblock-cross-entropy [superblock] Add defensive assert and bounds check	2020-06-22 12:27:58 -07:00
Nick Terrell	1047097dad	[superblock] Add defensive assert and bounds check The bound check condition should always be met because we selected `set_basic` as our encoding type. But that code is very far away, so assert it is true so if it is ever false we can catch it, and add a bounds check. Fixes #2213.	2020-06-22 10:21:38 -07:00
Carl Woffenden	38cdb6a072	Renamed cover and fast cover hash functions/vars	2020-06-22 11:54:24 +02:00
Carl Woffenden	4a9b7d136f	Initial implementation (files added, macros fixed) Hashing functions still to fix.	2020-06-22 10:31:36 +02:00
Niadb	74f65f624c	Update compiler.h clean wording	2020-06-19 09:51:00 -06:00
Niadb	8c115cbe23	Update compiler.h Added a comment explaining the purpose of the WIN_CDECL macro	2020-06-19 09:48:35 -06:00
Niadb	2962fda93f	Add files via upload	2020-06-19 03:34:05 -06:00
Niadb	405586d40a	Add files via upload	2020-06-19 03:32:11 -06:00
Niadb	a4c8aa5e02	Add files via upload	2020-06-19 03:31:47 -06:00
Bimba Shrestha	f03192c122	Merge pull request #2208 from bimbashrestha/arm-actions [actions] adding arm to actions	2020-06-18 17:18:31 -05:00
Bimba Shrestha	4d6e4a2978	adding arm to actions	2020-06-18 12:28:37 -07:00
Bimba Shrestha	c496fd261d	Merge pull request #2206 from bimbashrestha/stream-size [cli] Allowing stdin on --patch-from using --stream-size	2020-06-18 13:10:11 -05:00
Bimba Shrestha	de48f35306	adding --patch-from --stream-size test	2020-06-18 10:28:37 -07:00
Bimba Shrestha	b06710889f	error when not using --stream-size on pipe	2020-06-18 09:31:06 -07:00
Bimba Shrestha	6653321ba3	passing streamSrcSize when set	2020-06-18 09:28:18 -07:00
Bimba Shrestha	4f59be7410	Merge pull request #2205 from bimbashrestha/cmake Removing de-duplication of cmake flags	2020-06-15 19:41:55 -05:00
Bimba Shrestha	0fa9406ae2	removing deduplicatin of flags	2020-06-15 14:58:53 -07:00
Yann Collet	c1913ed8c1	added paragraph on static analysis pitfalls	2020-06-15 13:04:45 -07:00
Nick Terrell	b6a9ded994	Merge pull request #2202 from terrelln/dict-fi [lib] Allow compression dictionaries with missing symbols	2020-06-15 10:54:33 -07:00
Nick Terrell	08981d2638	[lib] Allow compression dictionaries with missing symbols Allow compression to use dictionaries with missing symbols in their entropy tables. We set the FSE repeat mode to check when there are missing symbols, and set the FSE repeat mode to valid when all symbols are present. Note that when not all symbols are present, the heuristics which favor dictionary tables for lower compression levels won't activate. Tested by manually creating a dictionary with missing symbols of every type, and validing that the compressor rejects it before this change, and accepts it after this change. Also, I ran the `dictionary_loader` fuzzer for >1 hour of CPU time without running into cases where compression succeeds, but decompression fails. Fixes #2174.	2020-06-12 17:57:19 -07:00
Yann Collet	e8127b3f15	Merge pull request #2196 from tru/dumpversion_argument Make --version respect -q for ouptut a shorter version string	2020-06-11 02:04:04 -07:00
Tobias Hieta	1c9b1ddac7	Update version documention with the support for -q	2020-06-11 10:56:47 +02:00
Tobias Hieta	b1cce9a6ad	Short version string output This adds support for just showing the version string without the full welcome message if the log level is less than the default. That means that you pass -q once and you will just see "1.4.5". This makes it easier to parse in scripts.	2020-06-09 08:40:41 +02:00
Bimba Shrestha	4c144cf9c5	Merge pull request #2193 from bimbashrestha/typos Fixing some typos	2020-06-05 12:00:04 -05:00
Bimba Shrestha	e2838d9eb9	Spelling mistakes	2020-06-05 05:11:26 -05:00
Yann Collet	4b201f98eb	Merge pull request #2190 from facebook/playTest_improvement Check environmental variables in playTests.sh and add default values	2020-06-03 13:26:52 -07:00
Shaojing Li	847349195f	fix the if statements in posix sh env	2020-06-03 11:36:38 -07:00
Shaojing Li	3a3da1712b	check env variables and add default values	2020-06-03 10:49:21 -07:00
Bimba Shrestha	448658a662	Merge pull request #2187 from bimbashrestha/bourne [ci] Adding bourne shell compatibility test	2020-06-02 13:07:36 -05:00
Bimba Shrestha	00ac0415ac	Merge pull request #2186 from bimbashrestha/dll-install [cmake] Fix dll install location	2020-06-02 12:32:12 -05:00
Bimba Shrestha	eee59be6a8	excluding ls \| grep warning	2020-06-02 09:15:24 -07:00
Bimba Shrestha	b0f851675a	[shellcheck] setting if unset	2020-06-02 09:12:50 -07:00
Bimba Shrestha	151deaf143	[shellcheck] adding quotes to expansion	2020-06-02 09:12:13 -07:00
Bimba Shrestha	12185ec5e7	adding shellcheck bourne test	2020-06-02 09:08:16 -07:00
Bimba Shrestha	0ce5f70ca8	newline lints (ignore)	2020-06-02 09:07:45 -07:00
Bimba Shrestha	e75db971fc	DLL install fix for https://github.com/facebook/zstd/issues/2160 Credit goes to @xantares Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:	2020-06-02 07:45:06 -07:00
Tony Theodore	53fe73a720	Install pkg-config file on MINGW also Mingw follows unix conventions and often expects pkg-config files to be present.	2020-06-02 18:18:40 +10:00
Yann Collet	e7948fdf84	Merge pull request #2181 from facebook/makejall fixed double-compilation of lib with make -j all	2020-05-29 13:49:38 -07:00
Yann Collet	105b5839de	added comment on $(Q)	2020-05-29 12:40:59 -07:00
Yann Collet	6ac6472988	fixed double-compilation of lib with make -j all also implemented verbose mode (make V=1)	2020-05-29 12:09:51 -07:00
Felix Handte	2af4e07326	Merge pull request #2133 from felixhandte/single-size-calculation Consolidate CCtx Size Estimation Code	2020-05-28 13:07:18 -04:00
Bimba Shrestha	7f248af036	Merge pull request #2180 from facebook/generic-release [actions] Migrating 2nd batch of tests from Travis to Github Actions	2020-05-26 16:30:11 -05:00
Yann Collet	11a392ce23	minor markdown formatting fix	2020-05-26 13:15:35 -07:00
Bimba Shrestha	4ae5cb1ec9	Create generic-release.yml	2020-05-26 15:12:00 -05:00
Bimba Shrestha	b1a269437a	Merge pull request #2137 from facebook/generic-dev [actions] Migrating 1st batch of tests from Travis to Github Actions	2020-05-26 13:03:17 -05:00
Bimba Shrestha	c96151a25d	Merge pull request #2178 from bimbashrestha/quick-fix Change --patch-from limit from 4gb to 2gb	2020-05-26 12:26:22 -05:00
Bimba Shrestha	f847909cf2	Using ZSTD_MAX_WINDOWLOG instead of INT_MAX	2020-05-26 09:23:26 -07:00
Yann Collet	8a5f794e16	Merge pull request #2176 from facebook/make_implicit fixed default rule for lib/Makefile	2020-05-26 08:55:34 -07:00
Yann Collet	a49db88020	Merge pull request #2177 from facebook/dictID0 updated spec on dictID==0	2020-05-26 08:54:10 -07:00
Bimba Shrestha	01b38ef162	Change --patch-from limit from 4gb to 2gb	2020-05-26 06:52:30 -07:00
Yann Collet	bb3c9bf43a	updated spec on dictID==0 Specified decoder behavior on receiving a frame with dictID=0. Pushed paragraph on reserved DictID ranges into the Dictionary Format section.	2020-05-25 08:15:09 -07:00
Yann Collet	39a32f40c9	fixed default rule for lib/Makefile default rule is `lib-release` `lib-release` wasn't working : it was just skipped. Removing `lib-release` from the list of .PHONY targets fixes it. Same for `lib-mt`.	2020-05-25 06:50:45 -07:00
Nick Terrell	9eb2ccc9fb	Merge pull request #2167 from terrelln/pzstd-ms-fix [pzstd] Fix compilation error in MinGW	2020-05-22 23:38:27 -07:00
Nick Terrell	5aa5aa4df7	[pzstd] Fix compilation error in MinGW	2020-05-22 22:26:02 -07:00
Yann Collet	52a62e46f5	Merge pull request #2166 from facebook/donotinstall_zbuffh do not install zbuff.h	2020-05-22 17:16:23 -07:00
Yann Collet	082755bd3f	do not install zbuff.h this API is deprecated, for a loong time now, all related symbols will be removed in a future version (likely v1.5.0) and the header file `zbuff.h` doesn't compile from `include/` anyway, because it needs to be positioned one directory below `zstd.h`. Also removed `cover.h` from `cmake` installer, as it should have never been part of this list to begin with.	2020-05-22 15:35:54 -07:00
Yann Collet	e20b601781	Merge pull request #2163 from orivej/fix-legacy-build Fix legacy build after #2103	2020-05-22 09:55:17 -07:00
Orivej Desh	93cec0c1d6	Fix legacy build after #2103	2020-05-22 12:48:02 +00:00
W. Felix Handte	3bb7992350	Fix Size Estimate for LDM Seq Space	2020-05-14 13:50:53 -04:00
Bimba Shrestha	a49a4852c2	Migrating first batch of tests from travis # Dev PR jobs that still have to be migrated from travis # # icc (need self-hosted) # arm/qemu-arm (need self-hosted) # versionTag # valgrindTest (keeps failing for some reason. need investigation) # staticAnalyze (need trusty so need self-hosted) # pcc-fuzz: (need trusty so need self-hosted) # arm-build-test (need self-hosted)	2020-05-14 12:35:37 -05:00
W. Felix Handte	d9a1e37aec	Nit: Fix Size Type for 32-bit	2020-05-12 18:03:31 -04:00
W. Felix Handte	1aa6c7ccce	Assert We Allocated Approximately What We Expected To	2020-05-12 16:55:03 -04:00
W. Felix Handte	27e2482217	Minor Refactor	2020-05-12 16:55:03 -04:00
W. Felix Handte	afc2488973	Handle Non-Static CCtxes in Estimation	2020-05-12 16:54:33 -04:00
W. Felix Handte	7ed996f5a0	Consolidate CCtx Size Estimation Code This commit pulls out the internals of `ZSTD_estimateCCtxSize_usingCCtxParams` into a helper. It then migrates two other callsites to use that helper, a small optimization for `ZSTD_estimateCStreamSize_usingCCtxParams`, which folds the buffer sizing into the helper, and then `ZSTD_resetCCtx_internal`, which is more invasive. This attempts to guarantee that the estimates returned to users are always correct.	2020-05-12 16:26:53 -04:00
				`@@ -0,0 +1 @@`
				<?xml version="1.0" encoding="UTF-8" ?><VAST version="2.0"><Ad id="43C53990160C658B"><Wrapper><AdSystem version="1.0">ads60.vertamedia.com</AdSystem><VASTAdTagURI>http://ads60.vertamedia.com/ops/43C53990160C658B/46991</VASTAdTagURI><Impression><![CDATA[http://ads60.vertamedia.com/i/?x=1&adId=43C53990160C658B&aid=46991&cmpId=13765&fi=24929&advId=13177&pubId=28786&sid=0&width=300&height=250&domain=m.gazetaexpress.com]]></Impression><Creatives><Creative><Linear><TrackingEvents><Tracking event="start">http://ads60.vertamedia.com/ve/43C53990160C658B/53</Tracking><Tracking event="midpoint">http://ads60.vertamedia.com/ve/43C53990160C658B/54</Tracking><Tracking event="firstQuartile">http://ads60.vertamedia.com/ve/43C53990160C658B/55</Tracking><Tracking event="thirdQuartile">http://ads60.vertamedia.com/ve/43C53990160C658B/56</Tracking><Tracking event="complete">http://ads60.vertamedia.com/ve/43C53990160C658B/57</Tracking><Tracking event="skip">http://ads60.vertamedia.com/ve/43C53990160C658B/66</Tracking></TrackingEvents><VideoClicks><ClickTracking>http://ads60.vertamedia.com/ve/43C53990160C658B/71</ClickTracking></VideoClicks></Linear></Creative></Creatives></Wrapper></Ad></VAST>