Document the following options:
--leftomost-captures
--invert-captures
And the following configurations:
re2c:leftmost-captures
re2c:invert-captures
And the new `(! ...)` syntax.
It cannot be a `enum class` because it's used in a bitfield and this
triggers compiler warnings and errors on some toolchains (see
https://github.com/skvadrik/re2c/issues/446 for details).
It cannot be a plain `enum` defaulting to `int` as well because of
-Wsigned-enum-bitfield warning (noticed when testing on OpenBSD with
clang-13).
On Unix systems, re2c uses POSIX functions (e.g. fdopen). Certain
systems, such as Cygwin and NetBSD, enable these funtions conditionally
based on certain preprocessor definitions.
Previously https://github.com/skvadrik/re2c/pull/376 fixed the build on
Cygwin by setting _POSIX_SOURCE. However, this broke the build on
FreeBSD which was fixed by commit
This worked for Autotools build system which uses -std=c++11 due to the
use of AC_USE_SYSTEM_EXTENSIONS, but it didn't work for CMake that used
-std=c++11 and disabled extensions with `set(CMAKE_CXX_EXTENSIONS OFF)`.
This commit drops `set(CMAKE_CXX_EXTENSIONS OFF)`, which effectively
enables -std=gnu++11 instead of -std=c++11 for CMake builds. This allows
us to get rid of the problematic _POSIX_SOURCE that breaks the build on
NetBSD.
This was tested on NetBSD-9.3, FreeBSD-13.2 and Cygwin with both
Autotools and CMake.
With `--flex-syntax` option re2c accepts raw (unquoted) strings.
If `--input-encoding utf8` is also enabled, re2c should parse raw UTF-8
characters in such strings. Previously it failed to do so because the
characters in raw string literals were limited to alphanumeric ASCII and
underscore (those allowed in indentifiers, as in normal mode unquoted
strings would be parsed as identifiers).
This commit fixes it: now in Flex mode and with UTF-8 input, if all
previous rules failed, re2c tries to lex the input as a UTF-8 character.
This fixes bug #447 "Re2go fail when compiling expression containing
combining character".
GCC-5.4 is unhappy that `GorPass` which is `enum class: uint32_t` is
stored in a bitfield of size 2 (2 bits). In practice it is safe since
the enum has only three values (0, 1, 2).
Reported in:
- #445 "make failure for re2c"
- #446 "make failure for re2c on centos7"
Change file context definition to cover more scenarios that include
filenames, e.g. the case when there is a depfile generated and appended
to the output.
This allows other projects that use bazel to utilize re2c as a tool.
Limitations:
. Input files (docs, lexer, parser) are not being re-generated.
. No tests are executed.
. Only the main binary is being build as nothing else is needed.
A user will load the package using bazel's 'http_archive' in their
WORKSPACE file.
In their BUILD files they can then use:
```
load("@re2c//bazel:re2c.bzl", "re2c_gen")
re2c_gen(
name = "file_re2c",
srcs = ["file.re2c"],
outs = [
"file.cc",
"file.h",
]
)
cc_library(
name = "file_cc",
srcs = ["file.cc"],
hdrs = ["file.h"],
)
```
The approach here is to automate as much as possible. In that way, all
source and header files will be found automatically (using glob). Also,
the provided bazel integration is kept to an absolute minimum.
An alternative is to use a `rules_foreign_cc` based approach. But that
would require much more bazel work and knowledge. Plus the result would
end up being a non-hermetic build, thereby bypassing bazel's advantages.
Further, using a rules_foreign_cc approach is much slower (for me ~x10).
Make option --invert-captures configurable. This requires postponing the
decision whether a group is capturing or not to the after-parsing stage
when the configurations for a block are known and the immutable AST is
converted to a regexp.
Add a test for the new configuration.
The new option inverts the meaning on capturing and non-capturing
parentheses: `(! ...)` becomes capturing, and `( ... )` becomes
non-capturing, including the implicit whole-regexp group, which also
becomes non-capturing.
The idea of the new option is to let the user decide which gropus should
be capturing and which are purely structural, and explicitly mark only
the capturing ones for submatch extraction.
The old syntax was `(? ...)`, the new syntax is `(! ...)`. This is in
preparation to add --invert-captures option that will flip the meaning
of capturing and non-capturing parentheses.
This fixes a warning on GitHub actions workflows: "The `set-output`
command is deprecated and will be disabled soon. Please upgrade to using
Environment Files".
The new option allows one to use POSIX syntax for capturing groups
with leftmost-greedy disambiguation. This comibination has already
been supported by libre2c, so the changes needed to expose it in re2c
are minimal.
In the case of POSIX disambiguation (with --posix-captures option) it is
necessary to add structural tags around non-capturing groups, otherwise
disambiguation will be incorrect.
Commit f519385213 that allowed the use of
implicit non-capturing groups didn't properly add structural parentheses
around implicit capturing groups (which are needed if the group contains
nested capturing groups).
Commit 1edd25d31d that added explicit
non-capturing groups repeated the same error, only for the newly added
syntax `(? ...)`.
This resulted in `(? "a"* | ("aa"))*` being parsed incorrectly, as if
leftmost greedy disambiguation was used instead of POSIX disambiguation,
giving priority to the shorter first alternative `"a"` rather than the
longer match `"aa"` as POSIX requires.
This commit ensures correctness for both explicit and implicit
non-capturing groups by adding proper structural tags for the case of
AST sub-node `REF`. It is expected that it will bring back some of the
overhead on POSIX disambiguation reduced by the introduction of
non-capturing groups.
Related bugs: #438, #308.
Rename tests for implicit non-capturing groups so that they are named
similar to the tests for explicit non-capturing groups, and thus easier
to locate and compare.
This sub-ode is used for named definitions and non-capturing groups. The
latter do not have a name (previously a dummy name "<group>" was given
to such groups, but it turned out that the name is unused anyway).
This is useful in POSIX mode (with --posix-captures option) in order to
avoid unnecessary captures and reduce the amount of tags.
The new syntax for non-capturing groups is `(? ...)`.
Add tests for the new feature. Note that test results coincide with the
corresponding test for implicit groups (where the same group is not
parenthesized at all, but implicit grouping is achieved with the help
of a named definition).
This addresses bugs #438 "POSIX captures processing" and #308
"Overloading precedence marker (e.g. (pattern)) with 'posix' group
markers (also parenthesis) causes loss of full functionality".
Allow the use of named definitions that cause implicit grouping and
circumvent the absence of syntax for non-capturing groups. This is not
allowed by the POSIX standard, but this feature is useful in practice
and many regexp engines support it.
Update the tests that should no longer result into a compilation error.
This addresses bugs #438 "POSIX captures processing" and #308
"Overloading precedence marker (e.g. (pattern)) with 'posix' group
markers (also parenthesis) causes loss of full functionality".
This reverts commit 8df0ed6737.
Reason for revert: D backend is not ready yet: there are no examples
and tests, the work on it has stopped. In the future support for D and
other languages should be added in a generic way via language syntax
files, rather than modifying re2c codegen to support each new language.
Previosuly they were allocated using AST allocator, which made it
necessary to keep AST memory until the main codegen pass. Now it is
possible to release AST memory after parsing.
Rust does not support line directives. Previosuly re2c generated stubs
that were replaced with empty statements and later removed by a follow
up codegen pass. Now the stubs are not generated at all in Rust mode.
Labels cannot be generated during main codegen phase: codegen for state
dispatch may add more use cases for block start labels. These labels may
come from blocks either before or after state dispatch that uses them,
which makes it difficult to decide in advance (especially since state
dispatch itself cannot be expanded before the main codegen phase).
Therefore label generation has to be delayed. However, there is no need
for a separate codegen pass, as it can be done by the rendering pass.
All the functionality has moved to the main codegen pass, as it no
longer needs to be delayed. Second delayed pass still remains: it
expands labels and removes line directives for rust.
It no longer needs to be delayed until late codegen phase, and there is
no need for a special code kind: it can be generated during the main
codegen phase.
It no longer needs to be delayed until late codegen phase (all blocks
are parsed before the main codegen phase, and the list of conditions is
known by that time).
YYGETSTATE is only needed in the YYFILL call, it doesn't matter in case
YYLESSTHAN check fails and YYFILL is not called at all. The value of
state variable cannot "leak" into semantic action, as it is reset to -1
immediateluy before executing the action.
Previosuly YYGETSTATE was generated inside of the YYLESSTHAN check if
the end-of-input rule $ was used, but outside if it wasn't used. This
commit unifies the behavior.
Previously they were allocated during codegen, which implied that
state dispatch in storable state mode had to be generated in the late
codegen phase (because fill labels are used in transitions for state
dispatch).
Use blocks should generate state goto even if it has previously been
generated. However, that should not affect normal blocks (and previosuly
it did, as the global boolean flags was reset when seeing a use block).
This is so that we can gather whole-program information during the
first pass and use it during the second pass (obliviating the need for
late codegen).
The only advantage of `OutputFragment` compared to just `CodeList` was
that it additionally stored indentation level. However, it turns out
that just using top indent in all fragments produces the same output
(some of the fragments are line directives that always use zero indent).
Previously parsing and early codegen were intermixed: early codegen for
each block was done before parsing for the next block started. This is
problematic, because it creates the need for a late codegen phase that
expands whole-program constructs such as state goto or condition enum.
And then an extra pass is needed to generate labels. Used-label analysis
is spread throughout all phases: some of it is needed before early
codegen (so that we know which goto transitions can be elided and avoid
generating redundant loop/switch cases, and also to decide the need for
backup statement if there is a loop through initial state). But some of
the used-label analysis happens during the late codegen phase (expansion
of state goto may add new uses for start labels).
Separating parsing and codegen will let us simplify codegen, get rid of
the late phase and make used-label analysis and label enumeration more
clean and compact.
Previously this was done only if there were declarations of local
variables. However, the cases when braces can be omitted are rare, and
detecting them makes the code more complex for very little value.
Start label is only needed for the first DFA in a block: although it was
previously attributed to a DFA, it should be attributed to the whole
block. Autogenerated start label is used by state dispatch in storable
state mode. It is generated before condition dispatch, which is also
generated once per block.
This patch moves start label and condition dispatch generation out of
the per-DFA codegen loop and makes it explicit that they are generated
once per block.
- In addition to a C++ compiler, Windows builds require a build system
(CMake or bash). Clarify the dependencies to say this.
- Python 3 is required for tests, and tests should always be available,
and both CMake and Automake give a hard error for missing Python 3,
so just say Python 3 is required.
- Update docs rebuild dependencies to specify Python3 and Docutils,
since that is what you need to install to make rst2man available.
- Add an explanation of how to use the CMake presets since they're pretty
good.
- A few other minor corrections.
- Prequisite check for Python3 docutils module instead of rst2man.
- Use `split_man.py` instead of `split_man.sh`.
- Use `rst2txt.py` instead of `rst2man` + `gen_help.sh`.
- Explicitly use `$(PYTHON) <scriptname>` when calling python scripts.
Regenerate files using haskell script test/encodings/unicode_groups.hs
that is based on Data.Charset library (charser-0.3.9 package). The
script generates tests in test/unicode directory inplace and include
file unicode_categories.re.inc that needs to be copied to
include/unicode_categories.re.
Update test results. As there are substantial changes in the standard,
updates are also nontrivial. They have been verified with
test/__run_unicode_tests.sh.
The three examples for C, Rust and Go were werified with __run_all.sh
scripts in examples/{c,go,rust) respectively.
Pretty much every Python install includes Docutils, and Docutils is 95%
of what is needed to bootstrap docs on Windows. Fill in the rest.
- `docutils_txt.py` is a Docutils text-output Writer class with support for
writing a C file.
- `rst2man.py` is a wrapper for Docutils converting to MAN file. (Copied
from Docutils repo.)
- `rst2txt.py` is a wrapper for Docutils converting to TXT file. (Same
as rst2man.py but it uses docutils_txt as the Writer.)
- `split_man.py` is a Python implementation of `split_man.sh`.
- Change CMake build to use the new scripts.
Additional changes:
- CMakeLists: give error message if docutils module not found.
- ci-cmake: Give hint about where to find Python3 so that the private
install is preferred over the system install. Enable docs regen for
Windows.
Without the change on `gcc-13` build warns as:
./src/parse/input.h: In member function 're2c::loc_t re2c::Input::cur_loc() const':
./src/parse/input.h:139:20: warning: unused variable 'p' [-Wunused-variable]
139 | const uint8_t* p = cur;
| ^
The change turns `DCHECK(x)` into a statement that uses `x`.
Previously if start conditions were used, bitmap definitions were
generated at the start of each condition, as they all had the same name
(`yybm` or a user-defined one). This was problematic in storable state
mode, as transitions to `YYFILL` labels bypassed bitmap declaration
(state dispatch is always at the beginning of a block).
Now we use condition-specific names for bitmaps (`yybm` plus underscore
and condition name), so they can all be defined together before state
dispatch, as it happens in blocks that do not use start conditions.
Both pairs of files contain lexer-related declarations and definitions,
the headers are usually included together, and the input.{h,cc} files
are very small.
Do not use a struct, instead pass individual parameters. They are often
used in semantic actions, and addressing them as struct fields adds too
much noise in the source code.
It performs almost identical to GOR1 (sometimes slightly better or
worse), but it has no proven worst-case bound.
Apparently both the updated test and benchmark erroneously flipped GOR1
and GTOP (but the results were so close it didn't matter).
Before the change parsers relied on `YYSTYPE` global variable.
Not very handy for libre2c if it's expected to be used in reentrant
or multi-threaded contexts.
The change enables `%define api.pure full` and adjusts lexer calls
to explicitly pass through `YYSTYPE` around.
`%define api.pure full` is preferred to `%pure-parser` (deprecated in
2008, around Bison 2.3b). Should be safe to use.
Autotools-based and cmake-based build systems had slightly different flag order:
Makefile.am: $(AM_V_GEN)$(BISON) --warnings --output=$@ --defines=$(@:cc=h) $<
Let's make them identical.
New `bison --warnings` gives the following helpful diagnostic:
GEN src/parse/parser.cc
src/parse/parser.ypp:60.2: warning: empty rule without %empty [-Wempty-rule]
60 | : /* empty */
| ^
| %empty
%empty was introduced in bison-2.7.90 around 2013.
The change allows catching fresh bison diagnostics. Current example:
GEN src/parse/parser.cc
src/parse/parser.ypp:60.2: warning: empty rule without %empty [-Wempty-rule]
60 | : /* empty */
| ^
| %empty
The warning itself will be fixed in another change.
Run tests as:
bash -c "ulimit -s 256; make check -j$(nproc)"
I also verified that the tests fail on CI without commit
460412913771e9b7b9506bc623470195f6ccf21e (the last change that was
needed to fix#219).
This fixes bug #219. Tested as follows (all tests passed):
bash -c "ulimit -s 256; time ./run_tests.py"
Original test that revealed the problematic recursive functions:
bash -c "ulimit -s 256; echo run | gdb --args ./re2c overflow-1.re"
Save this information in each subtree and compute it bottom-up on AST
construction. This means a slight constant overhead on construction for
non-POSIX case (where this information is unused), but it eliminates a
worst-case quadratic algorithm for the POSIX case (where the property
was recomputed for each alternative/concatenation subexpression).
Note that tags were counted as captures by the old implementation, but
not by the new one, as POSIX regexps cannot have standalone tags.
Lowercase type names prefixed with `_t` are used for containers and
utility structures. Domain-specific types use camel case.
Also, rename REG_REGLESS to REG_MULTIPASS (but keep the old constant
definition for compatibility).
Previously we used signed characters and `re2c:yych_conversion = 1;` to
convert from char to uint8_t in the generated lexer code (the lexer
compares `yych` to various literals which may be outside of the signed
ASCII range, depending on the lexical grammar). As a consequence,
-Wold-style-cast was disabled.
Unfortunately now we have to disable -Wreserved-identifiers, as it warns
about user-defined literal suffixes like `_u8` that must start with an
underscore (the standard reserves non-underscore prefix for the future).
Use four basic allocator types (defined as different structs, so not
possible to mix with each other by accident):
- AST allocator (may be shared between blocks)
- IR allocator (intermediate representations for a single DFA)
- DFA allocator (codegen for all DFA in a block)
- output allocator
Even /W4 emits too much (informational warnings that we are not going to
fix), and /Wall is /W4 plus some warnings that are off by default. See
https://docs.microsoft.com/en-us/cpp/build/reference/compiler-option-warning-level
Also disable some selected warnings that add to much noise (C4068 that
complains about unused pragmas and C4996 that complains about functions
like `fopen`).
Previously some of the warnings have been disabled with pragmas, but not
all warnings can be fixed this way, as some of the macros are emitted
too early (before the pragmas).
The fix is to "use" the macros in trivial `#ifndef` clauses.
They were added in C99, so should be available in C++11.
For int64_t / uint64_t use PRId64 / PRIu64 macros from <cinttypes>.
Also fix some formatting issues.
Assertions are used in tests and elsewhere to verify assumptions that
must be correct, or else the program should abort. Compiling them away
in release mode is dangerous. For debug-mode assertions we use `DCHECK`.
A previous fix of error handling revealed memory leaks caused by not
destructing common data structures like tags, rules and charset that
live through different intermediate representations and move from one to
another (regexp -> TNFA -> determinization context -> TDFA -> ADFA).
Previously they were stored as references, and the last IR (ADFA) took
the responsibility for destroying them. This works with normal control
flow, but if there is an error in one of the previous IRs, the program
exits without constructing and (what matters here) destructing ADFA.
The fix is to use move semantics introduced in C++11: instead of storing
references, store values and move common data structures from one IR to
another. This way they are always owned and destroyed by the same IR
that causes the error.
This patch makes all tests pass under valgrind:
$ ./run_tests.py --valgrind
$ LD_LIBRARY_PATH=.libs \
G_SLICE=always-malloc \
G_DEBUG=gc-friendly \
GLIBCXX_FORCE_NEW=1 \
valgrind \
--track-origins=yes \
--trace-children=yes \
--num-callers=50 \
--track-fds=yes \
--leak-check=full \
--show-reachable=yes \
--malloc-fill=0xa1 \
--free-fill=0xa1 \
--suppressions="$HOME/misc/valgrind/all.supp" \
.libs/test_libre2c
A long time ago `std::valarray` was used because it did not require copy
semantics for rules. But this depends on the libstdc++ implementation
(if I remember correctly, macOS implementation required copy semantics).
Instead of doing `exit(1)` after printing an error message, return
error code to the calling function and propagate it all the way back to
`main`. Avoid boilerplate and bloating the code with checks by using
consistent error codes throughout the codebase and wrapping function
calls in a `CHECK_RET` macro. Use another macro `RET_FAIL` for errors.
This change has reduced the number of failing tests under valgrind from
131 to 32 (previously all the tests that report an error failed, because
`exit(1)` bypassed the destructors and didn't free memory).
In CMake this is done by adding autogenerated sources to an "object
library", which is not really a library, but just a collection of files
that can be used as a library in the sense that other targets can depend
on it. The dependency can be achieved by either adding it explicitly to
another library with something like `target_link_libraries`, or by
adding the object library to the list of source files.
For libre2c the latter was done slightly differently (and incorrectly):
instead of listing object libraries in the list of sources, normal
sources were added to another object library, and the object libraries
for autogenerated and normal sources were listed as sibling entries on
the sources list. So there was no dependency from the object library for
normal sources on the object library for autogenerated sources.
This caused build errors when a normal source file included an
autogenerated header, which was not generated yet. The fix is to not use
object library for normal sources.
For some allocations, move them to slab allocators. For others, replace
them with temporary buffers. Avoid allocating and freeing temporary
values with `new`/`delete`, which was previously done in order not to
store "fat" value on the parser stack (also since non-POD values cannot
be stored in the `yylval` union).
Previosuly CMake checked for Bison, and if it was present, then it
regenerated parsers and updated bootstrap files. Now it checks
RE2C_REBUILD_PARSERS, and if it is enabled, then CMake fails if it
cannot find Bison, regenerates parsers and updates bootstrap files.
This fixes `make distcheck`, which previously failed when trying to
update bootstrap parser (it makes the source directory read-only. Now it
no longer fails because rebuilding parsers is disabled by default.
Also, cleanup Makefile.am implementation for rebuilding lexers with
RE2C_FOR_BUILD: there is no need to check that the re2c executable
exists, as it is done by configure.ac.
This is the `astyle` command used for autoformatting:
$ # run in source directory
$ astyle --style=google \
--max-code-length=100 \
--align-pointer=type \
--align-reference=type \
--keep-one-line-statements \
--suffix=none \
--recursive '*'
Although it is an interesting algorithm, re2c benchmarks show that it
performs worse than TDFA(1): on complex regular expressions staDFA are
often much larger and slower than TDFA(1). See the benchmarks on the
official re2c website: https://re2c.org/benchmarks/benchmarks.html, or
the paper https://arxiv.org/abs/2206.01398 for details.
StaDFA implementation adds complexity to the most complex part of re2c
code -- determinization, and it takes some effort to maintain.
The algorithm is not fully documented and formalized: the original paper
by Chowdhury is incomplete, and the implementation is re2c had to use a
few fixes and modifications to make it work.
The benchmarks for staDFA now use re2c-3.0 to build.
This fixes bug #405 reported by Daniel Macks
"Insufficient version-check for python (run_tests.py failures)".
For CMake, prefer manual check to avoid raising required CMake version
to 3.19, in which find_package() learned to do version check.
Changes in json2pgfplot.py:
- add mandatory argument --variant <dfa_aot | dfa_jit | nfa>
- for DFA-AOT benchmark show time and size results on the same page
- for DFA-AOT/DFA-JIT add header, drop labels on all but the 1st column
- make bars colorful
- for rows that don't fit replace three dots with a thick arrow
- automatically make DFA-JIT and NFA benchmarks relative
Also, remove some unnecessary cases from DFA JIT benchmark (they are
very similar to other cases and add little extra value).
The check makes no sense (parser is regenerated if there is bison, and
it is not related to re2c bootstrapping itself). The bootstrapping
process was changed in commit 967540566b,
and the check was likely left by mistake.
Ragel:
- Patch `/bin/bash` to `/usr/bin/env bash` to build Colm on NixOS.
- Configure Colm with `--disable-manual` to avoid Asciidoc dependency.
Kleenex:
- Use `cabal build` instead of the removed `cabal v1-sandbox`.
CI runs occasionally fail to fetch apt packages due to unavailable
remote. Let's try to cache heavyweight .deb packages and see if it
helps.
.deb files should not normally change frequently.
Configurations with the same origin TNFA state also have identical
transition tags (by TDFA(1) construction those are the lookahead tags in
the origin configuration), therefore backlinks for such configurations
are identical and can be collapsed into one (provided that indices in
other backlink arrays that point into this one are suitably adjusted).
Determinization context was needed at `regexec` time previously, when
tag sequences on TDFA transitions were fetched from the tag tree. Now
that they are stored as arrays, tag tree is no longer needed and there
is no reason to keep determinization context alive after TDFA is
constructed.
re2c test are so fast that text output in terminal takes comparable time
on slower terminals. Let's disable it by default. Can still be brought
back with --verbose option.
Otherwise there is a warning when compiling in C90 mode:
warning: commas at the end of enumerator lists are a C99-specific
feature [-Wc99-extensions]
This fixes bug #395 "commas at the end of enumerator lists".
The build was broken by https://github.com/skvadrik/re2c/pull/376, which
added define of _POSIX_SOURCE to unbreak the build on Cygwin. The build
on FreeBSD is fixed by additionally defining _POSIX_C_SOURCE, which is
required by fdopen.
Instead of passing compiler and linker option -m32, call 32-bit compiler
binaries directly (prefixed with "i686-pc-linux-gnu-"). This allows one
to run the script on NixOS where it's hard to get a system-wide multilib
environment.
Historically this was allowed and re2c swapped the bounds. However, it
most likely indicates an error in user code and there is only a single
occurrence in the tests (and the test in an artificial one), so although
the change is backwards incompatible there is low chance of breaking
real-world code.
This fixes second test case in the bug #394 "Stack overflow due to
recursion in src/dfa/dead_rules.cc" (the actual fix is to limit DFA size
but the test also has counted repetition with swapped bounds).
Instead of failing with an out of memory exception or crashing with a
stack overflow, emit an error message and exit. This is a partial fix
for bug #394 "Stack overflow due to recursion in src/dfa/dead_rules.cc",
where re2c hit stack overflow on a counted repetition regexp with high
upper bound.
The patch adds the following limits:
1. the number of NFA states
2. NFA depth (maximum length of a non-looping path from start to end)
3. the number of DFA states
3. total DFA size (sum total of all NFA substates in all DFA states)
There are tests for the first three limits, but not for the DFA size as
all examples that trigger this behavior take a long time to finish (a
few seconds), which increases test run time almost twice.
Function-like API is difficult to use in Rust, because API primitives
cannot be defined as functions or hygienic macros (they do not accept
all the necessary parameters), and closures are problematic due to the
borrow checker (in this test `Cell` is used as a workaround).
The new example parses semantic versions, while the old one parsed
IPv4 addresses. The new example demonstrates a conditional tag, as the
last version part may be missing.
This is necessary because some parts of codegen conditionally generate
newlines depending on the line directives option. In Rust mode, unless
line directives are disabled codegen generates redundant newlines (e.g.
for the `stags:re2c` directive).
YYPEEK statement is redundant if the assigned `yych` value is
overwritten before it is read. This may happen if a state has a single
transition to a non-move state (so there is no need to match on `yych`).
Transitions to a move state is different because a move state does not
have its own YYPEEK (it relies on a previously read `yych` value).
This change affects nearly every test (with the exception of tests for
errors that abort before code generation, and tests that use loop/switch
approach that have been updated earlier).
Since the change affects so many tests, it is impossible to verify
manually, but skeleton tests pass (and they actually compile and run the
generated code).
Currently DFA states with unused labels still have their own index, but
this will change soon and there won't be any state number to pass to
YYDEBUG. This patch prepares the code by removing YYDEBUG statements for
states with unused labels.
One exception is the initial label: it may be unused (if there are no
transitions into the initial state), but still have a valid index. Keep
YYDEBUG statements for the initial label as they may be useful.
Labels should be generated before DFA transitions, because the pass that
generates transitions performs used label analysis: for every transition
its destination state label is marked as used. However label indices
should be assigned after used label analysis, otherwise indices may be
assigned to unused labels.
This is particularly important with the loop/switch approach, as label
indices map to `yystate` switch cases which should have consecutive
numbers.
This patch does not change label numbering yet, it only prepares the
code for the change (so no tests are updated).
As long as the following state has no incoming transitions (its label is
unused), generate it as a continuation of the current state. This avoids
looping through the `yystate` switch only to return to the next case.
The optimization is based on used label analysis for DFA state labels.
This is necessary in order to generate better code with loop/switch
(avoid looping through the `yystate` switch in cases when the only
transition from a DFA state goes to the state immediately following the
current one). For that we need to know which DFA states are used (have
incoming transitions) before actually generating them.
Previously used label analysis was needed only to avoid generating
unused labels with goto/label approach. Since the decision whether to
output a label or not is taken at a late stage in codegen, it was ok to
perform the analysis when generating the states. With the loop/switch
approach this is no longer sufficient.
This patch moves used label analysis to an earlier codegen pass. A few
special labels are still handled at a late phase (such as the start
labels of blocks, some of which are only marked as used after expanding
`getstate` directives). These few exceptions do not affect DFA state
labels that matter for the loop/switch approach.
With storable states `yystate` may have value -1, so it needs to have
signed type. Conditions have the same type as they are assigned to
`yystate`.
Also add a test for the combination of storable state and conditions.
In loop/switch mode case -1 should be the same as case 0. Previously
the generated code handled these two as different cases, but they should
be handled as one case with a range of values.
This commit allows using ranges not only for characters (the switch on
`yych`), but also for numbers (the switch on `yystate`). In storable
state mode -1 is appended to the range of the first case 0.
With loop/switch approach (which is implied for the Rust backend) case
-1 is the same as case 0. Normally this is handled by falling through
from case -1 to case 0, but since Rust doesn't have fallthrough it is
necessary to set `yystate` to 0 and loop back to the head of the switch.
Semantic actions that start with `:=` are not enclosed in curly braces,
so re2c uses indentation to find the end of user-defined code: as long
as a line has nonzero indentation it is parsed as a continuation of
semantic action. In order to do that re2c needs to analyze if a newline
is followed by a whitespace. Previously this analysis was missing in the
case of newlines in single-line comments.
As Krishna noticed in bug #392 "Cannot compile when user c string has a
special pattern", `%{` in the middle of a string makes re2c think that
this is a start of a re2c block. The user has to braek the string in two
in order to separate `%` from `{`.
The problem is exacerbated by the fact that `%{` does not require -F,
--flex-support option. It affects all re2c programs.
In general re2c cannot be sure if `%{` appears in the middle of a
string, a comment, a macro defintion or some other lexical context where
it is a part of a larger lexeme and not a standalone block start marker.
This is because re2c doesn't parse user-defined code in-between re2c
blocks (doing that would require implementing multiple language
frontends in re2c, which is out of the question).
This commit mitigates the problem in #392 by restricting lexical context
wheer `%{` is recognized as a block start: the marker must be on a new
line, optionally preceded by whitespace characters. This does not cover
all the cases and it may break some re2c programs that used block start
`%{` in other lexical contexts, but it is a reasonable compromise
between usability and backards compatibility.
Reorder steps so that 'Get build dir' step is executd before build and
test steps. Otherwise if the build or test steps fail, subsequent steps
are skipped and the path for uploading artifacts is not set properly.
For example, --type-header should be just --header: it used to be for
`/*!types:re2c*/` directives initially, but after the addition of
`/*!header:re2c...*/` it may contain anything.
Also, --input is confusing and should be --api, as we refer to this as
"generic API" or "default API" in the documentation.
Configurations have evolved over time, and some of the names are quite
illogical. For example `re2c:flags:tags` should be just `re2c:tags`,
and `re2c:flags:unicode` should be `re2c:encoding:utf-32`. While we
cannot remove the old names (that would break existing programs), we can
still provide simpler and more logical aliases.
Take that opportunity to clean up configuration lexer a bit.
This allows to suppress generation of `unsafe { ... }` wrapper around
YYPEEK (for the Rust backend). Normally YYPEEK should avoid bounds
checks, as the lexer already does them (and goes to great lengths in
order to minimize the number of checks). However it may be useful to
enable bounds checks in YYPEEK for debug purposes, or just extra safety.
In that case the new option/configuration is needed to avoid warnings
emitted by the Rust compiler about redundant `unsafe`.
Also extend the script that compiles and runs examples for Rust to run
each test twice: first with the unsafe YYPEEK definition, and second
time after patching it to avoid `unsafe`. This allows to check that the
examples do not have trivial memory errors.
Normally sematic are surrounded by line directives, so they occupy
multiple lines and should be wrapped in curly braces. In Rust there are
no line directives, but the codegen was still confused by empty stubs
and rendered semantic actions as multiline code blocks.
In order to fix this, it was necessary to separate removal of empty
code blocks into a separate pass (previously it was a part of `combine`,
but `combine` is too specialized and does not remove all empty blocks).
It is necessary to initialize `yych` (even though the initial value will
never be used) because in Rust a potentially uninitialized variable is a
compilation error. However, in cases when the compiler is able to see
that the variable is overwritten on all code paths, it instead emits a
warning about unused assignments.
Suppress the warning with `#[allow(unused_assignments)]`.
Do indent cases with the default indent string. Do not auto-align cases
that fit on one line with tabs, use a single space before the case body.
Tab alignment didn't work very well and required special handling (for
example with --case-ranges it had to be turned off, because case length
varies more than one tab).
Over a thousand of tests were updated, but all the changes are trivial.
When lexing code blocks we need to lex string literals because they may
contain closing brace symbol that would otherwise be erroneously lexed
as a real closing brace.
However, single quote in Rust may be either the beginning of a char
literal as in '\u{1F600}', or a standalone one as in 'label. In the
latter case trying to lex a generic string literal will consume a
fragment of the file until the next single quote (if any) and result in
either a spurios parse error, or incorrect generated code. Therefore in
Rust we try to lex a char literal, or else consume the quote.
If the user has an outer loop that goes via YYGETSTATE switch, they
would expect the next iteration of the loop to start from the intiial
state rather than from some YYFILL state left after the last YYFILL
invocation. This especially matters when there is no explicit
`getstate:re2c` and no user-defined code between the YYGETSTATE switch
and the lexer entry (which is always the case in loop/switch mode).
A lot of (54) tests have been updated (they now have additional
YYSETSTATE invocations). One of them (conditions/condition_08_cbif.c)
previously resulted in a crashing program on input file "??(??)", and
now due to the addition of YYSETSTATE it doesn't crash.
Also, use state -1 rather than 0 as the default one in loop/switch mode.
Although it is safe to use 0 (cases always start from zero), using -1 is
unifies it with goto/label mode, where 0 cannot be used (start label
does not have to be `yy0`). And -1 is always safe for both modes, as
the YYGETSTATE switch always has either a default case or an explicit -1
case that goes to the initial state.
Options are managed by `Opt`, and warnings are managed by `Msg`. Both
are created in `main` and have the lifetime of the whole program, so it
is not difficult to connect them.
Previously trying to run rst2man.py with an output file in a nonexistent
directory resulted in the following build error:
$ cmake --preset=linux-gcc-debug-ootree-skeleton-full
$ cmake --build --preset=linux-gcc-debug-ootree-skeleton-full
...
[ 1%] Generating src/msg/help.cc
Unable to open destination file for writing:
OutputError: [Errno 2] No such file or directory:
'[...]/linux-gcc-debug-ootree-skeleton-full/src/msg/help.cc.1'
Otherwise variable definitions from "fast" configuration take precedence
over the "full" configuration, resulting in disabling some important
configrations, such as Go language support.
For example, if "fast" is inherited before "full" we have the following
incorrect configrations that sets RE2C_BUILD_RE2GO:BOOL="FALSE":
$ cmake -S .. --preset=linux-gcc-release-ootree-skeleton-full
Preset CMake variables:
CMAKE_BUILD_TYPE="Release"
CMAKE_CXX_COMPILER="g++"
CMAKE_C_COMPILER="gcc"
CMAKE_INSTALL_PREFIX:PATH="[CUT]/re2c/install"
RE2C_BUILD_LIBS:BOOL="TRUE"
RE2C_BUILD_RE2GO:BOOL="FALSE"
RE2C_FOR_BUILD="[CUT]/re2c/install/bin/re2c"
RE2C_REBUILD_DOCS:BOOL="TRUE"
RE2C_REBUILD_LEXERS:BOOL="TRUE"
But if "full" is inherited after "fast" we have the following (correct):
$ cmake -S .. --preset=linux-gcc-release-ootree-skeleton-full
Preset CMake variables:
CMAKE_BUILD_TYPE="Release"
CMAKE_CXX_COMPILER="g++"
CMAKE_C_COMPILER="gcc"
CMAKE_INSTALL_PREFIX:PATH="[CUT]/re2c/install-2"
RE2C_BUILD_LIBS:BOOL="TRUE"
RE2C_BUILD_RE2GO:BOOL="TRUE"
RE2C_FOR_BUILD="[CUT]/re2c/install/bin/re2c"
RE2C_REBUILD_DOCS:BOOL="TRUE"
RE2C_REBUILD_LEXERS:BOOL="TRUE"
Merge all conditions in one loop/switch where DFA for each condition has
its own set of cases. Use `yystate` variable to switch between
conditions in the same way as between DFA states.
The => rules set user-defined condition variable with YYSETCONDITION.
The :=> rules set `yystate` and continue to the head of the loop/switch.
Header file may contain blocks with regular expressions and rules. They
should be handled as well as the blocks in the output file.
This patch fixed a failing skeleton test.
Normally with the goto/label approach condition enumeration uses
implicit consecutive indices starting from zero. With the loop/switch
approach it is more convenient to represent conditions with their
initial DFA state index. This allows one to assign condition numbers to
the state variable `yystate` and get rid of the entry condition switch.
The `maxfill:re2c` and `maxnmatch:re2c` directives may specify an
optional list of block names. If the list is specified, the directives
compute maximum value over the specified blocks, otherwise they compute
global maximum across all blocks in the program.
Previously the global maximum was prepared in advance even if the
directives were not used. This patch delays the computation of global
YYMAXFILL / YYMAXNMATCH until they are needed, as is already done in
the case when the directive specifies a list of blocks.
The `stags:re2c` and `mtags:re2c` directives may specify an optional
list of block names. If the list is specified, the directives gather
condition nems from the specified blocks, otherwise they use a global
set of tag names gathered across all blocks.
Previously the global set was prepared in advance even if the directives
were not used, but a set for specific block list was constructed only if
needed. This patch delays global set construction until it is needed and
unifies it with the case of a specific block list.
The rationale is, if the user specified an explicit `types:re2c`
directive, they probably do not want the autogenerated enum as well. If
for some reason they need multiple enum definitions (either in the
output file or in the header file), they can use multiple `types:re2c`
directives.
This fixes a test for condition enum: previously re2c emitted an error
because one of the `types:re2c` directives listed block name for a block
defined in another file, and re2c failed to find it. Now it finds it.
Some compilers or language standards allow it, but generally it's more
likely to indicate an error in user code.
Fix a few tests that generated empty condition enum by mistake.
Replace a bunch of simple and similar tests with a few more
comprehensive ones that cover recent re2c features like named blocks and
block lists in the `types:re2c` directive, in addition to older features
such as the --type-header option.
One of the new tests reveals an error in re2c: it fails to find
conditions in header. The error will be fixed in a follow-up commit.
clang with -Weverything detects these as:
./src/util/free_list.h:9:16:
warning: identifier '_Ty' is reserved
because it starts with '_' followed by a capital letter [-Wreserved-identifier]
template<class _Ty>
^
Detected by clang's-Weverything:
./src/regexp/re.h:17:1: warning: struct 'Msg' was previously declared as a class;
this is valid, but may result in linker errors under the Microsoft C++ ABI [-Wmismatched-tags]
struct Msg;
^
With the loop/switch approach there is no `goto` statement, so it is
impossible to jump from a `getstate:re2c` block to an YYFILL label,
bypassing all the user-defined code in the middle (as it normally
happens with the goto/label approach).
Therefore `getstate:re2c` is forbidden with --loop-switch, and the
initial switch on storable state is merged into the `yystate` switch:
- case -1 falls through to case 0 (the initial state)
- transitions to YYFILL states are moved to cases that occupy
contiguous numbers after the last regular DFA state
In order to make it possible to wrap the generated lexer in a
user-defined loop, an `YYSETSTATE(1)` statement is emitted in each final
state before the user-defined semantic action: if the action jumps to
the head of the outer loop, the next iteration will start from state -1,
as the user might expect. Note that `YYSETSTATE(1)` is not needed with
goto/label approach, because one can jump directly to the start label.
Having configuration for the --loop-switch option is only useful if one
needs to enable loop/switch approach on a per-block basis. Mixing the
two approaches (goto/label and loop/switch) would mean that some of the
blocks have start labels and can use cross-block transitions, while
others cannot. This would create confusion for whole-program directives
like `getstate:re2c` that cover multiple blocks.
With the loop/switch approach `getstate:re2c` should be forbidden, as it
requires a cross-block transition from the state switch to the lexer
entry, which is unsupported in the absence of the `goto` statement.
Normally with the goto/label approach re2c uses a program-wide state
enumeration that starts from zero and increases with each block. Since
each state has a label, global enumeration allows one to have multiple
blocks in the same function without the risk of label collision.
Moreover, different blocks can be connected with cross-block `goto`
transitions, so all state labels should have different numbers.
The situation is different with the loop/switch approach: there are no
labels and no cross-block `goto` transtions; instead every state has a
switch on the `yystate` variable. Since different block have different
switches, it makes sense to start state enumeration from zero in each
block (so that switch cases start from zero, and the zeroth case
corresponds to the start state of every block).
The added tests are copy-pasted from examples, but the generated code is
different: it uses loop/switch constructs instead of goto/label. All
tests except for conditions and storable state are valid programs that
can be compiled and run without errors.
With this option, instead of the usual goto/label approach DFA is
wrapped in a loop with a `switch` statement on `yystate` variable. Each
DFA state is an individual `case`. Transitions between states are done
by setting the `yystate` variable to the numeric index of the target
state and executing `continue` that jumps to the head of the loop and
goes via the `switch` again. This is less efficient than direct jumps
with the goto/label approach, but it is the only way we can implement a
DFA for languages that do not have `goto` statement (such as Rust).
YYFILL labels are problematic for the loop/switch approach, because they
are normally in the middle of a DFA state (after the YYSKIP operation,
but before YYPEEK and transitions to other states). To fix this, we use
the --eager-skip option that moves YYSKIP operations from DFA states to
transitions.
The initial value of `yystate` is the numeric index (label) of the first
DFA state. It is _not_ the intiial label, because initial label does not
have a state (so there is no `case` in the `switch` for it), and it is
only needed to bypass YYSKIP operation in the first DFA state (which is
not there anyway because of the --eager-skip option).
Current limitations:
- Conditions are not supported. We need a per-condition `yystate`, and
ideally transitions between conditions should also use a loop/switch
approach instead of goto/label (it will be required for Rust, but it
is optional for C and Go).
- Storable state is not supported. We need to rewrite user-defined
code to use the same state variable as `yystate` and the one in
`YYGETSTATE`/`YYSETSTATE`, as well as to put `getstate` block
immediately before DFA entry (if there is any code between state
dispatch and DFA entry, it needs to be bypassed when re-entering
the lexer after YYFILL, which is easily done with goto/label
approach in C and Go, but not in Rust that has no `goto`).
Test results with the --loop-switch option enabled by default:
- Skeleton tests pass (320 soft errors).
- Examples for C and Go compile and run without errors, except for
conditions, storable state and cases when user-defined code needs to
be rewritten (e.g. when it uses an outer loop and `continue`/`break`
statements end up referring to the wrong loop).
The following has been added:
- option --loop-switch
- configuration `re2c:flags:loop-switch`
- configuration `re2c:variable:yystate`
Currently they are no-ops.
This is necessary to support the future --loop-switch option.
With --loop-switch we cannot have labels (goto targets) in the middle of
a state. This affects YYFILL labels which normally go after YYSKIP but
before YYPEEK. The --eager-skip option resolves the problem by moving
YYSKIP statements from states to transitions. The plan is to enforce it
automatically if --loop-switch is enabled, but since --loop-switch will
be a mutable option, --eager-skip also needs to be mutable.
The previous commit fixed bug #386 "#line directives around !include
contents" by reading every include directive twice: for the first time
on entry (when the file is included), and for the second time on exit
(when the file has been processed).
However that commit used a crude criterion to distinguish the first and
the second times: it compared the include file path (specified in the
include directive) to the path of the file on top of the stack. This
doesn't work for nested include files that have the same relative path.
This patch adds a test and fixes the problem by using a different
criterion: it checks if the parent file is topmost after popping all
finished files from stack (the comments in src/parse/scanner.cc explain
why this works in detail).
This fixes bug #386 "#line directives around !include contents"
reported by Cyan Ogilvie.
It is easy to generate line directive at the beginning of the include
file, but much more difficult to find the end of it: file contents are
just a stream of bytes in the input buffer, and the continuation of the
parent file follows immediately after the end of the inlude file.
Therefore we use a hack and "unread" the include directive in the parent
file before filling the buffer with the contents of the include file (as
if the include directive appeared after the contens of include file).
This makes the lexer read the directive for the second time; at this
point we know it's the same include directive because the include file
is already on top of the file stack. So we ignore the directive and
simply generate line info.
The current implementation has a problem with nested includes that have
identical relative file paths. This will be fixed in a follow-up patch
by looking at the resolved paths when comparing files.
This fixes bug #385 "mtag/stag automatic variable generation collision"
reported by Cyan Ogilvie.
Previously s-tag and m-tag variables used non-overlapping numbers
within a single DFA. With start conditions this is insufficient, as
variables in different conditions (DFAs) may have identical numbers.
If both tag variables are single-valued, or if they are both
multi-valued, having one variable is fine: there are no transitions to
or from the middle of DFA, so tag variables in different conditions are
never used at the same time.
However if one of the variables is single-valued and the other one
multi-valued (or vice versa), they cannot share the same tag variable.
To avoid collision in such cases, this patch gives m-tag variables an
extram "m" affix that makes them different from any s-tag variable.
S-tag variables do not get an "s" affix primarily to avoid updating over
600 tests. Tag variables are internal to re2c and the user should not
use them directly, so this change does not break user code.
Tags in trailing context may be fixed on the rightmost cursor position.
Their values must be updated before cursor is reset to the position of
the trailing context, otherwise the values would be incorrect.
Previously this was not so, which resulted in incorrect tag values.
This fixes bug #380 "Tags incorrectly set with lookahead" reported by
Terry Sanders .
Improve grammar, make some small style improvements, and keep the
lines consistently under 80 columns. Also update boostrap docs for
resulting changes.
Changes:
- use 'sh' instead of 'bash' as interpreter
- split build into stages 1 and 2 and rebuild lexers at stage 2
- double-quote shell variables to make shellcheck happy
Previously we used negative numbers for register versions that were
allocated for transitions on negative tags. This is pointless, because
state mapping may biject positive versions to negative ones or vice
versa, erasing the distinction. And it wasn't used anyway.
Exclude diagonal pairs from interference table. This does not affect
subsequent register allocation, because the algorithm that partitions
registers into non-interfering classes does not check is a register
interferes with itself. So the fix does not affect the generated code,
except for one test that draws the interference table.
Commit ec1dc2b3ec incorrectly compared
Wine path against `None` to determine if the tests run under Wine. That
doesn't work as expected because the default value is empty string, not
`None`, so the checks always succeeded and as a result the script did
unnecessary work (patched newlines in the generated output). That made
it a few times slower.
Since 3.19.8 CMake looks in the top source directory of a project for files
named CMakePresets.json and CMakeUserPresets.json. Neither file is required
to be present, but if the user tries to list or use presets, at least one of
the files must exist. If both files are present, they will effectively be
merged by reading the CMakePresets.json file first and then the
CMakeUserPresets.json file.
CMakePresets.json can be used to define presets for things like continuous
integration jobs or cross-compilation setups using toolchains provided as
part of the project. In general this feature allows us to simplify CI
configuration phase by taking out all the logic into presets-file and helps
us to make reproducible builds. From now on, you no longer need to carefully
read and copy all the flags locally in order to reproduce CI configuration or
build. Simple use the same one line command as at CI.
As a bonus, users of modern CMake can use a cross-platform and portable way
instead of build/__cmake*.sh scripts.
To see the list of available presets use:
cmake --list-presets
To configure re2c use:
cmake --preset=<NAME>
To build re2c use:
cmake --build --preset=<NAME>
To test re2c use:
cmake --build --preset=<NAME> --target tests
References:
- https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html
- https://gitlab.kitware.com/cmake/cmake/-/issues/21164
- https://cmake.org/cmake/help/latest/guide/user-interaction/index.html#presets
We need to copy multiple directories (test/ and examples/) into one
(temporary test directory). `shutil.copytree` is problematic for two
reasons:
1) it requires `dirs_exist_ok` parameter which was added only in
python-3.8
2) `make distcheck` fails when trying to copy the second directory
because it makes source files read-only
This may happen if a name is defined in global scope and inherited by a
`rules:re2c` block that is followed by another block that uses the rules
block with a `!use` directive:
/*!re2c
x = [x];
*/
/*!rules:re2c:R
// inherits 'x' definition from global scope
y = [y];
*/
/*!re2c
// inherits 'x' definition from global scope
// imports the same 'x' definition from 'R'
!use:R;
*/
At the time when symtab from the rules block is merged with the
inherited one, there is a collision on the global names that are present
in both symtabs. It's harmless.
Global options `-i` and `--no-debug-info` can be used instead.
Configurations allow to enable/disable `#line` directives on a more
fine-grained basis (for selected parts of the file), but there's not
much point in doing so and there are no known real-world use cases.
Having them as global options is convenient, because a global option
can be used in parser (unlike a configuration, which can change and for
which the final value is unknown until the whole block is parsed).
The few places are:
- condition enum (this one should always use whole-program options)
- state switch (this one should either use whole-program options, or
block-local options if this is a `use:re2c` block
Add comments.
Now it is possible to customize these directives:
/*!max:re2c
format = "static const int YYMAXFILL = @@";
*/
/*!maxnmatch:re2c
format = "static const int YYMAXNMATCH = @@";
*/
This commit breaks backward compatibilty, because it forbids arbitrary
garbage at the end of `max:re2c` directive. It was historically allowed
and even used in one test, but it doesn't add any practical value and
it's getting in the way of adding 'format' option (or any other option).
The following code that was previously accepted now triggers an error:
/*!max:re2c
... arbitrary g4rbage ...
*/
For example, emit an error on `/*!getstate:re2c:x:y:x */`. Previously
this resulted in code that would not compile (due to duplicate cases in
the generated state switch), but an explicit error message from re2c
is more user-friendly (and it covers all block lists, not just the case
of `getstate:re2c` directive).
Previously autogenerated names were identified by block line, but it may
be non-unique (it is possible to have multiple blocks on the same line,
included files may define blocks at the same line, and line directives
may reset line number).
Emit error messages instead of failing debug checks when:
- there are no blocks with code in the global state switch
- first named block on the list is empty (has no start label)
- a named block is missing (reworked error message)
Also, rewrite the code in a way that avoids looping over the block list
twice. Previosuly the first loop was used to to find a start label, and
the second loop was to generate switch cases. Now we do both in the same
loop and prepend the default case with the start label to other cases.
The first pass expands everything except labels (they have to wait until
the second pass because the first pass may generate some additional uses
for previously unused labels). The second pass expands labels.
State switch is generated differenly for `use:re2c` blocks and for all
other blocks.
For use blocks there is a block-local switch that is automatically
generated at the beginning of the block. Previously use blocks had a
separate enumeration of YYFILL states starting from zero, and they used
a block-specific prefix in YYFILL state labels. Now use blocks
participate in the global YYFILL state enumeration, and their labels no
longer require a block-specific prefix (because label numbers are unique
across the whole file).
For all other blocks there is a global state switch that is generated
either at the point where `getstate:re2c` directive is specified by the
user, or (if there are no user-defined `getstate:re2c` directives) at
the beginning of the first non-use block that has a start label. Code
generation of the global state switch hasn't changed.
Now it has become possible to generate a state switch for the subset of
blocks specified in a `getstate:re2c` directive. The code is still
slightly incorrect, as the start label for the first block on the list
is not always generated (only if this block is the first non-use block
in the file). This happens because the generation of labels happens too
early, before the generation of the state switch that marks the start
label as used.
The tests that have use blocks have been updated (only the labels and
the autogenerated YYFILL state numbers have changed).
As a result of this patch the cases in a state switch no longer have to
start from zero and be sequential: 0, 1, 2,... . Now they can start with
any number and consist of block-local sequential ranges (the numbers are
unique and non-decreasing). State numbering is an internal re2c detail,
so it should not affect any users. State -1 (or any other negative
number) is handled in the same way as before (as a special case).
To facilitate the generation of state switch based on block names, the
code has been rewritten to find blocks by their name. Each block now
stores start and end YYFILL indices and a link to the start label. This
information is used in the generation of state switch instead of an
implicit assumption that indices start from zero and occupy sequential
numbers. Use blocks are also handled via the new name-based mechanism:
they use a special autogenerated block name which is passed to the
autogenerated block-local state switch.
Start label is not always used, only in cases when there is a transition
to the start state or if it is explicitly enforced by the user.
Previously re2c did not include start label in the global label
enumeration unless it was actually generated. This required knowing in
advance if the label was used or not. Currently we do have such
knowledge, because the default transition in the state switch goes to
the first block in the file (except for reuse blocks, which have
individual state switch).
But this is problematic for the new `getstate:re2c` that is allowed to
specify a list of block names, because the default transition now needs
to go to the first block on the list, not the first block in the file.
This commit adds start label to the global label enumeration, even if it
is not generated. As a result, label numbers in some tests have changed.
Don't do anything with the specified names yet, just parse them and save
for future use. Follow-up changes will handle block names for each type
of block or directive.
The following block types now have an optional name:
/*!re2c[:name] ... */
/*!local:re2c[:name] ... */
The following directives now have an optional list of colon-separated
block names:
/*!max:re2c[:name1[:name2...]] ... */
/*!maxnmatch:re2c[:name1[:name2...]] ... */
/*!types:re2c[:name1[:name2...]] ... */
/*!getstate:re2c[:name1[:name2...]] ... */
/*!stags:re2c[:name1[:name2...]] ... */
/*!mtags:re2c[:name1[:name2...]] ... */
The new type of blocks differs from the usual `/*!re2c ... */` blocks
in that the names and configurations it defines are local: they have the
scope of a block and aren't visible outside of it in the global scope.
This allows one to reuse the same names multiple times in different
blocks, and to avoid "polluting" the global scope.
Symbol table (a.k.a. the mapping of named definitions to regexps) should
obey the same scoping rules as configurations. Before this commit it was
broken: there was one global symbol table that accumulated all named
definitions along the way.
Run tests to the end and produce results for all test cases that are
missing reference results. This is convenient when adding new tests, as
it allows one to drop *.re files in the test directory, run the tests
(failure is expected), manually verify the generated results and copy
them to the test directory.
Add escaping in some previously forgotten cases.
Avoid double escaping (previously this happened for some include files
for which file path was constructed as a concatenaton of an already
escaped directory part and the yet unescaped file path in the include
directive).
This only affects platforms that use backslashes in file paths (such as
Windows).
This fixes `make wtests` (--wine option to run_tests.py).
A few things were broken:
- re2c binary should have '.exe' file extension
- `re2c --version` on wine is rather slow, so drop the 2s timeout
Useful on systems with multiple python implementations.
While at it allow both 'python3' (preferred) and 'python' spellings.
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
Header files may be created in the same directory, so that one header
overwrites the other. It also causes races in run_tests.py when it tries
to clean up intermediate files created by each test.
The previous version of this function has multiple responsibilities and
actually had knowledge for which it was not intended.
These changes are about "S" in SOLID.
Previously the script divided all tests into chunks of equal size, which
is suboptimal as different tests take different time, and often large
tests end up in the same chunk. This adds imbalance and makes the test
runner slower than it could be with chunk size 1.
Also, use a more pythonic way to gather results from threads. The old
bash script did that via per-thread log files because it is hard to do
it otherwise in bash. But in python one can simply return results from
each thread, avoiding the overhead on file I/O.
The removed test was a trivial modification of another test (with -i
option turned off). It was useful long ago, but not anymore as we have
many tests with or without -i option.
The removed test was problematic because it created a header file with
the same name as the other test, which revealed a race in test runner
that removed the generated header before the other file processed it.
This got uncovered while debugging run_test.py, when I changed
multiprocessing.Pool.map() to multiprocessing.Pool.imap(). The race
happened only once in a few runs.
This pull request introduces a completely rewritten test runner
in Python.
The main goals of this change:
- Provide a OS-independent way to run tests (at least using CMake)
- Getting rid of dependence on Unix utilities in tests
(grep, cat, tr, etc.)
- Flexibility
- Ease of maintenance
This reverts commits:
- ec9eecd077
- f710983c23
By default if some tests fail, the corresponding temporary files are
kept (for investigation). For the passing tests temporary files are
removed. But in some cases it is convenient to keep temporary files for
the passing tests as well.
REgen procduces input data with variable RE size within each group, so
in order to generate truthful plot it is necessary to take that variance
into account.
For JS plot, compute the x-coordinate as the average RE size withing
each group (also average the y-coordinate within each group, as before).
For PGF plot, don't average RE size, instead use individual RE size as
the x-coordinate. Do a few transformations on the benchmark results:
remove outliers using centered moving median, then smooth the curve
using centered moving average.
Although leftmost greedy results are different from POSIX ones, the
relative difference of register/registerless variants is similar, so
it suffices to keep only the POSIX variants.
ReTdfa is a Java library that contains a family of algorithms for
regular expression parsing and submatch extraction based on TNFA and
TDFA. It is also a benchmark for various algorithms, including some
that are based on libre2c.
This commit also adds Java bindings to libre2c.
This should be done by the user, and the library may only store a
pointer to the allocated memory in the regex_t structure (this is
convenient for the foreign bindings).
The regex.h header is included by the users of the library. Including
other local headers in it is problematic, because the user then has to
specify additional include search paths (and do so by trial and error
rather than by some general rule).
Strictly speaking, this commit breaks C++98 compatibility as <stdint.h>
is non-standard in C++98. But in practice this is unlikely to cause any
problem, as libre2c is a recently added experimental library which
hardly has any users that do not have a C++11 toolchain.
- Fail with an error in some confusing cases like
`/*!include:re2c x.re */` that are likely to be unintentional syntax
errors (double quotes missing around the filename), but previously did
not fail and were treated like comments (no code was generated).
- Improve error messages for syntax errors such as a bad block name.
AFL tests had to be updated, because the original tests that crashed the
lexer now fail earlier due to an unrelated reason, so a space character
had to be added in some tests to make them hit the same error as before.
Within one block special rules *, $ and <!> cannot be redefined (it is
an error). However, when reusing other blocks it might be convenient to
override inherited special rules with local ones. This commit separates
inherited and local special rules by storing them in separate containers
that are merged at the end.
Also, this commit merges functions that validite and "normalize"
specifications into one function. This has to be done because some of
the validation checks must be performed before merging special rules,
and some must be done after that.
Now that `/*!rules:re2c ... */` and `/*!use:re2c ... */` blocks can be
mixed freely with the ordinary `/*!re2c ... */` blocks, there is no
reason not to allow reuse mode by default.
Also, fix memory leak in the parser (found by LeakSanitizer): don't
forget to free std::string that was allocated to hold block name in the
inline `!use` directive.
Historically re2c allows arbitrary text before the closing comment of
the directives such as `/*!ignore:re2c ... */` and `/*!max:re2c ... */`.
It only makes sense for the ignore directive (which is just a multiline
comment), but not for other directives. This commit disallows garbage at
the end of all directives, except `/*!ignore:re2c ... */` and (for the
reasons of backward compatibility) `/*!max:re2c ... */`.
The inline `!use:<block-name>;` directive can be used in the middle of
a re2c block. It allows one to merge a block named <block-name> into the
currect block: rules of the used block are added to the current set of
rules, and configurations are added to the current options. Overlapping
rules and configuraitons have the usual priority, as if the used block
was literally inserted in place of the `!use` directive (the first rule
wins, and the last configuration wins).
Named blocks allow one to reuse any rules block, not only the last one.
The syntax named blocks extends existing syntax for unnamed blocks:
/*!rules:re2c:<name>
...
*/
An optional name is specified after a semicolon. The block can then be
used by specifying its name in a use block:
/*!use:re2c:<name>
...
*/
This commit addresses issue #351 "include directive limitations",
reported by Henry Harvey.
An include directive in the middle of a block is useful in cases when
the include file contains additional rules overlapping with the rules in
the including file. In such cases the usual directive won't work, as it
would crate a new re2c block with its own set of rules (rather than
merge the included rules into the current set).
The inline directive syntax is slightly different from the usual re2c
directive syntax: comment markers '/*' and '*/' are dropped to avoid
nested multiline (which are unsupported in C/C++), and the ':re2c' part
is also dropped because it is redundant (the inline directive is already
inside of a re2c block).
The new option automatically adds tags for all subexpressions, as if
they were wrapped in capturing parentheses. This is only useful for
experimental purposes (to measure parsing speed in the case of fully
tagged RE and study performace degradation of different algorithms as
the density of submatch increases).
Unwinding tag history from its trie representation is rather slow. This
can be done in advance at regcomp() time, and the resulting t-string
fragments can be stored in backlinks (this is what regtstring() already
uses). For offset construction t-string fragments also need to include
negative tags.
This CL adds a new flag REG_TSTRING and a new API function regtstring()
that returns a t-string (a flattened representation a parse tree where
characters of the input string are interleaved with tags).
The goal of this algorithm is to construct parse results in the cheapest
possible way. There is no standard representation of a parse tree, and
any nontrivial representation would likely incur a significant overhead
(the algorithm would spend most of the time constructing parse results
rather doing the actual parsing).
The t-string representation requires very little effort to construct,
provided that the t-string fragments corresponding to the tagged paths
in the epsilon-closure are prepared in advance at regcomp() time. The
fragments are stored as arrays of tags, so that they can be easily
copied to the resulting t-string. Importantly, this does not require
unwinding of tag history stored in a trie form, which is rather slow.
A t-string is represented as a sequence of 16-bit values that are either
ASCII characters (that occupy the lower byte, leaving the higher byte as
zero) or tags (that are numeric value shifted to the upper half of the
16-bit range).
16-bit size for t-string characters is chosen to align with the
experimental Java implementation by Angelo Borsotti.
T-string allocation is done on the library side, and the t-string is
stored in the regexp structure. The user gets an immutable view of the
resulting t-string and is not expected to free memory after calling
regtstring(). Moreover, the library reuses the same memory on each
regtstring() call and reallocates it only if the new t-string does not
fit into the old memory area.
All in all, the new regtstring() API is primarily intended for
benchmarking the speed of the parsing algorithms, and not for "official"
use.
This patch adds a new flag REG_SUBHIST and new API functions regparse()
and regfreesub().
regparse() returns parse results as an array of nmatch size, where each
element is an array of offset pairs for the correspoinding capturing
group. If a group has matched repeatedly at different parts of the input
string, then its array will contain multiple offset pairs; otherwise, a
single pair. This differs from the standard POSIX API, where onl the
last offset pair for each group is returned.
The allocation of memory for the parse results is done by the library
(the user cannot know how much memory will be needed). The ownership of
parse results is transferred to the user, who is expected to deallocate
memory allocated by regparse() with regfreesub().
The measurements have been taken on an Intel(R) Core(TM) i7-8750H CPU
with 32 KiB L1 Data cache (x6 cores), 32 KiB L1 Instruction cache (x6
cores), 256 KiB L2 Unified cache (x6 cores), 9216 KiB L3 Unified cache,
32 GiB RAM.
Software versions:
- ragel-7.0.4
- kleenex built from Git at commit d474c60
- re2c built from Git at commit 1c3bead
- re2-2020-11-01
- GCC-10.2.0
- Clang-11.0.0
It reduces the visual noise on the plot: there are no exponents, but
the number of printed decimal places depends on the magnitude. In
combination with very high precision and some custom code that rounds
fractional part but keeps integer part intact, this allows one to have
numbers that are not too precise, but not too rounded either.
This format also has the advantage that it does not need math font, so
fonts like Comic Neue that don't have math support can be used as well.
The new option allows one to select the font to be used in the generated
PGF plot. Currently the supported fonts are 'comicneue' (Comic Neue) and
'cmbright' (Computer Modern Bright).
In https://bugs.gentoo.org/779187 Matt noticed errors when running ./configure
on dash:
./configure: 5339: test: x: unexpected operator
./configure: 5386: test: x: unexpected operator
The change is to use POSIX '=' comparison instead of bash-specific ==.
Patch-by: Matt Whitlock
Bug: https://bugs.gentoo.org/779187
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
`bmake distcheck` still fails with the same error if configured with
`--enable-benchmarks`, but now at least in the basic configuration it
succeeds. The error with `--enable-benchmarks` is:
Making distclean in benchmarks/submatch_nfa
bmake[2]: "[.]/re2c-2.0.3/_build/sub/benchmarks/submatch_nfa/Makefile"
line 407: Could not find ./.deps/bench_submatch_nfa-bench.Po
bmake[2]: Fatal errors encountered -- cannot continue
A recent commit bf5b965384 fixed an error
specific to build environments which have 32-bit size_t. It was not
captured by CI because the CI didn't use 32-bit builds on Linux/macOS,
and it didn't run tests on Windows (which has 32-bit size_t).
The error was in setting a constant to ~0u and comparing it to another
constant of type size_t, defined as std::numeric_limits<size_t>::max().
In 64-bit mode these are different values, but in 32-bit mode they are
the same, which triggered errors in the algorithm that finds fixed tag
candidates. The error manifested as a lot of broken tests.
Changes:
- give benchmarks more descriptive names
- make the first benchmark triple in the "rep" series use multiple
tags and the same counters as the other triples
Prefix all benchmarks with a number, where the first digit is the group
index (e.g. all "real-world" benchmarks start with 0), and the second
digit is the index of a benchmark inside group. Now it is possible to
run a selected subset of benchmarks, e.g. all "real-world" benchmarks:
$ ./run.py --filter '^0'
re2c benchmarks use two different end-of-input handling methods: the
default bounds checking method, and the more recent $-rule method based
on `re2c:eof` configuration. Although it is interesting to compare their
performance, presenting both methods on the plots makes them too
cluttered (and the performance of the two methods is similar compared to
other algorithms and lexer generators). So one of them is chosen as
primary (bounds checking) and its suffix "-bc" is dropped to simplify
labels on the plots.
Don't cut off parts of the plot because of too-small limits: set limits
based on the number of bars in each individual plot, and use "ymax" and
"ymin" to specify exact limits (rather than the previously used
"enlarge y limits" setting).
Compute maximum bar length ("xmax") based on the global average and
maximum values among all benchmarks. This is to avoid cases when a very
long bar dominates other bars and makes them appear too small. Such
outliers are cropped and marked with "..." to show that they overflow.
Previously the corring was done only in "relative" mode, because in
"absolute" mode it was unclear how to choose "xmax" value. Now that
"xmax" is computed based on global average and maximum, cropping is used
in both "relative" and "absolute" modes.
For DFA AOT submatch benchmarks we do not know the number of captures or
the regular expression size. In principle they could be extracted from
the generated file (for re2c only), or an option could be added to the
programs to print statistics (for re2c and ragel, but not kleenex, as it
autogenerates the program and the user controls only the regexp). But it
would be still problematic, as different engines have slightly different
regexps and ways to mark submatch positions.
Uniform scale means that there is a global scale (a common maximum value
shared by all benchmarks). Previously this was used only in "relative"
mode, when all algorithms of a given benchmark are normalized by the
first algorithm. Now it is also used in "absolute" mode.
The new options are:
--verify verify parse results on full data files
--quickverify verify parse results on small data samples
Replace the old verification script verify.sh with run.py.
Also, add options `--output <JSON file>` (to generate JSON output in a
form accepted by json2pgfplot.py) and `--filter-out <word>` (to mask
some of the benchmarks). The new script run.py can be used as follows:
$ ./run.py -h
usage: run.py [-h] [--output OUTPUT] [--filter FILTER]
[--filter-out FILTER_OUT] repcount
positional arguments:
repcount number of times to run one benchmark
optional arguments:
-h, --help show this help message and exit
--output OUTPUT JSON file to write benchmark results to
--filter FILTER pattern to select a subset of benchmarks by name
--filter-out FILTER_OUT
pattern to mask a subset of benchmarks by name
Previously the script grouped benchmarks with the same name together
only if they appeared sequentially in the input file, so that all
benchmarks in one group preceded any benchmarks in another group. Now
this restriction is relaxed: benchmarks with the same name are grouped
together regardless of their position.
This is needeed for ccache: it only knows how to cache the compilation
of a single file. Other types of compilations (multi-file compilation,
linking, etc) will silently fall back to running the real compiler.
Notice d benign warning when built with clang++ -Weverything:
src/dfa/dead_rules.cc:205:77: warning: for loop has empty body [-Wempty-body]
for (; sentcls < nsym && sentsym >= dfa.charset[sentcls + 1]; ++sentcls);
^
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
Noticed when built with clang++ -Weverything:
src/skeleton/generate_data.cc:325:35:
warning: declaration shadows a local variable [-Wshadow]
const mtag_t &t = tagtrie.head[tval];
^
src/skeleton/generate_data.cc:300:21:
note: previous declaration is here
for (size_t t = ltag; t < htag; ++t) {
^
Renamed `t` to `mt` to disambiguate.
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
Noticed when built with clang++ -Weverything:
./src/skeleton/mtag_trie.h:67:2: warning:
empty expression statement has no effect;
remove unnecessary ';' to silence this warning [-Wextra-semi-stmt]
{;
^
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
The idea is to replace registers and register operations with a runtime
log of backlinks that allow to reconstruct the matching TNFA path and
unwind tag history backwards from the final state to the initial state.
As the benchmarks show, on artificial inputs registerless TDFA can be
much faster than TDFA with registers (a few orders of magnitude). On
real-world inputs it is slightly slower (up to 2x).
Bit fields allow packing tag index and tag sign together in one 32-bit
value, which saves space. However it makes re2c slower (approximately
10% on benchmark bench_submatch_nfa/HTTP-RFC7230).
usage: json2pgfplot.py [-h] [--relative] input output
positional arguments:
input input JSON file
output output TeX file
optional arguments:
-h, --help show this help message and exit
--relative output CPU time relative to the first algorithm
Parent file is the one that contains `/*!include:re2c*/` directive, and
it is not necessarily the same as the most recently opened file (parent
file may have multiple include directives, in which case the most recent
file may be the sibling include file, not the parent one). The incorrect
behavior was introduced in 777476b0d8.
The new option is similar to GCC or Clang option `-MD -MF <filename>`.
It computes a transitive closure of all files included with
`/*!include:re2c ... */` directives and writes them to the specified
file in the form of a makefile rule:
<output> : <input1> <input2> ...
The new option requires that the output file is specified (otherwise it
is unclear how to generate dependency rule).
For other engines (kleenex and ragel) regeneration is optional (it is
controlled by CMake option RE2C_REGEN_BENCHMARKS and Autoconf option
--enable-benchmarks-regenerate). This is so because it may be
problematic to download and build other engines, and the results may
be hard to reproduce.
For re2c, it makes sense to always see the latest result.
Override default CMake 'clean' target by setting directory properties
CLEAN_NO_CUSTOM (which prevents removal of all outputs of custom
commands) and ADDITIONAL_CLEAN_FILES (which adds selected files to the
'clean' target). Use the combination of these properties to make the
'clean' target remove autogenerated C code, binaries and data files,
but keep generators (kleenex, ragel).
For libre2c benchmarks (bench_submatch_nfa and bench_submatch_dfa_jit)
the input is not large (unlike bench_submatch_dfa_aot), so it is
possible to hardcode it as C strings. This allows to avoid file I/O,
which is problematic because paths to the input files depend on the
relative location of the source and build directories.
- Drop ragel -T1/-F1 variants, as they are always slower than -G2.
- Drop re2c TDFA(1) with --eager-skip option in order to reduce the
number of re2c variants (the dropped variant is sometimes better,
sometimes worse than TDFA(1), but not by much, and not worth having
a separate variant).
For Autotools, the option is --enable-benchmarks-regenerate.
For CMake, the option is RE2C_REGEN_BENCHMARKS.
If the option is off (the default), C files for benchmarks are copied
from pre-generated files.
If the option is on, C files are generated with regexp engines.
Pre-generated files are updated. This requires building regexp engines
(kleenex, ragel, re2c), which works at the moment but may easily break
in the future, as it depends on downloading source code from the
internet and building it. Ragel is built from a versioned archive and
needs only a C++ compiler, but kleenex is a haskell program which
requires ghc and cabal (and it also needs patching for the recent ghc).
It consists of 3 independent subseries ("iter_a", "iter_b", "iter_c")
for artificial regexps with submatch extraction under repetition.
"iter_a" is R(n) = (a{n} #t)*.
"iter_b" is R(m,n,k) = ((a{m}|a{n}|a{k}) #t)*.
"iter_b" is R(m,n,k) = (a{m} #t1)* (a{n} #t2)* (a{k} #t3)*.
Ragel generates incorrect parse results on "iter_b" and "iter_c",
because actions collide and either they happen to often, or if
disambiguation operators <: and ** are used, some actions get removed
while they are still needed.
Kleenex generates very large output (tens of megabytes) for "iter_b"
(infeasible to compile with GCC or Clang on O3), and for "iter_c" it
causes OOM condition.
These are artificial benchmarks for concatenation of alternatives. The
regular expression is:
R(n,k) =
(@t_{1,1} a_1* @t_{1,2} | ... | @t_{1,2k+1} a_k* @t_{1,2k+2})
...
(@t_{n,1} a_1* @t_{n,2} | ... | @t_{n,2k+1} a_k* @t_{n,2k+2})
For example R(2,2) = (@a1 a* @a2 | @b1 b* @b2)(@a3 a* @a4 | @b3 b* @b4).
The benchmark includes cases R(1,2), R(1,4), R(1,8), R(2,2), R(2,4),
R(2,8), R(4,2), R(4,4), R(4,8).
For Ragel to handle these benchmarks correctly it is necessary to use
left-guarded concatenation <:.
The benchmarks show that sta-DFA performance degrades quickly with
growing n and k parameters.
Kleenex size and sta-DFA performance degrade quickly with increasing
level of nondeterminism (the k-parameter).
Current known failures are Ragel benchmarks ab2_4, ab2_8, ab4_2, ab4_4,
ab8_2. They have nondeterministic tags that require multiple tag
variables to store different possible tag values (until nondeterminism
is resolved, one of the values is selected, and the rest are discarded).
Ragel's actions are not designed for that: if two actions modify the
same variable, Ragel places them in some order on the same transition,
and the resulting value will be incorrect.
These are artificial benchmarks for sequences of 'a' and 'b' symbols
with varying degrees of tag nondeterminism. The regular expression is:
R(n,k) = @a_1 a* @b_1 a{k} b* ... @a_n a* @b_n a{k} b*
For example, R(2,4) = @a_1 a* @b_1 aaaa b* @a_2 a* @b_2 aaaa b*. The
benchmark includes cases R(2,0), R(2,4), R(2,8), R(4,0), R(4,2), R(4,4),
R(8,0), R(8,1), R(8,2).
Some of the benchmarks in this series cannot be handled correctly by
Ragel due to its lack of support for nondeterministic tags.
Kleenex size and sta-DFA performance degrade quickly with increasing
level of nondeterminism (the k-parameter).
The split is implemented by providing two sets of include files (one for
the default bounds checking method, and one for the $-rule method) and
passing different include options when building *-bc and *-eof benchmark
variants.
Previosuly cmake/Re2cCompilerFlags.cmake always prepended C++ compiler
options to CMAKE_CXX_FLAGS to avoid overriding user-defined options
passed on the command line with -DCMAKE_CXX_FLAGS. They would still be
overridden by build-type specific options in CMAKE_CXX_FLAGS_DEBUG,
CMAKE_CXX_FLAGS_RELEASE and so on, but at least if the user didn't set
any build type, then user-defined options would take proprity.
However, in MSVC builds CMake populates CMAKE_CXX_FLAGS with default
options, making it impossible (or just hard) to distinguish between
default and user options. So, for MSVC we override CMAKE_CXX_FLAGS.
This commit also fixes the order of CMAKE_CXX_FLAGS added by
cmake/Re2cCompilerFlags.cmake: previosuly the order was reversed, as
each flag was preopended individually. Now all flags are accumulated in
an intermediate variable, which is prepeded in one go.
-t, --type-header option has always been disabled for these targets,
but the recently added directive `/*!header:re2c:on*/` enforced header
generation even if the header option was disabled. Now it checks the
target as well.
This has the benefit of using a custom slab allocator for list elements,
as well as a simplified way of iterating through the elements (no need
to check for the end of sequence and wrap the iterator).
This patch speeds up skeleton generation for test/bug128.re by ~1.5x.
Note that `setvbuf(buffer, NULL, _IOFBF, 1024 * 1024);` does not achieve
the same effect (it makes no difference on systems where buffering is
already enabled).
Process each subpath of the multipath separately (instead of iterating
on all subpaths in lockstep). This allows to write data in the output
buffer sequentially.
At least on ext4 `open(O_TRUNC); write(); close()` works a lot
slower than `unlink(); open(); write(); close()`.
My speculation is that ext4 has to do non-trivial free block
accounting at `close()` time that were freed by `O_TRUNC`.
Before the patch:
```
$ rm 1.c.line8.input 1.c.line8.keys; time ../re2c bug128.re -o 1.c -Si
real 0m3,011s
user 0m2,091s
sys 0m0,907s
$ time ../re2c bug128.re -o 1.c -Si
real 0m9,506s
user 0m1,708s
sys 0m0,946s
```
After the patch:
```
$ rm 1.c.line8.input 1.c.line8.keys; time ../re2c bug128.re -o 1.c -Si
real 0m3,785s
user 0m2,615s
sys 0m1,152s
$ time ../re2c bug128.re -o 1.c -Si
real 0m3,570s
user 0m2,348s
sys 0m1,205s
```
It's a 3x speedup.
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
The test covers commit a9d582f9d2. Before
that commit re2c would either fail with assertion on the added test (if
it was built in debug mode), or generate incorrect code (with missing
tag operations on fallback transitions).
Also, be aware of sta-DFA tag actions when constructing tags on fallback
transition and when printing the automaton with --dump-adfa option.
Commit 56e46545b8 moved in-state tag
operations before YYFILL label in case $-rule is used. It erroneously
suppressed the generation of sta-DFA tag actions in final states or in
case of disabled YYFILL. This commit fixes that mistake and reinstates
sta-DFA tag actions in such cases.
Depending on the STL implementation these operations are required for
vectors of wrap iterators (in particular, push_back() method may require
a default assignment operator).
Previously tag values were stored in vectors (even thought most of the
tags are s-tags and have only one element in the history). For tests
with many tags this represenation was taking too much space and time
spent on vector initialization.
Now all tag values are stored as scalars (32-bit integers). For s-tags
the value is the offset. For m-tags the value is an index in the tag
trie (a prefix tree of tag histories that is encoded in an array). This
representation is faster and takes less space.
This allows to save space that was previously used for storing
intermediate tag values of tag variables as vectors
This partially fixes#331.
Process each subpath of the multipath separately (instead of iterating on all
subpaths in lockstep) in order to reduce the size of storage used for the
intermediate tag values. Previously the required storage was a matrix of size
M x N, where M is the width of the multipath and N is the maximum tag variable
number (prior to tag optimizations). Matrix elements are vectors, because some
tags are m-tags (they need to record the full history, not just the last value).
Allocating and initializing the matrix takes a long time. Now the required
storage is a vector of length N. The same storage is reused for all subpaths.
This partially fixes#331.
If $-rule is used the lexer may need to jump to the YYFILL label to
rescan the current input character after YYFILL supplies more input.
This may cause re-application of tag operations if they are generated
after the YYFILL label, which may produce incorrect results for non-
idempotent tag operations.
In-state tag operations are used either with sta-DFA (where all tag
operations are in-state and there are no on-transition operations), or
when operations on all transitions agree and can be hoisted out of
transitions into the state body.
If end-of-input rule $ is used, the lexer may need to rescan the current
input character after a successful return from YYFILL. Therefore any
operations that are hoisted out of transitions into the state body may
be re-applied. This is fine as long as the operations are idempotend
(subsequent application produces the same results).
Skip operation is not idempotent, because it increments the cursor every
time it is applied. This is only a problem with --eager-skip option, as
otherwise skip takes place before YYFILL label.
Tag operations may be idempotent or not, depending on whether the tags
used on the right hand side are also used on the left hand side of some
operations (eg 'x = y; y = z;' is non-idempotent because it overwrites
y).
This is a planned, intermediate change that prepares the following
things:
- Preparing to use different generators (Xcode, NMake, Ninja)
- Preparing to use rebuild and test documentation. This required
renaming some steps
- Preparing to build libs
- Preparing to use Autotools
- Preparing to make a full migration of CMake builds from Travis CI
Notable improvements include the unification of the Configure job
(now it is Fast Configure) for all platforms
There is no need for redirect output with > (which doesn't work on
Windows) as this is how rst2man works:
rst2man [options] [<source> [<destination>]]
For more see: https://github.com/skvadrik/re2c/issues/325
Windows does not have a native 'man' and 'sed' commands.
Therefore, we are looking for the Cygwin versions of 'man' and 'sed'.
For more see: https://github.com/skvadrik/re2c/issues/325
- Stripping headers may not work as expected, as man output differs
on different systems. This should work now.
- Simplified output parsing procedure by replacing the regular
expression by TERM=dumb.
- Stripping escape sequences.
- Added documentation about the intentions of this script.
For more see: https://github.com/skvadrik/re2c/issues/325
This is aimed to reduce assumptions, workarounds and potential
misunderstandings due to not a completely obvious behavior of
'fatal' function.
For more see discussion on https://github.com/skvadrik/re2c/pull/320
Fixes the following build scenario:
$ mkdir stage1 && cd stage1
$ cmake .. && make
$ cd .. && mkdir stage2 && cd stage2
$ cmake .. \
-DRE2C_REBUILD_LEXERS=yes \
-DRE2C_FOR_BUILD=$(pwd)/../stage1/re2c && make
This is a fix for linker flags that were mistakenly added after
discussion at https://github.com/skvadrik/re2c/pull/318.
Although STATIC_LINKER is technically a program for "getting one library
from a bundle of something", it works *completely* differently than
EXE_LINKER and SHARED_LINKER.
We do not use the functionality provided by FindBISON e.g. BISON_TARGET and
so on. All we need below is BISON_EXECUTABLE. Thus, I replaced find_package
by find_program and set BISON_EXECUTABLE explicitly. This is also fixes
searching executables on Windows systems.
-f, --storable-state option should not be used if YYFILL is disabled,
because without YYFILL the interrupt points do not necessarily
correspond to storable state labels (with generic API interrupts can
happen on any API invocation). This may cause subtle bugs when the
lexer is resumed from the wrong program point.
Instead of defining YYFILL to always return failure, disable it entirely
with `re2c:yyfill:enable = 0;`, but keep the check with
`re2c:yyfill:check = 1;`. This way re2c does not generate dead code that
invokes YYFILL and always fails.
Previously EOF rule was added only at the code generation level: if was
neither present as an NFA state, nor in any of the final DFA states.
This doesn't work in case there is a nullable loop and the default rule
is not specified, because the initial state gets merged with other DFA
states, which shouldn't match EOF rule. As a result, EOF rule never
matched.
The fix is to add EOF rule on the regexp level as a match-empty rule "".
- Added "preprint" footnote with a reference to the SPE article.
- Corrected submitted\revised/accepted dates.
- Added DOI.
- Changed bibliography style.
- Updated link to Glenn Fowler's page "An interpretation of the POSIX
regex standard". It is still accessible from the Wayback Machine, but
the links are not stable. The new link is to a manually uploded copy
of the page.
The problem with the path to include subdirectory is that the tests
can be run from different places relative to the source directory
(an out-of-tree build can be anywhere). Therefore instead of hardcoding
a few possible locations in the test itself, it is better to use a path
relative to @top_srcdir@ and compute it once for all tests.
This example uses include file unicode_categories.re and specifies
paths to include directory using -I option. Since the test can be
run both in in-tree and out-of-tree builds, different include paths
are needed to cover different cases.
In addition it is necessary to disable line directives, otherwise the
unstable path to the include file gets embedded in the output file.
The tests are generated with a Bash script from a list of test cases
that contain a regexp and a string. Some test cases use the same regexp
multiple times with a different string. For the autogenerated tests the
string is not used, so such test cases result in identical tests.
This patch changes the script to check for duplicates. The original set
of test cases remains unchanged and can be run with another script:
test/posix_captures/.run/__run.sh
Many tests have been updated, because some of the variable tags have
been turned into fixed tags.
The following script was used to check that skeleton tests for the new
re2c run without errors on the data (.input and .keys) generated both by
the new re2c and by the old re2c.
if [ $# -ne 3 ]; then
echo "usage: $0 <testdir> <re2c_new> <re2c_old>"
exit 1
fi
testdir="$1"
re2c_new="$2"
re2c_old="$3"
for f in `find $testdir -name '*.re'`; do
echo $f
# extract options
opts="$(head -n 2 $f \
| grep -E -a '^.*INPUT.*$' $f \
| sed -E 's/.*\$INPUT -o \$OUTPUT(.*)$/\1/')"
# run skeleton test with new re2c
$re2c_new --skeleton $opts $f -oa.c && gcc a.c -oa && ./a
# run skeleton test with new re2c on the data generated by old re2c
mv a.c b.c
$re2c_old --skeleton $opts $f -oa.c && gcc b.c -o b && ./b
# cleanup
rm -f *.keys *.input a b
done
Some exisitng tests have been changed to explictly set option
`--fixed-tags toplevel` or `--fixed-tags none`. These tests would be
affected by flipping the default from `toplevel` to `all`, because some
of the variable tags would become fixed, making the whole test useless,
as it no longer tests what it is supposed to test.
For some tests "nofix" variants have been added. Currently the generated
code is the same as for the base variant, but this will change after
flipping the default from `toplevel` to `all`.
With POSIX capturing groups one tag may correspond to multiple capturing
parentheses, as in ((e)). Previously re2c generated assignment chains
like `yypmatch[i] = yypmatch[i+2] = ...` and treated the whole chain as
a single l-value. This doesn't work in cases that are more complex than
a simple assignment, for example with generic API when the shift
distance is nonzero (it is not possible to shift the whole chain of
assignments in one operation).
This patch fixes the broken cases when a tag expands to multiple tags by
setting the first tag (which may require multiple statements or checks),
and then assigning the remaining tags to the first tag.
When applying fixed tags optimization to tags that are enclosed in
alternative or repetition it is necessary to check if the base tag has a
no-match value (in that case fixed tag should also be set to no-match).
For tags in top-level concatenation the check is not needed, because
they always match.
This commit implements the optimization, but does not enable it yet
(that would require updatimg many test results and will be done in a
separate commit).
This is related to bug #281 "Apply fixed tags optimization more
aggressively".
These rules are always needed (e.g. the file help.cc generated by these
commands is one of the source files that are required for the minimal
build). The rules were accidentally moved under conditional branch in
commit be688fdc6d, and were enabled only
if re2c is the root project. This caused a build error "Cannot find
source file" when trying to build re2c as a subproject:
├── other
│ ├── build
│ └── CMakeLists.txt
└── re2c
With the following other/CMakeLists.txt:
project(other)
add_subdirectory(../re2c re2c-build)
This is a partial fix for #302 "Using CMake add_subdirectory fails".
If the definition of a re2c block comes from an include file, use the name of
the file as well as the block line to generate a unique DFA name. The block
line itself is insufficient as there may be two blocks on the same line in
different files.
This has caused an error in one of the skeleton tests.
Rename Unicode tests. Rename is needed because Unicode tests are
clustered into groups of similar tests that have similar filenames,
and the previous rename left one tests of a grop having a short filename
without options, while the remaining tests have options in the filename.
Update test generator and script that runs Unicode tests.
This commit replaces many (~128) autogenerated layout tests with a few
hand-written tests. The autogenerated tests were not very useful as they
covered many similar trivial use cases, and only a few interesting ones.
This commit renames test files and patches filename strings embedded in
line directives, warnings, error messages, etc. in the generated output.
When possible, all options have been dropped from the filename. In cases
when this would cause collision due to multiple different tests being
renamed to the same file, only the first test has a filename without
options, and the remaining tests have filenames with options where all
special characters have been replaced with underscores.
Conversion has been done automatically with the following Bash script:
#!/bin/bash
for f in `find . -name '*.re'`; do
stem="$(basename $f | sed -E -e 's/^([^.]+)\..*$/\1/')"
ext="$(basename $f | grep -q -- '--lang(go)' && echo go || echo c)"
x="${f%.re}.$ext"
newf="$(dirname $f)/$stem.re"
newx="$(dirname $f)/$stem.$ext"
if [ "$f" == "$newf" ]; then
continue
elif [ -e "$newf" ]; then
sfx="$(basename $f | sed -E -e 's/^([^.]+)[.](.*)[.]re$/\2/' -e 's/[(.+-]+/_/g' -e 's/[)]//g')"
newf="$(dirname $f)/${stem}_$sfx.re"
newx="$(dirname $f)/${stem}_$sfx.$ext"
fi
a="$(echo $(basename ${f%.re}) | sed -E -e 's/([()])/\\\1/g' -e 's/\./\\./g')"
b="$(basename ${newf%.re})"
perl -p -e 's/'"$a"'/'"$b"'/g' "$f" > "$newf"
perl -p -e 's/'"$a"'/'"$b"'/g' "$x" > "$newx"
rm "$f" "$x"
done
This commit adds a comment with options to the beginning of each .re file.
Because of the added line all test results had to be updated (line numbers
in line directives, error messages, warnings, etc. have increased by one).
The testing script run_test.sh has been changed to use options in the
comments rather than options in the filenames.
Conversion has been done almost automatically (except for a few rare tricky
cases) with the following horrible Bash script:
#!/bin/bash
for f in `find . -name '*.re'`; do
switches=`basename "$f" | sed \
-e 's/^[^.]*\.re$//g' \
-e 's/^[^.]*\.\(.*\)\.re$/\1/g' \
-e 's,[+],/,g' \
-e 's/^\([^-]\)/-\1/' \
-e 's/--\([^ (-]*\)/ --\1/g' \
-e 's/(\([^)]*\))/ \1/g' \
`
ext=`basename "$f" | grep -q -- '--lang(go)' && echo go || echo c`
x="${f%.re}.$ext"
ff="$f.mod"
xx="$x.mod"
linedirs=`egrep -q -- '(#|//)line' "$x" && echo 1 || echo 0`
dot=`echo "$switches" | fgrep -q -- ' --emit-dot' && echo 1 || echo 0`
skel=`echo "$switches" | fgrep -q -- ' --skeleton' && echo 1 || echo 0`
if [ $ext == c ]; then
l='// re2c $INPUT -o $OUTPUT '"$switches"
else
switches=`echo "$switches" | sed 's/ --lang go//'`
l='//go:generate re2go $INPUT -o $OUTPUT '"$switches"
fi
nocomment=0
if [ "$(tail -n 1 $x | fgrep -q 'error: ' ; echo $?)" -eq 0 ]; then
nocomment=1
elif [ $dot -eq 1 ]; then
nocomment=1
elif [ $skel -eq 1 ]; then
nocomment=1
fi
echo "$l" > "$ff"
cat "$f" >> "$ff"
if [ $linedirs -eq 0 ]; then
head -n 1 "$x" > "$xx"
if [ $nocomment -eq 0 ]; then echo "$l" >> "$xx"; fi
tail -n +2 "$x" >> "$xx"
else
head -n 2 "$x" > "$x.head"
tail -n +3 "$x" > "$x.tail"
if [ $nocomment -eq 0 ]; then echo "$l" >> "$x.head"; fi
if [ $ext == c ]; then
e='s/^([ \t]*#line )([0-9]+)(.*\.re"|.*\.c")$/($1).($2+1).($3)/eg'
else
e='s/^(\/\/line ".*".)([0-9]+)$/($1).($2+1)/eg'
fi
perl -p -i -e "$e" "$x.tail"
cat "$x.head" "$x.tail" > "$xx"
rm "$x.head" "$x.tail"
fi
a="$(basename $f | sed -E -e 's/([()])/\\\1/g' -e 's/\./\\./g')"
perl -p -i -e 's/^(.*'"$a"'[:(])([0-9]+)(.*)$/($1).($2+1).($3)/eg' "$xx"
perl -p -i -e 's/^(.*shadowed by rule at line )([0-9]+)(.*)$/($1).($2+1).($3)/eg' "$xx"
perl -p -i -e 's/^(.* already defined at line )([0-9]+)(.*)$/($1).($2+1).($3)/eg' "$xx"
if [ $skel -eq 1 ]; then
perl -p -i -e 's/^(.*[_.]line)([0-9]+)(.*)$/($1).($2+1).($3)/eg' "$xx"
fi
done
find . -name '*.re' -o -name '*.c' -o -name '*.go' | xargs rm
for f in `find . -name '*.mod'` ; do mv "$f" "${f%.mod}" ; done
Re2c is a self-hosting lexer generator, therefore it needs re2c in order
to rebuild itself. It bootstraps using precompiled files in the bootstrap
subdirectory. Precompiled files can be used as long as the .re files are
not changed, otherwise lexers must be regenerated. Previosuly the build
system tried to autodetect whether to regenerate or to copy bootstrap
files by checking if the re2c executable exists in the build directory.
However, for the build system it means that dependency graph is changed
on every build, because bootstrap commands and regenerate commands have
different dependencies: bootstrap commands depend on the bootstrap
files, and regenerate commands depend on the .re files. It is not
possible to choose the commands statically, because the check for re2c
executable should run on every build rather than once at configure time.
CMake used a workaround and set both bootstrap and .re files as
dependencies for both commands, which caused weird rebuilds on Travis CI
(when the same lexer was regenerated multiple times during one build).
Automake used a different workaround (a pattern rule that didn't have
the dependency on bootstrap files at all).
This commit disables the checking mechanism and requires passing an
explicit configure parameter and a path to the re2c executable in order
to regenerate lexers. Otherwise, by default lexers are copied from
bootstrap files.
CMake: cmake ... -DRE2C_REBUILD_LEXERS=on -DRE2C_FOR_BUILD=<path>
Makefile.am: configure ... --enable-lexers RE2C_FOR_BUILD=<path>
CMakeLists.txt uses features such as HOMEPAGE_URL parameter in project()
command and add_compile_definitions() command that were introduced in
CMake version 3.12.
This fixes bug #297 "configure error on ubuntu 18.04 / cmake 3.10".
Normal blocks should use global options that are accumulated over all
non-reuse blocks, and use blocks should use options of the preceding
rules block, plus the options added by the use block itself. Before
this commit re2c failed to set up global options correctly, and as a
result the first normal block had incorrect options if it was preceded
by some reuse blocks.
Previously in condition mode re2c generated condition switch only for
the first re2c block in the file. This makes it impossible to have
multiple re2c blocks with conditions. The change in this commit doesn't
break backwards compatibility, because the case that was previously
supported (with one lexer per file) works in the same way as before,
and the previously unsupported case (with multiple lexers) now works.
This fixes bug #295 "Condition switch generated for only one lexer per
file" reported by Joshua Inscoe.
This change breaks backwards compatibility for some (rare) use cases.
For historical reasons re2c reset all API-related configurations to
default values in each `/*!use:re2c ... */` block. This behaviour is
illogical and inconvenient: the whole purpose of reuse mode is to share
configurations, definitions and rules between blocks, and resetting
selected configurations means that they have to be redefined in each
reuse block. This has caused confusion (see bug #291 "Defines in rules
block are not propagated to use blocks" reported by Joshua Inscoe).
The odd behavior was kept for backwards compatibility. It goes back to
the versions of re2c and was probably introduced accidentally or by
mistake. It is unlikely that many users rely on the old behavior: if
they need user-defined configurations in some of the blocks and default
configurations in other blocks, it is more likely that they would
explicitly reset them to defaults rather than rely on undocumented re2c
behavior.
The change affects the following configurations:
re2c:define:YYGETCONDITION
re2c:define:YYSETCONDITION
re2c:define:YYFILL
re2c:define:YYGETSTATE
re2c:define:YYSETSTATE
re2c:define:YYBACKUP
re2c:define:YYBACKUPCTX
re2c:define:YYCONDTYPE
re2c:define:YYCTXMARKER
re2c:define:YYCTYPE
re2c:define:YYCURSOR
re2c:define:YYDEBUG
re2c:define:YYLESSTNAN
re2c:define:YYLIMIT
re2c:define:YYMARKER
re2c:define:YYPEEK
re2c:define:YYRESTORE
re2c:define:YYRESTORECTX
re2c:define:YYRESTORETAG
re2c:define:YYSTAGN
re2c:define:YYSTAGP
re2c:define:YYMTAGN
re2c:define:YYMTAGP
re2c:define:YYSHIFT
re2c:define:YYSHIFTSTAG
re2c:define:YYSHIFTMTAG
re2c:define:YYSKIP
re2c:label:yyFillLabel
re2c:label:yyNext
re2c:variable:yyaccept
re2c:variable:yybm
re2c:variable:yych
re2c:variable:yyctable
re2c:variable:yytarget
Whenever an option is set or unset by the user, re2c needs to remember
that user-defined options have been updated and have diverged from the
real options. Any following attempt to read options must trigger
synchronization of user-defined and real options. Previously this was
handled correctly for all options except those with custom handlers.
The run_tests.sh script generates output for all examples and appends
the generated header at the end of the output file (to make it easier to
diff actual and expected output). Script that run examples from
autogenerated output need to cut off the appended header, otherwise the
example won't compile/run.
Before this patch the filename specified in `re2c:flags:t`,
`re2c:flags:type-header` configuration was relative to the directory
of the source file. It is inconvenient for out-of-tree builds, when the
output is generated in a directory different from the source directory
(which may be read-only). Therefore this patch makes the filename in
configuration relative to the directory of the output file.
As for the `re2c:flags:o`, `re2c:flags:output` configuration, it is
removed altogether. Making it relative to the source directory has the
same problems as with the header file, and making it absolute or
relative to the build directory (where re2c is run) is also bad,
because build-related information will have to be hard-coded in the
source file. Given that this is a relatively new configuraion and it
hasn't been documented in any released version of re2c, it is better to
delete it.
The change does not affect `-t`, `--type-header` and `-o`, `--output`
command-line options.
This fixes build error for in-tree builds wit hlexer regeneration,
broken after recent commit 5d130e89cc
that made filenames in configurations relative to the directory of the
source file.
Previously we did this to avoid tracking all the files generated by a
test run (optional header files, skeleton artifacts, etc.) However, this
approach has problems with tests that generate header in a nested
subdirectory: directory structure needs to exist (otherwise re2c won't
be able to open the header file), but re-creating it in a temporary
subdirectory is problematic (`find` collides with other threads running
in parallel, creating and removing directories).
Instead of using a sandbox, track all files generated by a test (that
turns out to be not so difficult with shell globs).
This patch adds a requirement that all tests that use header must
specify it explicitly in the test options (in filename) using the long
option --type-header. Also, this patch allows using plus sign in the
argument to --type-header option to substitute for slash (that cannot be
used in filenames). This allows one to specify headers in nested
subdirectories.
Prepend source file directory to these filenames when parsing
configurations and updating the corresponding options.
The affected configurations are:
`re2c:flags:o`, `re2c:flags:output`
`re2c:flags:t`, `re2c:flags:type-header`
The reason for this change is to make the contents of the source files
independent of the subdirectory where they are located relative to the
build directory where re2c is run. Otherwise it is necessary to hardcode
the relative path in the configuration and change it every time when
either the source file is moved to another subdirectory, or the build
directory is changed.
This change breaks backwards compatibility. However, the configurations
have been added not very long ago (in re2c-1.2) and not documented yet,
so it is likely that not many users will be affected. The corresponding
command-line options `-o`, `--output` and `-t`, `--type-header` remain
unchanged.
Previosly RE like `@t "xxx"` that use manual repetition were optimized,
but RE like `@t "x"{3}` that use fixed repetition counters were not,
even though they are unrolled to the same RE.
Changes:
- Use MANPAGER instead of PAGER (no significant difference).
- Set MANWIDTH=100 and to get deterministic line width.
- Tweak genhelp.sh as it was cutting off too many lines in the generated
output (presumably something has changed in the way man works).
Changes:
- Added new generic API primitives:
YYSHIFT
YYSHIFTSTAG
YYSHIFTMTAG
- Added new configurations:
re2c:api:style
re2c:api:sigil
- Rewrote the "user interface" section.
- Reordered configuraions and API primitives (grouped them by their
meaning instead of sorting alphabetically).
Changed IRE notation from expressions with operators like (i,J,e_1|e_2)
to more explicit Alt(i,J,e_1,e_2) to avoid confusion around the fact that
e_1|e_2 is neither an IRE nor a RE, but only a part of syntactic form.
The new option defaults to false. Also, add a corresponding boolean
configuration `re2c:flags:case-ranges = 0;`.
The new option allows to generate compact cases of the form
`case low ... high:` in switch statements. This is not standard C/C++
syntax, but some compilers (GCC, Clang, Tcc) support it.
This implements #290 "use switch range" suggested by algorithm314.
On a fresh checkout `autogen.sh` complains about missing `m4`:
```
$ ./autogen.sh
aclocal-1.16: warning: couldn't open directory 'm4': No such file or directory
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
...
```
The change always attempts to create the directory.
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
Previously the function was not erroneous, because only the head of the
newly constructed list was in a local stack variable, and the rest
consisted of reordered elements from the input list. The offending
pointer was initially set to the head of the list (local addr), and
then advanced to subsequent elements. The case when it was not advanced
and still pointed at the local variable was handled in a special way by
returning NULL.
This patch slightly rewrites topsort to use the output parameter
instead of a local variable as the head of the new topsorted list.
The typo in the macro name resulted in not using re2 library in libre2c
benchmark, even though it was found by CMake. The error was revealed by
Clang warning -Wunused-const-variable.
This patch fixes builds whn both options -DRE2C_BUILD_RE2GO and
-DRE2C_REBUILD_DOCS are set (previously in that case help.cc would be
generated twice, which caused a race and resulted in garbage output).
The desired behaviour is analogous of Automake BUILT_SOURCES. Buld all
autogenerated sources into an object library, and make that library a
dependency of the main target. Note that custom target that depends on
the autogenerated sources can't be used instead (although it may seem
the right solution at first), because somehow it triggers building of
its dependencies multiple, which ruins parallel builds.
Previously configuration `re2c:eof` was reset to default in skeleton
mode, resulting in a re2c error "EOF rule is used, but `re2c:eof` is
not set". Since re2c errors are not failures in skeleton tests (some
tests do actually test re2c errors), such cases increased the number
of "expected re2c errors", but did not show up as failed tests.
This patch increases the number of tests that run successfully with
--skeleton by some 28 test cases.
Soft failures are needed for skeleton tests: in some cases re2c fails to
set correct configurations and the test results in a re2c error, which
is not a hard error (because some tests do test re2c errors).
The generated code had `yyeofrule` labels, but jumps to `yyeof` labels,
so the program would not compile. This inconsistency was introduced in
commit 4253577f47.
Changes in the tests are numerous, but trivial (this can be verified
with changing `diff` to `diff -w` in run_tests.sh: given an option to
ingnore all space changes, diff finds no difference and all tests pass).
Reported by ubsan:
runtime error: reference binding to null pointer of type 'const struct Tag'
There was no actual crash, as the bound null value wasn't used.
Previously re2c allowed to use @@ or a custom string as a placeholder
for an argument in API primitives. With the addition of multi-argument
APIs like YYSHIFTSTAG and YYSHIFTMTAG this has become ambiguous. The
first attempt to resolve ambiguity was to use @@1, @@2, etc. This works,
but it is not perfect for two reasons. First, it is not clear how to
handle multi-digit suffixes: @@12 can be @1 followed by symbol 2, or it
can be @12 (but it is unlikely that re2c will ever have and API with 12
arguments, so it is most likely an error). Second, it is hard to
remember what is the meaning of each numbered placeholder.
This commit adds named placeholders of the form @@{name}. For APIs that
have only one argument short form @@ is still recognized (if the longer
form failed), but for multi-argument APIs only named form is allowed.
It is possible to globally override @@ with any string, for example to
have placeholders of the form ${name} or #{name}.
List of changes:
- Generic APIs YYSHIFT, YYSHIFTSTAG, YYSHIFTMTAG have been added.
- Generic APIs YYSTAGPD, YYMTAGPD have been removed. They were needed
for STADFA operations, which are now expressed using YYSTAGP /
YYSHIFTSTAG and YYMTAGP / YYSHIFTMTAG.
- Test results for tests that use generic API have been updated. Some
variable tags have turned into fixed tags. Tests with fixed-length
trailing context that do not use tags are no longer erroneous. All
skeleton tests now need to define new API primitives. STADFA tests
that use mtags now use YYMTAGP / YYSHIFTMTAG instead of YYMTAGPD.
Golang test with `re2c:decorate = 1;` that used tags in a hacky way
via pointers has been removed, because fixed tags cannot be handled
(and there is a better alternative with `re2c:decorate = 0;`).
This makes root directory less cluttered. Autoconf-specific
scripts like 'config.guess'. 'install-sh' and friends
are moved to build-aux/ now.
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
Changes:
- Allow defining YYMTAGP / YYMTAGN / YYMTAGPD with default API. These
primitives have no sensible defaults, so they always have to be
defined by the user.
- Don't generate semicolons with `re2c:decorate = 0;`.
- Generate semicolons in certain cases when they were missing.
- Add a TODO about generic API primitives for fixed tags.
Previously `yyeof` prefix was used for YYFILL labels in EOF rule mode,
but now they always use `yyFillLabel` prefix, so the shorter `yyeof`
is available for use in the EOF rule final state.
With `re2c:decorate = 1;` there should be no semicolon, as the user
should be able to fully control the generated text. With Go backend
there should be no semicolon because it is useless.
- Transform a few methods into functions to prevent accidental change
of class members (a function can accept parameters as read-only, or
not have them passed at all).
- Add methods `reset_<X>` for all options and use them.
- Add comments clarifying that user-defined options and default flags
should not be modified by the "fix" function.
- Reorder a few blocks in the "fix" function (no functional changes).
- Add a helper function for encodings.
This patch maked the following changes to code generation:
- Ensure that string `re2c:startlabel` configuration takes priority over
boolean `re2c:startlabel` configuration (this preserves the behavior
of previous re2c versions).
- In `-c` mode, if start label is forced, emit it only for the first
condition. For other conditions it's useless, as they already have
`yyc_<condition>` labels. Besides, numerical start labels would have
unstable numbers, and text start labels would have the same name for
different conditions (duplicate labels is an error).
Some tests have been removed as they are covered by the newly added
autogenerated tests.
For historic reasons in `-c` mode re2c started label enumeration from 1
rather than 0. Label 0 is reserved for the start state with `-f` option,
or if the start label is explicitly set with `re2c:startlabel`
configuration. In other cases there is no reason to skip index 0 (except
to avoid changes in the generated code). This patch makes a small change
in the code and drops the artificial increment.
The changes in tests are trivial: label numbers have been decreased
by 1 (or by 2, 4, etc. in tests with multiple reuse blocks). The
following perl script decrements all labels in the code generated by
re2c before the change (old.c) and compares the result (old1.c) with the
new code (new.c). The expected diff is empty. This doesn't work for all
tests (e.g. reuse mode), but in most cases it is enough to verify that
nothing else has changed:
perl -pe 's/(yy|YYDEBUG\()(\d+)/($1).($2-1)/eg' old.c > old1.c
diff old1.c new.c
Start label is needed in `-f` mode: it points to state 0 (the beginning
of block, before condition dispatch in `-c` mode). Initial label points
to the beginning of the DFA (after condition dispatch in `-c` mode).
Previously re2c reused the same label in cases when both start and
initial labels were needed and could be placed adjacent to each other.
Aliasing made it difficult to change the code, because changes of the
start label affected the initial label (and caused massive diffs in the
tests due to a shift in the label numbers).
This patch decouples the labels: now start label, if needed, has its own
label, and initial label always has it own label.
There are 29 changed tests. The changes can be grouped in 3 categories:
1. Trivial renumbering of labels. In most cases all labels except `yy0`
(which is the start label) have been shifted by 1. It is possible to
mechanically ensure that nothing else has changed using this script:
for f in `find . -name '*.orig'`; do
sed -r 's/(.*)(yy)([1-9][0-9]*)(.*)/cat << EOF\n\1\2$((\3 + 1))\4\nEOF/ge' $f > $f.mod
done
In most cases the ".mod" file (which is generated from the original
test result by incrementing all labels with sed) is identical to the
new tests result. In some cases manual verfication is necessary
because labels shifted by a varying number (e.g. due to multiple
reuse blocks in one file: shift was 2 for the first block, 4 for the
second block, etc.).
2. Flip of adjacent user-defined start label and re2c-defined start
label. This happened because previously re2c-defined start label
aliased to the initial label (which is emitted after the user-defined
start label), and re2c generated the the initial label (the start
label was hidden). Now there is no alias, and start label is no
longer hidden. The change is harmless because user-defined start
label and re2c-defined start label are always generated next to each
other.
3. Added jump from the start label to the initial label. Previosly the
state dispatch jumped directly to the initial label, and now it jumps
to the start label, and from there to the initial label. The extra
jump should be trivial to optimise for the compiler.
With `-f` each reuse block has its own state dispatch, and it needs its
own start label as well. Previously the initial state dispatch always
used label 0 (the start label of the first block), which is incorrect
for all but the first blocks.
This partially fixes#282 "Options -f, -c and -r do not combine well".
Previously the combination of `-r` and `-f` could not be used because
the generated code was inconsistent at least (or even incorrect). Now
that the combination has been fixed, it can be used for tests.
Since all reuse blocks have their own enumeration of YYFILL states
starting from zero and their own state dispatch (not shared with other
blocks, as in the case of non-reuse blocks), YYFILL labels in reuse
blocks have identical indices. This patch adds block ID infix to such
labels in order to prevent label collision if blocks are located in
the same function.
This partially fixes#282 "Options -f, -c and -r do not combine well".
This is necessary for consistency with YYFILL states (`-f` option),
which are block-local in reuse mode (`-r` option).
Re2c only recently started to allow mixing reuse and normal blocks,
and the primary user was re2c own lexer, so this change is unlikely
to break anyone.
This partially fixes#282 "Options -f, -c and -r do not combine well".
In `-f`, `--storable-state` mode re2c accumulates information about
YYFILL labels and fallback transitions (for EOF rule). For non-reuse
blocks this whole-program information is used to generate one common
state dispatch. Reuse blocks are handled differently: they have each
their own state dispatch, and YYFILL label numbers starting form zero.
Previously re2c incorrectly handled reuse blocks: instead of using
block-local list of states it used the list from the last block. This
patch fixes it.
This partially fixes#282 "Options -f, -c and -r do not combine well".
Last block options are used as "global" or "final" options in certain
cases for code generation: namely, when generating state dispatch and
condition enum, which are accumulated over all blocks. In reuse mode,
however, state dispatch is generated in every block, so it should use
block-local options.
This partially fixes#282 "Options -f, -c and -r do not combine well".
Exception: don't reuse mode for tests that set `-f` option and
check `YYGETSTATE` and initial state dispatch, as this mode is
broken (re2c always uses configurations of the last block).
Rework test generator for re2c configurations.
Also, don't reset options `re2c:decorate` and `re2c:placeholder` to
defaults (the latter is necessary in order to make the new tests pass).
A precondition to hoisting tag operations out of conditional branches
is that all transitions agree on tags, including the potential fallback
(final or default) transition. Therefore tag operations should not be
repeated on fallback transition. With s-tags repeating is redundant but
harmless, but with m-tags it could cause incorrect submatch results.
Previously with the combination of -f, --storable-state option and EOF
rule re2c generated `yyeof<k>` labels in the default branch of each
state. These labels were used as `goto` targets in the initial state
dispatch. Since Golang does not support jumps into nested blocks, this
patch removes `yyeof<k>` labels and inlines the corresponding fallback
transitions into the state dispatch.
This was a weird special case in order to preserve code generation
results during the major rework of the code generation subsystem. It is
no longer needed, as the changes in tests are trivial, concern only a
few skeleton tests and can be checked manually (they are caused by a
shift in label numbers).
`switch` statements occur in a few contexts in re2c output: switch on
input symbol, initial switch on conditions (with -c), initial switch on
state (with -f), switch on `yyaccept`. Previously different cases were
rendered in a slightly different way (some cases on the same line, some
on a new line; differences in spaces/tabs).
Now all `switch` statements are rendered uniformly (each case on a new
line), except for switch on input symbol which is unchanged (each cases
on the same line except for cases that contain multiline statements
like tag operations, skip with --eager-skip, or YYFILL with EOF rule).
It would be too difficult to change them, as they constitute the larger
part of re2c output. Also, this would make the generated code less
readable (one-line cases look condensed and aligned).
Massive changes in tests are trivial.
Previously goto elision was implemented by passing a null target state
instead of the actual DFA state to the code generation function. However
that prevented goto elision in some other cases with EOF rule, because
the decision whether to elide or not was based on the target state
(which was force-set to null).
This patch adds an explicit "elide" flag instead of the null "hint".
The combination of YYFILL with EOF rule and `-f` remains a problem due
to the Go limitation: the language disallows jumping into inner blocks
(re2c generates `yyeof<N>` labels inside of conditional branches).
Also, forbid using EOF rule with `re2c:yyfill:check = 0;`, as this
combination doesn't make sense (with EOF rule the check is necessary
to distingush sentinel symbol from EOF). Previously this configuration
was not supported (the check was generated despite the option).
The new tests are less uniform than the old tests: some of them contain
multiple blocks (when it is necessary to check the effect of previous
block settings on the next block), and some of them consist of multiple
files with a single block (when the tested part of the generated code
appears once per file, not once per block).
The new tests cover more cases: they add EOF rule and `re2c:decorate`,
`re2c:yyfill:check` configurations, as well as new combinations of
previously tested options.
All test cases are generated using one Haskell script.
Now instead of always defaulting to false, 're2c:*:naked' options
default to the value of 're2c:decorate' (unless they are explicitly set
by the user). The idea behind this change is to allow the user set one
option instead of a dozen, while still allowing a more fine-grained
control if necessary.
It is necessary to have different sets of default mutable options for
different language backends. For example, all Go programs need to set
generic API, so it makes sense to set it by default.
The difficulty in changing default mutable options based on the constant
(global) options is that re2c allows to intermix mutable and constant
options on the command line. Therefore by the time all constant options
have been parsed, some of the mutable options may have already been set.
It is impossible to tell which of them have been set from the option
values, because a user-set value may coincide with the default value.
Therefore this commit adds boolean flags indicating if a mutable option
has been explicitly set.
Also, add more tests for the Go backend that exercise the new option.
Update existing Go tests.
The new configuration has boolean values (zero is false, non-zero is
true). The default value is true. It is mostly useful with generic API.
It decides whether to generate decorations (e.g. parentheses "()") after
the invocation of YYPEEK, YYSKIP, etc. By setting 're2c:decorate = 0;'
it is possible to avoid functional style calls and generate more
free-form code for API primitives.
In the future this is meant to be an umbrella configuration for various
existing "re2c:define:X:naked" configurations.
Changes in the tests are trivial (despite large diffs). They are caused
by removal of redundant curly braces and one extra level of indentation
(previously generated in order to reduce the scope of bitmap
declaration, but useless because there are zero bitmaps in this case).
When EOF rule is used, a final state may be a fallback state even if
all outgoing paths from it are accepting, because the possibility of
YYFILL failure cretes additional default quasi-transition.
The "expand" pass generates code for stubs: parts of the code that
cannot be generated during the first code generation pass, because they
require some information that only becomes available after the full
program has been generated.
The "combine" pass folds sequences of simple skip/peek/backup statements
into complex statements. For example, it transforms two-statement
sequence 'yych = *YYCURSOR; ++YYCURSOR;' into 'yych = *YYCURSOR++;'.
Also, render yyeof<N> labels properly (without indentation) and don't
generate an empty statement after them unless they are at the end of a
block.
Previously labels of the form yyfillLable<N> were used only with -f,
--storable-state option. With EOF rule re2c generated label names
with underscores (unless -f was also enabled). Now re2c always uses
labels of the form yyfillLable<N>.
Although the current behavior was ok (only one of the arguments was
a complex expression, and it didn't rely on side-effects), such code
patterns are error-prone and may cause subtle bugs in the future.
Note:
Re2c uses C++98 standard. C++11 adds some guarantees in section 1.9.15:
(http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2011/n3242.pdf):
Every evaluation in the calling function (including other function
calls) that is not otherwise specifically sequenced before or after
the execution of the body of the called function is indeterminately
sequenced with respect to the execution of the called function.
And a footnote:
In other words, function executions do not interleave with each other.
But it is does not solve the problem of possible side-effects.
This is to avoid stack overflow on large RE (especially on instrumented
builds that have larger stack frames, like AddressSanitizer).
Partial fix for #219 "overflow-1.re test fails on system with small stack".
This is to avoid stack overflow on large RE (especially on instrumented
builds that have larger stack frames, like AddressSanitizer).
Partial fix for #219 "overflow-1.re test fails on system with small stack".
This is to avoid stack overflow on large RE (especially on instrumented
builds that have larger stack frames, like AddressSanitizer).
Partial fix for #219 "overflow-1.re test fails on system with small stack".
This is to avoid stack overflow on large RE (especially on instrumented
builds that have larger stack frames, like AddressSanitizer).
Stack overflow reported by Agostino Sarubbo.
Related to #219 "overflow-1.re test fails on system with small stack".
This is to avoid stack overflow on large RE (especially on instrumented
builds that have larger stack frames, like AddressSanitizer).
Stack overflow reported by Agostino Sarubbo.
Ensured that the new algorithm computes the same length as the old one
by adding temporary debug print of the computed length to both
algorithms and verifying that it coincides.
This is to avoid stack overflow on large RE (especially on instrumented
builds that have larger stack frames, like AddressSanitizer).
Stack overflow reported by Agostino Sarubbo.
This is to avoid stack overflow on large RE (especially on instrumented
builds that have larger stack frames, like AddressSanitizer).
Stack overflow reported by Agostino Sarubbo.
This allows to use EOF rule with multiple blocks in the same function
(previously re2c generated duplicate EOF labels, so the code did not
compile).
This fixes bug #280: "EOF rules with multiple blocks", reported by brk.
RelWithDebInfo keeps debug information, but also sets NDEBUG, which
disables asserts. For development, it is necessary to enable
optimizations, debug info and asserts (and also the additional debug
checks enabled with RE2C_DEBUG).
The crash happened in a rare case of a very long lexeme that doen't fit
into the buffer, forcing buffer reallocation.
The crash was caused by an incorrect calculation of the shift offset
(it was smaller than necessary). As a consequence, the data from buffer
start and up to the beginning of the current lexeme was not discarded
(as it should have been), resulting in less free space for new data than
expected.
CMake and Autoconf do not agree on the name of the variable: CMake uses
CMAKE_C_COMPILER, and Autoconf uses CC. Plain 'cc' doesn't work on systems
that do not have this alias, but it is unlikely that somenone will run
skeleton tests on such systems.
Previously the output program was represented as a list of blocks (one
block per each re2c comment in the source file). Each block consisted
of a list of heterogenous fragments. Some fragments contained text that
was generated immediately during the first pass of code generation;
other fragments were placeholders for parts of the program whose
generation was delayed (such as dispatch on YYFILL labels with -f option
that cannot be generated on the first pass because the full list of
labels is known only at the end of the first pass). Individual fragments
used ad-hoc methods to render the generated code, which made rendering
rather complex (for example, it was problematic to decide if the 'if'
statement should be rendered on a single line or on multiple lines).
This commit adds a new AST-like representation of the output program.
The top-level program is still a list of output blocks, but each block
is now an AST. This decouples code generation from rendering, which
simplifies both phases and allows easier customization for different
output language backends (such as C or Go).
Otherwise on Travis CI cmake-3.12.4 fails with errors like:
CMake Error at CMakeLists.txt:206 (install):
install TARGETS given no RUNTIME DESTINATION for executable target "re2c".
CMake Error at CMakeLists.txt:356 (install):
install TARGETS given no ARCHIVE DESTINATION for static library target
"libre2c_static".
CMake Error at CMakeLists.txt:365 (install):
install TARGETS given no LIBRARY DESTINATION for shared library target
"libre2c_shared".
The dependency should be relative to the build directory.
Tested as:
$ cmake -B build && cd build
$ make && make tests
(previously failed to build dependency run_tests.sh, now finishes
successfully).
Invoke it like this on Unix:
mkdir build
cd build
cmake .. \
[-DCMAKE_C_FLAGS=...] [-DCMAKE_CXX_FLAGS=...] \
[-DCMAKE_EXE_LINKER_FLAGS=...] \
[-DCMAKE_SHARED_LINKER_FLAGS=...] [-DCMAKE_STATIC_LINKER_FLAGS=...] \
[-DCMAKE_C_COMPILER=...] [-DCMAKE_CXX_COMPILER=...] \
[-DCMAKE_BUILD_TYPE=Debug|Release|RelWithDebInfo] \
[-DRE2C_REBUILD_DOCS=YES] \
[-DRE2C_BUILD_LIBS=YES [-DBUILD_SHARED_LIBS=YES|NO]] \
[-G Ninja]
make -j # or ninja, if "-G Ninja" was used
Or on Windows, install Visual Studio's CMake tools and then open this folder.
These labels are used by the code that handles unsuccessfull YYFILL:
the lexer falls back to the final state.
This fixes bug #276 "Example 01_fill.re in docs is broken"
(reported by Angelo Haller).
Changes:
- added an example of TNFA
- added an example of how mark() and enum() work
- changed the explicit submatch index to a set of indices to simplify handling
of repeated captures like ((e))
- changed benchmarks and their representation: use horizontal bar charts instead
of linear plots
- added naming conventions and renamed the variables throughout the paper
- re-worded the abstract and introduction
- multiple small tweaks
The comment suggested to avoid memset of boolean array, which elements
indicate if the tag has already been updated n the current step, with
an array of integers that store step number, and thus do not require
explicit zero-initialization (step numbers are growing monotonously).
After some expreiments this idea doesn't seem so good, because:
- Update function is called multiple times for the same step value (for
different configurations), thus requiring a more complex marker value
that would combine step number with configuration id.
- It doesn't seem to be the bottleneck (there is another memcpy, which
is unavoidable.
For example, (((x))) becomes (x), where the tag for the left parenthesis has
an associated range of submatch indices [0,2,4], and the tag for the right
parenthesis has an associated range [1,3,5].
The new option allows to specify the algorithm used to compute POSIX precedence
table. The 'complex' algorithm computes precedence table in one traversal of
tag history tree and has quadratic complexity in the number of TNFA states; it
is the default. The 'naive' algorithm has worst-case cubic complexity in the
number of TNFA states, but it is much simpler than 'complex' and may be slightly
faster in non-pathological cases.
The algorithm is described in the paper "STADFA: An efficient Subexpression
Matching Method" by Mohammad Imran Chowdhury.
The main difference between staDFA and TDFA is in the placement of tag actions:
staDFA places them in states, while TDFA places them on transitions. RE2C
implementation of staDFA differs from the description in the paper in a number
of aspects:
- We use TNFA with epsilon-transitions instead of Glushkov automaton used by
Chowdhury. Glushkov automaton is not ambiguity-preserving, unless it is
modified to have multi-transitions between a pair of states, which differ
only in the tag sequences assigned to them. Such multi-transitions are used
by the original Okui-Suzuki PAT, but there is a problem with this idea: the
number of multi-transitions may grow exponentially with the size of RE.
- We use the same disambiguation algorithms as those used for TDFA, not those
described by Chowdhury.
- We associate tag indices with TNFA states, and not symbols occurrences in RE.
In principle this is the same: for each occurrence of a given symbol there is
a distinct TNFA state (the same state corresponds to multiple symbols from
the same range, but not for multiple occurrences of the same symbol).
- We always use "delayed store" actions, as we need to know the values of tags
under repetition either on the last iteration (for s-tags), or on all
iterations (for m-tags).
- We compare lookahead tags when looking for an identical state.
- We compare POSIX precedence matrces when looking for an identical state.
- We use negative tags.
- We don't add "delayed store" actions for lookahead tags of the origin state
which do not have current transiiton symbol in their lookahead set.
- We create initial transfer actions for all tags and all sub-states of the
initial state. This ensures that there is an action for every final sub-state.
- We resolve cycles in tag commands using temporary variables.
Interference table is square in the number of tag variables, so 32-bit signed
indices into it overflow when the number of tag variables approaches 2^16.
With this fix overflow may still happen on platforms that have 32-bit size_t,
or on REs with too many tags.
Duplicate commands may arise as a result of tag optimizations, and should be
removed in all cases ("set', "copy" and "add" commands). In case of "set"
commands this can be done by sorting commands and removing adjacent duplicates.
In case of "add" commands sorting is not possible, but deduplication can still
be implemented using naive quadratic algorithm. Deduplication of "copy" commands
depends on the presence of cycles (if there are no cycles, it is possible to
sort commands lexicographically, then remove adjecent duplicates, then sort
commands topologically).
Accepting end pointer rather than assuming NULL-terminated list of commands is
convenient when the topsorted list is a sublist in the middle of a larger list.
Returning a pointer to the first non-trivial cycle is needed in ordert to break
the cycle by introducing a temporary variable; this is needed for implementing
staDFA.
This is needed in preparation for staDFA implementation. Variables local to
basic block may occur as a result of breaking cycles in a sequence of tag
commands by introducing temporary variables. TDFA mappings have no cycles
(such mappings are rejected), but staDFA transfer actions often do, because
the indices used in the actions are bound to TNFA substates, which may be
flipped in the origin and target staDFA states.
Ideally we should use different encoding (e.g. always allow 2 or 4 bytes
for history length), but that requires more complex serialization and
deserialization. Assert is a temporary measure.
M-tag pool is reallocated and moved when it runs out of space. However,
previously it contained an array of structs with pointers to elements of this
array. After moving these pointers where no longer valid.
In some cases this was causing increase in the number of tag variables after
optimizations, which is generally bad and also caused memory access violation
as the liveness/interference tables are allocated for the initial number.
The 0-th variable is superficial and does not correspond to a real tag version
(zero is a special version reserved for "no tag operation"). In the variable
allocation algorithm it is convenient to handle zero in the same way as other
variables (because arrays are indexed from zero). In most of the cases 0-th
variable ends up in a class that contains other (non-fake) variables, but if it
is alone in its class, then no tag variable should be allocated for this class.
This is needed in preparation for implementing staDFA algorithm, in which
tag operations are placed in states, not on transitions. The new CFG model
is able to accommodate both TDFA and staDFA, as well as a hypothetical
"mixed" case where tag operations occur both in states and on transitions.
This commit adds "canonical" marking of visited states and transitions, which
allows to avoid erroneous creation of dulicate arcs in the CFG.
Some of the helper functions have been rewritten from recursion to iteration.
This commit also adds CFG construction context to avoid passing too many
parameters to the helper functions.
With EOF rule, it is always possible that EOF happens before the next matching
rule, even if it matches all possible inputs. If EOF does happen, the lexer
needs to rollback to the last matched rule (so it is not dead).
Note: ninja_lexer.ib.c differs from the original lexer in the ninja project,
because the latter was generated with re2c-0.16, which used slightly
different constructs for complex expressions:
Re2c-0.16 used generated something like this:
++YYCURSOR;
if (yych = *YYCURSOR) ...
And the latest re2c generates something like this:
yych = *++YYCYRSOR;
if (yych) ...
The difference is insignificant.
Apparently some versions of bison generate token definitions as macros, not
as enum members. If a bison-generated header is included into another file
that contains identifiers identical to the names of parser tokens, all such
identifiers are macro-substituted to a numeric constant. Adding a prefix to
token names reduces the chances of name collision.
This fixes bug #261: "lib/parse.h:49:15: 错误：expected unqualified-id before
numeric constant" reported by qidizi.
The new analysis is meant to detect potential reads past the end of buffer
resulting from erroneous rules that allow sentinel symbol to occur in the
middle of lexeme. The analysis is only activated in cases bounds checking
is disabled with 're2c:yyfill:enable = 0;' and EOF rule is not used.
For example:
['] [^']* ['] { ... }
This may be a naive way to match single-quoted strings; it would crash the
lexer on input like "'aaa\0" (without the closing single quote).
This is important when re2c is packaged in distributions, so that one can
quickly check what license the re2c is distributed under. Distributions
typically use the release tarball, not the git repository, and so this commit
puts the LICENSE file into the release tarball also.
The ability to reset this configuration is necessary in order to be able to
mix re2c blocks that use EOF rule and blocks that don't.
This fixes bug #254: "Turn off re2c:eof".
Touched the following papers:
- 1994 paper "RE2C: a More Versatile Parser Generator"
- 2017 paper "Tagged Deterministic Finite Automata with Lookahead" (after arXiv publication)
- 2019 paper "Efficient POSIX Submatch Extraction on NFA"
All .pdf and .ps files have been moved to gh-pages-gen branch.
Note: in the non-EOF-rule case 're2c:yyfill:enable = 0;' results in suppression
of the whole 'YYLIMIT' check together with 'YYFILL' call. In case of EOF rule,
however, we are not bound by backward compatibility and can do a more logical
thing.
The current fix may sometimes cause generation of unused default label, because
at the time when it's generated we haven't geerated the code for EOF checks yet
and do nt know if default label is really needed. This is what happens in the
updated test cases.
Currently we need both libtool and slibtool:
- libtool fails to link self-contained DLL (it ignores '-static-libstdc++
-static-libgcc'), but manages to link EXEs with static library
- slibtool manages to link self-contained DLL, but then fails to link the
final EXEs with it (it passes '-lre2c' before some other predefined object
file that contains definitions from libgcc)
Instead, add only one negative tag (top-level closing one) and record
the remaining tags in its metadata. Use this metadata to update nested
negative tags during TNFA simulation.
Previous order was incorrect: fictive tags for both left and right
alternative or catenation were added before recursing into the right
alternative or catenation. This resulted in right fictive tags habving
higher precedence than left non-fictive tags.
In particular, test ((((a?)+)|(aa))+) and string "aaa" demonstrates why
we cannot replace SSSP (GOR1 or GTOP) with a liner-time DFS that sets
predecessor chains of TNFA states and sorts initial closure configurations
according to POSIX relation imposed by precedence matrix.
Consider what happens after consuming the first to 'a's (see the .dot
encoded TNFA below):
- initial states are 8, 14 and 9 (in the order imposed by D-matrix)
- start DFS from 8
- find path 8->7->6->5->4->3->2->22->21->11->10
- find path 8->7->6->5->4->3->2->22->21->20->19->18->17->16->15
- find path
8->7->6->5->4->3->2->22->21->20->19->18->17->16->14->13->17
compare it to the previous path to 17 and discard the new looping path
- find path
8->7->6->5->4->3->2->22->21->20->19->18->17->16->14->13->12->3
compare it to the previous path to 3 and discard the new looping path
- find path 8->7->6->5->4->3->2->1->0
- DFS from 8 complete, start DFS from 14
- find path 14, compare it to the previous path to 14 and accept the new
one as it is better (but do not attempt to propagate the improvement
and rescan states: this is DFS, not GOR1, and the whole idea is to
avoid rescanning and have linear complexity)
- DFS from 14 complete, start DFS from 9
- find path 9
The end, and we have an incorrect path to 15: the long path through the
outer loop from 8, rather than the short path through the inner loop
from 14. This further leads to incorrect results.
digraph NFA {
rankdir=LR
node[shape=Mrecord fontname=Courier height=0.2 width=0.2]
edge[arrowhead=vee fontname=Courier label=" "]
n23 [label="23"]
n23 -> n22 [label="/0↑(1)"]
n22 [label="22"]
n22 -> n21 [label="/2↑(2)"]
n21 [label="21"]
n21 -> n11
n21 -> n20 [color=lightgray]
n20 [label="20"]
n20 -> n19 [label="/4↓(3)"]
n19 [label="19"]
n19 -> n18 [label="/5↓(2)"]
n18 [label="18"]
n18 -> n17 [label="/6↑(3)"]
n17 [label="17"]
n17 -> n16 [label="/8↑(4)"]
n16 [label="16"]
n16 -> n15
n16 -> n14 [color=lightgray]
n15 [label="15"]
n15 -> n14 [label="97"]
n14 [label="14"]
n14 -> n13 [label="/9↑(3)"]
n13 [label="13"]
n13 -> n17
n13 -> n12 [color=lightgray]
n12 [label="12"]
n12 -> n3 [label="/7↑(2)"]
n11 [label="11"]
n11 -> n10 [label="/4↑(3)"]
n10 [label="10"]
n10 -> n9 [label="97"]
n9 [label="9"]
n9 -> n8 [label="97"]
n8 [label="8"]
n8 -> n7 [label="/5↑(2)"]
n7 [label="7"]
n7 -> n6 [label="/6↓(3)"]
n6 [label="6"]
n6 -> n5 [label="/8↓(4)"]
n5 [label="5"]
n5 -> n4 [label="/9↓(3)"]
n4 [label="4"]
n4 -> n3 [label="/7↓(2)"]
n3 [label="3"]
n3 -> n2 [label="/3↑(1)"]
n2 [label="2"]
n2 -> n22
n2 -> n1 [color=lightgray]
n1 [label="1"]
n1 -> n0 [label="/1↑(0)"]
n0 [label="0"] [fillcolor=gray]
}
In C/C++ rename() behaviour is implementation-defined. POSIX and Windows
have different behaviour when destination file exists: POSIX says rename()
should overwrite it, but Windows says rename() should fail.
Some newlines originate in user-defined code (including semantic actions
and code fragments in configurations and directives), some are generated
by re2c itself. In order to maintain consistency we convert all newlines
to LF when writing output to file.
This is more conveniet for build systems: if code generation is interrupted
for some reason, we don't end up with half-generated or empty output file.
Note that re2c there still can be discrepancy between the generated header
file and output file.
Previously used ~0U works incorrectly in such cases: ~0U equals
0xFFFFffff, and maximum (numerical) pointer value is 0xFFFFffffFFFFffff.
This caused incorrect comparison result when tracking include files that
have been fully processed and can be removed frm the include stack.
Problem report thanks to Denis Naumov.
The old alrorithm was straightforward: it computed precedence for each
pair of core closure items. This has worst-case cubic complexity in the
size of TNFA, because closure can have O(N) states and tag history length
can be also O(N). One example is ((a?){1,1000})*.
The new algorithm avoids that by computing parecedence for all pairs of
core items in one traversal of the shortest path tree constructed by
closure. It avoids traversing deadend branches (those that were cancelled
by search because a better path was found) by marking used nodes of the
tree from each leaf down to the root. The new algorithm has worst-case
quadratic behaviour. It can be slightly slower for simple cases when the
number of core items is low and histories are short.
The regexp is ((a?){1,200})*, and the input string is just "a".
Takes quibic time in the size of counter. This is caused by quadratic-
time computation of precedence matrix on each step (the number of TNFA
states in the closure approaches TNFA size), multiplied by the length
of compared histores (which also approaches TNFA size). Trie-based
algorithms are not affected, but they consume memory proportional to
the length of the input string, and so are also not practical.
Histories are stored in a trie. Each history is represented by an index
to its tail in the trie. In order to compare two histories, we can simply
unwind both indices, step by step, until they coincide or root index is
reached.
Related commits in libre2c:
062da2dca62a7158b401
Previously we used to unwing each history, store it as a linear sequence
in a separate buffer, and then iterate buffers from the beginning of
histories to fing the fork point.
This reconstruction step can be avoided, because we can find fork by
tracing both histories together from tail to root, element by element,
until their indices coincide (this is the fork point. Someone called
thos "two-fingers algorithm".
It doesn't show any difference on the tests, probably because previously
these values were set to zero, which unpacks to symmetrical (though
incorrect) values, and thus doesn't affect comparison.
During epsilon closure constructions, many operations only need to be
performed for core states. This allows to save some memory, e.g.
precedence matrix can be indexed by core states.
This option controls the formatting of locations in error messages
and warnings.
With --location-format gnu:
filename:line:column: ...
With --location-format msvc:
filename(line,column): ...
A lot of the match time was taken by eagerly computing precedence values
on each step for each pair of closure states. Most of these values are not
needed: RE may be unambigous; or the given input string may be unambigous,
even if there is ambiguity, it may take only a few comparisons to resolve.
All the rest is wasted effort.
We can avoid it by delaying precedence computation until necessary, and
then unwinding all the steps backwards, computing precedence for each step
and caching the computed values (so that the same pair of histories is not
compared twice). It is still the same incremental comparison as with
precedence matrices: we compare step by step, going from the fork frame to
the join frame. The result of comparison on each step is folded to a triple
of numbers and recorded in cache. It is important that we do record each
step, not just the first step, because the next pair of ambiguous histories
may unwind to the same pair of prefixes that was compared before.
For all this to work, it is necessary that we can store all history until
the very end, because at any step we might need to unwind an arbitrary
number of steps back. We also need to address individual subhistories
efficiently in order to use them as keys in the cache. All this is achieved
by storing history in the form of a trie and addressing individual
histories by indices in the trie. We also use trie to compute the resulting
tag values (instead of storing tags in registers at each step).
This is needed because we may want to use POSIX syntax with leftmost
greedy disambiguation. In that case, we still need parentheses-as-tags
attitude, but we don't want all the overhead and heuristics to spped
up POSIX closure.
libre2c_posix: added test for leftmost TNFA matcher.
POSIX standard says that:
The regex_t structure is defined in <regex.h> and contains at
least the following member: re_nsub (number of parenthesized
subexpressions).
Added fields:
- re_nsub: total number of submatch groups, required by POSIX standard
- pmatch: buffer for submatch results, usually supplied by the user to
regexec(), but we allow to do the allocation and storage in regex_t.
This is convenient for users that have hard time managing memory, e.g.
java bindings to libre2c_posix.
- regs: buffer for internal use by regexec(), strored in regex_t to
avoid repeated memory allocation on each call to regexec() with the
same regex.
Adjustments:
- configure.ac: pass win32-dll option to LT_INIT
- Makefile_libre2c_posix.am: use -no-undefined in LDFLAGS
- use slibtool: https://github.com/midipix-project/slibtool for windows
builds The problem with libtool is that it doesn't allow to link libstdc++
and libgcc statically, which is necessary to build portable DLLs with
Mingw. Libtool adds -nostdlib option to LDFLAGS and links some predefined
objects that pull in dependency on dynamic libstdc++ and libgcc, even in
the presence of -static-libstdc++ -static-libgcc.
Operator precedence was broken because re2c tried to parse whole strings
of characters at once instead of parsing one character at a time (in
much the same way as it would parse properly quotes string literals in
the original re2c format). This caused ab* being parsed as (ab)*, which
is clearly wrong (should be a(b)*).
This fixes bug #242:
"Operator precedence with --flex-syntax is broken."
Variadic functions cause subtle toolchain-specific errors because integer
literals used to initialize offsets may have different type than the type
expected by variadic function (e.g. int vs long).
Instead of using two stacks to weed out low-precedence initial
configurations with duplicate target state, let them remain on the
bottom of topsort stack, which effectively makes them ignored.
GTOP SSSP meand "Global Topsort Single Source Shortes Path".
It is well known that SSSP can be solved in linear time on DAGs (directed
acyclic graphs) by exploring graph nodes in topological order. In our case
TNFA is not a DAG (it may have cycles), but it is possible to compute fake
topologcal order by ignoring back edges.
The algorithm works by having a priority queue of nodes, where priorities
are indices of nodes in fake topological ordering. At each step, the node
with the minimal priority is popped from queue and explored. All nodes
reachable from it on admissible arcs are enqueued, unless they are already
on queue.
The resulting algorithm is of course not optimal: it can get stuck on
graphs with loops, because it will give priority to some of the loop nodes
compared to others for no good reason.
However the algorithm is simple and optimal for DAGs, therefore we keep it.
In lexer buffer nested files come before outer files. In lexer file
stack, however, outer files go before nested files (nested are at the
top). We want to break from the unreading cycle early, therefore we
proceed in reverse order of file offsets in buffer and break as soon
as the end offset is less than cursor (current position).
This fixes bug #229: re2c option -F (flex syntax) broken,
reported by Robert van Engelen.
A well-formed example that caused syntax error (flex-style raw literal
followed by one or more spaces and a curly brace):
/*!re2c
a {}
*/
The faulty behaviour goes back as far as re2c-0.13.6 (and supposedly
before that): in flex mode, raw literal may occur in various contexts
both as a regexp (string literal) and an identifier (named definition,
condiiton name). RE2C uses lookahead to infer the context and determine
the appropriate type of lexer token, but it missed some cases.
The fix has two sides. First, if reduces the number of contexts where
the general lexer may encounter raw literal (by using specialized lexers
for condition lists <x,y,...,z> and condition goto => and :=>). Second,
it fixes the lookahead regexps used for context inference.
Also added a bunch of tests (generated by a script).
The error was caused by assuming that a sequence of zeroes (used for
padding in YYFILL) cannot form a valid lexeme suffix. This is not the
case with strings, as they may contain arbitrary characters. The fix
is to manually loop over string characters in lexer, stopping at each
zero to check if it's the end of input.
Found by american fuzzy lop (thanks to Henri Salo).
```c
longest | (leftmost << 30);
```
assumes `longest` does not exceed 30 bits. It could if
it's a negative value originally.
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
signed shift overflow is not defined by C standard.
clang++ -fsanitize=undefined detects it as:
```
src/dfa/closure_posix.cc:207:32: runtime error: left shift of negative value -1
```
This change wraps bit shift arithmetics into unsigned types.
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
`xz` compresses twice as good as `gzip` on `re2c` sources:
```
$ ls -lh *1.1.1*
4,8M re2c-1.1.1.tar.gz
2,5M re2c-1.1.1.tar.xz
```
Switch `make dist` to `xz by default. `gzip` is still available
via `make dist-gzip`.
Reported-by: rofl0r
Bug: https://github.com/skvadrik/re2c/issues/221
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
On g++-3.4.6 re2c tests SIGSEGVed due to use of uninitialized data:
```
$ valgrind ... ./re2c -8 a.re -o foo.c
Conditional jump or move depends on uninitialised value(s)
at 0x432F23: re2c::tcpool_t::insert(re2c::tcmd_t const*) (tcmd.cc:202)
by 0x421FDA: re2c::freeze_tags(re2c::dfa_t&) (freeze.cc:45)
by 0x43A7FF: re2c::ast_to_dfa(re2c::spec_t const&, re2c::Output&) (compile.cc:88)
by 0x43B052: push_back (stl_iterator.h:614)
by 0x43B052: re2c::compile(re2c::Scanner&, re2c::Output&, re2c::Opt&) (???:0)
by 0x449D29: main (main.cc:31)
Uninitialised value was created by a heap allocation
at 0x403252F: operator new[](unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x42FC9E: re2c::find_state(re2c::determ_context_t&) (dfa.h:37)
by 0x429BD9: re2c::dfa_t::dfa_t(re2c::nfa_t const&, re2c::opt_t const*, std::string const&, re2c::Warn&) (determinization.cc:56)
by 0x43A76C: re2c::ast_to_dfa(re2c::spec_t const&, re2c::Output&) (compile.cc:69)
by 0x43B052: push_back (stl_iterator.h:614)
by 0x43B052: re2c::compile(re2c::Scanner&, re2c::Output&, re2c::Opt&) (???:0)
by 0x449D29: main (main.cc:31)
```
the problem here arose in default array constructor:
```c++
explicit dfa_state_t(size_t nchars)
: // ...
, tcmd(new tcmd_t*[nchars + 2]()) // +2 for final and fallback epsilon-transitions
// ...
```
g++-3.4.6 can't figure out zero-initialization rule (likely a gcc bug).
The change uses non-initializing new[] and memset() instead.
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
The error was in the code of the test itself: the special case of zero
wasn't handled correctrly by the function that prepares input data for
the test. As a result, zero-length input string was passed to the test,
which is unexpected: the tested function is an "unsafe" one (as the
name suggests) and is meant to be used on an already validated input.
RE2C used to perform the following optimization: when a POSIX capture is
under iteration, we only need to get tag values of the last iteration
(according to the POSIX standard). Therefore we can move the closing tag
out of loop.
This commit removes this optimization (as part of the effort to switch
from Kuklewicz POSIX disambiguation algorthm to Okui algorithm).
In other words, for RE (x)* re2c used to generate this "optimized" IRE:
1 (3 x)* 4 2
and now it generates the "canonical" IRE:
1 (3 x 4)* 2
Updated tests for '--posix-captures' that have been affected by the change.
Also, depth-first traversal was done in a slightly incorrect way:
we checked outgoing nodes for admissibility and pushed the corresponding
child states on stack all at once. This is not the same as checking
the first child and recursing into it, then checking the next child,
..., and so on (because we might discover the second child while exploring
the first, and admissiblitiy check for the second child *after* that
might yield false, while *before* exploring the first child it yielded
true).
This also fixes a error in the generation process: sometimes in case
of loops the current node's suffix was set before all of its children
were processed.
Updated test results (in some cases .input files became larger because
of the above fix, in some cases they became smaller because we now pick
the shortest suffix).
Added new test; this one was found by slyfox's fuzzer and revealed the
above bug.
Before tags were added to re2c, skeleton programs only used a limited
number of predefined names, such as 'yych', 'yystate', etc. With tags,
however, this is no longer true as tags may have any names. So now we need
to be more cautios when picking names for sekleton variables.
This patch is only a workaround to make all tests pass; the real solution
requires inventing a good naming scheme for skeleton programs and
regenerating all skeleton test results.
The error was found by slyfox's fuzzer (a randomly-generated skeleton test).
The bug in the code was, apparently, too early modification of the state's
estimated maximal distance to the end states: the distance was set before
all of the state's children were processed, which resulted in aborting the
accumulation of distance from the remaining children, and, as a consequence,
shorter than necessary max distance for the root itself.
The correct behaviour was broken somewhere in between 0.16 and 1.0:
re2c was forgetting to output the chunk of input file that precedes
the #line directive.
Reported by pskocik in #98.
GCC-4.2.1 is unable to compile code like this:
std::vector<int> v;
std::vector<int>::const_reverse_iterator i;
for (i = v.rbegin(); i != v.rend(); ++i) ;
It's unable to deduce const overload for 'rend':
"no match for ‘operator!=’ in ‘i != std::vector<_Tp, _Alloc>::rend()"
However, the following code compiles fine:
std::vector<int> v;
std::vector<int>::const_reverse_iterator i = v.rbegin(), e = v.rend();
for (i != e; ++i) ;
This was reported by Ryan Shmidt.
With '--posix-captures' RE2C stores submatch results in 'yynmatch'
(the total number of capturing groups for the matching rule) and
'yypmatch' (an array of submatch values for each group).
These variables should be user-defined, so that users can override
default implementation (e.g. make 'yypmatch' an array of integer
offsets rather than an array of pointers). Overriding is only possible
with generic API: if default API is used, then RE2C can autogenerate
'yynmatch' and 'yypmatch' (and so it did prior to this commit).
However, it is better to have the same behavior with both APIs; also,
it is coherent with '--tags' option (RE2C leaves tag definition to
the user).
Broken by commit fffb5932ee.
Path length were initialized with 0 instead 'DIST_ERROR', which caused
incorrect calculation of maximal path length. This in turn caused errors
in estimating the number of byted necessary to hold keys during data
generation in skeleton. The resulting keys were one-byte while maximal
path length was more than one byte, which (fortunately!) caused runtime
errors in skeleton programs.
Example of program that caused skeleton error:
/*!re2c
(@t [\x00] [^]{5,6})* {}
*/
The error was hidden for so long because in practice inputs that need
more than one-byte keys are rare, and fuzzer sets 'ulimit -t 10' when
running re2c, so most of such programs were simply aborted. Those that
were not aborted still had a chance of estimating key size correctly.
Use '$(@D)' instead of '$(dir $@)', as the latter requires secondary
expansion feature specific to GNU make; it causes build failures on
bmake.
'$(@D)', on the other hand, is a POSIX make feature documented e.g. in
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html
(section "Internal Macros").
The fixed rules do not presently trigger any build errors: target directory
is created by configure as it need to put autogenerated files doc/manpage.rst
and doc/help.rst in it.
However, this behaviour is occasional: if one removes the .in files, build
failure would be unmasked. So it makes sense to ensure that target directory
exists.
In some situations src/parse/ may not exist before a file is copied into the
directory. Ensure that this doesn't happen by creating the directory first.
Comparator is used in 'std::sort'.
All items in closure have unnique TNFA states, therefore we assumed
that the compared items must always be different. However, 'std::sort'
does not have this requirement and some implementations of it compare
the element with itself.
The removed assert caused crashes with old version of GCC (4.2).
Thanks to Sergei Trofimovich for debugging the issue.
In some situations src/parse/ may not exist before a file is copied into the
directory. Ensure that this doesn't happen by creating the directory first.
With CXXFLAGS='-fsanitize=undefined' GCC complains about unaligned access:
if custom allocator is used to allocate structs or other alignment-sensitive
things, then it must take care of the alignment (for example, add padding
to all unaligned blocks of memory which it allocates).
As for now, newline inside of a character string or class is an error:
re2c should emit clear error message. Different styles of newlines
should be recognized ("\n", "\r\n").
This commit fixes bug #162 reported by pauloscustodio:
Reading files with "rb" causes issues in Windows
This fix consists of two issues, both reported and fixed by pauloscustodio.
1. #162 "Open text files with "wb" causes issues on Windows"
Text files need to be opened for writing with "w", so that stdio does
the right thing in respect to the correct line endings for the current OS.
("\r\n" in Windows, "\n" in Linux).
2. #163 "Reading files with "rb" causes issues in Windows"
re2c reads input files in binary mode and writes the generated output in
text mode. This caused CR LF conversion to CR CR LF on Windows: first CR
comes from reading input in binary mode, second CR is added when writing
output in text mode. This only happened to those parts of input which are
not transformed by re2c: we used to copy-paste verbatim, now we patch line
endings. Now we convert all line endings to LF before writing the generated
code to file.
As suggested by sirzooro:
Please add symbol name to "can't find symbol" error message,
it would allow to quickly spot what is wrong. Now we have to
position cursor at given row and column to find that name.
Also tweaked error reporitng function to append "..." at the end
of the message if it didn't fit into buffer.
when default rules with conditions are used".
Default rule is handled in a special (delayed) way;
re2c uses different code for default rule than for normal rules.
This special code simply forgot to add condition name to the list
of conditions.
Thanks to sirzooro for bug report.
`make distcheck` protects source files from writing.
Test script run_tests.sh copies source files into build directory,
but the copied files inherit permissions, so `make distcheck` fails.
Line endings in the generated code depend on the target platform: e.g.,
"\r\n" on Windows vs. "\n" on Linux. However, reference test results are
(currently) generated on Linux and therefore contain "\n" line endings.
So we have to patch line endings in the generated code in order to pass
the tests on Windows.
Testing script did patch line endings in stdout and stderr, but forgot
to patch them in the generated file (it was broken since we started to
use '-o' option for testing). This commit fixes testing script.
It also deletes a couple of tests in which source code contains "\r\n"
instead of "\n". These tests are duplicates of other tests (they were
added by commit bd2875441c supposedly to
test that re2c preserves line endings in source code). They are broken
by current commit and fixing them is probably not worth of the effort.
Thanks to Abs62, who noted that under Windows (in MSYS) tests fail
because '2>"$outc.stderr"' dumps CRLF to file instead of LF
and proposed a fix:
sed -i 's/\r//g' "$outc.stderr"
Updated tests. Some error messages are more precise now, e.g. ill-formed
character classes and escape sequences: column points to the beginning
of the faulty lexeme rather than to the middle of it where the error
occured. Other error messages are less precise (lack column info), but
the column reported before was too inexact and didn't make much sense.
Previously we needed a different algorithm for non-orbit tags, because
disambiguation was based on both start and end tags. Non-orbit start tags
cannot be compared incrementally, like orbit tags, becuse default value
may be discovered on a later step than non-default value. Non-orbit end
tags do not have this problem: since negative tags are inserted at the end
of alternatives, default value is always discovered on the same step as
non-default value (provided that all higher-priority tags agree and
comparison reaches this tag at all).
Now that start tags are ignored, we can use incremental comparison for both
orbit and non-orbit subhistories, which simplifies the code.
Don't add closure items to "shadowed" set if there is an identical
"unshadowed" item: otherwise Goldberg-Radzik algorithm generates too much
"shadowed" items and the output becomes too noisy.
With leftmost greedy policy we can use simple depth-first search.
With POSIX policy we need Goldberg-Radzik algorithm, which is more complex
(and the necessity to accommodate both policies complicates it even more).
We forbid cycles of length 2 or more because they would need temporary
variable local to the basic block, which would complicate liveness
analysis. However, trivial cycles don't need a temporary.
Their position is fixed on some higher-priority tag (except the very
first tag, but in RE2C match is always anchored).
We cannot skip orbit start tag because the corresponding orbit end tag
is hoisted out of loop (by construction) and is, in fact, non-orbit.
It has O(M*N) worst-case complexity, where M is the number of nodes (states)
and N is the number of arcs (transitions).
Papers: 1993, "A heuristic improvement of the Bellman-Ford algorithm"
by Goldberg, Radzik and 1996, Shortest paths algorithms: Theory and
experimental evaluation" by Cherkassky, Goldberg, Radzik.
Comparing the last pair of subhistories is not enough: because of
the shortest-path algorithm mismatch may occur at some earlier point
(not at the last pair), in which case we still need to re-run the search.
Before we simplified POSIX disambiguation by inserting missing
captures, using three tags to represent captures under iteration
was the only way: we needed to guard where the first iteration starts,
where the last iteration starts and where the last iteration ends.
Therefore we used three tags: opening, closing and orbit.
Now that we insert missing pieces of capture hierarchy, we no longer
need a special opening tag for captures under iteration: there always
is a higher-priority tag that guards the start of the first iteration.
Massive test updates are mostly due to the fact that orbit tag is no
longer fixed (fixed tags go in the end): ~100 out of 147 tests.
Some other ~45 tests have two tag variables flipped. There are 4 tests
with non-trivial changes. All updated tests pass POSIX test
(/test/posix_captures/.run/__run.sh).
Mixing doesn't make much sense (there is no obvious use case).
It might be possible, but for now better forbid it before someone
starts to use it and it turns out to be impossible or very complex.
POSIX treats captured and non-captured subexpressions on equal terms.
However, non-captured subexpressions have no tags to guard them.
Previously we used default tags to infer ambiguous paths that correspond
to the missing captures: if one path had default tag and the other did
not, then desicion which path is better was made according to the leftmost
strategy. This algorithm works because in POSIX expressions without
captures have the property that leftmost path is the best path (for
example, 'a?' is greedy, it means 'a or epsilon').
However, this algorithm has one downside: because we may need leftmost
comparison, we have to impose leftmost order on NFA substates of each
DFA state, as well as maximize and orbit order for tags. This prevents
us from mapping perfectly mappable DFA states and we end up with a larger
DFA (which is sometimes folded back to smaller DFA, but not always).
Also, default-leftmost algorithm is more complex than inserting missing
hierarchy pieces: proving that it works is non-trivial.
In this case, instead of comparing histories by the leftmost criterion
(as it should be, if only one of them is bottom), we should assume that
from standpoint of this tag histories are equal and move on to other
tags.
The previous algorithm waited until the full epsilon-path is built,
then compared it to already existing paths. The new algorithm compares
and merges partial epsilon-paths as soon as they arrive at the same
NFA state, so the overall number of paths is bounded by the number of
NFA states at all times.
This is necassary for correct comparison of orbit tag histories:
if orbit tag is nested in an outer capture, this outer capture is
under repetition and there is an epsilon-path through it, then
this epsilon-path may contain pieces of orbit history that belong
to different iterations of outer capture; these pieces will be
glued together and the boundary between them will be lost.
Example: ((""){0,3}){0,2}.
However, in a common history we can always find boundaries
(they are marked by tags that correspond to outer captures).
If normalization of 'save' commands removes the last 'save' command,
then pointer to 'next' will point to the 'next' field of the removed
command and the chain of commands will break.
The previous attempt was to forbid 2-cycles; clearly it's not enough
and cycles of length greater than 2 also pose a problem.
We could allow cycles and deal with them by introducind a temporary
variable. However, this would create variables local to basic blocks,
which would complicate liveness analysis and dead code elimination.
Commands in the same basic block may have equal right hand sides,
e.g. 'x = y; z = y;'. Clearly in this case 'x' and 'z' do not interfere:
both versions are equal to 'y'. This optimization allows to deduplicate
cases like '("a" @x @y @z "b")*', where multiple tags have the same
values.
However, it is insufficient to find all commands in the current block
with RHS equal to current RHS. An example when this algorithm fails:
'x = y; w = z; z = y;', here 'x' and 'z' are assigned to the same RHS
'y', but if we merge them into one variable, 'w' will get the wrong
value.
The fix is as follows: first, ignore subsequent commands and consider
only commands that precede current command: if subsequent command that
sets LHS to the same value precedes any use of it, liveness propagation
through basic block would mark this LHS as dead and not interfering
anyway; otherwise (if use precedes setting to the same value), then it
indeed interferes with current LHS. This fixes the above example
'x = y; w = z; z = y;': when analysing 'x = y;' command we find that
'z' is alive (as before), but ignore subsequent 'z = y;' command (due
to the fix).
Second, when analysing preceding commands, calculate actual value for
each LHS (based on pessimistic assumption that on entry of basic block
all used versions are different). Formal RHS of some preceding command
may coincide with current formal RHS, but their values might differ:
'x = y; y = w; z = y;', here 'x = y;' and 'z = y;' have identical formal
RHS, but the value may be different. On the other hand, formal RHS
might be different, while their actual values are equal, as in
'x = y; w = y; z = w;'.
`make distcheck` protects source files from writing.
Test script run_tests.sh copies source files into build directory,
but the copied files inherit permissions, so `make distcheck` fails.
Don't rely on the fact that topsort leaves all-zero in-degree:
it doesn't if cycles are present. One could manually zero the remaining
entries after topsort, but zero initialization is more reliable.
And it's quite efficient since we only need to initialize the entries
we use.
When mapping states, we need to create a list of 'copy' commands.
However, if there is a 'save' command with LHS equal to RHS of the 'copy'
command, then instead of the 'copy' command we just fix LHS of the 'save'
command, as explained in this note:
/* note [save(X), copy(Y,X) optimization]
*
* save(X) command followed by a copy(Y,X) command can be optimized to
* save(Y). This helps reduce the number commands and versions (new version
* X is gone), but what is more important, it allows to put copy commands
* in front of save commands. This order is necessary when it comes to
* fallback commands.
*
* Note that in case of injective mapping there may be more than one copy
* command matching the same save command: save(X), copy(Y,X), copy(Z,X).
* In this case save command must be replicated for each copy command:
* save(Y), save(Z).
*
* For each save(X) command there must be at least one copy(Y,X) command
* (exactly one case of bijective mapping). This is because X version in
* save(X) command must be a new version which cannot occur in the older
* DFA state. Thus all save commands are transformed (maybe replicated) by
* copy commands, and some copy commands are erased by save commands.
*
* This optimization is applied after checking priority violation, so it
* cannot affect the check.
*/
However, the previous implementation would sometimes erroneously re-fix
the already fixed 'save' commands. This patch rewrites it in a way that
avoids re-fixing.
For simple tags without history 'save' command is self-sufficient:
it does not depend on anything. However, for tags with history 'save'
commands depend on history, which may be overwritten on the way from
accepting state to fallback transition. We must take care and backup
such overwritten histories when leaving the accepting state.
If history is not overwritten, we don't need a copy, but we still
have to propagate its liveness forward on all fallthrough paths
outgoing from accepting state.
First, fallback 'copy' commands should go before 'save' commands.
Second, codegen should not reorder 'copy' and 'save' commands.
These two bugs shadowed each other and everything seemed to work.
- removed 'YYCOPYTAG', use simple assignment insted
(it is hard to think of any other definition)
- renamed 'YYBACKUPTAG' to 'YYTAGP'
- added 'YYTAGN', 'YYTAGLISTP', 'YYTAGLISTN'
- removed 're2c:tags:default' configuration
This commit reverts commit 0b2ca9bf14:
"Pack lookahead tags together with tag versions in TDFA state."
Packing was possible because we were not interested in tag version
if closure item had this tag in lookahead set. However, if we are
to store the full history of each tag, we'll need this intermediate
version in spite of the fact that it is overwritten on the next
transition.
Liveness flows backwards through basic block, so it should be
propagated backwards starting with live-out set and applying
commands one by one in reverse order.
Previously we used separate lists for them; however the structure
of 'save' and 'copy' commands is almost identical, so its easy to
use one C struct for both types. Also, this simplifies analysis:
both types of commands are handled in a uniform way.
However, now we have to do list transformations like sorting and
deleleting duplicates in chunks.
This is a preliminary step to allow intermixing 'copy' and 'save'
commands.
Before this commit leftmost order was maintained implicitly by construction
of closure: NFA is traversed in left-first order, so the earliest item added
to closure is also the leftmost. Special care was taken to preserve this
order and never rearrange closure elements.
This is convenient if leftmost disambiguation strategy is the only one.
However, other strategies require explicit bookkeeping of disambiguation
information (POSIX needs tag history for orbit tags, maximize/minimize
needs most recent tag versions). Implicit handling of leftmost strategy
does not fall in line with other strategies.
By tracking leftmost order explicitly we achieve several goals:
- uniform handling of disambiguation strategies
- DFA states can be treated as good old sets again, which is more
conventional and means less work for minimization
POSIX tests changed (but the '/test/posix_captures/.run/__run.sh' script
still passes all tests) because there was an error in the order items were
added to closure, which probably prevented minimization (if the new item
dominated the existing one, the old one was substituted with the new one,
while it should have been removed and the new one appended at the end).
Skeleton tests changed because the order of NFA states in DFA state does
not matter, so more states can be merged by determinization.
Some options are immutable (target, output files; global switches like
conditions, reuse mode, storable states; support of flex syntax, etc.).
These options are passed as command-line arguments and never change.
It is safe to read them from any program point after parsing command-line
arguments.
Other options are configurable; they have block scope (may be specified
anywhere inside of the block and still affect the whole block).
Reading mutable options of yet unparsed block is not allowed because
they may affect the way RE2C parses current block (RE2C would be tempted
to base decisions on the latest option value, which may not be the final
one).
Configuration of these options either makes little sense (like changing
target on the fly) or is technically impossible to implement (the only
example is '-F, --flex-syntax' option which non-trivially affects the way
RE2C parses programs).
Changes in tests are caused by the lack of newline after '/*!ignore:re2c*/'
directive. There's no particular reason why it should be emitted and RE2C
code is simpler without it.
AST is immutable and independent of options: encoding expansion and other
option-sensitive transformations are delayed until AST is converted to
intermediate representation. Thus AST can be shared among different blocks
and conditions and reused with any set of options without modifications.
Changes in tests are trivial: warnings from the rules block are no longer
emitted (we don't compile AST to DFA for rules block anymore, since it
will be discarded anyway); skeleton tests no longer emit '.input' and
'.keys' files for rules block.
This allows us to delay things that should be delayed (like expanding
regexps with respect to current encoding), but which may throw an error
and need location info.
RE2C has named definitions, which allow to override operator precedence
without grouping parenthesis. However, in POSIX all parenthesis are
capturing: allowing some of them to be non-capturing would affect
disambiguation and might result in a different parse.
For now RE2C uses a crude criterion to determine if the given named
definition needs to be explicitely wrapped in parenthesis: if the topmost
constructor of the subexpression is either alternative, or catenation, or
another reference to another named definition, then wrapping is required.
This is an overapproximation, because it doesn't take the surrounding context
into account (e.g. catenation inside of an alternative doesn't need wrapping,
but catenation inside of an iteration does).
Tests are adopted from Glenn Fowler's testregex suite and its derivative
regex-posix-unittest by Christopher Kuklewicz.
Irrelevant tests have been dropped:
- tests for specific POSIX features not supported by RE2C, like
begin/end markers '$' and '^', backreferences, predefined character
classes (most of them come from 'basic' subdirectory)
- all 'leftassoc' tests: they are exact copies of 'rightassoc' tests,
differ only in match results and are supposed to fail
- all tests from 'categorize' that are supposed to fail
I also added a few tests of my own (in the 'other' subdirectory).
Tests are arranged as a bunch of '.dat' files (each file contains lines of
the form '<regexp> <string> <match>' or blank lines) and two scripts.
- '__run.sh' runs all tests (generates temporary '.re' file with the
given regexp, compiles it, runs against the given string and checks
that the result coincides with the given result)
- '__gen.sh' regenerates '.re' files for regular RE2C test suite (the
generated files do not contain any code, just the bare regexp)
All regexps are pre-patched to exclude '\x00' (terminating NULL) character
from subregexps like '.' or '[^a]: otherwise RE2C-generated programs would
greedily consume all characters and fail. Note that if only '.' was present
in the original tests, we might have used '\n' as terminator, but '[^a]'
includes newline. Also '()' subregexp was substituted with '("")'.
Instead of skipping fixed tags, skip orbit tags. For each orbit tag there is
a tag fixed on it, so if we check all fixed tags we'll surely check all orbit
tags. Anyway, with '--skeleton' all fixed tags correspond to orbit tags: other
fixed tags are forbidden with generic API (even zero-offset ones for now).
For some reason dead code elimination pass simply ignored 'copy' commands
and removed only dead 'save' commands. Strange as it seems, this went
unnoticed until now (when one of the POSIX tests compiled into C++ code
that triggered GCC warning about use of uninitialized variable).
With this option re2c interprets all parenthetised subexpressions
as POSIX capturing groups and returns submatch according to POSIX
disambiguation rules (in variables 'yynmatch' and 'yypmatch', local
to each action).
The algorithm for disambiguation was invented by Christopher Kuklewicz.
We'll need this for POSIX captures: in the prsence of orbit tag,
opening tag is needed only for disambiguation; after determinization
we can pretend that it is fixed on orbit tag with zero offset.
This will save some time and space on tag optimizations and maybe
reduce the number of tag variables.
This will be needed when we add POSIX captures: opening and closing
capture tags must have sequential numbers, but the tags may be located
far from each other in the regexp tree (other tags are possible on
the way from the opening tag to the closing tag).
This is necessary to avoid creating default tags for tags that
are duplicated by expansion of counted repetition. Default tags
should only be created if zero repetitions are acceptable; this case
is still expanded early.
Currently this makes no difference because tag duplication in NFA
is forbidden. Anyway, we are only interested in the last occurence
of each tag on the epsilon-path to the given NFA state.
This will make difference for POSIX disambiguation strategy: we'll
have to allow tag duplication in NFA (because of counted repetition
and iteration expansion).
Rationale: AST must be immutable, because it is shared among different
regexps, rules, conditions and blocks. NFA is mutable, but inconvenient
for some transformations (adding fixed and default tags, rewriting parts
of regexp, etc.). Before this commit all these transformations were done
in one pass (during convertion of AST to NFA) which made the code rather
complex. In order to add POSIX captures we'll need to add even more
complexity (tracking captures, creating tags for them, maintaining tag
order, etc.).
One must take fixed tags into account during determinization
if disambiguation policy is based on tag values.
Also, NFA construction is much easier when tags have common enumeration:
it allows to perdorm tag creation, tag fixing and adding default tags
in three distinct passes (for now they are intermixed and the code is
very complex).
We no longer need it; now that disambiguation policy is leftmost rather
than tag-centric, relative order of tag versions is irrelevant.
Tag versions have no impact on conflict resolution; any bijective
mapping is ok.
Instead of tracking priority for each tag version and resolving conflicting
configurations encountered by the epsilon-closure based on version priorities
(trying to minimize or maximize each tag), we now simply pick the leftmost
epsilon-path through the NFA.
NFA construction has to accommodate this policy: for example, greedy iteration
should be compiled so that always choosing the left alternative correpongs to
re-iteration, while choosing the right alternative corresponds to leaving the
iteration. The necessary adjustments have been done by earlier commit
03c65121da.
Now we can drop priority comparison when mapping TDFA states and stick to
simple bijection. This may add cycles in reordering commands. (Has not been
done yet.)
The rationale of switching to leftmost policy is this: with tag maximization/
minimization policy tags could influence the regexp: for example, regexp
"a"* @p "a"*
has different meaning depending on whether tag 'p' is minimized (first iteration
is non-greedy) or maximized (first iteration is greedy). Moreover, adding
another tag with the opposite policy and higher priority would change the
meaning (suppose 'q' is minimized and earlier tags have higher priority):
"a"* @q @p "a"*
This all makes regexps somewhat fragile, since adding just one tag may change
the whole regexp (and in the presence of other tags, it may change parse
results).
But even worse, with tag-centric disambiguation simple (untagged) subexpressions
are inferior to tagged ones. This contradicts intuition and even POSIX standard,
where a*(a*) is the same as (a*)(a*) . We don't keep to POSIX (and we can't
because of last-longest instead of first-longest semantics of iteration as shown
by the (<|<a|<ab|<aba|abab|baba|b>|>)* example), but even this somewhat
tag-centric standard respects non-captured parts of the regexp. Note that
Laurikari in his TRE library has to do special preprocessing of the regexp in
order to assign correct minimization/maximisation direction to each tag
depending on the context of the tag. His original tag-centric policy (open tag
is minimized and has higher priority than closing tag which is maximized,
nested tags have lower priority) is insufficient to accommodate POSIX.
By convention first alternative has higher priority.
So, for example, the following must be true:
r* = rr* | <empty>
r{n,m} = r{n} (r{m - n} | r{m - n - 1} | ... | r{1} | <empty>)
r{n,} = r{n} (rr* | <empty>)
For now we don't care about priorities: this is a preparatory step
before transition to greedy leftmost semantics for tags.
Found those as:
$ make CXXFLAGS="-Weverything -Werror -Wno-gnu-statement-expression -Wno-nested-anon-types"
- cleaned struct<->class declaration mismatch
- 'static' around local function
- 'RE2C_GXX_ATTRIBUTE ((noreturn))' around functions known to fail
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
Normal form is calculated for every new state before searching for
a mappable counterpart. All versions are renumbered to occupy consequent
numbers starting from one; version order is preserved.
One cannot rely the fact that final versions are [1 .. N] anyway (they are renamed
during tag optimization), so this is not convenient as it seems.
Now versions start from 1, which looks better (more logical) on the pictures.
/* note [tag hoisting, skip hoisting and tunneling]
*
* Tag hoisting is simple: if all transitions have the same commands,
* they can be hoisted out of conditional branches.
*
* Skip hoisting is only relevant with '--eager-skip' option.
* Normally this option is off and skip is lazy: it happens after
* transition to the next state, if this state is consuming.
* However, with '--eager-skip' skip happens before transition to the next
* state. Different transitions may disagree: some of them go to consuming
* states, others don't. If they agree, skip can be hoisted (just like tags).
*
* '--eager-skip' makes tag hoisting more complicated, because now we have
* to care about the type of automaton: lookahead TDFAs must skip after
* writing tags, while non-lookahead TDFAs must skip before writing tags.
* Therefore skip hoising cannot be done without tag hoisting in lookahead
* TDFAs, and vice versa with non-lookahead TDFAs.
* (Note that '--eager-skip' is implied by '--no-lookahead').
*
* Tunneling splits base states in two parts: head and body. Body has all
* the conditional branches (transitions on symbols), while head has just
* one unconditional jump to body.
*
* Normally tag hoisting should go before tunneling: hoisting may add new
* candidates to be merged by tunneling. However, with '--eager-skip' tag
* hoisting is interwined with skip hoisting, and the latter needs to know
* which states are consuming. This is not possible if tunneling is still
* to be done, because it may turn consuming states into non-consuming ones.
* Another option is to disallow splitting states with non-hoisted skip
* in the presence of '--eager-skip' (this way skip hoisting wouldn't need
* to know tunneling results), but it's much worse for tunneling.
*/
Found by slyfox's fuzzer. :)
Found by valgrind:
1,177 bytes in 1 blocks are definitely lost in loss record 1 of 1
at 0x4C2AE40: operator new[](unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x42153F: re2c::cfg_t::compact(re2c::cfg_t const&, int*) (compact.cc:12)
by 0x422088: re2c::optimize_tags(re2c::dfa_t&) (optimize.cc:18)
by 0x42DE3A: re2c::compile(re2c::spec_t const&, re2c::Output&) (compile.cc:82)
by 0x446742: re2c::parse(re2c::Scanner&, re2c::Output&) (parser.ypp:484)
by 0x402DC0: main (main.cc:38)
This reduces the overall number of versions greatly, which is crucial
for the space and time efficiency of tag liveness analysis.
Of course, bottom and normal transitions still have different versions.
Differents tags on the same transition also have different versions.
It is not incorrect to give them the same version: all tags updated by
one and the same transition are either set to bottom, or to current
input position (which is the same for all tags on the given transition).
But this would create many states with a very special "shape" that do
not map to other states. In fact, this would cause exponential blowup
in the number of states (even in simple cases).
After TDFA construction tag versions occupy non-contiguous range
of natural numbers: there may be "holes" between versions. This wastes
space (and time) spent on liveness analysis and subsequent optimization
passes. This commit adds a simple pass that renumbers versions so that
they occupy contiguios range.
This option allows to control program points in which cursor is
advanced to the next input character. Normally it happens after moving
to the next state (lazyly). With '--eager-skip' cursor is advanced
before moving to the next state (eagerly).
Eager mode presents some difficulties compared to the lazy mode:
transitions may disagree on whether advance cursor or not. Some of
them may go to accepting states and should not advance cursor, while
others go normal states and should advance cursor. In such cases
advancement must happen on a per-transition basis. If all transitions
agree on the matter, advancement can be hoisted out and performed on
a per-state basis, just as it happens in lazy mode.
re2c folds sequences of simple statements to complex tatements, e.g.:
++YYCURSOR;
yych = *YYCURSOR;
is folded to:
yych = *++YYCURSOR;
It is hard to fold expressions on the fly (at the time they are generated),
because one needs to forsee all the cases in advance. Some cases cannot be
anticipated (e.g. if substatements belong to different states, which is often
the case), so many statements remain unfolded.
This commit separates code generation and folding by delaying output of
skip, peek and backup primitives. Folding now is done in a separate pass
when all the substatements are already known.
Option scope is very poorly defined in re2c. Most options should have block
scope, but some have global scope (because they belong to global directives
like '/*!types:re2c ... */', '/*!tags:re2c ... */' or other global things
like generation date, etc.).
For now, options are applied immediately as they are parsed.
Code generation just reads the most recent state of options, so the output
depends not only on the location in source code when the option is defined,
but also on program points in which re2c generates and outputs code.
Generation and output may happen at different times, so they may use
different options. This is all very bad.
This commit is the first attempt to introduce scoped options: re2c now
makes a snapshot of options at the end of each block and uses this snapshot
for delayed code generation for this block (unless the given option must be
global).
This is not perfect (immediate code generation still uses 'seen-so-far'
options instead of block options), but at least it allows to delay code
generation for non-global things without loosing options.
Updates in test results are caused by the changed order of condition
compilation, see commit b7bbbf471778b487d0717c85a5e586b595e495ec.
Changes do not affect code generation, only the error messages.
Each condition has a unique name and unique number. Numbers are assigned
to conditions in the order they appear in source file. This order must
be preserved since some users rely on it (user code makes implicit
assumptions about condition order, see '-Wcondition-order').
However, once the order is preserved, the automata code can be emitted
in any order (user code can only address conditions by their start label,
and it can be anywhere in the output file).
This commit changes the order of automata: now they are emitted according
to their numbers, not names.
Test results have changed in cases when lexicographical order does not
coinside with number order. The changes touch some important files and
are not easy to verify. However, rearranging conditions in the generated
code back in lexicographical order makes the differences trivial (labels,
'YYDEBUG' parameter, 'YYFILL' labels, 'YYSETSTATE' parameter --- in other
words, everything that depends on state numbering). It is easy to see
that the structure of the code is not changed.
As a side effect of this commit a long-standing bug with setup rules
has been fixed: re2c forgot to clean up the list of setup rules together
with other data, which resulted in false errors about duplicate setup code.
Instead of sometimes generating code like this:
++YYCURSOR;
switch ((yych = *YYCURSOR)) {
re2c now always generates
yych = *++YYCURSOR;
switch (yych) {
There's not much difference: it's hard to say which would be better
optimized by C/C++ compiler. Most likely they would be optimized
equally well; practical experiments with GCC -O2 resulted in bit-to-bit
identical binaries.
However, dropping the first form allows to cut off a lot of tricky code
in re2c itself (a whole layer of code spread across multiple files was
tracking whether next input character has been already read or not).
Changes to the test results are multiple, but trivial; they have been
verified with skeleton.
A similar heuristic already existed, but it was imprecise and captured
only those cases when final state happened to be next to the one in
question.
A lot of tests were updated; however changes are trivial and I verified
them with skeleton.
If mapping is injective, there may be multiple copy commands per one
save command: save(X), copy(Y,X), copy(Z,X). In this case save command
must be replicated for each copy command: save(Y), save(Z).
One tempting way to track default value of tags is to forbid mapping
bottom (default) to non-bottom versions. This way we don't need to
explicitely save bottom to variable; bottom value is encoded in state
(once a tag becomes bottom, it will never become non-bottom again).
However, this may exponentially blow up the number of states.
Now '-Wnondeterministic-tags' analysis is done right after determinization,
while we still have all the kernels available. Instead of comparing
transition tags (which are available in closure items, but not in kernels)
we now compare tag versions: maximum number of distinct versions per tag is
the desired degree of nondeterminism. Kernel items that belong to another
rule do not count (all tags that don't belong to a particular rule should
have initial versions in all items belonging to this rule).
Added test with an example of pathological regexp: counted repetition
cannot be folded efficiently in DFA, and combined with nondeterministic
tags it results in arbitrary long chains of reordering commands.
Length of command choin grows linearly with the number of repetitions
(as DFA itself does).
This option controls how tag versions in DFA states are mapped to each
other during determinization. By default mapping is 'bijective', however
if it is set to 'injective' the new state must be less versatile than
the old one.
Tunneling optimization finds certain states and splits them into
two parts: First part consumes a character and unconditionally moves
to the second part. Second part does not consume characters, but
contains a switch on character value.
Second part is a 'base switch', which may be utilized by many other
states. So if the first part has hoisted tags, they should be dropped
in the second part (the aforementioned many other states may have hoisted
tags of there own or no hoisted tags at all).
Found by slyfox's fuzzer. :)
Yet unable to construct a test case that reveals the error: the original
test case found by fuzzer is not breaking anymore after recent commit
44045203c2 "Properly remove useless final states.".
When processing chain of 'copy' commands like 'x = y; y = z;'
one should first mark all left hand sides ('x' and 'y') as dead,
then mark all right hand sides as alive ('y' and 'z'). This way
'y' will end up alive, as it should be.
Found by slyfox's fuzzer. :)
Before this commit the algorithm considered final state useless only
if its rule is unreachable in this state. However, this leaves out
cases when final state is shadowed by other final states of the same
rule (longer alternatives of the same rule always match). In such
cases the rule is reachable because some (maybe all) outgoing accepting
paths match the same rule.
Now the algorithm considers final state useless if all outgoing paths
are accepting (there's no default transitions and default rule is
unreachable in all successors).
The order in which copy commands are executed is important:
'x = y; y = z;' is not the same as 'y = z; x = y;' (the latter
overwrites 'y' before its precious value is copied to 'x').
To avoid overwrites, commands should be topologically sorted.
This is always possible because there's no cyclic dependencies
by construction.
Found by slyfox's fuzzer. :)
This commit makes some important changes:
- Each DFA state under construction keeps a set of tag versions
(which version of each tag is actual for each NFA sub-state).
- Tag versions now have priorities which allow to disambuguate
between two competing configurations when building tagged epsilon-
closure. We try to maximaze value of each tag, thus versions
that correspond to greater tag value have greater priority.
- Comparison of DFA states (when trying to find an existing state
equal to the new one) has become more complex. We try to find
a similar enough (not necessarily identical) state: one that is
equal to the new state up to reordering of tag versions. Mapping
should be either bijective or injective (new state must be less
or equally versatile then the old one), default is bijective.
Mapping should not violate tag priorities: relative order of
versions must hold (absolute values of priorities don't have to
coincide).
- On each update tag gets new version: if tag is updated to bottom,
it gets negative version; otherwise it gets positive version.
- Different tags that updated by the same transition still get
different versions. This may seem to be a waste: we know exactly
that input position is equal for all tags on the same transition,
even more, for all tags on all outgoung transitions from the same
state, so we can alocate exactly one version for all of them.
However, allocating one version for different tags creates states
with a very specific tag configurations: these states don't map
to existing states and the automaton gets bloated (sometimes
exponentially). Furthermore, allocating independent versions is
good for optimizations.
- Second pass of optimizations: it only makes difference for one
test case (usually the first pass does a good job), but it's cheap
(all the necessary infrastructure is in place) and powerful.
- The resulting automaton is similar to Laurikari's TDFA, except
for one important difference: re2c's automaton makes use of
lookahead tags, while Laurikari's does not.
This commit reverts commit de3f9e70a4:
Keep fixed and variable tags together in one array.
Optimizations of variable tags get more complicated and fixed tags
should not get in the way.
This commit also drops check for tags in trailing context: there's
nothing special about them; no technical reason to forbid them.
NFA construction is already complex because of insertion of default tags,
now it gets even worse. However, keeping two separate passes requires
maintaining exactly the same order of traversal, which is fragile.
This is a preliminary step for tracking tag versions in closures / kernels.
For now closures and kernels store the same information, but it will
diverge when we start tracking tag versions: closure items will need
some extra-data that is needed during closure construction, but shouldn't
be in kernel.
Kernel representation should also allow efficient comparison for identitiy
or compatibility (for mapping).
At the time of skeleton construction DFA has just been build and
all tags in it are just raw pointers to lists of commands. These
pointers are unique for each transition (tags are not shared between
transitions). This means, comparing tags for different transitions
will always result in 'not equal', except if both transitions have
no tags (pointers are NULLs).
Found by slyfox's fuzzer. ;)
Instead of calculating all necessary keys, then attributing keys
of the first path to all paths, write proper keys for each path.
Found by slyfox's fuzzer. :)
Uninitialized tags come from alternative and iteration:
... @t ... | ...
(... @t ...)*
Tags under alternative preserve their initial value (given to them
before entering DFA) -- they don't have to be explicitely set to
default value (but it is convenient, otherwise we'd have to initialize
all tags every time before entering DFA).
Tags enclosed in iteration might be assigned positional values on
some iterarions, but not on the last one -- their default value must
be restored explicitely before each iteration.
This commit adds specail kind of 'save' commands that set the given
tag to default value (instead of current input position). We insert
these 'save default' commands for tags in alternatives: all tags that
are present on left alternative get default values at the end of right
alternative.
Later on (during DFA construction and optimizations) we treat default
value just like normal positional value: save it to a variable, copy
values between variables, etc. This way DFA gets additional commands,
but the number of states does not grow much.
Another way would be to treat default value as a special bottom value:
instead of saving it to variable, keep a dedicated bottom tag version
and forbid mixing bottom version with other versions. This way default
values would be encoded in the structure of DFA, so there would be no
need to explicitely save them. DFA would gain no additional default
commands, but new states would appear.
Test changes have been verified with '--skeleton'.
Test 'tags/interference.i--tags.re' have been completely rewritten --
the old version no longer captures what it should (see comment in the
test). The old vesion has been moved to 'tags/fallback6.i--tags.re'.
Before code generation each DFA state is assigned a specific action.
Actions are mutually exclussive except for one: inital state may
coincide with fallback state. The bug was in overriding fallback
action with initial action (we lost 'yyaccept' which cause the wrong
match).
The bug was found with '--skeleton'.
Tag may have different final versions in different accepting states
(currently either fallback version of the main version). Now we
reduce all final versions to one (this takes one additional 'copy'
command for normal tags and none for tags that have fallback copy).
Calculate final tag values during string generation and put them into
'.keys' file together with the length of the consumed / matched input
and the matching rule number. Each pack of keys only gets those tags
that belong to the matching rule (no tags for default paths or paths
corresponding to untagged rules). Tags do not include trailing context,
as it is already included into the length of matching input.
This commit should fix bug #163 reported by pauloscustodio:
Reading files with "rb" causes issues in Windows
re2c reads input files in binary mode and writes the generated output in
text mode. This caused CR LF conversion to CR CR LF on Windows: first CR
comes from reading input in binary mode, second CR is added when writing
output in text mode. This only happened to those parts of input which are
not transformed by re2c: we used to copy-paste verbatim, now we patch line
endings.
There's no need to pull the whole skeleton up to code generation phase:
skeleton itself is used only for string generation and tracing undefined
control flow, both things are done after DFA construction and before any
optimizations.
By the time code generation starts skeleton is obsolete: its structure
is no longer in sync with DFA and it shouldn't be used.
Code generation only needs some attributes (key size and default rule
index), which can be explicitely stored in DFA.
Skeleton must use unoptimized DFA right after determinization, otherwise
it won't be able to track optimization errors. Therefore skeleton must
either use a copy of DFA, or use the original before any optimizations.
Copy-pasting the whole DFA is a waste (especially when it comes to tag
commands, which are stored in linked lists), so we take the second approach.
However, this means that skeleton '.input' files are generated every time
DFA is constructed, not every time DFA is used for code generation.
This explains changes in tests: in reuse mode, we sometimes construct
DFA and never use it for code generation.
As for now, newline inside of a character string or class is an error:
re2c should emit clear error message. Different styles of newlines
should be recognized ("\n", "\r\n").
This commit fixes bug #162 reported by pauloscustodio:
Reading files with "rb" causes issues in Windows
For the most part character range is represented as a pair of bounds.
When generating data for skeleton programs we need to pick some
256 distinct characters from each range; only these characters will
participate in data generation.
Before this commit range expansion happened at the time of skeleton
construction. Now it is delayed until the time of dumping skeleton
paths to file. This means that expansion of particular range is done
multiple times (every time the corresponding skeleton multiarc is
dumped to file). However, data generation only happens with '--skeleton'
option, while skeleton construction is unavoidable.
Text files need to be opened for writing with "w", so that stdio does
the right thing in respect to the correct line endings for the current OS.
("\r\n" in Windows, "\n" in Linux).
Line endings in the generated code depend on the target platform: e.g.,
"\r\n" on Windows vs. "\n" on Linux. However, reference test results are
(currently) generated on Linux and therefore contain "\n" line endings.
So we have to patch line endings in the generated code in order to pass
the tests on Windows.
Testing script did patch line endings in stdout and stderr, but forgot
to patch them in the generated file (it was broken since we started to
use '-o' option for testing). This commit fixes testing script.
It also deletes a couple of tests in which source code contains "\r\n"
instead of "\n". These tests are duplicates of other tests (they were
added by commit bd2875441c supposedly to
test that re2c preserves line endings in source code). They are broken
by current commit and fixing them is probably not worth of the effort.
After inserting commands into common index some of them may share
representations in memory. Initially commands are stored in mutable
lists, which is very convenient for tag optimizations. However,
leaving this representation after indexing is dangerous, because
accidental modification of one command may affect other commands.
This commit hides mutable lists behind indices in hash table.
Table gives read-only access to the lists.
Table index is 32-bit, which is smaller than the previous 'index'
type (a pair of pointers). So in theory command comparison has
become a little faster.
DFA may be very large, but have only few tagged transitions.
In such cases doing tag optimizations on the whole DFA is a waste:
space aloocations and (respectively) processing time is proportional
to the number of all DFA transitions.
Instead, we spend a little effort to create CFG. It has one basic
block per each tagged transition or tagged final state, plus root
basic block.
Fixed-length arrays are widely used during DFA construction, where
we need to keep all versions of all tags in each NFA sub-state of the
DFA state under construction. These arrays are deduplicated, indexed
and stored in a hash table.
However, after determinization we deal with small (probably empty)
sets of tag versions that are updated on a given transition.
Corresponding arrays are sparse. We don't care about storage space
(arrays are deduplicated anyway), but traversing sparse arrays is
wasteful and the code is somewhat bloated.
We already used linked lists for 'copy' commands; now we also use
them for 'save' commands.
The previous commit 394fec90c1
"Use tag versions to implement fallback tags." added copy coalescing
and introduced serious changes to allocation algorithm. It simulated
the order of allocation used by previous algorithm to preserve test
results.
Now we drop this simulation and keep to the natural order: it is easy
to see that changes in test results are caused by different order of
tag version allocation.
This commit adds new data structure to represent copy commands for
tags: list of pairs (X, Y) which stand for command 'X = Y'.
Previously we used the same bit vectors that are used for constructing
DFA states, but this representation is inconvenient.
We no longer need special tag names (postfixed with underscore) for
fallback tags: we simply use a fresh version for each fallback tag.
Instead of treating tag as boolean variable (true - present,
false - absent) we now allow it to have many versions. Special
version value 'TAGVER_ZERO' denotes tag absense (corresponds to
'false').
For now, each tag has exactly one version except 'TAGVER_ZERO':
tag's number plus one.
We need tag versions because with only two values it's impossible
to expess more subtle states, e.g. initialized / uninitialized.
Tag versions will also allow to treat fallback tags like any other
tags (in particular, allow them participate in deduplication).
In future, when we add nondeterministic tags, versions will be used
to implement tag copies.
This is necessary to introduce tag versions: we'll have to
calculate liveness of tag versions (rather than tags) and won't be
able to keep liveness data in Tagpool.
This commit effectively reverts commit eeceb1d5ba,
"Use backwards propagation for liveness analyses on tags.", which changed
control flow analyses to backward propagation of liveness from final states.
The reason for rollback is that common control-flow analysis is more
general. Backward propagation is simple if tags are only used in final states.
While this is currently true, but we'll have to allow tag uses on arbitrary
arcs, and this will complicate backward propagation a lot.
Relative tags have some advantages:
- only base marker has to be updated by YYFILL
- ever-broken old code with overlapping trailing contexts
can be fixed without user's help (provided that re2c
autogenerates declarations of tag variables)
- fixed-length contexts may be used with generic API
(though it becomes less generic with relative tags,
as we rely on the concept of relative offset)
However, the generated code is less efficient in simple cases
(default API, no YYFILL): relativity adds extra arithmetics with base
marker (which implies dereference of base marker in most of the cases),
so the code is actually worse than hand-written.
Even with YYFILL it turns out to be more efficient to keep all tags
in a struct together with other pointers and update them once in a
while: YYFILL is called very infrequently (if properly implemented
by the user), while tag operations happen unconditionally.
However, making tags absolute means that re2c will fail with error
on old code with overlapping trailing contexts (unless they have fixed
length or can be deduplicated). These cases are probably rare.
Generic API can't make use of fixed tags, but becomes truly generic:
'YYBACKUPTAG / YYRESTORETAG / YYCOPYTAG' make less assumptions
about the input.
Reasoning:
- with YYFILL, one must use '/*!tags:re2c*/
- without YYFIL, one might still want to use '/*!tags:re2c*/'
(e.g. to avoid warnings about initialized variables)
- unconditional behaviour is easier to remember
- everyone who uses tags should know about '/*!tags:re2c*/'
- re2c code is simpler
Problems:
- without scoping rules multiple blocks with tags are troublesome
Note that actual code generation for '/*!tags:re2c*/' is still delayed.
This is a preparation step before making tags absolute (rather than
relative to context marker). Generic API should make no assumptions
about the input source; in particular, it should not rely on the
notion of distance between input positions.
Having one variable per tag name is convenient:
- tag calculation is not duplicated if tag is used multiple
times in the action body
- tag name is an l-value
- there's no need to patch actions
Variables are user-defined: users know which variables they need
(unlike distance variables 'yyt<N>', which are generated by re2c
and affected by things like tag fixedness, deduplication, etc.);
users may also want to initialize these variables in some particular
way.
Formatting of 'if' or 'case' statements depends on the number of
statements in the correspoonding code block: multi-statement blockss
must be wrapped in braces or extra-tabulated.
Estimating the number of lines has never been particularly elegant,
but it gets worse every time we add possible new instructions to
transitions.
We need to backup tags in fallback states: these tags may be
updated on some non-accepting path from fallback state, and if
the attempt to match longer rule fails, there's no way to restore
the overwritten tags.
Such tags may appear in self-overlapping rules, e.g.:
(@p "ab")* {}
We create a special backup variable for each potentially overwritten
tag. Backup tags do not participate in deduplication.
Bug was introduced in commit 5958a2366c
"Bind contexts (a.k.a. tags) to DFA transitions, not states.":
instead of comparing tags of single transition on one particular
symbol, we compared tags for all outgoing transitions at once
(probably copy-paste from Moore minimization).
The bug was revealed by multiple tests (including '--skeleton' ones)
as soon as I ran testsute with '--dfa-determinization table'.
State granularity is too crude: there might be
two different paths to the same state, with some tag alive on one
path but not on the other. If liveness is attributed to states, then
tag liveness on one path implies tag liveness in the join state,
which affects the other path. But if liveness is attributed to
edges, then liveness of one path won't affect liveness of the other.
re2c used to merge all possible fallback tags and backward-propagate
them from states that have default transitions. This is an
overapproximation; some tags are marked alive while they are not.
Liveness of fallback tags should be forward-propagated, because
this is the only way to track paths that really need these tags
(that is, paths that start in final state and end with default
transition *without passing through another final state*).
Note that sometimes we can get to the same default transition
following two different paths, one path may passing through final
state; the other is not. One path cares for fallback tags; the
other doesn't. We have no way to find out if we start bacward
prpagation of fallback tags from default transition.
By construction NFA has exactly one final state per rule.
Thus closure has at most one final item per rule (in other
words, all final items in closure belong to different rules).
The rule with the highest priority shadowes all other rules.
Final items that correspond to shadowed rules are useless
and should be removed as early as possible.
If we let such items remain in closure, they may prevent the
new DFA state from being merged with other states. But this
won't affect the resulting program: meaningless finalizing tags
will be removed by dead code elimination and after that DFA
minimization will merge equivalent final states.
But it is much easier and cleaner to remove useless items
immediately (and thas't what we do).
Some '--skeleton' tests changed: this is caused by collapsed
DFA states (shadowed final NFA states prevented them from
being collapsed before). All changes are verified with '--skeleton'.
Now that conflicting tag configurations are no longer merged,
we can insert them into 'Tagpool' immediately. This allows us
avoud ugly trick with 'union' in closure items.
This makes code somewhat clearer compared to using plain C arrays:
all function accept one parameter (vector) instead of two (array
start, array end/array length).
However, vectors have nontrivial structure and 'ord_hash_set_t'
is not suitable for them. That's why instead of 'ord_hash_set_t'
we now use 'lookup_t': just like 'ord_hash_set_t' it has constant
random access ang logarithmic insertion complexity, but unike
'ord_hash_set_t' it is not meant to be a container (rather an index).
All textbooks describe determinization (powerset construction)
like this:
S0 <- epsilon-closure of start NFA state
add S0 to DFA
while there is unmarked state S:
mark S
for each symbol X in the alphabet:
R <- the set of NFA states reachable from S on X
T <- epsilon-closure of R
if T is not in DFA
add T to DFA
In re2c the inner loop on alphabet symbols was split in two parts
and somewhat obfuscated; this commit makes re2c follow textbook
version more closely.
This commit effectively reverts commit 5cf441dc93
"Base '+' (one or more repetitions) on '*' (zero or more repetitions)."
It turn out, there is a good reason for using '+' as base operation:
'*' can be expressed in terms of '+' as 'r* ::= r+ | <empty>', while
'+' expands as 'r+ ::= r r*' and 'r' is duplicated.
Duplication becomes crucial in presence of tags: if the duplicated
subexpression has tags, then duplication causes an error.
As suggested by sirzooro:
Please add symbol name to "can't find symbol" error message,
it would allow to quickly spot what is wrong. Now we have to
position cursor at given row and column to find that name.
Also tweaked error reporitng function to append "..." at the end
of the message if it didn't fit into buffer.
when default rules with conditions are used".
Default rule is handled in a special (delayed) way;
re2c uses different code for default rule than for normal rules.
This special code simply forgot to add condition name to the list
of conditions.
Thanks to sirzooro for bug report.
Removing useless final states allows some further optimizations
which would otherwise be impossible (see the changed tests for
example).
However, removing useless final states revealed previously hidden
defects in calculating fallback states: in some tests spurious
new 'yyaccept' values appeared. It turned out that we used incorrect
(imprecise, too pessimistic) algorithm to calculate fallback states:
the given state was considered to be fallback state if it was final
and had transitions to non-default and non-final states. This
algorithm erroneousl captures cases when all paths from the given
state are accepting and long enough (longer than one transition).
The new algorithm to find fallback states uses information obtained
by rule liveness analyses (reachability of "none-rule" in particular).
Some important tests changed a lot; skeleton says the changes are
correct (as well as manual looking into the changes).
The previous algorithm handled loops incorrectly: if deep-first search
from current DFA state happened to visit looping paths before non-looping
paths, then result accumulated from non-looping paths was not propagated
into loopback states.
Now we use back propagation (from states that have transitions to default
state), which handles loops correctly.
We couldn't trace all unreachable nullable rules anyway, e.g.:
[^]?
Nullable part of this rule is unreachable, but it was reported.
Besides, there's nothing bad in reporting all probles at once.
Reversed DFA reflects state of the original DFA in the moment when
its reversed copy was built. Original DFA undergoes many changes;
it's hard (and needless) to keep reversed DFA in sync with the
original.
We only need reversed DFA in a couple of places, better make a new
copy in each case.
In commit a7a26d17ce we added fix-point
iterative algorithm to do liveness analyses on tags:
The usual algorithm for liveness analyses doesn't use DFS;
instead, it iterates on DFA states until fix point is reached
(next iteration adds no changes to live set compared to the
previous iteration). The fix applies this algorithm.
However, instead of iterating until fix point is reached, one may
use backwards propagation of live variables starting from the points
where these variables are used.
Tagpool now has a special buffer to accommodate the needs of
intermediate calculations involving tag sets. Tagpool is in charge
of these calculations now; it has various functions to merge,
mask and whatever one has to do with tag sets.
Example that revealed failure (added as test):
(@p "ac")* "b" { @p }
We used forward propagation: starting from initial state, walk DFA
in deep-first order; in each state, add tags associated with rule
or fallback set if necessary, then recurse to child states, then
merge all child sets into this node's live set.
This algorithm cannot propagate live tags in loops in cases when
DFS visits loopbacks before non-looping paths (tags don't propagate
into loopback states).
The usual algorithm for liveness analyses doesn't use DFS;
instead, it iterates on DFA states until fix point is reached
(next iteration adds no changes to live set compared to the
previous iteration). The fix applies this algorithm.
A more efficient algorithm may be possible; one that uses backwards
propagation instead of fix point (I'm not sure).
Some tags are merged to other tags during tag deduplication.
These tags are referenced by their distinct name in action code,
but in fact they are just aliases to their representative tag
(origin).
If one tag name, say 'p', is a prefix of another tag name, say 'p1',
and we try to substitute all occurences of '@p' in the given string,
we might occasionally substitute '@p1'.
I made a small re2c lexer that recognizes tag-like lexemes and
tries to match them agains rule's tags. If a tag's name matches the
recognized substring, than it is the perfect match (name cannot be
a prefix of a longer name because lexer recognizes the longest match).
If tags are explicitly enabled, re2c should always fail with error
on nondeterministic tags. There's no good in '-Werror-nondeterministic-tags'
diagnostic --- the user cannot reset it anyway.
This commit fixes unsuccessful attempt 2bfe75d57c
"Restore user warnings after temporarily enforcing a warning.", which
resulted in hiding tag errors if '-Wnondeterministic-tags' is not set.
Steps to reproduce:
$ echo -ne "&\x00" > A
$ re2c A
Segmentation fault
Analyses: when re2c finds NULL in the input file, it checks for the
end of input; if indeed it has reached the end of input, it stops.
Otherwise, it's just some NULL byte in the middle of input; it should
be handled like any other character.
The first case (NULL as end of input) was handled correctly, but
in the second case (NULL in the middle of input) re2c crashed:
someone forgot to put an appropriate 'goto' statement, which caused
completely ad-hoc control flow in lexer.
Steps to reproduce:
$ echo -ne "&\x00" > A
$ re2c A
Segmentation fault
Analyses: when re2c finds NULL in the input file, it checks for the
end of input; if indeed it has reached the end of input, it stops.
Otherwise, it's just some NULL byte in the middle of input; it should
be handled like any other character.
The first case (NULL as end of input) was handled correctly, but
in the second case (NULL in the middle of input) re2c crashed:
someone forgot to put an appropriate 'goto' statement, which caused
completely ad-hoc control flow in lexer.
This is a temporary (and quite inconvenient) hack; in fact,
warnings should be handled in the same way as options (via
user config / real config that are synced on each read access
just in case user config has been changed).
Fixed-length tag optimization applies only to top-level concatenation:
since we cannot predict which path lexer will choose, we cannot be
sure that any tags except top-level concatenation will ever be
initialized. If a tag may be uninitialized, we cannot fix other
tags on this one (we cannot even fix same-level tags relative to
each other, because fixed tags won't preserve the default value).
This is a very imortant change: it makes tracing tag conflicts as
simple as comparing tags on transitions during DFA construction.
If during determinization we get two identical transitions that
differ only in tags, then we have a tag conflict. Tags that cause
conflicts are called non-deterministic (since they don't allow
deterministic match).
This approach if very similar to Ville Laurikari's TDFA: as he does,
we first build TNFA and then apply determinization; however Laurikari's
TDFA uses complex bookkeeping to track all possible tag values,
while re2c simply forbids tags that cannot be matche efficiently.
Binding tags to transitions allows more fine-grained liveness
analyses, dead tag elimination and tag deduplication.
This is currently not necessary, but we'll need it to distinguish
between different rules when comparing tags: if rules are different,
we don't care about tag difference; otherwise tags must be equal.
- more elegant hanling of default case when generatin code
for 'switch' cases
- dumbed down complex logic behind the generation of sequential
'if' statements: explicitely list all different cases and
corresponding conditions
Thanks to Abs62, who noted that under Windows (in MSYS) tests fail
because '2>"$outc.stderr"' dumps CRLF to file instead of LF
and proposed a fix:
sed -i 's/\r//g' "$outc.stderr"
Directives like '/*!re2c:types*/' and '/*!re2c:contexts*/' have
global scope, they are not binded to any particular block and therefore
accumulate items (conditions/contexts/etc.) from the whole program.
There's currently no way to bind these directives to a particular
block (re2c will probably add scoping rules in future).
However, things like default context declaration or condition dispatch
have block scope (it's only natural that they are binded to the block
that generates them).
Unrelated change: context marker should be set for each condition,
re2c must generate it after condition label: this way code that skips
condition dispatch and jums directly to condition label will still
have context marker adjusted properly.
Now when lexer encounters the beginning of a new directive, it
dumps all the intermediate code to the output.
Before this commit re2c lexer dumped intermediate code on each
newline that occured in the input.
So this commit affects two aspects:
- Intermediate code is dumped much less often: it's good for
performance, but it becomes more probable that the intermediate
code will occasionally occupy too much buffer space and incur
buffer resize.
- Some re2c directives ignored characters right before the
(on the same line). These unfortunate characters are no longer
ignored.
re2c used to swallow newline that immediately followes end of directive:
for most of the directives re2c generates code block that ends with a
newline, so the generated code looks better if the newline is not doubled.
However, this unnecessarily complicates the lexer.
Directive is a new type of re2c block that consists of zero or more
of the following configurations (example values are the defaults):
line = "long @@;";
sep = "";
Renamed configuration: 're2c:ctxprefix' -> 're2c:contexts:prefix'
Added configuration: 're2c:contexts:expr = "@@"' that allows to
override the way context markers are accessed (but nottheir names).
Removed configuration: 're2c:define:YYDISTTYPE' (use the new
directive '/*!contexts:re2c ... */' instead).
- Rearranged some rules to avoid code duplication.
- Added production and error message about contexts in named definitions.
- Removed productions and error messages about missing expression
(simple 'syntax error' is enough, given that only part of the
errors was captured by the removed productions).
If default input API is used and fixed-length context is based on
the rightmost context (that is, YYCURSOR), then the following expression
(that calculates pointer corresponding to the given context):
(YYCTXMARKER + ((YYCURSOR - YYCTXMARKER) - yyctx))
can be optimized to:
(YYCURSOR - yyctx)
Note: unfortunately, as of GCC < 7.0, expressions like:
(YYCURSOR - 3) - (YYCURSOR - 5)
trigger warning:
warning: integer overflow in expression [-Woverflow]
See this GCC bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61240
Configurations:
"define:YYCTX" (defaults to 'YYCTX')
"define:YYDIST" (defaults to 'YYDIST')
"define:YYDISTTYPE" (defaults to 'long')
"ctxprefix" (defaults to 'yyctx')
Parser should simply construct AST; all the complex reasoning about
contexts (fixed vs variable) should be delayed until NFA is being
constructed. This way AST can be immutable and it's very easy to share
parts of AST between different conditions, etc.
Removed rule ranks and rank counter: rules are now stored in NFA-local
array and addressed by index.
This way it is more convenient to add various graph algorithms:
each algorithm can keep its own data in array indexed by skeleton
node numbers (instead of pushing all relevant data inside of the
node).
Just store pointers to skeleton nodes (instead of bookkeeping arcs,
contexts and rules in each path). All the necessary information can
be easily retrieved from nodes when path is baing dumped to file.
Three tests in '--skeleton' mode have been broken by this commit.
Actually, these are not breakages: these cases reveal incorrect
re2c-generated code. The change is due to the fact that skeleton
now doesn't simulate contexts that go *after* the matched rule:
------o------o------> ... (fallback to rule)
rule context
This patch fixes bug #137 "run_tests.sh fail when running configure
script with absolute path".
The fix: copy all test files into temporary build directory (which
is addressed by a relative filepath) and keep messing with relative
filepaths.
This patch fixes bug #137 "run_tests.sh fail when running configure
script with absolute path".
The fix: copy all test files into temporary build directory (which
is addressed by a relative filepath) and keep messing with relative
filepaths.
Rule information (line, attached code block, rank, shadowing set, etc.)
is used throughout the program. Before this patch, rule information was
inlined in RuleOp. It was inconvenient because RuleOp belongs to early
stages of compilation (AST, prior to NFA), but it had to live throughout
the whole program.
Static (that is, of fixed length) trailing contexts don't need
recording context position with YYCTXMARKER and restoring it
back on successful match. They can be tracked simply by decreasing
input position by context length.
The bug: the function treated 'r1 | r2' as 'r1 | r2'
(used 'r1' twice instead of 'r1' and 'r2').
As the function was rarely used the bug went unnoticed for about
two decades.
Before this patch [-Wmatch-empty-rule] was based on:
- DFA structural analyses (skeleton phase)
- rule reachability analyses (skeleton phase)
Now it is based on:
- NFA structural analyses (NFA phase)
- rule reachability analyses (skeleton phase)
It's much easier to find nullable rules in NFA than in DFA.
The problem with DFA is in rules with trailing context, both
dynamic and especially static (as it leaves no trace in DFA
states). re2c currently treats static context as dynamic, but
it will change soon.
On the other side NFA may give some false positives because of
unreachable rules:
[^] {}
"" {}
infinite rules:
[^]* {}
or self-shadowing rules:
[^]?
Reachability analyses in skeleton helps to filter out unreachable
and infinite rules, but not self-shadowing ones.
Before this commit, the list of rules depended on the order of NFA states
in each DFA state under construction (which is simply a matter of ordering
pointers to heap: the order can be different).
Now all rules for each DFA state are collected and the final choice of
rule is delayed until DFA is constructed, so the order of NFA states
no longer matters.
No serious changes intended (mostly cleanup and comments).
The underlying algorithm for finding strongly connected components
(SCC) remains the same: it's a slightly modified Tarjan's algorithm.
We now mark non-YYFILL states by setting YYFILL argument to zero,
which is only logical: why would anyone call YYFILL to provide zero
characters. In fact, re2c didn't generate 'YYFILL(0)' call itself,
but some remnants of YYFILL did remain (which caused changes in tests).
This commit fixes bug #132 "test failure on big endian archs with 0.15.3".
Tests failed because re2c with '--skeleton' option used host endianness
when serializing binary data to file. Expected test result was generated
on little-endian arch, while actual test was run on big-endian arch.
Only three tests failed (out of ~40 tests that are always run with
'--skeleton'), because in most cases data unit is 1 byte and endianness
doesn't matter.
The fix: re2c now converts binary data from host-endian to little-endian
before dumping it to file. Skeleton programs convert data back from
little-endian to host-endian when reading it from file (iff data unit
size is greater than 1 byte).
This commit fixes bug #132 "test failure on big endian archs with 0.15.3".
Tests failed because re2c with '--skeleton' option used host endianness
when serializing binary data to file. Expected test result was generated
on little-endian arch, while actual test was run on big-endian arch.
Only three tests failed (out of ~40 tests that are always run with
'--skeleton'), because in most cases data unit is 1 byte and endianness
doesn't matter.
The fix: re2c now converts binary data from host-endian to little-endian
before dumping it to file. Skeleton programs convert data back from
little-endian to host-endian when reading it from file (iff data unit
size is greater than 1 byte).
This mechanism was tricky and fragile; it cost us a most unfortunate
bug in PHP lexer: https://bugs.gentoo.org/show_bug.cgi?id=518904
(and a couple of other bugs).
Now that re2c does DFA minimization this is no longer needed. Hoooray!
The updated test changed because skeleton is constructed prior to
DFA minimization.
After minimization the resulting DFA is much smaller:
/*!re2c
[ac]{0,14} [a] [ac]{0,14} {}
*/
Was:
$ time re2c slow.re > slow.c && stat -c '%s' slow.c
real 1m54.837s
user 1m54.733s
sys 0m0.120s
5627102
Now:
$ time ./re2c slow.re > slow.c && stat -c '%s' slow.c
real 0m0.732s
user 0m0.684s
sys 0m0.048s
15078
Test results changed a lot; it is next to impossible to verify them
by hand. I therefore implemented two different minimization algorithms:
- "table filling" algorithm (simple and inefficient)
- Moore's algorithm (not so simple and efficient enough)
They produce identical minimized DFA (up to states relabelling), thus
giving some confidence in that the resulting DFA is correct.
I also checked the results with '--skeleton': re2c constructs
skeleton prior to reordering and minimization, therefore
skeleton-generated data is free of (potential) minimization errors.
ADFA stands for 'action DFA', that is, DFA with actions.
During DFA construction (aka NFA determinization) it is convenient
to represent DFA states as indexes to array of states.
Later on, while binding actions, it is more convanient to store
states in a linked list.
This partially fixes bug #128: "very slow DFA construction (resulting
in a very large DFA)". DFA construction is no longer slow, but the
resulting DFA is still too large and needs to be minimized.
Kleene star '*' (aka iteration, repetition, etc.) is a primitive
operation in regular expressions.
For some reason re2c used '+' as a primitive operation and expressed
'*' in terms of '+'. It is inconvenient, because all algorithms
described in literature are based on '*'.
Because we now express 'a+' as 'a* a', we have to set 'PRIVATE' attribute
on 'a': otherwize 'a' gets shared between the two occurences which causes
complex bugs.
Expressing 'a+' in a more intuitive way as 'a a*' rather than 'a* a'
causes the generated code to duplicate certain states. The generated code
is (supposedly correct), but re2c fails to deduplicate these states.
We therefore prefer 'a* a' expansion, which results in exactly the same
code as before.
Bootstrap lexer changed a lot: this change is caused by commit
a4c192f27a: "Reordered states in DFA.".
Changes in parser by this commit triggered lexer regeneration.
re2c used a complex and slow algorithm to split charset into
disjoint character ranges. This commit replaces old algorithm with
new (much simpler and quicker).
re2c test suite now runs 2x faster due to speedup in Unicode tests.
The worst dependency which 'include-what-you-use' fails to see
(and rightly so) is 'src/parse/lex.re' -> 'src/parse/parser.h'.
This dependency is caused by '#include "y.tab.h"' in 'src/parse/lex.re'.
Another ubiquitos issue is 'src/util/c99_stdint.h' ('include-what-you-use'
suggests to substitute it with '<stdint.h>').
And a couple of other dependencies that 'include-what-you-use' fails to see.
We use file extensions to encode re2c options.
Some (short) options are uppercase letters: e.g. '-D', '-F', '-S'.
There also short options for the same lowercase letters: '-d', '-f', '-s'.
This can cause filename collisions on platforms with case-insensitive
file extensions (e.g. Windows and OS X).
See bud #125: "[OS X] git reports changes not staged for commit
in newly cloned repository".
Fix: use long versions for options that uppercase options.
Disallowed uppercase options in 'run_tests.sh'.
The problem with pattern ordering first emerged on FreeBSD-10.2
(I was able to reproduce it with 'CXXFLAGS=-fsanitize=address').
Some tests failed because patterns reported by '-Wundefined-control-flow'
were sorted in different order than expected. This is because
patterns ordering was inconsistent: patterns were compared by length,
(it doesn't work for patterns of equal length). Now first ordering
criterion is length, and second criterion is lexicographical order.
This commit reduces the amount of memory consumed by '-Wundefined-control-flow':
re2c no longer allocates vectors on stack while deep-first-searching skeleton.
This commit also reduces the limit of memory for '-Wundefined-control-flow'
(64Mb edges -> 1Kb edges). Real-world programs rarely need that much.
The limit was so high to acommodate some few artificial tests (with lower
limit these tests cannot find shortest patterns).
This commit also removes the upper bound for the number of faulty patterns
reported by '-Wundefined-control-flow'. This bound was needed by the
artificial tests mentioned above: they produce lots of patterns.
Now these tests are limited with 1Kb of edges anyway.
Note that 1Kb limit is checked after each new pattern is added, so that
at least one pattern will fit in (even if it takes more than 1Kb).
Prefix of current path under construction is a multipath, because prefix
arcs have not been covered yet. Suffix can be a simple path (that is, a
multipath of width 1), because all alternative suffix arcs have already
been covered.
prefix suffix
_________ _________
... \ /
--------- o
_________/
But nothing prevents us from alternating suffix arcs also, as long as
suffix remains a single multipath:
_________ _________
... \ / ...
--------- o ---------
_________/ \_________
The resulting path's width is the maximum of prefix ans suffix width
(hence the growth in size of those tests in which suffix is wider
than prefix), but it only makes a small difference. And the generated
paths are more "variable".
If code units occupy 1 byte, then the generated path cover covers
*all* edges in the original DFA. If the size of code unit exceeds 1 byte,
then only some ~0x100 (or less) range values will be chosen
(including range bounds).
These defines were necessary to enable numeric limits definitions
(such as 'UINT32_MAX') in our local version of 'stdint.h' (which is
used on platforms that don't have system header 'stdint.h').
As noted by commit b237daed20, this
workaround doesn't work on FreeBSD, so re2c now uses 'numeric_limits.h'.
Warning was introduced in commit b237daed20:
compiler fails to recognise (or deliberately choses not to recognize)
'std::numeric_limits<...>::max()' as a special constant.
UINT32_MAX is conditionally defined only
for C compiler on FreeBSD but not for C++,
Stop using __STDC_LIMIT_MACROS workaround
as it does not work on FreeBSD.
Use std::numeric_limits<> from C++98 instead.
Signed-off-by: Sergei Trofimovich <siarheit@google.com>
Crashes observed on platforms OS X (clang-7.0.0) and FreeBSD-10.2 (clang-3.4).
First reported in bug #122 "clang does not compile re2c 0.15.x".
What caused NULL passed to 'operator <<': re2c always generates content of
header file (regardless of '-t --type-header' option), but the content is
dumped to file (and header filename initialized to non-NULL) only if the
option was enabled.
Fix: always initialize header filename to non-NULL string.
On OS X bison generates token enums as CPP macro
constants (y.tab.h):
#define CODE 260
while on my box it's
enum yytokentype {
...
CODE = 260,
...
};
That #define causes symbol collision as:
../src/parse/lex.re:169:38: error: expected unqualified-id
else if (opts->target == opt_t::CODE)
^
src/parse/y.tab.h:58:14: note: expanded from macro 'CODE'
#define CODE 260
Renamed enum entry to TOKEN_CODE.
See bug #122 "clang does not compile re2c 0.15.x".
Example of error on Mac OS X:
src/codegen/emit_dfa.cc:250:65: error: use of overloaded operator '<<' is ambiguous (with operand types 're2c::OutputFile' and 'const size_t'
(aka 'const unsigned long'))
o << indent(ind++) << "static void *" << opts->yyctable << "[" << conds << "] = {\n";
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~
./src/codegen/output.h:84:22: note: candidate function
friend OutputFile & operator << (OutputFile & o, char c);
^
./src/codegen/output.h:85:22: note: candidate function
friend OutputFile & operator << (OutputFile & o, uint32_t n);
^
./src/codegen/output.h:86:22: note: candidate function
friend OutputFile & operator << (OutputFile & o, uint64_t n);
^
On OS X 'size_t' is neither 'uint32_t' nor 'uint64_t', resolution is therefore ambiguous.
* master:
Updated version to 0.14.4.dev
Release 0.14.3.
Added simple test for yacc-style brackets (see patch #27)
Fixed '#27 re2c crashes reading files containing %{ %}' (patch by Rui)
Makefile.am: dropped distfiles for MSVC (they are broken anyway)
Added full another test for bug #57.
Updated version to 0.14.3.dev
Release 0.14.2.
Fixed bug #57: Wrong result only if another rule is present
Updated version to 0.14.2.dev
Release 0.14.1.
Pad version with '0' instead of nulls
Default rule '*' (not to be confused with 'none' rule) used to have
normal number just like other rules. Now that re2c has to distinguish
default rule fro other rules (because of [-Wunreachable-rules]),
it reserves a special number (UINT32_MAX - 1) for it.
With this option script runs re2c with '--skeleton' and
'-Werror-undefined-control-flow' and instead of comparing results with
reference test results, it compiles the generated skeleton programs and
runs them. If C compiler or binary return nonzero error status, script
reports an error. Note that cases when re2c failed to generate code are
not considered errors (re2c has lots of test cases for its errors).
re2c lacks submatch extraction; it would be much more convenient
to memorize input positions for some parts of regular expressions
than break each regexp in the middle and move parts to separate blocks.
Submatch extraction is dificult to implement in general, but supporting
submatch in some simple cases (like the case where trailing context is
allowed) would be not so difficult and most helpful.
This removes a lot of copy-pasting.
The change of error location in test is insignificant: the reported
location was incorrect and it still remains imprecise.
Warns about unreachable rules:
- rules that are shadowed by other rules, e.g. rule '[a]' is shadowed by
'[a] [^]'
- infinite rules that consume infinitely many characters and fail on
YYFILL, e.g. '[^]*'
- rules that contain never-matching link, e.g. '[]' with option
'--empty-class match-none'
default rule '*' should not be reported
re2c used to postpone merging default rules because rank counter could
only assign consequtive ranks to rules, and default rules must have
the lowest priority. Now rank counter has been modified to return
special value as defult rule rank.
It's not a bunch of unnecessary warnings I want to avoid, it's a bunch of
unnecessary runtime failures in programs generated with '--skeleton'
(failures caused by undefined control flow; re2c recogizes such cases
and the generated program reports a warning before failing).
Trialing contexts are currently broken (overlapping trailing contexts
cannot be tracked with a single 'YYCTXMARKER'). For now, re2c with
'--skeleton' mimics this incorrect behaviour: information about context
is lost by the time DFA is constructed, so skeleton has no way to
figure out the right order of things.
Prior to this commit backup of trailing context position was done
before advancing input position and re2c either had to emit
YYCTXMARKER = YYCURSOR + 1;
(with default input API), or
YYRESTORECTX ();
YYSKIP ();
(with custom input API).
The problem is that sometimes initial state doesn't sdvance input position
at all. Now re2c emits context backup after advancing input position and it
no longer needs '+1' or 'YYSKIP' hacks. It always backups the correct position.
'-r' is different from normal mode in two aspects:
- single DFA may be used multiple times (unchanged, we only
need a single copy for skeleton)
- DFA may be generated but not used at all
This commit changes the behaviour of three groups of options:
re2c:define:YYSETCONDITION
re2c:define:YYSETCONDITION@cond
re2c:define:YYSETCONDITION:naked
re2c:define:YYSETSTATE
re2c:define:YYSETSTATE@state
re2c:define:YYSETSTATE:naked (added by this commit)
re2c:define:YYFILL
re2c:define:YYFILL@len
re2c:yyfill:parameter
re2c:define:YYFILL:naked
The changes should be backwards compatible (meaning that old code that
compiled should still compile), but it may add empty statements or statements
with no effect for some configurations, e.g.:
YYSETCONDTITION(0);(0);
These changes were necessary to unify re2c behaviour, remove counter-intuitive
cases and make it possible to write comprehensible option descriptions.
In short, the changes are:
- 'naked' triggers generation of argument-in-braces and semicolon;
- 'parameter' triggers generation of argument-in-braces (when applicable,
'naked' has priority over 'parameter');
- argument templates ('@cond', '@state', '@len') don't force other
configurations, they also don't influence on argument-in-braces;
Added test generator and autogenerated tests.
This commit removes check (and error) for overwritten configurations
(like setting 're2c:define:YYCYRSOR' twice in the same block).
This check was in principle useful, but it was applied to somehow
randomly chosen set of parameters. If in future we'll feel a need
for such check, it should respect all options equally and report
warning rather than error.
Normally re2c generates single 'yyaccept' variable for all conditions.
With '--skeleton' re2c handles conditions separately, so each condition
needs (or needs not) its own 'yyaccept'.
Prior to this commit re2c used the same criterion to determine if
'yyaccept' is needed with '--skeleton' as it uses generally: whether
'yyaccept' was used in any of conditions. Now re2c looks if 'yyaccept'
was used with this particular condition.
Tests 'condtype_yysetcondition.c{s,g}.re' show the reason why I changed
how '-Wcondition-order' works in presence of 'YYSETCONDITION' calls:
programs generated from these tests work differently depending on
condition numbering. Explicit use of condition names cannot guarantee
that these explicit names were generated by re2c (and not hardcoded as
in these examples).
Determine maximal path length and maximal rule number while constructing
skeleton; take maximim of these two values; choose unsigned integer type
of minimal width capable of holding maximim.
Note: re2c operates on exact-width integers, but the generated program
doesn't (it might not have <stdint.h>). When generating the program,
re2c choses one of unsigned 'char', 'short', 'int' and 'long' types
(that one 'sizeof' which is equal to the disired key size). re2c makes
some implicit assumptions (generated program is run on the same platform
as re2c, byte consists of 8 bits, etc.). Perhaps re2c should hardcode
these assumptions in the generated program and check them on start.
A single key is formed of three values:
1. the length of string
2. the length of matched part of string
3. the number of matched rule
All these values are guaranteed to fit 32 bits, so for now we just
dump them as 'uint32_t' and read as 'unsigned int'. re2c asserts that
'sizeof (uint32_t) == sizeof (unsigned int)'.
Avoid structs, as they cause padding issues.
Maximal skeleton path length is a bit different from YYMAXFILL:
it assumes that loops are iterated once (unlike YYMAXFILL calculation,
which disregards loops) and returns zero for empty regexp.
We need to know it in order:
- to be sure it won't overflow
- to store keys in a compact form (yet to be done)
This commit also makes DFA and skeleton store condition name and
source file line corresponding to current condition: it gets quite
annoying to pass these things around. This change caused another
change of test results (line numbers in error messages changed
for tests that use '-r' and reuse old DFA (don't reconstruct DFA
in 'use:re2c' blocks).
This let us create skeletom right after DFA creation (but befor DFA
has been mangled in different ways), but call skeleton methods any time.
Undefined control flow is now checked at the time of real code generation,
that's why all those tests that use '-r' changed: re2c stopped reporting
'rules:re2c' blocks and reports 'use:re2c' blocks instead.
The algorithm now stores partially constructed paths in a compact
form and delays expansion until it reaches the end of current
branch of recursion. This has several advantages:
- no need to store large structures in memory
- write data to file in large chunks
- path expansion is faster than step-by-step construction
Speedup is actually tenfold (but the way keys are dumped to file is
still not optimized and spoils benchmarks). Real speedup can be
observed on such files as:
test/php20150211_zend_ini_scanner_trimmed.icF.re
, which cause ~5Gb dumps. Time has gone down: 8.5m -> 2.5m and will
be furter reduced to ~1m when key dumps are fixed. This will result
in generation speed ~20Mb/s which is quite good.
The simplest example I was able to come up with that reveals eternal
loop is the following:
/*!re2c
( [^acf] | "0b" | "a"[^] | "a0"[^] )+ {}
*/
The problem was caused by my assumption that from any node there is
at least one non-looping path to end node. The assumption is true;
what I didn't take into account was that all such paths may go via
nodes that have already occured twice on the way to current node
(their loop counter is greater than 1). In this case the algorithm
would find no path to end node. Since not all prefixes would have
been covered (exactly none of them) the algorithm would loop forever.
Such branch may be abandoned safely: the algorithm will later find
another path to current state without loops.
As soon as I realized the problem the fix was trivial: if all outgoing
arcs have been exhausted and none of them yielded any results, abandon
current branch.
There's a limitation on the size of input files for C/C++ compiler and
the compiled binary will have to contain all that data (and thus may grow
very large).
Storing data in binary form and reading it from file dynamically is
the way it should be.
Store length of strings instead of pointers to string start and end.
Storing pointers requires us to remember total length of strings already
written to file by the time we want to write next string. This is very
inconvenient if we want to dump strings as we perform DFS on graph:
we'll have to track size all the time (path-cover-generator already does
that, but all-paths-generator doesn't).
This adds a new local variable to the generated code ('token'), which
is used to backup cursor position when enterind DFA.
This is necessary to dump generated data as soon as possible instead
of keeping it until all data has been generated: we generate input
strings and keys simultaneously, but have to write all input strings
at once as one big string. Keys alone occupy lots of space, so
keeping only keys instead of keys and strings won't help.
We don't have any fallback algorithm in case path cover is too large.
We want to generate some paths anyway, so we have to construct path cover.
We shouldn't generate arbitrary large amounts of data.
With '--skeleton' re2c builds DFA skeleton and performs DFS in order to
estimate the size of data to be generated. It maintains size counter: as
soon as the counter reaches certain limit, DFS should stop.
Size counter is always checked when recursion returns.
Sometimes it is clear that size counter will overflow upon recursion
return (when arguments overflow already) and DFS can exit early (before
entering recursion).
This commit omits checks of some arguments (and correponding early exits
from DFS): first, path length is very unlikely to overflow (one has to
write/generate a regular expression with length of ~1Gb, in which case
skeleton generation won't be the worst problem); second, the number of
outgoing arcs in each vertex is also highly unlikely to exceed 1Gb limit.
Renamed '-Wnaked-default' to '-Wundefined-control-flow': the latter sounds
much scarier. :D
Completely changed the algorithm that is used to determine if default case
is not handled properly. Prior to this commit a simple and incorrent criterion
was used: whether there are code units that (alone) do not match any rule.
This gived false positives in cases like this:
[^] [^] { rule }
here all code units meet the criterion: no single code unit matches a rule.
But obviously, default case is handled properly, because any input string
matches 'rule' (strictly speaking, any input string of length 2 or more, but
that's YYFILL's problem).
The new algorithm is more complex (in terms of time and space), yet it is
less heuristic: re2c parforms exhaustive deep-first-search on graph skeleton
and collects all bad paths.
For some reason (which I yet do not understand) re2c uses size of DFA
state's "kernel" to determine whether a new state should be constructed
or some already constructed state (of the same "kernel" size) will do.
Since re2c knows "kernel" size for each state anyway, there's no need
to NULL-terminate "kernel" array while iterating over it: we can use
pre-calculated size instead.
Comparing against pointer to the end of array is easier than calculating
array length, then setting next-to-last element to NULL and iterating
while not NULL.
Turns out that re2c allows empty code blocks:
/*!re2c
<a> "a" :=
*/
re2c up to 0.14.3 handled this correctly (generated empty action).
Since then this behaviour has been broken: re2c started to autogenerate
jump to nonexistent condition.
In theory re2c makes no guarantee about the order of conditions in
the generated lexer. Users should use 'YYGETCONDITION'/'YYSETCONDITION'
interface and generate enum with condition names using either '-t'
option or '/*!types:re2c*/' directive. Only members of the generated
enum should be used with 'YYGETCONDITION' and 'YYSETCONDITION'.
This way code is independent of internal re2c condition numbering.
However, it turns out that some users (and notably, PHP) rely on
internal condition numbering: they manually hardcode condition numbers
and make re2c generate condition dispatch in such a way that it
doesn't mention condition names at all (e.g. with in the form of nested
'if' statements with '-b' or in the form of 'goto' table with '-g').
The code compiles, but change of code generation mode may break
compilation. Even worse, change of internal re2c condition numbering
may lead to epic runtime failures.
So re2c cannot just change internal condition numbering: it has to
preserve the existing numbering scheme (rather illogical and not
alphabetically sorted).
This commit tries to simplify tracking of condition numbers.
Consider the following example 1.re:
/*!re2c
"" {}
*/
With -if, re2c would generate correct dispatch:
$ re2c -if 1.re
/* Generated by re2c 0.14.3 on Tue Aug 25 10:41:45 2015 */
switch (YYGETSTATE()) {
default: goto yy0;
case 0: goto yyFillLabel0;
}
yy0:
YYSETSTATE(0);
yyFillLabel0:
{}
With -bif all positive YYGETSTATE() values will lead to yyFillLabel0,
which is clearly an error: values that are greater than 0 should lead
to yy0:
$ re2c -bif 1.re
/* Generated by re2c 0.14.3 on Tue Aug 25 10:40:32 2015 */
if (YYGETSTATE() < 0) {
goto yy0;
} else {
goto yyFillLabel0;
}
yy0:
YYSETSTATE(0);
yyFillLabel0:
{}
With -gif the error is different: all values greater than 0 now cause
undefined behaviour (access to memory not within array bounds):
$ re2c -gif 1.re
/* Generated by re2c 0.14.3 on Tue Aug 25 10:47:41 2015 */
{
static void *yystable[] = {
&&yyFillLabel0,
};
if (YYGETSTATE() < 0) {
goto yy0;
}
goto *yystable[YYGETSTATE()];
yy0:
YYSETSTATE(0);
yyFillLabel0:
{}
}
The situation gets even more tricky with re2c:state:abort configuration.
Besides, YYGETSTATE() macro is called multiple times with -b and -g,
which is not so good. Additional checks with -g don't help gain performance
either.
Fix: always generate simple switch.
The mechanism of updating bootstrap parser was broken by commit
b2bc24704e : "Support 'make distcheck'."
Before that commit bootstrap parser was updated every time when autogenerated
parser was updated. This approach contradicted with 'make distcheck' which
mounts top source directory read-only (including bootstrap files). That commit
tried to solve the problem by checking if the autogenerated parser has changed
and only then copying it to bootstrap parser.
I made two mistakes:
- philisophic mistake: autogenerated parser was actually altered by
'make distcheck' because of altered paths in '#line' info;
- shell scripting mistake: the check if the autogenerated parser has
changed was inverted, so bootstrap parser was never updated;
This commit makes use of two observations:
- bootstrap parser depends on the source parser, not on the autogenerated
parser;
- bootstrap parser shouldn't at all contain line information: at the time
it is copied to build directory the relative path from build directory
to source directory may be completely different (as it is with 'make
distcheck');
re2c allows configuration strings to contain escape sequencs as long
as the escaped code points fit into 1-byte range.
Finally both 'Scanner::unescape' functions can be removed along with
a whole bunch of [-Wsign-conversion] warnings. :)
Tests do not ensure that configurations are handled correctly or
have any impact on re2c behaviour. They only ensure that these
configuration names are recognized as valid.
Make re2c do all the dirty work instead of barely recognizing
configuration-like lexeme and manually matching the lexeme against
all allowed configuration names.
This commit adds many new tokens, which means that re2c-generated lexer
and bison-generated parser have both become larger. The difference
is not as great as one might expect: ~60 Kb -> ~80 Kb in both cases.
The problem starts with lexer: it should operate on unsigned chars
(as re2c wants). Then these casts won't be needed at all. Mixing
signed and unsigned chars is bad and lexer should be rewritten to
use unsigned chars.
Like C language, re2c allows all characters of current charset excapt for
newline and backslash to apper unescaped in string and class literals.
Added tests:
- all possible characters in strings and classes (omitting
newline, escaping backslash and quotes/closing bracket)
with different encodings (re2c up to 0.14.x would segfault
on this test with '-u' and '-w')
- string with newline
- string with unescaped backslash followed by newline (so that
it won't stick to next character)
First fix:
re2c used to check if code point exceeds maximal value for current
encoding when parsing it. When I moved parsing code points to lexer
I forgot the check.
Second fix:
I assumed that 'static_cast<uint32_t>' on 'signed char' zero-extends.
But I was wrong: it sign-extends. Need to cast to 'unsigned char'
instead.
Such strings can only contain ASCII letters, digits and uderscore:
no escapes. So there's very little parsing to do: just map code units
directly to code points.
This commit is a continuation of e42f271e4b:
"Parse character classes in lexer." It makes lexer parse individual code
points instead of barely recognizing string of code points.
Since the contents of case sensitive strings, case insensitive strings
and character classes are lexically the same (except for escaped quotes
or closing bracket), we can parse them all in the same lexer block.
Note that re2c allows any 1-byte code points. Maybe we should report
non-ASCII or non-printables.
Before this commit, lexer would barely recognize class as a whole
lexeme and pass it to functions that would further parse is by manually
picking individual code points and control characters out of the lexeme.
Heh, re2c is made for such kind of stuff (and it does it much better).
So now lexer parses individual code points or control characters and
stores them as a sequence of code points. Then an outer function
splits this sequence into ranges and individual characters.
I tried to preserve existing behaviour (judging from test suite only
the text of some error messages has changed). Added some autogenerated
tests (and the generator script itself), but these tests are not exhaustive.
Now "{0,}" is handled by general case for counted repetition. Had to
fix test because of swapped alternatives in construction of regexps
of the form "r*" and "r{n,".
Added some tests for counted repetition.
Got some errors about undefined 'UINT32_MAX' and others while trying
to build re2c with MINGW. These variables are defined conditionally in
<stdint.h> depending on the following defines:
__STDC_LIMIT_MACROS
__STDC_CONSTANT_MACROS
Fix: defined these symbols in the beginning of "src/util/c99_stdint.h".
Also found out that "src/util/c99_stdint.h" didn't compile in the
absense of system <stdint.h> (which is exactly the case when it was
written for). THe problem was caused by incorrect use of 'typedef'
statement; fixed easily. What a shame I didn't compile it before ;D
Using 'atoi' displeased me for several reasons:
- 'atoi' doesn't check for overflow
- by the time 'atoi' was called, re2c has already parsed input
string and knows that it is well-formed, no need to do it twice
- atoi returns 'int', custom function allows to refine types and
avoid casts between signed/unsigned values or [-Wsign-conversion]
warnings
Added test for new conversion functions.
Counted repetitions are regexps of the form:
r{n}
r{n,m}
r{n,}
They can be expressed in terms of concatenation, alternation and
iteration:
r{0} <empty regexp>
r{n} r{n-1} r
r{n,m} r{n} (r{0} | ... | r{m-n})
r{n,} r{n} r*
Reducing special regexp type to represent counted repetitions allows
us substitute complex code that compiles them to bytecode instructions
with simpler code that constructs counted repetitions using concatenation,
alternation and iteration.
Token length equals the difference between two pointers: YYCURSOR
value on the moment of successful match and YYCURSOR value when
entering DFA. This difference should be nonnegative and fit buffer
size.
These two variables ('ScannerState' class member 'cur' and local variable
'cursor') both played the role of YYCURSOR: YYCURSOR was defined to
'cursor', but the code had to adjust 'cur' to 'cursor' on every return
and restore it when reentering lexing procedures.
There was only two obscure places in which they were not completely
syncronized. Both looked the same:
cur = ptr > tok ? ptr - 1 : cursor;
However, 'ptr' is effectively YYMARKER and is never changed manually
by lexing procedures; 'tok' should point to the beginning of current
lexeme; this check doesn't make sense to me. This is an attempt to
resume lexing one character before YYMARKER, but what for? YYMARKER
isn't even used in Scanner::scan (look at the the generated code),
so 'ptr > tok' is always false.
This check is present since the initial commit, so I cannot trace back
it's origin.
As it doesn't break any tests I hereby simply remove it.
The bug was triggered by the following conditions: '-b'
(or any flag that implies '-b') and encoding with code units
wider than 1 byte (that is, UTF-16, UTF-32 and UCS-2).
The bug itself: re2c omitted 'goto <some state>;' when it
should have been generated. The absense of goto caused incorrect
control flow. One cas verify it in two ways using the minimal test
added by this commit:
- manually compare the code generated before and after fix
and ensure that newer version generates correct code;
- run the test with --skeleton, then try to add/remove
the goto statement, compile and run: the code without
goto will fail, the code with goto will not;
git-bisected commit which fixed bug:
commit e42003d805
Author: Ulya Trofimovich <skvadrik@gmail.com>
Date: Wed Mar 11 12:46:58 2015 +0000
Pass limited span instead of checking range all the time.
Also gound commit that introduced bug:
commit d10fb601f3
Author: helly <helly@642ea486-5414-0410-9d7f-a0204ed87703>
Date: Fri Dec 30 16:24:07 2005 +0000
- Allow to use -w with -b
# Still quite some work to do but it is possible
Much the same as previous commit, only this time YYCTYPE has been
erroneously set to 'char' instead of 'unsigned char. It didn't
cause any warnings or errors becase regular expressions contain
only ASCII range characters. However, relying on this is unsafe:
hypotheticlly re2c can generate some strange comparisons that compile
but fail in runtime.
Explicit cast makes sure thet all comparisons are between unsigned
values.
Source code is in ASCII: pointers have type 'char *', but re2c
makes an implicit assumption that YYCTYPE is unsigned when it
generates comparisons.
At first glance making YYCTYPE 'char' instead of 'unsigned char'
seems to fix the problem (no more warnings). But it's only because
re2c-generated lexer happens to make no comparisons with constants
larget than 0x7F. Lexer rules may change a bit and then we'll have
lots of warnings about 'comparison between signed and unsigned
values' and 'case label exceeds maximal value for type', not
mentioning that it may lead to real errors.
Explicit cast makes sure thet all comparisons are between unsigned
values.
Though all enum values are handled in switch, GCC still complains
that 'control reaches end of non-void function'. Adding default
clause helps and it's better than returning some spurious constant.
-y stands for yacc compatibility, we don't need it since we no longer
accept yacc instead of bison.
-d stands for defines file, we already specify that in makefile rule
for parser.
--no-lines is generally useless and didn't have any effect for some
reason (#line directives are present).
Found some errors while testing mingw-built re2c.exe on wine:
1. redirecting stderr to stdout resulted resulted in random
mix in tests that triggered re2c warnings
2. removing all '\r' symbols in the generated files didn't
work anymore since some tests that contain '\r' were added
Fixed:
1. redirect both to different files and cat manually
2. diff with -b flag on wine
The problem with the algorithm was that when one and the same skeleton
node is visited for the 3rd time, the algorithm would abandon this
branch and drop all the path prefixes that lead to it. It's ok when
bruteforcing all paths (the arcs in the dropped prefixes will occur in
other paths that lead to final states), but it's not ok for path cover:
some arcs may have occured only in these dropped prefixes).
The fix is easy: just check if the branch was cancelled and in case it
was try the same prefixes with another branch.
This small example demonstrates the problem:
/*!re2c
[^\x00]* [\x00] {}
*/
DFA skeleton looks like this:
state 0:
arcs to state 0: [0x01, 0xFF]
arcs to state 1: [0x00]
state 1: final
Before the fix, re2c would generate the following paths (provided
path cover construction is forced instead of generating all paths):
- 0x01,0x00
- 0x00
Clearly the [0xFF] arc looping from state 0 to state 0 is missing
(uncovered). After the fix:
- 0x01,0x00
- 0xFF,0x01,0x00
- 0x01,0xFF,0x00
- 0x00
Not a minimal path cover, but we don't need it. We just need a small
enough one.
Much the same as previous commit: array of length bounded with 32 bits,
first pointer points to some random element. second pointer points to
the beginning of array (unless there were some errors in calculatins
that would have likely crashed re2c anyway).
Fixes [-Wconversion] warning.
This time we deal with pointers in 'Ins' array (the length of array
equals the calculated size of regular expression, which is currently
of type uint32_t: larger regular expressions will surely crash re2c).
So we can guarantee that, if regexp was correctly compiled to instructions,
than the first pointer points to some instuction in the array and second
pointer points to the beginning of array, so this cast is safe.
If compilation was incorrect than re2c will fail anyway.
Fixes [-Wconversion] warning.
In theory, 'top' points at the top of stack and it's value is always greater
than or equal to 'stk' that points to stack bottom. Total stack size is
equal to 'DFA::nStates', which is of type uint32_t (DFAs larger than
2^32 states are currently not supported an will crash re2c).
In practice, 'stk' is not changed and 'top' is only incremented
before the cast (it's decremented afterwards. Note that the function
is self-recursive).
Fixes [-Wconversion] warning.
In theory, this function returns the number of 'Span's in the newly
constructed 'Span' array, so it must be a non-negative integer number
that fits into 32 bits and the cast is safe.
In practice, 'x0' variable stays unchanged in this function, while
'x' variable's value can only increase: whenever it can be (conditionally)
decremented, it is always unconditionally incremented. So 'x - x0'
must be non-negative.
Fixes [-Wconversion] warning.
'specMap' parameter can sometimes be NULL (when not in '-c' mode).
In code it was dereferenced before the check for '-c', but due to
compiler optimizations this was never revealed.
I found the bug while trying to measure the size of 'specMap'
before the check (caught segfault). Fixed by moving the check prior
to dereferencing.
Warning status can be in fact represented with only 2 bits, but
since it's engaged in arithmetic operations it gets promoted and
special precautions are needed. Much easier to ure 32 bits:
warnings are nowhere near performance/memory bottleneck.
Though this length shouldn't actually excedd 32 bits (othewise
too much data will be generated, and that is checked), it's better
to use size_t and don't care about the order of checks.
With --skeleton switch we need to generate lots of data: strings that
correspond to various paths in DFA and match given regular expression.
For small graphs we can afford to generate all paths, for large graphs
we can only generate path cover. Anyway we need to be able to estimate
the amount of data to be generated (measured in skeleton arcs). Since
it can easily exceed 32 bits (and 64 as well), calculations must stop
as soon as certain limit is reached.
8-bit and 16-bit unsigned integers used in arithmetic operations
are promoted to 32 bits before operation and then truncated back.
Theoretically this may change their value.
This fixes a lot of [-Wconversion] warnings.
This commit removes 10 shift/reduce conflicts in bison grammar for re2c.
These conflicts are caused by allowing flex-style named definitions
name regular-expression
to contain newlines and to be mixed with rules. It's not just some
conflicts in LALR(1) grammar, it is genuine ambiguity as can be observed
from the following example:
/*!re2c
name "a"
"b" "c" {}
*/
which can be parsed in two ways:
definition -> name "a"
rule -> "b" "c" {}
and
definition -> name "a" "b"
rule -> "c" {}
, both ways being perfectly valid.
This commit resolves ambiguity by forbidding newlines in flex-style
named definitions (conforming to flex syntax). Newline in these
definitions is treated in a special way: lexer emits token 'FID_END',
which marks the end of flex-style named definition in parser.
-W: enable all warnings
-Werror: turn all enabled (hereafter) warnings into errors
Specific warnings added by this commit:
-Wempty-character-class
-Wempty-rule
-Wnaked-default
For each warning there are -Wno-, -Werror-, -Wno-error- options.
This time I found all tests that are affected by this option
(that is, contain empty ranges) and added explicit option variants
for most of them (excluding 'test/empty_range.*' group, as we already
have 'test/bug61_positive.*' group that checks the same thing).
This option controls re2c actions when it encounters empty character
class (e.g. [], [^\0x00-\xFF] or [\0x00-\xFF]\[\0x00-\xFF]):
match-empty (default) - match on empty input
match-none - fail to match on any input
error - compilation error
This is a final fix for bug #61 "empty character class [] matches empty string".
Given the following code:
/*!re2c
[] {}
*/
/*!re2c
[^\x00-\xFF] {}
*/
/*!re2c
[\x00-\xFF]\[\x00-\xFF] {}
*/
re2c versions <=0.13.6 and >=0.13.7 behaved differently.
0.13.6 consistently considered that empty range should match empty string.
Since 0.13.7 empty positive range [] and empty difference (e.g. [a-z][a-z])
still match empty string, but empty negative range (e.g. [^\x00-\xFF])
matches nothing (always fails). The faulty commit is
28ee7c95bc
"Added UTF-8 encoding support and tests for it."
This commit brings back consistent behaviour of 0.13.6: empty range,
however it was constructed, always matches empty string. Whether this
behaviour is sane or not is another question.
NULL range represents empty range: range union and difference functions
return NULL for empty ranges. Thus NULL can be passed to functions
that construct regexp from range ('MatchOp', 'UTF8Range' and 'UTF16Range').
All these functions must behave return non-NULL for NULL ranges, since
further code relies on this.
Better have an assert than nothing until we handle such cases properly.
As for now, if the user inputs range like [9-0], it will be tranformed
to [0-9]. Later re2c should at least warn about such cases.
Ranges must be constructed so that linked ranges don't overlap and
are monotonous. This is always true for one-link ranges created by
range constructor, and we construct larger ranges from them using
union and difference functions (that maintain the invariant).
Inspired by sudden collision with '__COUNTER__' I occasionally got
while guarding 'src/util/counter.h'.
Now all headers use guards of the form '_RE2C_PATH_TO_HEADER_BASENAME_'.
Some compilers (e.g. clang++) would warn about [-Wreserved-id-macro],
but their reasonong about reserved macro names is quite crude and re2c
is not able to confirm their standards anyway (e.g. autoconf-generated
macro name 'SIZEOF___INT64').
As with labels, try to control how rule priorities are created:
make a special counter that creates new priorities and disallow
everyone but this counter do it.
New simpler implementation uses STL containers instaed of sparse
array. It's less efficient, but the place is not a bottleneck and
simplicity is more important than efficiency.
Now rule states and default state are added to DFA in separate
independent passes.
Removed 'bSaveOnHead' DFA property as it can be calculated right
before setting initial state action.
Bug description: sometimes re2c would generate code that backups
current input position (e.g. 'YYMARKER = YYCURSOR'), but wouldn't
generate code that restores backuped position (e.g. 'YYCURSOR =
YYMARKER').
Analyses: DFA may have overlapping rules (e.g. "a" and "aaa").
In such cases, if the shorter rule matched, lexer must attempt to
match the longer one. If the longer rule also mathed, then lexer
prefers it to the shorter rule. If the longer rule didn't match,
lexer must backtrack input position to the point when the shorter
rule matched. In order to be able to backtrack, re2c must generate
backup code (e.g. 'YYMARKER = YYCURSOR') and restore code (e.g.
'YYCURSOR = YYMARKER').
In some rare cases DFA has overlapping rules, but if the shorter rule
matched, then the longer rule will always match (perhaps on an
arbitrary long input string), e.g.:
/*!re2c
[^]+ "a" { 1st }
"b" { 2nd }
*/
In this cases there's no need to generate backup code for 2nd rule:
lexer will either encounter final "a" and the 1st rule will match
or YYFILL will not return; anyway, restore code will never be run.
re2c used to output backup code but not restore code in such cases.
This is the bug: backup code is useless without restore code and
should be omitted.
In future re2c should warn about such cases (when the shorter of
two overlapping rules is shadowed by the longer one).
The fix: postpone insertion of save actions (those with backup code)
untill it is known if restore code will be generated.
I also removed obsolete global variable 'bUsedYYMarker', which was
always set to 'true' (it should be per-DFA, not per-block configuration
anyway).
We have one 'yyaccept' initialization per re2c block. Each block
consists of one or more DFA (multiple DFA in '-c' mode in case of
multiple conditions). Each DFA may or may not use 'yyaccept'
(that is, save 'yyaccept' in some states and have a dispatch state
based on saved 'yyaccept' value).
Description of the bug: in '-c' mode, sometimes a DFA would have
states that save 'yyaccept', but no dispatch state that uses that
saved values. DFA didn't actually need 'yyaccept' (all the
assignments vanished if other conditions that need 'yyaccept' were
removed).
The essence of the bug: re2c decided whether to output 'yyaccept'
related stuff on a per-block basis: for multiple conditions in the
same block, the same decision was made (if any condition needed
'yyaccept', all of them would to output it).
The fix: 'yyaccept' initialization should be done on a per-block
basis, while assignments to 'yyaccept' should be done on a per-DFA
basis. Also, 'yyaccept' initialization must be delayed, while
assignments to 'yyaccept' must not.
Note: we may consider per-DFA 'yyaccept' initialization (have a
local 'yyaccept' variable per DFA). This wouldn't conflict with '-f'
switch (as it might seem) as long as we name all the variables
'yyaccept' and don't generate any 'yyaccept' initializations with '-f'.
As automake manual (chapter 27.6 "Flag Variables Ordering") states,
CXXFLAGS is a user variable and should be left for users to override
C++ compiler flags. Thus we should leave CXXFLAGS as is and modify
AM_CXXFLAGS insted.
Instead of unconditionally setting CXXFLAGS in Makefile.am,
check the presence of a flag in configure.ac. If the flag is
present (that is, an attempt to compile an empty C++ program with
this flag is successful), then it is added to CXXFLAGS.
Now one can add *any* compiler flag in configure.ac without
worrying about portability.
The only problem with 'make distcheck' was that we needed write
access to top source directory ('make' wanted to overwrite bootstrap
parser if it was built with bison and 'make check' wanted to create
temporary files in 'test/' directory).
This commit fixes it:
- 'make' doesn't try to overwrite bootstrap if it is identical
to the existing one (must always be true for 'make distcheck')
- testing script makes a temporary directory and keeps all
temporary files there. If some tests failed, temporary files
for them are left and test sources and reference results are
copied into temporary directory to make debug more convenient.
This commit makes use of 'make distcheck' in release script.
As I updated automake to version 1.15 it began to produce lots of
warnings about 'subdir-objects' not used when it should have been.
Turns out that 'subdir-objects' will be on by default in 2.0.
So I tried to turn on 'subdir-objects' and builds began to fail:
automake didn't expand '$(sourcedir)' and '$(builddir)' prefixes.
I erroneously prepended these prefixes in commit
38f526d044 "Updated build system.",
as commit message says:
...
Makefile.am:
- explicitly prefixed all file names with $(srcdir) or $(builddir)
...
But automake prepends these prefixes already where necessary, except
for custom rules with side effects: if a custom rule touches some
files that are not explicit targets or dependencies of this rule,
then automake won't find these files unless they are in build directory.
We have such side-effectful custom rules:
- parser rule produces multiple files and touches bootstrap files
- scanner rule touches bootstrap file
- doc rules touch bootstrap files
Multiple files is a common problem of make. Bootstrap introduces
circular dependency, since bootstrap files need to be updated after
they've been used. So it's hard to get rid of side effects in these
rules.
This commit enabels 'subdir-objects' feature and removes all prefixes
in variables and adds them in side-effectful custom rules (for files
from source directory, not for files from build directory). It also
makes use of '$@' and '$<' special variables in custom rules (which
makes side effects more explicit).
Still I don't yet fully understand how automake uses things like
'$(sourcedir)' and '$(builddir)' and their relation with 'subdir-objects'
(it's probably related with non-recursive makefiles).
Rationale: the equivalence of initial label to
're2c::label_counter_t::FIRST' is NOT a proper criterion
and pretty-printing shouldn't rely on it. The real criterion
is something like "(first re2c block OR any use block in '-r'
mode) AND first condition in'-c' mode", but it's spurious and
introduces unnecessary complications.
Droping this newline allows us drop equivalence operator for
labels.
Used the following bash script to ensure that all the changes
in tests are caused by missing newline(s):
#!/bin/bash
for f2 in *.temp
do
f1=${f2%.temp}
diff1=`diff $f1 $f2 | grep '^< ' | wc -l`
diff1_line=`diff $f1 $f2 | grep '^< #line' | wc -l`
diff1_newline=`diff $f1 $f2 | grep '^< $' | wc -l`
diff2=`diff $f1 $f2 | grep '^> ' | wc -l`
diff2_line=`diff $f1 $f2 | grep '^> #line' | wc -l`
# missing: only newlines and line directives
if [[ $diff1 -ne $((diff1_line + diff1_newline)) ]]
then
echo "FAIL1: $f1"
exit 1
fi
# added: only line directives
if [[ $diff2 -ne $diff2_line ]]
then
echo "FAIL2: $f1"
exit 1
fi
# the number of missing line directives
# equals to the number of added line directives
if [[ $diff1_line -ne $diff2_line ]]
then
echo "FAIL4: $f1"
exit 1
fi
done
echo "OK"
There are two configurations:
1. re2c:startlabel = <integer>;
2. re2c:startlabel = <string>;
The scope of these configurations is the scope of current re2c block.
Before this commit, given the following example:
/*!re2c
re2c:startlabel = "start";
[^]* {}
*/
re2c would generate the following code:
{
YYCTYPE yych;
goto yy0;
yy1:
start:
++YYCURSOR;
yy0:
if (YYLIMIT <= YYCURSOR) YYFILL(1);
yych = *YYCURSOR;
goto yy1;
{}
}
where "start:" falsely corresponds to "yy1:" rather to "yy0:".
(The important property of this example is that DFA has arrows
to initial state.) This commit fixes this behavior:
{
YYCTYPE yych;
start:
goto yy0;
yy1:
++YYCURSOR;
yy0:
if (YYLIMIT <= YYCURSOR) YYFILL(1);
yych = *YYCURSOR;
goto yy1;
{}
}
In '-f' mode, state dispatch generation can be triggered in two ways.
Both ways use 're2c::OutputFile::insert_state_goto', which generates
state dispatch only if it hasn't been already generated. The two ways
are:
1. Explicitly, using '/*!getstate:re2c*/'. In this case default state
must be state 0 because it's hardcoded in the invocation of
're2c::OutputFile::insert_state_goto' in 'src/parse/scanner_lex.re'.
2. Implicitly, in 're2c::DFA::emit'. In this case default state must
be state 0 because if 'prolog_label' is not 0, it means that it's
not the first time 're2c::DFA::emit' is called and state dispatch
has already been generated.
This commit makes it explicit that re2c always uses state 0.
Note:
Currently in '-f' mode re2c generates one global dispatch for
the whole file (the enumeration of yyFillLabel's is also global).
All re2c blocks share the same state dispatch, so in '-f' mode all
re2c blocks must reside in the same function and must be parts of
the same lexer (exception: in '-r' mode re2c generates one state
dispatch per use block).
This is clearly an ugly limitation: one is forced put disconnected
lexers in different files in '-f' mode.
Now re2c provides conditions as a way to express related blocks,
so if users used multiple blocks only for unrelated lexers, we
could safely limit the scope of state dispatch to a single block.
But conditions can conflict with other re2c features, they are
a bit broken and I'm pretty sure some users use multiple blocks
(e.g. I used to do it).
Thus we cannot just make '-f' generate state dispatch on per-block
basis (there're some other obstacles: it's not quite clear which
block '/*!getstate:re2c*/ directive is related to, etc.).
I leave the situation 'as is' until better times (when lexer-
parser loop is fixed and the whole code generation model is more
robust).
Initial state is different from all other states in that should
not advance YYCURSOR when it's entered first time (when DFA is
entered). But it should advance YYCURSOR like any normal state if
it's entered from any other part of DFA (other DFA states may well
lead to the inital state, e.g. '[^]*')
That's why inital state is split into two parts, each marked by a
separate label: normal state label points to the code that advances
YYCURSOR, and special label points right after that code (skips it).
It happens so that normal state label is equal to special initial
label plus one, but we shouldn't rely on that.
This is first step to postpone labelling states until all the
code is actually generated (in the form of some structure in
memory) and it's time to pretty-print it to file.
Labels shouldn't be mixed up with states (in particular, they
shouldn't be used as a state identifier).
This pass was used to gather statistics:
- which labels are used (to avoid 'unused label' warnings from
C/C++ compiler)
- in '-f' mode, how many times YYFILL is called (to generate
dispatch to the arrpopriate YYFILL call and resume lexing from
there)
This commit deals with second case: it makes counting YYFILL calls
independent of auxiliary pass. Counting relies on variable
're2c::last_fill_index', which is updated in 're2c::need' function
(note that it is crucial that 're2c::need' is always called prior
to generation of state dispatch).
Also fixed documentation for '-f'.
This is part of effort to reduce code generation to 'null device'
(the same code is generated twice only to gather statistics on
label usage). In order to reduce this evil pass, we should be able
to track label usage before code generation.
There're two major things to care about in this situation:
1. <*> rules must have the lowest priority: in order to guarantee
it, we re-iterate <*> regexps after all other rules have been
parsed and fix <*> regexps priority.
2. <*> regexps must be compiled to instrictions separately for each
condition: this is guaranteed by assigning them
're2c::RegExp::PRIVATE' attribute.
Note that 're2c::RuleOp::accept' member stores rule priority.
These priorities don't have to be consecutive, only the right order
must be maintained. Later on in 're2c::DFA::prepare' they result in
consecutive 'yyaccept' values.
Removed useless methods (most of them became useless after
're2c::Str' removal) and obsolete autoconf check for 'strndup'.
Removed 'Scanner::token' methods. They used
'Scanner::check_token_length', which was pretty useless:
1. checking for the lower should have always succeed, because
'Scanner::tok' is always set to buffer start in 'Scanner::fill'
and if the token was too long, it's start will be lost anyway.
2. checking for the upper bound could fail if re2c dev passed some
trash into it, but any normal function would do so and this is
no particular reason to have special runtime checks here.
Now substrings are constructed in lexer, where all the lengths
and bounds are easier to verify from lexing context (for re2c dev).
In most cases re2c accepts single-quoted or double-quoted strings
in regexp specifications, but if flex-like syntax if enabled, then
re2c accepts unquoted strings.
Before this commit, functions that parse strings into regexps
expected quoted strings, and we had to add quotes in case of
flex-like syntax. That was very inconvenient and in fact
unnecessary, since the first thing parsing functions do is get rid
of quotes.
Don't bother with named definitions in lexer, just pass them as
strings to parser. Parser will recognize named definitions, insert
them into symbol table and handle conflicts.
Use simple 'std::map' instead 're2c::Symbol' class (that hides
symbol table in class static member).
Use 'std::string' instead of 're2c::Str'. Due to bison limitations
we have to pass pointers to strings allocated on the heap and
carefully destroy them. The whole thing is quite error prone, so
maybe I'll make a small slab allocator for parser later on.
First, DFA is built ('re2c::DFA::DFA'), then it must be prepared
for code generation: some states must be split, backtracking points
must be marked, etc. ('re2c::DFA::prepare'), then finally code
can be generated ('re2c::DFA::genCode').
I haven't yet fully decided whether second stage (preparing) is
closer to DFA construction in general (and thus should be moved to
'src/dfa') or to code generation (and should be moved to 'src/codegen').
Since it deals a lot with bitmaps, second variant will suffice for
now. Perhaps later on I'll split preparation into general and
codegen-related parts.
're2c::BitMap' represents actual bitmaps used for code generation,
while 're2c::GoBitmap' (former 're2c::Bitmap') is a node type in
graph representation of the program right before code generation.
Names used in 're2c::Go' subsystem are really ugly and will be
renamed to something more sensible later on.
For some reason 're2c::Scanner' methods were defined in file
'src/codegen/code.cc'. Mpved them to 'src/parser/scanner.cc'
and renamed 'src/parser/scanner.re' to 'src/parser/scanner_lex.re'.
This is the first part of effort to reduce the total number of
interdependencies between different files.
"src/codegen/print.h" contains some pretty-printing functions
mostly used for code generation. There two other cases where.
these functions can also be useful:
1. debug: it's not yet well developed, just some messy chunks
of code that are commented out
2. error messages: character pretty-printing is not actually
very useful, since error messages mostly contain printable
characters (pieces of user-supplied input)
From the above I conclude that non-codegen files don't really need
"src/codegen/print.h". If this will change, I'll consider moving
this functions to src/util.
To enable mingw build, confugure with "--host i686-w64-mingw32".
Simple 'make' will then build re2c.exe executable for windows.
Note that 'make bootstrap' will not work: it will try to run
re2c.exe in order to recompile scanner and this will surely fail.
Testing with wine can be done using 'make wtests'.
If we configure with "--enable-docs", then docs get rebuild every
time make is executed, and we'll have to commit these insignificant
changes every time, which is bad.
Major changes:
- Moved all re2c source files to a separate directory. Top source
directory should only contain autotools source files and a few other
files like README.
- Enabled out-of-source builds (and wrote a simple script build.sh
that makes out-of-source build).
- Improved portable variant of <stdint.h> header (src/c99_stdint.h):
now it relies only on some few defines in configure-generated
src/config.h (instead of checking for MSVC version and relying
on MSVC-defined stuff). Implementation follows C99 standard closely.
- Removed all windows-related build stuff. It was no use keeping it:
it's been broken for a long time and I can't maintain it.
- Removed all RPM stuff: distro-maintainers use their own hacks anyway.
Makefile.am is definetely the wrong place to keep such things.
A separete script and .spec file is a better idea, but again, nobody
uses it.
- added make target 'bootstrap' to make mainteiners' job easier.
- Merged lessons and examples into one.
- Updated README and doc/index.html.
- Run tests in parallel by default.
Changes concerning particular files:
- configure.ac:
- removed autoconf version check: developers will need the right
version anyway (otherwize autoconf will reject to work);
users should use configure script provided by package
distribution
- removed GCC version (3 or above) check: it's pretty ancient
and I don't know which features are missing anyway
- removed some useless checks (which resulted in defines in
src/config,h used by no one)
- introduced some new checks (used by src/c99_stdint.h)
- followed the advices of autoupdate
Makefile.am:
- explicitly prefixed all file names with $(srcdir) or $(builddir)
- removed windows and RPM related rules and targets
- added target 'bootstrap'
- sorted out automake variables
Simplified 'generate_paths_cover' a bit: NULL-state case is not
really needed, it's artificial and should be embedded into the
only case that needs it.
Estimate the size of data to be generated rather than the number
of paths in DFA. Returns precise size or 'Skeleton::MAX_PATHS',
whatever is less. Estimate data size in both cases for all paths
or path cover.
Simplified traversal of outgoing arrows when extending prefixes.
Instead of adding outgoing arrow sets (which was erroneous anyway:
they should remain unmodified for another recursion entry), simply
wrap iterator and keep iterating until both ingoing and outgoing
arrows are counted.
More code cleanup: pre-initialize paths for final states and
default state, so that NULL can be used as a terminating condition
for path-generating function. Fixed memleaks.
Tried to simplify data generation a bit: construct skeleton states
so that there's no links to NULL state. Default state is
represented with a state that has no rule and zero outgoing
arrows. Final states are represented with a state that has a rule,
but zero outgoing labels.
Reduced the amount of generated data (no exponential growth now).
Generate enough paths to cover all DFA transition (well, not
exactly all: only upper and lower bounds of a range). Respect
cycles (loop once).
Use a separate DFA-like structure for DFA skeleton. This allows
to optimize skeleton states for fast traversal (group transitions
by destination state) and avoid messing with the original DFA.
Added 'count' method to estimate the the amount of data that will
be generated (it worked too slow for the origianl DFA with
ungrouped transitions).
Output input data to a separate file (otherwize we'll have to
keep all generated data in memory, cause output has a complex
structure and cannot be written to file until it's fully
generated)
This reduces memory usage significantly (so that there remain no
memory consumption problems with "--skeleton" switch). However
on some files re2c generated too much data, e.g. case-insensitive
strings:
/*!re2c
'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' {}
*/
Exponential growth is a bad thing; must deal with it somehow.
Time grows exponentially as well, of course.
Generate check for input position in final states: first check
that input position equals to the expected one; second advance
input position to the beginning of next DFA path (this may be
necessary when the generated lexer rollbacks: it should resume
with a new DFA path rather than some default characters remaining
after rollback).
Track exact number of characters to be consumed on each input
string. Note that it's not equal to string length, since many
strings end up with characters from default transitions: on such
strings lexer must rollback to the latest backtracking point (that
is, the latest accepting state).
Moved input-string-generation-from-DFA algorithm earlier, to the
point when DFA is barely constructed and contains only 'Match'
states and starting state (no saving, accepting or split states).
This simplifies string generation: if destination state is NULL,
it is either next to final state (if all spans lead to it), or
default state (if some spans lead to other states).
A naive attempt to generate input strings from DFA. The problem is,
if the number of spans equals 1, it's hard to determine whether
it's some kind of a 'transit' state or a normal state with just one
span. All states have spans, but do they use them? The situation is
further complicated with 'readCh' which makes it hard to trace how
actions influence input operations.
Generate prolog and epilog in the form of a for-loop. The body
of the loop is the hard-coded DFA. The code in DFA final states
is substituted with "continue" statements.
When making regexp alternative, only 'RegExp::PRIVATE' attribute
should be propagated. Propagating 'RegExp::SHARED' attribute is
a mistake, as can be observed from the following example:
name = "smth1";
"smth2" | name | "smth3" { ... }
name { ... }
Here, 'name' must have 'RegExp::PRIVATE' attribute, but it gets
broken after alternation. See #bug57 or test 'bug57.re' for full
working example.
This allows to move 'Go' initialization loop to 'DFA::prepare'
and thus avoid ugly check if it is already initialized (it can
happen in '-r' mode when the same DFA is used multiple times).
Now that we store 'State *' pointers instead of labels in
'CpgotoTable', relabelling won't affect the generated code.
(first part of effort to reduce codegen to null device)
- Properly destruct 'Go' graph (46 test failing with '--valgrind'
because of early exiting on errors)
- First, re2c builds a complex structure where it stores
all control flow codegen decisions: nested ifs or switches,
bitmaps or computed gotos, etc.
- Second, this structure is traversed and code is generated.
This differentiation is necessary to compute some statistics
(e.g. used labels) in advance, before code generation.
All tests pass.
The previous condition made more sense: it was clear that the
author intended to consider some frequent corner cases.
But the condition was very tangled and yet too heuristic,
so I substituted it with a meaningless, but simple one.
I'm planning to simplify it even more later on.
- draw a single arrow for all transitions between two given states
- label all arrows with corresponding character ranges in square
brackests (no "default" label, single characters also appear in
square brackets)
- .dot output became much smaller, thus pictures are drawn faster
and generally look better: e.g. it takes ~10x less time to draw
PHP lexer and the resulting graph is shaped better.
- Use 'open' function instead of checking return status
(one may forget to check return status, but if one forgets to
open file, the error will be obvious)
- Introduced separate file type for header.
Header is much simpler than output, it doesn't need delayed
code fragments and can be generated in destructor.
I was unsure if delayed generation was also needed for genCondGoto
and genCondTable; so I kept those enum members as a reminder. Now
I know that all conditions are known by the moment re2c block is
parsed and code generation starts.
Second pass was used because some information (which influences
early parts of the generated code, e.g enum with condition names
or YYMAXFILL definition) becomes available only at the end of first
pass.
I isolate all (I hope so) these things and generate stubs for them,
which are filled later. I restructured output as follows: the whole
output consists of source and header, each of them is a list of
blocks (corresponding to re2c blocks in source file), each block
is a list of code fragments (which can be either regular strings
with code or stubs that will be filled later).
The actual meaning of this primitive is to check if
there's not enough characters left in the input stream,
e.g. "(YYLIMIT - YYCURSOR) < n" or whatever else.
This implied modifying runtests.sh, as it couldn't handle
test names of the form "basename.--long-switch.re":
it inserted '-' in front of all switches.
It's impossible to correctly implement this switch in general,
since not all istreams support seek (simplest counterexample:
std::cin).
Even for simple istreams like std::istringstream, proper
error handling requires adding something like "YYERROR ()".
- moved enum and pretty-printing functions to a class
- renamed files 'input.{h,cc}' to 'input_api.{h,cc}'
- for "--input istream": moved input position increment to 'stmt_restorectx'
- main.cc: removed useless include
Double-escape special characters for dot.
Example:
17 -> 18 [label="\n"]
results in an "unlabeled" arrow in the rendered graph, but
17 -> 18 [label="\\n"]
is ok.
When one builds 'AltOp' from two 'RegExp's, one sometimes has to
break these 'RegExp's in pieces in order to merge their common prefix.
In such cases, if one of the original 'RegExp's has 'ins_access'
set to 'PRIVATE', it is lost (defaults to 'SHARED') after alternation.
This commit fixes Gentoo bug https://bugs.gentoo.org/show_bug.cgi?id=518904.
Sample error:
parser.y: In function `void re2c::parse(re2c::Scanner&, std::ostream&, std::ostream*)':
parser.y:564: error: `yyparse' undeclared (first use this function)
141.101.31.80 - - [23/Mar/2015:22:19:46 -0700] "GET /logs/access_130930.log HTTP/1.1" 404 73 "http://blondebitch.soup.io/" "Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
141.101.31.80 - - [23/Mar/2015:22:19:46 -0700] "GET /access_130930.log HTTP/1.1" 404 73 "http://blondebitch.soup.io/" "Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
141.101.31.80 - - [23/Mar/2015:22:19:46 -0700] "GET /logs/access_130930.log HTTP/1.1" 404 89 "http://blondebitch.soup.io/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0)"
141.101.31.80 - - [23/Mar/2015:22:19:47 -0700] "GET /access_130930.log HTTP/1.1" 404 73 "http://blondebitch.soup.io/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0)"
183.89.175.19 - - [23/Mar/2015:23:04:10 -0700] "GET /logs/access.log HTTP/1.1" 200 12589 "http://redlug.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"
72.81.177.197 - - [23/Mar/2015:23:06:15 -0700] "GET /socview/SV0302TroublesReview.htm HTTP/1.1" 200 3182 "https://www.google.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B466 Safari/600.1.4"
72.81.177.197 - - [23/Mar/2015:23:06:18 -0700] "GET /cwi.css HTTP/1.1" 404 89 "http://redlug.com/socview/SV0302TroublesReview.htm" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B466 Safari/600.1.4"
HEAD file://cwidoaajjb.pqzeztqtg.iagnuhmdq.npliaihmqn.yuptnwn/hlhkojvjvzmsgccwcpwhh/qfliekillmmdz/aypkwffmdojvpdodmrouzxuupqp/gjqnmbk/mnbtpocjbe/ttvqsflbsqirgahmutddqucg/nkycantossaikwulmgemuqttwx/foqeceoyiowyeftxpr/syjcvolygaguylfkx/gwgx?7B9Jq5CNzjzSFzJl7cCL4oUX HTTP/6.2
Loaded 100 of 5046 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Ulya Trofimovich