This reverts commit 5cea17c8fe.
Reason for revert: enough time has passed and most distributions have
updated to re2c version 4.0 or higher.
This fixes bug #507 "Update C/C++ examples".
lib/regexec_dfa.cc (regexec_dfa(), regparse_dfa()) and
lib/regexec_dfa_multipass.cc (forward_pass(), shared by
regexec_dfa_multipass<>, regparse_dfa_multipass<>, and
regtstring_dfa_multipass<>) read the next input character as
`*p++` into a plain `int32_t`/`chr`, then use it directly to index
`preg->char2class[]`. On platforms where `char` is signed (the
default for x86/x64), any input byte >= 0x80 sign-extends to a
negative value, so `char2class[c]` reads before the start of the
257-element `size_t` array allocated in regcomp() -- an
out-of-bounds heap read (observed to segfault in a minimal
regcomp()/regexec() reproduction with a high-byte input string).
Fix by reading the byte through `static_cast<uint8_t>(...)` before
using it as an index, matching the pattern already used for the
same purpose in the NFA matchers (regexec_nfa_leftmost.cc,
regexec_nfa_leftmost_trie.cc, regexec_nfa_posix.cc,
regexec_nfa_posix_trie.cc). The `== 0` end-of-input check is
unaffected since 0 is unchanged by the cast.
Verified: rebuilt libre2c (RE2C_BUILD_LIBS=ON) and confirmed
test_libre2c passes before and after the change; a targeted
regcomp()/regexec() check over all 254 non-newline byte values
0x01-0xFF segfaults on the pre-fix code and passes cleanly after
the fix.
They mostly duplicate the "ignore" policy (except for the categories that
include surrogates). These policies are also covered by the Unicode block tests.
Tests have changed due to the changes in the Unicode standard - the old test
were generated using Haskell "Charset" library, which supports unknown version
of Blocks.txt (none of the existing version matches exactly, but Unicode 4.0 is
somewhat close). There are also other changes in the tests, mostly
simplification and cleanup of the generated code.
The new generator is based on the official Blocks.txt file from the unicode.org
website. The generated result has changed; it's unclear why some of the blocks
are missing in the Haskell "Charset" library - most likely the library was not
updated for Unicode 15.1.0 (although the categories' definitions correspond to
Unicode 15.1.0).
This partially addresses bug #235 "Unicode character classes".
Don't use it in auto generated tests, because that creates a problem with
include paths: they are relative to the test directory, which makes the
error messages different based on the build directory (so the tests fail).
Use the old generator based on haskell "Charset" library in order to avoid
changes in the generated code - the library has some discrepancies with
https://unicode.org/Public/15.1.0/ucd/Blocks.txt.
Update test results - the changes have been verified with
test/__run_unicode_tests.sh.
This partially addresses bug #235 "Unicode character classes".
The new generator is based on the official UnicodeData.txt file from the
unicode.org website. After some experimentation, I found that the the
Unicode standard supported by the previously used Haskell "Charset" library
is 15.1.0, so this is the version used in the new generator. The generated
include file unicode_categories.re is identical to the old one, which
verifies correctness of the new generator.
This partially addresses bug #235 "Unicode character classes".
Previously edge cases like 0xFF were rendered as \u0ff rather than \xff.
The difference in the generated files was verified with:
`git diff --word-diff=color --word-diff-regex='-?\\[a-zA-Z0-9]+'`
Previously even if lower and upper characters were the same, the range
would still be rendered with a dash rather than a single character.
This commit removes useless duplication.
The difference in the generated files was verified with:
`git diff --word-diff=color --word-diff-regex='-?\\[a-zA-Z0-9]+'`
Regenerate files using haskell script test/encodings/unicode_groups.hs
that is based on Data.Charset library (charser-0.3.12 package). The
script generates tests in test/unicode directory inplace and include
file unicode_categories.re.inc that needs to be copied to
include/unicode_categories.re.
Update test results. As there are substantial changes in the standard,
updates are also nontrivial. They have been verified with
test/__run_unicode_tests.sh.
The example for C was werified with examples/c/__run_all.sh scripts.
Null pointer dereference happened because the root TNFA state was null:
there were no rules for a block, but determinization still happened.
In this case re2c should emit an error and never even attempt
determinization. It was properly handled for blocks with start
conditions, but not for normal blocks.
This addresses #571 "[Bug] Segmentation Fault (NULL Dereference) in
re2c::closure_leftmost_dfs during determinization".
Note: for Haskell and monadic lexers in general I considered adding `yyend`
variable in order to allow monadic YYEND (similar to the existing `yyfill`
variable). However, this is not yet useful even for Haskell, because it's always
easy to store the end in a record field and define YYEND as a simple non-monadic
comparison expression. It won't be late to add `yyend` variable later in a
backwards compatible way (if the variable is empty, generate code the old way,
otherwise do it the new way), therefore I decided to postpone `yyend`addition
until such time when it's needed.
This is necessary in order to accommodate negative tags, which are added at the
end of the other alternative (the one that doesn't have tags). During TNFA
construction $ is moved past negative tags and placed immediately before the
final TNFA state. This ensures that there are no tags on the $-transition and we
don't have to append tag operations to incoming transitions when getting rid of
the $-transitions in TDFA.
Do it right after TDFA construction. This is only strictly necessary for
self-shadowed rules, but it's better to handle all end-of-input rules
together (especially since the previous analysis was still a hack in
the reversed-TDFA structure).
Generate YYEND before YYFILL. This order is makes sense because YYEND should be
faster than YYFILL. In fact, it is the only possible order, as YYFILL may not
return if it's impossible to supply enough input characters.
Don't generate YYEND on transition in the case of "sentinel with bounds checks"
method; instead, generate it in state. This doesn't make any difference on the
tests because YYEND is only needed in the "end" states that have a single
transition.
Prevously end-of-input symbol $ was only allowed with "sentinel with
bounds check" method (when `re2c:eof` configuration is set to a
nonnegative value). This commit removes the above restriction and
enables YYEND checks for all end-of-input handling methods. Code
generation is not quite right yet, as in some of the cases YYEND checks
are emitted after YYFILL, while they should go before it.
Internally the main challengege was adding $-transitions in TDFA for all
end-of-input handling methods, and to hide them when necessary depending
on the method. E.g. -Wunreachable-rules analysis must ignore them except
for the $-final states, unless "sentinel with bounds checks" method is
used.
For the moment, YYEND is only supported for the "bounds checks with sentinel"
end-of-input handling method (the case of `re2c:eof` configuration). In this
case states that have transitions on symbos do not need YYEND check, as they
have the sentinel transition with YYLESSTHAN / YYFILL pair.
Namely, use `YYLIMIT <= YYCURSOR` rather than `YYLIMIT == YYCURSOR`.
The goal is to bridge the gap between YYEND and YYLESSTHAN in the case
of `re2c:eof`. $-final states that have outgoing transitions defer the
end of input check until matching `re2c:eof` symbol range on transition
(and then use YYLESSTHAN in the same way as non-$-final state would do).
But $-final states that are terminal (have no outgoing transitions on
symbols) use YYEND.
We can only bridge this gap in simple / record (not generic) API, and
only if YYFILL is disabled. Otherwise it has to be YYEND and YYLESSTHAN
as macros, and the distinction remains.
This commit does not compromise correctness of the generated code, as
YYCURSOR cannot read past YYLIMIT in a correct lexer, so if it was
correct before this commit it will be as correct after it.
Also, add the corresponding syntax file configuration `code:yyend` and
syntax file conditional `.yyfill.enable`.
The new API primitive will be used in the implementation of generalized
$ (the end of input symbol). If YYFILL is disabled, and the API used is
either simple or record (not generic API), then YYEND defaults to a
comparison `YYCURSOR == YYLIMIT`, otherwise it remains a macro to be
defined by the user.
We don't use a combination of YYLESSTHAN and YYFILL instead of the new
primitive for a number of reasons:
- Using YYEND allows us to unify all codegen cases instead of having
many different implementations based on YYFILL, storable state,
generic API, etc.
- Involving YYFILL would add runtime overhead: a possible function
call instead of a simple inplace check, useless reiteration via
YYFILL label (which is currently not even generated in these states)
This change breaks backwards compatibility. This is intentional, following
a period when -Wdeprecate-eof-rule has been a warning by default, now the
default is an error (-Werror-deprecate-eof-rule).
There are code generation changes: end of input rule is now handled as a
normal rue with positional precedence / inheritance order (it is no longer
a special rule like the default rule *). The changes in the code can be
observed with -Wno-error-deprecate-eof-rule (applied to the tests).
This reverts commit b0d740994c.
Reason for revert: don't keep signed YYCTYPE examples for languages
that don't allow the user to do the incorrect thing (Go, Swift forbid
comparison of signed 8-bit type to literals that overflow it, like 0xFF;
Python, JS, OCaml don't even have a signed 8-bit type).
Use `int` instead of `byte` for `YYCTYPE`, as `byte` is signed and therefore
incapable of holding the full range of literals generated by re2c (e.g. 0xFF).
use `Byte.toUnsignedInt` in YYPEEK in order to convert correctly from signed
bytes to `yych` of type `int`. Add a test that demonstrates the need for this.
Using `char` is fine, as it is 2-byte in Java and thus capable of holding
ASCII and UTF-8 range used in the examples.
This addresses bug #561 "Incorrectness with ASCII signedness and range
(in C/C++)".
Only one example required fixing `Int` -> `Word`, others use `Word8`.
This addresses bug #561 "Incorrectness with ASCII signedness and range
(in C/C++)".
The generated code compiles only with `#[allow(overflowing_literals)]`,
and it requires `re2c:yych:conversion = 1`.
There's no need to fix examples, as they use `u8` and `u32`.
This addresses bug #561 "Incorrectness with ASCII signedness and range
(in C/C++)".
The generated code for the test does not compile - Go rejects it:
signed_yyctype.go:34:9: cannot use yyinput[yycursor] (value of type byte)
as int8 value in assignment
signed_yyctype.go:35:14: 0xFF (untyped int constant 255) overflows int8
There's no need to fix examples, as they use `byte` (which is unsigned
8-bit type) or `uint32`.
This addresses bug #561 "Incorrectness with ASCII signedness and range
(in C/C++)".
There's no need to fix examples, as they use `char` (which is unsigned
8-bit type according to https://dlang.org/spec/type.html#basic-data-types)
or `uint8_t`/`uint32_t`.
This addresses bug #561 "Incorrectness with ASCII signedness and range
(in C/C++)".
This is needed as a last resort to allow releasing re2c on a system where some
of the language toolchains are broken (as it becomes excessively hard to keep
them all working and to keep the examples up to date for languages that don't
care about backwards compatibility, such as Zig).
This addresses bug #564 "End of input ($) rule deemed invalid when Posix
captures enabled". It is needed t allow implicit capture tags after $ in a
regexp, which was inadvertently forbidden in release 4.3.
It will also be necessary in order to accommodate negative tags added at the
end of alternatives that don't have tags.
During TNFA construction $ is moved past negative tags and placed immediately
before the final TNFA state. This ensures that there are no tags on the
$-transition and we don't have to append tag operations to incoming transitions
when getting rid of the $-transitions in TDFA.
This addresses bug #560 'Illformed C/C++ code due to missing "final"
conditions in generated enumeration'.
After much discussion (see the bug) it was decided that aborting in the initial
state dispatch for conditions without rules doesn't play well with the existing
configuration `re2c:cond:abort`. Some language backends (notably, C/C++ backend)
have it disabled by default, so the general-case default (no abort) would be at
odds with the special-case default for empty conditionss (abort).
This commit takes a different approach: always raise an error on conditions
without rules, but before doing that check if there are any `<*>` rules that
can be merged to them.
This reverts commit 34f6e5269f.
Reason for revert: to make sure we don't break existing code, e.g. if
the numeric constants are incompatible with the user-defined enum type.
This check should not compare condition numbers in goto/label mode,
as conditions get renumbered anyway. Previously it used to work because
all numbers were set to zero in goto/label mode, but it was broken by
a recent commit 8a4e4ebd54.
This addresses bug #560 'Illformed C/C++ code due to missing "final"
conditions in generated enumeration'.
Check that all conditions referenced by `=>` and `:=>` exist (have rules).
In the case of `:=>`, raise an error on non-existent conditions (because
the autogenerated semantic action would go to a nonexistent condition start
label). In the case of `=>` add the nonexistent condition to condition enum
(the user may want to set condition variable and return from the lexer to
handle it in user code), but make the initial dispatch abort on it (in case
it was a typo).
This addresses bug #560 'Illformed C/C++ code due to missing "final"
conditions in generated enumeration'.
Historically in goto/label mode conditions weren't assigned explicit
numeric identifiers - the numbers were assigned implicitly based on
position in the condition enumeration. This affected language backends
for C/C++, D, Go, V (those supporting goto/label model).
In loop/switch and rec/func modes condition numbers correspond to TDFA
state indices, so they are not consecutive and require explicit
initializers.
This commit makes the enumeration always have explcit initializers
regardless of the code model. The generated initializers are backwards
compatible (identical to the implicitly assigned ones).
Historically in goto/label mode conditions aren't assigned explicit
numeric identifiers - the numbers are assigned implicitly based on
position in the condition enumeration. However, this correlation is
erroneous: there's no guarantee that the languages that support `goto`
also have enums with implicit numbers.
For now, keep the erroneous correlation (to avoid changing the generated
code), but move it from re2c code to syntax file definitions.
Previously $ was a special end-of-input rule with precedence and
inheritance order similar to the special default rule *.
Now $ can be used in a regexp as part of a normal rule with normal
position-based precedence and inheritance order. However, this change would
break backwards compatibility in some cases, so for now there's a workaround
that enforces the old order and gives a warning. In the future the warning
will be turned to error, and the old order will no longer be enforced.
New options that control this behavior:
-Wdeprecated-eof-rule (turn on warning, on by default)
-Wno-deprecated-eof-rule (turn off warning)
-Werror-deprecated-eof-rule (turn on error, off by default for now)
-Wno-error-deprecated-eof-rule (turn off error)
The case when $ is in the middle of a regexp (like `[a] $ [b]`) are forbidden
and raise an error. The case when $ is at the end but not standalone (like
`[a] $ | [b]`) are forbidden at the moment, as they are not handled properly in
codegen, but they will be allowed later. For now, only the old-style standalone
rule `$` is handled properly.
This is the first part of the work needed for issue
https://github.com/skvadrik/re2c/issues/525.
If end-of-input rule `$` is used, re2c takes a different approach and
uses a different set of subroutines to find unreachable rules.
Previously these lacked "fallthough" analysis that is normally performed
without the `$` rule. As a result, some code was generated incorrectly
(as can be seen on the updated tests and the newly added test).
The CMAKE_POSITION_INDEPENDENT_CODE variable enables PIC when building
objects, and the CheckPIESupported module provides a
check_pie_supported() command, which enables position independent code
for executables (PIE) at the link step. Without this module, for
example, building at the link step in some cases might fail. For
example, on Haiku.
Avoid popping/pushing on-stack closure items and patch them in place if
possible. Note that using `emplace` instead of `push_back` method on
`std::vector` instead of this change is not nearly as much helpful: it adds
only a ~6% speedup compared to the ~42% speedup caused by this change.
It only affects leftmost greedy closure construction (which is the default).
This addresses bug https://github.com/skvadrik/re2c/discussions/544
"Unicode characters in UTF8 and generation performance".
The 42% speedup was measured on examples/c/unicode_identifier.re modified as
suggested in the comment
https://github.com/skvadrik/re2c/discussions/544#discussioncomment-13011740.
Make closure items smaller - don't store transition tags and tag versions
in them, as they can be retrieved from the origin kernel using origin TNFA
state for each closure item. Making the items smaller reduces the amount
of copying during closure construction.
Also, avoid extra copy between the main and auxiliary closure vectors;
construct the new TDFA state kernel right from the main vector. This required
a lot of changes in the code, as the kernel stores information as a struct of
arrays (this allows fast comparison and hashing of only the relevant parts of
the kernel), while the closure is an array of structs (this is more convenient
for closure construction and sorting).
Also, make the conditional in the kenel hashing function compile-time rather
than run-time.
There are no algorithmic changes; only the way information is organized and
propagated.
This addresses bug https://github.com/skvadrik/re2c/discussions/544
"Unicode characters in UTF8 and generation performance".
The 10% speedup was measured on examples/c/unicode_identifier.re modified as
suggested in the comment
https://github.com/skvadrik/re2c/discussions/544#discussioncomment-13011740.
Replace `code:cgoto_target` (which was the target expression in the computed
`goto` statement) with `code:cgoto` (which is the full statement).
Rename `code:cgoto_init` to `code:cgoto_data`.
Add `code:yytarget_filter` (similar to existing `code:yybm_filter`).
Local varaible of type `const char*` was passed as `const std::string&`
constructor parameter (so a temporary `std::string` had to be created)
and stored in a class. Fixed by storing the `const char*` pointer directly.
Previously we had `code:yytarget_goto`, `code:yyctable_goto` for computed
`goto` target expression, and `code:yytarget_elem`, `code:yyctable_elem`
for table initializers. The defintions for `yytarget` and `yyctable` were
almost identical, and the decision to keep them separate was taken only
so that each set of code templates could specify a different underlying
type for table elements. It was impossible to pass the type using
`code:type_yyctarget`, `code:type_yyctable` because these configurations
included type qualifiers like `static const`.
For the full discussion see this thread:
https://github.com/skvadrik/re2c/pull/537#issuecomment-2693462138
Now that we've got rid of type qualifiers in commit
a94d39b112, it has become possible to merge
code templates for `yytarget` and `yyctable` and avoid the duplication.
Code templates `code:array_local` and `code:array_global` can now use local
conditional `.const`, which is true if the array is immutable.
This also has the side-effect that type qualifiers such as `static const` in
C/C++ have been removed from type definitions `code:type_yybm`,
`code:type_yytarget` and `code:type_yyctable` (where they did not belong in
the first place).
Also, rename `yybm` to `YYBM` in Rust by default (this is an inner variable,
so changing default definition should not break backwards compatibility).
Rust and Java have been affected by this change, as their bitmap definitions
have been constified (which is reflected by the updated tests). Other language
backends either have used constant bitmaps already, or they don't have a notion
of immutable arrays.
While at it, fix description of `re2c:yyfn:sep` in the docs.
Regenerate bootstrap manpages and help (incorporates some earlier updates
for Java manpage that were omitted by mistake in commit
47afb07f90 that updated manpages for other
languages).
This is useful for --recursive-functions code model, as it's necessary to use
different syntax for function calls with an additional `try` keyword before the
call expression (which would be impossible to do before this change).
The test/codegen/zig/recursive_functions/basic.re test has been updated to
demonstrate the new functionality.
Adds support for yyfn:throw to the Swift syntax, allowing the user to throw
types that conform to the Error protocol within functions generated by the
recursive-functions code model. Non-empty settings of the `re2c:yyfn:throw`
property are appended to generated function definitions between parameters
and the return type, and generated function calls are prefixed with "try ".
The 'test/codegen/swift/recursive_functions/advanced.re' test has been updated
to test the new functionality as well as demonstrate practical use of throwing
functions in the recursive-functions model for handling runtime errors.
This reverts commit 85d9384b2f.
Reason for revert: the main motivation for this change was to support Swift
`throw` annotations. However, it turns out that attributes are unfit for the
purpose, since functions that throw also require slightly different call
expressions (they need to be prefixed with `try`). Although attributes may be
useful, let's remove them until we find a real-world use case.
For details, see the full conversation:
https://github.com/skvadrik/re2c/pull/538#discussion_r2008079095
code:yytarget_elem specifies the how the table elements for computed
goto tables are generated. If cgoto:relative = 1, then the table
entries contain an offset from a calculated base address.
code:yytarget_goto specifies the computed goto tables are generated.
code:yyctable_elem and code:yyctable_goto are used for the
corresponding computed gotos for conditions.
Bootstrap code has been regenerated.
Previously the code-generation did similar table generation across
the normal states, accept states and condtional variables.
This refactor introduces some utility functions that provides the same
logic across the different tables.
Instead of using absolute pointers are used in the cgoto tables,
we generate relative offsets from the earliest label offset
within the code. This offset type is hardcoded as `int' currently.
This way we can have more reliable measurements for the future benchmarks that
read input from a string rather than a file, and so run much faster and are
more affected by measurement errors.
They are interesting from the academic standoint (to compare DSST vs. TDFA),
but Kleenex is not flexible enough: it assumes fixed program template. The
generated code cannot be adapted to google-benchmark, which it important to
get precise time measurements.
Kleenex benchmarks results are available in the paper by Borsotti and
Trafimovich "A Closer Look at TDFA" (https://arxiv.org/abs/2206.01398).
For anyone interested, see the paper by Borsotti and Trafimovich
"A Closer Look at TDFA" (https://arxiv.org/abs/2206.01398). It has benchmark
results that cover the removed algorithms.
Some languages have special escape `\"` in character literals, so re2c should
recognize it as well.
This addresses bug #529 "[Rust] Escaped double-quotes in character literal
results in EOF error".
!pre_rule action is emitted immediately *before* any semantic action in the
current block / condition (it can be useful to have some common initialization
code). !post_rule action is the same, except that it is emitted *after* every
semantic action (it can be useful e.g. to emit unreachability checks).
This addresses bug #521 "Add special entry/exit rules".
This way it is possible to add new named special actions without having
to worry about exhausting all special symbols. Also, if we add mid-rule
actions in the future, this syntax can be used for them as well.
Remove incorrect assignment of default state to default rule state; it was
overwritten later, so it had no effect at all. Rule state should be kept
separate, as default state may be used for different things (e.g. it may be
used as the accept state that has `yyaccept` dispatch).
No changes in the generated code.
Previously initial state was marked using special initial action, which was set
only in rec/func mode. Now we explicitly store a pointer to initial state and
set it for all code models.
It's unclear where the previous condition came from; the intention seems to be
to exclude final states that have no outgoing transitions on symbols. With the
end-of-input rule `$` even if all characters fall into the same range as in
`[^]`, a separate range is split for the sentinel symbol, so the previous
condition could not capture any match states.
This is only needed in loop/switch and rec/func modes, as goto/label mode uses
special YYFILL labels, not the state labels. And most of the states were already
marked as used due to other transitions going into them. So the bug was not
revealed until the recent addition of the entry rule.
This addresses bug #521 "Add special entry/exit rules".
Previously we only checked that tags are parwise equal between two mapped
transitions, not between all transitions going to the bitmap state. This was a
regression caused by optimization 98f2a41393.
This fixes#523 "Possible regression with bit-vector optimizations between 3.x
and 4.x".
When all rules were inherited from other blocks, re2c failed to find correct
rule location and passed an invalid location, which caused a SIGSEGV in the
error reporting function.
This fixes bug #522 "Segfault when !use is present in a block with conditions".
In goto/label mode, if there is a loop via initial state, YYSKIP must be
bypassed when entering the DFA (visiting initial state for the first time).
Other modes generate YYSKIP on transitions, so they were not affected.
The bug was found by skeleton tests.
This addresses bug #521 "Add special entry/exit rules".
Also, rename "setup" to "exit action" to match the added "entry action"
(and also because it makes more sense).
This addresses bug #521 "Add special entry/exit rules".
Drop some not very useful debug checks.
Previously re2c failed with an error in the case when end-of-input rule `$` was
enabled and the special zero condition `<>` was used. Now it no longer fails and
generates code withot end-of-input check for the zero condition (which is
handled in a special way by the code, bypassing some of the transformations that
rely on the presence of end-of-input rule).
With captvars (either leftmost or POSIX ones) we need to generate names for the
variables that correspond to capturing parentheses. The difference between
leftmost and POSIX cases is that POSIX disambiguation algorithm is more complex
and it requres additional imlicit tags inserted in the regular expression.
These so-called "fictive" tags don't have valid submatch group indices, so
trying to use them caused integer overflow and a very long loop.
This is a fix for issue #519 "re2c hangs if posix-captvars is used".
We now have many options for captures, so we cannot assume that the
--posix-captures option is the one causing conflicts.
This partially addresses issue #518.
The generated code already handles the end of input, so it should use
`unsafe_get` to avoid redundant bounds checks. If the user wants to use `get`,
they can always do so by overriding YYPEEK.
Update all examples and remove '\x00' character at the end of string literals,
as they are already NULL-terminated.
Previously C escape sequences were used for all languages, which resulted in
incorrect escapes generated for some languages, e.g. `\f` in OCaml is not a
valid escape, while in C it is.
This commit adds syntax configuration `special_escapes` to all syntax files.
Its value is a string of all special characters that require escaping.
This is a fix for #516 "re2ocaml can generate invalid escape sequences".
0x20 (space ' ') is printable, but others like '\f' are not. And 0x20 is already
covered by the `is_print` part of the check.
This patch does not change existing behavior, as the code handled all
non-printable space characters as escape sequences before the check, so they
were printed correctly.
Update pre-generated benchmark files (trivial line directive changes).
Disable regeneration of Kleenex files, as Kleenex is not actively maintained and
fails to build with the recent cabal and Haskell ecosystem.
2nd attempt to fix issue #515 "version 4.0.1 re2rust default generate C code".
This time, instead of trying to split the object library, the code that needs
the split has been moved out of the problematic library.
Also there are some small improvements in the way default syntax strings are
declared (added `static constexpr`) and linked (moved them to header files).
Generation script has been fixed to generate correct string names.
This reverts commit 9cfb87edaf.
Reason for revert: it's no good trying to build multiple object libraries for
different languages in parallel, as they all try to copy files to the same
bootstrap directory, causing conflicts and build errors. Avoiding such conflicts
was the reason the object library got added in the first place (so that
autogenerated sources could be built once before all other sources).
Previously target compile definitions weren't properly added to the
autogenerated sources, as they were built into a separate object library
(to ensure that they are built before the rest of the sources). Now the
object library is built separately for each language with proper options.
This addresses issue #515 "version 4.0.1 re2rust default generate C code".
Some of the examples use C++ data structures like `std::ifstream`, `std::vector`
etc. Leave them as is for now.
Update docs.
This addresses bug #484 "Example is not C".
Keep examples compatible with older re2c versions until most of the world has
updated to re2c-4.0 or later. When it happens, this commit should be reverted.
This is to fix building with bmake on FreeBSD.
Replace all pattern rules with POSIX-compatible suffix rules. As the latter are
much less flexible than pattern rules, this required creating copies or symlinks
with appropriate names for source files that don't fit into the naming scheme
implied by suffix rules.
Also, dependencies on custom build scripts had to be dropped, as suffix rules
don't allow additional prerequisites.
This warns on `static const std::map<...>`.
re2c is single-threaded, so there's no risk of race conditions when one
thread is terminating and the other one tries to access the object.
Mostly a cospetic change. I hope to get to the state when `re2c` can be
built with `-flto -Werror=odr`. Right now it fails as:
src/parse/parser.cc:121:6: error: type 'yysymbol_kind_t' violates the C++ One Definition Rule [-Werror=odr]
This change does not fix it, but at least it clearly solits the exposed
API namespace:
https://www.gnu.org/software/bison/manual/html_node/Multiple-Parsers.html
These parts are used inconsistently in configuration names, e.g. we have
`re2c:tags:expression` which has to be scoped under `tags:` but it's also a
definition, so it should be scoped under `define:` (and other examples like
this). In fact, most of the string configurations are defines of some sort.
Also, change `conf:` prefix in syntax files to `re2c:`, as in
configurations. From now on, "c" in "re2c" stands not for C/C++ language,
but rather "code" or "compiler", so it's good for all languages.
Also, use prefix `.` (dot) for conditionals in syntax files to make it
easy to distinguish them from variables in code templates.
Also, fix warnings and error messages to use backtick quotes consistently.
Also, fix some comments in the code that still used the word "directive" for
special-purpose blocks.
Automake: generalize generation rules for docs (make split_man.py infer
language from its second argument, as POSIX make does not seem to have a way
to pass languge name to the rule). Explicitly list all example files (provide
a `find` command to generate the list in a comment).
CMake: Add forgotten install files for re2java and re2js. Use GLOB_RECURSIVE
with CONFIGURE_DEPENDS to add examples to the list of source file.
Also, update description of submatch extraction and change captures example:
use the one without POSIX-style array `yypmatch`, as not all languages support
it (Haskell doesn't, as it has no mutable arrays in the standard library).
Remove syntax configurations `code:bit_and`, `implicit_bool_conversion`
as they are no longer needed. Strictly speaking, `code:bit_and` was also
used in generation of computed goto, but it's fine to hardcode `&` for
now, as computed goto are only implemented in C/C+. Later they will be
reimplemented using a syntax configuration similar to the bitmap ones.
This unifies bitmap transition with other transitions that are constructed
during the analyze phase and removes the need for special handling during
the generate phase.
This commit splits table initialization (populating it with values) and code
generation for the initialized table. The former is moved to analyze phase, and
the latter remains in the generate phase.
Bitmap values are needed when generating preconditions for bitmap transitions.
Currently this is done during generate phase, but there is a plan to move it to
analyse phase so that bitmap transitions could be generated using the same code
primitives as other transitions (linear IF).
Some functional languages like OCaml require that IF statement has an ELSE part.
This requirement may be relaxed if the statement under IF returns void, but this
is not necessarily the case in re2c-generated code (it depends on how the user
defines YYFN: they may choose it to be a void or non-void function). Besides,
other functional languages like Haskell do not have this relaxation.
The added test works correctly both before and after this change, however
it shows incorrect behavior if tags are not checked for equivalence at all
(in that case some URI fragments are missing in the test output).
These directives allow the user to simplify declaration of tags variables,
which is especially useful when the variables are for capturing paretheses.
Also, adjust commnets in submatch examples.
Previously it was recomputed from the numeric value every time. Although
the recomputation was trivial (comparing against zero), it made the code
more verbose and less readable.
This commit adds the following options and the corresponding configurations:
--captures (alias for existing --leftmost-captures)
--leftmost-captvars, --captvars (alias for --leftmost-captvars)
--posix-captvars
Variables are named `yytl<n>` and `yytr<n>` for the left and right
parentheses of n-th capturing group.
Warnings like -Wundefined-control-flow should be on by default, as they report
severe issues with the code that should be fixed. If the user wants to suppress
such warnings, they will have to do it explicitly on the command-line.
The only purpose of this configuration was to make the lexer fail on standalone
single quotes for languages that do not allow them. Removing it does not affect
how the lexer works: it still attempts to lex single quote as the beginning of a
char literal and looks for a matching closing quote. The only thing changed is
that now the lexer does not raise an error on standalone single quotes for
languages that do not allow them; it simply ignores them in the same way it does
for languages that do allow them (e.g. identifiers in Haskell/OCaml, numbers in
C++, labels/lifetimes in Rust, etc.). This goes along with the general re2c
behaviour: it does not attempt to parse any languge - it only does the bare
minimum of work needed to parse re2c blocks correctly.
This takes care of bug #482 "Handle single quotes correctly in the lexer".
The lexer does not attempt to lex all lexemes that may contain a standalone
single quote - those would differ a lot for different languages. Instead, it
tries to lex paired single quotes (character literals) which are similar enough
in different languages to allow a single generic lexer rule. If this attempt
fails, lexer checks if standalone single quotes are allowed (if yes, it just
skips the single quote and resumes lexing; if not, it fails with a syntax
error). This way lexer doesn't need to know about different types of lexemes
that may contain a standalone single quote.
This fixes a problem with libre2c introduced by the previous commit
8e9bfda737 which required that all code options
are defined (non-null). The problem was caused by testing if `line_dirs` option
is defined: the test happened in the code that is shared with libre2c, while
normally such tests happen in the code that is not shared. This commit moves
`line_dirs` initialization to config files, so the faulty test is removed
entirely.
It also changes the way that implied imuutable options are set: former
`set_default_*` functions that soft-set an option (mark it as not user-defined)
are replaced with `set_*` functions that hard-set the option (mark it as
user-defined). It shouldn't matter for immutable options, as they cannot be
redefined by the user.
an option only set an option specified in a syntax
file if it's not explicitly defined by the user. No tests have changed.
This commit adds a new warning: -Wundefined-syntax-config. It is not an
error, so that old syntax files stay valid as new syntax configurations
are added. Warnings allow the user notice that new configurations are
available and it's time to update their syntax file.
Not all languages support `yych:conversion` (this commit adds support
for it in C/C++, D, Go, Rust and Zig backends). Conversion is only
useful for backends that implement simple/record API and have a type
system strict enough to prevent implicit conversion.
Don't adjust file start/end pointers in cases when we reached the end of
the include file when reading from it, as YYLIMIT may point outside of
the portion of the input buffer occupied by this file. It may happen if
the previous `read()` invocation drained this file, then read some data
from the parent file (the one that contains include directive), and then
YYFILL got triggered while YYCURSOR was still in this file, so this file
could not be popped off the stack yet.
The error was rare - I found it by a lucky accident when the buffer got
resized to a certain size due to a change in syntax file. This triggered
YYFILL in the right moment (in encodings/unicode_identifier.re example).
This way the user may provide a syntax file with only a few overrides
instead of copy-pasting default definitions for all configurations.
As an example, add a test for the Haskell backend that pretends to be a
test for the Python backend, but overrides default Python config with
Haskell config. We don't use default C config in this case as it
defines a few optional configurations for oneline switch cases/ifs that
are not overridden by the Haskell config. The generated code is
identical to examples/haskell/01_basic.re.
Also, remove a stale TODO comment.
This should be configurable, as in some languages type system is strict
and defaulting to any particular constant means that tags must have the
type of this constant (and by extension cursor, marker, ctxmarker and
limit). It would still be possible to override them, but it would
require using generic API and providing definitions for all primitives.
In generic API `tags:negative` defaults to the tag variable that is set
to the negative tag value (substituited by re2c). We keep this behaviour
for backwards compatibility.
Also, slightly change parser rules for `%{`: instead of requiring that
it is on a new line (possibly preceded by whitespace), allow arbitrary
text before it on the same line, but require a newline immediately after
it. This simplifies parser rules, and it should be good enough to avoid
parsing `%{` occurrences in the middle of a string or comment by mistake
in most of the cases.
Previously their names were hardcoded in syntax files, now they delegate to
configurations `define:YYCURSOR`, `define:YYMARKER`, `define:YYLIMIT` and
`define:YYCTXMARKER`. This results in slightly longer field names, but also
more consistent with default and generic APIs.
In some cases it may be necessary to override the default names, e.g. Go
requires public struct fields to be uppercase, while private ones should be
lowercase.
Previously their names were hardcoded in syntax files, now they delegate to
configurations `variable:yyaccept`, `variable:yycond` and `variable:yystate`.
With the new record API defined via syntax files, it may actually be useful to
delegate some `code:` syntax configurations to generic API primitives. It allows
one to customize the generated code for record API by redefining only a few
selected primitives.
Add a new option `--api record` and a new configuration `re2c:api = record`.
Use the new record API in syntax file for C. Switch some C examples to it.
Previously API definitions like YYPEEK, YYSKIP etc. were always strings, both in
syntax files and in source files. Now in syntax files they are code templates;
in particular, they should use variable `sigil` instead of substring `@@`. In
source files they are still strings: there's no need for code templates there,
and if fact it's not possible for them to become code templates, as source files
allow unquoted strings which would be mis-parsed as variables in code templates.
Using code templates in syntax files allows one to make conditional language-
specific defaults for APIs depending on mutable options, such as the current
API, API style, etc. Previously it was only possible to use immutable options as
predicates. Using variable substitution for `sigil` allows us to automatically
replace default sigil `@@` with a user-defined one.
Previously sustitutions were always done in the free-form API style,
while the function style was inconsistent: for most primitives
substitutions weren't done, but for a few primitives they were done.
This doesn't matter in real life, as the function style assumes that the
primitives are function names, which cannot contain the sigil string
"@@" (unless it was redefined to some alphanumeric string by the user,
which is unlikely).
Previously it was used to specify optional modifiers. Syntax configs for
some languages utilized modifiers for mutability annotations (Rust) or
strictness annotations (Haskell). More generally, it was used in cases
when the formal parameter syntax differs from the argument syntax.
However, modifiers were not flexible enough: they can only be added on
one side, while sometimes both left and right context is needed.
This patch makes the 3rd component mean the whole formal parameter. It's
still optional: when not specified, 3rd component defaults to the 1st
component (argument). However, if parameter syntax differs from the
argument syntax, it is now possible to specify arbitrary text in each.
Instead of always emitting error prefix `re2c:` emit `re2go:` for
re2go, `re2rust:` for re2rust, etc. Note that this doesn't affect the
runtime --lang option, e.g. for `re2c --lang go` the error still starts
with `re2c:`.
No tests, as the testing script replaces all binary names with --lang.
Provide a less verbose interface for generating language-specific expressions
during "generate" codegen pass. Encapsulate some of the details in helper
methods of the `opt_t` class.
The new primitives and configurations allow one to override `yyaccept`
syntax for a single source file or for a given language backend.
For existing languages the new primitives have default definitions:
nothing changes in the code generation, and no new definitions are
required from the user. The change is fully backwards compatible.
Namely, remove examples that use end-of-input rule `$`, and leave
examples that are based on the sentinel method (as they are simpler and
demonstrate the technique well enough.
This condition is true when there's more than one IF branch. It is
needed for Haskell, which uses a different syntax for one-branch IF
(the `when` statement).
This is needed because the check itself is language-specific, and also
the IF construct is problematic in some languages (e.g. in Haskell it
would be impossible to get the updated state out of the IF branch
without making the IF expression part of the YYSHIFTSTAG operation.
The following configurations have been added:
- code:shiftstag
- code:shiftmtag
- code:stagp
- code:stagn
- code:mtagp
- code:mtagn
- code:copystag
- code:copymtag
Mark `code:shiftstag` and `code:shiftmtag` as oneline configurations
(in fact they should be named one-statement, not oneline).
Note that the user can define YYSHIFTSTAG/YYSHIFTMTAG as multi-statement
without the curly braces, and thus break the assumption re2c makes about
the generated code. But it is also the case with YYFILL (and has always
been), so we don't make the situation much worse. This requirement
should be documented, though.
Previously default API had some shortcuts, e.g. in case of fixed tags
the operations of copying base tag and subtracting offset were fused
into one. This is no longer the case.
Replace the complex `code:yyrestorectx` configuration that had many
conditionals with a few simple configurations `code:yyrestorectx`,
`code:yyrestoretag`, `code:yyshift`. Common language-independent logic
belongs in the re2c source code (to avoid duplicating it in each syntax
file, and also to avoid unecessarily complicating syntax files).
Previously re2c generated assignment chains like this:
t1 = t2 = ... = tN = YYCURSOR;
Now it generates a sequence of simple statements:
t1 = YYCURSOR;
t2 = YYCURSOR;
...
tN = YYCURSOR;
Syntax configuration `code:assign` has been updated accordingly:
the `lhs` variable no longer needs to be a list variable.
For most languages they have default values (in syntax files) that never
need to be configured manually by the user. However, for languages like
haskell it is impossible to provide a reasonable default value - that's
why we need these primitives.
This reverts commit 8fb3d50f8d6dd8aace87631bb1a961b6e13c4536.
Reason for revert: change of plans: no need to rename these primitives
unless we add YYGET{S,M}TAG, and the latter can be avoided if we use the
3rd component of YYFN elements for arguments (as opposed to parameters).
E.g. if some API primitive X is defined as "...@@..." in syntax file,
and the user specifies sigil "$" instead of "@@", then default X
definition should become "...$...".
Previously this was only done for `re2c:cond:goto`, as no API primitives
have default definitions that contain sigil. But we don't really know
default API definitions, as the user may provide a custom syntax file
with custom definitions.
Error on disabling unsupported features (previously re2c only errored
on enabling them).
Add `unsafe` and `monadic` features and tests for them.
Update tests for existing unsupported features.
The new configurations are needed to support monadic lexers in the
upcoming Haskell backend.
Configuration `monadic` affects control flow constructs: when it is on,
the "do" keyword is generated before nested blocks.
Configuration `variable:yyfill" allows one to decouple YYFILL invocation
from the check whether YYFILL has failed. This is needed becausse YYFILL
performs I/O (like reading from file), so it should return the updated
state of the world together with the status whether it has failed or not.
The new option cotrols whether to strip curly braces from semantic
actions, and whether to wrap the whole program in curly braces.
Previously this was deduced from other options, e.g. from
`indentation_sensitive` in Python. However, in some other languages this
cannot be so easily deduded (e.g. Vlang uses curly bracees in constructs
like `if`, `switch`, etc., but it rejects a compound statement wrapped
in curly braces, unless it's separated from the previous statement with
a semicolon).
Replace many copies of examples for each language/codegen model
combination with just two tests for all auxiliary code models: a basic
one and an advanced one that covers storable states, conditions, tags,
end-of-input rule $ and `yyaccept`. Other tests are left only for C/C++
backend, as it is sufficient to have them for one backend (e.g. to test
errors specific to this codegen model).
The rationale for this change is: when new backends are added to re2c,
the amount of tests grows significantly, slowing down test run time for
no good purpose (most of the new backends are experimental). Also, it
slows down the addition of new languages (e.g. Vlang supports all three
codegen models --- goto/label, loop/switch and rec/func --- but adding
three full sets of examples for it is an overkill).
Also, remove semicolon from `code:abort` definition in Rust syntax file,
so that it can be used in one-line switch-case context.
Update test results accordingly.
It may be used in syntax file - but not necesarily. C, D and Go backends
don't use it. Rust, on the other hand, requires that all variables are
initialized.
Previously there was a hardcoded 0 constant in the syntax file for Rust.
Now the initializer is 0 or '\x00' depending on `variable:yych:literals`
configuration and the current encoding (the same condition which is used
to decide how to print YYCTYPE literals).
They are usually faster than switch cases on the tests, e.g. on the
newly added test/switch_vs_ifs.re
nested ifs took 0.17752790451049805
switch cases took 0.6691319942474365
As this case is trivial, it's easier to copy-paste the following case
than to add a portable fallthrough annotation (I'm not aware of an easy
way to do it in C++11).
Put syntax configurations in a struct and make it part of the immutable
options instead of storing them in a map. This simplifies their handling
and access, as there's no need to check if a key is in the map, etc.
Merge syntax file lexer/parser into configuration lexer/parser, reuse
existing methods.
Rename list syntax configurations and define them as lists of strings in
quotes (rather than words) so that we can reuse configuration parser for
lists.
Remove separate "features" syntax configurations and replace them with a
single list of features (to be augmented in future commits).
Referencing another `code:*` configuration in the middle of a definition
is not allowed (that would be hard to implement, as appending/prepending
to `StxCodes` list when using it in another configuration would alter
the original list). However, referencing another configuration as an
entire definition is ok, as it doesn't alter the existing configuration.
OCaml supports character ranges like 'a'..'z', but it does not support
numeric ranges like 0..9, therefore previously all ranges were rendered
in expanded form, like 'a'|'b'|'c'|...|'z'. This results in sometimes
very long lines with a lot of alternatives in match expressions.
This commit fixes it by adding a conditional into syntax file
definition. It checks if the range is of char type. If so, then the
range is rendered compactly.
One exception is the case of multibyte encoding: in that case
YYCTYPE literals are rendered as numbers (because in many languages
char type cannot hold values more than 1-byte).
Replace syntax configuration `yyctype_literals` with
`variable:yych:literals` that may have values `char`, `hex` and
`char_or_hex`.
Don't give a syntax error on backticks if the language does not support
backtick-quoted strings. Instead, simply skip them and move on.
Incorrect behaviour was introduced in commit
624aa6d72059e6180bab7298128e86b5f337e07c.
This is a very unlikely and useless case (trailing context that is
always empty), it is not worth specializing the code. In such case we
might as well generate a no-op operatio that decrements cursor by zero.
(Previously the operation was omitted, but an empty statement was still
generated.)
This is safe because each semantic action is placed into a separate
function, so there won't be any name collisions with local variables.
It is needed for language backends that don't use curly braces for
compound statements (like OCaml).
Add `have_retval` conditional to `code:fncall`/`code:tailcall` and
`retval` variable to `code:fncall` only (it is not needed in
`code:tailcall` as the latter returns immediately).
Also, change `code:fncall` to be a statement, like `code:tailcall`
(generate `retval = ...` if there is a return value).
This was problematic because previously newline was always rendered in
oneline statements (when rendering the statement itself, without any
knowledge of the surrounding context).
Now newline is no longer considered part of the oneline statement: if
needed, it should be added in the surrounding syntactic construct (in a
syntax file).
In some languages type modifiers go on the left of function parameters,
while the type itself is on the right (like `mut n: usize` in Rust). For
such cases, we need to allow optional third component of an argument
descriptor in YYFN. So, change its format from a list of 2*N strings to
a list of N strings with semicolon-separated parts (name, type, optional
modifiers).
Add `fn:sep` configuration to allow the users configure the separator.
Precompute function parameters and arguments, as they are common for all
function definitions / calls.
Move individual configuration initialization from re2c code to syntax
configs (previously it ws done in a post-initialization fixup step, but
now there's no need for it).
Previously empty header filename was concatenated with directory path,
resulting into incorrect header filepath. Change this to only prepend
directory path if the header filename is nonempty.
Previously this was done by comparing filepaths, however it broke on
windows as one of the filepaths was escaped (and on windows filepaths
have backslashes, so the escaped name differed from the original).
The same lexer state is used for all lexer files, so YYMAXFILL must be
the maximum of the individual YYMAXFILL values for each file. Previously
YYMAXFILL value for the main lexer was used everywhere, but this is
incorrect and may cause unexpected end of input errors on other lexers.
Previously all configurations were handled by the lexer, but the
grammar for conditionals is more complex (it allows nested conditional
expressions). These will be used in syntax files to set configuration
defaults for a language based on the global options.
One part of the old lexer still remains: the lexer for string
configurations in source files. Unlike syntax files, source files may
have raw (unquoted) string configurations, which may contain special
symbols used in other parts of the grammar (e.g in the new syntax for
conditionals). Switching to the new parser would require excluding these
symbols and thus breaking backwards compatibility.
The added configurations are named in the same way as in source files,
except that they use prefix `conf:` rather than `re2c:`. E.g. for a
source file configuration `re2c:state:abort` the corresponding syntax
file configuration is is a `conf:state:abort`.
Check that tags and YYSKIP are the same as on transition to which the
control flow falls through if the fallback transition is elided.
(Previously the code assumed that tags and YYSKIP are the same, but it
is not obvious why it should be so.)
Fallback transition is part of an IF/THEN/ELSE statement. In rec/func
mode both THEN and ELSE must end in a tailcall, therefore elisiion is
forbidden (except if YYFILL is disabled, then the whole IF/THEN/ELSE can
be replaced with a tailcall to to-state).
Normally final tags are user-defined local variables in the function
that wraps the generated DFA. However, in rec/func mode each DFA state
is a separate autogenerated function, so there's no place for the user
to define local variables. Therefore in rec/func mode final tags are
part of the state that is passed from one function to another.
We could theoretically autogenerate final tag declarations before
setting them in the final states, so that they are still local
variables. This would require adding YYSTAGTYPE / YYMTAGTYPE. But what's
worse, it would't work for languages like OCaml, where variables are
immutable bindings: fixed tags with generic API require a few operations
(set, then shift). If, however, final tags are part of the state struct,
they can be updated.
It is possible to add support for skeleton in rec/func mode, but it will
take some effort as skeleton programs use a lot of local variables that
must be passed as function parameters. Disable it for now.
If `getstate:re2c` is global (it has no explicit block list, assuming
that it covers all blocks in a file) then use global options accumulated
over the whole file. Otherwise use options of the first block listed in
the directive. For block-level labels/function always use block options.
The following configurations have been added:
`code:skip`
`code:yybackup`
`code:yyskip_peek`
`code:yypeek_skip`
`code:yyskip_backup`
`code:yybackup_skip`
`code:yybackup_peek`
`code:yyskip_backup_peek`
`code:yybackup_peek_skip`
Also, tweak syntax configuration `code:recursive_functions` to assume
toplevel functions rather than closures (as most languages do not allow
co-recursive closures, and some don't have closures at all).
In functional languages both branches must have the same type, and
C-style early returns are not possible.
In goto/label and loop/switch mode early returns are still preferred,
because the "else" part is often elided.
This is because in functional languages like OCaml "if/else-if/else" is
an expression where all branches must have the same type (except perhaps
for a single "if" branch that may have void type.
One exception is the case of a single branch that goes to the next state
(in that case "if/else-if/else" construct is not needed).
This is part of a general effort to unify rendering of oneline if/else
statements (either render them all on the same line without braces, or
all on a new line with braces).
Users that prefer always rendering if/else statements in braces should
be able to do so by commenting out `code:if_then_else_oneline`
configuration in the syntax file (`--syntax` option).
Note that default syntax file include/syntax/<language> (the one that is
used without `--syntax` option) is built into re2c; changing it requires
rebuilding re2c with configure option `--enable-syntax` or CMake option
`-DRE2C_REBUILD_SYNTAX=on`.
This is part of a general effort to unify rendering of oneline if/else
statements (either render them all on the same line without braces, or
all on a new line with braces).
Users that prefer always rendering if/else statements in braces should
be able to do so by commenting out `code:if_then_else_oneline`
configuration in the syntax file (`--syntax` option).
Note that default syntax file include/syntax/<language> (the one that is
used without `--syntax` option) is built into re2c; changing it requires
rebuilding re2c with configure option `--enable-syntax` or CMake option
`-DRE2C_REBUILD_SYNTAX=on`.
(It is not used anyway, as there is always a multi-line "else" branch
that makes the whole "if-then-else" construct render as multiline.)
This is part of a general effort to unify rendering of oneline if/else
statements (either render them all on the same line without braces, or
all on a new line with braces).
Users that prefer always rendering if/else statements in braces should
be able to do so by commenting out `code:if_then_else_oneline`
configuration in the syntax file (`--syntax` option).
Note that default syntax file include/syntax/<language> (the one that is
used without `--syntax` option) is built into re2c; changing it requires
rebuilding re2c with configure option `--enable-syntax` or CMake option
`-DRE2C_REBUILD_SYNTAX=on`.
This is part of a general effort to unify rendering of oneline if/else
statements (either render them all on the same line without braces, or
all on a new line with braces).
Users that prefer always rendering if/else statements in braces should
be able to do so by commenting out `code:if_then_else_oneline`
configuration in the syntax file (`--syntax` option).
Note that default syntax file include/syntax/<language> (the one that is
used without `--syntax` option) is built into re2c; changing it requires
rebuilding re2c with configure option `--enable-syntax` or CMake option
`-DRE2C_REBUILD_SYNTAX=on`.
This is part of a general effort to unify rendering of oneline if/else
statements (either render them all on the same line without braces, or
all on a new line with braces).
Users that prefer always rendering if/else statements in braces should
be able to do so by commenting out `code:if_then_else_oneline`
configuration in the syntax file (`--syntax` option).
Note that default syntax file include/syntax/<language> (the one that is
used without `--syntax` option) is built into re2c; changing it requires
rebuilding re2c with configure option `--enable-syntax` or CMake option
`-DRE2C_REBUILD_SYNTAX=on`.
This is part of a general effort to unify rendering of oneline if/else
statements (either render them all on the same line without braces, or
all on a new line with braces).
Users that prefer always rendering if/else statements in braces should
be able to do so by commenting out `code:if_then_else_oneline`
configuration in the syntax file (`--syntax` option).
Note that default syntax file include/syntax/<language> (the one that is
used without `--syntax` option) is built into re2c; changing it requires
rebuilding re2c with configure option `--enable-syntax` or CMake option
`-DRE2C_REBUILD_SYNTAX=on`.
These are currently used only for skeleton backend (which supports only
C/C++), but later they will be used for rec/func codegen model.
Split codegen for function definitions and function calls. Make it
simpler (don't try to format differently based on the length of
arguments, just output all the arguments on the same line).
Some function calls in skeleton programs are syntactically too complex
(they are part of OR expressions, or they have multiline arguments) ---
for such function calls do not use syntax configurations and fallback to
generating text. This works because skeleton supports only one language
backend.
If there is a need for more complex syntax constructs that contain
function calls in the future, it will be possible to add them as
separate syntax configurations (for that purpose `code:fncall` is an
expression, not a statement).
Pass `yych` instead of YYPEEK as the second argument to YYDEBUG.
Also, move YYDEBUG invocation in the initial state past YYPEEK so that
`yych` is properly set to the first symbol.
The motivation for this change is to decouple codegen for YYPEEK and
YYDEBUG statements and avoid having two syntax configurations for YYPEEK
(one for the expression that was previously used in YYDEBUG, and one for
the statement). Also, this helps in cases where YYPEEK modifies internal
lexer state, e.g. reads a character from file and moves file pointer.
Replace syntax configuration `abort_requires_include` with
`abort_in_default_case`. Enable it for Go and Rust, but not for C/C++ as
it requires an `#include` directive that must be provided by the user.
Update Go tests. Rust tests do not need updates because they use a
different codegen model (loop/switch) that has no initial switch on
conditions.
Instead, provide default values:
- empty string for string-valued configurations
- false for boolean-valued ones
- no output for `code:*` configurations
Also, fix an error in libre2c initialization: it now has to manually set
the target option, as it does not load syntax file (and so the options
are not properly initialized).
This is the first example of a single string valued configuration.
Leave it in the "code" group for now (`code:...`) but add a special
`Stx`method that returns an error if the value is not a single string.
Make the default value empty string, as we don't want to specify such
configurations in syntax files for languages that don't need them.
Options are passed everywhere by const pointer, as they should not be
modified anywhere in the program once they have been set for a block.
Syntax configurations also should be immutable, with the exception of
mutable stacks that are used when evaluating immutable configurations.
This was necessary to fix https://github.com/skvadrik/re2c/issues/201,
where line directives suppression was configured using configuration
`re2c:flags:no-debug-info` (the problem was that it didn't affect the
first block, and the first line directive was generated contrary to the
user's intent).
Now this workaround is no longer needed as the configuration
`re2c:flags:no-debug-info` was deprecated in re2c-2.2 and later removed.
Dropping this allows us to get rid of default copy semantics for `opt_t`
which is necessary in preparation to store `Stx` reference there
(the compiler won't generate default defintions in that case).
Some `Stx::*` methods are now called from src/options.opt.cc, but the
source file that defines them (src/options/syntax.cc) weren't added to
the list of library sources.
The fix is to make these methods inline, as they are small and the other
alternative (to include src/options/syntax.cc) would pull in even more
unrelated source files for the definition of DEFAULT_SYNTAX_{C,GO,RUST}.
Conditionals in syntax files were based on options, which caused a
dependency cycle: options required syntax configurations to initialize
default values, and evaluating syntax configurations required options.
Now syntax configurations do not depend on options.
The only conditional was in `char_literals` (a test for EBCDIC).
It can be as well hardcoded in the re2c source code, as there is nothing
language-specific in it.
The next step will be to simplify non-code syntax configurations and
allow only single word values or word lists (no conditionals).
Commit 1d9319548bc97313426499ad7b7aa6dfdfe71ac2 added a fix-up pass that
removes line directives (if they are not needed). It turns out that this
is unnecessary, as line directives cannot be enabled/disabled with a
configuration --- only with a command-line option. This option is
available early at program start, so there's no need for a clean-up pass
as we can test the option before adding line directives.
Line directives are generated by default for C and Go, but not for Rust
(which does not support them at all). They can be turned off/on with a
command-line option or configuration. From the codegen perspective,
there are multiple ways to avoid generating line directives:
1) do not add them in the first place
2) remove them during fix-up phase
3) render them as empty during rendering phase
1 is the most efficient, but it canot be used with configuration, as the
latter may be parsed later than the directive must be generated (some of
them are generated early during parsing, at which point the block
options are not known yet).
3 is the easiest to implement, but it confuses the checks for oneline
statements and results in ugly multiline formatting and lots of trivial
test updates.
Previously we used a combination of 1 and 3: for Rust directives were
not generated (language is an immutable option which is known at program
start), and for other languages directives were generated but not
rendered.
This commit changes the behaviour to 2 and makes it uniform for all
languages. C tests for which re2c is now able to properly detect oneline
statements have been updated.
It worked without a semicolon as well because panic was the last
statement in a block. However, this was unintentional and it may change
in the future.
Rust has an unusual syntax for labels: they have to start with a single
quote. But it is inconvenient to encode the quote in a syntax config ---
it's easier to encode it in the name, since the only kind of label used
in Rust is the yyloop label (which has a string name that is configured
with `re2c:label:yyloop` that can easily encode the quote as well).
Reuse code item for `code:var` configuration.
Change Go codegen to assume that YYMAXFILL and YYMAXNMATCH are unsigned
and update test results accordingly. (Other codegens already assume that
YYMAXFILL and YYMAXNMATCH are unsigned).
Namely:
- add `have_init` conditional instead of using `jumps.loop_switch`
- add `type` variable instead of using `storable_state` conditional
and `code:type_cond_enum` configuration for the underlying type
These changes make `code:enum` more generic and usable not only for the
specific case of condition enum, but for any enum (although it's not
needed at the moment, as condition enum is the only use).
Remove obsolete `GenEnum` class.
Don't add `static const` type qualifiers in the C syntax file. Instead,
add them as part of the type specified in the re2c source code. This
will be fixed later, as the type should not be hardcoded based on the
language (it will become a config in the syntax file).
Prepare for rendering tables using syntax configuration.
Update test results. There is a lot of changes, but they are trivial:
- drop optional comma at the end of array initializer for C
- change formatting: `void *` to `void* `
- add explicit array size in array definitions
Update syntax files for C/Go/Rust accordingly. Remove unnecessary
syntax configuration `code:yyctype`, as it is already configurable
via `re2c:define:YYCTYPE`.
This was mean to be used with `code_loop` for languages that have a
label preceding a loop, so that re2c would know whether to indent the
label or not (the default assumption being to indent everything).
However, now that we use explicit `topindent` variable everywhere,
there's no need for `label_indent`.
This is so that syntax files for languages that don't need special
handling for oneline statements (like Go) do not have to handle
`oneline` conditional (which may be confusing and unexpected).
List bounds need to be evaluated at codegen time (not earlier), because
the indices specified in syntax file may be negative and must be
converted to non-negative (the size of a list varies and is not known
until codegen time). To avoid duplicating the same code for every list
handler, this commit moves it to the generic function that renders
syntax trees.
Also, add `eval_cond` method to evaluate conditionals in syntax files
and fix the generic `eval_cond` method to respect configuration-local
conditionals.
Previously re2c was built with syntax file for C, re2go with that for Go
and re2rust with that for Rust. However each binary needs all three
configs, as the language may be changed dynamically with --lang option.
It is the first codegen function that is based on syntax configuration.
The chosen configuration is generic enough: it has lists with specific
bounds, conditionals, global and local variables.
Previously it was parsed as part of complex expressions, which should
not be allowed by the grammar as it makes it too lax and introduces
meaningless constructs.
Also, remove parenthesized expressions other than conditionals, as they
aere not used (they also caused LR conflicts in grammar).
This simplifies the find_package() call. The version in find_package()
has been initially supported in CMake 2 and 3. Version ranges
(versionMin...[<]versionMax), on the other hand, are only supported by
CMake 3.19+.
And this also helps a bit when adding re2c as a FetchContent dependency
in CMake project where finding Python is not necessary and can be
disabled with CMAKE_DISABLE_FIND_PACKAGE_Python3 variable.
This addresses bug https://github.com/skvadrik/re2c/issues/471
"Examples in re2rust manual have undefined behavior".
In Rust, it is UB to call `get_unchecked` on a slice that may be empty,
and many examples had functions that did just that (when invoked on
incorrect input data, since they use some form of sentinel method to
handle the end of input, which relies on the input being sentinel-
terminated).
The introductory example has been fixed to avoid UB and use simple
indexing with bounds checks instead of `get_unchecked`. Other examples
that had UB are now annotated with `assert` that checks conditions under
which the functions are safe to invoke.
Also, eof/02_bounds_checks.re example now uses a more efficient way to
construct YYMAXFILL-padded input string.
It loks like at least in `windows` `actions/upload-artifact@v4` CI step
fails to collect files at `${{ steps.build-info.outputs.BUILD_DIR }}/test_[0-9]*`
My theory is that `glob` is not able to match paths like
`D:/a/re2c/re2c/install-2`
against theit unix-like form of
`/d/a/re2c/re2c/install-2`
I suspect that `pwd` returns us the second form while
`${{ github.workspace }}` might keep an original form.
Lex char literals properly as single characters or escape sequences
rather than lexing them as single-quoted strings. Add syntax rules from
C/C++, Go and Rust (they are similar and do not contradict).
Add a Go test for string and character literals (prior to this patch it
caused a syntax error).
Note that the same lexer method is used for indented code blocks and
code blocks in curly braces.
This is a revert of commit 77714fce94.
While it is easier to always generate braces, in some languages (Go) it
is a compilation error to jump into the wrapped block (a scenario that
happens if there is an outer state switch generated with `getstate:re2c`
directive.
This regression was found because one of the examples for Go didn't
compile (examples/state/push.go). Now it does.
Document the following options:
--leftomost-captures
--invert-captures
And the following configurations:
re2c:leftmost-captures
re2c:invert-captures
And the new `(! ...)` syntax.
It cannot be a `enum class` because it's used in a bitfield and this
triggers compiler warnings and errors on some toolchains (see
https://github.com/skvadrik/re2c/issues/446 for details).
It cannot be a plain `enum` defaulting to `int` as well because of
-Wsigned-enum-bitfield warning (noticed when testing on OpenBSD with
clang-13).
On Unix systems, re2c uses POSIX functions (e.g. fdopen). Certain
systems, such as Cygwin and NetBSD, enable these funtions conditionally
based on certain preprocessor definitions.
Previously https://github.com/skvadrik/re2c/pull/376 fixed the build on
Cygwin by setting _POSIX_SOURCE. However, this broke the build on
FreeBSD which was fixed by commit
This worked for Autotools build system which uses -std=c++11 due to the
use of AC_USE_SYSTEM_EXTENSIONS, but it didn't work for CMake that used
-std=c++11 and disabled extensions with `set(CMAKE_CXX_EXTENSIONS OFF)`.
This commit drops `set(CMAKE_CXX_EXTENSIONS OFF)`, which effectively
enables -std=gnu++11 instead of -std=c++11 for CMake builds. This allows
us to get rid of the problematic _POSIX_SOURCE that breaks the build on
NetBSD.
This was tested on NetBSD-9.3, FreeBSD-13.2 and Cygwin with both
Autotools and CMake.
With `--flex-syntax` option re2c accepts raw (unquoted) strings.
If `--input-encoding utf8` is also enabled, re2c should parse raw UTF-8
characters in such strings. Previously it failed to do so because the
characters in raw string literals were limited to alphanumeric ASCII and
underscore (those allowed in indentifiers, as in normal mode unquoted
strings would be parsed as identifiers).
This commit fixes it: now in Flex mode and with UTF-8 input, if all
previous rules failed, re2c tries to lex the input as a UTF-8 character.
This fixes bug #447 "Re2go fail when compiling expression containing
combining character".
GCC-5.4 is unhappy that `GorPass` which is `enum class: uint32_t` is
stored in a bitfield of size 2 (2 bits). In practice it is safe since
the enum has only three values (0, 1, 2).
Reported in:
- #445 "make failure for re2c"
- #446 "make failure for re2c on centos7"
Change file context definition to cover more scenarios that include
filenames, e.g. the case when there is a depfile generated and appended
to the output.
This allows other projects that use bazel to utilize re2c as a tool.
Limitations:
. Input files (docs, lexer, parser) are not being re-generated.
. No tests are executed.
. Only the main binary is being build as nothing else is needed.
A user will load the package using bazel's 'http_archive' in their
WORKSPACE file.
In their BUILD files they can then use:
```
load("@re2c//bazel:re2c.bzl", "re2c_gen")
re2c_gen(
name = "file_re2c",
srcs = ["file.re2c"],
outs = [
"file.cc",
"file.h",
]
)
cc_library(
name = "file_cc",
srcs = ["file.cc"],
hdrs = ["file.h"],
)
```
The approach here is to automate as much as possible. In that way, all
source and header files will be found automatically (using glob). Also,
the provided bazel integration is kept to an absolute minimum.
An alternative is to use a `rules_foreign_cc` based approach. But that
would require much more bazel work and knowledge. Plus the result would
end up being a non-hermetic build, thereby bypassing bazel's advantages.
Further, using a rules_foreign_cc approach is much slower (for me ~x10).
Make option --invert-captures configurable. This requires postponing the
decision whether a group is capturing or not to the after-parsing stage
when the configurations for a block are known and the immutable AST is
converted to a regexp.
Add a test for the new configuration.
The new option inverts the meaning on capturing and non-capturing
parentheses: `(! ...)` becomes capturing, and `( ... )` becomes
non-capturing, including the implicit whole-regexp group, which also
becomes non-capturing.
The idea of the new option is to let the user decide which gropus should
be capturing and which are purely structural, and explicitly mark only
the capturing ones for submatch extraction.
The old syntax was `(? ...)`, the new syntax is `(! ...)`. This is in
preparation to add --invert-captures option that will flip the meaning
of capturing and non-capturing parentheses.
This fixes a warning on GitHub actions workflows: "The `set-output`
command is deprecated and will be disabled soon. Please upgrade to using
Environment Files".
The new option allows one to use POSIX syntax for capturing groups
with leftmost-greedy disambiguation. This comibination has already
been supported by libre2c, so the changes needed to expose it in re2c
are minimal.
In the case of POSIX disambiguation (with --posix-captures option) it is
necessary to add structural tags around non-capturing groups, otherwise
disambiguation will be incorrect.
Commit f519385213 that allowed the use of
implicit non-capturing groups didn't properly add structural parentheses
around implicit capturing groups (which are needed if the group contains
nested capturing groups).
Commit 1edd25d31d that added explicit
non-capturing groups repeated the same error, only for the newly added
syntax `(? ...)`.
This resulted in `(? "a"* | ("aa"))*` being parsed incorrectly, as if
leftmost greedy disambiguation was used instead of POSIX disambiguation,
giving priority to the shorter first alternative `"a"` rather than the
longer match `"aa"` as POSIX requires.
This commit ensures correctness for both explicit and implicit
non-capturing groups by adding proper structural tags for the case of
AST sub-node `REF`. It is expected that it will bring back some of the
overhead on POSIX disambiguation reduced by the introduction of
non-capturing groups.
Related bugs: #438, #308.
Rename tests for implicit non-capturing groups so that they are named
similar to the tests for explicit non-capturing groups, and thus easier
to locate and compare.
This sub-ode is used for named definitions and non-capturing groups. The
latter do not have a name (previously a dummy name "<group>" was given
to such groups, but it turned out that the name is unused anyway).
This is useful in POSIX mode (with --posix-captures option) in order to
avoid unnecessary captures and reduce the amount of tags.
The new syntax for non-capturing groups is `(? ...)`.
Add tests for the new feature. Note that test results coincide with the
corresponding test for implicit groups (where the same group is not
parenthesized at all, but implicit grouping is achieved with the help
of a named definition).
This addresses bugs #438 "POSIX captures processing" and #308
"Overloading precedence marker (e.g. (pattern)) with 'posix' group
markers (also parenthesis) causes loss of full functionality".
Allow the use of named definitions that cause implicit grouping and
circumvent the absence of syntax for non-capturing groups. This is not
allowed by the POSIX standard, but this feature is useful in practice
and many regexp engines support it.
Update the tests that should no longer result into a compilation error.
This addresses bugs #438 "POSIX captures processing" and #308
"Overloading precedence marker (e.g. (pattern)) with 'posix' group
markers (also parenthesis) causes loss of full functionality".
This reverts commit 8df0ed6737.
Reason for revert: D backend is not ready yet: there are no examples
and tests, the work on it has stopped. In the future support for D and
other languages should be added in a generic way via language syntax
files, rather than modifying re2c codegen to support each new language.
Previosuly they were allocated using AST allocator, which made it
necessary to keep AST memory until the main codegen pass. Now it is
possible to release AST memory after parsing.
Rust does not support line directives. Previosuly re2c generated stubs
that were replaced with empty statements and later removed by a follow
up codegen pass. Now the stubs are not generated at all in Rust mode.
Labels cannot be generated during main codegen phase: codegen for state
dispatch may add more use cases for block start labels. These labels may
come from blocks either before or after state dispatch that uses them,
which makes it difficult to decide in advance (especially since state
dispatch itself cannot be expanded before the main codegen phase).
Therefore label generation has to be delayed. However, there is no need
for a separate codegen pass, as it can be done by the rendering pass.
All the functionality has moved to the main codegen pass, as it no
longer needs to be delayed. Second delayed pass still remains: it
expands labels and removes line directives for rust.
It no longer needs to be delayed until late codegen phase, and there is
no need for a special code kind: it can be generated during the main
codegen phase.
It no longer needs to be delayed until late codegen phase (all blocks
are parsed before the main codegen phase, and the list of conditions is
known by that time).
YYGETSTATE is only needed in the YYFILL call, it doesn't matter in case
YYLESSTHAN check fails and YYFILL is not called at all. The value of
state variable cannot "leak" into semantic action, as it is reset to -1
immediateluy before executing the action.
Previosuly YYGETSTATE was generated inside of the YYLESSTHAN check if
the end-of-input rule $ was used, but outside if it wasn't used. This
commit unifies the behavior.
Previously they were allocated during codegen, which implied that
state dispatch in storable state mode had to be generated in the late
codegen phase (because fill labels are used in transitions for state
dispatch).
Use blocks should generate state goto even if it has previously been
generated. However, that should not affect normal blocks (and previosuly
it did, as the global boolean flags was reset when seeing a use block).
This is so that we can gather whole-program information during the
first pass and use it during the second pass (obliviating the need for
late codegen).
The only advantage of `OutputFragment` compared to just `CodeList` was
that it additionally stored indentation level. However, it turns out
that just using top indent in all fragments produces the same output
(some of the fragments are line directives that always use zero indent).
Previously parsing and early codegen were intermixed: early codegen for
each block was done before parsing for the next block started. This is
problematic, because it creates the need for a late codegen phase that
expands whole-program constructs such as state goto or condition enum.
And then an extra pass is needed to generate labels. Used-label analysis
is spread throughout all phases: some of it is needed before early
codegen (so that we know which goto transitions can be elided and avoid
generating redundant loop/switch cases, and also to decide the need for
backup statement if there is a loop through initial state). But some of
the used-label analysis happens during the late codegen phase (expansion
of state goto may add new uses for start labels).
Separating parsing and codegen will let us simplify codegen, get rid of
the late phase and make used-label analysis and label enumeration more
clean and compact.
Previously this was done only if there were declarations of local
variables. However, the cases when braces can be omitted are rare, and
detecting them makes the code more complex for very little value.
Start label is only needed for the first DFA in a block: although it was
previously attributed to a DFA, it should be attributed to the whole
block. Autogenerated start label is used by state dispatch in storable
state mode. It is generated before condition dispatch, which is also
generated once per block.
This patch moves start label and condition dispatch generation out of
the per-DFA codegen loop and makes it explicit that they are generated
once per block.
- In addition to a C++ compiler, Windows builds require a build system
(CMake or bash). Clarify the dependencies to say this.
- Python 3 is required for tests, and tests should always be available,
and both CMake and Automake give a hard error for missing Python 3,
so just say Python 3 is required.
- Update docs rebuild dependencies to specify Python3 and Docutils,
since that is what you need to install to make rst2man available.
- Add an explanation of how to use the CMake presets since they're pretty
good.
- A few other minor corrections.
- Prequisite check for Python3 docutils module instead of rst2man.
- Use `split_man.py` instead of `split_man.sh`.
- Use `rst2txt.py` instead of `rst2man` + `gen_help.sh`.
- Explicitly use `$(PYTHON) <scriptname>` when calling python scripts.
Regenerate files using haskell script test/encodings/unicode_groups.hs
that is based on Data.Charset library (charser-0.3.9 package). The
script generates tests in test/unicode directory inplace and include
file unicode_categories.re.inc that needs to be copied to
include/unicode_categories.re.
Update test results. As there are substantial changes in the standard,
updates are also nontrivial. They have been verified with
test/__run_unicode_tests.sh.
The three examples for C, Rust and Go were werified with __run_all.sh
scripts in examples/{c,go,rust) respectively.
Pretty much every Python install includes Docutils, and Docutils is 95%
of what is needed to bootstrap docs on Windows. Fill in the rest.
- `docutils_txt.py` is a Docutils text-output Writer class with support for
writing a C file.
- `rst2man.py` is a wrapper for Docutils converting to MAN file. (Copied
from Docutils repo.)
- `rst2txt.py` is a wrapper for Docutils converting to TXT file. (Same
as rst2man.py but it uses docutils_txt as the Writer.)
- `split_man.py` is a Python implementation of `split_man.sh`.
- Change CMake build to use the new scripts.
Additional changes:
- CMakeLists: give error message if docutils module not found.
- ci-cmake: Give hint about where to find Python3 so that the private
install is preferred over the system install. Enable docs regen for
Windows.
Without the change on `gcc-13` build warns as:
./src/parse/input.h: In member function 're2c::loc_t re2c::Input::cur_loc() const':
./src/parse/input.h:139:20: warning: unused variable 'p' [-Wunused-variable]
139 | const uint8_t* p = cur;
| ^
The change turns `DCHECK(x)` into a statement that uses `x`.
Previously if start conditions were used, bitmap definitions were
generated at the start of each condition, as they all had the same name
(`yybm` or a user-defined one). This was problematic in storable state
mode, as transitions to `YYFILL` labels bypassed bitmap declaration
(state dispatch is always at the beginning of a block).
Now we use condition-specific names for bitmaps (`yybm` plus underscore
and condition name), so they can all be defined together before state
dispatch, as it happens in blocks that do not use start conditions.
Both pairs of files contain lexer-related declarations and definitions,
the headers are usually included together, and the input.{h,cc} files
are very small.
Do not use a struct, instead pass individual parameters. They are often
used in semantic actions, and addressing them as struct fields adds too
much noise in the source code.
It performs almost identical to GOR1 (sometimes slightly better or
worse), but it has no proven worst-case bound.
Apparently both the updated test and benchmark erroneously flipped GOR1
and GTOP (but the results were so close it didn't matter).
Before the change parsers relied on `YYSTYPE` global variable.
Not very handy for libre2c if it's expected to be used in reentrant
or multi-threaded contexts.
The change enables `%define api.pure full` and adjusts lexer calls
to explicitly pass through `YYSTYPE` around.
`%define api.pure full` is preferred to `%pure-parser` (deprecated in
2008, around Bison 2.3b). Should be safe to use.
Autotools-based and cmake-based build systems had slightly different flag order:
Makefile.am: $(AM_V_GEN)$(BISON) --warnings --output=$@ --defines=$(@:cc=h) $<
Let's make them identical.
New `bison --warnings` gives the following helpful diagnostic:
GEN src/parse/parser.cc
src/parse/parser.ypp:60.2: warning: empty rule without %empty [-Wempty-rule]
60 | : /* empty */
| ^
| %empty
%empty was introduced in bison-2.7.90 around 2013.
The change allows catching fresh bison diagnostics. Current example:
GEN src/parse/parser.cc
src/parse/parser.ypp:60.2: warning: empty rule without %empty [-Wempty-rule]
60 | : /* empty */
| ^
| %empty
The warning itself will be fixed in another change.
Run tests as:
bash -c "ulimit -s 256; make check -j$(nproc)"
I also verified that the tests fail on CI without commit
460412913771e9b7b9506bc623470195f6ccf21e (the last change that was
needed to fix#219).
This fixes bug #219. Tested as follows (all tests passed):
bash -c "ulimit -s 256; time ./run_tests.py"
Original test that revealed the problematic recursive functions:
bash -c "ulimit -s 256; echo run | gdb --args ./re2c overflow-1.re"
Save this information in each subtree and compute it bottom-up on AST
construction. This means a slight constant overhead on construction for
non-POSIX case (where this information is unused), but it eliminates a
worst-case quadratic algorithm for the POSIX case (where the property
was recomputed for each alternative/concatenation subexpression).
Note that tags were counted as captures by the old implementation, but
not by the new one, as POSIX regexps cannot have standalone tags.
Lowercase type names prefixed with `_t` are used for containers and
utility structures. Domain-specific types use camel case.
Also, rename REG_REGLESS to REG_MULTIPASS (but keep the old constant
definition for compatibility).
Previously we used signed characters and `re2c:yych_conversion = 1;` to
convert from char to uint8_t in the generated lexer code (the lexer
compares `yych` to various literals which may be outside of the signed
ASCII range, depending on the lexical grammar). As a consequence,
-Wold-style-cast was disabled.
Unfortunately now we have to disable -Wreserved-identifiers, as it warns
about user-defined literal suffixes like `_u8` that must start with an
underscore (the standard reserves non-underscore prefix for the future).
Use four basic allocator types (defined as different structs, so not
possible to mix with each other by accident):
- AST allocator (may be shared between blocks)
- IR allocator (intermediate representations for a single DFA)
- DFA allocator (codegen for all DFA in a block)
- output allocator
Even /W4 emits too much (informational warnings that we are not going to
fix), and /Wall is /W4 plus some warnings that are off by default. See
https://docs.microsoft.com/en-us/cpp/build/reference/compiler-option-warning-level
Also disable some selected warnings that add to much noise (C4068 that
complains about unused pragmas and C4996 that complains about functions
like `fopen`).
Previously some of the warnings have been disabled with pragmas, but not
all warnings can be fixed this way, as some of the macros are emitted
too early (before the pragmas).
The fix is to "use" the macros in trivial `#ifndef` clauses.
They were added in C99, so should be available in C++11.
For int64_t / uint64_t use PRId64 / PRIu64 macros from <cinttypes>.
Also fix some formatting issues.
Assertions are used in tests and elsewhere to verify assumptions that
must be correct, or else the program should abort. Compiling them away
in release mode is dangerous. For debug-mode assertions we use `DCHECK`.
A previous fix of error handling revealed memory leaks caused by not
destructing common data structures like tags, rules and charset that
live through different intermediate representations and move from one to
another (regexp -> TNFA -> determinization context -> TDFA -> ADFA).
Previously they were stored as references, and the last IR (ADFA) took
the responsibility for destroying them. This works with normal control
flow, but if there is an error in one of the previous IRs, the program
exits without constructing and (what matters here) destructing ADFA.
The fix is to use move semantics introduced in C++11: instead of storing
references, store values and move common data structures from one IR to
another. This way they are always owned and destroyed by the same IR
that causes the error.
This patch makes all tests pass under valgrind:
$ ./run_tests.py --valgrind
$ LD_LIBRARY_PATH=.libs \
G_SLICE=always-malloc \
G_DEBUG=gc-friendly \
GLIBCXX_FORCE_NEW=1 \
valgrind \
--track-origins=yes \
--trace-children=yes \
--num-callers=50 \
--track-fds=yes \
--leak-check=full \
--show-reachable=yes \
--malloc-fill=0xa1 \
--free-fill=0xa1 \
--suppressions="$HOME/misc/valgrind/all.supp" \
.libs/test_libre2c
A long time ago `std::valarray` was used because it did not require copy
semantics for rules. But this depends on the libstdc++ implementation
(if I remember correctly, macOS implementation required copy semantics).
Instead of doing `exit(1)` after printing an error message, return
error code to the calling function and propagate it all the way back to
`main`. Avoid boilerplate and bloating the code with checks by using
consistent error codes throughout the codebase and wrapping function
calls in a `CHECK_RET` macro. Use another macro `RET_FAIL` for errors.
This change has reduced the number of failing tests under valgrind from
131 to 32 (previously all the tests that report an error failed, because
`exit(1)` bypassed the destructors and didn't free memory).
In CMake this is done by adding autogenerated sources to an "object
library", which is not really a library, but just a collection of files
that can be used as a library in the sense that other targets can depend
on it. The dependency can be achieved by either adding it explicitly to
another library with something like `target_link_libraries`, or by
adding the object library to the list of source files.
For libre2c the latter was done slightly differently (and incorrectly):
instead of listing object libraries in the list of sources, normal
sources were added to another object library, and the object libraries
for autogenerated and normal sources were listed as sibling entries on
the sources list. So there was no dependency from the object library for
normal sources on the object library for autogenerated sources.
This caused build errors when a normal source file included an
autogenerated header, which was not generated yet. The fix is to not use
object library for normal sources.
For some allocations, move them to slab allocators. For others, replace
them with temporary buffers. Avoid allocating and freeing temporary
values with `new`/`delete`, which was previously done in order not to
store "fat" value on the parser stack (also since non-POD values cannot
be stored in the `yylval` union).
Previosuly CMake checked for Bison, and if it was present, then it
regenerated parsers and updated bootstrap files. Now it checks
RE2C_REBUILD_PARSERS, and if it is enabled, then CMake fails if it
cannot find Bison, regenerates parsers and updates bootstrap files.
This fixes `make distcheck`, which previously failed when trying to
update bootstrap parser (it makes the source directory read-only. Now it
no longer fails because rebuilding parsers is disabled by default.
Also, cleanup Makefile.am implementation for rebuilding lexers with
RE2C_FOR_BUILD: there is no need to check that the re2c executable
exists, as it is done by configure.ac.
This is the `astyle` command used for autoformatting:
$ # run in source directory
$ astyle --style=google \
--max-code-length=100 \
--align-pointer=type \
--align-reference=type \
--keep-one-line-statements \
--suffix=none \
--recursive '*'
Although it is an interesting algorithm, re2c benchmarks show that it
performs worse than TDFA(1): on complex regular expressions staDFA are
often much larger and slower than TDFA(1). See the benchmarks on the
official re2c website: https://re2c.org/benchmarks/benchmarks.html, or
the paper https://arxiv.org/abs/2206.01398 for details.
StaDFA implementation adds complexity to the most complex part of re2c
code -- determinization, and it takes some effort to maintain.
The algorithm is not fully documented and formalized: the original paper
by Chowdhury is incomplete, and the implementation is re2c had to use a
few fixes and modifications to make it work.
The benchmarks for staDFA now use re2c-3.0 to build.
This fixes bug #405 reported by Daniel Macks
"Insufficient version-check for python (run_tests.py failures)".
For CMake, prefer manual check to avoid raising required CMake version
to 3.19, in which find_package() learned to do version check.
Changes in json2pgfplot.py:
- add mandatory argument --variant <dfa_aot | dfa_jit | nfa>
- for DFA-AOT benchmark show time and size results on the same page
- for DFA-AOT/DFA-JIT add header, drop labels on all but the 1st column
- make bars colorful
- for rows that don't fit replace three dots with a thick arrow
- automatically make DFA-JIT and NFA benchmarks relative
Also, remove some unnecessary cases from DFA JIT benchmark (they are
very similar to other cases and add little extra value).
The check makes no sense (parser is regenerated if there is bison, and
it is not related to re2c bootstrapping itself). The bootstrapping
process was changed in commit 967540566b,
and the check was likely left by mistake.
Ragel:
- Patch `/bin/bash` to `/usr/bin/env bash` to build Colm on NixOS.
- Configure Colm with `--disable-manual` to avoid Asciidoc dependency.
Kleenex:
- Use `cabal build` instead of the removed `cabal v1-sandbox`.
CI runs occasionally fail to fetch apt packages due to unavailable
remote. Let's try to cache heavyweight .deb packages and see if it
helps.
.deb files should not normally change frequently.
Configurations with the same origin TNFA state also have identical
transition tags (by TDFA(1) construction those are the lookahead tags in
the origin configuration), therefore backlinks for such configurations
are identical and can be collapsed into one (provided that indices in
other backlink arrays that point into this one are suitably adjusted).
Determinization context was needed at `regexec` time previously, when
tag sequences on TDFA transitions were fetched from the tag tree. Now
that they are stored as arrays, tag tree is no longer needed and there
is no reason to keep determinization context alive after TDFA is
constructed.
re2c test are so fast that text output in terminal takes comparable time
on slower terminals. Let's disable it by default. Can still be brought
back with --verbose option.
Otherwise there is a warning when compiling in C90 mode:
warning: commas at the end of enumerator lists are a C99-specific
feature [-Wc99-extensions]
This fixes bug #395 "commas at the end of enumerator lists".
2022-02-04 07:28:29 +00:00
4861 changed files with 1294889 additions and 1556585 deletions
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Ulya Trofimovich