This reverts commit 5cea17c8fe.
Reason for revert: enough time has passed and most distributions have
updated to re2c version 4.0 or higher.
This fixes bug #507 "Update C/C++ examples".
They mostly duplicate the "ignore" policy (except for the categories that
include surrogates). These policies are also covered by the Unicode block tests.
Tests have changed due to the changes in the Unicode standard - the old test
were generated using Haskell "Charset" library, which supports unknown version
of Blocks.txt (none of the existing version matches exactly, but Unicode 4.0 is
somewhat close). There are also other changes in the tests, mostly
simplification and cleanup of the generated code.
The new generator is based on the official Blocks.txt file from the unicode.org
website. The generated result has changed; it's unclear why some of the blocks
are missing in the Haskell "Charset" library - most likely the library was not
updated for Unicode 15.1.0 (although the categories' definitions correspond to
Unicode 15.1.0).
This partially addresses bug #235 "Unicode character classes".
Don't use it in auto generated tests, because that creates a problem with
include paths: they are relative to the test directory, which makes the
error messages different based on the build directory (so the tests fail).
Use the old generator based on haskell "Charset" library in order to avoid
changes in the generated code - the library has some discrepancies with
https://unicode.org/Public/15.1.0/ucd/Blocks.txt.
Update test results - the changes have been verified with
test/__run_unicode_tests.sh.
This partially addresses bug #235 "Unicode character classes".
The new generator is based on the official UnicodeData.txt file from the
unicode.org website. After some experimentation, I found that the the
Unicode standard supported by the previously used Haskell "Charset" library
is 15.1.0, so this is the version used in the new generator. The generated
include file unicode_categories.re is identical to the old one, which
verifies correctness of the new generator.
This partially addresses bug #235 "Unicode character classes".
Previously edge cases like 0xFF were rendered as \u0ff rather than \xff.
The difference in the generated files was verified with:
`git diff --word-diff=color --word-diff-regex='-?\\[a-zA-Z0-9]+'`
Previously even if lower and upper characters were the same, the range
would still be rendered with a dash rather than a single character.
This commit removes useless duplication.
The difference in the generated files was verified with:
`git diff --word-diff=color --word-diff-regex='-?\\[a-zA-Z0-9]+'`
Regenerate files using haskell script test/encodings/unicode_groups.hs
that is based on Data.Charset library (charser-0.3.12 package). The
script generates tests in test/unicode directory inplace and include
file unicode_categories.re.inc that needs to be copied to
include/unicode_categories.re.
Update test results. As there are substantial changes in the standard,
updates are also nontrivial. They have been verified with
test/__run_unicode_tests.sh.
The example for C was werified with examples/c/__run_all.sh scripts.
Null pointer dereference happened because the root TNFA state was null:
there were no rules for a block, but determinization still happened.
In this case re2c should emit an error and never even attempt
determinization. It was properly handled for blocks with start
conditions, but not for normal blocks.
This addresses #571 "[Bug] Segmentation Fault (NULL Dereference) in
re2c::closure_leftmost_dfs during determinization".
Note: for Haskell and monadic lexers in general I considered adding `yyend`
variable in order to allow monadic YYEND (similar to the existing `yyfill`
variable). However, this is not yet useful even for Haskell, because it's always
easy to store the end in a record field and define YYEND as a simple non-monadic
comparison expression. It won't be late to add `yyend` variable later in a
backwards compatible way (if the variable is empty, generate code the old way,
otherwise do it the new way), therefore I decided to postpone `yyend`addition
until such time when it's needed.
This is necessary in order to accommodate negative tags, which are added at the
end of the other alternative (the one that doesn't have tags). During TNFA
construction $ is moved past negative tags and placed immediately before the
final TNFA state. This ensures that there are no tags on the $-transition and we
don't have to append tag operations to incoming transitions when getting rid of
the $-transitions in TDFA.
Do it right after TDFA construction. This is only strictly necessary for
self-shadowed rules, but it's better to handle all end-of-input rules
together (especially since the previous analysis was still a hack in
the reversed-TDFA structure).
Generate YYEND before YYFILL. This order is makes sense because YYEND should be
faster than YYFILL. In fact, it is the only possible order, as YYFILL may not
return if it's impossible to supply enough input characters.
Don't generate YYEND on transition in the case of "sentinel with bounds checks"
method; instead, generate it in state. This doesn't make any difference on the
tests because YYEND is only needed in the "end" states that have a single
transition.
Prevously end-of-input symbol $ was only allowed with "sentinel with
bounds check" method (when `re2c:eof` configuration is set to a
nonnegative value). This commit removes the above restriction and
enables YYEND checks for all end-of-input handling methods. Code
generation is not quite right yet, as in some of the cases YYEND checks
are emitted after YYFILL, while they should go before it.
Internally the main challengege was adding $-transitions in TDFA for all
end-of-input handling methods, and to hide them when necessary depending
on the method. E.g. -Wunreachable-rules analysis must ignore them except
for the $-final states, unless "sentinel with bounds checks" method is
used.
For the moment, YYEND is only supported for the "bounds checks with sentinel"
end-of-input handling method (the case of `re2c:eof` configuration). In this
case states that have transitions on symbos do not need YYEND check, as they
have the sentinel transition with YYLESSTHAN / YYFILL pair.
Namely, use `YYLIMIT <= YYCURSOR` rather than `YYLIMIT == YYCURSOR`.
The goal is to bridge the gap between YYEND and YYLESSTHAN in the case
of `re2c:eof`. $-final states that have outgoing transitions defer the
end of input check until matching `re2c:eof` symbol range on transition
(and then use YYLESSTHAN in the same way as non-$-final state would do).
But $-final states that are terminal (have no outgoing transitions on
symbols) use YYEND.
We can only bridge this gap in simple / record (not generic) API, and
only if YYFILL is disabled. Otherwise it has to be YYEND and YYLESSTHAN
as macros, and the distinction remains.
This commit does not compromise correctness of the generated code, as
YYCURSOR cannot read past YYLIMIT in a correct lexer, so if it was
correct before this commit it will be as correct after it.
Also, add the corresponding syntax file configuration `code:yyend` and
syntax file conditional `.yyfill.enable`.
The new API primitive will be used in the implementation of generalized
$ (the end of input symbol). If YYFILL is disabled, and the API used is
either simple or record (not generic API), then YYEND defaults to a
comparison `YYCURSOR == YYLIMIT`, otherwise it remains a macro to be
defined by the user.
We don't use a combination of YYLESSTHAN and YYFILL instead of the new
primitive for a number of reasons:
- Using YYEND allows us to unify all codegen cases instead of having
many different implementations based on YYFILL, storable state,
generic API, etc.
- Involving YYFILL would add runtime overhead: a possible function
call instead of a simple inplace check, useless reiteration via
YYFILL label (which is currently not even generated in these states)
This change breaks backwards compatibility. This is intentional, following
a period when -Wdeprecate-eof-rule has been a warning by default, now the
default is an error (-Werror-deprecate-eof-rule).
There are code generation changes: end of input rule is now handled as a
normal rue with positional precedence / inheritance order (it is no longer
a special rule like the default rule *). The changes in the code can be
observed with -Wno-error-deprecate-eof-rule (applied to the tests).
This reverts commit b0d740994c.
Reason for revert: don't keep signed YYCTYPE examples for languages
that don't allow the user to do the incorrect thing (Go, Swift forbid
comparison of signed 8-bit type to literals that overflow it, like 0xFF;
Python, JS, OCaml don't even have a signed 8-bit type).
Use `int` instead of `byte` for `YYCTYPE`, as `byte` is signed and therefore
incapable of holding the full range of literals generated by re2c (e.g. 0xFF).
use `Byte.toUnsignedInt` in YYPEEK in order to convert correctly from signed
bytes to `yych` of type `int`. Add a test that demonstrates the need for this.
Using `char` is fine, as it is 2-byte in Java and thus capable of holding
ASCII and UTF-8 range used in the examples.
This addresses bug #561 "Incorrectness with ASCII signedness and range
(in C/C++)".
Only one example required fixing `Int` -> `Word`, others use `Word8`.
This addresses bug #561 "Incorrectness with ASCII signedness and range
(in C/C++)".
The generated code compiles only with `#[allow(overflowing_literals)]`,
and it requires `re2c:yych:conversion = 1`.
There's no need to fix examples, as they use `u8` and `u32`.
This addresses bug #561 "Incorrectness with ASCII signedness and range
(in C/C++)".
The generated code for the test does not compile - Go rejects it:
signed_yyctype.go:34:9: cannot use yyinput[yycursor] (value of type byte)
as int8 value in assignment
signed_yyctype.go:35:14: 0xFF (untyped int constant 255) overflows int8
There's no need to fix examples, as they use `byte` (which is unsigned
8-bit type) or `uint32`.
This addresses bug #561 "Incorrectness with ASCII signedness and range
(in C/C++)".
There's no need to fix examples, as they use `char` (which is unsigned
8-bit type according to https://dlang.org/spec/type.html#basic-data-types)
or `uint8_t`/`uint32_t`.
This addresses bug #561 "Incorrectness with ASCII signedness and range
(in C/C++)".
This is needed as a last resort to allow releasing re2c on a system where some
of the language toolchains are broken (as it becomes excessively hard to keep
them all working and to keep the examples up to date for languages that don't
care about backwards compatibility, such as Zig).
This addresses bug #564 "End of input ($) rule deemed invalid when Posix
captures enabled". It is needed t allow implicit capture tags after $ in a
regexp, which was inadvertently forbidden in release 4.3.
It will also be necessary in order to accommodate negative tags added at the
end of alternatives that don't have tags.
During TNFA construction $ is moved past negative tags and placed immediately
before the final TNFA state. This ensures that there are no tags on the
$-transition and we don't have to append tag operations to incoming transitions
when getting rid of the $-transitions in TDFA.
This addresses bug #560 'Illformed C/C++ code due to missing "final"
conditions in generated enumeration'.
After much discussion (see the bug) it was decided that aborting in the initial
state dispatch for conditions without rules doesn't play well with the existing
configuration `re2c:cond:abort`. Some language backends (notably, C/C++ backend)
have it disabled by default, so the general-case default (no abort) would be at
odds with the special-case default for empty conditionss (abort).
This commit takes a different approach: always raise an error on conditions
without rules, but before doing that check if there are any `<*>` rules that
can be merged to them.
This reverts commit 34f6e5269f.
Reason for revert: to make sure we don't break existing code, e.g. if
the numeric constants are incompatible with the user-defined enum type.
This check should not compare condition numbers in goto/label mode,
as conditions get renumbered anyway. Previously it used to work because
all numbers were set to zero in goto/label mode, but it was broken by
a recent commit 8a4e4ebd54.
This addresses bug #560 'Illformed C/C++ code due to missing "final"
conditions in generated enumeration'.
Check that all conditions referenced by `=>` and `:=>` exist (have rules).
In the case of `:=>`, raise an error on non-existent conditions (because
the autogenerated semantic action would go to a nonexistent condition start
label). In the case of `=>` add the nonexistent condition to condition enum
(the user may want to set condition variable and return from the lexer to
handle it in user code), but make the initial dispatch abort on it (in case
it was a typo).
This addresses bug #560 'Illformed C/C++ code due to missing "final"
conditions in generated enumeration'.
Historically in goto/label mode conditions weren't assigned explicit
numeric identifiers - the numbers were assigned implicitly based on
position in the condition enumeration. This affected language backends
for C/C++, D, Go, V (those supporting goto/label model).
In loop/switch and rec/func modes condition numbers correspond to TDFA
state indices, so they are not consecutive and require explicit
initializers.
This commit makes the enumeration always have explcit initializers
regardless of the code model. The generated initializers are backwards
compatible (identical to the implicitly assigned ones).
Historically in goto/label mode conditions aren't assigned explicit
numeric identifiers - the numbers are assigned implicitly based on
position in the condition enumeration. However, this correlation is
erroneous: there's no guarantee that the languages that support `goto`
also have enums with implicit numbers.
For now, keep the erroneous correlation (to avoid changing the generated
code), but move it from re2c code to syntax file definitions.
Previously $ was a special end-of-input rule with precedence and
inheritance order similar to the special default rule *.
Now $ can be used in a regexp as part of a normal rule with normal
position-based precedence and inheritance order. However, this change would
break backwards compatibility in some cases, so for now there's a workaround
that enforces the old order and gives a warning. In the future the warning
will be turned to error, and the old order will no longer be enforced.
New options that control this behavior:
-Wdeprecated-eof-rule (turn on warning, on by default)
-Wno-deprecated-eof-rule (turn off warning)
-Werror-deprecated-eof-rule (turn on error, off by default for now)
-Wno-error-deprecated-eof-rule (turn off error)
The case when $ is in the middle of a regexp (like `[a] $ [b]`) are forbidden
and raise an error. The case when $ is at the end but not standalone (like
`[a] $ | [b]`) are forbidden at the moment, as they are not handled properly in
codegen, but they will be allowed later. For now, only the old-style standalone
rule `$` is handled properly.
This is the first part of the work needed for issue
https://github.com/skvadrik/re2c/issues/525.
If end-of-input rule `$` is used, re2c takes a different approach and
uses a different set of subroutines to find unreachable rules.
Previously these lacked "fallthough" analysis that is normally performed
without the `$` rule. As a result, some code was generated incorrectly
(as can be seen on the updated tests and the newly added test).
Avoid popping/pushing on-stack closure items and patch them in place if
possible. Note that using `emplace` instead of `push_back` method on
`std::vector` instead of this change is not nearly as much helpful: it adds
only a ~6% speedup compared to the ~42% speedup caused by this change.
It only affects leftmost greedy closure construction (which is the default).
This addresses bug https://github.com/skvadrik/re2c/discussions/544
"Unicode characters in UTF8 and generation performance".
The 42% speedup was measured on examples/c/unicode_identifier.re modified as
suggested in the comment
https://github.com/skvadrik/re2c/discussions/544#discussioncomment-13011740.
Make closure items smaller - don't store transition tags and tag versions
in them, as they can be retrieved from the origin kernel using origin TNFA
state for each closure item. Making the items smaller reduces the amount
of copying during closure construction.
Also, avoid extra copy between the main and auxiliary closure vectors;
construct the new TDFA state kernel right from the main vector. This required
a lot of changes in the code, as the kernel stores information as a struct of
arrays (this allows fast comparison and hashing of only the relevant parts of
the kernel), while the closure is an array of structs (this is more convenient
for closure construction and sorting).
Also, make the conditional in the kenel hashing function compile-time rather
than run-time.
There are no algorithmic changes; only the way information is organized and
propagated.
This addresses bug https://github.com/skvadrik/re2c/discussions/544
"Unicode characters in UTF8 and generation performance".
The 10% speedup was measured on examples/c/unicode_identifier.re modified as
suggested in the comment
https://github.com/skvadrik/re2c/discussions/544#discussioncomment-13011740.
Replace `code:cgoto_target` (which was the target expression in the computed
`goto` statement) with `code:cgoto` (which is the full statement).
Rename `code:cgoto_init` to `code:cgoto_data`.
Add `code:yytarget_filter` (similar to existing `code:yybm_filter`).
Ulya Trofimovich