Update pre-generated benchmark files (trivial line directive changes).
Disable regeneration of Kleenex files, as Kleenex is not actively maintained and
fails to build with the recent cabal and Haskell ecosystem.
2nd attempt to fix issue #515 "version 4.0.1 re2rust default generate C code".
This time, instead of trying to split the object library, the code that needs
the split has been moved out of the problematic library.
Also there are some small improvements in the way default syntax strings are
declared (added `static constexpr`) and linked (moved them to header files).
Generation script has been fixed to generate correct string names.
This reverts commit 9cfb87edaf.
Reason for revert: it's no good trying to build multiple object libraries for
different languages in parallel, as they all try to copy files to the same
bootstrap directory, causing conflicts and build errors. Avoiding such conflicts
was the reason the object library got added in the first place (so that
autogenerated sources could be built once before all other sources).
Previously target compile definitions weren't properly added to the
autogenerated sources, as they were built into a separate object library
(to ensure that they are built before the rest of the sources). Now the
object library is built separately for each language with proper options.
This addresses issue #515 "version 4.0.1 re2rust default generate C code".
Some of the examples use C++ data structures like `std::ifstream`, `std::vector`
etc. Leave them as is for now.
Update docs.
This addresses bug #484 "Example is not C".
Keep examples compatible with older re2c versions until most of the world has
updated to re2c-4.0 or later. When it happens, this commit should be reverted.
This is to fix building with bmake on FreeBSD.
Replace all pattern rules with POSIX-compatible suffix rules. As the latter are
much less flexible than pattern rules, this required creating copies or symlinks
with appropriate names for source files that don't fit into the naming scheme
implied by suffix rules.
Also, dependencies on custom build scripts had to be dropped, as suffix rules
don't allow additional prerequisites.
This warns on `static const std::map<...>`.
re2c is single-threaded, so there's no risk of race conditions when one
thread is terminating and the other one tries to access the object.
Mostly a cospetic change. I hope to get to the state when `re2c` can be
built with `-flto -Werror=odr`. Right now it fails as:
src/parse/parser.cc:121:6: error: type 'yysymbol_kind_t' violates the C++ One Definition Rule [-Werror=odr]
This change does not fix it, but at least it clearly solits the exposed
API namespace:
https://www.gnu.org/software/bison/manual/html_node/Multiple-Parsers.html
These parts are used inconsistently in configuration names, e.g. we have
`re2c:tags:expression` which has to be scoped under `tags:` but it's also a
definition, so it should be scoped under `define:` (and other examples like
this). In fact, most of the string configurations are defines of some sort.
Also, change `conf:` prefix in syntax files to `re2c:`, as in
configurations. From now on, "c" in "re2c" stands not for C/C++ language,
but rather "code" or "compiler", so it's good for all languages.
Also, use prefix `.` (dot) for conditionals in syntax files to make it
easy to distinguish them from variables in code templates.
Also, fix warnings and error messages to use backtick quotes consistently.
Also, fix some comments in the code that still used the word "directive" for
special-purpose blocks.
Automake: generalize generation rules for docs (make split_man.py infer
language from its second argument, as POSIX make does not seem to have a way
to pass languge name to the rule). Explicitly list all example files (provide
a `find` command to generate the list in a comment).
CMake: Add forgotten install files for re2java and re2js. Use GLOB_RECURSIVE
with CONFIGURE_DEPENDS to add examples to the list of source file.
Also, update description of submatch extraction and change captures example:
use the one without POSIX-style array `yypmatch`, as not all languages support
it (Haskell doesn't, as it has no mutable arrays in the standard library).
Remove syntax configurations `code:bit_and`, `implicit_bool_conversion`
as they are no longer needed. Strictly speaking, `code:bit_and` was also
used in generation of computed goto, but it's fine to hardcode `&` for
now, as computed goto are only implemented in C/C+. Later they will be
reimplemented using a syntax configuration similar to the bitmap ones.
This unifies bitmap transition with other transitions that are constructed
during the analyze phase and removes the need for special handling during
the generate phase.
This commit splits table initialization (populating it with values) and code
generation for the initialized table. The former is moved to analyze phase, and
the latter remains in the generate phase.
Bitmap values are needed when generating preconditions for bitmap transitions.
Currently this is done during generate phase, but there is a plan to move it to
analyse phase so that bitmap transitions could be generated using the same code
primitives as other transitions (linear IF).
Some functional languages like OCaml require that IF statement has an ELSE part.
This requirement may be relaxed if the statement under IF returns void, but this
is not necessarily the case in re2c-generated code (it depends on how the user
defines YYFN: they may choose it to be a void or non-void function). Besides,
other functional languages like Haskell do not have this relaxation.
The added test works correctly both before and after this change, however
it shows incorrect behavior if tags are not checked for equivalence at all
(in that case some URI fragments are missing in the test output).
These directives allow the user to simplify declaration of tags variables,
which is especially useful when the variables are for capturing paretheses.
Also, adjust commnets in submatch examples.
Previously it was recomputed from the numeric value every time. Although
the recomputation was trivial (comparing against zero), it made the code
more verbose and less readable.
This commit adds the following options and the corresponding configurations:
--captures (alias for existing --leftmost-captures)
--leftmost-captvars, --captvars (alias for --leftmost-captvars)
--posix-captvars
Variables are named `yytl<n>` and `yytr<n>` for the left and right
parentheses of n-th capturing group.
Warnings like -Wundefined-control-flow should be on by default, as they report
severe issues with the code that should be fixed. If the user wants to suppress
such warnings, they will have to do it explicitly on the command-line.
The only purpose of this configuration was to make the lexer fail on standalone
single quotes for languages that do not allow them. Removing it does not affect
how the lexer works: it still attempts to lex single quote as the beginning of a
char literal and looks for a matching closing quote. The only thing changed is
that now the lexer does not raise an error on standalone single quotes for
languages that do not allow them; it simply ignores them in the same way it does
for languages that do allow them (e.g. identifiers in Haskell/OCaml, numbers in
C++, labels/lifetimes in Rust, etc.). This goes along with the general re2c
behaviour: it does not attempt to parse any languge - it only does the bare
minimum of work needed to parse re2c blocks correctly.
This takes care of bug #482 "Handle single quotes correctly in the lexer".
The lexer does not attempt to lex all lexemes that may contain a standalone
single quote - those would differ a lot for different languages. Instead, it
tries to lex paired single quotes (character literals) which are similar enough
in different languages to allow a single generic lexer rule. If this attempt
fails, lexer checks if standalone single quotes are allowed (if yes, it just
skips the single quote and resumes lexing; if not, it fails with a syntax
error). This way lexer doesn't need to know about different types of lexemes
that may contain a standalone single quote.
This fixes a problem with libre2c introduced by the previous commit
8e9bfda737 which required that all code options
are defined (non-null). The problem was caused by testing if `line_dirs` option
is defined: the test happened in the code that is shared with libre2c, while
normally such tests happen in the code that is not shared. This commit moves
`line_dirs` initialization to config files, so the faulty test is removed
entirely.
It also changes the way that implied imuutable options are set: former
`set_default_*` functions that soft-set an option (mark it as not user-defined)
are replaced with `set_*` functions that hard-set the option (mark it as
user-defined). It shouldn't matter for immutable options, as they cannot be
redefined by the user.
an option only set an option specified in a syntax
file if it's not explicitly defined by the user. No tests have changed.
This commit adds a new warning: -Wundefined-syntax-config. It is not an
error, so that old syntax files stay valid as new syntax configurations
are added. Warnings allow the user notice that new configurations are
available and it's time to update their syntax file.
Not all languages support `yych:conversion` (this commit adds support
for it in C/C++, D, Go, Rust and Zig backends). Conversion is only
useful for backends that implement simple/record API and have a type
system strict enough to prevent implicit conversion.
Don't adjust file start/end pointers in cases when we reached the end of
the include file when reading from it, as YYLIMIT may point outside of
the portion of the input buffer occupied by this file. It may happen if
the previous `read()` invocation drained this file, then read some data
from the parent file (the one that contains include directive), and then
YYFILL got triggered while YYCURSOR was still in this file, so this file
could not be popped off the stack yet.
The error was rare - I found it by a lucky accident when the buffer got
resized to a certain size due to a change in syntax file. This triggered
YYFILL in the right moment (in encodings/unicode_identifier.re example).
This way the user may provide a syntax file with only a few overrides
instead of copy-pasting default definitions for all configurations.
As an example, add a test for the Haskell backend that pretends to be a
test for the Python backend, but overrides default Python config with
Haskell config. We don't use default C config in this case as it
defines a few optional configurations for oneline switch cases/ifs that
are not overridden by the Haskell config. The generated code is
identical to examples/haskell/01_basic.re.
Also, remove a stale TODO comment.
This should be configurable, as in some languages type system is strict
and defaulting to any particular constant means that tags must have the
type of this constant (and by extension cursor, marker, ctxmarker and
limit). It would still be possible to override them, but it would
require using generic API and providing definitions for all primitives.
In generic API `tags:negative` defaults to the tag variable that is set
to the negative tag value (substituited by re2c). We keep this behaviour
for backwards compatibility.
Also, slightly change parser rules for `%{`: instead of requiring that
it is on a new line (possibly preceded by whitespace), allow arbitrary
text before it on the same line, but require a newline immediately after
it. This simplifies parser rules, and it should be good enough to avoid
parsing `%{` occurrences in the middle of a string or comment by mistake
in most of the cases.
Previously their names were hardcoded in syntax files, now they delegate to
configurations `define:YYCURSOR`, `define:YYMARKER`, `define:YYLIMIT` and
`define:YYCTXMARKER`. This results in slightly longer field names, but also
more consistent with default and generic APIs.
In some cases it may be necessary to override the default names, e.g. Go
requires public struct fields to be uppercase, while private ones should be
lowercase.
Previously their names were hardcoded in syntax files, now they delegate to
configurations `variable:yyaccept`, `variable:yycond` and `variable:yystate`.
With the new record API defined via syntax files, it may actually be useful to
delegate some `code:` syntax configurations to generic API primitives. It allows
one to customize the generated code for record API by redefining only a few
selected primitives.
Add a new option `--api record` and a new configuration `re2c:api = record`.
Use the new record API in syntax file for C. Switch some C examples to it.
Previously API definitions like YYPEEK, YYSKIP etc. were always strings, both in
syntax files and in source files. Now in syntax files they are code templates;
in particular, they should use variable `sigil` instead of substring `@@`. In
source files they are still strings: there's no need for code templates there,
and if fact it's not possible for them to become code templates, as source files
allow unquoted strings which would be mis-parsed as variables in code templates.
Using code templates in syntax files allows one to make conditional language-
specific defaults for APIs depending on mutable options, such as the current
API, API style, etc. Previously it was only possible to use immutable options as
predicates. Using variable substitution for `sigil` allows us to automatically
replace default sigil `@@` with a user-defined one.
Previously sustitutions were always done in the free-form API style,
while the function style was inconsistent: for most primitives
substitutions weren't done, but for a few primitives they were done.
This doesn't matter in real life, as the function style assumes that the
primitives are function names, which cannot contain the sigil string
"@@" (unless it was redefined to some alphanumeric string by the user,
which is unlikely).
Previously it was used to specify optional modifiers. Syntax configs for
some languages utilized modifiers for mutability annotations (Rust) or
strictness annotations (Haskell). More generally, it was used in cases
when the formal parameter syntax differs from the argument syntax.
However, modifiers were not flexible enough: they can only be added on
one side, while sometimes both left and right context is needed.
This patch makes the 3rd component mean the whole formal parameter. It's
still optional: when not specified, 3rd component defaults to the 1st
component (argument). However, if parameter syntax differs from the
argument syntax, it is now possible to specify arbitrary text in each.
Instead of always emitting error prefix `re2c:` emit `re2go:` for
re2go, `re2rust:` for re2rust, etc. Note that this doesn't affect the
runtime --lang option, e.g. for `re2c --lang go` the error still starts
with `re2c:`.
No tests, as the testing script replaces all binary names with --lang.
Provide a less verbose interface for generating language-specific expressions
during "generate" codegen pass. Encapsulate some of the details in helper
methods of the `opt_t` class.
The new primitives and configurations allow one to override `yyaccept`
syntax for a single source file or for a given language backend.
For existing languages the new primitives have default definitions:
nothing changes in the code generation, and no new definitions are
required from the user. The change is fully backwards compatible.
Namely, remove examples that use end-of-input rule `$`, and leave
examples that are based on the sentinel method (as they are simpler and
demonstrate the technique well enough.
This condition is true when there's more than one IF branch. It is
needed for Haskell, which uses a different syntax for one-branch IF
(the `when` statement).
This is needed because the check itself is language-specific, and also
the IF construct is problematic in some languages (e.g. in Haskell it
would be impossible to get the updated state out of the IF branch
without making the IF expression part of the YYSHIFTSTAG operation.
The following configurations have been added:
- code:shiftstag
- code:shiftmtag
- code:stagp
- code:stagn
- code:mtagp
- code:mtagn
- code:copystag
- code:copymtag
Mark `code:shiftstag` and `code:shiftmtag` as oneline configurations
(in fact they should be named one-statement, not oneline).
Note that the user can define YYSHIFTSTAG/YYSHIFTMTAG as multi-statement
without the curly braces, and thus break the assumption re2c makes about
the generated code. But it is also the case with YYFILL (and has always
been), so we don't make the situation much worse. This requirement
should be documented, though.
Previously default API had some shortcuts, e.g. in case of fixed tags
the operations of copying base tag and subtracting offset were fused
into one. This is no longer the case.
Replace the complex `code:yyrestorectx` configuration that had many
conditionals with a few simple configurations `code:yyrestorectx`,
`code:yyrestoretag`, `code:yyshift`. Common language-independent logic
belongs in the re2c source code (to avoid duplicating it in each syntax
file, and also to avoid unecessarily complicating syntax files).
Previously re2c generated assignment chains like this:
t1 = t2 = ... = tN = YYCURSOR;
Now it generates a sequence of simple statements:
t1 = YYCURSOR;
t2 = YYCURSOR;
...
tN = YYCURSOR;
Syntax configuration `code:assign` has been updated accordingly:
the `lhs` variable no longer needs to be a list variable.
For most languages they have default values (in syntax files) that never
need to be configured manually by the user. However, for languages like
haskell it is impossible to provide a reasonable default value - that's
why we need these primitives.
This reverts commit 8fb3d50f8d6dd8aace87631bb1a961b6e13c4536.
Reason for revert: change of plans: no need to rename these primitives
unless we add YYGET{S,M}TAG, and the latter can be avoided if we use the
3rd component of YYFN elements for arguments (as opposed to parameters).
E.g. if some API primitive X is defined as "...@@..." in syntax file,
and the user specifies sigil "$" instead of "@@", then default X
definition should become "...$...".
Previously this was only done for `re2c:cond:goto`, as no API primitives
have default definitions that contain sigil. But we don't really know
default API definitions, as the user may provide a custom syntax file
with custom definitions.
Error on disabling unsupported features (previously re2c only errored
on enabling them).
Add `unsafe` and `monadic` features and tests for them.
Update tests for existing unsupported features.
The new configurations are needed to support monadic lexers in the
upcoming Haskell backend.
Configuration `monadic` affects control flow constructs: when it is on,
the "do" keyword is generated before nested blocks.
Configuration `variable:yyfill" allows one to decouple YYFILL invocation
from the check whether YYFILL has failed. This is needed becausse YYFILL
performs I/O (like reading from file), so it should return the updated
state of the world together with the status whether it has failed or not.
The new option cotrols whether to strip curly braces from semantic
actions, and whether to wrap the whole program in curly braces.
Previously this was deduced from other options, e.g. from
`indentation_sensitive` in Python. However, in some other languages this
cannot be so easily deduded (e.g. Vlang uses curly bracees in constructs
like `if`, `switch`, etc., but it rejects a compound statement wrapped
in curly braces, unless it's separated from the previous statement with
a semicolon).
Replace many copies of examples for each language/codegen model
combination with just two tests for all auxiliary code models: a basic
one and an advanced one that covers storable states, conditions, tags,
end-of-input rule $ and `yyaccept`. Other tests are left only for C/C++
backend, as it is sufficient to have them for one backend (e.g. to test
errors specific to this codegen model).
The rationale for this change is: when new backends are added to re2c,
the amount of tests grows significantly, slowing down test run time for
no good purpose (most of the new backends are experimental). Also, it
slows down the addition of new languages (e.g. Vlang supports all three
codegen models --- goto/label, loop/switch and rec/func --- but adding
three full sets of examples for it is an overkill).
Also, remove semicolon from `code:abort` definition in Rust syntax file,
so that it can be used in one-line switch-case context.
Update test results accordingly.
It may be used in syntax file - but not necesarily. C, D and Go backends
don't use it. Rust, on the other hand, requires that all variables are
initialized.
Previously there was a hardcoded 0 constant in the syntax file for Rust.
Now the initializer is 0 or '\x00' depending on `variable:yych:literals`
configuration and the current encoding (the same condition which is used
to decide how to print YYCTYPE literals).
They are usually faster than switch cases on the tests, e.g. on the
newly added test/switch_vs_ifs.re
nested ifs took 0.17752790451049805
switch cases took 0.6691319942474365
As this case is trivial, it's easier to copy-paste the following case
than to add a portable fallthrough annotation (I'm not aware of an easy
way to do it in C++11).
Put syntax configurations in a struct and make it part of the immutable
options instead of storing them in a map. This simplifies their handling
and access, as there's no need to check if a key is in the map, etc.
Merge syntax file lexer/parser into configuration lexer/parser, reuse
existing methods.
Rename list syntax configurations and define them as lists of strings in
quotes (rather than words) so that we can reuse configuration parser for
lists.
Remove separate "features" syntax configurations and replace them with a
single list of features (to be augmented in future commits).
Referencing another `code:*` configuration in the middle of a definition
is not allowed (that would be hard to implement, as appending/prepending
to `StxCodes` list when using it in another configuration would alter
the original list). However, referencing another configuration as an
entire definition is ok, as it doesn't alter the existing configuration.
OCaml supports character ranges like 'a'..'z', but it does not support
numeric ranges like 0..9, therefore previously all ranges were rendered
in expanded form, like 'a'|'b'|'c'|...|'z'. This results in sometimes
very long lines with a lot of alternatives in match expressions.
This commit fixes it by adding a conditional into syntax file
definition. It checks if the range is of char type. If so, then the
range is rendered compactly.
One exception is the case of multibyte encoding: in that case
YYCTYPE literals are rendered as numbers (because in many languages
char type cannot hold values more than 1-byte).
Replace syntax configuration `yyctype_literals` with
`variable:yych:literals` that may have values `char`, `hex` and
`char_or_hex`.
Don't give a syntax error on backticks if the language does not support
backtick-quoted strings. Instead, simply skip them and move on.
Incorrect behaviour was introduced in commit
624aa6d72059e6180bab7298128e86b5f337e07c.
This is a very unlikely and useless case (trailing context that is
always empty), it is not worth specializing the code. In such case we
might as well generate a no-op operatio that decrements cursor by zero.
(Previously the operation was omitted, but an empty statement was still
generated.)
This is safe because each semantic action is placed into a separate
function, so there won't be any name collisions with local variables.
It is needed for language backends that don't use curly braces for
compound statements (like OCaml).
Add `have_retval` conditional to `code:fncall`/`code:tailcall` and
`retval` variable to `code:fncall` only (it is not needed in
`code:tailcall` as the latter returns immediately).
Also, change `code:fncall` to be a statement, like `code:tailcall`
(generate `retval = ...` if there is a return value).
This was problematic because previously newline was always rendered in
oneline statements (when rendering the statement itself, without any
knowledge of the surrounding context).
Now newline is no longer considered part of the oneline statement: if
needed, it should be added in the surrounding syntactic construct (in a
syntax file).
In some languages type modifiers go on the left of function parameters,
while the type itself is on the right (like `mut n: usize` in Rust). For
such cases, we need to allow optional third component of an argument
descriptor in YYFN. So, change its format from a list of 2*N strings to
a list of N strings with semicolon-separated parts (name, type, optional
modifiers).
Add `fn:sep` configuration to allow the users configure the separator.
Precompute function parameters and arguments, as they are common for all
function definitions / calls.
Move individual configuration initialization from re2c code to syntax
configs (previously it ws done in a post-initialization fixup step, but
now there's no need for it).
Previously empty header filename was concatenated with directory path,
resulting into incorrect header filepath. Change this to only prepend
directory path if the header filename is nonempty.
Previously this was done by comparing filepaths, however it broke on
windows as one of the filepaths was escaped (and on windows filepaths
have backslashes, so the escaped name differed from the original).
The same lexer state is used for all lexer files, so YYMAXFILL must be
the maximum of the individual YYMAXFILL values for each file. Previously
YYMAXFILL value for the main lexer was used everywhere, but this is
incorrect and may cause unexpected end of input errors on other lexers.
Previously all configurations were handled by the lexer, but the
grammar for conditionals is more complex (it allows nested conditional
expressions). These will be used in syntax files to set configuration
defaults for a language based on the global options.
One part of the old lexer still remains: the lexer for string
configurations in source files. Unlike syntax files, source files may
have raw (unquoted) string configurations, which may contain special
symbols used in other parts of the grammar (e.g in the new syntax for
conditionals). Switching to the new parser would require excluding these
symbols and thus breaking backwards compatibility.
The added configurations are named in the same way as in source files,
except that they use prefix `conf:` rather than `re2c:`. E.g. for a
source file configuration `re2c:state:abort` the corresponding syntax
file configuration is is a `conf:state:abort`.
Check that tags and YYSKIP are the same as on transition to which the
control flow falls through if the fallback transition is elided.
(Previously the code assumed that tags and YYSKIP are the same, but it
is not obvious why it should be so.)
Fallback transition is part of an IF/THEN/ELSE statement. In rec/func
mode both THEN and ELSE must end in a tailcall, therefore elisiion is
forbidden (except if YYFILL is disabled, then the whole IF/THEN/ELSE can
be replaced with a tailcall to to-state).
Normally final tags are user-defined local variables in the function
that wraps the generated DFA. However, in rec/func mode each DFA state
is a separate autogenerated function, so there's no place for the user
to define local variables. Therefore in rec/func mode final tags are
part of the state that is passed from one function to another.
We could theoretically autogenerate final tag declarations before
setting them in the final states, so that they are still local
variables. This would require adding YYSTAGTYPE / YYMTAGTYPE. But what's
worse, it would't work for languages like OCaml, where variables are
immutable bindings: fixed tags with generic API require a few operations
(set, then shift). If, however, final tags are part of the state struct,
they can be updated.
It is possible to add support for skeleton in rec/func mode, but it will
take some effort as skeleton programs use a lot of local variables that
must be passed as function parameters. Disable it for now.
If `getstate:re2c` is global (it has no explicit block list, assuming
that it covers all blocks in a file) then use global options accumulated
over the whole file. Otherwise use options of the first block listed in
the directive. For block-level labels/function always use block options.
The following configurations have been added:
`code:skip`
`code:yybackup`
`code:yyskip_peek`
`code:yypeek_skip`
`code:yyskip_backup`
`code:yybackup_skip`
`code:yybackup_peek`
`code:yyskip_backup_peek`
`code:yybackup_peek_skip`
Also, tweak syntax configuration `code:recursive_functions` to assume
toplevel functions rather than closures (as most languages do not allow
co-recursive closures, and some don't have closures at all).
In functional languages both branches must have the same type, and
C-style early returns are not possible.
In goto/label and loop/switch mode early returns are still preferred,
because the "else" part is often elided.
This is because in functional languages like OCaml "if/else-if/else" is
an expression where all branches must have the same type (except perhaps
for a single "if" branch that may have void type.
One exception is the case of a single branch that goes to the next state
(in that case "if/else-if/else" construct is not needed).
This is part of a general effort to unify rendering of oneline if/else
statements (either render them all on the same line without braces, or
all on a new line with braces).
Users that prefer always rendering if/else statements in braces should
be able to do so by commenting out `code:if_then_else_oneline`
configuration in the syntax file (`--syntax` option).
Note that default syntax file include/syntax/<language> (the one that is
used without `--syntax` option) is built into re2c; changing it requires
rebuilding re2c with configure option `--enable-syntax` or CMake option
`-DRE2C_REBUILD_SYNTAX=on`.
This is part of a general effort to unify rendering of oneline if/else
statements (either render them all on the same line without braces, or
all on a new line with braces).
Users that prefer always rendering if/else statements in braces should
be able to do so by commenting out `code:if_then_else_oneline`
configuration in the syntax file (`--syntax` option).
Note that default syntax file include/syntax/<language> (the one that is
used without `--syntax` option) is built into re2c; changing it requires
rebuilding re2c with configure option `--enable-syntax` or CMake option
`-DRE2C_REBUILD_SYNTAX=on`.
(It is not used anyway, as there is always a multi-line "else" branch
that makes the whole "if-then-else" construct render as multiline.)
This is part of a general effort to unify rendering of oneline if/else
statements (either render them all on the same line without braces, or
all on a new line with braces).
Users that prefer always rendering if/else statements in braces should
be able to do so by commenting out `code:if_then_else_oneline`
configuration in the syntax file (`--syntax` option).
Note that default syntax file include/syntax/<language> (the one that is
used without `--syntax` option) is built into re2c; changing it requires
rebuilding re2c with configure option `--enable-syntax` or CMake option
`-DRE2C_REBUILD_SYNTAX=on`.
This is part of a general effort to unify rendering of oneline if/else
statements (either render them all on the same line without braces, or
all on a new line with braces).
Users that prefer always rendering if/else statements in braces should
be able to do so by commenting out `code:if_then_else_oneline`
configuration in the syntax file (`--syntax` option).
Note that default syntax file include/syntax/<language> (the one that is
used without `--syntax` option) is built into re2c; changing it requires
rebuilding re2c with configure option `--enable-syntax` or CMake option
`-DRE2C_REBUILD_SYNTAX=on`.
This is part of a general effort to unify rendering of oneline if/else
statements (either render them all on the same line without braces, or
all on a new line with braces).
Users that prefer always rendering if/else statements in braces should
be able to do so by commenting out `code:if_then_else_oneline`
configuration in the syntax file (`--syntax` option).
Note that default syntax file include/syntax/<language> (the one that is
used without `--syntax` option) is built into re2c; changing it requires
rebuilding re2c with configure option `--enable-syntax` or CMake option
`-DRE2C_REBUILD_SYNTAX=on`.
This is part of a general effort to unify rendering of oneline if/else
statements (either render them all on the same line without braces, or
all on a new line with braces).
Users that prefer always rendering if/else statements in braces should
be able to do so by commenting out `code:if_then_else_oneline`
configuration in the syntax file (`--syntax` option).
Note that default syntax file include/syntax/<language> (the one that is
used without `--syntax` option) is built into re2c; changing it requires
rebuilding re2c with configure option `--enable-syntax` or CMake option
`-DRE2C_REBUILD_SYNTAX=on`.
These are currently used only for skeleton backend (which supports only
C/C++), but later they will be used for rec/func codegen model.
Split codegen for function definitions and function calls. Make it
simpler (don't try to format differently based on the length of
arguments, just output all the arguments on the same line).
Some function calls in skeleton programs are syntactically too complex
(they are part of OR expressions, or they have multiline arguments) ---
for such function calls do not use syntax configurations and fallback to
generating text. This works because skeleton supports only one language
backend.
If there is a need for more complex syntax constructs that contain
function calls in the future, it will be possible to add them as
separate syntax configurations (for that purpose `code:fncall` is an
expression, not a statement).
Pass `yych` instead of YYPEEK as the second argument to YYDEBUG.
Also, move YYDEBUG invocation in the initial state past YYPEEK so that
`yych` is properly set to the first symbol.
The motivation for this change is to decouple codegen for YYPEEK and
YYDEBUG statements and avoid having two syntax configurations for YYPEEK
(one for the expression that was previously used in YYDEBUG, and one for
the statement). Also, this helps in cases where YYPEEK modifies internal
lexer state, e.g. reads a character from file and moves file pointer.
Replace syntax configuration `abort_requires_include` with
`abort_in_default_case`. Enable it for Go and Rust, but not for C/C++ as
it requires an `#include` directive that must be provided by the user.
Update Go tests. Rust tests do not need updates because they use a
different codegen model (loop/switch) that has no initial switch on
conditions.
Instead, provide default values:
- empty string for string-valued configurations
- false for boolean-valued ones
- no output for `code:*` configurations
Also, fix an error in libre2c initialization: it now has to manually set
the target option, as it does not load syntax file (and so the options
are not properly initialized).
This is the first example of a single string valued configuration.
Leave it in the "code" group for now (`code:...`) but add a special
`Stx`method that returns an error if the value is not a single string.
Make the default value empty string, as we don't want to specify such
configurations in syntax files for languages that don't need them.
Options are passed everywhere by const pointer, as they should not be
modified anywhere in the program once they have been set for a block.
Syntax configurations also should be immutable, with the exception of
mutable stacks that are used when evaluating immutable configurations.
This was necessary to fix https://github.com/skvadrik/re2c/issues/201,
where line directives suppression was configured using configuration
`re2c:flags:no-debug-info` (the problem was that it didn't affect the
first block, and the first line directive was generated contrary to the
user's intent).
Now this workaround is no longer needed as the configuration
`re2c:flags:no-debug-info` was deprecated in re2c-2.2 and later removed.
Dropping this allows us to get rid of default copy semantics for `opt_t`
which is necessary in preparation to store `Stx` reference there
(the compiler won't generate default defintions in that case).
Some `Stx::*` methods are now called from src/options.opt.cc, but the
source file that defines them (src/options/syntax.cc) weren't added to
the list of library sources.
The fix is to make these methods inline, as they are small and the other
alternative (to include src/options/syntax.cc) would pull in even more
unrelated source files for the definition of DEFAULT_SYNTAX_{C,GO,RUST}.
Conditionals in syntax files were based on options, which caused a
dependency cycle: options required syntax configurations to initialize
default values, and evaluating syntax configurations required options.
Now syntax configurations do not depend on options.
The only conditional was in `char_literals` (a test for EBCDIC).
It can be as well hardcoded in the re2c source code, as there is nothing
language-specific in it.
The next step will be to simplify non-code syntax configurations and
allow only single word values or word lists (no conditionals).
Commit 1d9319548bc97313426499ad7b7aa6dfdfe71ac2 added a fix-up pass that
removes line directives (if they are not needed). It turns out that this
is unnecessary, as line directives cannot be enabled/disabled with a
configuration --- only with a command-line option. This option is
available early at program start, so there's no need for a clean-up pass
as we can test the option before adding line directives.
Line directives are generated by default for C and Go, but not for Rust
(which does not support them at all). They can be turned off/on with a
command-line option or configuration. From the codegen perspective,
there are multiple ways to avoid generating line directives:
1) do not add them in the first place
2) remove them during fix-up phase
3) render them as empty during rendering phase
1 is the most efficient, but it canot be used with configuration, as the
latter may be parsed later than the directive must be generated (some of
them are generated early during parsing, at which point the block
options are not known yet).
3 is the easiest to implement, but it confuses the checks for oneline
statements and results in ugly multiline formatting and lots of trivial
test updates.
Previously we used a combination of 1 and 3: for Rust directives were
not generated (language is an immutable option which is known at program
start), and for other languages directives were generated but not
rendered.
This commit changes the behaviour to 2 and makes it uniform for all
languages. C tests for which re2c is now able to properly detect oneline
statements have been updated.
It worked without a semicolon as well because panic was the last
statement in a block. However, this was unintentional and it may change
in the future.
Rust has an unusual syntax for labels: they have to start with a single
quote. But it is inconvenient to encode the quote in a syntax config ---
it's easier to encode it in the name, since the only kind of label used
in Rust is the yyloop label (which has a string name that is configured
with `re2c:label:yyloop` that can easily encode the quote as well).
Reuse code item for `code:var` configuration.
Change Go codegen to assume that YYMAXFILL and YYMAXNMATCH are unsigned
and update test results accordingly. (Other codegens already assume that
YYMAXFILL and YYMAXNMATCH are unsigned).
Namely:
- add `have_init` conditional instead of using `jumps.loop_switch`
- add `type` variable instead of using `storable_state` conditional
and `code:type_cond_enum` configuration for the underlying type
These changes make `code:enum` more generic and usable not only for the
specific case of condition enum, but for any enum (although it's not
needed at the moment, as condition enum is the only use).
Remove obsolete `GenEnum` class.
Don't add `static const` type qualifiers in the C syntax file. Instead,
add them as part of the type specified in the re2c source code. This
will be fixed later, as the type should not be hardcoded based on the
language (it will become a config in the syntax file).
Prepare for rendering tables using syntax configuration.
Update test results. There is a lot of changes, but they are trivial:
- drop optional comma at the end of array initializer for C
- change formatting: `void *` to `void* `
- add explicit array size in array definitions
Update syntax files for C/Go/Rust accordingly. Remove unnecessary
syntax configuration `code:yyctype`, as it is already configurable
via `re2c:define:YYCTYPE`.
This was mean to be used with `code_loop` for languages that have a
label preceding a loop, so that re2c would know whether to indent the
label or not (the default assumption being to indent everything).
However, now that we use explicit `topindent` variable everywhere,
there's no need for `label_indent`.
This is so that syntax files for languages that don't need special
handling for oneline statements (like Go) do not have to handle
`oneline` conditional (which may be confusing and unexpected).
List bounds need to be evaluated at codegen time (not earlier), because
the indices specified in syntax file may be negative and must be
converted to non-negative (the size of a list varies and is not known
until codegen time). To avoid duplicating the same code for every list
handler, this commit moves it to the generic function that renders
syntax trees.
Also, add `eval_cond` method to evaluate conditionals in syntax files
and fix the generic `eval_cond` method to respect configuration-local
conditionals.
Previously re2c was built with syntax file for C, re2go with that for Go
and re2rust with that for Rust. However each binary needs all three
configs, as the language may be changed dynamically with --lang option.
It is the first codegen function that is based on syntax configuration.
The chosen configuration is generic enough: it has lists with specific
bounds, conditionals, global and local variables.
Previously it was parsed as part of complex expressions, which should
not be allowed by the grammar as it makes it too lax and introduces
meaningless constructs.
Also, remove parenthesized expressions other than conditionals, as they
aere not used (they also caused LR conflicts in grammar).
This simplifies the find_package() call. The version in find_package()
has been initially supported in CMake 2 and 3. Version ranges
(versionMin...[<]versionMax), on the other hand, are only supported by
CMake 3.19+.
And this also helps a bit when adding re2c as a FetchContent dependency
in CMake project where finding Python is not necessary and can be
disabled with CMAKE_DISABLE_FIND_PACKAGE_Python3 variable.
This addresses bug https://github.com/skvadrik/re2c/issues/471
"Examples in re2rust manual have undefined behavior".
In Rust, it is UB to call `get_unchecked` on a slice that may be empty,
and many examples had functions that did just that (when invoked on
incorrect input data, since they use some form of sentinel method to
handle the end of input, which relies on the input being sentinel-
terminated).
The introductory example has been fixed to avoid UB and use simple
indexing with bounds checks instead of `get_unchecked`. Other examples
that had UB are now annotated with `assert` that checks conditions under
which the functions are safe to invoke.
Also, eof/02_bounds_checks.re example now uses a more efficient way to
construct YYMAXFILL-padded input string.
It loks like at least in `windows` `actions/upload-artifact@v4` CI step
fails to collect files at `${{ steps.build-info.outputs.BUILD_DIR }}/test_[0-9]*`
My theory is that `glob` is not able to match paths like
`D:/a/re2c/re2c/install-2`
against theit unix-like form of
`/d/a/re2c/re2c/install-2`
I suspect that `pwd` returns us the second form while
`${{ github.workspace }}` might keep an original form.
Lex char literals properly as single characters or escape sequences
rather than lexing them as single-quoted strings. Add syntax rules from
C/C++, Go and Rust (they are similar and do not contradict).
Add a Go test for string and character literals (prior to this patch it
caused a syntax error).
Note that the same lexer method is used for indented code blocks and
code blocks in curly braces.
This is a revert of commit 77714fce94.
While it is easier to always generate braces, in some languages (Go) it
is a compilation error to jump into the wrapped block (a scenario that
happens if there is an outer state switch generated with `getstate:re2c`
directive.
This regression was found because one of the examples for Go didn't
compile (examples/state/push.go). Now it does.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Ulya Trofimovich