3 Commits
Author SHA1 Message Date
Eric S. Raymond 585889745c Clean up the indentation and brace usage in the skeleton file.
It was in a mix of several different styles tht made it hard
to read.  I've massaged it all ibto K&R with tabs and mandatory
braces.  No logic changes.
2020-09-27 14:26:29 -04:00
Will Estes f8fd9731d4 build: set dependency info involving cpp-skel.h 2020-09-26 20:09:29 -04:00
Eric S. Raymond 695cab8a1e Fold documentation on how to write bac ends into the Texinfo manual 2020-09-26 20:08:02 -04:00
4 changed files with 1217 additions and 1280 deletions
+118 -9
View File
@@ -192,7 +192,7 @@ FAQ
* How can I build a two-pass scanner?::
* How do I match any string not matched in the preceding rules?::
* I am trying to port code from AT&T lex that uses yysptr and yysbuf.::
* Is there a way to make flex treat NULL like a regular character?::
* Is there a way to make flex treat NUL like a regular character?::
* Whenever flex can not match the input it says "flex scanner jammed".::
* Why doesn't flex have non-greedy operators like perl does?::
* Memory leak - 16386 bytes allocated by malloc.::
@@ -269,6 +269,7 @@ Appendices
* Bison Bridge::
* M4 Dependency::
* Common Patterns::
* Adding More Target Languages
Indices
@@ -734,7 +735,7 @@ if X is @samp{a}, @samp{b}, @samp{f}, @samp{n}, @samp{r}, @samp{t}, or
@samp{v}, then the ANSI-C interpretation of @samp{\x}. Otherwise, a
literal @samp{X} (used to escape operators such as @samp{*})
@cindex NULL character in patterns, syntax of
@cindex NUL character in patterns, syntax of
@item \0
a NUL character (ASCII code 0)
@@ -5148,10 +5149,10 @@ the version of flex that was used to create the serialized tables.
@item th_name[]
Contains the name of this table set. The default is @samp{yytables},
and is prefixed accordingly, e.g., @samp{footables}. Must be NULL-terminated.
and is prefixed accordingly, e.g., @samp{footables}. Must be NUL-terminated.
@item th_pad64[]
Zero or more NULL bytes, padding the entire header to the next 64-bit boundary
Zero or more NUL bytes, padding the entire header to the next 64-bit boundary
as calculated from the beginning of the header.
@end table
@@ -5239,7 +5240,7 @@ The table data. This array may be a one- or two-dimensional array, of type
@code{td_flags}, @code{td_hilen}, and @code{td_lolen} fields.
@item td_pad64[]
Zero or more NULL bytes, padding the entire table to the next 64-bit boundary as
Zero or more NUL bytes, padding the entire table to the next 64-bit boundary as
calculated from the beginning of this table.
@end table
@@ -5447,7 +5448,7 @@ publish them here.
* How can I build a two-pass scanner?::
* How do I match any string not matched in the preceding rules?::
* I am trying to port code from AT&T lex that uses yysptr and yysbuf.::
* Is there a way to make flex treat NULL like a regular character?::
* Is there a way to make flex treat NUL like a regular character?::
* Whenever flex can not match the input it says "flex scanner jammed".::
* Why doesn't flex have non-greedy operators like perl does?::
* Memory leak - 16386 bytes allocated by malloc.::
@@ -6060,8 +6061,8 @@ what they're doing, and then replace @code{input()} with an appropriate definiti
@code{YY_INPUT}. You shouldn't need to (and must not) replace
@code{flex}'s @code{unput()} function.
@node Is there a way to make flex treat NULL like a regular character?
@unnumberedsec Is there a way to make flex treat NULL like a regular character?
@node Is there a way to make flex treat NUL like a regular character?
@unnumberedsec Is there a way to make flex treat NUL like a regular character?
Yes, @samp{\0} and @samp{\x00} should both do the trick. Perhaps you have an ancient
version of @code{flex}. The latest release is version @value{VERSION}.
@@ -8167,6 +8168,7 @@ See @ref{Top, , , bison, the GNU Bison Manual}.
* Bison Bridge::
* M4 Dependency::
* Common Patterns::
* Retargeting Flex::
@end menu
@node Makefiles and Flex, Bison Bridge, Appendices, Appendices
@@ -8413,7 +8415,7 @@ removing such sequences from your code.
@code{m4} is only required at the time you run @code{flex}. The generated
scanner is ordinary C or C++, and does @emph{not} require @code{m4}.
@node Common Patterns, ,M4 Dependency, Appendices
@node Common Patterns,Retargeting Flex,M4 Dependency, Appendices
@section Common Patterns
@cindex patterns, common
@@ -8561,6 +8563,113 @@ to appear in a URI, including spaces and control characters. See
@end table
@node Retargeting Flex, ,Common Patterns, Appendices
@section Retargeting Flex
@cindex retargeting
@cindex language independence
This appendix describes how to add support for a new target language
to Flex.
The flex code has been factored to isolate knowledge of the specifics
of each target language from the logic for building the lexer state
tables.
The only assumption that is absolutely baked into all of Flex is that
the bodies of initializers for arrays of integers consist of decimal
numeric literals separated by commas (and optional whitespace).
Otherwise, knowledge of each target langage's syntax lives in two
places: (1) a table of language-specific syntax-generator methods,
and (2) A language-specific skeleton file.
For example: The methods for the C and C++ back end live in a source
file named @file{cpp_backend.c} (so named because both languages use the C
preprocessor), and in a skeleton file named @file{cpp-flex.skl}.
All the code that accesses language-specific code generators goes
through a global pointer named "backend" to a method table. The
results of these generators are used to fill in some parts of the
language-specific skeleton file and conditionalize others.
Read the definition of struct backend_t in @file{src/flexdefs.h}, and
attached comments, to get a feel for the methods. Don't worry
about understandng table generator names at first.
To write support for a language, you'll want to do the following
steps:
@enumerate
@item
Clone one of the existing back-end/skeleton pairs. If the language
you are supporting is named @var{foo}, you should create files named
@file{foo_backend.c} and @file{foo-flex.skl}.
@item
Add @file{foo_backend.c} to COMMON_SOURCES in @file{src/Makefile.am}. Add the
name of your skeleton file to EXTRA_DIST.
@item
Add a production to @file{src/Makefile.am} parallel to the one that
produces @file{cpp-skel.h}. Your objective is to make a string list
initializer from your skeleton file that can be linked with flex and
is pointed at by the skel nember of your language back end.
@item
Add some logic to @file{main.c} that enables the new back end with a
new command-line option. Following this step you should be
able to run Flex on a specification and get code out in the
language of whatever back end you cloned.
@item
The interesting part: mutate your new back end and skeleton so they
produce code in your desired target langage.
@item
Add a test suite for your back end. You should be able to clone
one of the existing sets of test loads to get good coverage. Note
that is highly unlikely your back end will be accepted into the
flex distribution without a test suite.
@end enumerate
Syntactically C-like languages such as Go, Rust, and Java should be easy
targets. Almost anything generally descended from Algol shouldn't be
much more difficult; this certainly includes the whole
Pascal/Modula/Oberon family.
A hint about the interesting part:
@itemize
@item
Don't bother supporting non-reentrant parser generation.
The interface of original Lex with all those globals hanging out
needs to be supported in C for backwards compatibility, but
there is no need to carry it forward.
@end itemize
The following assumptions in the code might trip you up and
require fixes outside a back end.
@enumerate
@item
The item separator in data initializers is a comma.
@item
The language has a case-arm syntax that looks like
a (possibly empty) prefix, followed by a value
expression, followed by a colon.
@item
Either case arms can be stacked as in C; that is, there is
an implicit fallthrough if the case arm has no code. Or,
there is an explicit fallthrough keyword that enables this,
as in Go.
@end enumerate
By putting a @code{yyterminate()} call in the fallthrough member
and a null pointer in the endcase member, you could handle
languages like Pascal.
@node Indices, , Appendices, Top
@unnumbered Indices
+5 -4
View File
@@ -46,6 +46,7 @@ flex_SOURCES = \
$(COMMON_SOURCES)
nodist_flex_SOURCES = \
cpp-skel.h \
stage1scan.c
flex_CFLAGS = $(AM_CFLAGS) $(WARNINGFLAGS)
@@ -54,7 +55,6 @@ COMMON_SOURCES = \
buf.c \
ccl.c \
cpp_backend.c \
cpp_skel.h \
dfa.c \
ecs.c \
filter.c \
@@ -91,13 +91,14 @@ include_HEADERS = \
EXTRA_DIST = \
cpp-flex.skl \
mkskel.sh \
gettext.h \
backend.adoc
gettext.h
CLEANFILES = stage1scan.c stage1flex$(EXEEXT)
CLEANFILES = stage1scan.c stage1flex$(EXEEXT) cpp-skel.h
MAINTAINERCLEANFILES = c9pp-skel.h
flex-cpp_backend.$(OBJEXT): cpp-skel.h
cpp-skel.h: cpp-flex.skl mkskel.sh flexint_shared.h tables_shared.h tables_shared.c
$(SHELL) $(srcdir)/mkskel.sh cpp $(srcdir) $(m4) $(VERSION) > $@.tmp
mv -f $@.tmp $@
-94
View File
@@ -1,94 +0,0 @@
= How to add a support for a new language to flex
= Theory
The flex code was historically written to generate parsers in C, but
it has factored to isolate knowledge of the specifics of each target
languageas from the logic for byukilding the lexer state tables much
as possible.
The only assumption that is absolutely baked into all of flex is that
the bodies of initializers for arrays of integers consist of decimal
numeric kiterals sepaerated by commas (and optional whitespace).
Otherwise, knowledge of each target langage's syntax lives in two
places: (1) a table of langyuge-specific syntax-generator methods,
and (2) A language-specific skeleton file.
For example: The methods for the C and C++ back end live in a source
file named cpp_backend.c (so named because both languages use the C
preprocessor), and in a skeleton file names cpp-flex.skl.
Syntactically C-like languages such as Go, Rust, and Java should be easy
target. Alnost anything generally descended from Algol shouldn't be
much more difficult; this certainly includes the whole
Pascal/Modula/Oberon family.
= Writing a new backend
All the code that accesses language-specific code generators goes
through a global pointer named "backend" to a method table. The
results of these generators are used to fill in some parts of the
language-specifoc skeleton file amd conditionalize other.
Read the definition of struct backend_t in src/flexdefs.h, and
attached comments, to get a feel for the methods. Don't worry
about understandng table generator names at first.
To write support for a language, you'll want to do the following
steps:
1. Clone one of the existing back-end/skeleton pairs. If the language
you are supporting is named "foo", you should create files named
foo_backend.c and foo-flex.skl.
2. Add foo_backend.c to COMMON_SOURCES in src.Makefile.am. Add the
name of your skeleton file to EXTRA_DIST.
3. Add a production to src/Makefile.am parallel to the one that
priduces cpp-skel.h. Your objecting is to make a string list
initializer from your skeleton file that can be linked with flex
and is opointed at by the skel nember of your language back end.
4. Add some logic to main.c that enables the new back end with a
new command-line option. Following this step you should be
able to run flex on a specification and get code out in the
language of whatever back end you cloned.
5. The interesting part: mutate your new back end and skeleton so they
produce code in your desired target langage.
6. Write a test suite for your back end. You should be able to clone
one of the existing sets of test loads to get good coverage. Note
that is highly unlikely your back end will be accepted into the
flex distribution without a test suite.
A hint about step 5:
* Don't bother supporting non-reentrant parser generation.
The interface of original lex with all those globals hanging out
needs to be supported in C for backwards compatibility, but
there
The following assumptions in the code might trip you up and
require fixes outside a back end.
1. The language has a case-arm syntax that looks like
a (possibly empty) prefix, followed by a value
expression, followed by a colon.
2. Either case arms can be stacked as in C; that is, there is
an implicit fallthrough if the case arm has no code. Or,
there is an explicit fallthrough keyword that enables this,
as in Go.
3. The item separator in data initializers is a comma.
By putting a yyterminate() call in the fallthrough member
and a null pointer in the endcase member, you could handle
languages like Pascal.
// end
+1094 -1173
View File
File diff suppressed because it is too large Load Diff