_altname is a char[BUFSIZ+1] that neither constructor touches, so CurrName()
and the AltNameOf paths can read uninitialised heap until something calls
AltName(). Start it empty.
getline(buf, BUFSIZ, ';') on a stream whose next BUFSIZ bytes contain no
semicolon sets failbit and leaves buf unchanged. The scan loop only tests
eof(), which never arrives once failbit is set, so it never terminates - hand
it any binary file and it hangs instead of reporting that the file is not
STEP. Treat a failed extraction the same as end of file.
STEPnode::_null had no default: STEPnode has no constructor, so a node built
by a subclass that does not assign it starts out with whatever was on the
heap, and is_null() then answers at random. Give STEPnode a constructor that
starts nodes null.
EntityNode::StrToVal() and ::STEPread() additionally have to set it per
outcome - they resolve an entity reference and must report null exactly when
the reference did not resolve. Before this, a node whose reference failed to
resolve could still report itself non-null.
A data section indexes its instances from its own constructor, through
addLazyInstance(), under the section id it WILL be given - and
lazyFileReader::initP21() registers the section only afterwards, and only
if its scan succeeded. A section that fails part way therefore leaves
every instance it had already indexed naming a section that does not
exist.
typeFromFile() then indexes _dataSections with it unconditionally, and
loadInstance() does the same behind an assert() that Release compiles
out. On an empty vector that is an access violation. sourceRecord()
already makes exactly this test; the other two now make it too and answer
0 with a diagnostic, as they already do for an instance found in no
section.
The instances stay in the index - totalInstanceCount() still counts them
and getInstances() still lists them. What changes is that asking for one
of them fails the way every other unloadable instance fails.
Reachable from a thirteen-line file, added as test/lazy_broken_section.stp
and checked by the lazy_index test: a DATA section holding one instance
and then a line that is not one. Without the two changes below it, that
test dies with SIGSEGV inside typeFromFile().
Note that the new break in loadInstance() leaves `inst` null, and
isNilSTEPentity( 0 ) is false, so the null would be cached as a loaded
instance were it not for the guard added in "lazy: do not cache a null
instance as a loaded one" earlier in this branch. With the bounds test
alone the process survives the two calls and then dies on the way out.
nextInstance()'s "invalid instance, so clear everything" branch deletes
i.refs but leaves the pointer, while componentTypes two lines below is
deleted and nulled, and i.name is assigned 0 right after. The struct is
returned by value with the stale pointer still in it.
Nothing frees it twice today: the same branch zeroes loc.begin, and both
callers - the data section constructor's scan loop and indexScope() -
stop on that before looking at anything else. So this is the same kind of
uniformity fix as the two commits before it rather than a live bug, and
it costs one line to stop being a question.
nextInstance() explicitly initialises every member of the local
namedLazyInstance except `name`, and the assignment it does get is conditional:
if( _file.good() ) i.name = getDelimitedKeyword( ";( /\\" );
Nothing is wrong today. When that guard is false the closing "invalid instance"
branch is reached on the same condition and assigns `i.name = 0` before the
struct is returned, so the uninitialised value is never observed - but that is a
control-flow argument spanning forty lines, not an invariant the code states.
It also stops holding the moment getDelimitedKeyword() can fail softly. It
cannot upstream, because it abort()s on a keyword with no delimiter after it; a
caller that would rather report bad input than terminate the host process has to
make it return 0, and then `i.name` is read while uninitialised.
One line, and the struct is uniformly initialised where it is declared.
isNilSTEPentity( 0 ) returns false, so `if( !isNilSTEPentity( inst ) )` in
loadInstance() is TRUE for a null pointer, and every branch of the switch above
it can leave one:
* case 0 and default leave `inst` at the value returned by
_instancesLoaded.find(), which is 0 on a cache miss;
* case 1 sets `inst = 0` explicitly on the cancellation path added for batch
loading.
A null therefore reaches the success arm and is inserted into _instancesLoaded,
while _loadedInstanceCount, _materializations, _cacheHighWater and the resident
source-byte accounting are all advanced for an instance that was never
materialized. The lazyRefs construction that follows is harmless - its init()
rejects two null arguments - so this is accounting corruption rather than a
crash.
It does not settle either. releaseBatch() skips the entry with `if( !inst )
continue;` BEFORE the `--_loadedInstanceCount` and the erase from
_batchOwnedInstances, so the phantom can never be evicted: the counter stays
high for the life of the manager and _batchOwnedInstances grows across batches.
cacheStatistics() reports the drift.
Guard the pointer as well as the nil sentinel. A null then takes the existing
else arm, which already emits LAZY_DIAGNOSTIC_ERROR "SDAI instance
materialization failed" - the right diagnostic for what happened.
"cmake: declare target link interfaces explicitly" (0b950dd1) made SC_ADDLIB
publish its LINK_LIBRARIES as PUBLIC. That is the right change, but it is not
compatible with the way src/cldai/CMakeLists.txt names its static dependency:
SC_ADDLIB(stepdai-static STATIC SOURCES ${DAI_SRCS}
LINK_LIBRARIES $<JOIN:${_libdeps},-static >-static)
_libdeps holds one entry, steputils, so the generator expression is an elaborate
way of writing steputils-static. While the dependency was linked privately it
expanded once, on the link line of stepdai-static, and was correct. Published as
PUBLIC it lands in INTERFACE_LINK_LIBRARIES and is evaluated again in every
consumer, where it comes back out mangled and reaches the linker verbatim:
LINK : fatal error LNK1104: cannot open file $<JOIN:steputils,-static.obj
p21read_sdai_<schema> is the first consumer to hit it, so with BUILD_STATIC_LIBS=ON
no schema executable links. Reproduced on develop tip - nothing after 0b950dd1
touches either file, so the static build is broken there too.
Building the list with a foreach costs nothing, reads the same, and keeps working
if _libdeps ever grows. The shared branch is untouched; this is inside
if(BUILD_STATIC_LIBS).
When an embedding application installs a structured diagnostic callback,
do not also print the same materialization errors to stderr. This keeps
large batches of unsupported companion-schema entities from flooding
converter output while preserving legacy standalone diagnostics.
Materializing each instance repeated the same FILE_SCHEMA warning.
Keep a thread-safe claim flag on the lazy file reader so concurrent
loads and multiple DATA sections report the ambiguity once per input
file.
Disable leak detection only while the instrumented exp2cxx build tool
generates schema sources. LeakSanitizer cannot run under common tracers
and sandboxes, which otherwise makes an ASan build fail before its tests
can start.
Preserve the component keyword vector already parsed while indexing a
complex Part 21 instance, and expose it through a read-only lazy manager
API. This lets clients interpret complex instances without reparsing raw
source text or materializing the SDAI graph.
Judy nodes pack variable-width keys at byte offsets which are not
guaranteed to meet judyvalue alignment. Access these keys and caller
byte buffers through memcpy so lazy indexing is portable and visible
to optimizers without undefined behavior.
Also mask the radix iterator shift at a word boundary. Without the
mask, the final byte requests a shift by 64 on 64-bit builds.
Reference validation already reports targets absent from DATA sections.
Do not add those IDs to a materialization closure, where they cannot
be loaded and only produce duplicate or misleading limit errors.
Cover the closure behavior with the existing missing-reference fixture.
Provide explicit ASan, UBSan, and TSan options with compile and
link probes. Keep thread instrumentation in an incompatible,
standalone configuration and preserve frame pointers for useful
diagnostics.
- Correct ownership of stack-backed and allocated entity attributes.
- Typed accessor delegation for redefined attributes.
- Correct destruction when an attribute is redefined to a different type.
- Complete aggregate descriptor initialization.
- The API-v2 exp2python link dependency.
- Regression tests covering those cases.
Write generated files to sibling temporary files and replace a target
only when content differs. Track owned paths in a versioned manifest
and remove obsolete generated files without touching unrelated files.
Strip generator source paths from output and add regression coverage.
Treat nonliteral aggregate bounds as expression records instead of
reading an inactive integer union member. This also removes process
addresses that made packed AP242 output nondeterministic.
On AP242e4, an identical generation changes zero of ten generated
source/header timestamps and leaves the 390-byte manifest unchanged.
The following CMake build falls from 2.01 to 0.06 seconds and peak RSS
from 118,108 to 18,432 KiB. Generation remains 0.40 seconds.
The NIST AP242e4 sample still creates and validates all 10,609
instances with no errors or warnings and writes the same 949,685-byte
semantic result.
Keep full descriptor metadata as the default and add an opt-in
structural profile for conversion-oriented late-bound API v2 users.
Strip descriptive and constraint text while retaining the complete
runtime descriptor graph. Pass the profile through CMake and correct
the scanner's compact late-bound file lists.
AP242e4 full-to-structural measurements:
Generated output falls from 54,839 lines and 3,304,007 bytes to
29,993 lines and 1,525,897 bytes. The packed schema source shrinks
68.1%. The schema DSO falls from 2,178,296 to 720,120 bytes, and peak
compile RSS drops 6.8%. File count and 49 ELF relocations are
unchanged.
The NIST AP242e4 sample still creates and validates all 10,609
instances with no errors or warnings and writes the same 949,685-byte
semantic result. Twenty read/write runs improve from 4.76 to 4.71
seconds and peak RSS falls from 82,276 to 78,900 KiB.
Construct SELECT, enumeration, and aggregate values from descriptors so
compact output needs no generated type classes or factories. Preserve
enumeration elements and renamed schema items in the packed image.
Make each public schema header independently includable through the small
SchemaModule interface. Keep runtime and model declarations internal.
AP242e4 output falls from 865 to 10 files and clean compile/link time
falls from 18.71 to 1.21 seconds. The schema DSO falls from 3,503,352
to 2,178,296 bytes, with ELF relocations falling from 9,620 to 49.
Replace late-bound descriptor procedures with validated, offset-based
schema and complex metadata images. Use an explicit per-load context
for reentrant module construction while retaining the version 1
compatibility path.
AP242e4 generated output falls from 9,060,249 to 4,865,657 bytes and
clean compile/link time falls from 41.49 to 18.71 seconds. The schema
DSO falls from 7,470,304 to 3,503,352 bytes and ELF relocations fall
from 33,221 to 9,620.
Replace generated SchemaModuleSlot pointer arrays with a versioned,
16-byte SchemaModuleImage. Shared initialization helpers capture
entity, type, and attribute descriptors in stable generated-ID order
and transfer them to SchemaModule without one relocated pointer per
descriptor.
Retain the slot-based initialization entry point for existing API v2
output, and leave API v1 generation unchanged.
For AP242e4, generated output falls from 123,552 to 117,046 lines
(-5.3%) and from 9,804,680 to 9,060,249 bytes (-7.6%). The schema
module source falls from 808,907 to 64,367 bytes (-92.0%).
The schema DSO falls from 7,728,384 to 7,470,304 bytes (-3.3%), while
ELF relocations fall from 39,719 to 33,221 (-16.4%). Comparable clean
compile times are unchanged, and paired read/write runs improve by 0.6%.
The real file still creates and validates 10,609 instances with zero
errors and warnings. Its 949,685-byte rewrite differs only by its
timestamp.
Resolve each flattened inherited attribute layout once after schema
initialization. Late-bound instance construction now follows descriptor
and index records directly instead of recursively traversing supertypes
and repeating derived and redefined attribute name searches.
Keep the cache outside EntityDescriptor so its object layout and API v1
ABI remain unchanged. Invalidate cached subtype layouts when descriptor
graphs are edited, and retain lazy preparation for non-module callers.
For the AP242e4 sample's most common entity, constructing 1,531,000
DIRECTION instances improved from 0.84s to 0.63s (-25.0%); user time
improved from 0.82s to 0.61s (-25.6%). Peak RSS increased by 0.9% and
libstepcore grew by 1.6%. Twenty full read/write runs remained at 4.9s.
The real file still created and validated 10,609 instances with zero
errors and warnings. Its 949,685-byte rewrite changed only by its
timestamp.
Add stable entity, type, and attribute IDs through SchemaModule. Make
late-bound compatibility names opt-in, table-drive rules and complex
metadata, and emit entity metadata directly into bounded chunks.
AP242e4 late-bound output changed as follows:
* generated files: 5,719 to 905 (-84.2%)
* generated lines: 254,220 to 123,552 (-51.4%)
* generated bytes: 13,942,523 to 9,804,680 (-29.7%)
* generation wall time: 2.74s to 0.56s (-79.6%)
* compile/link wall time: 157.68s to 37.02s (-76.5%)
* compile/link peak RSS: 593,320 KiB to 199,156 KiB (-66.4%)
* shared library size: 9,956,640 to 7,716,096 bytes (-22.5%)
* read/write wall time: 0.26s to 0.25s
ELF relocations increased from 28,843 to 39,719 due to descriptor-slot
tables. The packed schema image stage is intended to remove this cost.
The real AP242 sample created and validated 10,609 instances with zero
errors and warnings. Rewritten output matched except for its timestamp.
- API v2 table-driven schema, entity, type, attribute, and inheritance
initialization via include/clstepcore/ schemaInit.h.
- --late-bound/-T, producing descriptor-backed entities without
generated early-bound entity classes.
- CMake controls:
- SC_EXP2CXX_API_VERSION=2
- SC_EXP2CXX_LATE_BOUND=ON
- Runtime late-bound construction and owned attribute storage in
src/clstepcore/entityDescriptor.cc and src/clstepcore/STEPattribute.cc.
- Documentation and regression coverage, including incompatible option
handling.
AP242e4 generated-output results:
Mode Lines Bytes
━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━ ━━━━━━━━━━
Default v1 1,210,640 57.57 MB
──────────────────── ─────────── ──────────
API v2 early-bound 509,001 25.20 MB
──────────────────── ─────────── ──────────
API v2 late-bound 254,123 13.94 MB
The late-bound AP242 library compiled under C++11 and successfully
processed the BRL-CAD NIST sample: 10,609 instances
created and validated, with zero errors and warnings.
Opt in with either:
exp2cxx --api-version 2 schema.exp
or:
cmake -DSC_EXP2CXX_API_VERSION=2 ...
Version 1 remains the default, and the regression test verifies default
output is byte-identical to explicit v1
output.
API v2 now provides:
- Compact SELECT classes backed by shared descriptor-driven runtime code
instead of repeated generated parsing, assignment, and serialization
bodies.
- Generic Set* and typed getter APIs in
include/clstepcore/sdaiSelect.h:109.
- Dependency-bounded entity headers and source-owned dependencies in
src/exp2cxx/classes_entity.c:968.
- Out-of-line entity creators.
- Unity chunks that no longer include the all-entity header.
- C++11 output; C++17 did not offer enough additional benefit for this
tranche.
The command-line handling is in src/exp2cxx/fedex_main.c:85, while
compact SELECT generation begins in src/exp2cxx/selects.c:702.
Measured using the BRL-CAD ap242e4.exp, Release builds, eight parallel
jobs, and 64:8 unity chunks:
Metric API v1 Final API v2 Reduction
━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━ ━━━━━━━━━━━
Generated C++ lines 1,210,640 522,445 56.8%
──────────────────────── ─────────────── ────────────── ───────────
Generated bytes 57,566,051 27,202,644 52.7%
──────────────────────── ─────────────── ────────────── ───────────
Schema build wall time 183.22 s 88.31 s 51.8%
──────────────────────── ─────────────── ────────────── ───────────
Compiler CPU time 641.53 s 378.49 s 41.0%
──────────────────────── ─────────────── ────────────── ───────────
Peak RSS 1,035,112 KiB 916,412 KiB 11.5%
──────────────────────── ─────────────── ────────────── ───────────
Object-file bytes 70,472,952 40,031,088 43.2%
──────────────────────── ─────────────── ────────────── ───────────
Shared-library text 34,361,400 19,420,467 43.5%
──────────────────────── ─────────────── ────────────── ───────────
Generation wall time 1.52 s 0.69 s 54.6%
The 35 AP209/AP214 crashes came from inverse resolution inspecting an
instance still being materialized. The loader now defers such inverse
passes until explicit attributes are complete, avoids partial duplicate
inverse population, and guards invalid/null referrers. See
src/cllazyfile/lazyInstMgr.cc:301 and src/cllazyfile/lazyRefs.h:90.
Handle nested SCOPE constructs and export lists while indexing Part 21
records, preserving scope-owned source records and their references.
Add scoped-record coverage.
Schemas commonly contain many more entity implementations than type
implementations. Accept N:N chunk limits and use smaller defaults so
both kinds of generated unity files stay manageable.
Check cancellation while finding dependencies, pinning records, and
resolving recursive references. Discard an object if cancellation
interrupts its materialization.
Generate schema source manifests and bounded unity translation
units, then make schema targets depend on the generated absolute
paths. This avoids oversized compiler inputs while retaining
unity-build performance.
Track indexed record bounds and expose the source-byte footprint of
materialized instances in cache statistics. Avoid promoting batch-owned
cache hits during reference resolution.
Clifford Yapp