When an embedding application installs a structured diagnostic callback,
do not also print the same materialization errors to stderr. This keeps
large batches of unsupported companion-schema entities from flooding
converter output while preserving legacy standalone diagnostics.
Materializing each instance repeated the same FILE_SCHEMA warning.
Keep a thread-safe claim flag on the lazy file reader so concurrent
loads and multiple DATA sections report the ambiguity once per input
file.
Disable leak detection only while the instrumented exp2cxx build tool
generates schema sources. LeakSanitizer cannot run under common tracers
and sandboxes, which otherwise makes an ASan build fail before its tests
can start.
Preserve the component keyword vector already parsed while indexing a
complex Part 21 instance, and expose it through a read-only lazy manager
API. This lets clients interpret complex instances without reparsing raw
source text or materializing the SDAI graph.
Judy nodes pack variable-width keys at byte offsets which are not
guaranteed to meet judyvalue alignment. Access these keys and caller
byte buffers through memcpy so lazy indexing is portable and visible
to optimizers without undefined behavior.
Also mask the radix iterator shift at a word boundary. Without the
mask, the final byte requests a shift by 64 on 64-bit builds.
Reference validation already reports targets absent from DATA sections.
Do not add those IDs to a materialization closure, where they cannot
be loaded and only produce duplicate or misleading limit errors.
Cover the closure behavior with the existing missing-reference fixture.
Provide explicit ASan, UBSan, and TSan options with compile and
link probes. Keep thread instrumentation in an incompatible,
standalone configuration and preserve frame pointers for useful
diagnostics.
- Correct ownership of stack-backed and allocated entity attributes.
- Typed accessor delegation for redefined attributes.
- Correct destruction when an attribute is redefined to a different type.
- Complete aggregate descriptor initialization.
- The API-v2 exp2python link dependency.
- Regression tests covering those cases.
Write generated files to sibling temporary files and replace a target
only when content differs. Track owned paths in a versioned manifest
and remove obsolete generated files without touching unrelated files.
Strip generator source paths from output and add regression coverage.
Treat nonliteral aggregate bounds as expression records instead of
reading an inactive integer union member. This also removes process
addresses that made packed AP242 output nondeterministic.
On AP242e4, an identical generation changes zero of ten generated
source/header timestamps and leaves the 390-byte manifest unchanged.
The following CMake build falls from 2.01 to 0.06 seconds and peak RSS
from 118,108 to 18,432 KiB. Generation remains 0.40 seconds.
The NIST AP242e4 sample still creates and validates all 10,609
instances with no errors or warnings and writes the same 949,685-byte
semantic result.
Keep full descriptor metadata as the default and add an opt-in
structural profile for conversion-oriented late-bound API v2 users.
Strip descriptive and constraint text while retaining the complete
runtime descriptor graph. Pass the profile through CMake and correct
the scanner's compact late-bound file lists.
AP242e4 full-to-structural measurements:
Generated output falls from 54,839 lines and 3,304,007 bytes to
29,993 lines and 1,525,897 bytes. The packed schema source shrinks
68.1%. The schema DSO falls from 2,178,296 to 720,120 bytes, and peak
compile RSS drops 6.8%. File count and 49 ELF relocations are
unchanged.
The NIST AP242e4 sample still creates and validates all 10,609
instances with no errors or warnings and writes the same 949,685-byte
semantic result. Twenty read/write runs improve from 4.76 to 4.71
seconds and peak RSS falls from 82,276 to 78,900 KiB.
Construct SELECT, enumeration, and aggregate values from descriptors so
compact output needs no generated type classes or factories. Preserve
enumeration elements and renamed schema items in the packed image.
Make each public schema header independently includable through the small
SchemaModule interface. Keep runtime and model declarations internal.
AP242e4 output falls from 865 to 10 files and clean compile/link time
falls from 18.71 to 1.21 seconds. The schema DSO falls from 3,503,352
to 2,178,296 bytes, with ELF relocations falling from 9,620 to 49.
Replace late-bound descriptor procedures with validated, offset-based
schema and complex metadata images. Use an explicit per-load context
for reentrant module construction while retaining the version 1
compatibility path.
AP242e4 generated output falls from 9,060,249 to 4,865,657 bytes and
clean compile/link time falls from 41.49 to 18.71 seconds. The schema
DSO falls from 7,470,304 to 3,503,352 bytes and ELF relocations fall
from 33,221 to 9,620.
Replace generated SchemaModuleSlot pointer arrays with a versioned,
16-byte SchemaModuleImage. Shared initialization helpers capture
entity, type, and attribute descriptors in stable generated-ID order
and transfer them to SchemaModule without one relocated pointer per
descriptor.
Retain the slot-based initialization entry point for existing API v2
output, and leave API v1 generation unchanged.
For AP242e4, generated output falls from 123,552 to 117,046 lines
(-5.3%) and from 9,804,680 to 9,060,249 bytes (-7.6%). The schema
module source falls from 808,907 to 64,367 bytes (-92.0%).
The schema DSO falls from 7,728,384 to 7,470,304 bytes (-3.3%), while
ELF relocations fall from 39,719 to 33,221 (-16.4%). Comparable clean
compile times are unchanged, and paired read/write runs improve by 0.6%.
The real file still creates and validates 10,609 instances with zero
errors and warnings. Its 949,685-byte rewrite differs only by its
timestamp.
Resolve each flattened inherited attribute layout once after schema
initialization. Late-bound instance construction now follows descriptor
and index records directly instead of recursively traversing supertypes
and repeating derived and redefined attribute name searches.
Keep the cache outside EntityDescriptor so its object layout and API v1
ABI remain unchanged. Invalidate cached subtype layouts when descriptor
graphs are edited, and retain lazy preparation for non-module callers.
For the AP242e4 sample's most common entity, constructing 1,531,000
DIRECTION instances improved from 0.84s to 0.63s (-25.0%); user time
improved from 0.82s to 0.61s (-25.6%). Peak RSS increased by 0.9% and
libstepcore grew by 1.6%. Twenty full read/write runs remained at 4.9s.
The real file still created and validated 10,609 instances with zero
errors and warnings. Its 949,685-byte rewrite changed only by its
timestamp.
Add stable entity, type, and attribute IDs through SchemaModule. Make
late-bound compatibility names opt-in, table-drive rules and complex
metadata, and emit entity metadata directly into bounded chunks.
AP242e4 late-bound output changed as follows:
* generated files: 5,719 to 905 (-84.2%)
* generated lines: 254,220 to 123,552 (-51.4%)
* generated bytes: 13,942,523 to 9,804,680 (-29.7%)
* generation wall time: 2.74s to 0.56s (-79.6%)
* compile/link wall time: 157.68s to 37.02s (-76.5%)
* compile/link peak RSS: 593,320 KiB to 199,156 KiB (-66.4%)
* shared library size: 9,956,640 to 7,716,096 bytes (-22.5%)
* read/write wall time: 0.26s to 0.25s
ELF relocations increased from 28,843 to 39,719 due to descriptor-slot
tables. The packed schema image stage is intended to remove this cost.
The real AP242 sample created and validated 10,609 instances with zero
errors and warnings. Rewritten output matched except for its timestamp.
- API v2 table-driven schema, entity, type, attribute, and inheritance
initialization via include/clstepcore/ schemaInit.h.
- --late-bound/-T, producing descriptor-backed entities without
generated early-bound entity classes.
- CMake controls:
- SC_EXP2CXX_API_VERSION=2
- SC_EXP2CXX_LATE_BOUND=ON
- Runtime late-bound construction and owned attribute storage in
src/clstepcore/entityDescriptor.cc and src/clstepcore/STEPattribute.cc.
- Documentation and regression coverage, including incompatible option
handling.
AP242e4 generated-output results:
Mode Lines Bytes
━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━ ━━━━━━━━━━
Default v1 1,210,640 57.57 MB
──────────────────── ─────────── ──────────
API v2 early-bound 509,001 25.20 MB
──────────────────── ─────────── ──────────
API v2 late-bound 254,123 13.94 MB
The late-bound AP242 library compiled under C++11 and successfully
processed the BRL-CAD NIST sample: 10,609 instances
created and validated, with zero errors and warnings.
Opt in with either:
exp2cxx --api-version 2 schema.exp
or:
cmake -DSC_EXP2CXX_API_VERSION=2 ...
Version 1 remains the default, and the regression test verifies default
output is byte-identical to explicit v1
output.
API v2 now provides:
- Compact SELECT classes backed by shared descriptor-driven runtime code
instead of repeated generated parsing, assignment, and serialization
bodies.
- Generic Set* and typed getter APIs in
include/clstepcore/sdaiSelect.h:109.
- Dependency-bounded entity headers and source-owned dependencies in
src/exp2cxx/classes_entity.c:968.
- Out-of-line entity creators.
- Unity chunks that no longer include the all-entity header.
- C++11 output; C++17 did not offer enough additional benefit for this
tranche.
The command-line handling is in src/exp2cxx/fedex_main.c:85, while
compact SELECT generation begins in src/exp2cxx/selects.c:702.
Measured using the BRL-CAD ap242e4.exp, Release builds, eight parallel
jobs, and 64:8 unity chunks:
Metric API v1 Final API v2 Reduction
━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━ ━━━━━━━━━━━
Generated C++ lines 1,210,640 522,445 56.8%
──────────────────────── ─────────────── ────────────── ───────────
Generated bytes 57,566,051 27,202,644 52.7%
──────────────────────── ─────────────── ────────────── ───────────
Schema build wall time 183.22 s 88.31 s 51.8%
──────────────────────── ─────────────── ────────────── ───────────
Compiler CPU time 641.53 s 378.49 s 41.0%
──────────────────────── ─────────────── ────────────── ───────────
Peak RSS 1,035,112 KiB 916,412 KiB 11.5%
──────────────────────── ─────────────── ────────────── ───────────
Object-file bytes 70,472,952 40,031,088 43.2%
──────────────────────── ─────────────── ────────────── ───────────
Shared-library text 34,361,400 19,420,467 43.5%
──────────────────────── ─────────────── ────────────── ───────────
Generation wall time 1.52 s 0.69 s 54.6%
The 35 AP209/AP214 crashes came from inverse resolution inspecting an
instance still being materialized. The loader now defers such inverse
passes until explicit attributes are complete, avoids partial duplicate
inverse population, and guards invalid/null referrers. See
src/cllazyfile/lazyInstMgr.cc:301 and src/cllazyfile/lazyRefs.h:90.
Handle nested SCOPE constructs and export lists while indexing Part 21
records, preserving scope-owned source records and their references.
Add scoped-record coverage.
Schemas commonly contain many more entity implementations than type
implementations. Accept N:N chunk limits and use smaller defaults so
both kinds of generated unity files stay manageable.
Check cancellation while finding dependencies, pinning records, and
resolving recursive references. Discard an object if cancellation
interrupts its materialization.
Generate schema source manifests and bounded unity translation
units, then make schema targets depend on the generated absolute
paths. This avoids oversized compiler inputs while retaining
unity-build performance.
Track indexed record bounds and expose the source-byte footprint of
materialized instances in cache statistics. Avoid promoting batch-owned
cache hits during reference resolution.
Add a lazy Part 21 reader and instance manager that index records
and resolve SDAI instances on demand. Include diagnostics, cache
management, reference indexes, and baseline coverage for lazy reads.
A redefined attribute supplies the inherited physical-file slot even
when that original attribute is derived. Prefer the redefinition
consistently during formatting and STEP writing, remove debug output,
and cover the behavior with unit tests.
Modern Lemon accepts allocator callbacks with a context parameter. Adapt
the standard C allocator functions so generated EXPRESS parsers remain
compatible.
Use modern target_link_libraries scopes so libraries publish their
transitive requirements and executables keep theirs private. This
preserves schema test linkage with current CMake versions.
Clifford Yapp