Initial planning session for a bu_parallel rework

bu_parallel_upgrade.txt

@@ -0,0 +1,324 @@
+bu_parallel recursive scheduling upgrade
+========================================
+
+Status
+------
+
+This document proposes a replacement for the recursive scheduling and thread
+ID bookkeeping in src/libbu/parallel.c.  It records the constraints behind the
+current implementation, the recommended C++17 design, and an incremental plan
+for implementing and validating it.
+
+
+Confirmed source of the Valgrind warnings
+-----------------------------------------
+
+The following declaration is the source of both observed Valgrind messages:
+
+    static struct parallel_info mapping[MAX_PSW*MAX_PSW];
+
+MAX_PSW is currently 4096 and struct parallel_info is 32 bytes on this LP64
+build.  The array therefore contributes exactly 512 MiB to libbu's BSS.  The
+complete libbu BSS is 563,028,096 bytes in the observed build.
+
+Preloading only libbu.so.20 into /bin/true under Valgrind reproduces both
+"set address range perms: large range" warnings while reporting zero bytes in
+use at exit.  The dynamic loader emits the warnings while establishing memory
+permissions on libbu's unusually large RW ELF segment.  They are not reports
+of heap leaks.
+
+
+Why the array is squared
+------------------------
+
+Commit 145c80ce922fae38105c01aacae1da690e0955c1 deliberately changed the
+mapping from MAX_PSW entries to MAX_PSW*MAX_PSW entries.  It was an explicitly
+temporary attempt to accommodate recursive bu_parallel() calls.
+
+The present algorithm uses a parallel_info record both as a live thread record
+and as the concurrency scope for a nested invocation.  If an outer invocation
+starts N callbacks, each callback can call bu_parallel(..., 0) and inherit an
+independent limit of N.  The implementation can consequently create about N*N
+simultaneous parent and child records rather than sharing one limit of N.
+
+There are two additional historical constraints:
+
+* A parent ID cannot be released when a nested bu_parallel() returns if the
+  parent callback is still alive.  It may retain per-ID state and may invoke
+  bu_parallel() again.  Commit 988887e815 documents this case.
+
+* bu_parallel_id() values are used throughout BRL-CAD to index MAX_PSW-sized
+  lock-free resource arrays.  The value may be arbitrary, but it must remain in
+  the range [0, MAX_PSW).  Commit af04ca7bb4 documents this requirement.
+
+The squared mapping postpones exhaustion for one level of recursion, but it
+does not solve arbitrary nesting and it permits IDs that violate the public
+MAX_PSW array contract.  Increasing MAX_PSW from 1024 to 4096 in commit
+9aa03acf836 made the 48/64-core recursive test fit, but also increased the
+squared mapping from about 32 MiB to 512 MiB.
+
+Reducing the mapping to MAX_PSW without changing the scheduling model is not a
+valid fix.  It restores the failure that the 2016 change was trying to avoid.
+
+
+Required invariants
+-------------------
+
+The replacement must maintain the following invariants:
+
+1. Inside a callback, func_cpu_id and bu_parallel_id() identify the same
+   logical resource slot.
+
+2. Every live logical callback context has an ID in [0, MAX_PSW), and no two
+   simultaneously live contexts share an ID.
+
+3. A callback retains its logical ID across a nested bu_parallel() call and
+   until the callback itself returns.  Nested callbacks receive other IDs.
+
+4. bu_parallel() remains synchronous: it does not return until all callbacks
+   requested by that invocation have completed.
+
+5. ncpu == 0 inherits the enclosing executor's concurrency width instead of
+   multiplying it at every recursive level.
+
+6. Recursive fork/join cannot deadlock merely because all physical workers are
+   currently executing parent callbacks.
+
+7. Explicit requests are bounded by MAX_PSW.  Exhaustion must produce a clear,
+   deterministic failure before any out-of-range ID is exposed.
+
+8. Bookkeeping memory is proportional to actual workers, active calls, and
+   nesting depth.  It must not be proportional to MAX_PSW squared.
+
+9. Existing C callers and the exported C ABI remain compatible.
+
+
+Recommended design: a C++17 fork/join executor
+----------------------------------------------
+
+C++17 makes it practical to replace the platform-specific recursive thread
+creation and manual counters with a small fork/join executor using std::thread,
+std::mutex, std::condition_variable, atomics, and RAII.
+
+The preferred organization is to put the scheduler in a C++ implementation
+file and retain C-callable entry points.  It should not be necessary to convert
+all of parallel.c immediately.  CPU discovery and platform helpers can remain
+in C while the current parallel_cpp11thread.cpp is replaced or superseded by a
+new parallel_executor.cpp.  If moving bu_parallel() cleanly requires renaming
+parallel.c to parallel.cpp, its exported functions must retain C linkage.
+
+Do not enable the existing CPP11THREAD implementation as-is.  It does not
+implement BRL-CAD logical IDs, affinity behavior, recursive limits, nested
+fork/join progress, or the MAX_PSW resource contract.
+
+The executor should contain these conceptual components:
+
+1. Physical executor
+
+   An outermost bu_parallel() creates an executor with a bounded number of
+   physical workers.  An explicit ncpu selects the requested width, clamped to
+   MAX_PSW.  A zero value uses bu_avail_cpus().  A nested invocation reuses the
+   executor already recorded in thread-local state and never creates N more OS
+   threads for each parent callback.
+
+   Starting with an executor owned by each outermost call avoids introducing a
+   process-lifetime thread pool, complicated fork behavior, and new shutdown
+   ordering.  A persistent process-wide executor can be considered later if
+   measurements justify it.
+
+2. Logical ID pool
+
+   Maintain a process-wide pool of exactly MAX_PSW logical resource IDs.  IDs
+   are acquired immediately before a callback begins and released immediately
+   after that callback returns.  A free list or bitmap avoids scanning a large
+   mapping table.
+
+   A nested callback running on the same physical std::thread as its suspended
+   parent must acquire a different logical ID.  An RAII guard saves the
+   physical thread's previous thread-local ID, installs the child's ID, and
+   restores the parent ID when the child returns.  The parent ID remains
+   reserved for the lifetime of the suspended parent stack.
+
+   If nesting consumes all MAX_PSW logical IDs, bu_parallel() must fail with a
+   diagnostic rather than wait forever or index beyond a caller's resource
+   array.  The API currently returns void, so the existing bu_bomb-style hard
+   failure is the compatible behavior unless a future API adds error returns.
+
+3. Parallel call state
+
+   Each invocation has dynamically allocated call state containing at least:
+
+       callback and data
+       requested callback count
+       next unclaimed callback
+       number of incomplete callbacks
+       parent/root executor association
+       completion mutex/condition
+       failure state
+
+   Callback work should be claimed lazily.  Do not allocate or enqueue N*N job
+   objects for a recursive N-by-N call.  A bounded number of runnable tokens
+   can refer to one call state while workers atomically claim its remaining
+   callback invocations.  Memory then remains proportional to executor width
+   and the number of active calls.
+
+4. Helping wait
+
+   A non-worker caller may sleep on the call's completion condition.  A worker
+   making a nested bu_parallel() call must not simply block, since all workers
+   could otherwise block waiting for children that have no thread on which to
+   run.
+
+   Instead, a waiting worker repeatedly executes available work until its
+   nested call completes.  This is the standard fork/join "help while waiting"
+   rule.  It keeps the fixed worker set making progress at arbitrary nesting
+   depth and replaces the current one-second polling loop.
+
+   Every re-entered callback gets its own logical ID guard, even when it runs
+   on the same physical std::thread as a suspended parent.
+
+5. Affinity
+
+   Affinity belongs to physical workers, not logical callback IDs.  Set it once
+   when a worker starts, using a physical worker ordinal.  Logical IDs may be
+   larger than the available CPU count and may change as one worker helps
+   nested calls.
+
+6. Callback ID semantics
+
+   The callback's func_cpu_id remains a logical resource ID and must match
+   bu_parallel_id().  It is not a stable callback ordinal and is not guaranteed
+   to start at zero or be contiguous.  Documentation that promises increasing
+   callback numbers starting at zero should be corrected.  Code needing a work
+   ordinal should claim work from its shared data, as existing self-dispatching
+   callbacks generally do.
+
+
+Why this fixes the recursive case
+---------------------------------
+
+For an N-way outer call in which every callback makes an N-way nested call:
+
+* The executor still has N physical workers, not N*N workers.
+* All N*N nested callback invocations are performed, but workers claim them
+  incrementally while waiting for their own nested joins.
+* Each suspended outer callback retains its logical ID.
+* Each currently executing nested callback obtains another logical ID.
+* The usual two-level case needs on the order of 2*N live logical IDs rather
+  than N*N mapping entries.
+* Deeper nesting consumes logical IDs according to live stack depth.  The hard
+  MAX_PSW bound remains explicit and safe instead of silently corrupting a
+  per-ID array.
+
+
+Optional short-term containment
+-------------------------------
+
+If the loader warnings must be removed before the executor is ready, replace
+the monolithic squared array with stable, lazily allocated blocks while
+preserving its current logical capacity.  For example, a static top-level
+array of block pointers can address blocks of 1024 parallel_info records,
+allocating a block only when an ID in that block is first needed.  Individual
+blocks must never move because thread_context currently retains pointers to
+parallel_info records.
+
+A free list or high-water hint should accompany the blocks so ID allocation
+does not scan the entire MAX_PSW*MAX_PSW address space.  This would reduce
+libbu's initial BSS and eliminate the two Valgrind loader warnings without
+reintroducing the immediate recursive-capacity regression.
+
+This is containment only.  It retains the possibility of returning an ID
+greater than or equal to MAX_PSW and does not bound recursive thread creation.
+It must not be presented as the correctness fix or allowed to delay the ID and
+scheduler work above.
+
+
+Implementation plan
+-------------------
+
+Phase 1: characterize and strengthen tests
+
+* Add instrumentation available in test builds to report the maximum logical
+  ID, number of simultaneously live logical contexts, physical worker count,
+  and nesting depth.
+* Extend test_parallel with two- and three-level recursion.
+* Test repeated nested calls from one parent, preserving the case described by
+  commit 988887e815.
+* Exercise ncpu values 1, 2, the available CPU count, 48, and 64 where the host
+  permits them without making every test host create thousands of threads.
+* Add concurrent independent outermost bu_parallel() calls from ordinary
+  application threads.
+* Assert that every observed callback ID is less than MAX_PSW and that
+  func_cpu_id equals bu_parallel_id().
+
+Phase 2: implement the executor behind an internal switch
+
+* Add the C++17 executor and logical ID pool without deleting the legacy path.
+* Keep both implementations selectable at build time or by a test-only switch
+  so identical workloads can be compared.
+* Use RAII for logical IDs, TLS restoration, mutex ownership, worker joining,
+  and cleanup after thread creation failures.
+* Catch std::system_error from worker creation, stop accepting work, join all
+  workers already created, and issue the normal libbu fatal diagnostic.
+
+Phase 3: implement recursive helping and lazy work claims
+
+* Make nested calls reuse their current executor.
+* Implement help-while-waiting and completion notification.
+* Generate callback work lazily so recursive fan-out does not allocate a job
+  object per callback.
+* Confirm that nested explicit ncpu requests control the number of callback
+  invocations but do not create a second unbounded set of OS threads.
+
+Phase 4: switch the default and remove legacy mapping
+
+* Run the complete libbu, librt, MGED, and ray-tracing regression suites using
+  the new executor.
+* Stress under ThreadSanitizer and run the recursive tests repeatedly to expose
+  bookkeeping races.
+* Compare representative ray-tracing and BREP-preparation performance against
+  the legacy implementation.
+* Make the executor the default, then remove parallel_mapping(),
+  parallel_wait_for_slot(), the squared mapping, and obsolete platform thread
+  creation branches.
+* Remove the Valgrind warning FIXME once libbu's BSS no longer contains the
+  squared array, and verify by preloading libbu into /bin/true under Valgrind.
+
+Phase 5: documentation and cleanup
+
+* Update bu/parallel.h to describe arbitrary bounded logical IDs, executor
+  reuse during recursion, and explicit failure at MAX_PSW exhaustion.
+* Update the TODO item concerning recursive bu_parallel limits and known races.
+* Decide whether the old parallel_cpp11thread.cpp should be replaced, renamed,
+  or removed to avoid two competing implementations.
+* Consider a persistent executor only as a separate performance project after
+  fork behavior, library teardown, and application concurrency are specified.
+
+
+Validation checklist
+--------------------
+
+The upgrade is ready to replace the legacy path only when all of the following
+are true:
+
+* No callback observes an ID outside [0, MAX_PSW).
+* No simultaneously live callback contexts share an ID.
+* A parent retains its ID before and after every nested call.
+* Two- and three-level recursive tests complete without deadlock.
+* Recursive ncpu == 0 never increases the physical worker count.
+* Multiple independent outer calls complete without ID collisions.
+* Thread creation and logical ID exhaustion fail cleanly.
+* ThreadSanitizer reports no scheduler bookkeeping races.
+* Existing libbu and application regression tests pass.
+* Representative performance is not materially worse than the legacy path.
+* libbu no longer contains the 512 MiB mapping BSS.
+* Preloading libbu under Valgrind produces neither large-range warning.
+
+
+Non-goals
+---------
+
+This upgrade should not initially attempt to remove MAX_PSW-sized resource
+arrays throughout BRL-CAD, change the public callback signature, introduce a
+permanent global thread pool, or guarantee a callback ordinal through
+func_cpu_id.  Those are separate API and architecture projects.

misc/tools/CMakeLists.txt

@@ -45,14 +45,6 @@ cmakefiles(
   asciiquack/vendor/cxxopts.hpp
   db2adoc/db2adoc.py
   db2adoc/db2adoc.xsl
-  brep_corpus_audit.py
-  step_corpus_capture.py
-  step_corpus_run.py
-  step_corpus_summary.py
-  step_corpus_verify.py
-  step_external_corpus_expectations.json
-  step_local_external_corpus_expectations.json
-  step_dependency_subset.py
   inactvhdrs/CMakeLists.txt
   inactvhdrs/inactvhdrs.cpp
 )

misc/tools/brep_corpus_audit.py

@@ -1,619 +0,0 @@
-#!/usr/bin/env python3
-"""Audit BRep wireframe and fast shaded realizations in a directory of .g files.
-
-The companion ``brep-audit`` executable does one BRep per process.  This
-driver inventories each database, runs those isolated checks, checkpoints
-every result to JSON Lines, and writes JSON and Markdown summaries.  Geometry
-findings do not make the driver fail: they are the intended work list.
-"""
-
-import argparse
-import datetime
-import json
-import os
-import re
-import signal
-import subprocess
-import sys
-import tempfile
-import time
-from collections import Counter
-from concurrent.futures import ThreadPoolExecutor, as_completed
-from pathlib import Path
-
-
-FORMAT = "brlcad-brep-corpus-audit-v1"
-HELPER_FORMAT = "brlcad-brep-realization-audit-v1"
-
-
-def file_identity(path):
-    try:
-        info = path.stat()
-        return {"size": info.st_size, "mtime_ns": info.st_mtime_ns}
-    except OSError:
-        return None
-
-
-def text_tail(value, limit=8000):
-    return (value or "")[-limit:]
-
-
-def last_phase(stderr):
-    phases = re.findall(r"^brep-audit: phase=([a-z]+)$", stderr or "", re.MULTILINE)
-    return phases[-1] if phases else None
-
-
-def process_rss_bytes(pid):
-    """Read current Linux RSS, returning zero when it is unavailable."""
-    try:
-        with open("/proc/{}/status".format(pid), "r", encoding="ascii") as stream:
-            for line in stream:
-                if line.startswith("VmRSS:"):
-                    return int(line.split()[1]) * 1024
-    except (OSError, ValueError, IndexError):
-        pass
-    return 0
-
-
-def kill_process(process):
-    try:
-        if os.name == "posix":
-            os.killpg(process.pid, signal.SIGKILL)
-        else:
-            process.kill()
-    except (OSError, ProcessLookupError):
-        pass
-
-
-def file_tail(stream, limit):
-    stream.flush()
-    size = os.fstat(stream.fileno()).st_size
-    stream.seek(max(0, size - limit))
-    return stream.read().decode("utf-8", errors="replace"), size > limit
-
-
-def run_process(command, timeout, max_rss_mib=0):
-    """Run and monitor one helper, retaining enough detail for diagnostics."""
-    started = time.monotonic()
-    with tempfile.TemporaryFile() as stdout_file, \
-            tempfile.TemporaryFile() as stderr_file:
-        try:
-            process = subprocess.Popen(
-                command,
-                stdout=stdout_file,
-                stderr=stderr_file,
-                start_new_session=(os.name == "posix"),
-            )
-        except OSError as error:
-            return {
-                "returncode": 127,
-                "stdout": "",
-                "stdout_truncated": False,
-                "stderr": str(error),
-                "elapsed_seconds": time.monotonic() - started,
-                "observed_peak_rss_bytes": 0,
-                "termination_reason": "launch_error",
-            }
-
-        peak_rss = 0
-        termination_reason = None
-        while True:
-            elapsed = time.monotonic() - started
-            remaining = timeout - elapsed
-            if remaining <= 0:
-                termination_reason = "timeout"
-                kill_process(process)
-                process.wait()
-                break
-            try:
-                process.wait(timeout=min(0.1, remaining))
-                peak_rss = max(peak_rss, process_rss_bytes(process.pid))
-                break
-            except subprocess.TimeoutExpired:
-                rss = process_rss_bytes(process.pid)
-                peak_rss = max(peak_rss, rss)
-                if max_rss_mib and rss > max_rss_mib * 1024 * 1024:
-                    termination_reason = "rss_limit"
-                    kill_process(process)
-                    process.wait()
-                    break
-
-        stdout, stdout_truncated = file_tail(
-            stdout_file, 16 * 1024 * 1024
-        )
-        stderr, _ = file_tail(stderr_file, 64 * 1024)
-
-    return {
-        "returncode": process.returncode,
-        "stdout": stdout,
-        "stdout_truncated": stdout_truncated,
-        "stderr": stderr,
-        "elapsed_seconds": time.monotonic() - started,
-        "observed_peak_rss_bytes": peak_rss,
-        "termination_reason": termination_reason,
-    }
-
-
-def synthetic_record(database, object_name, status, issue, run, signature):
-    return {
-        "format": HELPER_FORMAT,
-        "scope": "brep" if object_name else "database",
-        "database": str(database),
-        "object": object_name,
-        "status": status,
-        "reference": None,
-        "wireframe": None,
-        "shaded": None,
-        "issues": [issue],
-        "audit_signature": signature,
-        "database_identity": file_identity(database),
-        "process": {
-            "return_code": run["returncode"],
-            "elapsed_seconds": run["elapsed_seconds"],
-            "observed_peak_rss_bytes": run["observed_peak_rss_bytes"],
-            "termination_reason": run["termination_reason"],
-            "last_phase": last_phase(run["stderr"]),
-            "stdout_tail": text_tail(run["stdout"]),
-            "stderr_tail": text_tail(run["stderr"]),
-        },
-    }
-
-
-def parse_helper_json(output):
-    for line in reversed(output.splitlines()):
-        if not line.lstrip().startswith("{"):
-            continue
-        try:
-            value = json.loads(line)
-        except json.JSONDecodeError:
-            continue
-        if isinstance(value, dict):
-            return value
-    return None
-
-
-def audit_one(helper, database, object_name, args, signature):
-    command = [
-        str(helper),
-        "--ratio-min", str(args.ratio_min),
-        "--ratio-max", str(args.ratio_max),
-        "--tess-abs", str(args.tess_abs),
-        "--tess-rel", str(args.tess_rel),
-        "--tess-norm", str(args.tess_norm),
-    ]
-    if args.address_space_limit_mib:
-        command.extend([
-            "--memory-limit-mib", str(args.address_space_limit_mib)
-        ])
-    command.extend([str(database), object_name])
-    run = run_process(command, args.timeout, args.max_rss_mib)
-
-    if run["termination_reason"] == "timeout":
-        phase = last_phase(run["stderr"])
-        return synthetic_record(
-            database, object_name, "timeout",
-            "{}_timeout".format(phase) if phase else "process_timeout",
-            run, signature
-        )
-    if run["termination_reason"] == "rss_limit":
-        phase = last_phase(run["stderr"])
-        return synthetic_record(
-            database, object_name, "memory_limit",
-            "{}_rss_limit_exceeded".format(phase)
-            if phase else "process_rss_limit_exceeded",
-            run, signature
-        )
-    if run["termination_reason"] == "launch_error":
-        return synthetic_record(
-            database, object_name, "error", "helper_launch_failed", run, signature
-        )
-
-    record = parse_helper_json(run["stdout"])
-    if record is None:
-        issue = (
-            "process_signal_{}".format(-run["returncode"])
-            if run["returncode"] is not None and run["returncode"] < 0
-            else "helper_output_not_json"
-        )
-        return synthetic_record(
-            database, object_name, "error", issue, run, signature
-        )
-    if record.get("format") != HELPER_FORMAT:
-        return synthetic_record(
-            database, object_name, "error", "unexpected_helper_format",
-            run, signature
-        )
-
-    record["scope"] = "brep"
-    record["audit_signature"] = signature
-    record["database_identity"] = file_identity(database)
-    record["process"] = {
-        "return_code": run["returncode"],
-        "elapsed_seconds": run["elapsed_seconds"],
-        "observed_peak_rss_bytes": run["observed_peak_rss_bytes"],
-        "termination_reason": None,
-        "last_phase": last_phase(run["stderr"]),
-        "stdout_tail": "",
-        "stderr_tail": text_tail(run["stderr"]),
-    }
-    expected_return = 0 if record.get("status") == "ok" else 1
-    if run["returncode"] != expected_return:
-        record["status"] = "error"
-        record.setdefault("issues", []).append("helper_exit_status_mismatch")
-    return record
-
-
-def list_breps(helper, database, timeout, signature):
-    run = run_process([str(helper), "--list", str(database)], timeout)
-    if run["termination_reason"] == "timeout":
-        return [], synthetic_record(
-            database, "", "timeout", "brep_inventory_timeout", run, signature
-        )
-    if run["returncode"] != 0:
-        return [], synthetic_record(
-            database, "", "error", "brep_inventory_failed", run, signature
-        )
-    if run["stdout_truncated"]:
-        return [], synthetic_record(
-            database, "", "error", "brep_inventory_output_too_large",
-            run, signature
-        )
-    names = sorted(set(line.strip() for line in run["stdout"].splitlines()
-                       if line.strip()))
-    return names, None
-
-
-def git_state(start):
-    try:
-        revision = subprocess.check_output(
-            ["git", "-C", str(start), "rev-parse", "HEAD"],
-            stderr=subprocess.DEVNULL, text=True,
-        ).strip()
-        dirty = bool(subprocess.check_output(
-            ["git", "-C", str(start), "status", "--porcelain"],
-            stderr=subprocess.DEVNULL, text=True,
-        ).strip())
-        return revision, dirty
-    except (OSError, subprocess.CalledProcessError):
-        return "unknown", None
-
-
-def read_checkpoint(path, signature):
-    records = {}
-    if not path.is_file():
-        return records
-    with path.open("r", encoding="utf-8") as stream:
-        for number, line in enumerate(stream, 1):
-            try:
-                record = json.loads(line)
-            except json.JSONDecodeError:
-                print("ignoring malformed checkpoint line {} in {}".format(
-                    number, path), file=sys.stderr)
-                continue
-            if record.get("audit_signature") != signature:
-                continue
-            records[(record.get("database", ""), record.get("object", ""))] = record
-    return records
-
-
-def all_issues(record):
-    issues = list(record.get("issues") or [])
-    for mode in ("wireframe", "shaded"):
-        result = record.get(mode)
-        if result:
-            issues.extend(result.get("issues") or [])
-    return list(dict.fromkeys(str(issue) for issue in issues))
-
-
-def format_vector(values):
-    if not values:
-        return "-"
-    return " × ".join("{:.5g}".format(value) for value in values)
-
-
-def format_geometry(result, primitive):
-    if not result:
-        return "-"
-    ratios = result.get("dimension_ratios")
-    ratio_text = ""
-    if ratios:
-        ratio_text = "; ratios " + "/".join(
-            "-" if value is None else "{:.3g}".format(value)
-            for value in ratios
-        )
-    return "{} points, {} {}{}".format(
-        result.get("vertices", 0), result.get("primitives", 0),
-        primitive, ratio_text,
-    )
-
-
-def markdown_escape(value):
-    return str(value).replace("|", "\\|").replace("\n", "<br>")
-
-
-def worklist_markdown(summary, input_directory):
-    totals = summary["totals"]
-    lines = [
-        "# BRep realization audit",
-        "",
-        "Revision: `{}`{}".format(
-            summary["revision"], " (dirty)" if summary.get("dirty") else ""
-        ),
-        "",
-        (
-            "{} databases, {} BReps: {} okay, {} geometry findings, "
-            "{} timeouts, {} memory-limit stops, and {} process errors."
-        ).format(
-            totals["databases"], totals["breps"], totals["ok"], totals["fail"],
-            totals["timeout"], totals["memory_limit"], totals["error"],
-        ),
-        "",
-        (
-            "Accepted generated/reference dimension ratios: {}–{}. "
-            "Shaded tessellation tolerance: abs {}, rel {}, norm {}."
-        ).format(
-            summary["options"]["ratio_min"], summary["options"]["ratio_max"],
-            summary["options"]["tess_abs"], summary["options"]["tess_rel"],
-            summary["options"]["tess_norm"],
-        ),
-        "",
-        "## Work list",
-        "",
-    ]
-    findings = [
-        record for record in summary["results"]
-        if record.get("status") != "ok"
-    ]
-    if not findings:
-        lines.append("No findings.")
-        return "\n".join(lines) + "\n"
-
-    lines.extend([
-        "| Database | BRep | Status | Reference dimensions | Wireframe | Shaded | Peak RSS MiB | Issues |",
-        "|---|---|---|---|---|---|---:|---|",
-    ])
-    for record in findings:
-        database = Path(record.get("database", ""))
-        try:
-            database_name = str(database.relative_to(input_directory))
-        except ValueError:
-            database_name = str(database)
-        wire = record.get("wireframe")
-        shaded = record.get("shaded")
-        peaks = [
-            int(result.get("peak_rss_bytes") or 0)
-            for result in (wire, shaded) if result
-        ]
-        process = record.get("process") or {}
-        peaks.append(int(process.get("observed_peak_rss_bytes") or 0))
-        reference = record.get("reference") or {}
-        lines.append("| {} | {} | {} | {} | {} | {} | {:.1f} | {} |".format(
-            markdown_escape(database_name),
-            markdown_escape(record.get("object") or "(inventory)"),
-            markdown_escape(record.get("status", "error")),
-            markdown_escape(format_vector(reference.get("dimensions"))),
-            markdown_escape(format_geometry(wire, "segments")),
-            markdown_escape(format_geometry(shaded, "triangles")),
-            max(peaks, default=0) / (1024.0 * 1024.0),
-            markdown_escape(", ".join(all_issues(record)) or "unspecified"),
-        ))
-    return "\n".join(lines) + "\n"
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("input_directory", type=Path)
-    parser.add_argument("output_directory", type=Path)
-    parser.add_argument(
-        "--brep-audit", type=Path, default=Path(".build/bin/brep-audit")
-    )
-    parser.add_argument("-j", "--jobs", type=int, default=1,
-                        help="BRep processes to run concurrently (default: 1)")
-    parser.add_argument("--timeout", type=float, default=300.0,
-                        help="seconds allowed per BRep")
-    parser.add_argument("--list-timeout", type=float, default=120.0,
-                        help="seconds allowed to inventory one database")
-    parser.add_argument("--ratio-min", type=float, default=0.5)
-    parser.add_argument("--ratio-max", type=float, default=2.0)
-    parser.add_argument("--tess-abs", type=float, default=0.0)
-    parser.add_argument("--tess-rel", type=float, default=0.01)
-    parser.add_argument("--tess-norm", type=float, default=0.0)
-    parser.add_argument(
-        "--max-rss-mib", type=int, default=0,
-        help="kill a BRep helper above this observed RSS (zero disables)"
-    )
-    parser.add_argument(
-        "--address-space-limit-mib", type=int, default=0,
-        help="set the helper RLIMIT_AS in MiB (zero disables)"
-    )
-    parser.add_argument("--only-file",
-                        help="regular expression matched against .g filenames")
-    parser.add_argument("--only-object",
-                        help="regular expression matched against BRep names")
-    parser.add_argument("--resume", action="store_true",
-                        help="retain checkpointed results with identical options")
-    args = parser.parse_args()
-
-    if args.jobs < 1:
-        parser.error("--jobs must be positive")
-    if args.timeout <= 0 or args.list_timeout <= 0:
-        parser.error("timeouts must be positive")
-    if args.ratio_min <= 0 or args.ratio_max < args.ratio_min:
-        parser.error("dimension ratio limits are invalid")
-    if min(args.tess_abs, args.tess_rel, args.tess_norm) < 0:
-        parser.error("tessellation tolerances cannot be negative")
-    if args.max_rss_mib < 0 or args.address_space_limit_mib < 0:
-        parser.error("memory limits cannot be negative")
-
-    input_directory = args.input_directory.resolve()
-    output_directory = args.output_directory.resolve()
-    helper = args.brep_audit.resolve()
-    if not helper.is_file():
-        parser.error("brep-audit not found: {}".format(helper))
-    file_matcher = re.compile(args.only_file) if args.only_file else None
-    object_matcher = re.compile(args.only_object) if args.only_object else None
-    databases = sorted(
-        path.resolve() for path in input_directory.iterdir()
-        if path.is_file() and path.suffix.lower() == ".g" and
-        (not file_matcher or file_matcher.search(path.name))
-    )
-    if not databases:
-        parser.error("no matching .g files in {}".format(input_directory))
-
-    options = {
-        "ratio_min": args.ratio_min,
-        "ratio_max": args.ratio_max,
-        "tess_abs": args.tess_abs,
-        "tess_rel": args.tess_rel,
-        "tess_norm": args.tess_norm,
-        "timeout": args.timeout,
-        "list_timeout": args.list_timeout,
-        "max_rss_mib": args.max_rss_mib,
-        "address_space_limit_mib": args.address_space_limit_mib,
-        "only_file": args.only_file,
-        "only_object": args.only_object,
-        "helper": str(helper),
-        "helper_size": helper.stat().st_size,
-        "helper_mtime_ns": helper.stat().st_mtime_ns,
-    }
-    signature = options
-    output_directory.mkdir(parents=True, exist_ok=True)
-    checkpoint_path = output_directory / "brep-audit-results.jsonl"
-    records = (
-        read_checkpoint(checkpoint_path, signature) if args.resume else {}
-    )
-
-    tasks = []
-    for index, database in enumerate(databases, 1):
-        print("[inventory {}/{}] {}".format(
-            index, len(databases), database.name), flush=True)
-        names, error_record = list_breps(
-            helper, database, args.list_timeout, signature
-        )
-        if error_record:
-            for key in list(records):
-                if key[0] == str(database):
-                    records.pop(key)
-            records[(str(database), "")] = error_record
-            continue
-        records.pop((str(database), ""), None)
-        if object_matcher:
-            names = [name for name in names if object_matcher.search(name)]
-        current_names = set(names)
-        identity = file_identity(database)
-        for key, record in list(records.items()):
-            if key[0] != str(database):
-                continue
-            if key[1] not in current_names or \
-                    record.get("database_identity") != identity:
-                records.pop(key)
-        tasks.extend((database, name) for name in names)
-
-    # Rewrite a resumed checkpoint with only the selected run signature, then
-    # append each new result immediately so an interrupted corpus run is usable.
-    with checkpoint_path.open("w", encoding="utf-8") as checkpoint:
-        for record in sorted(
-                records.values(),
-                key=lambda item: (item.get("database", ""), item.get("object", ""))):
-            checkpoint.write(json.dumps(record, sort_keys=True) + "\n")
-        checkpoint.flush()
-
-    pending = [
-        task for task in tasks if (str(task[0]), task[1]) not in records
-    ]
-    resumed = len(tasks) - len(pending)
-    if resumed:
-        print("resuming {} completed BRep records".format(resumed), flush=True)
-
-    completed = 0
-    with checkpoint_path.open("a", encoding="utf-8") as checkpoint:
-        with ThreadPoolExecutor(max_workers=args.jobs) as pool:
-            futures = {
-                pool.submit(
-                    audit_one, helper, database, object_name, args, signature
-                ): (database, object_name)
-                for database, object_name in pending
-            }
-            for future in as_completed(futures):
-                database, object_name = futures[future]
-                try:
-                    record = future.result()
-                except Exception as error:  # Preserve the rest of a corpus run.
-                    run = {
-                        "returncode": 125,
-                        "stdout": "",
-                        "stderr": repr(error),
-                        "elapsed_seconds": 0.0,
-                        "observed_peak_rss_bytes": 0,
-                        "termination_reason": "driver_error",
-                    }
-                    record = synthetic_record(
-                        database, object_name, "error",
-                        "audit_driver_exception", run, signature
-                    )
-                records[(str(database), object_name)] = record
-                checkpoint.write(json.dumps(record, sort_keys=True) + "\n")
-                checkpoint.flush()
-                completed += 1
-                print("[audit {}/{}] {}:{} {}".format(
-                    completed, len(pending), database.name, object_name,
-                    record["status"],
-                ), flush=True)
-
-    results = sorted(
-        records.values(),
-        key=lambda item: (item.get("database", ""), item.get("object", "")),
-    )
-    statuses = Counter(
-        record.get("status", "error")
-        for record in results if record.get("scope") == "brep"
-    )
-    database_errors = [
-        record for record in results if record.get("scope") == "database"
-    ]
-    revision, dirty = git_state(Path(__file__).resolve().parents[2])
-    summary = {
-        "format": FORMAT,
-        "generated_utc": datetime.datetime.now(
-            datetime.timezone.utc
-        ).isoformat(),
-        "revision": revision,
-        "dirty": dirty,
-        "input_directory": str(input_directory),
-        "output_directory": str(output_directory),
-        "helper": str(helper),
-        "options": options,
-        "totals": {
-            "databases": len(databases),
-            "database_inventory_errors": len(database_errors),
-            "breps": sum(statuses.values()),
-            "ok": statuses["ok"],
-            "fail": statuses["fail"],
-            "timeout": statuses["timeout"],
-            "memory_limit": statuses["memory_limit"],
-            "error": statuses["error"] + len(database_errors),
-        },
-        "results": results,
-    }
-    summary_path = output_directory / "brep-audit-summary.json"
-    worklist_path = output_directory / "brep-audit-worklist.md"
-    summary_path.write_text(
-        json.dumps(summary, indent=2, sort_keys=True) + "\n",
-        encoding="utf-8",
-    )
-    worklist_path.write_text(
-        worklist_markdown(summary, input_directory), encoding="utf-8"
-    )
-    totals = summary["totals"]
-    print(
-        "audited {} BReps: {} okay; {} work items; reports in {}".format(
-            totals["breps"], totals["ok"],
-            totals["breps"] - totals["ok"] + len(database_errors),
-            output_directory,
-        )
-    )
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())

misc/tools/step_corpus_capture.py

@@ -1,127 +0,0 @@
-#!/usr/bin/env python3
-"""Capture standard swrast overview images for converted STEP databases.
-
-Each database is opened in its own gsh process so the display manager and GED
-state are isolated.  The two modes are written to separate directories and
-use the database filename (without its final ``.g``) as the image name.
-"""
-
-import argparse
-import json
-import os
-import re
-import subprocess
-from concurrent.futures import ThreadPoolExecutor
-from pathlib import Path
-
-
-def image_stem(path):
-    return path.name[:-2] if path.name.lower().endswith(".g") else path.stem
-
-
-def capture(gsh, model, output, mode, azimuth, elevation, timeout):
-    output.mkdir(parents=True, exist_ok=True)
-    image = output / (image_stem(model) + ".png")
-    try:
-        tops = subprocess.run(
-            [str(gsh), "--new-cmds", str(model), "tops"],
-            text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
-            timeout=timeout,
-        ).stdout.splitlines()
-    except subprocess.TimeoutExpired:
-        return {"model": str(model), "mode": mode, "image": str(image),
-                "returncode": 124, "complete": False,
-                "log": "timed out while listing top-level objects"}
-    tops = [line.strip().rstrip("/") for line in tops
-            if line.strip().endswith("/") and line.strip().count("/") == 1]
-    if not tops:
-        return {"model": str(model), "mode": mode, "image": str(image),
-                "returncode": 1, "complete": False,
-                "log": "database has no top-level objects"}
-    commands = [
-        "dm attach swrast SWDM",
-        "draw -m{} {}".format(mode, " ".join(tops)),
-        "autoview",
-        "ae {} {}".format(azimuth, elevation),
-        "screengrab -D SWDM {}".format(image),
-        "quit",
-    ]
-    try:
-        proc = subprocess.run(
-            [str(gsh), "--new-cmds", str(model)],
-            input="\n".join(commands) + "\n",
-            text=True,
-            stdout=subprocess.PIPE,
-            stderr=subprocess.STDOUT,
-            timeout=timeout,
-        )
-    except subprocess.TimeoutExpired as error:
-        return {"model": str(model), "mode": mode, "image": str(image),
-                "returncode": 124, "complete": False,
-                "log": str(error.stdout or "")[-4000:] + "\ntimed out during capture"}
-    return {
-        "model": str(model),
-        "mode": mode,
-        "image": str(image),
-        "returncode": proc.returncode,
-        "complete": proc.returncode == 0 and image.is_file() and image.stat().st_size > 0,
-        "log": proc.stdout[-4000:],
-    }
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("input_directory", type=Path)
-    parser.add_argument("output_directory", type=Path)
-    parser.add_argument("--gsh", type=Path, default=Path(".build/bin/gsh"))
-    parser.add_argument("--jobs", type=int, default=min(2, os.cpu_count() or 1))
-    parser.add_argument("--timeout", type=int, default=300,
-                        help="maximum seconds per model and view")
-    parser.add_argument("--mode", choices=("wireframe", "shaded", "both"), default="both")
-    parser.add_argument("--only", help="regular expression matched against model names")
-    parser.add_argument("--azimuth", type=float, default=35.0)
-    parser.add_argument("--elevation", type=float, default=25.0)
-    args = parser.parse_args()
-    if args.jobs < 1:
-        parser.error("--jobs must be positive")
-    if args.timeout < 1:
-        parser.error("--timeout must be positive")
-    if not args.gsh.is_file():
-        parser.error("gsh not found: {}".format(args.gsh))
-
-    matcher = re.compile(args.only) if args.only else None
-    models = sorted(path for path in args.input_directory.glob("*.g")
-                    if not matcher or matcher.search(path.name))
-    if not models:
-        parser.error("no converted .g models in {}".format(args.input_directory))
-    modes = [("wireframe", 0), ("shaded", 1)] if args.mode == "both" else [
-        (args.mode, 0 if args.mode == "wireframe" else 1)
-    ]
-    jobs = []
-    for name, mode in modes:
-        target = args.output_directory / name
-        jobs.extend((model, target, mode) for model in models)
-
-    with ThreadPoolExecutor(max_workers=args.jobs) as pool:
-        results = list(pool.map(
-            lambda item: capture(args.gsh, item[0], item[1], item[2],
-                                  args.azimuth, args.elevation, args.timeout), jobs
-        ))
-    args.output_directory.mkdir(parents=True, exist_ok=True)
-    manifest = {
-        "format": "brlcad-step-corpus-capture-v1",
-        "azimuth": args.azimuth,
-        "elevation": args.elevation,
-        "jobs": args.jobs,
-        "results": results,
-    }
-    (args.output_directory / "capture-manifest.json").write_text(
-        json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8"
-    )
-    failed = [item for item in results if not item["complete"]]
-    print("captured {}/{} images".format(len(results) - len(failed), len(results)))
-    return 1 if failed else 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

misc/tools/step_corpus_run.py

@@ -1,501 +0,0 @@
-#!/usr/bin/env python3
-"""Run the schema-dispatching BRL-CAD STEP importer over an external corpus.
-
-Unknown schemas are recorded in the run manifest but are not passed to the
-converter.  Each supported input gets a structured report, a diagnostic log,
-and a tab-separated status record consumed by ``step_corpus_summary.py``.
-Nested input directories are scanned recursively, and existing complete
-records may be resumed.
-"""
-
-import argparse
-import datetime
-import hashlib
-import json
-import os
-import platform
-import re
-import signal
-import subprocess
-import sys
-import time
-from pathlib import Path
-
-
-STEP_SUFFIXES = {".stp", ".step", ".p21"}
-INCOMPLETE_PROCESS_STATUSES = {124, 130, 137, 143}
-
-
-def schema_names(path):
-    with path.open("rb") as stream:
-        header = stream.read(2 * 1024 * 1024).decode("latin-1", errors="replace")
-    match = re.search(
-        r"FILE_SCHEMA\s*\(\s*\((.*?)\)\s*\)", header,
-        re.IGNORECASE | re.DOTALL,
-    )
-    if not match:
-        return []
-    return [
-        value.replace("''", "'").upper()
-        for value in re.findall(r"'((?:''|[^'])*)'", match.group(1))
-    ]
-
-
-def schema_family(names):
-    """Mirror the public host's supported-family and interim-profile routing."""
-    families = set()
-    legacy_ap203e2 = {
-        "CONFIGURATION_CONTROL_3D_DESIGN_MIM_LF",
-        "CCD_CLA_GVP_AST",
-        "CCD_CLA_GVP_AST_ASD",
-    }
-    interim_companions = (
-        "GEOMETRIC_VALIDATION_PROPERTIES_MIM",
-        "GEOMETRIC_VALIDATION_PROPERTY_REPRESENTATION_MIM",
-        "SHAPE_APPEARANCE_LAYER_MIM",
-        "SHAPE_APPEARANCE_LAYERS_MIM",
-    )
-    for name in names:
-        if "AP203_CONFIGURATION_CONTROLLED" in name or any(
-                name.startswith(alias) for alias in legacy_ap203e2):
-            families.add("AP203e2")
-        elif "AUTOMOTIVE_DESIGN" in name:
-            families.add("AP214")
-        elif "CONFIG_CONTROL_DESIGN" in name:
-            families.add("AP203")
-        elif "AP242" in name or "MANAGED_MODEL_BASED_3D_ENGINEERING" in name:
-            families.add("AP242")
-
-    if families == {"AP203"} and any(
-            name.startswith(companion)
-            for name in names for companion in interim_companions):
-        return "AP203e2"
-    return next(iter(families)) if len(families) == 1 else "unsupported"
-
-
-def artifact_stem(path, input_directory):
-    relative = path.relative_to(input_directory).as_posix()
-    readable = re.sub(r"[^A-Za-z0-9_.-]+", "_", relative)
-    # Sanitizing path separators and punctuation is not injective.  Retain a
-    # readable prefix while making nested and similarly named inputs safe.
-    identity = hashlib.sha256(relative.encode("utf-8")).hexdigest()[:12]
-    return "{}__{}_".format(readable, identity)
-
-
-def legacy_artifact_stem(path, input_directory):
-    """Return the pre-v2 stem so an in-progress corpus can be resumed."""
-    relative = path.relative_to(input_directory).as_posix()
-    return re.sub(r"[^A-Za-z0-9_.-]+", "_", relative) + "_"
-
-
-def completed_status(path):
-    try:
-        status = int(path.read_text(encoding="utf-8").split("\t", 1)[0])
-    except (OSError, ValueError):
-        return False
-    return status not in INCOMPLETE_PROCESS_STATUSES
-
-
-def synthetic_report(input_path, status, message):
-    return {
-        "format": "brlcad-step-import-report-v1",
-        "input": str(input_path),
-        "output": "",
-        "exit_status": status,
-        "coverage": {
-            "geometry_attempted": 0,
-            "geometry_written": 0,
-            "geometry_skipped": 0,
-        },
-        "validation": {
-            "invalid_breps": 0,
-            "output_failures": 0,
-            "repairs": 0,
-            "inferred_curves": 0,
-        },
-        "diagnostics": [{
-            "severity": "error",
-            "entity_id": 0,
-            "entity_type": "",
-            "attribute": "",
-            "message": message,
-            "count": 1,
-            "aggregated": False,
-        }],
-        "timings_us": {},
-        "peak_rss_bytes": 0,
-        "skipped_items": [],
-        "corpus_run_incomplete": True,
-    }
-
-
-def file_sha256(path):
-    digest = hashlib.sha256()
-    with path.open("rb") as stream:
-        for block in iter(lambda: stream.read(1024 * 1024), b""):
-            digest.update(block)
-    return digest.hexdigest()
-
-
-def executable_identity(path):
-    stat = path.stat()
-    return {
-        "path": str(path),
-        "size": stat.st_size,
-        "mtime_ns": stat.st_mtime_ns,
-        "sha256": file_sha256(path),
-    }
-
-
-def schema_plugin_paths(step_g):
-    """Locate build/install-tree schema modules used by the plugin host."""
-    plugin_directory = step_g.parent.parent / "libexec" / "step"
-    if not plugin_directory.is_dir():
-        return []
-    return sorted(
-        path for path in plugin_directory.glob("libstep-schema-*")
-        if path.is_file()
-    )
-
-
-def schema_plugin_identities(step_g):
-    """Identify build/install-tree schema modules used by the plugin host."""
-    return [executable_identity(path) for path in schema_plugin_paths(step_g)]
-
-
-def linked_library_identities(executables, runtime_root):
-    """Hash resolved runtime libraries supplied by the selected build tree.
-
-    Converter and plugin hashes do not detect a rebuilt geometry or STEPcode
-    shared library.  On platforms with ``ldd``, retain every resolved library
-    below the executable's build/install root so a long run cannot be mistaken
-    for a homogeneous baseline after one of those dependencies changes.
-    """
-    libraries = {}
-    for executable in executables:
-        try:
-            result = subprocess.run(
-                ["ldd", str(executable)], check=False, capture_output=True,
-                text=True,
-            )
-        except OSError:
-            return []
-        if result.returncode:
-            continue
-        for line in result.stdout.splitlines():
-            match = re.search(r"=>\s+(/\S+)", line)
-            if not match:
-                match = re.match(r"\s*(/\S+)", line)
-            if not match:
-                continue
-            path = Path(match.group(1)).resolve()
-            try:
-                path.relative_to(runtime_root)
-            except ValueError:
-                continue
-            if path.is_file():
-                libraries[str(path)] = path
-    return [executable_identity(libraries[name]) for name in sorted(libraries)]
-
-
-def host_snapshot():
-    """Capture lightweight run context without adding a monitoring dependency."""
-    snapshot = {
-        "utc": datetime.datetime.now(datetime.timezone.utc).replace(
-            microsecond=0
-        ).isoformat(),
-    }
-    try:
-        snapshot["load_average"] = list(os.getloadavg())
-    except (AttributeError, OSError):
-        pass
-    try:
-        values = {}
-        with Path("/proc/meminfo").open("r", encoding="ascii") as stream:
-            for line in stream:
-                key, value = line.split(":", 1)
-                if key in ("MemTotal", "MemAvailable"):
-                    values[key] = int(value.split()[0]) * 1024
-        if values:
-            snapshot["memory_bytes"] = values
-    except (OSError, ValueError):
-        pass
-    return snapshot
-
-
-def cpu_model():
-    try:
-        with Path("/proc/cpuinfo").open("r", encoding="ascii") as stream:
-            for line in stream:
-                if line.lower().startswith("model name"):
-                    return line.split(":", 1)[1].strip()
-    except (OSError, IndexError):
-        pass
-    return platform.processor()
-
-
-def write_manifest(path, manifest):
-    """Atomically preserve progress so an interrupted long run is auditable."""
-    temporary = path.with_suffix(path.suffix + ".tmp")
-    temporary.write_text(
-        json.dumps(manifest, indent=2, sort_keys=True) + "\n",
-        encoding="utf-8",
-    )
-    temporary.replace(path)
-
-
-def terminate_process(process, force=False):
-    try:
-        if os.name == "nt":
-            process.kill() if force else process.terminate()
-        else:
-            os.killpg(process.pid, signal.SIGKILL if force else signal.SIGTERM)
-    except ProcessLookupError:
-        pass
-
-
-def run_converter(command, log, timeout):
-    """Run one converter and terminate its complete process group on timeout."""
-    process = subprocess.Popen(
-        command, stdout=log, stderr=subprocess.STDOUT,
-        start_new_session=os.name != "nt",
-    )
-    try:
-        return process.wait(timeout=timeout if timeout > 0 else None), False
-    except subprocess.TimeoutExpired:
-        terminate_process(process)
-        try:
-            process.wait(timeout=30)
-        except subprocess.TimeoutExpired:
-            terminate_process(process, force=True)
-            process.wait()
-        return 124, True
-    except BaseException:
-        terminate_process(process)
-        try:
-            process.wait(timeout=30)
-        except subprocess.TimeoutExpired:
-            terminate_process(process, force=True)
-            process.wait()
-        raise
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("input_directory", type=Path)
-    parser.add_argument("output_directory", type=Path)
-    parser.add_argument("--step-g", type=Path, default=Path(".build/bin/step-g"))
-    parser.add_argument("-j", "--jobs", type=int, default=min(12, os.cpu_count() or 1),
-                        help="geometry workers inside each converter")
-    parser.add_argument("--previous", type=Path,
-                        help="previous corpus-summary.json for trend deltas")
-    parser.add_argument("--expectations", type=Path,
-                        help="checksum-pinned corpus expectation manifest")
-    parser.add_argument("--allow-missing-expectations", action="store_true",
-                        help="allow expectation entries absent from a filtered run")
-    parser.add_argument("--run-id", help="stable label for this run")
-    parser.add_argument("--resume", action="store_true")
-    parser.add_argument("--exact", action="store_true")
-    parser.add_argument("--timeout", type=float, default=0.0,
-                        help="outer wall-clock limit per file in seconds (0 disables)")
-    parser.add_argument("--stall-timeout", type=float,
-                        help="override step-g's no-progress timeout in seconds")
-    parser.add_argument(
-        "--only", help="regular expression matched against relative input paths"
-    )
-    parser.add_argument("--list", action="store_true",
-                        help="classify inputs without running converters")
-    args = parser.parse_args()
-
-    if args.jobs < 1 or args.timeout < 0:
-        parser.error("--jobs must be positive and --timeout must be nonnegative")
-    if args.stall_timeout is not None and args.stall_timeout <= 0:
-        parser.error("--stall-timeout must be positive")
-    input_directory = args.input_directory.resolve()
-    output_directory = args.output_directory.resolve()
-    output_directory.mkdir(parents=True, exist_ok=True)
-    matcher = re.compile(args.only) if args.only else None
-    inputs = sorted(
-        path for path in input_directory.rglob("*")
-        if path.is_file() and path.suffix.lower() in STEP_SUFFIXES and
-        (not matcher or matcher.search(path.relative_to(input_directory).as_posix()))
-    )
-    if not inputs:
-        parser.error("no matching STEP files in {}".format(input_directory))
-
-    run_id = args.run_id or datetime.datetime.now(
-        datetime.timezone.utc
-    ).strftime("%Y%m%dT%H%M%SZ")
-    step_g = args.step_g.resolve()
-    if not step_g.is_file():
-        parser.error("step-g not found: {}".format(step_g))
-    plugin_paths = schema_plugin_paths(step_g)
-    runtime_root = step_g.parent.parent.resolve()
-    manifest = {
-        "format": "brlcad-step-corpus-run-v1",
-        "run_id": run_id,
-        "started_utc": datetime.datetime.now(datetime.timezone.utc).replace(
-            microsecond=0
-        ).isoformat(),
-        "host": {
-            "hostname": platform.node(),
-            "platform": platform.platform(),
-            "machine": platform.machine(),
-            "cpu_model": cpu_model(),
-            "logical_cpus": os.cpu_count(),
-            "initial_snapshot": host_snapshot(),
-        },
-        "input_directory": str(input_directory),
-        "output_directory": str(output_directory),
-        "options": {
-            "jobs": args.jobs,
-            "exact": args.exact,
-            "recursive": True,
-            "timeout_seconds": args.timeout,
-            "stall_timeout_seconds": args.stall_timeout,
-        },
-        "step_g": executable_identity(step_g),
-        "schema_plugins": [executable_identity(path) for path in plugin_paths],
-        "linked_libraries": linked_library_identities(
-            [step_g] + plugin_paths, runtime_root
-        ),
-        "expectations": str(args.expectations.resolve()) if args.expectations else None,
-        "files": [],
-    }
-    manifest_path = output_directory / "run-manifest.json"
-    write_manifest(manifest_path, manifest)
-
-    for index, input_path in enumerate(inputs, 1):
-        relative_input = input_path.relative_to(input_directory).as_posix()
-        declared_schemas = schema_names(input_path)
-        declared_schema = declared_schemas[0] if declared_schemas else ""
-        family = schema_family(declared_schemas)
-        record = {
-            "input": str(input_path),
-            "declared_schema": declared_schema,
-            "declared_schemas": declared_schemas,
-            "schema": family,
-        }
-        if args.list:
-            print("{}\t{}\t{}".format(
-                family, ";".join(declared_schemas), relative_input
-            ))
-            manifest["files"].append(record)
-            write_manifest(manifest_path, manifest)
-            continue
-        input_stat = input_path.stat()
-        record["input_identity"] = {
-            "size": input_stat.st_size,
-            "mtime_ns": input_stat.st_mtime_ns,
-            "sha256": file_sha256(input_path),
-        }
-        if family == "unsupported":
-            record["result"] = "skipped"
-            record["reason"] = "unrecognized schema"
-            manifest["files"].append(record)
-            write_manifest(manifest_path, manifest)
-            print("[{}/{}] skip {} ({})".format(
-                index, len(inputs), relative_input, family
-            ))
-            continue
-
-        stem = artifact_stem(input_path, input_directory)
-        report_path = output_directory / (stem + ".json")
-        log_path = output_directory / (stem + ".log")
-        status_path = output_directory / (stem + ".status")
-        if args.resume:
-            legacy_stem = legacy_artifact_stem(input_path, input_directory)
-            legacy_report = output_directory / (legacy_stem + ".json")
-            legacy_status = output_directory / (legacy_stem + ".status")
-            if legacy_report.is_file() and completed_status(legacy_status):
-                stem = legacy_stem
-                report_path = legacy_report
-                log_path = output_directory / (stem + ".log")
-                status_path = legacy_status
-        if args.resume and report_path.is_file() and completed_status(status_path):
-            record["result"] = "resumed"
-            record["report"] = report_path.name
-            manifest["files"].append(record)
-            write_manifest(manifest_path, manifest)
-            print("[{}/{}] keep {}".format(index, len(inputs), relative_input))
-            continue
-
-        command = [
-            str(step_g), "-D", "-j", str(args.jobs),
-            "--report", str(report_path),
-        ]
-        if args.exact:
-            command.append("--exact")
-        if args.stall_timeout is not None:
-            command.extend(["--stall-timeout", str(args.stall_timeout)])
-        command.append(str(input_path))
-        print("[{}/{}] {} {}".format(index, len(inputs), family, relative_input),
-              flush=True)
-        record.update({"result": "running", "host_before": host_snapshot()})
-        manifest["files"].append(record)
-        write_manifest(manifest_path, manifest)
-        started = time.monotonic()
-        with log_path.open("wb") as log:
-            try:
-                status, timed_out = run_converter(command, log, args.timeout)
-                message = "converter exited without writing a structured report"
-                if timed_out:
-                    message = "converter exceeded the {:.1f}-second outer timeout".format(
-                        args.timeout
-                    )
-            except OSError as error:
-                status = 127
-                timed_out = False
-                message = "could not execute converter: {}".format(error)
-        elapsed = time.monotonic() - started
-        if not report_path.is_file():
-            report_path.write_text(
-                json.dumps(synthetic_report(input_path, status, message), indent=2) + "\n",
-                encoding="utf-8",
-            )
-        status_path.write_text(
-            "{}\t{:.3f}\t{}\t{}\n".format(status, elapsed, family, input_path),
-            encoding="utf-8",
-        )
-        record.update({
-            "result": "completed",
-            "process_exit_status": status,
-            "elapsed_seconds": elapsed,
-            "timed_out": timed_out,
-            "report": report_path.name,
-            "log": log_path.name,
-            "host_after": host_snapshot(),
-        })
-        write_manifest(manifest_path, manifest)
-
-    manifest["finished_utc"] = datetime.datetime.now(
-        datetime.timezone.utc
-    ).replace(microsecond=0).isoformat()
-    manifest["host"]["final_snapshot"] = host_snapshot()
-    write_manifest(manifest_path, manifest)
-    if not args.list:
-        summary_command = [
-            sys.executable,
-            str(Path(__file__).with_name("step_corpus_summary.py")),
-            str(output_directory),
-            "--run-id", run_id,
-        ]
-        if args.previous:
-            summary_command.extend(["--previous", str(args.previous.resolve())])
-        summary_status = subprocess.call(summary_command)
-        if summary_status or not args.expectations:
-            return summary_status
-        verify_command = [
-            sys.executable,
-            str(Path(__file__).with_name("step_corpus_verify.py")),
-            str(output_directory),
-            str(args.expectations.resolve()),
-        ]
-        if args.allow_missing_expectations:
-            verify_command.append("--allow-missing")
-        return subprocess.call(verify_command)
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())

misc/tools/step_corpus_summary.py

@@ -1,939 +0,0 @@
-#!/usr/bin/env python3
-"""Summarize one BRL-CAD STEP converter corpus run and compare it to another.
-
-The corpus runner writes one ``NAME.status`` record and one ``NAME.json``
-converter report per input.  A status record is tab separated::
-
-    exit_status<TAB>elapsed_seconds<TAB>schema<TAB>input_path
-
-This utility intentionally consumes the converters' structured reports rather
-than scraping diagnostics.  It writes a durable JSON record suitable for trend
-plots and a compact Markdown companion for review.
-"""
-
-import argparse
-import datetime
-import json
-import math
-import os
-import statistics
-import subprocess
-import sys
-from collections import Counter
-from pathlib import Path
-
-
-FORMAT = "brlcad-step-corpus-summary-v2"
-
-
-INCOMPLETE_PROCESS_STATUSES = {124, 130, 137, 143}
-
-
-def run_complete(item):
-    """Return whether an input reached a normal converter terminal state."""
-    if "run_complete" in item:
-        return bool(item["run_complete"])
-    return int(item.get("process_exit_status", 0)) not in INCOMPLETE_PROCESS_STATUSES
-
-
-def read_json(path):
-    with path.open("r", encoding="utf-8") as stream:
-        return json.load(stream)
-
-
-def git_state(start):
-    try:
-        revision = subprocess.check_output(
-            ["git", "-C", str(start), "rev-parse", "HEAD"],
-            stderr=subprocess.DEVNULL,
-            text=True,
-        ).strip()
-        dirty = bool(
-            subprocess.check_output(
-                ["git", "-C", str(start), "status", "--porcelain"],
-                stderr=subprocess.DEVNULL,
-                text=True,
-            ).strip()
-        )
-        return revision, dirty
-    except (OSError, subprocess.CalledProcessError):
-        return "unknown", None
-
-
-def failure_category(reason):
-    text = (reason or "").lower()
-    if any(word in text for word in ("budget", "expired", "timed out", "timeout")):
-        return "work_budget"
-    if any(word in text for word in (
-        "not solid", "non-solid", "solid validation", "validate as a solid"
-    )):
-        return "non_solid"
-    if "structural" in text:
-        return "structural"
-    if "exact" in text or "conversion failed" in text:
-        return "exact_conversion"
-    if "opennurbs" in text:
-        return "structural"
-    if "no importer" in text or "unsupported" in text:
-        return "unsupported"
-    if "output" in text or "database" in text or "write" in text:
-        return "output"
-    return "other"
-
-
-def parse_status(path):
-    fields = path.read_text(encoding="utf-8").rstrip("\n").split("\t", 3)
-    if len(fields) != 4:
-        raise ValueError("{} is not a four-field corpus status record".format(path))
-    return {
-        "process_exit_status": int(fields[0]),
-        "elapsed_seconds": float(fields[1]),
-        "schema": fields[2],
-        "input": fields[3],
-    }
-
-
-def occurrence_depth(occurrences):
-    """Return the maximum product-assembly depth represented by occurrences."""
-    children = {}
-    vertices = set()
-    for occurrence in occurrences:
-        parent = int(occurrence.get("parent_product_id") or 0)
-        child = int(occurrence.get("child_product_id") or 0)
-        if parent <= 0 or child <= 0:
-            continue
-        children.setdefault(parent, set()).add(child)
-        vertices.update((parent, child))
-
-    active = set()
-    memo = {}
-
-    def visit(vertex):
-        if vertex in memo:
-            return memo[vertex]
-        if vertex in active:
-            # A cycle is invalid, but retain a useful finite metric and let the
-            # converter's structured diagnostics report the underlying issue.
-            return 0
-        active.add(vertex)
-        depth = max((1 + visit(child)
-                     for child in children.get(vertex, ())), default=0)
-        active.remove(vertex)
-        memo[vertex] = depth
-        return depth
-
-    return max((visit(vertex) for vertex in vertices), default=0)
-
-
-def report_record(status_path):
-    status = parse_status(status_path)
-    report_path = status_path.with_suffix(".json")
-    if not report_path.is_file():
-        raise ValueError("missing report paired with {}".format(status_path))
-    report = read_json(report_path)
-    coverage = report.get("coverage") or {}
-    validation = report.get("validation") or {}
-    timings = report.get("timings_us") or {}
-    performance = report.get("performance") or {}
-    calibration = performance.get("calibration") or {}
-    stage_timings = performance.get("stages") or {}
-    skipped = report.get("skipped_items") or []
-    occurrences = report.get("occurrence_details") or []
-    configuration_records = report.get("configuration_records") or []
-    representation_coverage = report.get("representation_coverage") or []
-    attempted = int(coverage.get("geometry_attempted") or 0)
-    written = int(coverage.get("geometry_written") or 0)
-    skipped_count = int(coverage.get("geometry_skipped") or len(skipped))
-    categories = Counter(failure_category(item.get("reason")) for item in skipped)
-    invalid_breps = int(validation.get("invalid_breps") or 0)
-    invalid_breps_written = int(validation.get("invalid_breps_written") or 0)
-    invalid_breps_rejected = int(validation.get("invalid_breps_rejected") or 0)
-    output_failures = int(validation.get("output_failures") or 0)
-    inferred_curves = int(validation.get("inferred_curves") or 0)
-    if invalid_breps_written:
-        categories["invalid_brep_preserved"] += invalid_breps_written
-    if output_failures:
-        categories["output"] += output_failures
-    configuration_types = Counter(
-        item.get("type") or "UNKNOWN" for item in configuration_records
-    )
-    representation_statuses = Counter(
-        item.get("status") or "unknown" for item in representation_coverage
-    )
-    converter_status = int(
-        report.get("exit_status", status["process_exit_status"])
-    )
-    intentionally_empty = (
-        status["process_exit_status"] == 6 and converter_status == 6 and
-        attempted == 0 and bool(representation_coverage) and
-        all(item.get("status") == "intentionally_non_geometric"
-            for item in representation_coverage)
-    )
-    input_path = str(report.get("input") or status["input"])
-    result = {
-        "name": os.path.basename(input_path),
-        "input": input_path,
-        "schema": status["schema"],
-        "report": report_path.name,
-        "process_exit_status": status["process_exit_status"],
-        "converter_exit_status": converter_status,
-        "run_complete": status["process_exit_status"] not in
-                        INCOMPLETE_PROCESS_STATUSES,
-        "elapsed_seconds": status["elapsed_seconds"],
-        "geometry_attempted": attempted,
-        "geometry_written": written,
-        "geometry_skipped": skipped_count,
-        "success_fraction": (float(written) / attempted) if attempted else 0.0,
-        "products": int(coverage.get("products") or 0),
-        "occurrences": int(coverage.get("occurrences") or 0),
-        "assembly_usage_count": len(report.get("assembly_usages") or []),
-        "occurrence_detail_count": len(occurrences),
-        "occurrence_max_depth": occurrence_depth(occurrences),
-        "occurrence_methods": sorted({
-            item.get("shape_method") or "unknown" for item in occurrences
-        }),
-        "styles_extracted": int(coverage.get("styles_extracted") or 0),
-        "styles_applied": int(coverage.get("styles_applied") or 0),
-        "layers_extracted": int(coverage.get("layers_extracted") or 0),
-        "materials_extracted": int(coverage.get("materials_extracted") or 0),
-        "properties_extracted": int(coverage.get("properties_extracted") or 0),
-        "configuration_record_count": len(configuration_records),
-        "configuration_record_types": dict(sorted(configuration_types.items())),
-        "configuration_metadata_count": len(
-            report.get("configuration_metadata") or {}
-        ),
-        "product_alternative_count": len(
-            report.get("product_alternatives") or []
-        ),
-        "usage_substitute_count": len(report.get("usage_substitutes") or []),
-        "entity_type_counts": dict(sorted(
-            (coverage.get("entity_counts") or {}).items()
-        )),
-        "unsupported_entity_counts": dict(sorted(
-            (coverage.get("unsupported_counts") or {}).items()
-        )),
-        "representation_status_counts": dict(sorted(
-            representation_statuses.items()
-        )),
-        "clean": status["process_exit_status"] not in
-                 INCOMPLETE_PROCESS_STATUSES and attempted > 0 and
-                 written == attempted and
-                 converter_status == 0 and
-                 invalid_breps_written == 0 and output_failures == 0 and
-                 inferred_curves == 0,
-        "intentionally_empty": intentionally_empty,
-        "invalid_breps": invalid_breps,
-        "invalid_breps_written": invalid_breps_written,
-        "invalid_breps_rejected": invalid_breps_rejected,
-        "output_failures": output_failures,
-        "repairs": int(validation.get("repairs") or 0),
-        "inferred_curves": inferred_curves,
-        "tolerance_mm": validation.get("tolerance_mm"),
-        "peak_rss_bytes": int(report.get("peak_rss_bytes") or 0),
-        "load_us": int(timings.get("load") or 0),
-        "index_us": int(timings.get("index") or 0),
-        "inventory_us": int(timings.get("inventory") or 0),
-        "convert_us": int(timings.get("convert") or 0),
-        "calibration": {
-            "queries": int(calibration.get("queries") or 0),
-            "elapsed_us": int(calibration.get("elapsed_us") or 0),
-            "scalar_queries_per_second": float(
-                calibration.get("scalar_queries_per_second") or 0.0
-            ),
-            "parallel_queries_per_second": float(
-                calibration.get("parallel_queries_per_second") or 0.0
-            ),
-        },
-        "stage_timings_us": {
-            name: int(values.get("total_us") or 0)
-            for name, values in sorted(stage_timings.items())
-        },
-        "failure_categories": dict(sorted(categories.items())),
-        "skipped_items": [
-            {
-                "entity_id": item.get("entity_id"),
-                "entity_type": item.get("entity_type"),
-                "reason": item.get("reason"),
-                "category": failure_category(item.get("reason")),
-            }
-            for item in skipped
-        ],
-    }
-    return result
-
-
-def totals(files):
-    by_schema = {}
-    for schema in sorted(set(item["schema"] for item in files)):
-        by_schema[schema] = totals_base(
-            [item for item in files if item["schema"] == schema]
-        )
-    result = totals_base(files)
-    result["by_schema"] = by_schema
-    return result
-
-
-def totals_base(files):
-    completed = [item for item in files if run_complete(item)]
-    incomplete = [item for item in files if not run_complete(item)]
-    process_statuses = Counter(
-        str(item["process_exit_status"]) for item in completed
-    )
-    converter_statuses = Counter(
-        str(item["converter_exit_status"]) for item in completed
-    )
-    attempted = sum(item["geometry_attempted"] for item in completed)
-    written = sum(item["geometry_written"] for item in completed)
-    categories = Counter()
-    entity_types = Counter()
-    unsupported_types = Counter()
-    configuration_types = Counter()
-    representation_statuses = Counter()
-    occurrence_methods = set()
-    for item in completed:
-        categories.update(item["failure_categories"])
-        entity_types.update(item.get("entity_type_counts") or {})
-        unsupported_types.update(item.get("unsupported_entity_counts") or {})
-        configuration_types.update(item.get("configuration_record_types") or {})
-        representation_statuses.update(
-            item.get("representation_status_counts") or {}
-        )
-        occurrence_methods.update(item.get("occurrence_methods") or [])
-    return {
-        "files": len(files),
-        "completed_files": len(completed),
-        "inconclusive_files": len(incomplete),
-        "clean_files": sum(1 for item in completed if item["clean"]),
-        "intentionally_empty_files": sum(
-            1 for item in completed if item.get("intentionally_empty")
-        ),
-        "process_exit_status_counts": dict(sorted(process_statuses.items())),
-        "converter_exit_status_counts": dict(sorted(converter_statuses.items())),
-        "geometry_attempted": attempted,
-        "geometry_written": written,
-        "geometry_skipped": sum(item["geometry_skipped"] for item in completed),
-        "success_fraction": (float(written) / attempted) if attempted else 0.0,
-        "products": sum(item.get("products", 0) for item in completed),
-        "occurrences": sum(item.get("occurrences", 0) for item in completed),
-        "assembly_usage_count": sum(
-            item.get("assembly_usage_count", 0) for item in completed
-        ),
-        "occurrence_detail_count": sum(
-            item.get("occurrence_detail_count", 0) for item in completed
-        ),
-        "occurrence_max_depth": max(
-            (item.get("occurrence_max_depth", 0) for item in completed),
-            default=0,
-        ),
-        "occurrence_methods": sorted(occurrence_methods),
-        "styles_extracted": sum(
-            item.get("styles_extracted", 0) for item in completed
-        ),
-        "styles_applied": sum(
-            item.get("styles_applied", 0) for item in completed
-        ),
-        "layers_extracted": sum(
-            item.get("layers_extracted", 0) for item in completed
-        ),
-        "materials_extracted": sum(
-            item.get("materials_extracted", 0) for item in completed
-        ),
-        "properties_extracted": sum(
-            item.get("properties_extracted", 0) for item in completed
-        ),
-        "configuration_record_count": sum(
-            item.get("configuration_record_count", 0) for item in completed
-        ),
-        "configuration_metadata_count": sum(
-            item.get("configuration_metadata_count", 0) for item in completed
-        ),
-        "product_alternative_count": sum(
-            item.get("product_alternative_count", 0) for item in completed
-        ),
-        "usage_substitute_count": sum(
-            item.get("usage_substitute_count", 0) for item in completed
-        ),
-        "invalid_breps": sum(
-            item.get("invalid_breps", 0) for item in completed
-        ),
-        "invalid_breps_written": sum(
-            item.get("invalid_breps_written", 0) for item in completed
-        ),
-        "invalid_breps_rejected": sum(
-            item.get("invalid_breps_rejected", 0) for item in completed
-        ),
-        "output_failures": sum(
-            item.get("output_failures", 0) for item in completed
-        ),
-        "repairs": sum(item.get("repairs", 0) for item in completed),
-        "inferred_curves": sum(
-            item.get("inferred_curves", 0) for item in completed
-        ),
-        "elapsed_seconds": sum(item["elapsed_seconds"] for item in completed),
-        "load_seconds": sum(item.get("load_us", 0) for item in completed) / 1.0e6,
-        "index_seconds": sum(item.get("index_us", 0) for item in completed) / 1.0e6,
-        "inventory_seconds": sum(
-            item.get("inventory_us", 0) for item in completed
-        ) / 1.0e6,
-        "convert_seconds": sum(
-            item.get("convert_us", 0) for item in completed
-        ) / 1.0e6,
-        "inconclusive_elapsed_seconds": sum(
-            item["elapsed_seconds"] for item in incomplete
-        ),
-        "observed_inconclusive_geometry_attempted": sum(
-            item["geometry_attempted"] for item in incomplete
-        ),
-        "observed_inconclusive_geometry_written": sum(
-            item["geometry_written"] for item in incomplete
-        ),
-        "observed_inconclusive_geometry_skipped": sum(
-            item["geometry_skipped"] for item in incomplete
-        ),
-        "peak_file_rss_bytes": max(
-            (item["peak_rss_bytes"] for item in files), default=0
-        ),
-        "entity_type_counts": dict(sorted(entity_types.items())),
-        "unsupported_entity_counts": dict(sorted(unsupported_types.items())),
-        "configuration_record_types": dict(sorted(configuration_types.items())),
-        "representation_status_counts": dict(
-            sorted(representation_statuses.items())
-        ),
-        "failure_categories": dict(sorted(categories.items())),
-    }
-
-
-def positive_ratio(current, previous):
-    return (float(current) / float(previous)) if previous > 0 else None
-
-
-def median_ratio(current_files, previous_files, field):
-    ratios = [
-        positive_ratio(current.get(field, 0), previous.get(field, 0))
-        for current, previous in zip(current_files, previous_files)
-    ]
-    ratios = [value for value in ratios if value is not None and value > 0]
-    return statistics.median(ratios) if ratios else None
-
-
-def geometric_mean_ratio(current_files, previous_files, field):
-    ratios = [
-        positive_ratio(current.get(field, 0), previous.get(field, 0))
-        for current, previous in zip(current_files, previous_files)
-    ]
-    ratios = [value for value in ratios if value is not None and value > 0]
-    return math.exp(sum(math.log(value) for value in ratios) / len(ratios)) \
-        if ratios else None
-
-
-def performance_comparison(current_files, previous_files):
-    current = totals_base(current_files)
-    previous = totals_base(previous_files)
-    fields = {
-        "elapsed": "elapsed_seconds",
-        "load": "load_seconds",
-        "convert": "convert_seconds",
-        "peak_file_rss": "peak_file_rss_bytes",
-    }
-    aggregate_ratios = {
-        label: positive_ratio(current[field], previous[field])
-        for label, field in fields.items()
-    }
-    per_file_fields = {
-        "elapsed": "elapsed_seconds",
-        "load": "load_us",
-        "convert": "convert_us",
-        "peak_rss": "peak_rss_bytes",
-    }
-    return {
-        "comparable_files": len(current_files),
-        "current": {field: current[field] for field in fields.values()},
-        "previous": {field: previous[field] for field in fields.values()},
-        "aggregate_ratios": aggregate_ratios,
-        "median_file_ratios": {
-            label: median_ratio(current_files, previous_files, field)
-            for label, field in per_file_fields.items()
-        },
-        "geometric_mean_file_ratios": {
-            label: geometric_mean_ratio(current_files, previous_files, field)
-            for label, field in per_file_fields.items()
-        },
-    }
-
-
-def converter_options(configuration):
-    options = (configuration or {}).get("options") or {}
-    return {
-        "jobs": options.get("jobs"),
-        "exact": bool(options.get("exact", False)),
-        "stall_timeout_seconds": options.get("stall_timeout_seconds"),
-    }
-
-
-def converter_identity(configuration):
-    configuration = configuration or {}
-    executable = configuration.get("step_g") or {}
-    plugins = configuration.get("schema_plugins") or []
-    libraries = configuration.get("linked_libraries") or []
-    return {
-        "host": executable.get("sha256"),
-        "schema_plugins": [
-            (os.path.basename(item.get("path") or ""), item.get("sha256"))
-            for item in plugins
-        ],
-        "linked_libraries": [
-            (os.path.basename(item.get("path") or ""), item.get("sha256"))
-            for item in libraries
-        ],
-    }
-
-
-def host_identity(configuration):
-    host = (configuration or {}).get("host") or {}
-    return {
-        key: host.get(key)
-        for key in ("hostname", "platform", "machine", "cpu_model",
-                    "logical_cpus")
-    }
-
-
-def host_observations(manifest):
-    snapshots = []
-    host = manifest.get("host") or {}
-    for name in ("initial_snapshot", "final_snapshot"):
-        if isinstance(host.get(name), dict):
-            snapshots.append(host[name])
-    for item in manifest.get("files", []):
-        for name in ("host_before", "host_after"):
-            if isinstance(item.get(name), dict):
-                snapshots.append(item[name])
-
-    loads = [item.get("load_average") for item in snapshots]
-    loads = [item for item in loads if isinstance(item, list) and len(item) == 3]
-    available = [
-        (item.get("memory_bytes") or {}).get("MemAvailable")
-        for item in snapshots
-    ]
-    available = [int(value) for value in available if value is not None]
-    result = {"snapshot_count": len(snapshots)}
-    if loads:
-        labels = ("one_minute", "five_minutes", "fifteen_minutes")
-        result["load_average"] = {
-            label: {
-                "minimum": min(float(item[index]) for item in loads),
-                "mean": statistics.mean(float(item[index]) for item in loads),
-                "maximum": max(float(item[index]) for item in loads),
-            }
-            for index, label in enumerate(labels)
-        }
-    if available:
-        result["available_memory_bytes"] = {
-            "minimum": min(available),
-            "maximum": max(available),
-        }
-    return result
-
-
-def compare(current_files, previous, current_configuration=None,
-            previous_configuration=None):
-    if not previous:
-        return None
-    old_files = {item["input"]: item for item in previous.get("files", [])}
-    changes = []
-    comparable_current = []
-    comparable_previous = []
-    for current in current_files:
-        old = old_files.get(current["input"])
-        if not run_complete(current):
-            changes.append({
-                "input": current["input"],
-                "change": "inconclusive",
-                "process_exit_status": current["process_exit_status"],
-            })
-            continue
-        if not old:
-            changes.append({"input": current["input"], "change": "new"})
-            continue
-        if not run_complete(old):
-            changes.append({"input": current["input"],
-                            "change": "previous_inconclusive"})
-            continue
-        comparable_current.append(current)
-        comparable_previous.append(old)
-        written_delta = current["geometry_written"] - old.get("geometry_written", 0)
-        skipped_delta = current["geometry_skipped"] - old.get("geometry_skipped", 0)
-        inferred_delta = current.get("inferred_curves", 0) - \
-                         old.get("inferred_curves", 0)
-        old_skips = {
-            (item.get("entity_id"), item.get("entity_type"), item.get("reason"))
-            for item in old.get("skipped_items", [])
-        }
-        new_skips = {
-            (item.get("entity_id"), item.get("entity_type"), item.get("reason"))
-            for item in current.get("skipped_items", [])
-        }
-        if (written_delta or skipped_delta or inferred_delta or
-                bool(old.get("clean")) != current["clean"] or
-                old_skips != new_skips):
-            changes.append({
-                "input": current["input"],
-                "written_delta": written_delta,
-                "skipped_delta": skipped_delta,
-                "inferred_curves_delta": inferred_delta,
-                "became_clean": current["clean"] and not old.get("clean"),
-                "regressed_from_clean": bool(old.get("clean")) and not current["clean"],
-                "fixed_skips": [list(item) for item in sorted(old_skips - new_skips)],
-                "new_skips": [list(item) for item in sorted(new_skips - old_skips)],
-            })
-    previous_totals = totals(comparable_previous)
-    current_totals = totals(comparable_current)
-    input_hash_pairs = [
-        (current.get("input_sha256"), old.get("input_sha256"))
-        for current, old in zip(comparable_current, comparable_previous)
-    ]
-    input_hashes_verified = bool(input_hash_pairs) and all(
-        current and old and current == old
-        for current, old in input_hash_pairs
-    )
-    options_match = bool(
-        current_configuration and previous_configuration
-    ) and converter_options(current_configuration) == converter_options(
-        previous_configuration
-    )
-    hosts_match = bool(
-        current_configuration and previous_configuration
-    ) and host_identity(current_configuration) == host_identity(
-        previous_configuration
-    )
-    current_observations = (
-        current_configuration or {}
-    ).get("host_observations") or {}
-    previous_observations = (
-        previous_configuration or {}
-    ).get("host_observations") or {}
-    loads_recorded = bool(
-        current_observations.get("load_average") and
-        previous_observations.get("load_average")
-    )
-    return {
-        "previous_run_id": previous.get("run_id"),
-        "geometry_written_delta": current_totals["geometry_written"] -
-                                  int(previous_totals.get("geometry_written") or 0),
-        "geometry_skipped_delta": current_totals["geometry_skipped"] -
-                                  int(previous_totals.get("geometry_skipped") or 0),
-        "clean_files_delta": current_totals["clean_files"] -
-                             int(previous_totals.get("clean_files") or 0),
-        "regressed_clean_files": [
-            item["input"] for item in changes if item.get("regressed_from_clean")
-        ],
-        "newly_clean_files": [
-            item["input"] for item in changes if item.get("became_clean")
-        ],
-        "inconclusive_files": [
-            item["input"] for item in changes
-            if item.get("change") == "inconclusive"
-        ],
-        "configuration": {
-            "current_options": (current_configuration or {}).get("options"),
-            "previous_options": (previous_configuration or {}).get("options"),
-            "converter_options_match": options_match,
-            "host_identity_match": hosts_match,
-            "input_hashes_verified": input_hashes_verified,
-            "load_observations_recorded": loads_recorded,
-            "timing_context_comparable": options_match and hosts_match and
-                                         input_hashes_verified and
-                                         loads_recorded,
-            "executable_match": bool(
-                (current_configuration or {}).get("step_g") and
-                (previous_configuration or {}).get("step_g")
-            ) and converter_identity(current_configuration) ==
-                  converter_identity(previous_configuration),
-            "current_outer_timeout_seconds": (
-                (current_configuration or {}).get("options") or {}
-            ).get("timeout_seconds"),
-            "previous_outer_timeout_seconds": (
-                (previous_configuration or {}).get("options") or {}
-            ).get("timeout_seconds"),
-            "current_host_identity": host_identity(current_configuration),
-            "previous_host_identity": host_identity(previous_configuration),
-            "current_host_observations": current_observations,
-            "previous_host_observations": previous_observations,
-        },
-        "performance": performance_comparison(
-            comparable_current, comparable_previous
-        ),
-        "changes": changes,
-    }
-
-
-def run_configuration(run_directory):
-    path = Path(run_directory) / "run-manifest.json"
-    if not path.is_file():
-        return None
-    manifest = read_json(path)
-    return {
-        "manifest": str(path.resolve()),
-        "options": manifest.get("options"),
-        "step_g": manifest.get("step_g"),
-        "schema_plugins": manifest.get("schema_plugins"),
-        "linked_libraries": manifest.get("linked_libraries"),
-        "host": manifest.get("host"),
-        "host_observations": host_observations(manifest),
-        "started_utc": manifest.get("started_utc"),
-        "finished_utc": manifest.get("finished_utc"),
-    }
-
-
-def attach_input_identities(files, run_directory):
-    path = Path(run_directory) / "run-manifest.json"
-    if not path.is_file():
-        return
-    manifest = read_json(path)
-    identities = {
-        record.get("input"): record.get("input_identity") or {}
-        for record in manifest.get("files") or []
-    }
-    for item in files:
-        identity = identities.get(item["input"]) or {}
-        if identity.get("sha256"):
-            item["input_sha256"] = identity["sha256"]
-            item["input_size"] = identity.get("size")
-
-
-def markdown(summary):
-    total = summary["totals"]
-    def count_list(values):
-        return ", ".join(
-            "{} {}".format(count, name)
-            for name, count in sorted(
-                values.items(), key=lambda item: (-item[1], item[0])
-            )
-        )
-
-    lines = [
-        "# STEP corpus run {}".format(summary["run_id"]),
-        "",
-        "Revision: `{}`{}".format(
-            summary["revision"], " (dirty)" if summary.get("dirty") else ""
-        ),
-        "",
-        "| Schema | Files clean | Intentional empty | Geometry written | Success |",
-        "|---|---:|---:|---:|---:|",
-    ]
-    for schema, values in sorted(total["by_schema"].items()):
-        lines.append("| {} | {}/{} | {} | {}/{} | {:.2%} |".format(
-            schema, values["clean_files"], values["completed_files"],
-            values["intentionally_empty_files"],
-            values["geometry_written"], values["geometry_attempted"],
-            values["success_fraction"],
-        ))
-    lines.append("| Combined | {}/{} | {} | {}/{} | {:.2%} |".format(
-        total["clean_files"], total["completed_files"],
-        total["intentionally_empty_files"], total["geometry_written"],
-        total["geometry_attempted"], total["success_fraction"],
-    ))
-    lines.extend([
-        "",
-        "Assembly coverage: {} product(s), {} occurrence(s), maximum depth "
-        "{}, using {}.".format(
-            total["products"], total["occurrences"],
-            total["occurrence_max_depth"],
-            ", ".join(total["occurrence_methods"]) or "no occurrence method",
-        ),
-        "Presentation and metadata: {} style(s) extracted, {} applied, {} "
-        "layer(s), {} material(s), {} property value(s), and {} retained "
-        "configuration record(s).".format(
-            total["styles_extracted"], total["styles_applied"],
-            total["layers_extracted"], total["materials_extracted"],
-            total["properties_extracted"],
-            total["configuration_record_count"],
-        ),
-        "Validation: {} invalid B-Rep(s), {} preserved and {} rejected; {} "
-        "bounded repair(s) and {} tagged curve inference(s) applied.".format(
-            total["invalid_breps"], total["invalid_breps_written"],
-            total["invalid_breps_rejected"],
-            total["repairs"],
-            total["inferred_curves"],
-        ),
-    ])
-    if total["failure_categories"]:
-        lines.append("Geometry omission categories: {}.".format(
-            count_list(total["failure_categories"])
-        ))
-    lines.append("Converter terminal statuses: {}.".format(
-        count_list(total["converter_exit_status_counts"])
-    ))
-    if total["unsupported_entity_counts"]:
-        lines.append("Unsupported STEP entity inventory: {}.".format(
-            count_list(total["unsupported_entity_counts"])
-        ))
-    observations = ((summary.get("run_configuration") or {}).get(
-        "host_observations") or {})
-    one_minute = (observations.get("load_average") or {}).get("one_minute")
-    if one_minute:
-        lines.extend([
-            "",
-            "Host one-minute load over {} boundary samples: {:.2f} mean "
-            "({:.2f}-{:.2f}).".format(
-                observations["snapshot_count"], one_minute["mean"],
-                one_minute["minimum"], one_minute["maximum"],
-            ),
-        ])
-    if total["inconclusive_files"]:
-        lines.append("")
-        lines.append(
-            "Inconclusive (timed out or interrupted): {} file(s); partial "
-            "geometry is excluded from totals and deltas.".format(
-                total["inconclusive_files"]
-            )
-        )
-    lines.extend([
-        "", "## Per file", "",
-        "| Schema | Input | Status | Written | Skipped | Inferred | Reported load s | "
-        "Reported convert s | Wall s | Peak RSS MiB |",
-        "|---|---|---|---:|---:|---:|---:|---:|---:|---:|",
-    ])
-    for item in summary["files"]:
-        lines.append("| {} | {} | {} | {}/{} | {} | {} | {:.1f} | {:.1f} | {:.1f} | {:.1f} |".format(
-            item["schema"], item["name"].replace("|", "\\|"),
-            "clean" if item["clean"] else
-            ("intentional empty" if item.get("intentionally_empty") else
-             ("partial" if run_complete(item) else "inconclusive")),
-            item["geometry_written"], item["geometry_attempted"],
-            item["geometry_skipped"], item.get("inferred_curves", 0),
-            item["load_us"] / 1.0e6,
-            item["convert_us"] / 1.0e6, item["elapsed_seconds"],
-            item["peak_rss_bytes"] / (1024.0 * 1024.0),
-        ))
-    delta = summary.get("delta")
-    if delta:
-        lines.extend([
-            "", "## Delta", "",
-            "Written geometry: {:+d}; skipped geometry: {:+d}; clean files: {:+d}.".format(
-                delta["geometry_written_delta"], delta["geometry_skipped_delta"],
-                delta["clean_files_delta"],
-            ),
-        ])
-        if delta["regressed_clean_files"]:
-            lines.append("Regressed clean files: {}.".format(
-                ", ".join(delta["regressed_clean_files"])
-            ))
-        if delta["inconclusive_files"]:
-            lines.append("Not compared because the run was incomplete: {}.".format(
-                ", ".join(delta["inconclusive_files"])
-            ))
-        performance = delta.get("performance") or {}
-        ratios = performance.get("aggregate_ratios") or {}
-        if performance.get("comparable_files"):
-            lines.extend([
-                "",
-                "Performance over {} comparable files (current/previous): "
-                "wall {:.3f}x and peak-file RSS {:.3f}x. Reported phase "
-                "ratios are load {:.3f}x and conversion {:.3f}x; phase "
-                "boundaries may differ across binaries.".format(
-                    performance["comparable_files"],
-                    ratios.get("elapsed") or 0.0,
-                    ratios.get("peak_file_rss") or 0.0,
-                    ratios.get("load") or 0.0,
-                    ratios.get("convert") or 0.0,
-                ),
-            ])
-        configuration = delta.get("configuration") or {}
-        if not configuration.get("converter_options_match"):
-            lines.append(
-                "Run options differ or could not be verified; performance "
-                "ratios are informational only."
-            )
-        if not configuration.get("host_identity_match"):
-            lines.append(
-                "Host identity differs or could not be verified; timing "
-                "ratios are not controlled benchmarks."
-            )
-        if not configuration.get("input_hashes_verified"):
-            lines.append(
-                "Input hashes differ or were not recorded in both runs; "
-                "timing ratios are informational only."
-            )
-        if not configuration.get("load_observations_recorded"):
-            lines.append(
-                "Both runs did not record host-load observations; "
-                "timing ratios are not controlled benchmarks."
-            )
-        else:
-            current_load = configuration["current_host_observations"][
-                "load_average"
-            ]["one_minute"]["mean"]
-            previous_load = configuration["previous_host_observations"][
-                "load_average"
-            ]["one_minute"]["mean"]
-            lines.append(
-                "Recorded one-minute load mean was {:.2f} current versus "
-                "{:.2f} previous.".format(current_load, previous_load)
-            )
-        if configuration.get("timing_context_comparable"):
-            lines.append(
-                "Host, input hashes, converter options, and host-load "
-                "telemetry are comparable; unrecorded background activity "
-                "may still affect timings."
-            )
-    return "\n".join(lines) + "\n"
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("run_directory", type=Path)
-    parser.add_argument("--previous", type=Path,
-                        help="previous corpus-summary.json for delta reporting")
-    parser.add_argument("--output", type=Path,
-                        help="output JSON (default: RUN_DIRECTORY/corpus-summary.json)")
-    parser.add_argument("--run-id", help="stable run label; defaults to directory name")
-    parser.add_argument("--no-markdown", action="store_true")
-    args = parser.parse_args()
-
-    run_directory = args.run_directory.resolve()
-    status_paths = sorted(run_directory.glob("*.status"))
-    if not status_paths:
-        parser.error("{} contains no *.status corpus records".format(run_directory))
-    try:
-        files = [report_record(path) for path in status_paths]
-    except (OSError, ValueError, json.JSONDecodeError) as error:
-        print("step_corpus_summary.py: {}".format(error), file=sys.stderr)
-        return 2
-    attach_input_identities(files, run_directory)
-
-    revision, dirty = git_state(Path(__file__).resolve().parents[2])
-    previous = read_json(args.previous) if args.previous else None
-    current_configuration = run_configuration(run_directory)
-    previous_configuration = run_configuration(
-        previous.get("run_directory")
-    ) if previous and previous.get("run_directory") else None
-    generated = datetime.datetime.now(datetime.timezone.utc).replace(
-        microsecond=0
-    ).isoformat()
-    summary = {
-        "format": FORMAT,
-        "run_id": args.run_id or run_directory.name,
-        "generated_utc": generated,
-        "run_directory": str(run_directory),
-        "revision": revision,
-        "dirty": dirty,
-        "run_configuration": current_configuration,
-        "totals": totals(files),
-        "files": files,
-    }
-    summary["delta"] = compare(
-        files, previous, current_configuration, previous_configuration
-    )
-
-    output = args.output or (run_directory / "corpus-summary.json")
-    output.parent.mkdir(parents=True, exist_ok=True)
-    with output.open("w", encoding="utf-8") as stream:
-        json.dump(summary, stream, indent=2, sort_keys=True)
-        stream.write("\n")
-    if not args.no_markdown:
-        output.with_suffix(".md").write_text(markdown(summary), encoding="utf-8")
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())

misc/tools/step_corpus_verify.py

@@ -1,288 +0,0 @@
-#!/usr/bin/env python3
-"""Verify a recorded STEP corpus run against checksum-pinned expectations.
-
-External STEP samples are often valuable but not suitable for copying into
-BRL-CAD.  This tool keeps their evidence reproducible without copying the
-files: an expectation manifest records the source, accepted local filenames,
-SHA-256 digest, and report assertions.  The verifier consumes the structured
-reports and run manifest produced by ``step_corpus_run.py``.
-"""
-
-import argparse
-import hashlib
-import json
-import sys
-from pathlib import Path
-
-
-FORMAT = "brlcad-step-corpus-verification-v1"
-
-
-def read_json(path):
-    with path.open("r", encoding="utf-8") as stream:
-        return json.load(stream)
-
-
-def file_sha256(path):
-    digest = hashlib.sha256()
-    with path.open("rb") as stream:
-        for block in iter(lambda: stream.read(1024 * 1024), b""):
-            digest.update(block)
-    return digest.hexdigest()
-
-
-def dotted_value(document, path):
-    value = document
-    for component in path.split("."):
-        if not isinstance(value, dict) or component not in value:
-            raise KeyError(path)
-        value = value[component]
-    return value
-
-
-def occurrence_depth(occurrences):
-    children = {}
-    vertices = set()
-    for occurrence in occurrences:
-        parent = int(occurrence.get("parent_product_id") or 0)
-        child = int(occurrence.get("child_product_id") or 0)
-        if parent <= 0 or child <= 0:
-            continue
-        children.setdefault(parent, set()).add(child)
-        vertices.update((parent, child))
-
-    active = set()
-    memo = {}
-
-    def visit(vertex):
-        if vertex in memo:
-            return memo[vertex]
-        if vertex in active:
-            raise ValueError("occurrence product graph contains a cycle")
-        active.add(vertex)
-        depth = max((1 + visit(child) for child in children.get(vertex, ())),
-                    default=0)
-        active.remove(vertex)
-        memo[vertex] = depth
-        return depth
-
-    return max((visit(vertex) for vertex in vertices), default=0)
-
-
-def derived_values(report):
-    occurrences = report.get("occurrence_details") or []
-    metadata = report.get("product_metadata") or []
-    materials = [
-        material
-        for product in metadata
-        for material in product.get("materials") or []
-    ]
-    material_properties = [
-        prop
-        for material in materials
-        for prop in material.get("properties") or []
-    ]
-    material_numeric_values = sorted({
-        value
-        for prop in material_properties
-        for value in prop.get("values") or []
-        if isinstance(value, (int, float))
-    })
-    return {
-        "assembly_usage_count": len(report.get("assembly_usages") or []),
-        "occurrence_detail_count": len(occurrences),
-        "occurrence_max_depth": occurrence_depth(occurrences),
-        "occurrence_methods": sorted({
-            item.get("shape_method") or "" for item in occurrences
-        }),
-        "skipped_entity_ids": sorted({
-            int(item.get("entity_id") or 0)
-            for item in report.get("skipped_items") or []
-            if int(item.get("entity_id") or 0) > 0
-        }),
-        "product_metadata_count": len(metadata),
-        "material_count": len(materials),
-        "material_identifiers": sorted({
-            material.get("identifier") or "" for material in materials
-        }),
-        "material_names": sorted({
-            material.get("name") or "" for material in materials
-        }),
-        "material_property_count": len(material_properties),
-        "material_property_names": sorted({
-            prop.get("name") or "" for prop in material_properties
-        }),
-        "material_property_units": sorted({
-            prop.get("units") or "" for prop in material_properties
-            if prop.get("units")
-        }),
-        "material_property_numeric_values": material_numeric_values,
-        "validation_property_count": sum(
-            len(product.get("validation_properties") or [])
-            for product in metadata
-        ),
-    }
-
-
-def compare_expected(report, expected):
-    combined = dict(report)
-    combined["derived"] = derived_values(report)
-    checks = []
-    errors = []
-    for path, wanted in sorted((expected.get("equals") or {}).items()):
-        try:
-            actual = dotted_value(combined, path)
-        except KeyError:
-            errors.append("{} is missing".format(path))
-            continue
-        checks.append(path)
-        if actual != wanted:
-            errors.append("{} is {!r}, expected {!r}".format(
-                path, actual, wanted
-            ))
-    for path, wanted in sorted((expected.get("minimum") or {}).items()):
-        try:
-            actual = dotted_value(combined, path)
-        except KeyError:
-            errors.append("{} is missing".format(path))
-            continue
-        checks.append(path)
-        if not isinstance(actual, (int, float)) or actual < wanted:
-            errors.append("{} is {!r}, expected at least {!r}".format(
-                path, actual, wanted
-            ))
-    return checks, errors, combined["derived"]
-
-
-def markdown(result):
-    lines = [
-        "# STEP external corpus verification",
-        "",
-        "Expectation manifest: `{}`".format(result["expectations"]),
-        "",
-        "| Input | Source/producer | Result | Checks |",
-        "|---|---|---:|---:|",
-    ]
-    for item in result["files"]:
-        source = item.get("producer") or item.get("source_page") or ""
-        lines.append("| {} | {} | {} | {} |".format(
-            item["name"].replace("|", "\\|"),
-            source.replace("|", "\\|"),
-            item["result"], len(item.get("checks") or []),
-        ))
-        for error in item.get("errors") or []:
-            lines.append("  - **{}:** {}".format(item["name"], error))
-    lines.extend([
-        "",
-        "Verified {}/{} present files; {} manifest entries were missing.".format(
-            result["passed_files"], result["present_files"],
-            result["missing_files"],
-        ),
-    ])
-    return "\n".join(lines) + "\n"
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("run_directory", type=Path)
-    parser.add_argument("expectations", type=Path)
-    parser.add_argument("--allow-missing", action="store_true",
-                        help="do not fail for expectation entries absent from the run")
-    parser.add_argument("--output", type=Path,
-                        help="output JSON (default: RUN/corpus-verification.json)")
-    args = parser.parse_args()
-
-    run_directory = args.run_directory.resolve()
-    manifest_path = run_directory / "run-manifest.json"
-    if not manifest_path.is_file():
-        parser.error("{} does not contain run-manifest.json".format(run_directory))
-    expectations_path = args.expectations.resolve()
-    run = read_json(manifest_path)
-    expectations = read_json(expectations_path)
-    if expectations.get("format") != "brlcad-step-corpus-expectations-v1":
-        parser.error("unsupported expectation manifest format")
-
-    run_files = {}
-    for record in run.get("files") or []:
-        input_name = Path(record.get("input") or "").name
-        if input_name:
-            run_files[input_name] = record
-
-    results = []
-    passed = 0
-    present = 0
-    missing = 0
-    for expectation in expectations.get("files") or []:
-        names = [expectation["name"]] + list(expectation.get("aliases") or [])
-        record = next((run_files[name] for name in names if name in run_files), None)
-        item = {
-            "name": expectation["name"],
-            "source_page": expectation.get("source_page"),
-            "producer": expectation.get("producer"),
-            "checks": [],
-            "errors": [],
-        }
-        if record is None:
-            missing += 1
-            item["result"] = "missing"
-            if not args.allow_missing:
-                item["errors"].append("input was not present in the corpus run")
-            results.append(item)
-            continue
-
-        present += 1
-        input_path = Path(record["input"])
-        actual_hash = file_sha256(input_path) if input_path.is_file() else None
-        item["input"] = str(input_path)
-        item["sha256"] = actual_hash
-        item["checks"].append("sha256")
-        recorded_hash = (record.get("input_identity") or {}).get("sha256")
-        if recorded_hash and recorded_hash != actual_hash:
-            item["errors"].append(
-                "input changed after the run manifest recorded its SHA-256"
-            )
-        if actual_hash != expectation.get("sha256"):
-            item["errors"].append("SHA-256 is {}, expected {}".format(
-                actual_hash, expectation.get("sha256")
-            ))
-
-        report_name = record.get("report")
-        report_path = run_directory / report_name if report_name else None
-        if not report_path or not report_path.is_file():
-            item["errors"].append("structured converter report is missing")
-        else:
-            report = read_json(report_path)
-            checks, errors, derived = compare_expected(
-                report, expectation.get("expected") or {}
-            )
-            item["checks"].extend(checks)
-            item["errors"].extend(errors)
-            item["derived"] = derived
-            item["report"] = str(report_path)
-
-        item["result"] = "pass" if not item["errors"] else "fail"
-        if item["result"] == "pass":
-            passed += 1
-        results.append(item)
-
-    failed = sum(1 for item in results if item.get("errors"))
-    result = {
-        "format": FORMAT,
-        "expectations": str(expectations_path),
-        "run_directory": str(run_directory),
-        "passed_files": passed,
-        "present_files": present,
-        "missing_files": missing,
-        "failed_files": failed,
-        "files": results,
-    }
-    output = args.output or (run_directory / "corpus-verification.json")
-    output.parent.mkdir(parents=True, exist_ok=True)
-    output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n",
-                      encoding="utf-8")
-    output.with_suffix(".md").write_text(markdown(result), encoding="utf-8")
-    return 1 if failed else 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())

misc/tools/step_dependency_subset.py

@@ -1,449 +0,0 @@
-#!/usr/bin/env python3
-"""Extract a small, forward-complete STEP geometry subset.
-
-This helper is intended for local diagnosis and for identifying the minimal
-geometry that a project-owned synthetic regression test must reproduce.  It
-copies the dependency closure of one or more STEP face entities and wraps them
-in an OPEN_SHELL product, retaining the source entity numbers so converter
-diagnostics remain directly comparable to the original file.  A diagnostic
---closed-shell mode instead uses CLOSED_SHELL/MANIFOLD_SOLID_BREP so reductions
-can exercise topology rules which only apply to asserted solids.  A face subset
-is not necessarily a valid closed shell, so that mode is for failure reduction,
-not evidence that the extracted result is valid geometry.  The output keeps the
-input model's provenance and is not automatically suitable for committing or
-redistribution.
-"""
-
-import argparse
-import re
-from pathlib import Path
-
-
-ADVANCED_FACE = re.compile(
-    r"^[ \t]*#[0-9]+[ \t]*=[ \t]*ADVANCED_FACE[ \t]*\(",
-    re.IGNORECASE,
-)
-FILE_SCHEMA = re.compile(
-    r"FILE_SCHEMA\s*\(\s*\(\s*'([^']+)'", re.IGNORECASE | re.DOTALL
-)
-ENTITY_TYPE = re.compile(
-    r"^[ \t]*#[0-9]+[ \t]*=[ \t]*([A-Z0-9_]+)[ \t]*\(",
-    re.IGNORECASE,
-)
-
-
-def instance_end(text, start):
-    """Return the first record-ending semicolon outside strings/comments."""
-    in_string = False
-    in_comment = False
-    offset = start
-    while offset < len(text):
-        current = text[offset]
-        following = text[offset + 1] if offset + 1 < len(text) else ""
-        if in_comment:
-            if current == "*" and following == "/":
-                in_comment = False
-                offset += 2
-                continue
-        elif in_string:
-            if current == "'":
-                if following == "'":
-                    offset += 2
-                    continue
-                in_string = False
-        elif current == "/" and following == "*":
-            in_comment = True
-            offset += 2
-            continue
-        elif current == "'":
-            in_string = True
-        elif current == ";":
-            return offset + 1
-        offset += 1
-    raise ValueError("unterminated STEP instance starting at byte {}".format(start))
-
-
-def instances(text):
-    """Return Part 21 instances without recognizing text in strings/comments."""
-    result = {}
-    offset = 0
-    line_start = True
-    in_string = False
-    in_comment = False
-    while offset < len(text):
-        current = text[offset]
-        following = text[offset + 1] if offset + 1 < len(text) else ""
-        if in_comment:
-            if current == "*" and following == "/":
-                in_comment = False
-                offset += 2
-                continue
-        elif in_string:
-            if current == "'":
-                if following == "'":
-                    offset += 2
-                    continue
-                in_string = False
-        elif current == "/" and following == "*":
-            in_comment = True
-            offset += 2
-            continue
-        elif current == "'":
-            in_string = True
-        elif current == "\n" or current == "\r":
-            line_start = True
-        elif line_start and current in " \t":
-            pass
-        elif line_start and current == "#":
-            digit_end = offset + 1
-            while digit_end < len(text) and text[digit_end].isdigit():
-                digit_end += 1
-            separator = digit_end
-            while separator < len(text) and text[separator] in " \t":
-                separator += 1
-            if digit_end > offset + 1 and separator < len(text) and \
-                    text[separator] == "=":
-                entity_id = int(text[offset + 1:digit_end])
-                end = instance_end(text, offset)
-                if entity_id in result:
-                    raise ValueError(
-                        "duplicate STEP entity #{}".format(entity_id)
-                    )
-                result[entity_id] = text[offset:end].strip()
-                offset = end
-                line_start = False
-                continue
-            line_start = False
-        else:
-            line_start = False
-        offset += 1
-    return result
-
-
-def references(record):
-    """Yield entity references outside Part 21 strings and comments."""
-    offset = 0
-    in_string = False
-    in_comment = False
-    while offset < len(record):
-        current = record[offset]
-        following = record[offset + 1] if offset + 1 < len(record) else ""
-        if in_comment:
-            if current == "*" and following == "/":
-                in_comment = False
-                offset += 2
-                continue
-        elif in_string:
-            if current == "'":
-                if following == "'":
-                    offset += 2
-                    continue
-                in_string = False
-        elif current == "/" and following == "*":
-            in_comment = True
-            offset += 2
-            continue
-        elif current == "'":
-            in_string = True
-        elif current == "#":
-            digit_end = offset + 1
-            while digit_end < len(record) and record[digit_end].isdigit():
-                digit_end += 1
-            if digit_end > offset + 1:
-                yield int(record[offset + 1:digit_end])
-                offset = digit_end
-                continue
-        offset += 1
-
-
-def dependency_closure(records, roots):
-    pending = list(roots)
-    selected = set()
-    while pending:
-        entity_id = pending.pop()
-        if entity_id in selected:
-            continue
-        record = records.get(entity_id)
-        if record is None:
-            raise ValueError("STEP entity #{} is not present".format(entity_id))
-        selected.add(entity_id)
-        for referenced_id in references(record):
-            if referenced_id != entity_id and referenced_id not in selected:
-                pending.append(referenced_id)
-    return selected
-
-
-def record_type(record):
-    match = ENTITY_TYPE.match(record)
-    return match.group(1).upper() if match else ""
-
-
-def adjacent_faces(records, faces):
-    """Return ADVANCED_FACE entities sharing an EDGE_CURVE with faces."""
-    reverse = {}
-    for entity_id, record in records.items():
-        for referenced_id in references(record):
-            reverse.setdefault(referenced_id, set()).add(entity_id)
-
-    edge_curves = {
-        entity_id
-        for entity_id in dependency_closure(records, faces)
-        if record_type(records[entity_id]) == "EDGE_CURVE"
-    }
-    oriented_edges = {
-        entity_id
-        for edge_id in edge_curves
-        for entity_id in reverse.get(edge_id, ())
-        if record_type(records[entity_id]) == "ORIENTED_EDGE"
-    }
-    edge_loops = {
-        entity_id
-        for oriented_id in oriented_edges
-        for entity_id in reverse.get(oriented_id, ())
-        if record_type(records[entity_id]) == "EDGE_LOOP"
-    }
-    face_bounds = {
-        entity_id
-        for loop_id in edge_loops
-        for entity_id in reverse.get(loop_id, ())
-        if record_type(records[entity_id]) in {
-            "FACE_BOUND", "FACE_OUTER_BOUND"
-        }
-    }
-    return {
-        entity_id
-        for bound_id in face_bounds
-        for entity_id in reverse.get(bound_id, ())
-        if record_type(records[entity_id]) == "ADVANCED_FACE"
-    }
-
-
-def quoted(value):
-    return value.replace("'", "''")
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("input", type=Path)
-    parser.add_argument("output", type=Path)
-    parser.add_argument(
-        "--face",
-        type=int,
-        action="append",
-        default=[],
-        help="ADVANCED_FACE entity ID to retain (repeatable)",
-    )
-    parser.add_argument(
-        "--face-range",
-        action="append",
-        default=[],
-        metavar="FIRST:LAST",
-        help="inclusive contiguous ADVANCED_FACE ID range to retain (repeatable)",
-    )
-    parser.add_argument(
-        "--face-neighbor-depth",
-        type=int,
-        default=0,
-        metavar="N",
-        help=(
-            "also retain ADVANCED_FACE neighbors reachable through N layers "
-            "of shared EDGE_CURVEs"
-        ),
-    )
-    parser.add_argument("--name", default="reduced_step_regression")
-    parser.add_argument(
-        "--context",
-        type=int,
-        help=(
-            "source GEOMETRIC_REPRESENTATION_CONTEXT entity ID to preserve; "
-            "recommended when source angle or length units are not SI defaults"
-        ),
-    )
-    parser.add_argument(
-        "--uncertainty",
-        type=float,
-        default=1.0e-6,
-        help="fixture uncertainty in millimetres (default: 1e-6)",
-    )
-    parser.add_argument(
-        "--closed-shell",
-        action="store_true",
-        help=(
-            "wrap the face subset as CLOSED_SHELL/MANIFOLD_SOLID_BREP "
-            "instead of OPEN_SHELL/SHELL_BASED_SURFACE_MODEL"
-        ),
-    )
-    args = parser.parse_args()
-    faces = list(args.face)
-    for face_range in args.face_range:
-        fields = face_range.split(":", 1)
-        if len(fields) != 2:
-            parser.error("--face-range must be FIRST:LAST")
-        try:
-            first, last = (int(field) for field in fields)
-        except ValueError:
-            parser.error("--face-range must contain integer entity IDs")
-        if first > last:
-            parser.error("--face-range FIRST must not exceed LAST")
-        faces.extend(range(first, last + 1))
-    faces = sorted(set(faces))
-    if not faces:
-        parser.error("at least one --face or --face-range is required")
-    if args.face_neighbor_depth < 0:
-        parser.error("--face-neighbor-depth must be nonnegative")
-
-    source = args.input.read_text(encoding="latin-1")
-    records = instances(source)
-    for entity_id in faces:
-        record = records.get(entity_id)
-        if record is None:
-            parser.error("STEP entity #{} is not present".format(entity_id))
-        if not ADVANCED_FACE.match(record):
-            parser.error(
-                "STEP entity #{} is not an ADVANCED_FACE".format(entity_id)
-            )
-    expanded_faces = set(faces)
-    frontier = set(faces)
-    for _ in range(args.face_neighbor_depth):
-        neighbors = adjacent_faces(records, frontier) - expanded_faces
-        if not neighbors:
-            break
-        expanded_faces.update(neighbors)
-        frontier = neighbors
-    faces = sorted(expanded_faces)
-    closure_roots = list(faces)
-    if args.context is not None:
-        closure_roots.append(args.context)
-    selected = dependency_closure(records, closure_roots)
-    schema_match = FILE_SCHEMA.search(source)
-    schema = (
-        schema_match.group(1)
-        if schema_match
-        else "AUTOMOTIVE_DESIGN { 1 0 10303 214 3 1 1 1 }"
-    )
-
-    next_id = max(records) + 1
-    wrapper_ids = list(range(next_id, next_id + 18))
-    (
-        shell_id,
-        model_id,
-        length_id,
-        angle_id,
-        solid_angle_id,
-        uncertainty_id,
-        context_id,
-        application_id,
-        protocol_id,
-        product_context_id,
-        definition_context_id,
-        product_id,
-        formation_id,
-        definition_id,
-        definition_shape_id,
-        representation_id,
-        shape_definition_id,
-        _unused_id,
-    ) = wrapper_ids
-
-    safe_name = quoted(args.name)
-    face_refs = ",".join("#{}".format(entity_id) for entity_id in faces)
-    if args.closed_shell:
-        wrapper = [
-            "#{}=CLOSED_SHELL('{} shell',({}));".format(
-                shell_id, safe_name, face_refs
-            ),
-            "#{}=MANIFOLD_SOLID_BREP('{} model',#{});".format(
-                model_id, safe_name, shell_id
-            ),
-        ]
-    else:
-        wrapper = [
-            "#{}=OPEN_SHELL('{} shell',({}));".format(
-                shell_id, safe_name, face_refs
-            ),
-            "#{}=SHELL_BASED_SURFACE_MODEL('{} model',(#{}));".format(
-                model_id, safe_name, shell_id
-            ),
-        ]
-    if args.context is None:
-        wrapper.extend(
-            [
-                "#{}=(LENGTH_UNIT() NAMED_UNIT(*) "
-                "SI_UNIT(.MILLI.,.METRE.));".format(length_id),
-                "#{}=(NAMED_UNIT(*) PLANE_ANGLE_UNIT() "
-                "SI_UNIT($,.RADIAN.));".format(angle_id),
-                "#{}=(NAMED_UNIT(*) SI_UNIT($,.STERADIAN.) "
-                "SOLID_ANGLE_UNIT());".format(solid_angle_id),
-                "#{}=UNCERTAINTY_MEASURE_WITH_UNIT(LENGTH_MEASURE({:.17g}),"
-                "#{},'distance_accuracy_value','maximum gap value');".format(
-                    uncertainty_id, args.uncertainty, length_id
-                ),
-                "#{}=(GEOMETRIC_REPRESENTATION_CONTEXT(3) "
-                "GLOBAL_UNCERTAINTY_ASSIGNED_CONTEXT((#{})) "
-                "GLOBAL_UNIT_ASSIGNED_CONTEXT((#{},#{},#{})) "
-                "REPRESENTATION_CONTEXT('fixture','3D'));".format(
-                    context_id,
-                    uncertainty_id,
-                    length_id,
-                    angle_id,
-                    solid_angle_id,
-                ),
-            ]
-        )
-    representation_context_id = (
-        args.context if args.context is not None else context_id
-    )
-    wrapper.extend(
-        [
-        "#{}=APPLICATION_CONTEXT('automotive design');".format(application_id),
-        "#{}=APPLICATION_PROTOCOL_DEFINITION("
-        "'fixture','automotive_design',2010,#{});".format(
-            protocol_id, application_id
-        ),
-        "#{}=PRODUCT_CONTEXT('',#{},'mechanical');".format(
-            product_context_id, application_id
-        ),
-        "#{}=PRODUCT_DEFINITION_CONTEXT('part definition',#{},'design');".format(
-            definition_context_id, application_id
-        ),
-        "#{}=PRODUCT('{}','{}','reduced corpus regression fixture',(#{}));".format(
-            product_id, safe_name, safe_name, product_context_id
-        ),
-        "#{}=PRODUCT_DEFINITION_FORMATION_WITH_SPECIFIED_SOURCE("
-        "'1','',#{},.MADE.);".format(formation_id, product_id),
-        "#{}=PRODUCT_DEFINITION('design','',#{},#{});".format(
-            definition_id, formation_id, definition_context_id
-        ),
-        "#{}=PRODUCT_DEFINITION_SHAPE('','',#{});".format(
-            definition_shape_id, definition_id
-        ),
-        "#{}={}('',(#{}),#{});".format(
-            representation_id,
-            "ADVANCED_BREP_SHAPE_REPRESENTATION" if args.closed_shell else
-            "MANIFOLD_SURFACE_SHAPE_REPRESENTATION",
-            model_id,
-            representation_context_id,
-        ),
-        "#{}=SHAPE_DEFINITION_REPRESENTATION(#{},#{});".format(
-            shape_definition_id, definition_shape_id, representation_id
-        ),
-        ]
-    )
-
-    lines = [
-        "ISO-10303-21;",
-        "HEADER;",
-        "FILE_DESCRIPTION(('BRL-CAD reduced corpus regression fixture'),'2;1');",
-        "FILE_NAME('{}','',('BRL-CAD'),('BRL-CAD'),'','','');".format(safe_name),
-        "FILE_SCHEMA(('{}'));".format(quoted(schema)),
-        "ENDSEC;",
-        "DATA;",
-    ]
-    lines.extend(records[entity_id] for entity_id in sorted(selected))
-    lines.extend(wrapper)
-    lines.extend(("ENDSEC;", "END-ISO-10303-21;", ""))
-    args.output.write_text("\n".join(lines), encoding="latin-1")
-
-
-if __name__ == "__main__":
-    main()

misc/tools/step_external_corpus_expectations.json

@@ -1,178 +0,0 @@
-{
-  "format": "brlcad-step-corpus-expectations-v1",
-  "description": "Checksum-pinned external samples used for independent producer qualification without copying their exchange files into BRL-CAD.",
-  "redistribution": "The linked STEP Tools pages state All Rights Reserved, while the STEPcode collective work is distributed under its 3-clause BSD COPYING terms. Obtain each sample from its stated source; this manifest does not independently grant redistribution rights.",
-  "files": [
-    {
-      "name": "cubsomcy.stp",
-      "aliases": ["ap203_cubsomcy.stp"],
-      "sha256": "eb8fb40f56523fb154b435066f7b2719a88e15e30d0b8d8e6d01dd99339ddcf1",
-      "source_page": "https://www.steptools.com/docs/stpfiles/ap203/index.html",
-      "producer": "CATIA / Alcatel Space Industries",
-      "expected": {
-        "equals": {
-          "exit_status": 0,
-          "coverage.products": 1,
-          "coverage.occurrences": 0,
-          "coverage.geometry_attempted": 2,
-          "coverage.geometry_written": 2,
-          "coverage.geometry_skipped": 0,
-          "coverage.outcome": "complete",
-          "coverage.entity_counts.ADVANCED_BREP_SHAPE_REPRESENTATION": 1,
-          "coverage.entity_counts.FACETED_BREP_SHAPE_REPRESENTATION": 1,
-          "derived.occurrence_max_depth": 0,
-          "derived.skipped_entity_ids": []
-        }
-      }
-    },
-    {
-      "name": "moon_buggy_asm.stp",
-      "aliases": ["ap203_moon_buggy_asm.stp"],
-      "sha256": "2afaaf50cd7ec15e36367be0a1e41730e7af5440ed974b62a0a21cf6346289dd",
-      "source_page": "https://www.steptools.com/docs/stpfiles/ap203/index.html",
-      "producer": "STEPNet / EDS Unigraphics 10.5",
-      "expected": {
-        "equals": {
-          "exit_status": 1,
-          "coverage.products": 20,
-          "coverage.occurrences": 23,
-          "coverage.geometry_attempted": 12,
-          "coverage.geometry_written": 9,
-          "coverage.geometry_skipped": 3,
-          "coverage.outcome": "partial",
-          "coverage.entity_counts.MAPPED_ITEM": 23,
-          "derived.assembly_usage_count": 23,
-          "derived.occurrence_detail_count": 23,
-          "derived.occurrence_max_depth": 4,
-          "derived.occurrence_methods": ["mapped"],
-          "derived.skipped_entity_ids": [32060, 32220, 32820]
-        }
-      }
-    },
-    {
-      "name": "123Block_Color.stp",
-      "aliases": ["ap203e2_123Block_Color.stp"],
-      "sha256": "4bbdd752fd3c4e7ebfee633da210b9e3df62c583a604e874ca5e9c36545d2be4",
-      "source_page": "https://www.steptools.com/docs/stpfiles/ap203e2/index.html",
-      "producer": "CATIA / Boeing",
-      "expected": {
-        "equals": {
-          "exit_status": 0,
-          "coverage.products": 1,
-          "coverage.geometry_attempted": 1,
-          "coverage.geometry_written": 1,
-          "coverage.geometry_skipped": 0,
-          "coverage.outcome": "complete",
-          "coverage.styles_extracted": 1,
-          "coverage.styles_applied": 1,
-          "derived.skipped_entity_ids": []
-        }
-      }
-    },
-    {
-      "name": "boxy_with_linearsize.stp",
-      "aliases": ["ap203e2_boxy_with_linearsize.stp"],
-      "sha256": "e120b982c5bb5c8b62c99bb03d5ae46c16d9200fcb53be6521c201be98ead6d6",
-      "source_page": "https://www.steptools.com/docs/stpfiles/ap214/index.html",
-      "producer": "STEP-NC Machine / STEP Tools, Inc.",
-      "expected": {
-        "equals": {
-          "exit_status": 3,
-          "coverage.products": 1,
-          "coverage.geometry_attempted": 1,
-          "coverage.geometry_written": 0,
-          "coverage.geometry_skipped": 1,
-          "coverage.outcome": "failed",
-          "coverage.entity_counts.DATUM": 3,
-          "coverage.entity_counts.PLUS_MINUS_TOLERANCE": 1,
-          "derived.skipped_entity_ids": [26]
-        }
-      }
-    },
-    {
-      "name": "as1-ac-214.stp",
-      "aliases": ["ap214_as1-ac-214.stp"],
-      "sha256": "1bb1a0e55dc4a0169e329529eb520596690f8745077c1bdd2f9fa41fa5a2aa95",
-      "source_page": "https://www.steptools.com/docs/stpfiles/ap214/index.html",
-      "producer": "AutoCAD / STEPnet",
-      "expected": {
-        "equals": {
-          "exit_status": 0,
-          "coverage.products": 9,
-          "coverage.occurrences": 13,
-          "coverage.geometry_attempted": 5,
-          "coverage.geometry_written": 5,
-          "coverage.geometry_skipped": 0,
-          "coverage.outcome": "complete",
-          "coverage.styles_extracted": 5,
-          "coverage.styles_applied": 5,
-          "coverage.layers_extracted": 1,
-          "derived.assembly_usage_count": 13,
-          "derived.occurrence_detail_count": 13,
-          "derived.occurrence_max_depth": 3,
-          "derived.occurrence_methods": ["referenced"],
-          "derived.skipped_entity_ids": []
-        }
-      }
-    },
-    {
-      "name": "as1-ug-214.stp",
-      "aliases": ["ap214_as1-ug-214.stp"],
-      "sha256": "53c5069f5d6b59435eb32ac743cc82e53bd3607954c94785d989d579d0e29426",
-      "source_page": "https://www.steptools.com/docs/stpfiles/ap214/index.html",
-      "producer": "Unigraphics / STEPnet",
-      "expected": {
-        "equals": {
-          "exit_status": 0,
-          "coverage.products": 9,
-          "coverage.occurrences": 13,
-          "coverage.geometry_attempted": 5,
-          "coverage.geometry_written": 5,
-          "coverage.geometry_skipped": 0,
-          "coverage.outcome": "complete",
-          "coverage.styles_extracted": 5,
-          "coverage.styles_applied": 5,
-          "coverage.layers_extracted": 3,
-          "coverage.properties_extracted": 42,
-          "derived.assembly_usage_count": 13,
-          "derived.occurrence_detail_count": 13,
-          "derived.occurrence_max_depth": 3,
-          "derived.occurrence_methods": ["referenced"],
-          "derived.validation_property_count": 42,
-          "derived.skipped_entity_ids": []
-        }
-      }
-    },
-    {
-      "name": "dm1-id-214.stp",
-      "sha256": "25e727cdc4d738064da7907d604bd4ceeb4e941c6fdc201f3df04c0b94671661",
-      "source_page": "STEPcode data/ap214e3/dm1-id-214.stp (3-clause BSD collective work)",
-      "producer": "I-DEAS Master Series 9 / SDRC",
-      "expected": {
-        "equals": {
-          "exit_status": 0,
-          "coverage.products": 7,
-          "coverage.occurrences": 7,
-          "coverage.geometry_attempted": 3,
-          "coverage.geometry_written": 3,
-          "coverage.geometry_skipped": 0,
-          "coverage.outcome": "complete",
-          "coverage.styles_extracted": 3,
-          "coverage.styles_applied": 3,
-          "coverage.materials_extracted": 3,
-          "coverage.properties_extracted": 22,
-          "coverage.entity_counts.MAKE_FROM_USAGE_OPTION": 3,
-          "derived.assembly_usage_count": 7,
-          "derived.occurrence_detail_count": 7,
-          "derived.occurrence_max_depth": 1,
-          "derived.occurrence_methods": ["referenced"],
-          "derived.material_count": 3,
-          "derived.material_identifiers": ["AMS 4928", "AMS 5613", "AMS 5662"],
-          "derived.material_names": ["Greek Ascoloy", "Inconel 718", "Titanium 6-4"],
-          "derived.validation_property_count": 19,
-          "derived.skipped_entity_ids": []
-        }
-      }
-    }
-  ]
-}

misc/tools/step_local_external_corpus_expectations.json

@@ -1,46 +0,0 @@
-{
-  "format": "brlcad-step-corpus-expectations-v1",
-  "description": "Checksum-pinned independent-producer samples from locally retained, non-redistributed corpora whose original download pages were not recorded.",
-  "redistribution": "These records provide reproducible local qualification evidence only. They do not grant permission to copy the exchange files into BRL-CAD or redistribute them.",
-  "files": [
-    {
-      "name": "Leopard2-A6.step",
-      "sha256": "ef8f21f6454f7d952b619ed1e41dab824f31f393e81a3e138392c53605a0ab94",
-      "source_page": "Local non-redistributed GrabCAD corpus; original download page not recorded",
-      "producer": "Autodesk Inventor 2018 / ST-Developer v17",
-      "expected": {
-        "equals": {
-          "exit_status": 1,
-          "coverage.products": 14,
-          "coverage.occurrences": 338,
-          "coverage.geometry_attempted": 31,
-          "coverage.geometry_written": 31,
-          "coverage.geometry_skipped": 0,
-          "coverage.outcome": "partial",
-          "coverage.styles_extracted": 2425,
-          "coverage.styles_applied": 31,
-          "coverage.materials_extracted": 13,
-          "coverage.properties_extracted": 26,
-          "coverage.properties_invalid": 13,
-          "validation.invalid_breps": 1,
-          "validation.invalid_breps_written": 1,
-          "validation.invalid_breps_rejected": 0,
-          "derived.assembly_usage_count": 338,
-          "derived.occurrence_detail_count": 338,
-          "derived.occurrence_max_depth": 1,
-          "derived.occurrence_methods": ["referenced"],
-          "derived.product_metadata_count": 13,
-          "derived.material_count": 13,
-          "derived.material_identifiers": [""],
-          "derived.material_names": ["Generic", "SAE 1020"],
-          "derived.material_property_count": 26,
-          "derived.material_property_names": ["Generic", "SAE 1020", "density measure"],
-          "derived.material_property_units": ["gram^1*centimetre^3"],
-          "derived.material_property_numeric_values": [1, 7.84],
-          "derived.validation_property_count": 0,
-          "derived.skipped_entity_ids": []
-        }
-      }
-    }
-  ]
-}

remaining_step_work.txt

@@ -1,508 +0,0 @@
-Remaining STEP Import/Export Work
-=================================
-
-Status: 2026-08-05
-
-
-Purpose
--------
-
-This is the current implementation and qualification plan for the public
-step-g and g-step converters.  It replaces recommended_practices_todo.txt,
-save_invalid_plan.txt, and step_refactor.txt.  It is not a development
-changelog: completed work is summarized only to define the supported baseline,
-and the detailed sections describe work which remains.
-
-Status terms used below are:
-
-* COMPLETE -- implemented and covered by focused tests;
-* ACTIVE -- being qualified or reduced from real inputs;
-* REMAINING -- useful work with a defined scope;
-* BLOCKED -- requires a missing prerequisite and must not be approximated; and
-* DEFERRED -- outside the present converter objective.
-
-A feature is complete only when schema-valid fixtures, permissive and strict
-policy tests, and structured report assertions prove that geometry or semantics
-did not disappear.  Independent-producer evidence is required wherever a
-BRL-CAD export and re-import could otherwise repeat the same mistake.
-
-
-Supported Baseline
-------------------
-
-Architecture and schemas
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-The public converter architecture is COMPLETE:
-
-* step-g and g-step are schema-dispatching hosts;
-* AP203 edition 1, AP203 edition 2, AP214 edition 3, and AP242 editions 1
-  through 4 are bidirectional plugins;
-* AP242 generated bindings remain edition-specific because the entity sets and
-  some Part 21 physical layouts differ;
-* the AP242 export alias targets edition 4, while automatic import probes
-  editions 4 through 1;
-* common conversion behavior is schema-neutral and schema-only behavior is
-  under ap203, ap203e2, ap214, or ap242; and
-* obsolete schema-specific public converter programs are not built.
-
-Explicit --schema ap242e1 through ap242e4 selection remains useful when a file
-uses only the editions' common physical subset.  Interim identifiers
-CONFIGURATION_CONTROL_3D_DESIGN_MIM_LF, CCD_CLA_GVP_AST, and
-CCD_CLA_GVP_AST_ASD route to AP203e2.  Supported multi-schema AP203e2 headers
-also route correctly; unknown companion schemas remain visible, and genuinely
-ambiguous supported-AP combinations fail rather than guessing.
-
-The STEPcode lazy reader supports multiple DATA sections, SCOPE,
-source-ordered complex components, bounded diagnostics, type census output,
-and schema-independent retention of unknown configuration records.  AP203
-Recommended Practice 8 fixtures cover referenced, mapped, and replicated
-shapes, multi-level hierarchy, SHUO and usage semantics, multiple SDR forms,
-wireframes, configuration data, and SCOPE.
-
-STEPcode and build tractability
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-BRL-CAD uses STEPcode's compact generated API v2, packed schema modules,
-precomputed attribute layouts, bounded unity chunks, and generated source
-manifests.  Entity and type chunk limits are independently tunable; 64:8 is the
-qualified default.  STEP plugins retain optimized compilation but disable LTO.
-
-On the qualification host, a serial AP242e4 binding build without LTO fell
-from 8m25s and 6.22 GB peak RSS at 64:64 to 6m12s and 1.33 GB at 64:8.  NIST
-CTC 03 showed no measurable conversion benefit from LTO.
-
-The importer is split into schema-neutral common build units for session and
-cache handling, document traversal, job scheduling, geometric sets, topology,
-periodic bands, pullback, seam repair, loop repair, validation, and
-finalization.  Schema plugins consume those objects directly, preserving
-static Factory registration without a new runtime ABI.  A local geometry
-repair edit therefore recompiles its topical unit and relinks only requested
-plugins.  Revisit these boundaries only in response to a measured dependency,
-ownership, or rebuild problem.
-
-Default policy and loss accounting
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The user-facing default is deliberately permissive:
-
-* safe topology repair is enabled;
-* exact and strict modes are disabled;
-* storage-safe invalid BReps are preserved and explicitly tagged;
-* useful partial output is published with status 1 and a structured report;
-* --strict is transactional; and
-* --reject-invalid-objs restores invalid-BRep rejection.
-
-Recovery order is source-faithful construction, bounded safe repair, tagged
-modeling inference for a proven source contradiction, marked storage-safe
-invalid fallback, and finally omission.  Scale-bounded inference is attempted
-only in a detached whole-item retry after the ordinary candidate fails.  It is
-committed only when either:
-
-* the rebuilt BRep passes normal structural and solidness validation; or
-* permissive invalid preservation proves that a structurally safe open result
-  corresponds one-for-one to source EDGE_CURVEs with exactly one authored
-  ORIENTED_EDGE use.  Such a result remains invalid and is tagged
-  `step:invalidity=authored_open_topology`; it is never presented as a solid.
-
-Inference is disabled by exact, strict, no-repair, and reject-invalid policy.
-Reports and affected database objects retain source edge identifiers,
-discrepancy, safe and inference limits, declared tolerance, inference kind,
-and explanatory detail.  Rejected inference transactions do not contaminate
-ordinary results or diagnostics.
-
-Every product-bound representation, representation item, selected BRL-CAD
-object, and reachable child receives a handled, intentionally non-geometric,
-skipped, malformed, unsupported, or failed disposition.  Import and export
-distinguish complete, partial, empty, and failed outcomes.  The pre-conversion
-entity census describes the input and does not imply conversion support.
-
-A preserved invalid BRep is not, by itself, proof of bad authoring.  Unless an
-independent topology proof establishes a source contradiction, it remains an
-unresolved converter result tagged `step:geometry_status=invalid_preserved`
-and `step:source_validity=unresolved`.  It forces a partial result and remains
-in the failure ratchet.
-
-Geometry
-~~~~~~~~
-
-The following geometry paths are COMPLETE for their documented subsets:
-
-* rigid product hierarchies, repeated occurrences, transforms, and shared
-  product/occurrence emission in every supported schema;
-* optional native CSG import/export in AP203e2, AP214, and AP242, including
-  supported Boolean trees, exact mapped primitives, BRep leaves, transforms,
-  and RECTANGULAR_PYRAMID;
-* exact compatible extruded and revolved area/face solids and linear or
-  circular SWEPT_DISK_SOLID in AP203e2, AP214, and AP242;
-* AP242 triangulated faces, simple/complex surface sets, shells, and solids,
-  including strips, fans, pnindex, normals, welding, closure, and winding;
-* analytic and spline BRep import, periodic and singular topology, voids,
-  bounded repair, transactional inference, and explicit invalid fallback;
-* solid-BRep export/re-import for sphere, cylinder, cone, torus, box, open
-  NURBS faces with voids, rational Cobb surfaces, and BREP_WITH_VOIDS;
-* mated librt BRep fallback for EHY, EPA, HYP, PART, PIPE, EXTRUDE, REVOLVE,
-  solid BOT, closed ARS, and supported NMG input;
-* sketch recovery for reordered or reversed segments, zero-length segments,
-  inner loops, lines, circles, arcs, Beziers, and rational/non-rational NURBS;
-  and
-* mixed GEOMETRIC_SET curves, points, and CURVE_BOUNDED_SURFACEs with partial
-  set loss accounting and shared bounded pullback validation.
-
-The detailed native-CSG primitive and operator matrix is maintained in
-src/conv/step/README.csg.md.  AP203e1 has no native-CSG path; non-union AP203e1
-trees are BLOCKED on reliable Boolean evaluation.
-
-STEP B-spline curves and surfaces share exact conversion from EXPRESS full knot
-vectors to OpenNURBS endpoint-truncated storage.  Valid unclamped closed
-surfaces are clamped and only numerically equal seam controls are normalized,
-subject to dense before/after audits.
-
-Presentation, units, configuration, and PMI
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-AP203e2, AP214, and AP242 share colour, transparency, style, layer, material,
-property, validation-value, and recursive unit-graph extraction.  Export
-round-trips product identity and revision, presentation, multiple materials,
-typed properties, known SI and exact conversion units, derived units,
-occurrence labels, and reference designators.  AP203e1 explicitly reports
-presentation or property requests it cannot author.
-
-One constrained safe-mode repair handles an identified producer defect where
-a positive density's complete unit graph is mass^1 times length^3 instead of
-mass^1 times length^-3.  The value is retained, the denominator dimension is
-corrected, provenance is recorded, and exact, strict, and no-repair modes
-reject the graph.  This is not a general unit-guessing rule.
-
---output-units, --angle-units, and --uncertainty form one schema-neutral export
-policy covering BReps, native CSG, cone angles, and rigid occurrence
-translations.
-
-Configuration data is retained as a typed reference graph with exact Part 21
-right-hand sides.  Cross-schema authoring covers approvals, provenance,
-assignments, effectivity, alternatives, substitutes, categories, people,
-organizations, addresses, security classifications, documents, date/time
-chains, and their legal schema-specific assignment forms.
-
-DATUM supports point, line, plane, frame, and bounded-target forms, semantic
-roles, axes, dimensions, identifiers, validation, transforms, and versioned
-legacy-compatible database storage.  ANNOT supports model-space planes,
-per-segment roles and styles, UTF-8 text, runtime TrueType outlines,
-transforms, and legacy screen-space compatibility.  The hardened private
-struetype code is shared from libbu; LGPL OSIFont remains runtime data and is
-not compiled into a binary.
-
-AP242 PMI stage 1 retains connected semantic, presentation, association, and
-dependency records with source values, component order, references, product
-ownership, and native disposition.  Exactly resolved point, axis, and plane
-datums become DATUM objects.  Supported graphical content becomes model-space
-ANNOT geometry; ambiguous content remains semantic-only.  Malformed
-presentation is partial in permissive mode and transactional in strict mode.
-
-Safety and test infrastructure
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-BRL-CAD and STEPcode support ASan+UBSan and a separate TSan mode.  Release and
-ASan+UBSan have covered the complete STEP/AP range; focused parallel, failure,
-native-CSG, and BRep-export paths have run under TSan.  On the qualification
-host TSan must run through `setarch x86_64 -R` because its runtime otherwise
-fails before main.  Repeated -j8 versus -j1 output comparison remains part of
-the determinism gate.
-
-
-Validation Snapshot
--------------------
-
-In-tree tests include success, malformed, partial, strict, round-trip,
-sanitizer, and output-validity cases across every supported schema.  The last
-complete focused Release STEP matrix contains 197 tests.  Official NIST MBE PMI
-files are stored under db/nist with documented provenance.  Five AP203 MIM
-files exercise AP203e2, and 17 AP242 files span editions 1 through 4.
-
-Checksum-pinned, non-redistributed STEP Tools and STEPcode samples cover CATIA,
-Unigraphics, AutoCAD, I-DEAS, STEPnet, and STEP Tools producers.  Important
-independent modern-schema evidence currently includes:
-
-* AP242 Leopard2-A6: 31/31 geometry items, 14 products, 338 occurrences,
-  2,425 styles, 13 material identities, and 26 material values;
-* AP242 Open Source Rover: six product levels, 713 products, 1,390 rigid
-  occurrences, and 569 applied styles; and
-* AP203e2 ROTATOR_ASM: eight product levels, 126 participating products, 404
-  usages, 671 product-property records, and 3,991 styles.
-
-The last complete unified corpus baseline is the 1,138-file dry run at
-`.build/step-corpus-full-j8-validity-ladder-20260804-post-speaker`:
-
-* schemas: 616 AP203, 375 AP214, four AP203e2, and 143 AP242 files;
-* files: 1,057 clean; process statuses were 1,064 complete, 69 useful partial,
-  three intentionally empty, one failed, and one SIGSEGV;
-* geometry: 41,765 of 41,992 attempted items written (99.46 percent), 221 exact
-  omissions, and six unsupported-representation omissions;
-* invalid fallback: 386 structurally invalid but storage-safe BReps preserved,
-  none rejected;
-* aggregate wall/conversion time: 36,796.2/34,678.9 seconds; and
-* maximum per-file RSS: 3.62 GB.
-
-This remains the comparison baseline, not the current expected result.  The
-crash and several omission groups have since been fixed in selected working
-runs.  Its timing delta from the preceding baseline is not a controlled
-regression result because host load and available memory differed materially.
-Preserve both manifests and compare future runs only under similar load.
-
-The corpus runner records hashes, schemas, converter/plugin/shared-library
-identities, reports, logs, wall time, peak RSS, host load, memory availability,
-geometry outcomes, hierarchy depth, metadata, and failure categories.  Dry
-mode still parses, constructs, validates, and repairs geometry; it omits only
-database serialization.  Historical `.g` files are comparative artifacts, not
-correctness oracles, and must remain intact.
-
-Selected post-baseline qualifications establish:
-
-* cargo ship now terminates normally; its two remaining selected omissions are
-  explicit missing required FACE_SURFACE references;
-* hAVYARD now terminates normally after rejected seam transactions reacquire
-  topology owned by the restored BRep; a direct owned lifetime fixture remains
-  to be added;
-* SNAP #801290 is a valid solid after a face-scoped horn-torus keyhole repair;
-* all ten historical T90 omissions serialize: six valid solids, two
-  source-open BReps tagged `authored_open_topology`, and two storage-safe
-  partial fallbacks whose contradictions exceed bounded inference limits;
-* paired 135-file AP203/AP242 Recurta runs write all 139 attempted items per
-  schema as valid BReps, with three tagged inferences per schema; and
-* the selected 421-face speaker solid now produces a validated 419-face solid
-  through a whole-item shared-spur transaction.
-
-The T90 serialized selection is recorded in
-`.build/t90_omissions_final_write.json`; it attempted and wrote ten of ten,
-skipped none, and explicitly preserved four invalid objects.  The file-level
-partial status is intentional and does not represent an omission.
-
-Two measured performance targets remain useful:
-
-* Recurta `10012.step`, 992 faces: 78.7 seconds in the final qualification;
-  profiling attributes most time to periodic trim-orientation repair and
-  repeated transaction work, not preflight construction; and
-* chess king, 13,234 faces: about 616.8 seconds and 2.92 GB in its selected
-  clean run.
-
-
-Remaining Work in Recommended Order
------------------------------------
-
-1. Close the current corpus ratchet -- ACTIVE
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Complete selected work before starting another expensive unified run:
-
-1. Add a compact rejected-seam-transaction lifetime fixture, qualify it under
-   ASan+UBSan, and retain the complete hAVYARD real-file check.
-2. Sample and reduce the largest post-baseline omission sets:
-
-   * Garrett: 49;
-   * 2-litre engine: 49, including 48 GEOMETRIC_SET items;
-   * Open Source Rover: 16;
-   * Epoxy granite VMC: seven; and
-   * BLANCO: five.
-
-3. Retest the former 58-omission GEOMETRIC_SET group with the completed bounded
-   surface path.  The independent b/c/d compatibility files are already
-   qualified except for six d-wireframe boundary/pcurve disagreements.
-4. Reduce the Leopard2-A6 export/re-import failure at STEP edge #460691.  One
-   of 31 exported BReps fails while splitting a pcurve whose candidate
-   fragment is 36.4476 model units from its subedge.  This is a geometry
-   interoperability problem, not schema selection or assembly loss.
-5. Retain MOUSE as a source edge/surface mismatch unless independent geometry
-   proves a narrower correction.
-6. Revisit T90 #20240 and #20307 only if independent geometry supports a
-   narrower reconstruction.  Their marked partial results are preferable to
-   inventing a false solid.
-7. Remove repeated full-BRep scans and copies from large periodic repair
-   transactions while preserving exact rollback behavior.
-8. Run focused Release, ASan+UBSan, TSan where applicable, and -j1/-j8
-   determinism gates.  Then run the complete 1,138-file qualification.
-
-Corpus fixes require a compact reproducer, permissive and strict assertions,
-structured report coverage, and a selected real-file check.  Do not introduce
-broad repair heuristics solely to improve corpus percentages.
-
-The next unified run is acceptable only if every file reaches a normal terminal
-report; there is no crash, hang, schema-load failure, corrupt output, or
-unexplained zero-geometry result; and all omissions and invalid fallbacks are
-grouped by owned root cause.
-
-2. Expand independent AP203e2/AP242 evidence -- REMAINING
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Locate and pin independent examples for:
-
-* AP203e2 material and property association variants;
-* AP242 materials beyond the Inventor MAKE_FROM_USAGE_OPTION pattern;
-* mixed representation contexts and large multiple-product files; and
-* AP203e2/AP242 sweeps, NURBS, shells, voids, and mixed BRep/CSG assemblies.
-
-Promote small distributable reductions into in-tree fixtures.  For external
-inputs record source, producer, SHA-256, and structured expectations.  A
-BRL-CAD-authored round trip alone cannot close this item.
-
-3. Complete AP242 semantic PMI and MBD -- REMAINING
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Stage 1 graph retention and exact native point/axis/plane/curve creation are
-complete.  Remaining interpretation and authoring work is to:
-
-* normalize dimension type/value, tolerance type and magnitude,
-  material-condition modifiers, datum-reference compartments, and zones;
-* preserve stable product, occurrence, face, edge, and vertex target links;
-* distinguish semantic-only, graphical-only, and validated combined PMI;
-* map representable text, leaders, extension/dimension lines, arrowheads,
-  fills, area targets, and supported symbols;
-* explicitly report graphical content which cannot be mapped faithfully;
-* author retained semantic and supported presentation records in AP242; and
-* qualify against NIST PMI and independent AP242 files, not only hand-authored
-  fixtures.
-
-An ANNOT outline is not proof that original text or tolerance semantics were
-recovered.  Do not infer missing semantics from graphical similarity.
-
-4. Complete tessellated coverage and optional export -- REMAINING
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Add faithful mappings for tessellated point, curve, wire, geometric-set, and
-repositioned-item forms.  Add explicit optional AP242 BOT export using the
-shared unit and uncertainty policy.  Tessellation must not silently replace
-the eventual evaluated-BRep default for analytic solids.
-
-5. Broaden existing-BRep interoperability -- REMAINING
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Reduce the six d-wireframe boundary/pcurve disagreements and independently
-classify each as an authored tolerance violation or adapter defect.  Preserve
-faithful partial-set output until that proof exists; do not stretch pcurves or
-widen model uncertainty merely to reach complete counts.
-
-Expand independent-producer coverage for rational and non-rational NURBS,
-open shells, nested BREP_WITH_VOIDS, reversed proxies/faces, difficult trims,
-periodic and singular surfaces, and closed curves.  Define a legal policy for
-one BRL-CAD object containing disconnected material volumes.  Add wireframe
-and surface-curve export for databases without face topology.
-
-Verification must include STEPcode readback, strict BRL-CAD re-import,
-topology and material-ray comparison, and an independent STEP validator or
-consumer where available.  Textual identity is insufficient.
-
-Sketch containment still needs a complete loop hierarchy for disjoint outer
-regions and nested material islands.  REVOLVE should split profiles at sharp
-C0/tangent discontinuities when needed to avoid pathological wireframe and CDT
-refinement.
-
-DSP, deprecated HF, EBM, and VOL are a separate scalability and fidelity
-project.  They require bounded-error patching or faithful faceting, discrete
-boundary simplification, source-semantic ray/material comparisons, and hard
-validity, memory, and complexity limits.  A single high-degree DSP surface or
-unbounded voxel-to-BRep expansion is not an acceptable general solution.
-
-6. Complete product and configuration mappings -- REMAINING
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Define scoped mappings and loss policy for region identifiers,
-material/shader strings, and selected arbitrary BRL-CAD attributes.  Do not
-conflate opaque attributes with typed STEP properties.
-
-Broaden normalized named-unit conveniences only where schemas agree; the
-recursive retained graph remains authoritative.  Extend authoring for
-remaining configuration identities, relationships, complex addresses, and
-application edges only when required source values exist.  Expand
-import-export-import tests to multiple products, occurrences, shared complex
-effectivities, and schema-specific graphs.
-
-7. Resolve assembly and native-CSG edge cases -- REMAINING
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Define explicit policy for scale, mirror, and future non-rigid occurrence
-transforms.  Preserve or explicitly report nested combination identity inside
-one CSG expression; STEP CSG_SELECT cannot directly use another CSG_SOLID as
-an operand.  Expand stable naming and independent multi-level mixed BRep/CSG
-evidence across AP203e2, AP214, and AP242.
-
-NOT and guard/no-op Boolean nodes have no lossless finite STEP CSG encoding.
-AP242 CYCLIDE_SEGMENT_SOLID remains unsupported because recognizing a faithful
-BRL-CAD PIPE subset requires additional geometric semantics; similar
-dimensions alone are insufficient.
-
-8. Implement remaining exact sweeps -- REMAINING
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Add nonlinear swept-disk directrices only when OpenNURBS can construct a
-faithful pipe with validated frame continuity and self-intersection handling.
-Ellipse, spline, and composite-curve cases remain.  Implement drafted and
-trimmed face sweeps only after referenced termination, offset, trim intent,
-and per-edge draft can be evaluated exactly.  Inherited depth or angle is not
-a faithful fallback.
-
-
-Blocked and Deferred Work
--------------------------
-
-Evaluated-BRep default export -- BLOCKED
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The broadly interoperable default should eventually evaluate arbitrary
-Boolean trees into final BReps.  Current Boolean evaluation is not reliable
-enough.  Continue evaluator work and validation; when it is ready, make
-evaluated BRep the default without weakening optional exact native CSG.
-
-STEPcode upstreaming -- DEFERRED maintenance
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The generated-binding scalability work required by step-g and g-step is
-complete.  Remaining STEPcode work is upstream preparation and maintenance:
-retain API v1 compatibility where required, and address correctness, cache,
-generation, or resource defects when measured.  Reopen generator scalability
-only for a demonstrated regression or a new schema exceeding current limits.
-
-Additional application protocols -- DEFERRED
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Do not build every ISO SMRL schema speculatively.  AP224 machining features and
-AP238 STEP-NC are the strongest future geometry/manufacturing candidates once
-representative inputs and mapping goals exist.  AP209 analysis and AP210
-electronics require domain-specific semantics.  AP232 and AP239 are primarily
-metadata protocols and are not geometry targets merely because they share
-integrated resources.
-
-ISO SMRL and historical STEPmod selection, provenance, and redistribution
-policy are documented in STEP(5).
-
-
-Definition of Done
-------------------
-
-Every new decoder, emitter, or repair must include, as applicable:
-
-1. an independent hand-authored or external fixture;
-2. a supported case and a malformed or explicitly unsupported case;
-3. permissive and transactional strict assertions;
-4. structured report checks proving exhaustive loss accounting;
-5. schema-specific coverage where legal alternatives or physical layouts
-   differ;
-6. round-trip geometry, topology, semantic, and unit checks;
-7. Release, sanitizer, and deterministic parallel tests;
-8. selected corpus reruns for every affected real file; and
-9. matching step-g, g-step, STEP(5), CSG matrix, and plan documentation.
-
-The overall STEP effort is broadly robust when:
-
-* every input and selected output object is exhaustively accounted for;
-* the unified corpus has no crash, hang, schema-load failure, corrupt output,
-  or unexplained zero-geometry result;
-* AP203e2 and AP242 have independent producer evidence comparable to the
-  relevant AP203/AP214 baseline;
-* supported tessellation, rigid assemblies, presentation, configuration, and
-  PMI round-trip with explicit losses;
-* existing BReps interoperate across analytic, NURBS, periodic, singular,
-  shell, void, orientation, and trim cases with independent consumers;
-* native CSG remains optional, exact, and explicit about unsupported content;
-* parallel, sanitizer, deterministic-output, performance, and memory gates
-  remain green; and
-* reliable Boolean evaluation enables evaluated-BRep default export without
-  regressing native CSG.