Weekly benchmarks (#1758)
* support weekly benchmarks * updated for perftest * rem clipper * other tests * parse cmake output + use ember-benchmark-cases.json * add only required STl files for ember tests * remove ember tests+fix corrupted stl file * shell script to py + minor changes * mac-os * more info of machine with linux fallback * storage for weekly benchmarks * rebase * binary stl * clean * . * fix ember phase capture and drop removed phases - merge stderr into stdout in run_ember_case so man_bench diagnostic output is not silently dropped - remove Assembly/Triangulation/Simplification/Sorting from INDEPENDENT_PHASES; their Timer::Print calls were removed in PR #1745 and they always produced 0.0; only Intersect12 and Winding03 phases are still reported - update build_ember_summary table to match the 4 remaining phases Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert runner * add frontend + trigger on publish + san extended test * fix formatting * TEMP: allow workflow_dispatch to exercise data-publish steps for testing Not for merge - revert before this PR lands. Only relaxing the if: gate so a manual run can validate the folded run: > block and sanitizer arg passing reviewers asked about, without waiting for a scheduled/release run. * Revert "TEMP: allow workflow_dispatch to exercise data-publish steps for testing" This reverts commit e00c5f5784eefa27255681dce4105bd45be3daa0. * Move weekly sanitizer lane to Linux, fix artifact path, derive compiler - Move build_sanitizer/test_sanitizer in weekly_benchmark.yml from macos-15 to ubuntu-24.04, matching the existing PR-triggered sanitizer lane in manifold.yml, since nothing about ASan/UBSan requires macOS here. Drops the compiler overrides and macOS-only setup steps. - Download sanitizer build artifacts into build/ instead of the workspace root, so manifold_test's baked-in RPATH to libsamples can actually resolve. It was crashing on load with "Library not loaded: @rpath/libsamples.3.dylib" because the artifact download flattened the build/ directory away. - Capture the sanitizer build's own cmake configure log and derive SANITIZER_SUMMARY_COMPILER from it at runtime instead of hardcoding the string, reusing system_metadata.py's existing helpers. - Trim set -euo pipefail to set -u in the data-publish steps, since GitHub Actions' default bash invocation already applies -e/pipefail. - Remove the duplicate weeklybenchmark-test job from manifold.yml in favor of testing weekly_benchmark.yml directly via workflow_dispatch. * Merge dashboard pages + remove dead JS fallbacks + gtest JSON output + simplify perf summary formatting + clean up .gitignore * Stop forcing clang-18 for sanitizer builds, use runner default (gcc) * small changes * -Wno-maybe-uninitialized * review changes * dead code * fix cmke read * change flags * remove redundant flags and schema version * /dashboard route + remove upload route + change dead and retention changes --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Anshul
co-authored by
Claude Sonnet 4.6
Claude Sonnet 4.6
parent
422ab6fcee
commit
ff42ddc885
@@ -22,6 +22,8 @@ jobs:
|
||||
mkdir public
|
||||
mv ./samples ./public
|
||||
mv ./docs ./public
|
||||
# weekly benchmark dashboard, served at /dashboard
|
||||
mv ./dashboard ./public
|
||||
|
||||
- name: Download manifoldCAD editor, examples and documentation
|
||||
uses: dawidd6/action-download-artifact@v6
|
||||
|
||||
@@ -420,46 +420,11 @@ jobs:
|
||||
. -B build
|
||||
ASAN_OPTIONS=detect_container_overflow=0 cmake --build build
|
||||
|
||||
build_sanitizer:
|
||||
name: Sanitizer build (ASan+UBSan)
|
||||
timeout-minutes: 30
|
||||
runs-on: ubuntu-24.04
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Install apt packages
|
||||
uses: awalsh128/cache-apt-pkgs-action@v1.6.0
|
||||
with:
|
||||
packages: libgtest-dev libtbb-dev clang-18 cmake
|
||||
version: 1
|
||||
- name: Build (clang-18 + asan/ubsan)
|
||||
run: bash ./scripts/ci/sanitizer_build.sh
|
||||
- name: Upload sanitizer build artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: sanitizer-build
|
||||
path: build/
|
||||
retention-days: 5
|
||||
|
||||
test_sanitizer:
|
||||
name: Sanitizer (ASan+UBSan)
|
||||
needs: build_sanitizer
|
||||
timeout-minutes: 30
|
||||
runs-on: ubuntu-24.04
|
||||
env:
|
||||
ASAN_OPTIONS: detect_container_overflow=0:strict_init_order=1
|
||||
UBSAN_OPTIONS: print_stacktrace=1
|
||||
SANITIZER_TEST_TIMEOUT_SEC: 1200
|
||||
SANITIZER_SUBSET: ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && 'extended' || 'core' }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Download sanitizer build artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: sanitizer-build
|
||||
path: .
|
||||
- name: Test sanitizer subset
|
||||
shell: bash
|
||||
run: bash ./scripts/ci/sanitizer_test.sh
|
||||
sanitizer:
|
||||
uses: ./.github/workflows/sanitizer.yml
|
||||
with:
|
||||
subset: core
|
||||
retention-days: 1
|
||||
|
||||
build_mxe:
|
||||
name: MXE (TBB:${{matrix.parallelization == 'ON'}})
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
name: Sanitizer
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
subset:
|
||||
description: Which gtest subset to run - core or extended.
|
||||
type: string
|
||||
required: true
|
||||
retention-days:
|
||||
description: >
|
||||
Retention for the shared sanitizer build artifact. It only exists to
|
||||
hand the build from the build job to the test job, so this is short.
|
||||
type: number
|
||||
default: 1
|
||||
test-timeout-sec:
|
||||
description: Per-run timeout applied inside sanitizer_test.sh.
|
||||
type: number
|
||||
default: 1200
|
||||
outputs:
|
||||
build-result:
|
||||
description: Result of the sanitizer build job.
|
||||
value: ${{ jobs.build_sanitizer.result }}
|
||||
test-result:
|
||||
description: Result of the sanitizer test job.
|
||||
value: ${{ jobs.test_sanitizer.result }}
|
||||
|
||||
jobs:
|
||||
build_sanitizer:
|
||||
name: Sanitizer build (ASan+UBSan)
|
||||
timeout-minutes: 30
|
||||
runs-on: ubuntu-24.04
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Install apt packages
|
||||
uses: awalsh128/cache-apt-pkgs-action@v1.6.0
|
||||
with:
|
||||
packages: libgtest-dev cmake
|
||||
version: 1
|
||||
- name: Build (asan/ubsan)
|
||||
run: ./scripts/ci/sanitizer_build.sh
|
||||
- name: Upload sanitizer build artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: sanitizer-build
|
||||
path: build/
|
||||
retention-days: ${{ inputs.retention-days }}
|
||||
|
||||
test_sanitizer:
|
||||
name: Sanitizer ${{ inputs.subset }} subset (ASan+UBSan)
|
||||
needs: build_sanitizer
|
||||
timeout-minutes: 30
|
||||
runs-on: ubuntu-24.04
|
||||
env:
|
||||
ASAN_OPTIONS: detect_container_overflow=0:strict_init_order=1
|
||||
UBSAN_OPTIONS: print_stacktrace=1
|
||||
SANITIZER_TEST_TIMEOUT_SEC: ${{ inputs.test-timeout-sec }}
|
||||
SANITIZER_SUBSET: ${{ inputs.subset }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Download sanitizer build artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: sanitizer-build
|
||||
path: build
|
||||
- name: Test sanitizer subset
|
||||
shell: bash
|
||||
run: bash ./scripts/ci/sanitizer_test.sh
|
||||
@@ -0,0 +1,77 @@
|
||||
name: Weekly Benchmarks
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
release:
|
||||
types: [published]
|
||||
schedule:
|
||||
- cron: "0 2 * * 0"
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
concurrency:
|
||||
group: weekly-benchmark
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
sanitizer:
|
||||
uses: ./.github/workflows/sanitizer.yml
|
||||
with:
|
||||
subset: extended
|
||||
retention-days: 1
|
||||
|
||||
weekly_benchmark:
|
||||
name: Weekly benchmarks
|
||||
needs: sanitizer
|
||||
if: ${{ !cancelled() }}
|
||||
timeout-minutes: 60
|
||||
runs-on: macos-15
|
||||
env:
|
||||
WEEKLY_BENCHMARK_REPEATS: 5
|
||||
WEEKLY_BENCHMARK_THREADS: 1
|
||||
CMAKE_OSX_ARCHITECTURES: arm64
|
||||
DATA_BRANCH: benchmark-data
|
||||
RELEASE_TAG: ${{ github.event_name == 'release' && github.event.release.tag_name || '' }}
|
||||
SANITIZER_SUMMARY_SUBSET: extended
|
||||
SANITIZER_SUMMARY_RUNNER: ubuntu-24.04
|
||||
SANITIZER_SUMMARY_OS: Linux
|
||||
SANITIZER_BUILD_RESULT: ${{ needs.sanitizer.outputs.build-result }}
|
||||
SANITIZER_TEST_RESULT: ${{ needs.sanitizer.outputs.test-result }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.x'
|
||||
- name: Install brew packages
|
||||
run: brew install googletest assimp tbb cmake
|
||||
- name: Run weekly benchmarks
|
||||
shell: bash
|
||||
run: python3 ./scripts/run_weekly_benchmarks.py . ./bench/weekly "${WEEKLY_BENCHMARK_REPEATS}"
|
||||
- name: Report weekly benchmarks
|
||||
run: python3 ./scripts/report_weekly_benchmarks.py . ./bench/weekly "${WEEKLY_BENCHMARK_REPEATS}"
|
||||
- name: Publish weekly benchmark data
|
||||
if: github.event_name == 'release' || github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
|
||||
shell: bash
|
||||
run: |
|
||||
# github-actions[bot]'s identity for committing with the built-in
|
||||
# token, see https://github.com/actions/checkout#push-a-commit-using-the-built-in-token
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
git fetch origin "${DATA_BRANCH}"
|
||||
git worktree add benchmark-data "origin/${DATA_BRANCH}"
|
||||
|
||||
python3 ./scripts/publish_weekly_benchmark_results.py \
|
||||
./bench/weekly ./benchmark-data \
|
||||
--release-tag "${RELEASE_TAG}" \
|
||||
--sanitizer-subset "${SANITIZER_SUMMARY_SUBSET}" \
|
||||
--sanitizer-build-result "${SANITIZER_BUILD_RESULT}" \
|
||||
--sanitizer-test-result "${SANITIZER_TEST_RESULT}" \
|
||||
--sanitizer-runner "${SANITIZER_SUMMARY_RUNNER}" \
|
||||
--sanitizer-os "${SANITIZER_SUMMARY_OS}"
|
||||
|
||||
git -C benchmark-data add weekly
|
||||
git -C benchmark-data commit -m "add weekly benchmark result ${GITHUB_RUN_ID}"
|
||||
git -C benchmark-data push origin "HEAD:${DATA_BRANCH}"
|
||||
+1
-1
@@ -133,7 +133,7 @@ CMakeSettings.json
|
||||
[._]sw[a-p]
|
||||
|
||||
# Benchmark stl files
|
||||
extras/ember_tests/testfiles/raw_meshes
|
||||
extras/ember_tests/testfiles/raw_meshes/*
|
||||
|
||||
#Eclipse project files
|
||||
/.project
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
body.container {
|
||||
max-width: 1800px;
|
||||
margin: 0 auto;
|
||||
padding: 28px 64px;
|
||||
color: #222;
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
}
|
||||
|
||||
a {
|
||||
color: #2563c9;
|
||||
}
|
||||
|
||||
.top-nav {
|
||||
margin-bottom: 14px;
|
||||
font-family: ui-monospace, SFMono-Regular, Consolas, monospace;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
h1 {
|
||||
margin: 10px 0 16px;
|
||||
font-size: 26px;
|
||||
}
|
||||
|
||||
h2 {
|
||||
margin-top: 18px;
|
||||
}
|
||||
|
||||
details {
|
||||
margin-bottom: 12px;
|
||||
}
|
||||
|
||||
summary {
|
||||
cursor: pointer;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.controls,
|
||||
#scale-select-form {
|
||||
margin: 8px 0 12px;
|
||||
}
|
||||
|
||||
#scale-select-form label,
|
||||
.controls label {
|
||||
margin-right: 14px;
|
||||
}
|
||||
|
||||
.controls {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 12px;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.data-source,
|
||||
#page-latest,
|
||||
#status-line {
|
||||
margin: 8px 0;
|
||||
}
|
||||
|
||||
#page-latest p {
|
||||
margin: 3px 0;
|
||||
}
|
||||
|
||||
.top-nav a.active {
|
||||
font-weight: 600;
|
||||
color: inherit;
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.graphs {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
gap: 18px 30px;
|
||||
margin-top: 8px;
|
||||
}
|
||||
|
||||
.chart-box {
|
||||
min-width: 0;
|
||||
min-height: 420px;
|
||||
}
|
||||
|
||||
table {
|
||||
border-collapse: collapse;
|
||||
width: 100%;
|
||||
margin-top: 8px;
|
||||
}
|
||||
|
||||
th,
|
||||
td {
|
||||
border-bottom: 1px solid #ddd;
|
||||
padding: 6px 8px;
|
||||
text-align: left;
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
th {
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
#error-box {
|
||||
padding: 10px;
|
||||
border: 1px solid #c44;
|
||||
background: #fff5f5;
|
||||
color: #8a1f1f;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
|
||||
@media screen and (max-width: 900px) {
|
||||
body.container {
|
||||
padding: 18px;
|
||||
}
|
||||
|
||||
.graphs {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,536 @@
|
||||
'use strict';
|
||||
|
||||
const DEFAULT_DATA_ROOT =
|
||||
'https://raw.githubusercontent.com/elalish/manifold/benchmark-data';
|
||||
|
||||
const CHARTS = [
|
||||
{
|
||||
element: 'perf-time-chart',
|
||||
metric: 'perf_time',
|
||||
title: 'perfTest size sweep time',
|
||||
unit: 'Milliseconds',
|
||||
valueSuffix: ' ms',
|
||||
},
|
||||
{
|
||||
element: 'perf-rss-chart',
|
||||
metric: 'perf_rss',
|
||||
title: 'perfTest peak RSS',
|
||||
unit: 'MB',
|
||||
valueSuffix: ' MB',
|
||||
},
|
||||
{
|
||||
element: 'ember-full-chart',
|
||||
metric: 'ember_full',
|
||||
title: 'Ember full phase time',
|
||||
unit: 'Milliseconds',
|
||||
valueSuffix: ' ms',
|
||||
},
|
||||
{
|
||||
element: 'ember-phase-chart',
|
||||
metric: 'ember_phase',
|
||||
title: 'Ember individual phase time',
|
||||
unit: 'Milliseconds',
|
||||
valueSuffix: ' ms',
|
||||
maxSeries: 16,
|
||||
},
|
||||
{
|
||||
element: 'gtest-time-chart',
|
||||
metric: 'gtest_time',
|
||||
title: 'Existing gtest time',
|
||||
unit: 'Milliseconds',
|
||||
valueSuffix: ' ms',
|
||||
},
|
||||
];
|
||||
|
||||
const state = {
|
||||
dataRoot: '',
|
||||
branchRoot: '',
|
||||
index: null,
|
||||
records: [],
|
||||
scale: 'linear',
|
||||
};
|
||||
|
||||
function qs(id) {
|
||||
return document.getElementById(id);
|
||||
}
|
||||
|
||||
function setText(id, text) {
|
||||
const element = qs(id);
|
||||
if (element) {
|
||||
element.textContent = text;
|
||||
}
|
||||
}
|
||||
|
||||
function setStatus(text) {
|
||||
setText('status-text', text);
|
||||
}
|
||||
|
||||
function setError(message) {
|
||||
const box = qs('error-box');
|
||||
if (!box) {
|
||||
return;
|
||||
}
|
||||
if (!message) {
|
||||
box.hidden = true;
|
||||
box.textContent = '';
|
||||
return;
|
||||
}
|
||||
box.hidden = false;
|
||||
box.textContent = message;
|
||||
}
|
||||
|
||||
function normalizeRoot(root) {
|
||||
return (root || DEFAULT_DATA_ROOT).replace(/\/+$/, '');
|
||||
}
|
||||
|
||||
function withNoCache(url) {
|
||||
const separator = url.includes('?') ? '&' : '?';
|
||||
return `${url}${separator}t=${Date.now()}`;
|
||||
}
|
||||
|
||||
async function fetchJson(url) {
|
||||
const response = await fetch(withNoCache(url), {cache: 'no-store'});
|
||||
if (!response.ok) {
|
||||
throw new Error(`${response.status} ${response.statusText}: ${url}`);
|
||||
}
|
||||
return response.json();
|
||||
}
|
||||
|
||||
async function loadIndex(dataRoot) {
|
||||
const branchRoot = normalizeRoot(dataRoot);
|
||||
const weeklyRoot = `${branchRoot}/weekly`;
|
||||
const indexUrl = `${weeklyRoot}/index.json`;
|
||||
|
||||
try {
|
||||
const index = await fetchJson(indexUrl);
|
||||
return {branchRoot, weeklyRoot, index};
|
||||
} catch (error) {
|
||||
throw new Error(`Unable to load benchmark index.\n${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
function resultUrl(entry) {
|
||||
// result_path is always a weekly/-prefixed path relative to branchRoot -
|
||||
// see publish_weekly_benchmark_results.py's dated_rel construction.
|
||||
return `${state.branchRoot}/${entry.result_path}`;
|
||||
}
|
||||
|
||||
function shortSha(sha) {
|
||||
return sha ? sha.slice(0, 8) : 'unknown';
|
||||
}
|
||||
|
||||
function formatDate(raw) {
|
||||
if (!raw) {
|
||||
return 'unknown';
|
||||
}
|
||||
const date = new Date(raw);
|
||||
if (Number.isNaN(date.getTime())) {
|
||||
return raw;
|
||||
}
|
||||
return date.toISOString().slice(0, 10);
|
||||
}
|
||||
|
||||
function runLabel(entry) {
|
||||
const label = formatDate(entry.timestamp);
|
||||
return entry.release_tag ? `${label} (${entry.release_tag})` : label;
|
||||
}
|
||||
|
||||
function sanitizerSummary(entry) {
|
||||
const sanitizer = entry.sanitizer;
|
||||
if (!sanitizer) {
|
||||
return '-';
|
||||
}
|
||||
const subset = sanitizer.subset || 'sanitizer';
|
||||
const build = sanitizer.build_result || 'unknown';
|
||||
const test = sanitizer.test_result || 'unknown';
|
||||
return build === 'success' ? `${subset}: ${test}` :
|
||||
`${subset}: build ${build}`;
|
||||
}
|
||||
|
||||
function statKey(stat, suffix = '') {
|
||||
return `${stat}${suffix}`;
|
||||
}
|
||||
|
||||
function currentStat() {
|
||||
return qs('stat-select')?.value || 'median';
|
||||
}
|
||||
|
||||
function metricItems(record, metricId, stat) {
|
||||
const suites = record.result.suites || {};
|
||||
|
||||
if (metricId === 'perf_time') {
|
||||
return (suites.perf_size_sweep?.workloads || [])
|
||||
.map((workload) => ({
|
||||
name: workload.benchmark || `nTri=${workload.n_tri}`,
|
||||
value: workload.time_ms?.[statKey(stat, '_ms')] ?? null,
|
||||
}));
|
||||
}
|
||||
|
||||
if (metricId === 'perf_rss') {
|
||||
return (suites.perf_size_sweep?.workloads || [])
|
||||
.map((workload) => ({
|
||||
name: workload.benchmark || `nTri=${workload.n_tri}`,
|
||||
value: workload.peak_rss_mb?.[stat] ?? null,
|
||||
}));
|
||||
}
|
||||
|
||||
if (metricId === 'ember_full') {
|
||||
return (suites.weekly_ember_phase?.cases || [])
|
||||
.map(
|
||||
(caseItem) => ({
|
||||
name: `case ${caseItem.case_index}`,
|
||||
value: caseItem.full_phase_sum_ms?.[statKey(stat, '_ms')] ?? null,
|
||||
}));
|
||||
}
|
||||
|
||||
if (metricId === 'ember_phase') {
|
||||
const items = [];
|
||||
for (const caseItem of suites.weekly_ember_phase?.cases || []) {
|
||||
for (const [phase, stats] of Object.entries(caseItem.phases || {})) {
|
||||
items.push({
|
||||
name: `case ${caseItem.case_index} / ${phase}`,
|
||||
value: stats?.[statKey(stat, '_ms')] ?? null,
|
||||
});
|
||||
}
|
||||
}
|
||||
return items;
|
||||
}
|
||||
|
||||
if (metricId === 'gtest_time') {
|
||||
return (suites.existing_gtests?.tests || [])
|
||||
.map((test) => ({
|
||||
name: test.name,
|
||||
value: test.time_ms?.[statKey(stat, '_ms')] ?? null,
|
||||
}));
|
||||
}
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
function buildSeries(metricId, stat, maxSeries) {
|
||||
const categories = state.records.map((record) => runLabel(record.entry));
|
||||
const names = new Set();
|
||||
const byRun = state.records.map((record) => {
|
||||
const map = new Map();
|
||||
for (const item of metricItems(record, metricId, stat)) {
|
||||
names.add(item.name);
|
||||
map.set(item.name, item.value);
|
||||
}
|
||||
return map;
|
||||
});
|
||||
|
||||
const latestMap = byRun[byRun.length - 1] || new Map();
|
||||
const sortedNames = [...names].sort((a, b) => {
|
||||
const bValue = latestMap.get(b) ?? -Infinity;
|
||||
const aValue = latestMap.get(a) ?? -Infinity;
|
||||
return bValue - aValue || a.localeCompare(b);
|
||||
});
|
||||
|
||||
const visibleNames = sortedNames.slice(0, maxSeries || sortedNames.length);
|
||||
const series = visibleNames.map((name) => ({
|
||||
type: 'line',
|
||||
name,
|
||||
animation: false,
|
||||
data: byRun.map((run) => {
|
||||
const value = run.get(name) ?? null;
|
||||
if (state.scale === 'log' &&
|
||||
value !== null && value <= 0) {
|
||||
return null;
|
||||
}
|
||||
return value;
|
||||
}),
|
||||
}));
|
||||
|
||||
return {categories, series, totalSeries: sortedNames.length};
|
||||
}
|
||||
|
||||
function renderHighchartsChart(config) {
|
||||
const stat = currentStat();
|
||||
const {categories, series, totalSeries} = buildSeries(
|
||||
config.metric,
|
||||
stat,
|
||||
config.maxSeries,
|
||||
);
|
||||
|
||||
Highcharts.chart({
|
||||
chart: {
|
||||
renderTo: config.element,
|
||||
zooming: {
|
||||
type: 'xy',
|
||||
},
|
||||
type: 'line',
|
||||
},
|
||||
title: {
|
||||
text: config.title,
|
||||
},
|
||||
subtitle: {
|
||||
text: totalSeries > series.length ?
|
||||
`Showing top ${series.length} of ${totalSeries} series by latest ${
|
||||
stat}` :
|
||||
`${stat} across ${categories.length} loaded runs`,
|
||||
},
|
||||
yAxis: {
|
||||
title: {text: config.unit},
|
||||
min: state.scale === 'linear' ? 0 : undefined,
|
||||
type: state.scale === 'log' ? 'logarithmic' : undefined,
|
||||
},
|
||||
xAxis: {
|
||||
categories,
|
||||
title: {text: 'Run date'},
|
||||
},
|
||||
tooltip: {
|
||||
shared: false,
|
||||
valueSuffix: config.valueSuffix,
|
||||
},
|
||||
series,
|
||||
});
|
||||
}
|
||||
|
||||
function renderCharts() {
|
||||
const chartsOnPage = CHARTS.filter((chart) => qs(chart.element));
|
||||
if (!chartsOnPage.length) {
|
||||
return;
|
||||
}
|
||||
if (!window.Highcharts) {
|
||||
throw new Error('Highcharts was not loaded.');
|
||||
}
|
||||
for (const chart of chartsOnPage) {
|
||||
renderHighchartsChart(chart);
|
||||
}
|
||||
}
|
||||
|
||||
function renderLatestMetadata() {
|
||||
const latest = state.records[state.records.length - 1];
|
||||
if (!latest) {
|
||||
setText('latest-run', '-');
|
||||
setText('latest-cpu', '-');
|
||||
setText('latest-compiler', '-');
|
||||
return;
|
||||
}
|
||||
|
||||
const entry = latest.entry;
|
||||
setText(
|
||||
'latest-run',
|
||||
`${formatDate(entry.timestamp)} / ${shortSha(entry.commit_sha)}`);
|
||||
setText('latest-cpu', entry.cpu_model || 'unknown');
|
||||
setText('latest-compiler', entry.compiler || 'unknown');
|
||||
renderLatestTable(latest);
|
||||
}
|
||||
|
||||
function renderLatestTable(latest) {
|
||||
const container = qs('latest-metadata-table');
|
||||
if (!container) {
|
||||
return;
|
||||
}
|
||||
container.replaceChildren();
|
||||
|
||||
const entry = latest.entry;
|
||||
const sanitizer = entry.sanitizer || {};
|
||||
const rows = [
|
||||
['Run ID', entry.run_id],
|
||||
['Trigger', entry.trigger],
|
||||
['Release tag', entry.release_tag || '-'],
|
||||
['Timestamp', entry.timestamp],
|
||||
['Commit', entry.commit_sha || 'unknown'],
|
||||
['Workflow', entry.workflow || 'unknown'],
|
||||
['Runner', entry.runner || 'unknown'],
|
||||
['OS', entry.os || 'unknown'],
|
||||
['Compiler', entry.compiler || 'unknown'],
|
||||
['CPU', entry.cpu_model || 'unknown'],
|
||||
['CPU count', entry.cpu_count || 'unknown'],
|
||||
['Sanitizer subset', sanitizer.subset || '-'],
|
||||
['Sanitizer build', sanitizer.build_result || '-'],
|
||||
['Sanitizer test', sanitizer.test_result || '-'],
|
||||
];
|
||||
|
||||
const table = document.createElement('table');
|
||||
const tbody = document.createElement('tbody');
|
||||
for (const [label, value] of rows) {
|
||||
const tr = document.createElement('tr');
|
||||
const th = document.createElement('th');
|
||||
th.textContent = label;
|
||||
const td = document.createElement('td');
|
||||
td.textContent = value;
|
||||
tr.append(th, td);
|
||||
tbody.appendChild(tr);
|
||||
}
|
||||
table.appendChild(tbody);
|
||||
container.appendChild(table);
|
||||
}
|
||||
|
||||
function renderRunTable(records) {
|
||||
const container = qs('run-table');
|
||||
if (!container) {
|
||||
return;
|
||||
}
|
||||
container.replaceChildren();
|
||||
|
||||
const table = document.createElement('table');
|
||||
const thead = document.createElement('thead');
|
||||
const headerRow = document.createElement('tr');
|
||||
for (const text
|
||||
of ['date', 'release', 'commit', 'runner', 'cpu', 'sanitizer', 'run',
|
||||
'json']) {
|
||||
const th = document.createElement('th');
|
||||
th.textContent = text;
|
||||
headerRow.appendChild(th);
|
||||
}
|
||||
thead.appendChild(headerRow);
|
||||
table.appendChild(thead);
|
||||
|
||||
const tbody = document.createElement('tbody');
|
||||
for (const record of [...records].reverse()) {
|
||||
const entry = record.entry;
|
||||
const tr = document.createElement('tr');
|
||||
for (const text
|
||||
of [formatDate(entry.timestamp),
|
||||
entry.release_tag || '-',
|
||||
shortSha(entry.commit_sha),
|
||||
entry.runner || 'unknown',
|
||||
entry.cpu_model || 'unknown',
|
||||
sanitizerSummary(entry),
|
||||
]) {
|
||||
const td = document.createElement('td');
|
||||
td.textContent = text;
|
||||
tr.appendChild(td);
|
||||
}
|
||||
|
||||
const runCell = document.createElement('td');
|
||||
if (entry.github_run_url) {
|
||||
const runLink = document.createElement('a');
|
||||
runLink.href = entry.github_run_url;
|
||||
runLink.textContent = entry.run_id || 'run';
|
||||
runCell.appendChild(runLink);
|
||||
} else {
|
||||
runCell.textContent = entry.run_id || '-';
|
||||
}
|
||||
tr.appendChild(runCell);
|
||||
|
||||
const jsonCell = document.createElement('td');
|
||||
const jsonLink = document.createElement('a');
|
||||
jsonLink.href = resultUrl(entry);
|
||||
jsonLink.textContent = 'result.json';
|
||||
jsonCell.appendChild(jsonLink);
|
||||
tr.appendChild(jsonCell);
|
||||
|
||||
tbody.appendChild(tr);
|
||||
}
|
||||
table.appendChild(tbody);
|
||||
container.appendChild(table);
|
||||
}
|
||||
|
||||
function renderDashboard() {
|
||||
renderLatestMetadata();
|
||||
renderCharts();
|
||||
renderRunTable(state.records);
|
||||
}
|
||||
|
||||
function selectedRunCount(total) {
|
||||
const raw = qs('run-count')?.value || '10';
|
||||
return raw === 'all' ? total : Math.min(Number(raw), total);
|
||||
}
|
||||
|
||||
async function loadDashboard() {
|
||||
setError('');
|
||||
setStatus('Loading data');
|
||||
setText('data-summary', 'Fetching benchmark index...');
|
||||
const loadButton = qs('load-button');
|
||||
if (loadButton) {
|
||||
loadButton.disabled = true;
|
||||
}
|
||||
|
||||
try {
|
||||
const root = state.dataRoot || DEFAULT_DATA_ROOT;
|
||||
const loaded = await loadIndex(root);
|
||||
state.dataRoot = root;
|
||||
state.branchRoot = loaded.branchRoot;
|
||||
state.index = loaded.index;
|
||||
|
||||
const runs = [...loaded.index.runs].sort(
|
||||
(a, b) => a.timestamp.localeCompare(b.timestamp),
|
||||
);
|
||||
const count = selectedRunCount(runs.length);
|
||||
const selected = runs.slice(-count);
|
||||
|
||||
setText(
|
||||
'data-summary',
|
||||
`Fetching ${selected.length} result file${
|
||||
selected.length === 1 ? '' : 's'}...`);
|
||||
state.records = await Promise.all(
|
||||
selected.map(async (entry) => ({
|
||||
entry,
|
||||
result: await fetchJson(resultUrl(entry)),
|
||||
})),
|
||||
);
|
||||
|
||||
setStatus('Loaded');
|
||||
setText(
|
||||
'data-summary',
|
||||
`${runs.length} run${runs.length === 1 ? '' : 's'} in index, ${
|
||||
state.records.length} loaded. Latest: ${
|
||||
loaded.index.latest_run_id || 'unknown'}.`,
|
||||
);
|
||||
renderDashboard();
|
||||
} catch (error) {
|
||||
setStatus('Load failed');
|
||||
setText('data-summary', 'The dashboard could not load benchmark data.');
|
||||
setError(error.stack || error.message || String(error));
|
||||
} finally {
|
||||
if (loadButton) {
|
||||
loadButton.disabled = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function handleScaleChange(event) {
|
||||
const value = event.target.value;
|
||||
if (value !== 'linear' && value !== 'log') {
|
||||
console.error('Invalid scale');
|
||||
return;
|
||||
}
|
||||
if (state.scale !== value) {
|
||||
state.scale = value;
|
||||
renderDashboard();
|
||||
}
|
||||
}
|
||||
|
||||
const TABS = ['graphs', 'latest', 'runs', 'help'];
|
||||
|
||||
function currentTab() {
|
||||
const hash = window.location.hash.replace(/^#/, '');
|
||||
return TABS.includes(hash) ? hash : 'graphs';
|
||||
}
|
||||
|
||||
function showTab(tab) {
|
||||
for (const name of TABS) {
|
||||
const page = qs(`page-${name}`);
|
||||
if (page) {
|
||||
page.hidden = name !== tab;
|
||||
}
|
||||
}
|
||||
for (const link of document.querySelectorAll('.top-nav a[data-tab]')) {
|
||||
link.classList.toggle('active', link.dataset.tab === tab);
|
||||
}
|
||||
if (tab === 'graphs') {
|
||||
renderCharts();
|
||||
}
|
||||
}
|
||||
|
||||
function init() {
|
||||
const params = new URLSearchParams(window.location.search);
|
||||
state.dataRoot = params.get('data') || DEFAULT_DATA_ROOT;
|
||||
setText('data-root-label', state.dataRoot);
|
||||
|
||||
window.addEventListener('hashchange', () => showTab(currentTab()));
|
||||
showTab(currentTab());
|
||||
|
||||
qs('load-button')?.addEventListener('click', loadDashboard);
|
||||
qs('stat-select')?.addEventListener('change', renderDashboard);
|
||||
qs('run-count')?.addEventListener('change', loadDashboard);
|
||||
qs('linear-scale-input')?.addEventListener('input', handleScaleChange);
|
||||
qs('log-scale-input')?.addEventListener('input', handleScaleChange);
|
||||
|
||||
loadDashboard();
|
||||
}
|
||||
|
||||
document.addEventListener('DOMContentLoaded', init);
|
||||
@@ -0,0 +1,118 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Manifold weekly benchmarks</title>
|
||||
<link rel="stylesheet" href="dashboard.css">
|
||||
</head>
|
||||
<body class="container">
|
||||
<div class="top-nav">> <a href="#graphs" data-tab="graphs">graphs</a>, <a href="#latest" data-tab="latest">latest</a>, <a href="#runs" data-tab="runs">runs</a>, <a href="#help" data-tab="help">help</a>.</div>
|
||||
|
||||
<div class="controls">
|
||||
<label for="run-count">Runs</label>
|
||||
<select id="run-count">
|
||||
<option value="10">latest 10</option>
|
||||
<option value="25">latest 25</option>
|
||||
<option value="all">all</option>
|
||||
</select>
|
||||
<button id="load-button" type="button">Reload</button>
|
||||
</div>
|
||||
|
||||
<div class="data-source"><strong>Data source:</strong> <code id="data-root-label">-</code></div>
|
||||
<p id="status-line"><strong>Status:</strong> <span id="status-text">Loading data</span></p>
|
||||
<pre id="error-box" hidden></pre>
|
||||
|
||||
<section id="page-graphs">
|
||||
<h1>Manifold weekly benchmarks</h1>
|
||||
|
||||
<details open>
|
||||
<summary>What data is in the dashboard?</summary>
|
||||
<p>
|
||||
The dashboard reads weekly benchmark JSON files from the <code>benchmark-data</code>
|
||||
branch. It shows the <code>perfTest</code> size sweep, selected Ember phase
|
||||
timings, and selected existing gtests. The charts use Highcharts line graphs,
|
||||
matching the approach used by the rustc-perf dashboard.
|
||||
</p>
|
||||
</details>
|
||||
|
||||
<form id="scale-select-form">
|
||||
<label for="linear-scale-input">
|
||||
<input id="linear-scale-input" type="radio" name="scale-select" value="linear" checked>
|
||||
Linear-scale
|
||||
</label>
|
||||
<label for="log-scale-input">
|
||||
<input id="log-scale-input" type="radio" name="scale-select" value="log">
|
||||
Log-scale
|
||||
</label>
|
||||
</form>
|
||||
|
||||
<div class="controls">
|
||||
<label for="stat-select">Statistic</label>
|
||||
<select id="stat-select">
|
||||
<option value="median">median</option>
|
||||
<option value="mean">mean</option>
|
||||
<option value="min">min</option>
|
||||
<option value="max">max</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div class="graphs">
|
||||
<div id="perf-time-chart" class="chart-box"></div>
|
||||
<div id="perf-rss-chart" class="chart-box"></div>
|
||||
<div id="ember-full-chart" class="chart-box"></div>
|
||||
<div id="ember-phase-chart" class="chart-box"></div>
|
||||
<div id="gtest-time-chart" class="chart-box"></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="page-latest" hidden>
|
||||
<h1>Latest weekly benchmark run</h1>
|
||||
<p>Run: <span id="latest-run">-</span></p>
|
||||
<p>CPU: <span id="latest-cpu">-</span></p>
|
||||
<p>Compiler: <span id="latest-compiler">-</span></p>
|
||||
<p id="data-summary">Waiting for benchmark index...</p>
|
||||
<div id="latest-metadata-table"></div>
|
||||
</section>
|
||||
|
||||
<section id="page-runs" hidden>
|
||||
<h1>Weekly benchmark runs</h1>
|
||||
<div id="run-table"></div>
|
||||
</section>
|
||||
|
||||
<section id="page-help" hidden>
|
||||
<h1>Manifold benchmark dashboard help</h1>
|
||||
|
||||
<details open>
|
||||
<summary>What data is in the dashboard?</summary>
|
||||
<p>
|
||||
The dashboard reads weekly benchmark JSON files from the <code>benchmark-data</code>
|
||||
branch. The default source is <code>https://raw.githubusercontent.com/elalish/manifold/benchmark-data</code>.
|
||||
For local testing against a fork, append <code>?data=<raw benchmark-data branch URL></code>
|
||||
to the dashboard URL. The URL must point at the branch root (e.g.
|
||||
<code>https://raw.githubusercontent.com/<you>/manifold/benchmark-data</code>),
|
||||
not the <code>weekly/</code> subdirectory.
|
||||
Release-triggered benchmark runs are stored in the same timeline and marked
|
||||
with their release tag.
|
||||
</p>
|
||||
</details>
|
||||
|
||||
<h2>Tabs</h2>
|
||||
<p><a href="#graphs">graphs</a> shows Highcharts line charts for weekly benchmark trends.</p>
|
||||
<p><a href="#latest">latest</a> shows metadata for the latest loaded run.</p>
|
||||
<p><a href="#runs">runs</a> lists loaded historical runs and links to raw JSON.</p>
|
||||
|
||||
<h2>Measured suites</h2>
|
||||
<ul>
|
||||
<li><code>perfTest</code> size sweep time and peak RSS.</li>
|
||||
<li>Selected Ember phase timings.</li>
|
||||
<li>Selected existing gtests.</li>
|
||||
<li>Sanitizer subset summary status from the weekly workflow.</li>
|
||||
<li>Release tags, when a run was triggered by a published release.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<script src="https://cdn.jsdelivr.net/npm/highcharts@12.4.0/highcharts.js"></script>
|
||||
<script src="dashboard.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -15,10 +15,11 @@ decompress and put the `raw_meshes` directory inside `testfiles`.
|
||||
1. Download the dataset.
|
||||
2. Build the binary with `MANIFOLD_PAR` and `ASSIMP_ENABLE` enabled. The
|
||||
binary should be inside `build/extras` (relative to the project root).
|
||||
3. Run `python do_ember_tests.py`. This will take 10 minutes to an hour
|
||||
depending on your machine. At the end, a `benchmark.csv` file will be
|
||||
generated.
|
||||
4. Run `python analyze_ember_tests.py` to analyze the result.
|
||||
3. Run the selected Ember benchmark cases through
|
||||
`scripts/run_weekly_benchmarks.py`, or run `man_bench` directly with the
|
||||
case arguments from `testfiles/ember-benchmark-cases.json`.
|
||||
4. Run `python analyze_ember_tests.py` to analyze the full `benchmark.csv`
|
||||
result.
|
||||
|
||||
## Files
|
||||
|
||||
@@ -30,13 +31,8 @@ decompress and put the `raw_meshes` directory inside `testfiles`.
|
||||
file with the argument and timing information.
|
||||
- `testfiles/test_file/ember-benchmark-cases.json`: JSON file describing each
|
||||
test case.
|
||||
- `do_ember_tests.py` reads the description of the files and transforms
|
||||
from the supplemental material of the Ember paper, and executes
|
||||
man_bench on each pair. This creates a benchmark.csv file (remember
|
||||
to remove that file again if you want a clean new run of data).
|
||||
- `analyze_ember_tests.py` reads the benchmark.csv file and calculates
|
||||
statistics and does a plot.
|
||||
- `bug-case.json` is a single test case that takes abnormally long.
|
||||
- `m3max_benchmarks.csv` is the benchmarks file gotten by running on a
|
||||
Macbook Pro M3 Max (12 performance cores, 4 efficiency cores).
|
||||
|
||||
|
||||
@@ -1,69 +0,0 @@
|
||||
import os
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import pathlib
|
||||
|
||||
ember_tests = pathlib.Path(__file__).parent
|
||||
root = ember_tests.parent.parent
|
||||
|
||||
meshdir = ember_tests.joinpath("testfiles/raw_meshes")
|
||||
testspecfile = ember_tests.joinpath("testfiles/ember-benchmark-cases.json")
|
||||
man_bench = root.joinpath("build/extras/man_bench")
|
||||
|
||||
threads_cases = [1, 2, 4, 6, 8, 10, 12, 14, 16]
|
||||
|
||||
meshfiles = os.listdir(meshdir)
|
||||
|
||||
id_to_file = {}
|
||||
for f in meshfiles:
|
||||
base, ext = os.path.splitext(f)
|
||||
if base and ext:
|
||||
try:
|
||||
baseid = int(base)
|
||||
id_to_file[baseid] = f
|
||||
except:
|
||||
pass
|
||||
|
||||
|
||||
def file_for_id(id):
|
||||
if id in id_to_file:
|
||||
file = id_to_file[id]
|
||||
fullfile = meshdir.joinpath(file)
|
||||
return fullfile
|
||||
return None
|
||||
|
||||
|
||||
with open(testspecfile, "r") as file:
|
||||
spec_data = json.load(file)
|
||||
|
||||
for threads in threads_cases:
|
||||
spec_index = 0
|
||||
for spec in spec_data:
|
||||
id1 = spec["id_a"]
|
||||
id2 = spec["id_b"]
|
||||
tr1 = spec["transform_a"]
|
||||
tr2 = spec["transform_b"]
|
||||
file1 = file_for_id(id1)
|
||||
if not file1:
|
||||
print("no test file for ", id1)
|
||||
file2 = file_for_id(id2)
|
||||
if not file2:
|
||||
print("no test file for ", id2)
|
||||
args = [man_bench, file1, file2]
|
||||
for i in [1, 2]:
|
||||
args.append("-t%d" % i)
|
||||
tr = tr1 if i == 1 else tr2
|
||||
for col in ["col0", "col1", "col2", "col3"]:
|
||||
for row in ["x", "y", "z"]:
|
||||
v = tr[col][row]
|
||||
args.append("%f" % v)
|
||||
if threads != 0:
|
||||
args.append("--threads")
|
||||
args.append("%d" % threads)
|
||||
print("test", spec_index, "threads", threads)
|
||||
cp = subprocess.run(args)
|
||||
if cp.returncode != 0:
|
||||
print("return code", cp.returncode)
|
||||
sys.exit(1)
|
||||
spec_index = spec_index + 1
|
||||
@@ -155,7 +155,7 @@ static MeshGL do_boolean(const std::string& file1, const std::string& file2,
|
||||
|
||||
static void usage() {
|
||||
std::cout << "Usage: %s file1 file2 [-t1 <12 floats>] [-t2 <12 floats>] "
|
||||
"[--threads <int>]\n";
|
||||
"[--threads <int>] [--verbose <int>]\n";
|
||||
}
|
||||
|
||||
int main(int argc, const char** argv) {
|
||||
@@ -170,6 +170,7 @@ int main(int argc, const char** argv) {
|
||||
mat3x4 transform1 = linalg::identity;
|
||||
mat3x4 transform2 = linalg::identity;
|
||||
int threads = 1000;
|
||||
int manifold_verbose = 0;
|
||||
if (argc > 3) {
|
||||
int argi = 3;
|
||||
while (argi < argc) {
|
||||
@@ -201,12 +202,25 @@ int main(int argc, const char** argv) {
|
||||
return 1;
|
||||
}
|
||||
argi += 2;
|
||||
} else if (strcmp(argv[argi], "--verbose") == 0) {
|
||||
if (argi + 1 >= argc) {
|
||||
usage();
|
||||
return 1;
|
||||
}
|
||||
try {
|
||||
manifold_verbose = std::stoi(argv[argi + 1]);
|
||||
} catch (...) {
|
||||
usage();
|
||||
return 1;
|
||||
}
|
||||
argi += 2;
|
||||
} else {
|
||||
usage();
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
ManifoldParams().verbose = manifold_verbose;
|
||||
if (verbose) {
|
||||
std::cout << "files: " << file1 << ", " << file2 << "\n";
|
||||
if (threads > 0) {
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Regular → Executable
+7
-7
@@ -1,17 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
#!/usr/bin/env -S bash -euo pipefail
|
||||
|
||||
# Only CXX flags are set: manifold is declared `project(manifold LANGUAGES CXX)`,
|
||||
# so cmake never reads CMAKE_C_FLAGS.
|
||||
#
|
||||
# MANIFOLD_STRICT is deliberately OFF here - it adds -Werror, and this lane is
|
||||
# for finding runtime memory/UB bugs, not for enforcing warnings.
|
||||
cmake \
|
||||
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
|
||||
-DBUILD_SHARED_LIBS=ON \
|
||||
-DMANIFOLD_STRICT=ON \
|
||||
-DMANIFOLD_PYBIND=OFF \
|
||||
-DMANIFOLD_STRICT=OFF \
|
||||
-DMANIFOLD_DEBUG=ON \
|
||||
-DMANIFOLD_ASSERT=ON \
|
||||
-DMANIFOLD_PAR=OFF \
|
||||
-DCMAKE_C_COMPILER=clang-18 \
|
||||
-DCMAKE_CXX_COMPILER=clang++-18 \
|
||||
-DCMAKE_C_FLAGS="-fsanitize=address,undefined" \
|
||||
-DCMAKE_CXX_FLAGS="-fsanitize=address,undefined" \
|
||||
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address,undefined" \
|
||||
-DCMAKE_SHARED_LINKER_FLAGS="-fsanitize=address,undefined" \
|
||||
|
||||
@@ -11,38 +11,25 @@ case "${SANITIZER_SUBSET}" in
|
||||
SANITIZER_GTEST_FILTER="${SANITIZER_GTEST_FILTER_EXTENDED}"
|
||||
;;
|
||||
*)
|
||||
echo "::warning::Unknown SANITIZER_SUBSET=${SANITIZER_SUBSET}, falling back to core."
|
||||
SANITIZER_GTEST_FILTER="${SANITIZER_GTEST_FILTER_CORE}"
|
||||
echo "::error::Unknown SANITIZER_SUBSET=${SANITIZER_SUBSET}"
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
SANITIZER_TEST_BIN=""
|
||||
for CANDIDATE in \
|
||||
./build/test/manifold_test \
|
||||
./test/manifold_test \
|
||||
./sanitizer-build/test/manifold_test \
|
||||
./sanitizer-build/build/test/manifold_test; do
|
||||
if [ -f "${CANDIDATE}" ]; then
|
||||
SANITIZER_TEST_BIN="${CANDIDATE}"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -z "${SANITIZER_TEST_BIN}" ]; then
|
||||
SANITIZER_TEST_BIN="$(find . -type f -name manifold_test | head -n 1)"
|
||||
fi
|
||||
if [ -z "${SANITIZER_TEST_BIN}" ]; then
|
||||
echo "::error::Could not find manifold_test after downloading sanitizer artifacts."
|
||||
# The sanitizer build artifact is always downloaded into build/.
|
||||
SANITIZER_TEST_BIN=./build/test/manifold_test
|
||||
if [ ! -f "${SANITIZER_TEST_BIN}" ]; then
|
||||
echo "::error::Could not find ${SANITIZER_TEST_BIN} after downloading sanitizer artifacts."
|
||||
echo "::group::Artifact layout"
|
||||
find . -maxdepth 4 -type d | sort
|
||||
echo "::endgroup::"
|
||||
exit 127
|
||||
fi
|
||||
chmod +x "${SANITIZER_TEST_BIN}" || true
|
||||
# Artifact downloads do not preserve the executable bit.
|
||||
chmod +x "${SANITIZER_TEST_BIN}"
|
||||
|
||||
set +e
|
||||
timeout "${SANITIZER_TEST_TIMEOUT_SEC}" \
|
||||
"${SANITIZER_TEST_BIN}" --gtest_filter="${SANITIZER_GTEST_FILTER}"
|
||||
timeout "${SANITIZER_TEST_TIMEOUT_SEC}" "${SANITIZER_TEST_BIN}" --gtest_filter="${SANITIZER_GTEST_FILTER}"
|
||||
TEST_RC="$?"
|
||||
set -e
|
||||
if [ "${TEST_RC}" -eq 124 ]; then
|
||||
|
||||
@@ -0,0 +1,426 @@
|
||||
#!/usr/bin/env python3
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from compare_pr_perf_guard import compute_stats
|
||||
from compare_pr_perf_guard import parse_suite as parse_perf_run_suite
|
||||
from system_metadata import resolve_metadata as resolve_system_metadata
|
||||
|
||||
SCHEMA_VERSION = "1.0.0"
|
||||
EMBER_SUITE = "weekly_ember_phase"
|
||||
PERF_SUITE = "perf_size_sweep"
|
||||
GTEST_SUITE = "existing_gtests"
|
||||
|
||||
EMBER_CASE_PATTERN = re.compile(r"^### case\s+([0-9]+)(?:\s+\([0-9]+\s+vs\s+[0-9]+\))?")
|
||||
PHASE_PATTERN = re.compile(r"^-+\s+([0-9]+)\s+ms for\s+(.*)$")
|
||||
VERTS_PATTERN = re.compile(r"^[0-9]+\s+verts and\s+[0-9]+\s+tris$")
|
||||
|
||||
INDEPENDENT_PHASES = [
|
||||
"Intersect12 P->Q",
|
||||
"Intersect12 Q->P",
|
||||
"Winding03 P",
|
||||
"Winding03 Q",
|
||||
]
|
||||
|
||||
|
||||
def summarize(values: list[float], suffix: str = "") -> dict:
|
||||
stats = compute_stats(values)
|
||||
fields = {
|
||||
name: stats[name]
|
||||
for name in ("samples", "mean", "median", "sd", "min", "max")
|
||||
}
|
||||
return {
|
||||
**{f"{name}{suffix}": value for name, value in fields.items()},
|
||||
"n_runs": stats["n_runs"],
|
||||
}
|
||||
|
||||
|
||||
def run_files(suite_dir: Path) -> list[Path]:
|
||||
files = sorted(suite_dir.glob("run*.txt"))
|
||||
if not files:
|
||||
raise RuntimeError(f"No run*.txt files found in {suite_dir}")
|
||||
return files
|
||||
|
||||
|
||||
def load_ember_specs(source_dir: Path) -> list[dict]:
|
||||
spec_path = (
|
||||
source_dir
|
||||
/ "extras"
|
||||
/ "ember_tests"
|
||||
/ "testfiles"
|
||||
/ "ember-benchmark-cases.json"
|
||||
)
|
||||
return json.loads(spec_path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def parse_ember_run(run_path: Path, run_index: int) -> dict:
|
||||
cases = []
|
||||
current = None
|
||||
accepting_phases = False
|
||||
|
||||
def finish_current() -> None:
|
||||
if current is not None:
|
||||
cases.append(current)
|
||||
|
||||
for raw_line in run_path.read_text(encoding="utf-8").splitlines():
|
||||
line = raw_line.strip().lstrip("\ufeff")
|
||||
case_match = EMBER_CASE_PATTERN.match(line)
|
||||
if case_match:
|
||||
finish_current()
|
||||
current = {
|
||||
"case_index": int(case_match.group(1)),
|
||||
"phases_ms": {},
|
||||
}
|
||||
accepting_phases = True
|
||||
continue
|
||||
|
||||
if current is None or not accepting_phases:
|
||||
continue
|
||||
|
||||
phase_match = PHASE_PATTERN.match(line)
|
||||
if phase_match:
|
||||
phase = phase_match.group(2).strip()
|
||||
if phase in INDEPENDENT_PHASES and phase not in current["phases_ms"]:
|
||||
current["phases_ms"][phase] = float(phase_match.group(1))
|
||||
continue
|
||||
|
||||
if VERTS_PATTERN.match(line.strip()):
|
||||
accepting_phases = False
|
||||
|
||||
finish_current()
|
||||
if not cases:
|
||||
raise RuntimeError(f"No Ember benchmark cases found in {run_path}")
|
||||
|
||||
for case in cases:
|
||||
for phase in INDEPENDENT_PHASES:
|
||||
case["phases_ms"].setdefault(phase, 0.0)
|
||||
|
||||
return {"path": str(run_path), "run_index": run_index, "cases": cases}
|
||||
|
||||
|
||||
def parse_ember_suite(suite_dir: Path, source_dir: Path) -> dict:
|
||||
runs = [parse_ember_run(path, i + 1) for i, path in enumerate(run_files(suite_dir))]
|
||||
ember_specs = load_ember_specs(source_dir)
|
||||
case_order = [case["case_index"] for case in runs[0]["cases"]]
|
||||
|
||||
cases = []
|
||||
for position, case_index in enumerate(case_order):
|
||||
spec = ember_specs[case_index]
|
||||
phase_samples = {phase: [] for phase in INDEPENDENT_PHASES}
|
||||
full_phase_samples = []
|
||||
intersect12_share_samples = []
|
||||
|
||||
for run in runs:
|
||||
phases = run["cases"][position]["phases_ms"]
|
||||
full_phase_sum = sum(phases[phase] for phase in INDEPENDENT_PHASES)
|
||||
full_phase_samples.append(full_phase_sum)
|
||||
intersect12_sum = phases["Intersect12 P->Q"] + phases["Intersect12 Q->P"]
|
||||
intersect12_share_samples.append(
|
||||
intersect12_sum / full_phase_sum if full_phase_sum else 0.0
|
||||
)
|
||||
for phase in INDEPENDENT_PHASES:
|
||||
phase_samples[phase].append(phases[phase])
|
||||
|
||||
phase_metrics = {
|
||||
phase: summarize(samples, "_ms") for phase, samples in phase_samples.items()
|
||||
}
|
||||
dominant_phase = max(
|
||||
INDEPENDENT_PHASES, key=lambda phase: phase_metrics[phase]["mean_ms"]
|
||||
)
|
||||
cases.append(
|
||||
{
|
||||
"case_index": case_index,
|
||||
"id_a": spec["id_a"],
|
||||
"id_b": spec["id_b"],
|
||||
"full_phase_sum_ms": summarize(full_phase_samples, "_ms"),
|
||||
"intersect12_share": summarize(intersect12_share_samples),
|
||||
"dominant_phase": dominant_phase,
|
||||
"phases": phase_metrics,
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"type": "ember_phase",
|
||||
"description": "Selected Ember boolean phase timing cases",
|
||||
"phase_order": INDEPENDENT_PHASES,
|
||||
"case_order": case_order,
|
||||
"cases": cases,
|
||||
"runs": runs,
|
||||
}
|
||||
|
||||
|
||||
def ntri_from_benchmark_key(benchmark: str) -> int | str:
|
||||
prefix = "nTri="
|
||||
if benchmark.startswith(prefix):
|
||||
value = benchmark[len(prefix) :]
|
||||
if value.isdigit():
|
||||
return int(value)
|
||||
return benchmark
|
||||
|
||||
|
||||
def parse_perf_suite(suite_dir: Path) -> dict:
|
||||
parsed = parse_perf_run_suite(suite_dir)
|
||||
workload_order = [
|
||||
ntri_from_benchmark_key(benchmark) for benchmark in parsed["benchmark_order"]
|
||||
]
|
||||
workloads = []
|
||||
for benchmark, n_tri in zip(parsed["benchmark_order"], workload_order):
|
||||
metrics = parsed["benchmarks"][benchmark]
|
||||
time_ms_samples = [
|
||||
sample * 1000.0 for sample in metrics["timing_sec"]["samples"]
|
||||
]
|
||||
workloads.append(
|
||||
{
|
||||
"benchmark": benchmark,
|
||||
"n_tri": n_tri,
|
||||
"time_ms": summarize(time_ms_samples, "_ms"),
|
||||
"peak_rss_mb": metrics["peak_rss_mb"],
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"type": "perf_size_sweep",
|
||||
"description": "perfTest sphere boolean size sweep",
|
||||
"workload_order": workload_order,
|
||||
"workloads": workloads,
|
||||
"runs": parsed["runs"],
|
||||
}
|
||||
|
||||
|
||||
def parse_gtest_run(run_path: Path, run_index: int) -> dict:
|
||||
# run_path is a gtest --gtest_output=json report; a passing test has no
|
||||
# "failures" key, matching the old parser's "only count [ OK ] lines"
|
||||
# behavior. time is a string in seconds, e.g. "0.024s" or "0s".
|
||||
report = json.loads(run_path.read_text(encoding="utf-8-sig"))
|
||||
tests = []
|
||||
for testsuite in report.get("testsuites", []):
|
||||
for test in testsuite.get("testsuite", []):
|
||||
if "failures" in test:
|
||||
continue
|
||||
name = f"{test['classname']}.{test['name']}"
|
||||
time_ms = float(test["time"].rstrip("s")) * 1000.0
|
||||
tests.append({"name": name, "time_ms": time_ms})
|
||||
return {"path": str(run_path), "run_index": run_index, "tests": tests}
|
||||
|
||||
|
||||
def gtest_run_files(suite_dir: Path) -> list[Path]:
|
||||
files = sorted(suite_dir.glob("run*.json"))
|
||||
if not files:
|
||||
raise RuntimeError(f"No run*.json files found in {suite_dir}")
|
||||
return files
|
||||
|
||||
|
||||
def parse_gtest_suite(suite_dir: Path) -> dict:
|
||||
runs = [
|
||||
parse_gtest_run(path, i + 1)
|
||||
for i, path in enumerate(gtest_run_files(suite_dir))
|
||||
]
|
||||
test_order = [item["name"] for item in runs[0]["tests"]]
|
||||
|
||||
tests = []
|
||||
for position, name in enumerate(test_order):
|
||||
samples = [run["tests"][position]["time_ms"] for run in runs]
|
||||
tests.append({"name": name, "time_ms": summarize(samples, "_ms")})
|
||||
|
||||
return {
|
||||
"type": "existing_gtests",
|
||||
"description": "Selected existing regression tests used as weekly benchmark signals",
|
||||
"test_order": test_order,
|
||||
"tests": tests,
|
||||
"runs": runs,
|
||||
}
|
||||
|
||||
|
||||
def resolve_metadata(args: argparse.Namespace) -> dict:
|
||||
metadata = resolve_system_metadata(args)
|
||||
return {"schema_version": SCHEMA_VERSION, **metadata}
|
||||
|
||||
|
||||
def build_ember_summary(suite: dict) -> list[str]:
|
||||
lines = []
|
||||
lines.append("#### Ember Phase Timings")
|
||||
lines.append("")
|
||||
lines.append(
|
||||
"| Case | Dominant phase | Full mean (ms) | Intersect12 share | P->Q | Q->P | Winding P | Winding Q | Runs |"
|
||||
)
|
||||
lines.append("|---:|---|---:|---:|---:|---:|---:|---:|---:|")
|
||||
|
||||
for case in sorted(
|
||||
suite["cases"],
|
||||
key=lambda entry: entry["full_phase_sum_ms"]["mean_ms"],
|
||||
reverse=True,
|
||||
):
|
||||
phases = case["phases"]
|
||||
lines.append(
|
||||
"| {case_index} | {dominant_phase} | {full:.2f} | {share:.3f} | "
|
||||
"{p_to_q:.2f} | {q_to_p:.2f} | {winding_p:.2f} | "
|
||||
"{winding_q:.2f} | {runs} |".format(
|
||||
case_index=case["case_index"],
|
||||
dominant_phase=case["dominant_phase"],
|
||||
full=case["full_phase_sum_ms"]["mean_ms"],
|
||||
share=case["intersect12_share"]["mean"],
|
||||
p_to_q=phases["Intersect12 P->Q"]["mean_ms"],
|
||||
q_to_p=phases["Intersect12 Q->P"]["mean_ms"],
|
||||
winding_p=phases["Winding03 P"]["mean_ms"],
|
||||
winding_q=phases["Winding03 Q"]["mean_ms"],
|
||||
runs=case["full_phase_sum_ms"]["n_runs"],
|
||||
)
|
||||
)
|
||||
|
||||
lines.append("")
|
||||
lines.append(
|
||||
"Note: phase timings cover `Intersect12` and `Winding03` only; `Intersections (total)` is excluded from the denominator."
|
||||
)
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def build_perf_summary(suite: dict) -> list[str]:
|
||||
lines = []
|
||||
lines.append("#### perfTest Size Sweep")
|
||||
lines.append("")
|
||||
lines.append(
|
||||
"| nTri | Mean (ms) | Median (ms) | Min (ms) | Max (ms) | "
|
||||
"Peak RSS mean (MB) | Peak RSS min (MB) | Peak RSS max (MB) | Runs |"
|
||||
)
|
||||
lines.append("|---:|---:|---:|---:|---:|---:|---:|---:|---:|")
|
||||
for workload in suite["workloads"]:
|
||||
timing = workload["time_ms"]
|
||||
peak_rss = workload["peak_rss_mb"]
|
||||
# peak_rss_mb can be None if RSS data was missing for every repeat
|
||||
# (see compare_pr_perf_guard.py) - show N/A rather than crashing.
|
||||
if peak_rss:
|
||||
rss_mean = f"{peak_rss['mean']:.2f}"
|
||||
rss_min = f"{peak_rss['min']:.2f}"
|
||||
rss_max = f"{peak_rss['max']:.2f}"
|
||||
else:
|
||||
rss_mean = rss_min = rss_max = "N/A"
|
||||
lines.append(
|
||||
"| {n_tri} | {mean_ms:.2f} | {median_ms:.2f} | {min_ms:.2f} | "
|
||||
"{max_ms:.2f} | {rss_mean} | {rss_min} | {rss_max} | "
|
||||
"{n_runs} |".format(
|
||||
n_tri=workload["n_tri"],
|
||||
rss_mean=rss_mean,
|
||||
rss_min=rss_min,
|
||||
rss_max=rss_max,
|
||||
**timing,
|
||||
)
|
||||
)
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def build_gtest_summary(suite: dict) -> list[str]:
|
||||
lines = []
|
||||
lines.append("#### Existing Regression Tests")
|
||||
lines.append("")
|
||||
lines.append("| Test | Mean (ms) | Median (ms) | Min (ms) | Max (ms) | Runs |")
|
||||
lines.append("|---|---:|---:|---:|---:|---:|")
|
||||
if not suite["tests"]:
|
||||
lines.append("| No tests selected | 0 | 0 | 0 | 0 | 0 |")
|
||||
for test in suite["tests"]:
|
||||
timing = test["time_ms"]
|
||||
lines.append(
|
||||
"| {name} | {mean_ms:.2f} | {median_ms:.2f} | {min_ms:.2f} | "
|
||||
"{max_ms:.2f} | {n_runs} |".format(
|
||||
name=test["name"],
|
||||
**timing,
|
||||
)
|
||||
)
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def build_summary(suites: dict, metadata: dict, repeats: int) -> str:
|
||||
lines = []
|
||||
lines.append("### Weekly Benchmarks")
|
||||
lines.append("")
|
||||
lines.append(f"Commit: `{metadata['commit_sha'] or 'unknown'}`")
|
||||
lines.append(f"Runner: `{metadata['runner'] or 'unknown'}`")
|
||||
lines.append(f"OS: `{metadata['os'] or 'unknown'}`")
|
||||
lines.append(f"Compiler: `{metadata['compiler'] or 'unknown'}`")
|
||||
lines.append(f"CPU: `{metadata['cpu_model'] or 'unknown'}`")
|
||||
lines.append(f"CPU count: `{metadata['cpu_count'] or 'unknown'}`")
|
||||
if metadata.get("cpu_model_identifier"):
|
||||
lines.append(f"CPU model identifier: `{metadata['cpu_model_identifier']}`")
|
||||
if metadata.get("cpu_physical_count"):
|
||||
lines.append(f"CPU physical cores: `{metadata['cpu_physical_count']}`")
|
||||
if metadata.get("cpu_performance_core_count") is not None:
|
||||
lines.append(
|
||||
f"CPU performance cores: `{metadata['cpu_performance_core_count']}`"
|
||||
)
|
||||
if metadata.get("cpu_efficiency_core_count") is not None:
|
||||
lines.append(
|
||||
f"CPU efficiency cores: `{metadata['cpu_efficiency_core_count']}`"
|
||||
)
|
||||
lines.append(f"Repeats: `{repeats}`")
|
||||
lines.append("")
|
||||
|
||||
if EMBER_SUITE in suites:
|
||||
lines.extend(build_ember_summary(suites[EMBER_SUITE]))
|
||||
if PERF_SUITE in suites:
|
||||
lines.extend(build_perf_summary(suites[PERF_SUITE]))
|
||||
if GTEST_SUITE in suites:
|
||||
lines.extend(build_gtest_summary(suites[GTEST_SUITE]))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def parse_suites(root_dir: Path, source_dir: Path) -> dict:
|
||||
suites = {}
|
||||
ember_dir = root_dir / "ember_phase"
|
||||
perf_dir = root_dir / "perf_size_sweep"
|
||||
gtest_dir = root_dir / "existing_gtests"
|
||||
|
||||
if ember_dir.exists():
|
||||
suites[EMBER_SUITE] = parse_ember_suite(ember_dir, source_dir)
|
||||
|
||||
if perf_dir.exists():
|
||||
suites[PERF_SUITE] = parse_perf_suite(perf_dir)
|
||||
|
||||
if gtest_dir.exists():
|
||||
suites[GTEST_SUITE] = parse_gtest_suite(gtest_dir)
|
||||
|
||||
if not suites:
|
||||
raise RuntimeError(f"No weekly benchmark suites found in {root_dir}")
|
||||
return suites
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Parse weekly benchmark runs.")
|
||||
parser.add_argument("--suite-dir", required=True, type=Path)
|
||||
parser.add_argument("--source-dir", default=Path("."), type=Path)
|
||||
parser.add_argument("--markdown-out", required=True, type=Path)
|
||||
parser.add_argument("--json-out", required=True, type=Path)
|
||||
parser.add_argument("--repeats", required=True, type=int)
|
||||
parser.add_argument("--commit-sha")
|
||||
parser.add_argument("--workflow")
|
||||
parser.add_argument("--runner")
|
||||
parser.add_argument("--os-name")
|
||||
parser.add_argument("--compiler")
|
||||
parser.add_argument("--cpu-model")
|
||||
parser.add_argument("--cpu-count", type=int)
|
||||
args = parser.parse_args()
|
||||
|
||||
metadata = resolve_metadata(args)
|
||||
suites = parse_suites(args.suite_dir, args.source_dir)
|
||||
suite_names = list(suites.keys())
|
||||
markdown = build_summary(suites, metadata, args.repeats)
|
||||
payload = {
|
||||
"metadata": metadata,
|
||||
"config": {
|
||||
"repeats": args.repeats,
|
||||
"suites": suite_names,
|
||||
},
|
||||
"suites": suites,
|
||||
}
|
||||
|
||||
args.markdown_out.write_text(markdown + "\n", encoding="utf-8")
|
||||
args.json_out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,151 @@
|
||||
#!/usr/bin/env python3
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
|
||||
# Metadata fields copied into each index entry. The index is a lightweight
|
||||
# listing, so it deliberately carries a subset - the full metadata (cmake
|
||||
# details, per-core CPU counts, ...) stays in the run's result.json.
|
||||
INDEX_METADATA_FIELDS = (
|
||||
"commit_sha",
|
||||
"workflow",
|
||||
"runner",
|
||||
"os",
|
||||
"compiler",
|
||||
"cpu_model",
|
||||
"cpu_count",
|
||||
)
|
||||
|
||||
|
||||
def utc_now() -> datetime.datetime:
|
||||
return datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0)
|
||||
|
||||
|
||||
def parse_timestamp(raw: str) -> datetime.datetime:
|
||||
return datetime.datetime.fromisoformat(raw.replace("Z", "+00:00")).astimezone(
|
||||
datetime.timezone.utc
|
||||
)
|
||||
|
||||
|
||||
def json_dump(path: Path, payload: dict) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(
|
||||
json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def github_run_url(run_id: str) -> str | None:
|
||||
server = os.getenv("GITHUB_SERVER_URL")
|
||||
repo = os.getenv("GITHUB_REPOSITORY")
|
||||
if not server or not repo or not os.getenv("GITHUB_RUN_ID"):
|
||||
return None
|
||||
return f"{server}/{repo}/actions/runs/{run_id}"
|
||||
|
||||
|
||||
def load_index(index_path: Path) -> dict:
|
||||
if not index_path.exists():
|
||||
return {"schema_version": SCHEMA_VERSION, "runs": []}
|
||||
return json.loads(index_path.read_text(encoding="utf-8-sig"))
|
||||
|
||||
|
||||
def optional_field(payload: dict, key: str, value: str | int | None) -> None:
|
||||
if value not in (None, ""):
|
||||
payload[key] = value
|
||||
|
||||
|
||||
def sanitizer_summary(args: argparse.Namespace) -> dict | None:
|
||||
summary = {}
|
||||
optional_field(summary, "subset", args.sanitizer_subset)
|
||||
optional_field(summary, "build_result", args.sanitizer_build_result)
|
||||
optional_field(summary, "test_result", args.sanitizer_test_result)
|
||||
optional_field(summary, "runner", args.sanitizer_runner)
|
||||
optional_field(summary, "os", args.sanitizer_os)
|
||||
return summary or None
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Publish weekly benchmark result files.")
|
||||
parser.add_argument("suite_dir", type=Path)
|
||||
parser.add_argument("data_dir", type=Path)
|
||||
parser.add_argument("--release-tag")
|
||||
parser.add_argument("--sanitizer-subset")
|
||||
parser.add_argument("--sanitizer-build-result")
|
||||
parser.add_argument("--sanitizer-test-result")
|
||||
parser.add_argument("--sanitizer-runner")
|
||||
parser.add_argument("--sanitizer-os")
|
||||
args = parser.parse_args()
|
||||
|
||||
result_path = args.suite_dir / "result.json"
|
||||
summary_path = args.suite_dir / "summary.md"
|
||||
if not result_path.exists():
|
||||
raise FileNotFoundError(f"Missing weekly benchmark result: {result_path}")
|
||||
if not summary_path.exists():
|
||||
raise FileNotFoundError(f"Missing weekly benchmark summary: {summary_path}")
|
||||
|
||||
result = json.loads(result_path.read_text(encoding="utf-8-sig"))
|
||||
metadata = result["metadata"]
|
||||
timestamp = parse_timestamp(metadata["timestamp"])
|
||||
timestamp_iso = timestamp.isoformat().replace("+00:00", "Z")
|
||||
run_id = os.getenv("GITHUB_RUN_ID") or timestamp.strftime("%Y%m%dT%H%M%SZ")
|
||||
|
||||
dated_rel = (
|
||||
Path("weekly")
|
||||
/ timestamp.strftime("%Y")
|
||||
/ timestamp.strftime("%m")
|
||||
/ timestamp.strftime("%d")
|
||||
/ run_id
|
||||
)
|
||||
dated_dir = args.data_dir / dated_rel
|
||||
dated_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
entry = {
|
||||
"run_id": run_id,
|
||||
"timestamp": timestamp_iso,
|
||||
"date": timestamp.strftime("%Y-%m-%d"),
|
||||
**{name: metadata[name] for name in INDEX_METADATA_FIELDS},
|
||||
"result_path": str(dated_rel / "result.json").replace("\\", "/"),
|
||||
"summary_path": str(dated_rel / "summary.md").replace("\\", "/"),
|
||||
"github_run_url": github_run_url(run_id),
|
||||
}
|
||||
if args.release_tag:
|
||||
entry["release_tag"] = args.release_tag
|
||||
sanitizer = sanitizer_summary(args)
|
||||
if sanitizer:
|
||||
entry["sanitizer"] = sanitizer
|
||||
entry["trigger"] = "release" if args.release_tag else "weekly"
|
||||
|
||||
result_dest = dated_dir / "result.json"
|
||||
summary_dest = dated_dir / "summary.md"
|
||||
stored_result = dict(result)
|
||||
stored_result["storage"] = entry
|
||||
json_dump(result_dest, stored_result)
|
||||
shutil.copy2(summary_path, summary_dest)
|
||||
|
||||
json_dump(args.data_dir / "weekly" / "latest.json", stored_result)
|
||||
|
||||
index_path = args.data_dir / "weekly" / "index.json"
|
||||
index = load_index(index_path)
|
||||
runs = [run for run in index.get("runs", []) if str(run.get("run_id")) != run_id]
|
||||
runs.append(entry)
|
||||
runs.sort(key=lambda run: run.get("timestamp", ""))
|
||||
index = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"generated_at": utc_now().isoformat().replace("+00:00", "Z"),
|
||||
"latest_run_id": run_id,
|
||||
"runs": runs,
|
||||
}
|
||||
json_dump(index_path, index)
|
||||
|
||||
print(f"Published weekly benchmark result to {result_dest}")
|
||||
print(f"Updated {index_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,75 @@
|
||||
#!/usr/bin/env python3
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import parse_weekly_benchmarks as pwb
|
||||
|
||||
|
||||
def print_file(path: Path) -> None:
|
||||
sys.stdout.write(path.read_text(encoding="utf-8-sig"))
|
||||
|
||||
|
||||
def raw_run_files(suite_dir: Path) -> list[Path]:
|
||||
return sorted(
|
||||
path
|
||||
for path in suite_dir.rglob("run*.txt")
|
||||
if path.is_file() and "build" not in path.relative_to(suite_dir).parts
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if hasattr(sys.stdout, "reconfigure"):
|
||||
sys.stdout.reconfigure(encoding="utf-8")
|
||||
|
||||
parser = argparse.ArgumentParser(description="Report weekly benchmark results.")
|
||||
parser.add_argument("source_dir", type=Path)
|
||||
parser.add_argument("suite_dir", type=Path)
|
||||
parser.add_argument("repeats", type=int)
|
||||
args = parser.parse_args()
|
||||
|
||||
summary_path = args.suite_dir / "summary.md"
|
||||
result_path = args.suite_dir / "result.json"
|
||||
|
||||
metadata = pwb.resolve_metadata(argparse.Namespace(suite_dir=args.suite_dir))
|
||||
suites = pwb.parse_suites(args.suite_dir, args.source_dir)
|
||||
markdown = pwb.build_summary(suites, metadata, args.repeats)
|
||||
payload = {
|
||||
"metadata": metadata,
|
||||
"config": {"repeats": args.repeats, "suites": list(suites.keys())},
|
||||
"suites": suites,
|
||||
}
|
||||
summary_path.write_text(markdown + "\n", encoding="utf-8")
|
||||
result_path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
print("::group::Weekly benchmark summary")
|
||||
print_file(summary_path)
|
||||
print("::endgroup::")
|
||||
|
||||
print("::group::Weekly benchmark result.json")
|
||||
print_file(result_path)
|
||||
print("::endgroup::")
|
||||
|
||||
print("::group::Weekly benchmark raw outputs")
|
||||
for run_file in raw_run_files(args.suite_dir):
|
||||
print(f"--- {run_file} ---")
|
||||
print_file(run_file)
|
||||
print("::endgroup::")
|
||||
|
||||
step_summary = os.getenv("GITHUB_STEP_SUMMARY")
|
||||
if step_summary:
|
||||
with Path(step_summary).open("a", encoding="utf-8") as out:
|
||||
out.write(summary_path.read_text(encoding="utf-8"))
|
||||
out.write("\n")
|
||||
out.write(
|
||||
"Raw logs: open this step and expand `Weekly benchmark result.json` "
|
||||
"and `Weekly benchmark raw outputs` groups.\n"
|
||||
)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,326 @@
|
||||
#!/usr/bin/env python3
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from benchmark_pr_perf_guard import NUM_SIZES as PERF_SIZE_COUNT
|
||||
from benchmark_pr_perf_guard import run_measured_perf_size
|
||||
|
||||
DEFAULT_EMBER_CASES = "16 84 667 695 260 406 551 582"#valid cases 1 to 1000
|
||||
DEFAULT_GTEST_FILTER = ":".join(
|
||||
[
|
||||
"Manifold.DeepChainDoesNotOverflowNumLeaves",
|
||||
"Boolean.BatchBoolean",
|
||||
"CrossSection.BatchBoolean",
|
||||
"Polygon.Sponge4",
|
||||
"Polygon.Zebra1",
|
||||
"Polygon.Zebra3",
|
||||
"ExecutionContextFromMeshGL.CancelConcurrent",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BuildContext:
|
||||
source_dir: Path
|
||||
out_dir: Path
|
||||
repeats: int
|
||||
build_dir: Path
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BenchmarkSuite:
|
||||
name: str
|
||||
target: str
|
||||
binary: str
|
||||
run: object
|
||||
|
||||
|
||||
def parse_cases(raw: str) -> list[int]:
|
||||
cases = []
|
||||
for part in raw.replace(",", " ").split():
|
||||
cases.append(int(part))
|
||||
if not cases:
|
||||
raise ValueError("No weekly benchmark cases were provided")
|
||||
return cases
|
||||
|
||||
|
||||
def transform_args(prefix: str, transform: dict) -> list[str]:
|
||||
args = [prefix]
|
||||
for col in ("col0", "col1", "col2", "col3"):
|
||||
for row in ("x", "y", "z"):
|
||||
args.append(f"{float(transform[col][row]):f}")
|
||||
return args
|
||||
|
||||
|
||||
def run_command(
|
||||
args: list[str], cwd: Path | None = None, log_path: Path | None = None
|
||||
) -> subprocess.CompletedProcess:
|
||||
print("+ " + " ".join(args))
|
||||
sys.stdout.flush()
|
||||
if log_path is None:
|
||||
return subprocess.run(args, check=True, cwd=cwd)
|
||||
|
||||
log_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
result = subprocess.run(
|
||||
args,
|
||||
check=False,
|
||||
cwd=cwd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
)
|
||||
output = result.stdout
|
||||
if output:
|
||||
print(output, end="")
|
||||
log_path.write_text("+ " + " ".join(args) + "\n" + output, encoding="utf-8")
|
||||
if result.returncode != 0:
|
||||
raise subprocess.CalledProcessError(
|
||||
result.returncode, args, output=result.stdout
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def configure_build(ctx: BuildContext) -> None:
|
||||
cmake_args = [
|
||||
"cmake",
|
||||
"-S",
|
||||
str(ctx.source_dir),
|
||||
"-B",
|
||||
str(ctx.build_dir),
|
||||
"-DCMAKE_BUILD_TYPE=Release",
|
||||
"-DMANIFOLD_STRICT=OFF",
|
||||
"-DMANIFOLD_PAR=ON",
|
||||
# TIMING gates the Boolean3 phase Timer output that the ember suite
|
||||
# parses - without it those phases silently report zero.
|
||||
"-DMANIFOLD_TIMING=ON",
|
||||
# ASSIMP is needed to load the ember test meshes.
|
||||
"-DASSIMP_ENABLE=ON",
|
||||
]
|
||||
if sys.platform == "darwin":
|
||||
cmake_args.append(
|
||||
f"-DCMAKE_OSX_ARCHITECTURES={os.getenv('CMAKE_OSX_ARCHITECTURES', 'arm64')}"
|
||||
)
|
||||
|
||||
run_command(
|
||||
cmake_args,
|
||||
log_path=ctx.out_dir / "cmake_configure.log",
|
||||
)
|
||||
|
||||
|
||||
def build_targets(ctx: BuildContext, suites: list[BenchmarkSuite]) -> None:
|
||||
targets = list(dict.fromkeys(suite.target for suite in suites))
|
||||
run_command(["cmake", "--build", str(ctx.build_dir), "--target", *targets])
|
||||
|
||||
|
||||
def find_binary(ctx: BuildContext, name: str) -> Path:
|
||||
candidates = [
|
||||
ctx.build_dir / "extras" / name,
|
||||
ctx.build_dir / "test" / name,
|
||||
ctx.build_dir / "bin" / name,
|
||||
ctx.build_dir / name,
|
||||
ctx.build_dir / "extras" / f"{name}.exe",
|
||||
ctx.build_dir / "test" / f"{name}.exe",
|
||||
ctx.build_dir / "bin" / f"{name}.exe",
|
||||
ctx.build_dir / f"{name}.exe",
|
||||
]
|
||||
for candidate in candidates:
|
||||
if candidate.exists() and candidate.is_file():
|
||||
return candidate
|
||||
joined = ", ".join(str(candidate) for candidate in candidates)
|
||||
raise FileNotFoundError(f"{name} binary not found in expected paths: {joined}")
|
||||
|
||||
|
||||
def get_mesh(mesh_id: int, mesh_dir: Path) -> Path:
|
||||
path = mesh_dir / f"{mesh_id}.stl"
|
||||
if path.exists() and path.stat().st_size > 0:
|
||||
return path
|
||||
raise FileNotFoundError(f"Required Ember benchmark mesh missing: {path}")
|
||||
|
||||
|
||||
def load_ember_specs(spec_path: Path, cases: list[int]) -> list[dict]:
|
||||
specs = json.loads(spec_path.read_text(encoding="utf-8"))
|
||||
selected = []
|
||||
for case_index in cases:
|
||||
spec = dict(specs[case_index])
|
||||
spec["case_index"] = case_index
|
||||
selected.append(spec)
|
||||
return selected
|
||||
|
||||
|
||||
def write_ember_cases_metadata(out_dir: Path, specs: list[dict]) -> None:
|
||||
with (out_dir / "cases.csv").open("w", newline="", encoding="utf-8") as f:
|
||||
writer = csv.DictWriter(f, fieldnames=["case_index", "id_a", "id_b"])
|
||||
writer.writeheader()
|
||||
for spec in specs:
|
||||
writer.writerow(
|
||||
{
|
||||
"case_index": spec["case_index"],
|
||||
"id_a": spec["id_a"],
|
||||
"id_b": spec["id_b"],
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def run_ember_case(
|
||||
binary: Path, spec: dict, mesh_dir: Path, threads: int, verbose: int, cwd: Path
|
||||
) -> str:
|
||||
mesh_a = get_mesh(int(spec["id_a"]), mesh_dir)
|
||||
mesh_b = get_mesh(int(spec["id_b"]), mesh_dir)
|
||||
args = [
|
||||
str(binary),
|
||||
str(mesh_a),
|
||||
str(mesh_b),
|
||||
*transform_args("-t1", spec["transform_a"]),
|
||||
*transform_args("-t2", spec["transform_b"]),
|
||||
"--threads",
|
||||
str(threads),
|
||||
"--verbose",
|
||||
str(verbose),
|
||||
]
|
||||
result = subprocess.run(
|
||||
args, check=True,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
|
||||
text=True, cwd=cwd,
|
||||
)
|
||||
return result.stdout
|
||||
|
||||
|
||||
def run_ember_suite(ctx: BuildContext, binary: Path) -> None:
|
||||
out_dir = ctx.out_dir / "ember_phase"
|
||||
mesh_dir = (
|
||||
ctx.source_dir / "extras" / "ember_tests" / "testfiles" / "raw_meshes"
|
||||
)
|
||||
spec_file = (
|
||||
ctx.source_dir
|
||||
/ "extras"
|
||||
/ "ember_tests"
|
||||
/ "testfiles"
|
||||
/ "ember-benchmark-cases.json"
|
||||
)
|
||||
cases = parse_cases(os.getenv("WEEKLY_BENCHMARK_CASES", DEFAULT_EMBER_CASES))
|
||||
threads = int(os.getenv("WEEKLY_BENCHMARK_THREADS", "1"))
|
||||
verbose = int(os.getenv("WEEKLY_BENCHMARK_VERBOSE", "2"))
|
||||
|
||||
specs = load_ember_specs(spec_file, cases)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
write_ember_cases_metadata(out_dir, specs)
|
||||
|
||||
for run_index in range(1, ctx.repeats + 1):
|
||||
run_file = out_dir / f"run{run_index}.txt"
|
||||
benchmark_csv = out_dir / "benchmark.csv"
|
||||
benchmark_csv.unlink(missing_ok=True)
|
||||
with run_file.open("w", encoding="utf-8") as out:
|
||||
for spec in specs:
|
||||
out.write(f"### case {spec['case_index']}\n")
|
||||
stdout = run_ember_case(
|
||||
binary.resolve(),
|
||||
spec,
|
||||
mesh_dir.resolve(),
|
||||
threads,
|
||||
verbose,
|
||||
out_dir.resolve(),
|
||||
)
|
||||
out.write(stdout)
|
||||
if not stdout.endswith("\n"):
|
||||
out.write("\n")
|
||||
|
||||
if benchmark_csv.exists():
|
||||
benchmark_csv.replace(out_dir / f"benchmark_run{run_index}.csv")
|
||||
|
||||
print(f"completed Ember benchmark run {run_index}/{ctx.repeats}")
|
||||
sys.stdout.flush()
|
||||
|
||||
|
||||
def run_perf_size_sweep_suite(ctx: BuildContext, binary: Path) -> None:
|
||||
out_dir = ctx.out_dir / "perf_size_sweep"
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
for run_index in range(1, ctx.repeats + 1):
|
||||
run_file = out_dir / f"run{run_index}.txt"
|
||||
with run_file.open("w", encoding="utf-8") as handle:
|
||||
for size_index in range(PERF_SIZE_COUNT):
|
||||
block, status = run_measured_perf_size(binary, size_index)
|
||||
handle.write(f"### perfTest size_index={size_index}\n{block}\n\n")
|
||||
handle.flush()
|
||||
if status != 0:
|
||||
raise RuntimeError(
|
||||
f"perfTest exited with status {status} "
|
||||
f"(run {run_index}, size_index={size_index}, dir={out_dir})"
|
||||
)
|
||||
print(f"completed perfTest run {run_index}/{ctx.repeats}")
|
||||
sys.stdout.flush()
|
||||
|
||||
|
||||
def run_existing_gtests_suite(ctx: BuildContext, binary: Path) -> None:
|
||||
out_dir = ctx.out_dir / "existing_gtests"
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
gtest_filter = os.getenv("WEEKLY_BENCHMARK_GTEST_FILTER", DEFAULT_GTEST_FILTER)
|
||||
for run_index in range(1, ctx.repeats + 1):
|
||||
# gtest's --gtest_output=json emits structured per-test results
|
||||
# directly, so parse_weekly_benchmarks.py doesn't need to regex the
|
||||
# console text; relative path avoids Windows drive-letter colons
|
||||
# being misread as the format:path separator.
|
||||
json_name = f"run{run_index}.json"
|
||||
result = subprocess.run(
|
||||
[
|
||||
str(binary),
|
||||
f"--gtest_filter={gtest_filter}",
|
||||
f"--gtest_output=json:{json_name}",
|
||||
],
|
||||
cwd=out_dir,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if result.stderr:
|
||||
print(result.stderr, end="", file=sys.stderr)
|
||||
(out_dir / f"run{run_index}.txt").write_text(result.stdout, encoding="utf-8")
|
||||
print(f"completed existing gtest run {run_index}/{ctx.repeats}")
|
||||
sys.stdout.flush()
|
||||
|
||||
|
||||
# Add future benchmark suites here.
|
||||
# emits run<N>.txt files that parse_weekly_benchmarks.py can collect.
|
||||
SUITES = [
|
||||
BenchmarkSuite("ember_phase", "man_bench", "man_bench", run_ember_suite),
|
||||
BenchmarkSuite(
|
||||
"perf_size_sweep", "perfTest", "perfTest", run_perf_size_sweep_suite
|
||||
),
|
||||
BenchmarkSuite(
|
||||
"existing_gtests", "manifold_test", "manifold_test", run_existing_gtests_suite
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Run weekly benchmark suites.")
|
||||
parser.add_argument("source_dir", type=Path)
|
||||
parser.add_argument("out_dir", type=Path)
|
||||
parser.add_argument("repeats", type=int)
|
||||
args = parser.parse_args()
|
||||
|
||||
ctx = BuildContext(
|
||||
source_dir=args.source_dir.resolve(),
|
||||
out_dir=args.out_dir.resolve(),
|
||||
repeats=args.repeats,
|
||||
build_dir=(args.out_dir / "build").resolve(),
|
||||
)
|
||||
|
||||
ctx.out_dir.mkdir(parents=True, exist_ok=True)
|
||||
configure_build(ctx)
|
||||
build_targets(ctx, SUITES)
|
||||
for suite in SUITES:
|
||||
print(f"::group::Run weekly benchmark suite: {suite.name}")
|
||||
binary = find_binary(ctx, suite.binary)
|
||||
suite.run(ctx, binary)
|
||||
print("::endgroup::")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -3,9 +3,39 @@ import argparse
|
||||
import datetime
|
||||
import os
|
||||
import platform
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
CMAKE_SUMMARY_PATTERN = re.compile(r"^--\s+([A-Z0-9_]+):\s*(.*)$")
|
||||
|
||||
|
||||
def parse_cmake_configure_log(log_path: Path) -> dict:
|
||||
if not log_path.exists():
|
||||
return {}
|
||||
|
||||
values = {}
|
||||
for line in log_path.read_text(encoding="utf-8", errors="ignore").splitlines():
|
||||
match = CMAKE_SUMMARY_PATTERN.match(line.strip())
|
||||
if match:
|
||||
values[match.group(1)] = match.group(2).strip()
|
||||
|
||||
return {
|
||||
"version": values.get("CMAKE_VERSION"),
|
||||
"generator": values.get("CMAKE_GENERATOR"),
|
||||
"build_type": values.get("CMAKE_BUILD_TYPE"),
|
||||
"cxx_compiler_id": values.get("CMAKE_CXX_COMPILER_ID"),
|
||||
"cxx_compiler_version": values.get("CMAKE_CXX_COMPILER_VERSION"),
|
||||
}
|
||||
|
||||
|
||||
def cmake_compiler(cmake: dict) -> str | None:
|
||||
compiler_id = cmake.get("cxx_compiler_id")
|
||||
compiler_version = cmake.get("cxx_compiler_version")
|
||||
if compiler_id and compiler_version:
|
||||
return f"{compiler_id} {compiler_version}"
|
||||
return compiler_id or compiler_version
|
||||
|
||||
|
||||
def detect_compiler() -> str | None:
|
||||
# best compiler fingerprint for metadata
|
||||
@@ -50,6 +80,8 @@ def int_or_none(value: str | None) -> int | None:
|
||||
|
||||
|
||||
def default_cpu_model() -> str | None:
|
||||
# /proc/cpuinfo gives a better CPU name than platform.processor()
|
||||
# on GitHub-hosted Ubuntu runners.
|
||||
cpuinfo = Path("/proc/cpuinfo")
|
||||
if not cpuinfo.exists():
|
||||
return platform.processor() or None
|
||||
@@ -94,14 +126,21 @@ def cpu_details() -> dict:
|
||||
def resolve_metadata(args: argparse.Namespace) -> dict:
|
||||
# resolve metadata from args first, then GitHub env vars
|
||||
timestamp = datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0)
|
||||
suite_dir = getattr(args, "suite_dir", None)
|
||||
cmake = parse_cmake_configure_log(suite_dir / "cmake_configure.log") if suite_dir else {}
|
||||
cpu = cpu_details()
|
||||
cpu_count = getattr(args, "cpu_count", None) or cpu["logical_count"]
|
||||
return {
|
||||
"commit_sha": args.commit_sha or os.getenv("GITHUB_SHA"),
|
||||
"workflow": args.workflow or os.getenv("GITHUB_WORKFLOW"),
|
||||
"runner": args.runner or os.getenv("RUNNER_NAME"),
|
||||
"os": args.os_name or os.getenv("RUNNER_OS"),
|
||||
"compiler": args.compiler or detect_compiler(),
|
||||
"cpu_model": cpu["model"],
|
||||
"commit_sha": getattr(args, "commit_sha", None) or os.getenv("GITHUB_SHA"),
|
||||
"workflow": getattr(args, "workflow", None) or os.getenv("GITHUB_WORKFLOW"),
|
||||
"runner": getattr(args, "runner", None) or os.getenv("RUNNER_NAME"),
|
||||
"os": getattr(args, "os_name", None) or os.getenv("RUNNER_OS"),
|
||||
"compiler": getattr(args, "compiler", None)
|
||||
or cmake_compiler(cmake)
|
||||
or detect_compiler(),
|
||||
"cmake": cmake,
|
||||
"cpu_model": getattr(args, "cpu_model", None) or cpu["model"],
|
||||
"cpu_count": cpu_count,
|
||||
"cpu_brand": cpu["brand"],
|
||||
"cpu_model_identifier": cpu["model_identifier"],
|
||||
"cpu_arch": cpu["arch"],
|
||||
|
||||
Reference in New Issue
Block a user