Reorganize shootout benchmarks into a single directory (#260)

* Move all `shootout` benchmarks into a single directory

Now that #251 and #256 make it possible for more than one benchmark to
live in a single directory, this change moves all of the shootout
artifacts into a single directory. This simply performs the file
movement; subsequent commits will make necessary tweaks.

* Rename shootout benchmarks in `*.suite` files

* Enable native benchmarking in new `shootout` directory

This change refactors how the shootout native benchmarks are built. The
`Dockerfile.native` file is retained and is expected to be _the_ way to
build the native shared libraries for this kind of benchmarking. A
`build-native.sh` script is included in the directory to (a) be used by
`Dockerfile.native` and (b) for building the native benchmarks in
environments where running Docker may not be possible.

Now that all of the benchmarks are built in one directory, the native
libraries cannot all be named `benchmark.so`. Because of this and the
hard-coded path expected by the native engine (see #259), this change
also modifies the associated `*-native.sh` scripts to set up a temporary
directory that looks like the `benchmark.so` environment that was there
previously. This additional logic could be removed once #259 is fixed.

* Remove the original `shootout-*` directories

These are all migrated over to be a part of the single `shootout`
directory.

* Update verbiage in native GitHub action

* Update `ackermann` to use new `*.input` paths

The new file structure for `shootout` now expects these paths to look
like `shootout-ackermann.*.input`.

* Fix `heapsort` allocation

When we allocate the array to sort, we should do so with items of size
`double` (64 bits) instead of `double*` (32 bits in WebAssembly). I am
very confused as to why this benchmark worked previously, but when I
recompiled it prior to this change, it would invariably fail due to
accessing addresses beyond the memory bounds.

* Recompile `shootout` benchmarks with wasi-sdk v20

* Tweak native scripts

This change fixes some issues highlighted by CI:
- it adds more verbose output to see which commands are executed
- it improves the documentation to clarify how to use certain flags
- it fixes slight mistakes in the scripts missed by previous refactoring
- and, __most especially__, it alters the order of the parameters passed
  to compile the native libraries.

This last change is indicative of the fragility of the native
benchmarks: apparently moving `-lengine` to the end was necessary for
the linker to understand which library provides `bench_start` and
`bench_end`.

* Update documentation

Now that `Dockerfile.native` relies on a script, `build-native.sh`,
instead of the Cargo build system, the documentation for building native
libraries has to change.

.github/workflows/benchmarks_native.yaml

@@ -7,14 +7,14 @@ on:
     branches: [ main ]
 
 jobs:
-
-  Build_Native_X86-64:
+  native:
+    name: Build and run native benchmarks
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
     - name: Install nightly
       uses: actions-rs/toolchain@v1
       with:
         toolchain: nightly
-    - name: Rebuild benchmarks docker
-      run: benchmarks/build-all-native.sh --run
+    - name: Rebuild and run native benchmarks using Docker
+      run: benchmarks/build-all-native.sh --run

benchmarks/.gitignore

@@ -1,2 +1,2 @@
-*/benchmark.so
+*/*.so
 */target

benchmarks/README.md

@@ -71,53 +71,30 @@ through the `validate` command:
 $ cargo run -- validate path/to/benchmark.wasm
 ```
 
+## Compatibility Requirements for Native Execution
 
-## Compatibility Requirements for the Native Engine
-
-In addition to knowing the performance of the benchmark when targeting Wasm,
-a user may also want to know the performance of that same benchmark when native
-is the original target. To facilitate this, Sightglass includes a native
-engine (similar to the Wasm engine) that can execute a natively compiled program
-such that the same high level source can be used to compile both Wasm and native
-targets without change. Because it is intended that the same high-level source be used,
-requirements listed in the section "Minimal Technical Requirement" above
-are required. In addition though, the benchmark folder is required to supply a Cargo based
-build support that targets a benchmark.so file copied to the base of the benchmark's
-target directory. Specifically:
-
-* A Cargo.toml must be included and the base of the benchmark directory, <benchmark>/Cargo.toml,
-such that "cargo run" produces a benchmark.so dynamically linked library file at the base of the
-target folder, <benchmark>/target/benchmark.so
-
-* The "main" function must be renamed to "native_entry". For C and C++ based source this can be done
-with a simple define directive passed to CC. For example to compile a shootout benchmark the following
-command is used:
-
-
- ```
- let output = Command::new("cc")
-        .args([
-            "-O3",
-            "-Dmain=native_entry",
-            "-fPIC",
-            "-I.",
-            "-shared",
-            "-o",
-            "./target/benchmark.so",
-            "benchmark.c",
-        ])
-        .output()
-        .expect("failed to compile native benchmark");
-    io::stdout().write_all(&output.stdout).unwrap();
-    io::stderr().write_all(&output.stderr).unwrap();
-````
-
-* Also, a symbolic link should be included to Dockerfile.native to allow building in a container. Because this Dockerfile
-will simply copy the directory context and call cargo run as if compiling directly on host, there should be no need to
-supply a custom Dockerfile to make a container based build work.
-
-* Support for native is optional. If a benchmark is added for Wasm, corresponding build support for native is optional and
-will not break CI if it is not included.
+Sightglass can also measure the performance of a subset of benchmarks compiled
+to native code (i.e., not WebAssembly). To compile these benchmarks without
+changing their source code, this involves a delicate interface with the [native
+engine] with some additional requirements beyond the [Minimal Technical
+Requirements] noted above:
+
+[native engine]: ../engines/native
+[Minimal Technical Requirements]: #minimal-technical-requirements
+
+* Generate an ELF shared library linked to the [native engine] shared library to
+  provide definitions for `bench_start` and `bench_end`.
+
+* Rename the `main` function to `native_entry`. For C- and C++-based source this
+  can be done with a simple define directive passed to `cc` (e.g.,
+  `-Dmain=native_entry`).
+
+* Provide reproducible builds via a `Dockerfile.native` file (see
+  [`build-native.sh`](./build-native.sh)).
+
+Note that support for native execution is optional: adding a WebAssembly
+benchmark does not imply the need to support its native equivalent &mdash; CI
+will not fail if it is not included.
 
 ## Additional Requirements
 

benchmarks/all.suite

@@ -94,23 +94,23 @@ meshoptimizer/benchmark.wasm
 noop/benchmark.wasm
 pulldown-cmark/benchmark.wasm
 regex/benchmark.wasm
-shootout-ackermann/benchmark.wasm
-shootout-base64/benchmark.wasm
-shootout-ctype/benchmark.wasm
-shootout-ed25519/benchmark.wasm
-shootout-fib2/benchmark.wasm
-shootout-gimli/benchmark.wasm
-shootout-heapsort/benchmark.wasm
-shootout-keccak/benchmark.wasm
-shootout-matrix/benchmark.wasm
-shootout-memmove/benchmark.wasm
-shootout-minicsv/benchmark.wasm
-shootout-nestedloop/benchmark.wasm
-shootout-random/benchmark.wasm
-shootout-ratelimit/benchmark.wasm
-shootout-seqhash/benchmark.wasm
-shootout-sieve/benchmark.wasm
-shootout-switch/benchmark.wasm
-shootout-xblabla20/benchmark.wasm
-shootout-xchacha20/benchmark.wasm
+shootout/shootout-ackermann.wasm
+shootout/shootout-base64.wasm
+shootout/shootout-ctype.wasm
+shootout/shootout-ed25519.wasm
+shootout/shootout-fib2.wasm
+shootout/shootout-gimli.wasm
+shootout/shootout-heapsort.wasm
+shootout/shootout-keccak.wasm
+shootout/shootout-matrix.wasm
+shootout/shootout-memmove.wasm
+shootout/shootout-minicsv.wasm
+shootout/shootout-nestedloop.wasm
+shootout/shootout-random.wasm
+shootout/shootout-ratelimit.wasm
+shootout/shootout-seqhash.wasm
+shootout/shootout-sieve.wasm
+shootout/shootout-switch.wasm
+shootout/shootout-xblabla20.wasm
+shootout/shootout-xchacha20.wasm
 spidermonkey/benchmark.wasm

benchmarks/build-all-native.sh

@@ -2,9 +2,11 @@
 
 # Either build all of the benchmark's native target or a random subset of them.
 #
-# Usage: ./build-all-native.sh <number of benchmarks>
-# - <number of benchmarks>, an optional number of benchmarks to build; if provided, this script will
-#   randomize the list of benchmarks and pick a subset of them to build
+# Usage: ./build-all-native.sh <number of benchmarks> [--run]
+# - <number of benchmarks>, an optional number of benchmarks to build; if
+#   provided, this script will randomize the list of benchmarks and pick a
+#   subset of them to build
+# - --run, if set, this script will attempt to run the built benchmarks
 
 set -e
 
@@ -26,6 +28,8 @@ for DOCKERFILE in $DOCKERFILES; do
     BENCHMARK_DIR=$(dirname $DOCKERFILE)
     $BUILD_SCRIPT $BENCHMARK_DIR
     if [[ $* == *$FLAG* ]]; then
-        $RUN_SCRIPT $BENCHMARK_DIR/target/benchmark.so
+        for BENCHMARK in $(find $BENCHMARK_DIR -name '*.so'); do
+            $RUN_SCRIPT $BENCHMARK
+        done
     fi
 done

benchmarks/build-native.sh

@@ -1,35 +1,27 @@
 #!/usr/bin/env bash
 
-# Build a single native benchmark
+# Build a single native benchmark.
+#
+# Usage: ./build-native.sh [--host] <path to benchmark directory>
 
 set -e
 
 BENCHMARKS_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 SIGHTGLASS_BASE=$(dirname $BENCHMARKS_DIR)
 ENGINE=$SIGHTGLASS_BASE/engines/native/libengine.so
-
-BENCHMARK_DIR="";
-
-for ARG in "$@"; do
-    if [[ -d $ARG ]]; then
-        BENCHMARK_DIR=$ARG;
-        break
-    fi
-done
+BENCHMARK_DIR=$1;
 
 print_header() {
     >&2 echo
     >&2 echo ===== $@ =====
 }
 
-
-if [[ $BENCHMARK_DIR == "" ]]; then
+if [[ ! -d $BENCHMARK_DIR ]]; then
     echo "Unknown benchmark directory; usage: ./build-native.sh [--host] <path to benchmark directory>"
     exit 1
 fi
 
-
-# Build engine if not availble since it needs to be present for linking
+# Build engine if not availble since it needs to be present for linking.
 if [[ ! -f $ENGINE ]]; then
     cd $SIGHTGLASS_BASE/engines/native/libengine/
     cargo build --release
@@ -40,29 +32,26 @@ fi
 # Build directly on host
 FLAG='--host'
 if [[ $* == *$FLAG* ]]; then
-
-print_header "Build Native Benchmark Using Host"
-(set -x; cd $BENCHMARK_DIR && cargo run --release)
-
-
+    print_header "Build native benchmark directly on host (no Docker)"
+    (set -x; cd $BENCHMARK_DIR && ./build-native.sh)
 else
-
-BENCHMARK_NAME=$(readlink -f $BENCHMARK_DIR | xargs basename)
-IMAGE_NAME=sightglass-benchmark-$BENCHMARK_NAME-native
-
-# To allow the use of symlinks in the benchmark directories (docker ignores them), we `tar` up the
-# directory and `--dereference` (i.e., follow) all symlinks provided.
-print_header "Create build context"
-TMP_TAR=$(mktemp /tmp/sightglass-benchmark-dir-XXXXXX.tar)
-(set -x; cd $BENCHMARK_DIR && ln -f -s ../../engines/native/libengine.so ./libengine.so && tar --create --file $TMP_TAR --dereference --verbose . && rm libengine.so)
-
-# Build the benchmark and extract the build results from the container
-print_header "Build Native Benchmark Using Host"
-(set -x; docker build -f Dockerfile.native --tag $IMAGE_NAME - < $TMP_TAR)
-CONTAINER_ID=$(set -x; docker create $IMAGE_NAME)
-(set -x; mkdir -p $BENCHMARK_DIR/target; docker cp --follow-link $CONTAINER_ID:/benchmark/target/. $BENCHMARK_DIR/target/;)
-
-# Copy host files to container and build inside a container
-
+    BENCHMARK_NAME=$(readlink -f $BENCHMARK_DIR | xargs basename)
+    IMAGE_NAME=sightglass-benchmark-$BENCHMARK_NAME-native
+
+    # To allow the use of symlinks in the benchmark directories (docker ignores
+    # them), we `tar` up the directory and `--dereference` (i.e., follow) all
+    # symlinks provided.
+    print_header "Create build context"
+    TMP_TAR=$(mktemp /tmp/sightglass-benchmark-dir-XXXXXX.tar)
+    (set -x; cd $BENCHMARK_DIR && \
+        ln -f -s $ENGINE ./libengine.so && \
+        tar --create --file $TMP_TAR --dereference --verbose . && \
+        rm libengine.so)
+
+    # Build the benchmark and extract the build results from the container.
+    print_header "Build native benchmark in Docker"
+    (set -x; docker build -f Dockerfile.native --tag $IMAGE_NAME - < $TMP_TAR)
+    CONTAINER_ID=$(set -x; docker create $IMAGE_NAME)
+    (set -x; docker cp --follow-link $CONTAINER_ID:/benchmark/. $BENCHMARK_DIR/;)
 fi
 

benchmarks/run-native.sh

@@ -1,40 +1,22 @@
 #!/usr/bin/env bash
 
-# Run a single native benchmark that is already built
+# Run a single native benchmark that is already built.
 #
-# Usage: ./run-native.sh <path-to-benchmark-folder/target/benchmark.so>
+# Usage: ./run-native.sh <path-to-benchmark-folder/benchmark.so>
 
 set -e
 
 BENCHMARKS_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 SIGHTGLASS_BASE=$(dirname $BENCHMARKS_DIR)
-
-SIGHTGLASS="cargo +nightly run --release --bin sightglass-cli --"
-
+SIGHTGLASS="cargo run --release --bin sightglass-cli --"
 ENGINE=$SIGHTGLASS_BASE/engines/native/libengine.so
-
-BENCHMARK_NATIVE_SO="";
-
-for ARG in "$@"; do
-    if [[ -f $ARG ]]; then
-        BENCHMARK_NATIVE_SO=$ARG;
-        break
-    fi
-done
+BENCHMARK_NATIVE_SO=$1
 
 if [[ -z $BENCHMARK_NATIVE_SO ]]; then
-    echo "Missing benchmark"
-    echo "Usage: run-native.sh <path-to-benchmark-folder/target/benchmark.so>"
-    exit
+    echo "Usage: run-native.sh <path-to-benchmark-folder/benchmark.so>"
+    exit 1
 fi
 
-BENCHMARK_NATIVE_SO="$(realpath $BENCHMARK_NATIVE_SO)"
-
-print_header() {
-    >&2 echo
-    >&2 echo ===== $@ =====
-}
-
 # If an engine is not available, build it.
 if [[ ! -f $ENGINE ]]; then
     cd $SIGHTGLASS_BASE/engines/native/libengine/
@@ -43,10 +25,24 @@ if [[ ! -f $ENGINE ]]; then
     cd - > /dev/null
 fi
 
-# Run a benchmark with the newly created library
+# Because of some hard-coding in the native engine (TODO:
+# https://github.com/bytecodealliance/sightglass/issues/259), we need to set up
+# the temporary directory with the hard-coded paths.
+BENCHMARK_NATIVE_SO="$(realpath $BENCHMARK_NATIVE_SO)"
+MD5SUM=$(md5sum $BENCHMARK_NATIVE_SO | awk '{ print $1 }')
+TMP_BENCHMARK_DIR=/tmp/sightglass-benchmark-native-$MD5SUM
+mkdir -p $TMP_BENCHMARK_DIR
+(set -x; ln -fs $BENCHMARK_NATIVE_SO $TMP_BENCHMARK_DIR/benchmark.so)
+BENCHMARK_DIR=$(dirname $BENCHMARK_NATIVE_SO)
+NAME=$(basename $BENCHMARK_NATIVE_SO .so);
+for FILE in $(find $BENCHMARK_DIR -name "$NAME*.input"); do
+    (set -x; ln -fs $FILE $TMP_BENCHMARK_DIR/)
+done
+
+# Run a benchmark with the native library.
 cd $SIGHTGLASS_BASE
-BENCH_DIR=$(dirname $BENCHMARK_NATIVE_SO)
-echo $BENCH_DIR
-LD_LIBRARY_PATH=./engines/native/ $SIGHTGLASS benchmark --engine engines/native/libengine.so --working-dir $BENCH_DIR -- $BENCHMARK_NATIVE_SO
+ENGINE_DIR=$(dirname $ENGINE)
+(set -x; LD_LIBRARY_PATH=$ENGINE_DIR $SIGHTGLASS benchmark --engine $ENGINE \
+    --processes 1 --iterations-per-process 3 \
+    --working-dir $TMP_BENCHMARK_DIR -- $TMP_BENCHMARK_DIR/benchmark.so)
 cd - > /dev/null
-