Adding a benchmark for wasi-nn #201

brianjjones · 2022-09-15T20:04:21Z

This benchmark will measure the performance of
wasi-nn by measuring the time required to complete a single inference.

This benchmark will measure the performance of wasi-nn by measuring the time required to complete a single inference.

brianjjones · 2022-09-15T20:10:29Z

One issue that needs to be addressed is that this benchmark requires OpenVINO to run. So I would think it would be best for it to be an optional benchmark that doesn't get executed automatically by run-all.sh. Thoughts on the best course of action? I already have setup.sh ready to go, which will install it for you, but obviously it shouldn't be run when calling run-all.sh.

* Renaming to image-classification to better describe what it does. * Updated setup.sh to unpack a local copy of OpenVINO rather than installing. * Updated the README to inform users how to use this benchmark. * Changed the name of the .wasm file from benchmark.wasm to image-classification-benchmark.wasm so that it doesn't get picked up by run_all.sh

brianjjones · 2022-10-18T22:31:22Z

@abrown This PR is ready for review now. It's been updated with the changes we discussed. The one possible issue is that the current build.sh script expects the file created in Docker to be named benchmark.wasm. Perhaps we could change this to look for any file that matches *-benchmark.wasm? This way, when build_all.sh or build.sh is called on image-classification, it names it image-classification-benchmark.wasm instead. This prevents run_all.sh from accidentally picking up the image-classification benchmark.

abrown

I think this mostly looks good minus the comments I made below. I am interested also to hear @fitzgen's opinion on this PR as well.

abrown · 2022-10-25T00:08:44Z

benchmarks/image-classification/Dockerfile

+ADD rust-benchmark rust-benchmark
+WORKDIR /usr/src/rust-benchmark
+RUN cargo build --release --target=wasm32-wasi
+RUN cp target/wasm32-wasi/release/image-classification-benchmark.wasm /benchmark.wasm


Yeah, I agree with your comment above about changing the build.sh to be a more liberal "copier." The problem is that I'm not sure you can use glob patterns like *benchmark.wasm with Docker commands? If you can, then great, we should do that; otherwise we may have to modify build.sh to copy everything from some directory, e.g., /sightglass-output.

I'm actually surprised by this question: how does the "rebuild" CI not fail? It seems like it should because this Dockerfile emits a benchmark.wasm but you have checked in a image-classification-benchmark.wasm — that should result in a failure when CI realizes we are not building the same files.

I think the answer is that build-all.sh picks a random subset of benchmarks to build, so this will eventually fail, just not every time. (Rebuilding the benchmarks takes a while so in an effort to speed up CI we just do a subset). You might want to run build.sh on this benchmark and see what happens.

(On a side note, I think at some point there was a Git check to make sure none of the bytes had changed, e.g., git diff --exit-code. That is no longer there so let me open an issue... #210)

@abrown Not sure about why the CI didn't fail, but I just made some changes so now build.sh allows for different wasm file names besides benchmark.wasm. Now you can either pass the filename.wasm to build.sh, or it will look in the benchmark you are building for the name and use that.

abrown · 2022-10-25T00:11:24Z

benchmarks/image-classification/setup.sh

+FILENAME=l_openvino_toolkit_ubuntu20_2022.2.0.7713.af16ea1d79a_x86_64
+[ ! -d ${WASI_NN_DIR}/openvino ] && wget -nc -q --no-check-certificate https://storage.openvinotoolkit.org/repositories/openvino/packages/2022.2/linux/${FILENAME}.tgz -O ${WASI_NN_DIR}/${FILENAME}.tgz
+[ ! -d ${WASI_NN_DIR}/openvino ] && tar -C ${WASI_NN_DIR} -zxf ${WASI_NN_DIR}/${FILENAME}.tgz  && mv ${WASI_NN_DIR}/${FILENAME} ${WASI_NN_DIR}/openvino || echo "OpenVINO is already there, skipping..."
+source ${WASI_NN_DIR}/openvino/setupvars.sh


Could this script also retrieve mobilenet.bin, mobilenet.xml, and the test.jpg? The additional weight isn't a lot for those files (i.e., <14MB) but it might be nice to avoid downloading it if users don't want to.

Sure, I guess I was just doing it this way because all the other benchmarks included everything needed to just run the provided benchmark.wasm. But since this requires you run setup.sh regardless, I don't see why they can't be downloaded.

Update: I'm keeping the original test.jpg because the old .bgr file doesn't seem to work with the example.

abrown · 2022-10-25T00:13:42Z

engines/wasmtime/build.rs

@@ -66,7 +66,7 @@ fn main() {
    // Build the engine library.
    section("Building the engine");
    exec(
-        &["cargo", "build", "--release", "-p", "wasmtime-bench-api"],
+        &["cargo", "build", "--release", "-p", "wasmtime-bench-api", "--features", "wasi-nn"],


Did we figure out if this was actually necessary? I think it would be nicer (but this is a nit, not a real complained) if it could be removed either a) because it is unnecessary or b) by an upstream change to build in wasi-nn by default to bench-api.

Yes this isn't needed, I'll take it out.

abrown · 2022-10-25T00:14:45Z

benchmarks/image-classification/rust-benchmark/src/main.rs

+// the pixel precision to FP32. The resulting BGR pixel vector is then returned.
+fn image_to_tensor(path: String, height: u32, width: u32) -> Vec<u8> {
+    let pixels = Reader::open(path).unwrap().decode().unwrap();
+    let dyn_img: DynamicImage = pixels.resize_exact(width, height, image::imageops::Triangle);


Is there an issue anywhere (or TODO) to remind us to replace with the new and improved image2tensor crate?

I've added a TODO:

abrown · 2022-10-25T00:15:19Z

benchmarks/image-classification/rust-benchmark/Cargo.toml

+publish = false
+
+[dependencies]
+image = { version = "0.23.14", default-features = false, features = ["gif", "jpeg", "ico", "png", "pnm", "tga", "tiff", "webp", "bmp", "hdr", "dxt", "dds", "farbfeld"]  }


I think we're only using "jpeg" here, right?

I suppose there's no reason currently to pull in the other image types.

Removing un-needed "--features Adding TODO for image2tensor Removing un-needed dependencies Download the model files

Changes to build.sh that allow benchmarks to have a different name than benchmark.wasm. This allows them to opt out of ./run-all.sh if they choose. Users can provide the expected .wasm name if they want. If one isn't provided, it will look for the filename in the benchmark being built.

jlb6740 · 2022-11-14T19:20:37Z

Hi @brianjjones
Without being sure of the steps to follow to run this I did this:

cd benchmarks; ./build.sh image-classification/
cc image-classification ; "Read the readme" ; ./setup.sh
cc ../../ ; cargo run -- benchmark --engine engines/wasmtime/libengine.so --engine-flags=--wasi-modules=experimental-wasi-nn -- benchmarks/image-classification/ # but no wasm file.

Realized I should run ./build.sh after I ran.

cd benchmarks; ./build.sh image-classification/
cc../ ; cargo run -- benchmark --engine engines/wasmtime/libengine.so --engine-flags=--wasi-modules=experimental-wasi-nn -- benchmarks/image-classification/image-classification-benchmark.wasm

But here there is this error:

 Running `target/debug/sightglass-cli benchmark --engine engines/wasmtime/libengine.so --engine-flags=--wasi-modules=experimental-wasi-nn -- benchmarks/image-classification/image-classification-benchmark.wasm`
Host error: BackendError(BackendAccess(library loading error

Caused by:
system failed to load shared libraries (see https://github.com/intel/openvino-rs/blob/main/crates/openvino-finder): the shared library at /opt/intel/openvino_2021/deployment_tools/inference_engine/lib/intel64/libinference_engine_c_api.so could not be opened: libinference_engine.so: cannot open shared object file: No such file or directory))

What are the correct steps?

brianjjones · 2022-11-15T02:20:37Z

@jlb6740 For whatever reason sourcing the setupvars.sh script from setup.sh doesn't work. So you'll need to call the following:

source benchmarks/image-classification/openvino/setupvars.sh

After that, your cargo run should work.

brianjjones · 2022-11-15T02:21:47Z

Also I'll update the README with more clear instructions.

brianjjones · 2022-11-15T22:13:47Z

@jlb6740 I updated the instructions in the README, let me know if that's more clear and works for you.

jlb6740 · 2022-11-18T03:23:47Z

Looks good to me. Thanks for the updates.

Adding a benchmark for wasi-nn

66a1aa5

This benchmark will measure the performance of wasi-nn by measuring the time required to complete a single inference.

abrown mentioned this pull request Oct 5, 2022

Organize benchmarks into suites #202

Closed

brianjjones added 2 commits October 18, 2022 13:34

Merge branch 'bytecodealliance:main' into wasi-nn-bench

01f66a7

jlb6740 requested review from jlb6740 and abrown October 18, 2022 22:52

abrown reviewed Oct 25, 2022

View reviewed changes

brianjjones added 2 commits October 25, 2022 17:11

Various minor changes

44fc373

Removing un-needed "--features Adding TODO for image2tensor Removing un-needed dependencies Download the model files

Updating README and setup.sh

2f88f7d

brianjjones force-pushed the wasi-nn-bench branch from 18ba286 to 2f88f7d Compare November 15, 2022 22:27

jlb6740 approved these changes Nov 18, 2022

View reviewed changes

jlb6740 merged commit 49f6739 into bytecodealliance:main Nov 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding a benchmark for wasi-nn #201

Adding a benchmark for wasi-nn #201

brianjjones commented Sep 15, 2022

brianjjones commented Sep 15, 2022 •

edited

Loading

brianjjones commented Oct 18, 2022

abrown left a comment

abrown Oct 25, 2022

abrown Oct 25, 2022 •

edited

Loading

brianjjones Oct 26, 2022

abrown Oct 25, 2022

brianjjones Oct 25, 2022

brianjjones Oct 26, 2022

abrown Oct 25, 2022

brianjjones Oct 25, 2022

abrown Oct 25, 2022

brianjjones Oct 25, 2022

abrown Oct 25, 2022

brianjjones Oct 25, 2022

jlb6740 commented Nov 14, 2022

brianjjones commented Nov 15, 2022

brianjjones commented Nov 15, 2022

brianjjones commented Nov 15, 2022

jlb6740 commented Nov 18, 2022

Adding a benchmark for wasi-nn #201

Adding a benchmark for wasi-nn #201

Conversation

brianjjones commented Sep 15, 2022

brianjjones commented Sep 15, 2022 • edited Loading

brianjjones commented Oct 18, 2022

abrown left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abrown Oct 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlb6740 commented Nov 14, 2022

brianjjones commented Nov 15, 2022

brianjjones commented Nov 15, 2022

brianjjones commented Nov 15, 2022

jlb6740 commented Nov 18, 2022

brianjjones commented Sep 15, 2022 •

edited

Loading

abrown Oct 25, 2022 •

edited

Loading