The Unexpected Solution to Testing WebAssembly...

As Extism continues to grow, and its users bring it to more projects, one question has come up over and over: How do we test Extism plugins?

Historically, testing WebAssembly artifacts has been… tricky at best.

What are my options?

You could run your .wasm in a standalone runtime via shell scripts. But we’ve found that pretty inflexible – especially when it comes to testing against a host function interface.

You could build out a whole new host program to simulate your environment. But that’s a whole lot of effort, and now you’ve got to keep your test host in sync with your production host.

You could wave a white flag and write language-native tests, mocking out calls to the host. But that only gives you so much confidence — you frequently want to test the integration of plugin and host, and language-native tests won’t give you that.

None of these really check the boxes for developers who want the familiarity of unit tests and a test runner to execute them.

So we built that.

I’m excited to introduce a small –but mighty!– feature of a bigger platform we’re working on: XTP. There will be much more to share soon on XTP, but if you’re eager to learn more, sign up for our waitlist!

So… about that testing framework?

Read along below for a walkthrough, but if you’re more of a video person, I’ve recorded a quick demo to show how testing with xtp works:

xtp CLI provides a test harness to execute Extism tests, written as (you guessed it) Extism plugins. We’re starting off with a comprehensive TypeScript library which enables you to write a set of test assertions like this:

  import { Test } from "@dylibso/xtp-test";

export function test() {
  // call your plugin's `example` function, passing it some test input
  const output = Test.call("example", "testing");
  // assert that the output is not empty
  Test.assertNotEqual("example not null", output.offset, 0);
  // check that the output is an expected value
  Test.assertEqual("example output", output.readString(), "Hello, world!");
  // run a benchmark / timing test and verify plugin execution performance
  Test.assert(
    "example runtime",
    Test.timeNanoseconds("example", "abc") <= 10000,
  );

  // this is an Extism plugin, so it expects a return value!
  return 0;
}

Once you compile this to an Extism plugin (using the js-pdk), you can run it using the xtp CLI.

Running the test is as easy as:

  xtp plugin test plugin.wasm --with test.wasm

You’ll see some output that looks like:

Download the xtp CLI by running:

  curl https://static.dylibso.com/cli/install.sh | sudo sh

Linking up test and plugin modules

xtp plugin test host linking and calling

For plugins with more advanced functionality who use Host Functions, you can implement your Host Functions as plugins too, and as long as the function signatures match, you can supply them to the test runner like so:

  xtp plugin test plugin.wasm --with test.wasm --host mock-host.wasm

We think xtp plugin test is a great way to build confidence that your Wasm artifacts work – and there’s more to come! We’d love your feedback; as always, at Dylibso, our goal is to make WebAssembly easy to use so you can unlock its power in your application –today! Check out the docs for @dylibso/xtp-test on npm and start testing your plugins today!

We’ll be hanging out in the #xtp channel on our Discord, so drop by and show off your tests, ask us questions, & help us improve it!