A staggering number of languages and platforms support WebAssembly (Wasm). Runtimes are available in every major browser, via the most popular edge and serverless providers, across a dizzying number of IoT devices, and on globe-spanning blockchain compute platforms. When you compile your program to Wasm, you can run it nearly anywhere. But that power isn’t exactly easy to harness today.

While programmers are accustomed to modeling their work in terms of their business logic – classes like User, Item, or Cart – Wasm works in terms of numbers and linear memory, making it difficult to communicate with guest Wasm programs about even basic types, like strings¹. And while Wasm is broadly supported, it’s not natively available for every language platform, and specifications are adopted by language compilers and runtimes at different rates. This makes it difficult to compile to Wasm and run it “out of the box”.

This is where Extism comes in. Extism streamlines the developer experience across languages and platforms for everyone – whether you’re compiling to Wasm or embedding Wasm programs. And you can use it today.

How does Extism help?

So how does Extism help make WebAssembly easier to use? With two assertions:

It should be a no-brainer to host WebAssembly guest programs from applications written in your language of choice.
It should be dead simple to communicate between WebAssembly and the host program.

Extism makes it easy to use Wasm in your application by providing host SDKs in an ever growing list of languages. We use runtimes native to the host language – like Golang’s Wazero or JS’s web platform – where available. Where that’s not available, we distribute libextism shared objects, and allow for easy installation with the Extism CLI. libextism wraps Wasmtime, an industrial-grade Wasm runtime. With this approach, Extism brings the ability to host Wasm programs to environments that may be years away from being able to do so natively.

Making it dead simple to communicate with Wasm is less straightforward. It involves abstracting away the hard parts of Wasm by providing constructs to enable HTTP requests, error handling, memory management, and the ability to pass types that are more complex than numbers between the host and the WebAssembly. A lot of great work is being done in the Wasm community to define specifications like WASI and the Component Model that make Wasm development easier. We at Dylibso are participating in these efforts and working to make sure Extism supports the Component Model as it develops. Extism already has support for WASI (preview 1) and will continue to seamlessly integrate new standards and bring them to as many languages as possible as they solidify.

To better illustrate the hard parts of using Wasm, let’s create a function that takes a string as input, modifies it, and returns a new string, then compile that to Wasm and run it. Seems simple enough, right? Maybe get a glass of water just in case.

Passing strings without Extism

When using Wasm, it is often helpful to bear in mind the guest/host relationship. The Wasm binary being executed is the “guest”, and the “host” is the program that executes that binary.

Let’s start by writing a guest program that defines a function called say_hello which accepts a string as input and returns a string as output. At least, that’s how we’d like to think about the guest program. In actuality, since this code is being compiled to Wasm, our function can only accept and return numeric types, so we have to pass our input string and recieve our output string another way. In this case, we’ll store the input and output strings in linear memory and pass their size and addresses (pointers) between the host and guest through the guest’s exported say_hello function².

Ok, we’ve got a plan, let’s give it a go (in Rust)!

  // src/lib.rs:

use std::mem;
use std::ptr;
use std::slice;

#[no_mangle]
pub extern "C" fn say_hello(mem_addr: *const u8, len: usize, ret_len_addr: *mut u32) -> i32 {
    // read input string
    let data: Vec<u8> = unsafe { slice::from_raw_parts(mem_addr, len).to_vec() };

    // create output string
    let greeting = format!(
        "Hello there, {}.  That was a lot of memory management to pass a string!",
        String::from_utf8_lossy(&data)
    );

    // determine the length of the output string and set that value in the address of ret_len_addr
    unsafe {
        ptr::write(ret_len_addr, greeting.len() as u32);
    };

    greeting.as_ptr() as i32
}

#[no_mangle]
pub extern "C" fn alloc() -> *const u8 {
    let mut buf = Vec::with_capacity(1024);
    let ptr = buf.as_mut_ptr();

    // tell Rust not to clean this up
    mem::forget(buf);

    ptr
}

#[no_mangle]
pub unsafe extern "C" fn dealloc(ptr: &mut u8) {
    let _ = Vec::from_raw_parts(ptr, 0, 1024);
}

We start off by including the std::slice, std::mem and std::ptr modules from the Rust Standard Library. These modules allow us to perform operations on raw pointers. The #[no_mangle] attribute instructs the compiler to preserve the function name as authored during compilation. This is necessary so that the runtime knows how to reference the function when it is being executed by a host. On the following line, pub extern "C" fn say_hello indicates that the say_hello function is intended to be called by some external code across an ABI. In many cases, this is used to compile your code into a .dylib or .so to be linked into some other program. However, because we are compiling to Wasm, we are indicating to the compiler that the function is to be exported from our Wasm module so that a host (or some other module) may call it.

Now we’ve got to come up with some numeric parameters that will allow us to express a string. We’ll use Wasm’s linear memory in combination with function arguments and returns to facilitate passing complex data into and out of the say_hello function. Before we start passing pointers around, let’s have a quick sidebar about memory management.

In order to use Wasm’s linear memory in a somewhat responsible manner, we need to be careful about how we allocate and deallocate memory. This is where the alloc and dealloc functions we defined as guest exports come into play. These functions serve as a naive implementation of a memory allocator that can only allocate and deallocate 1024 bytes at a time. This allows the host and guest to allocate linear memory without corrupting one another, however this naive approach is bound to lead to an excess of memory allocation. Spoiler alert: Extism takes additional care into ensuring efficient memory allocation and abstracts this away for the developer. Ok let’s get back to breaking down the say_hello function.

The first parameter, mem_addr: *const u8, represents a raw pointer to the location in linear memory where the host placed the input string. The next parameter, len: usize represents the length of the input string in bytes.

This is enough for us to go and retrieve our input string, so let’s zoom in on that before we describe how we return a string - I haven’t forgotten about the third parameter!

  
// ...
let data: Vec<u8> = unsafe { slice::from_raw_parts(mem_addr, len).to_vec() };
let greeting = format!(
    "Hello there, {}.",
    String::from_utf8_lossy(&data)
);
// ...

First we use slice::from_raw_parts to initialize a Vec<u8> from the location and length in memory that the host passed in. This is an unsafe operation in Rust because we are dealing with raw pointers, so the compiler cannot guarantee the memory is allocated with what we expect it to be (or even at all!). This is ok though since we have full control over linear memory as authors of both the guest and the host. On the following line, we decode our Vec<u8> from raw bytes to a UTF-8 string, and save that to a new variable called greeting. Our guest code has now created a new String called greeting which simply appends the string we got from our input to some hard-coded string. Let’s look at how we return this string to the host. This is where that third parameter to say_hello comes into play:

  
// set the address pointed to by `ret_len_addr` to the length of the new string
unsafe {
    ptr::write(ret_len_addr, greeting.len() as u32);
};

greeting.as_ptr() as i32

In order for the host to receive the output string, it needs to know the location and size of the string that the guest created. We again need to perform an unsafe operation by writing a u32 integer (which holds the length of the new string, greeting) into the memory address passed to us by the host as a function parameter (ret_len_addr). With the length of the output string written to a place in linear memory that the host knows about, we can return a pointer to the output string’s memory address, and now the host has all it needs (the memory address and length) to find and decode the return string. You can compile this code to WebAssembly by running: cargo build --target=wasm32-unknown-unknown (the source code for this example is also available here).

Before we get into how to do this in Extism, let’s take a quick look at the host code just to show you how to run the guest WebAssembly module we created above. I left comments in-line rather than going through it in depth.

  // /src/main.rs

// we're using Wasmtime as our WebAssembly runtime, but there are a number of runtimes available
use wasmtime::*;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // checkout the wasmtime crate docs to learn more about embedding wasmtime into your rust applications - https://docs.wasmtime.dev/examples-rust-embed.html
    let engine = Engine::default();
    let mut store = Store::new(&engine, ());
    let module = Module::new(
        &engine,
        // the location of the Wasm binary we compiled from our guest code
        include_bytes!(
            "../../no-extism-guest/target/wasm32-unknown-unknown/debug/no_extism_guest.wasm"
        ),
    )?;
    let instance = Instance::new(&mut store, &module, &[])?;
    let say_hello = instance.get_typed_func::<(i32, i32, i32), i32>(&mut store, "say_hello")?;
    let alloc = instance.get_typed_func::<(), i32>(&mut store, "alloc")?;
    let dealloc = instance.get_typed_func::<i32, ()>(&mut store, "dealloc")?;

    // get an instance of linear memory
    let memory = instance
        .get_memory(&mut store, "memory")
        .ok_or(anyhow::format_err!("failed to find `memory` export"))?;

    // allocate memory to hold the input string and the output string return length
    let input_addr = alloc.call(&mut store, ())?;
    let output_len_addr = alloc.call(&mut store, ())?;

    // define the input string and write the string as binary to the memory we allocated for the input string
    let input_string = b"non-extism host!";
    memory.write(&mut store, input_addr as usize, input_string)?;

    // invoke the `say_hello` function and save the memory address to where the output will be saved
    let output_addr = say_hello.call(
        &mut store,
        (
            input_addr,                // input memory address
            input_string.len() as i32, // input length
            output_len_addr as i32,    // output length memory address
        ),
    )?;

    // we are storing the length of the output string as an unsigned 32 bit integer in linear memory, so let's
    // allocate 4 bytes, read the data from linear memory, and store it in the len_buffer
    let mut len_buffer = [0u8; 4];
    memory.read(&store, output_len_addr as usize, &mut len_buffer)?;

    // convert the bytes into an i32 (WebAssembly is always little endian)
    let len = i32::from_le_bytes(len_buffer);

    // let's now create a new Vec<u8> to store the bytes of the output string and fit it to size
    let mut v = Vec::<u8>::new();
    v.resize(len as usize, 0);

    // read the memory address returned to us by the `say_hello` function, which points to the location
    // in linear memory where our WebAssembly module stored the output string
    memory.read(&store, output_addr.try_into().unwrap(), &mut v)?;

    // decode the byte vector into a UTF8 string and print to the console!
    println!("{}", String::from_utf8_lossy(&v));

    // deallocate all of the memory that we allocated
    // this is technically unnecessary because the program is about to end, but that may not always be the case
    dealloc.call(&mut store, input_addr)?;
    dealloc.call(&mut store, output_len_addr)?;

    Ok(())
}

We did it! We passed a string into a Wasm function and got a string back in return! After a well deserved pat on the back, let’s now take a look at how to do this with Extism.

Passing strings with Extism

Luckily, there’s a lot less to go through with Extism. Here is the equivalent code:

  // /src/lib.rs

use extism_pdk::*;

#[plugin_fn]
pub fn say_hello(input: String) -> FnResult<String> {
    let greeting = format!("Hello, {}", input);
    Ok(greeting)
}

and the host code:

  // /src/main.rs

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let ctx = extism::Context::new();
    let mut plugin = extism::Plugin::new(
        &ctx,
        include_bytes!("../../yes-extism/target/wasm32-unknown-unknown/debug/yes_extism.wasm"),
        [],
        false,
    )?;
    let data = plugin.call("say_hello", "extism host!")?;
    println!("{}", String::from_utf8_lossy(data));
    Ok(())
}

As tempting as it is to drop the mic, let’s be civil and talk a bit about what Extism is doing.

One of the things you may have already noticed is that you no longer need to manage Wasm memory directly in either the guest or the host. Extism abstracts away the “hard part” of managing your own memory by providing constructs for clean and efficient memory management that you can rely on throughout the runtime of your program. Oh, and Extism doesn’t only work with strings, you can use JSON, Protobuf, raw binary.. whatever!

The Extism Plugin Development Kit (“PDK”) provides constructs for dealing with input, output, errors, and memory (as seen in this small program), but it also provides mechanisms for http, configuration, plugin scoped variables… the list will continue to grow with the needs of the community. As for the host code, let’s take another look and talk about some of the key benefits we get from using Extism:

  // /src/main.rs

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let ctx = extism::Context::new();
    let mut plugin = extism::Plugin::new(
        &ctx,
        include_bytes!("../../yes-extism/target/wasm32-unknown-unknown/debug/yes_extism.wasm"),
        [],
        false,
    )?;
    let data = plugin.call("say_hello", "extism host!")?;
    println!("{}", String::from_utf8_lossy(data));
    Ok(())
}

Not only is this code easier to reason about, but we preserved the flexibility we need while abstracting away the initialization of the Wasm runtime. As a result, we could swap it out for a runtime that aligns more with the specific needs of your program with minimal changes. Extism accounts for the nuances of Wasm runtimes and compilers to achieve a smooth and interoperable developer experience across a growing list of languages and platforms.

The future

As the primary maintainers of Extism, we keep a close eye on the core WebAssembly specification and related specifications like WASI and the Component Model. We track Wasm feature adoption in runtimes throughout the ecosystem to make sure that Extism developers have access to the latest capabilities Wasm has to offer. Our goal with Extism is to make it painless to unlock all of Wasm’s capability, even as those capabilities grow.

We’ve had a blast bringing the speed, security, portability and composability of Wasm to developers in so many languages and this is just the beginning! Extism is already used in production by many projects and, later this year, we’ll release Extism 1.0. Extism 1.0 comes with a promise that our tooling and libraries are officially stable. Beyond 1.0, we are excited to continue to advance Extism in alignment with the core WebAssembly specification and community, and will continue to ensure a smooth and consistent developer experience for those who are producing or executing WebAssembly.

Say “Hi!”

Have a question about anything in this article, Extism, or WebAssembly in general? Head on over to our friendly discord and drop us a note!

Shout-outs

A big thanks to my colleagues Chris Dickinson and Ben Eckel for their sage guidance in writing this article!

There’s work to improve this story happening via the Wasm Component Model. Dylibso is involved in these efforts — we’ll talk a little more about how Extism relates later in this post, and more about the Component Model in a later blog post! ↩
Radu Matei has a great write up on linear memory for those who want to dig deeper into Wasm memory. ↩

Why Extism?

Why Extism?

How does Extism help?

Passing strings without Extism

Passing strings with Extism

The future

Say “Hi!”

Shout-outs

Why Extism?

Why Extism?

How does Extism help?

Passing strings without Extism

Passing strings with Extism

The future

Say “Hi!”

Shout-outs

Footnotes