The Plug-in System Hiding in Your App

As we are approaching Extism 1.0, I’ve been thinking about how to make bootstrapping a plug-in system easier. Extism makes working with Wasm easier, but we’ve yet to tackle the high level tools needed to make plug-in systems easy.

The challenge is that designing, building, documenting, and securing a plug-in system from scratch still takes a lot of effort. It can be daunting to start from a blank slate. So I started to think about how you can reuse existing work to make a plug-in system. This led me to the realization that there are common patterns in our web apps that map almost 1-to-1 with a plug-in system, and we just need to glue them to Extism to reuse that work.

What even is a plug-in system?

A joke we often like to repeat is: “If you squint, everything is a plug-in”. It’s a very fuzzy concept. But for us the simple definition is it is a decoupled boundary in your application where you allow 3rd parties to provide their own implementation. Extism Plug-ins, as Wasm modules, consist of two primary components: exports and imports.

Exports are functions that the plug-in presents to the outside world (our application, or “host”). Our application decides when and where to call these exports. We typically call the plug-in’s exports when some event happens in our system.

Imports are functions that the plug-in can import from the host application. These are typically the capabilities you wish to give a plug-in author. Remember: Wasm can’t affect the outside world unless you give it an import function to do so.

So how do these 3 properties – third party code, events, and capabilities – map to our application?

Web Integrations

In the web world, we have some common patterns for integrating with 3rd parties. The two most common tools are HTTP APIs and Webhooks. And it turns out, if you squint… this is a plug-in system:

For people intimately familiar with plug-in systems, you may not recognize it as such because you wouldn’t normally design a plug-in system where the plug-ins are potentially separated from their host by an ocean. But the nice thing here is that if you have an application like this, you’ve already done a lot of the work needed to start a good plug-in system.

If you have an HTTP API, you’ve decided what capabilities you want to give a third party to manipulate your application. If you have Webhooks, you’ve decided what events in your system a third party would want to know about and you’ve designed those event’s types. You’ve also likely documented these functions and types well. And most importantly, you’ve already assumed that the input into these systems can come from an untrusted party. So you’ve sanitized, secured, and rate limited these inputs.

So, given that Webhooks are our system’s events, and HTTP APIs are our system’s capabilities, a quick way to bootstrap a plug-in system could be to just move this “Customer Logic” piece right into our app, and wrap that in an Extism plug-in:

How do we map this to Extism?

These interfaces are fundamentally tied to HTTP. But what is the interface of an HTTP request exactly? It’s just a function call that optionally takes some bytes as input and optionally returns some bytes as output. That maps directly to Extism’s interface. For example, a JSON HTTP endpoint like POST /customers/ could map to an Extism function like: createCustomer(req: Json<CustomerParams>) -> Json<Customer>. So for this case, our application can make a createCustomer host function that calls the same code as our endpoint and passes that down to the plug-in as an import. Now the plug-in has the capability to securely create customers in our application.

Example App: Lago

After fleshing out this idea, I wanted to try it on a real application and start building out some tools. Our founding team, having all met at Recurly, has a soft-spot for subscription billing and Rails. And that led me to look at Lago which is an open source subscription billing platform in Rails.

Lago is a really nice Rails application that’s still quite small, simple, and readable. And it has all of the traditional components that we talked about along with a domain that is very familiar to me.

OpenAPI

Lago has put a lot of work into designing their APIs. By documenting their API with OpenAPI, the Lago developers didn’t let that work go to waste. OpenAPI is a machine readable Interface Definition Language (or “IDL”) for web APIs. OpenAPI schemas describe all the operations in a given API as well as the types of the inputs and outputs of those operations. They’re a perfect source of information from which to bootstrap our plug-in environment.

Lago PDK

In Extism we have the concept of a language “Plug-in Development Kit” (or “PDK”.) This is a library which helps you write an Extism Plug-in in the language of your choice. For an example, see the Rust PDK. You could, of course, just give your users the Rust PDK and a description of the interface then let them write bindings themselves, but that’s a pretty bad developer experience which would result in a ton of duplication of effort as well as mistakes. The pattern we’ve seen from Extism users is to provide their own application specific PDK wrapper for their users. See Extism user proto as an example which provides end-users with a proto-pdk library that gives them all the nice Rust interfaces they need to write a proto plug-in in Rust.

Our goal here is to create a Lago Rust PDK with as little work as possible.

Generating a PDK from OpenAPI

The OpenAPI project already comes with a project to generate code for you. However these all generate HTTP clients. All we need to do is modify the Rust generator to generate Extism imports instead. I was able to do this by forking the rust generator and making some modifications to the mustache templates. You can find this repo on GitHub.

So to generate our Lago Rust PDK. All we need to do is run the openapi-generator with our Extism templates:

  # Let's create a folder to hold all this code
mkdir extism-lago
cd extism-lago
git init

# Now let's pull down the openapi generator templates
git submodule add git@github.com:extism/openapi-rs-pdk-template.git templates

# Now all we need to do is run the openapi-generator
# and it will generate our PDK rust crate. lagoapi.yaml comes from lago's codebase
openapi-generator generate -i lagoapi.yaml -g rust \
    -o lago_pdk --template-dir templates/ \
    --additional-properties=packageName=lago_pdk

This will generate a rust crate at ./lago_pdk. Instead of the HTTP client it generates Extism host function imports like this where each function is an operation id:

  // types not shown for brevity

#[host_fn]
extern "ExtismHost" {
    pub fn create_customer(params: Json<CreateCustomerParams>)-> Json<ResponseContent<Customer>>;
    pub fn delete_applied_coupon(params: Json<DeleteAppliedCouponParams>)-> Json<ResponseContent<AppliedCoupon>>;
    pub fn destroy_customer(params: Json<DestroyCustomerParams>)-> Json<ResponseContent<Customer>>;
    pub fn find_all_customer_past_usage(params: Json<FindAllCustomerPastUsageParams>)-> Json<ResponseContent<CustomerPastUsage>>;
    pub fn find_all_customers(params: Json<FindAllCustomersParams>)-> Json<ResponseContent<CustomersPaginated>>;
    pub fn find_customer(params: Json<FindCustomerParams>)-> Json<ResponseContent<Customer>>;
    pub fn find_customer_current_usage(params: Json<FindCustomerCurrentUsageParams>)-> Json<ResponseContent<CustomerUsage>>;
    pub fn get_customer_portal_url(params: Json<GetCustomerPortalUrlParams>)-> Json<ResponseContent<GetCustomerPortalUrl200Response>>;
}

Using from Rails

This generates the interfaces in our plug-in, but what about the host functions that must satisfy this interface? We can use OpenAPI to generate those too. First let’s look at how host functions work in Ruby. If i want to create the create_customer host function using our Ruby SDK, I can create a HostEnvironment where create_customer is a method on that object:

  class LagoPluginEnvironment
  include Extism::HostEnvironment
  register_import :create_customer, [Extism::ValType::I64], [Extism::ValType::I64]

  def create_customer(plugin, inputs, outputs, _user_data)
    # get our customer params as json
    customer_params = plugin.input_as_json(inputs.first)
    # create our customer, this is pseudocode
    customer_object = call_rails_action :create_customer, customer_params
    # write our created customer back as json
    plugin.output_json(outputs.first, customer_object.as_json)
  end
end

So all we need to do is write some Ruby meta-programming that generates these from our OpenAPI file. Thats where extism_openapi_rb comes in. This creates a mixin to the HostEnvironment that allows us to pre-generate these host functions from the OpenAPI file:

  class LagoPluginEnvironment
  include ExtismOpenapi::HostEnvironment
  register_openapi File.join(__dir__, '..', 'openapi-lago.yaml')
end

Now we can use this HostEnvironment like normal, but pass in some stuff it needs to invoke the actions properly:

  env = LagoPluginEnvironment.new(
  base_url: 'http://api.lago.dev/api/v1',
  auth: {
    type: :bearer,
    token: '983a0a35-f35f-4fab-b0e1-922d108a2204'
  }
)
path = 'my-lago-plugin.wasm'
manifest = Extism::Manifest.from_path(path)
plugin = Extism::Plugin.new(manifest, environment: env)

Now this plugin has all it needs, regardless of language, to manipulate our app.

Exports

What about exports? How should we trigger the plug-ins and what should the API be? As I mentioned before, Webhooks are a good abstraction for this. However this can be tricky to automate as there aren’t as many tools and conventions around Webhooks. Also Webhooks have built into them the assumption of low reliability and high latency. As such they are mostly fire-and-forget. You’d never pause some user request in your application waiting for a webhook response. The beauty of an Extism based system is you can run these plug-ins in low latency operations now. You’re no longer constrained to the limitations of Webhooks.

You can consult user defined logic in real-time.

So where do we get these events from exactly if not Webhooks? We could design them ourselves but remember we’re looking for the easy mode bootstrapped solution. Where do we have structured conventions around events in our system? I think there are a few places that are worth exploring, but an easy place to tap into is ActiveRecord Callbacks.

ActiveRecord Callbacks

ActiveRecord Callbacks are part of the Rails ORM and are essentially lifecycle events around records. You can have them trigger these callbacks when you create, save, update, or destroy a record. It can also trigger on events like validation.

To tap into these events, we can use a little meta-programming:

  module PluginCallbacks
  def self.included(base)
    %i[before after].each do |temporal|
      %i[validation create save update destroy].each do |action|
        define_export base.model_name, temporal, action
        base.send cb_name.to_sym, "#{record_name}_#{temporal}_#{action}"
      end
    end
  end
end

When we include this module on an ActiveRecord model, it will hook the ActiveRecord events up to the exports of a plug-in. define_export can check to see if the export exists and then ask the plug-in what to do on this event if it does:

  # this is pseudocode, don't take it literally:
def define_export(record_name, temporal, action)
  # ...
  export_fn_name = "#{record_name}_#{temporal}_#{action}"
  define_method export_fn_name do
    if plugin.has_function?(export_fn_name)
      puts "Plugin has the function #{export_fn_name}"
      result = plugin.call(export_fn_name, record.as_json)
      # ...
    else
      puts "Plugin does not have function #{export_fn_name} so we're doing nothing"
    end
  end
  # ...
end

Conclusion and Demo

Once I had the underlying ruby and OpenAPI repos done, it only took a few hours of experimenting to build the basic plug-in system. I’m leaving out the details about how it’s done in the Lago Rails app because I think the code is too specific to the conventions of their application, but feel free to reach out to me on the Extism Discord if you want some details or help doing something similar. I’m happy to consult!

Here is the demo in action: