Running a Javascript VM in Golang: Data Transforms via User Scripts

Javascript VM in Golang

Introduction

Integrating user-defined scripts into backend systems can significantly enhance the flexibility and power of your applications. In this article, we will explore how to run a JavaScript Virtual Machine (VM) within a Go (Golang) application. Specifically in this example, we’ll focus on performing HTTP body and headers transformations via user-provided scripts.

We will be using the goja library, a JavaScript VM written in Go, to execute JavaScript code. This setup allows developers to provide custom transformation logic for http payloads and headers.

The Project

Inhooks is an open source project aiming to build a lightweight incoming webhooks gateway solution. As some users have requested a way to transform http messages before forwarding them to targets, I have been looking for a proper solution.

Simplified Inhooks Architecture for Transforms

High-Level Overview

Our system listens for incoming HTTP requests, processes the payloads using a JavaScript VM, and applies user-defined transformations to the HTTP body and headers. This allows dynamic and customizable data processing, which can be very useful in various integration scenarios.

Features

• Execute JavaScript in a Go application: Run Javascript user scripts provided at runtime.
• Transform HTTP payloads and headers: Give users a powerful and easy-to-use method to modify incoming requests data.
• Security and Error handling: The VM is isolated and does not have access to the network or file system. Robust error handling ensures smooth operation and easy debugging.

Code Walkthrough

Let’s dive into the implementation. Below is a snippet of the core transformation function written in Go. This function takes in the original HTTP body and headers, applies the user-defined transformation script, and returns the transformed data.

package main

import (
    "fmt"
    "github.com/dop251/goja"
    "net/http"
    "encoding/json"
)

// transformPayload function to apply user-defined JavaScript transformations
func transformPayload(jsScript string, body []byte, headers http.Header) ([]byte, http.Header, error) {
    // Create a new JavaScript VM
    vm := goja.New()

    // Set the HTTP body in the VM
    bodyStr := string(body)
    err := vm.Set("bodyStr", bodyStr)
    if err != nil {
        return nil, nil, fmt.Errorf("failed to set bodyStr: %w", err)
    }

    // Marshal headers to JSON and set in the VM
    headersStr, err := json.Marshal(headers)
    if err != nil {
        return nil, nil, fmt.Errorf("failed to marshal headers to JSON: %w", err)
    }
    err = vm.Set("headersStr", string(headersStr))
    if err != nil {
        return nil, nil, fmt.Errorf("failed to set headersStr: %w", err)
    }

    // Prepare the full script with user function
    fullScript := fmt.Sprintf(`
        /* User Function */
        %s
        /* End User Function */

        const headers = JSON.parse(headersStr);
        var results = transform(bodyStr, headers);
        [results[0], results[1]];
    `, jsScript)

    // Run the script
    val, err := vm.RunString(fullScript)
    if err != nil {
        return nil, nil, fmt.Errorf("failed to execute JavaScript: %w", err)
    }

    // Get the results from the script
    results := val.Export().([]interface{})
    if len(results) != 2 {
        return nil, nil, fmt.Errorf("expected 2 results in js transform, got %d", len(results))
    }

    // Extract and type assert the transformed payload and headers
    transformedPayloadStr, ok := results[0].(string)
    if !ok {
        return nil, nil, fmt.Errorf("expected payload to be of type string, got %T", results[0])
    }
    transformedHeadersTemp, ok := results[1].(map[string]interface{})
    if !ok {
        return nil, nil, fmt.Errorf("expected headers to be of type map[string]interface{}, got %T", results[1])
    }

    // Rebuild the header object
    transformedHeaders := http.Header{}
    for k, values := range transformedHeadersTemp {
        valuesArr, ok := values.([]interface{})
        if !ok {
            return nil, nil, fmt.Errorf("expected header values to be of type []string, got %T", values)
        }

        stringValuesArr := make([]string, len(valuesArr))
        for i, value := range valuesArr {
            stringValue, ok := value.(string)
            if !ok {
                return nil, nil, fmt.Errorf("expected header value to be of type string, got %T", value)
            }
            stringValuesArr[i] = stringValue
        }
        transformedHeaders[k] = stringValuesArr
    }

    // Return the final results
    return []byte(transformedPayloadStr), transformedHeaders, nil
}

What the code above does is:

1. Setting Up the VM: We initialize a new JavaScript VM using goja.New() and set the HTTP body and headers as variables within the VM.
2. Running the Script: The user-defined script is embedded within a template that parses and transforms the body and headers.
3. Handling Results: The transformed payload and headers are extracted, type-checked, and returned.

The users then provide a user script such as the one below in the Inhooks configuration file. This example handles a JSON body, adds a http header X-INHOOKS-TRANSFORMED, converts the body.msg field to upper case if it is present and deletes the body.my_dummy_key field.

      function transform(bodyStr, headers) {
        const body = JSON.parse(bodyStr);

        // add a header
        headers["X-INHOOKS-TRANSFORMED"] = ["1"];
        // capitalize the message if present
        if (body.msg) {
          body.msg = body.msg.toUpperCase();
        }
        // delete a key from the body
        delete body.my_dummy_key;

        return [JSON.stringify(body), headers];
      }

The following unit tests make sure the code is working properly:

package services

import (
 "context"
 "net/http"
 "testing"
 "time"

 "github.com/didil/inhooks/pkg/lib"
 "github.com/didil/inhooks/pkg/models"
 "github.com/stretchr/testify/assert"
)

func TestMessageTransformer_Transform_Javascript(t *testing.T) {
 config := &lib.TransformConfig{
  JavascriptTimeout: 5000 * time.Millisecond,
 }

 mt := NewMessageTransformer(config)

 m := &models.Message{
  Payload: []byte(`{
    "name": "John Doe",
    "age": 30,
    "locations": ["New York", "London", "Tokyo"],
    "scores": [85, 90, 78, 92]
   }`),
  HttpHeaders: http.Header{
   "Content-Type":  []string{"application/json"},
   "X-Request-Id":  []string{"123"},
   "Authorization": []string{"Bearer token123"},
  },
 }
 transformDefinition := &models.TransformDefinition{
  Type: models.TransformTypeJavascript,
  Script: `
   function transform(bodyStr, headers) {
    const body = JSON.parse(bodyStr);
    body.username = body.name;
    delete body.name;
    delete body.age;
    body.location_count = body.locations.length;
    body.average_score = body.scores.reduce((a, b) => a + b, 0) / body.scores.length;
    headers["X-AUTH-TOKEN"] = [headers.Authorization[0].split(' ')[1]];
    delete headers.Authorization;
    return [JSON.stringify(body), headers];
   }
   `,
 }
 err := mt.Transform(context.Background(), transformDefinition, m)
 assert.NoError(t, err)

 assert.JSONEq(t, `{"username":"John Doe","locations":["New York","London","Tokyo"],"scores":[85,90,78,92],"average_score":86.25,"location_count":3}`, string(m.Payload))
 assert.Equal(t, http.Header{"Content-Type": []string{"application/json"}, "X-AUTH-TOKEN": []string{"token123"}, "X-Request-Id": []string{"123"}}, m.HttpHeaders)
}

func TestMessageTransformer_Transform_Javascript_Error(t *testing.T) {
 config := &lib.TransformConfig{
  JavascriptTimeout: 5000 * time.Millisecond,
 }

 mt := NewMessageTransformer(config)

 m := &models.Message{
  Payload: []byte(`{
    "name": "John Doe",
    "age": 30,
    "locations": ["New York", "London", "Tokyo"],
    "scores": [85, 90, 78, 92]
   }`),
  HttpHeaders: http.Header{
   "Content-Type":  []string{"application/json"},
   "X-Request-Id":  []string{"123"},
   "Authorization": []string{"Bearer token123"},
  },
 }
 transformDefinition := &models.TransformDefinition{
  Type: models.TransformTypeJavascript,
  Script: `
   function transform(bodyStr, headers) {
    const body = JSON.parse(bodyStr);
    throw new Error("random error while in the transform function");
    return [JSON.stringify(body), headers];
   }
   `,
 }
 err := mt.Transform(context.Background(), transformDefinition, m)
 assert.ErrorContains(t, err, "failed to transform message: failed to execute JavaScript: Error: random error while in the transform function")
}

Conclusion

Adding a JavaScript VM to the Go project turned out to be quite easy and allows easily customizable data transformations. The goja library provides an efficient and straightforward way to execute JavaScript code in Go without using V8/CGO. Another option would have been to use lua scripts via gopher-lua for example. That could be an idea for a future project.

Feel free to try out the code, adapt it to your own needs and share your feedback. Happy coding!

Get in touch

As a Freelance Software Engineer with over 13 years experience, I have been collaborating remotely with companies of all sizes and all over the globe, to build and improve their software solutions in a robust and time-efficient way. I’m based in Lisbon, Portugal and currently specialize in Backend Development with Golang/Python/Typescript as well as Cloud Engineering using AWS. You can check my github and linkedin profiles. Please get in touch (adil@ledidil.com) if you’re looking for help on your projects.