Note: If you are just looking to have a minimal TypeScript runtime you can check out Deno's great blog post series on embedding deno_core to create a custom JS/TS runtime. This blog post is for those that want a battery-included runtime with FS and IO capabilities that supports module import via HTTP.

In this blog post, we will take an iterative approach to build a progressively complex and complete TypeScript runtime (that also supports JavaScript) in your Rust project. We will go into just the right amount of detail so that you will be comfortable working with Deno's APIs in the end.

This series of articles will be a condensate of around 6 months of my experience playing with Deno.

The blog post roughly follows these steps:

1. Build an isolate that executes JavaScript

2. Bind Rust functions and states to the isolate

3. Add support for TypeScript

4. Add support for importing via HTTP

5. Build a multi-threaded runtime that manages multiple isolates (Part 2)

Prerequisites

• Intermediate level of Rust

• Basic understanding of async in Rust

• (Optional) Knowledge in multithreading, atomic, mutex, channels

• (Optional) Knowledge of how JavaScript works

Project Setup

I will be using VSCode + rust-analyzer. If you are coding along, you can use whatever IDE + LSP combo you are comfortable with.

Initialize project:

cargo init embed_deno
code embed_deno

Add dependencies to your Cargo.toml file

[dependencies]
anyhow = "1.0.72"
basic_deno_ts_module_loader = "0.1.1"
deno_core = "0.195.0"
deno_error_mapping = "0.1.0"
deno_runtime = "0.120.0"
serde = "*"
serde_derive = "*"
tokio = { version = "1.29.1", features = ["full"] }
serde_v8 = "0.107.0"

Since the runtime will get complex quickly, I will keep everything related to the runtime in a separate module.

src
├── ts_runtime
│   ├── mod.rs
│   ├── ops.rs
│   └── isolate.rs
└── main.rs
tests
├──

A Basic JavaScript Isolate

Our isolate will have one simple responsibility - take in a piece of JS code and execute it till the end.

// isolate.rs

pub use anyhow::Result;
use std::path::Path;

pub struct Isolate;

impl Isolate {
    pub fn new() -> Self {
        Isolate
    }

    pub async fn run(&self, path_to_main_module: &Path) -> Result<()> {
        todo!()
    }
}

Implementing run

The main API of deno_runtime is a MainWorker.

to create a MainWorker, we use its bootstrap_from_options constructor, which expects a URI to the Javascript file on disk, a PermissionsContainer specifying what permission this isolate has, and a WorkerOptions where we can bind our functions and states from Rust, and pass in a custom module loader.

We will get to all of that later, let's keep it simple for now:

// isolate.rs

pub use anyhow::Result;
use deno_core::ModuleSpecifier;
use deno_runtime::permissions::{Permissions, PermissionsContainer};
use std::path::Path;

pub struct Isolate;

impl Isolate{
    pub fn new() -> Self {
        Isolate
    }

    pub async fn run(&self, path_to_main_module: &Path) -> Result<()> {
        let main_module = ModuleSpecifier::from_file_path(path_to_main_module).map_err(|_| {
            anyhow::anyhow!(
                "Failed to create module specifier from path: {:?}",
                path_to_main_module
            )
        })?;
        let mut main_worker = deno_runtime::worker::MainWorker::bootstrap_from_options(
            main_module.clone(),
            PermissionsContainer::new(Permissions::allow_all()),
            Default::default(),
        );
        main_worker.execute_main_module(&main_module).await?;
        Ok(())
    }
}

We have created a worker with all permissions and the a WorkerOptions.

Calling execute_main_module will, unsurprisingly, execute the main module.

Let's test it out!

// main.rs

use std::path::Path;

use ts_runtime::isolate;

pub mod ts_runtime;

#[tokio::main]
async fn main() {
    let isolate = runtime::Isolate::new();
    let path_to_main_module = Path::new("tests/helloworld.js").canonicalize().unwrap();
    isolate.run(&path_to_main_module).await.unwrap();
}

The helloworld.js file is just

// tests/helloworld.js
console.log('Hello World')

Do cargo run aaaaaaaaaaand..

Oh.

Welp time for a quick coffee break.

If everything goes well, you should see "Hello World!" printed on your console

Nice!

What we have right now is basically a fully fledge JavaScript runtime (think Node.js), and we can do basically anything JavaScript can do:

// tests/fib.js

// get current time in milliseconds
const now = Date.now();

function fib(n) {
    return n < 2 ? n : fib(n - 1) + fib(n - 2);
}
// compute fib(40) synchronously
const res = fib(40);
// log elapsed time in milliseconds
console.log(`Fibonacci of 40 is ${res} in ${Date.now() - now}ms`);

(if you are curious the same function took ~880ms in Rust w/ debug build)

Binding Rust Functions and States

Our isolate as it is is already very useful, but we can give it superpowers by binding functions and states from Rust!

An op is a Rust function callable from the Deno runtime. Any Rust function that takes in serde Serialize-able parameters and returns Deserialize-able value can be an op, you just need to annotate the function with #[op].

// src/ts_runtime/ops.rs

use deno_core::op;

#[op]
fn op_hello_from_rust(s: String) {
    println!("[Rust] Hello, {}!", s);
}

#[op]
async fn op_hello_from_rust_async(s: String) -> String {
    println!("[Rust] Hello, {}!", s);
    format!("Hello, {}!", s)
}

To register these ops with Deno, we need to pass in a custom WorkerOptions to MainWorker.

We first build the ops into extensions:

 let ext = deno_core::Extension::builder("generic_deno_extension_builder")
            .ops(vec![
                op_hello_from_rust::DECL,
                op_hello_from_rust_async::DECL,
            ])
            .build();

Then construct a WorkerOptions with the extension

let worker_options = WorkerOptions {
    extensions: vec![ext],
    ..Default::default()
};

which can be passed into the MainWorker constructor:

// isolate.rs

pub use anyhow::Result;
use deno_core::{ModuleSpecifier, Op};
use deno_runtime::{
    permissions::{Permissions, PermissionsContainer},
    worker::WorkerOptions,
};
use std::path::Path;

use super::ops::{op_hello_from_rust, op_hello_from_rust_async};

pub struct Isolate;

impl Isolate{
    pub fn new() -> Self {
        Isolate
    }

    pub async fn run(&self, path_to_main_module: &Path) -> Result<()> {
        let main_module = ModuleSpecifier::from_file_path(path_to_main_module).map_err(|_| {
            anyhow::anyhow!(
                "Failed to create module specifier from path: {:?}",
                path_to_main_module
            )
        })?;
        let ext = deno_core::Extension::builder("generic_deno_extension_builder")
            .ops(vec![
                op_hello_from_rust::DECL,
                op_hello_from_rust_async::DECL,
            ])
            .build();
        let worker_options = WorkerOptions {
            extensions: vec![ext],
            ..Default::default()
        };
        let mut main_worker = deno_runtime::worker::MainWorker::bootstrap_from_options(
            main_module.clone(),
            PermissionsContainer::new(Permissions::allow_all()),
            worker_options,
        );
        main_worker.execute_main_module(&main_module).await?;
        Ok(())
    }
}

To access these ops in JS:

// tests/ops.js

const core = Deno[Deno.internal].core;
const { ops } = core;

ops.op_hello_from_rust("Deno");

const response = await core.opAsync("op_hello_from_rust_async", "Deno");

console.log("[JS]", response);

Remember to switch the path in main.rs

// main.rs

use std::path::Path;

use ts_runtime::isolate;

pub mod ts_runtime;

#[tokio::main]
async fn main() {
    let isolate = isolate::Isolate::new();
    let path_to_main_module = Path::new("tests/ops.js").canonicalize().unwrap();
    isolate.run(&path_to_main_module).await.unwrap();
}

cargo run should now give you something like

Custom Types

Ops can take in any Serialize-able parameter and can return any Derialize-able value.

We will be implementing an op that does vec2 addition as an example.

#[derive(serde::Deserialize, serde::Serialize)]
struct Vec2 {
    x: f64,
    y: f64,
}

#[op]
fn op_add_vec2(a: Vec2, b: Vec2) -> Vec2 {
    Vec2 {
        x: a.x + b.x,
        y: a.y + b.y,
    }
}

Make sure to register the op!

let ext = deno_core::Extension::builder("generic_deno_extension_builder")
            .ops(vec![
                op_hello_from_rust::DECL,
                op_hello_from_rust_async::DECL,
                op_add_vec2::DECL,
            ])
            .build();

In JS:

const first = { x: 1, y: 2 };
const second = { x: 3, y: 4 };

const result = ops.op_add_vec2(first, second);

console.log("[JS]", result);

States in ops

Since ops are called just like normal Rust functions in the end, you can track your states with global variables using OnceCell or lazy_static. However, you will have to prove to the compiler that your global mutable variable is thread-safe by using a Mutex or RWLock. If you don't need to share your state between threads you can have Deno manage your states for the ops via OpState.

#[op]
fn op_add_state(state: &mut OpState, num: i32) -> i32 {
    // borrow_mut() is called on OpState
    let ret = *state.borrow::<i32>();
    state.borrow_mut::<i32>().add_assign(num);
    ret
}

#[op]
async fn op_add_state_async(state: Rc<RefCell<OpState>>, num: i32) -> i32 {
    // the first borrow() is called on Rc RefCell and NOT on the actual OpState
    let ret = *state.borrow().borrow::<i32>();
    state.borrow_mut().borrow_mut::<i32>().add_assign(num);
    ret
}

We need to first initialize the states:

let ext = deno_core::Extension::builder("generic_deno_extension_builder")
            .ops(vec![
                op_hello_from_rust::DECL,
                op_hello_from_rust_async::DECL,
                op_add_vec2::DECL,
                op_add_state::DECL,
                op_add_state_async::DECL,
            ])
            // initialize the state
            .state(|state| {
                state.put(0);
            })
            .build();

The ops can be called normally in JS:

// tests/ops.js

...
console.log("[JS]", ops.op_add_state(1)); // outputs 0 
console.log("[JS]", await core.opAsync("op_add_state_async", 2)); // outputs 1

You may have noticed our state is "indexed" by the type of state. This means we can only store one object per type. If you need to store a complex state you can create a custom struct that has all of your states and access it accordingly.

It's 2023, Let's make it TypeScript

If you didn't know, Deno does not actually run TypeScript directly. The execution engine it uses, v8, only supports JavaScript. So, any TypeScript code needs to be transpiled into JavaScript first.

This is something we can do via a custom ModuleLoader. A module loader's job is to, well, load modules, so it is invoked whenever a import statement is encountered. We will also have the module loader transpile any TypeScript code.

If this sounds complicated, well that's because it is. The actual module loader in Deno is 800+ lines long, with caching, type-checking, transpiling, and npm package supports.

That doesn't mean we can't make something simpler. In fact, all we have to do is

1. Read the URL from the import statement, if the URI is a file path then just read the file content, if not we will assume it to be a remote URL, so we make an HTTP GET request to fetch the code

2. If the code is in TypeScript, transpile it to JavaScript

So I have done exactly that, with a lot of help from one of Deno's contributors, Bartek Iwańczuk, I wrote a very basic module loader and published it as a package on crate.io.

To use it, simply cargo add basic_deno_ts_module_loader, and in the code where we initialize a WorkerOption, pass in the default initialization:

// src/ts_runtime/isolate.rs
// in run() method
let worker_options = WorkerOptions {
            extensions: vec![ext],
            // add the custom module loader
            module_loader : Rc::new(basic_deno_ts_module_loader::TypescriptModuleLoader::default()),
            ..Default::default()
        };

To make sure both the transpiling and HTTP import work, let's write some TypeScript code that imports from the Deno standard library remotely

// tests/http_import.ts

import { assert } from "https://deno.land/std@0.195.0/assert/mod.ts";

function test_four(num1: number, num2: number): boolean {
    return (num1 + num2) === 4;
}

assert(test_four(2, 2));
console.log("Asserted that 2 + 2 = 4")

Changing the parameter in main and cargo run:

Success!

Error Mapping

Note: I really have no idea what I'm talking about. I'm not even sure if my solution is 100% correct. The only info I could find is this issue

There are instances where we need to use the instanceof operator to check the type of error at runtime.

For example, one of the ways to check if a file exists in Deno:

try {
    await Deno.lstat("./nonexist.file");
} catch (e) {
    if (e instanceof Deno.errors.NotFound) {
        console.log("File not found");
    } else {
        console.log("Unknown error ", e);
    }
}

One would expect "File not found" to be printed when running this code, but we will always get "Unknown error" instead. This is because we didn't tell Deno how to map errors correctly.

To do so, initialize the get_error_class_fn field of WorkerOption:

let worker_options = WorkerOptions {
            extensions: vec![ext],
            module_loader: Rc::new(basic_deno_ts_module_loader::TypescriptModuleLoader::default()),
            get_error_class_fn: Some(&deno_error_mapping::get_error_class_name),
            ..Default::default()
        };

Re-running the script should now yield the correct result.