Check out the Lodestone project here:

Note: Lodestone plugins are called "macros." However, since the word “plugin” is commonly used in the context of Minecraft server. "macros" will be called "plugins" for the rest of this post.

I still remember a discussion at the inception of the Lodestone project, when the team consisted of just 3 friends who played Minecraft more than they should have. It went something like this:

a 3-hour debate ensues on what to name this project

Little did I know, this far future would come in just a year.

Motivation for a Plugin System

Well, it basically boils down to what 2022 Peter said - the ability for end users to extend Lodestone with functionalities they wish to use. As an open-source project, we expect Lodestone to receive a lot of feature requests that are too niche to implement. The plugin system would allow users to adapt Lodestone to their needs.

Some examples of plugins include scheduled restarts for instances, scheduled server backups, and chat monitoring.

Plugins are also a great way to keep the project active when we are too busy to maintain it. By curating a community around plugins, Lodestone can thrive for years to come.

Let's build it from scratch! - First iteration

After finishing an assembler, lexer, and a top-down parser for a C-like language in my compiler course, I felt confident in building a brand new language for Lodestone plugins.

"How hard could it be? We just need to come up with a formal programmer, a parser to parse that grammar, a compiler to compile it to some form of assembly, and an interpreter to run the assembly code!"

Totally doable! Ok, well… maybe not, but that didn’t stop me. So taking heavy inspiration from a dialect of MIPS that was taught in the course, I hacked together this in an afternoon when I really should have been doing my actual compiler assignment.

To spare you the reading, I made an assembly-styled, interpreted, procedural, dynamically typed language that can receive events asynchronously and interact with an instance's console.

This sounds quite fancy until you realize:

• There is no control flow except for goto (no conditionals, loops, while, for ..)

• No variable scoping, so every variable is global

• A whopping 2 data types: string and int

• No composite data type support

• No functions, so to implement a subroutine you use jalr:

> let ret = $31
> jalr procedure
say $3
> goto ret
> label procedure
> add $3 1 2
> goto $31

• No dynamic memory (something I was going to add but gave up)

• No package manager

• No way to import/export code…

The list goes on.

Oh, and it executes slower than Python. (Probably because I was spamming regex everywhere)

It became obvious that no developer in the world would ever use my crappy flavor of MIPS to develop a plugin for Lodestone. Even if I could make the language something more C-like, no one would learn a new and domain-specific programming language with no community, no support, and no libraries just to write plugins. Needless to say, this idea was very quickly scrapped.

What did I learn?

This whole ordeal gave me a more concrete understanding of how the language of the plugin system should look like

• Developer ergonomic. There should be a reasonably sized community around the language already, so developers are able to easily pick it up and start writing

• Some concurrency/async support. The language must be able to receive and handle events asynchronously

• Interoping with Rust. The runtime of this language needs to be embedded into our project and needs to support binding APIs from Rust in order to interact with Lodestone's internal state

• Safety. Yes having an arbitrary code execution engine built-in is already hella unsafe, but that doesn't mean we should allow a badly programmed plugin to cause undefined behavior in the rest of the program

Lua? Lua! - Second iteration

After stumbling across a Reddit post asking for the difference between the rlua and mlua crate on the second floor of my school's math building at 1 in the morning, I had a realization that other people have tried to do something exactly like this before, and Lua was the de facto choice when it came to embedding a language.

So, I spent the next hour swapping out the assembly interpreter with the rlua crate when I really should have been studying for my exams.

At a glance, everything looks great. Lua has a very simple and minimal runtime, and when combined with rlua's great documentation and examples, I was able to export a bunch of APIs from Lodestone into the Lua runtime. Now I could write something like:

while (true)
do
    player, message = await_player_message()
    if contains_bad_words(message) then
        send_stdin("say Warning: " .. player .. " said a bad word!")
end

In this case, the await_player_message and send_stdin functions are both defined in Rust and do what their names suggest.

I also wrote an auto-backup script:

os.execute("mkdir " .. PATH_TO_INSTANCE .. "/backups")

BACKUP_INTERVAL = 10080
WORLD_NAME = "world"

while true do
    local interval = BACKUP_INTERVAL
    local flag = true
    while interval > 0 do
        if interval < 60 and flag then
            send_stdin("say Backup in less than " .. interval .. " seconds")
            send_stdin("save-all")
            flag = false
        end
        delay_sec(1)
        interval = interval - 1
    end
    os.execute("cp -r " .. PATH_TO_INSTANCE .. "/" .. WORLD_NAME .. " " .. PATH_TO_INSTANCE .. "/backups/" .. os.date(WORLD_NAME.."-%Y-%b-%d-%H:%M"))
    print("tick")

end

It was at this point that I started to see some of the problems with Lua

Firstly, note the use of os.execute(".."). Lua provides such a minimum runtime that it doesn't even include a file system API, forcing me to resort to shell commands, which are not ergonomic and not cross-platform.

Second, the lack of any types. Lua is a dynamically typed language, so working with complex data structures from Lodestone is a pain.

Third, the lack of enum or ADTs. Lodestone makes heavy use of Rust enums, which makes any integration with Lua challenging.

Fourth, the lack of async. Lua uses coroutines and threads for asynchronous execution, both of which do not play well with Rust's future, which we utilize heavily in Lodestone.

Lua is also a rather strange language in my opinion. The BASIC style syntax would be quite unfamiliar for a developer that is used to braces like me, not to mention the use of tables instead of object orientation, the pcall error handling, and of course the infamous 1-indexing.

Most crucially, Lua is missing a good package manager, which makes pulling and sharing code from others very difficult. Ideally, the plugin developer will have access to a modern package manager such as Rust's Cargo, Node's npm, or Deno's HTTP resolve.

Deno...Deno....

Wait, Deno is written in Rust! - Final Form

Deno, for those that don't know, is a JavaScript runtime similar to Node, with first-class TypeScript support. Deno is written in Rust, with most of its async stuff powered by tokio, which also happens to be Lodestone's choice of async runtime.

Most importantly, Deno provides the entirety of its runtime, including FS, IO, and networking functions as a library crate that we can just pull into the project as a dependency!

There is a lot to like about the TypeScript + Deno combo:

• Statically typed and enum support

• Great community support

• Remote package support (can potentially tab into the npm package ecosystem)

• Very easy to upload and share packages

• Similar async semantic

• Powered by tokio

• A fully-fledged runtime with batteries included

I can't say the same about the integration process for Deno though. The entire process is arduous and took me well over 4 months, and I couldn't have done it without the generous help provided by one of Deno's maintainers, Bartek Iwańczuk, a process I plan to write a blogpost about in the future.

So, what does Lodestone's plugin system look like right now?

import { EventStream } from "https://raw.githubusercontent.com/Lodestone-Team/lodestone-macro-lib/main/events.ts";
import { Instance } from "https://raw.githubusercontent.com/Lodestone-Team/lodestone-macro-lib/main/instance.ts";

const current = await Instance.current()

const eventStream = new EventStream(current.getUUID(), await current.name());

EventStream.emitDetach();

while (true) {
    const {player, message} = await eventStream.nextPlayerMessage();
    if (containsBadWords(message)) {
        current.sendCommand(`say Warning: ${player} said a bad word!`)
    }
}

There is quite a bit of boilerplate which I'm working to reduce, but it should be clear what the code does.

The backup plugin is as beautiful as its lack of os.execute :

import { format } from "https://deno.land/std@0.177.1/datetime/format.ts";
import { copy } from "https://deno.land/std@0.191.0/fs/copy.ts";
import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
import { EventStream } from "https://raw.githubusercontent.com/Lodestone-Team/lodestone-macro-lib/main/events.ts";
import { MinecraftJavaInstance } from "https://raw.githubusercontent.com/Lodestone-Team/lodestone-macro-lib/main/instance.ts";

const currentInstance = await MinecraftJavaInstance.current();

const eventStream = new EventStream(currentInstance.getUUID(), await currentInstance.name());

const backupFolderRelative = "backups";

const delaySec = 60 * 60;

const instancePath = await currentInstance.path();
const backupFolder = `${instancePath}/${backupFolderRelative}`;
EventStream.emitDetach();
while (true) {
    eventStream.emitConsoleOut("[Backup Macro] Backing up world...");
    if (await currentInstance.state() == "Stopped") {
        eventStream.emitConsoleOut("[Backup Macro] Instance stopped, exiting...");
        break;
    }

    const now = new Date();
    const now_str = format(now, "yy-MM-dd_HH");
    try {
        await copy(`${instancePath}/world`, `${backupFolder}/backup_${now_str}`);
    } catch (e) {
        console.log(e)
    }

    await sleep(delaySec);
}

If you would like a detailed explanation of how everything works, such as the EventStream.emitDetach() call, check out our wiki page here.

The Future and Beyond

I'm very satisfied with being able to achieve something I thought was so far-fetched in only a year. I can confidently say the plugin system is my best piece of work yet. Through this journey, I was able to learn so much about Rust, asynchronous programming, how a Javascript runtime like Deno works, and the v8 event loop, among countless other topics. It gave me a newfound appreciation for stuff I used to take for granted.

The plugin system also directly enabled me to create Lodestone Atom, a system that allows one to implement their custom instance logic in Typescript.

Of course, this doesn't mean the plugin system is perfect - it's far from it. I'd be ecstatic if you could try it and provide us with any feedback and suggestions. Any contribution is also welcome!