Max Bernstein's Blog

Sorry for marking all the posts as unread

Wed, 31 Jan 2024 00:00:00 +0000

I noticed that the URLs were all a little off (had two slashes instead of one) and went in and fixed it. I did not think everyone's RSS software was going to freak out the way it did. PS: this is a special RSS-only post that is not visible on the site. Enjoy.

Heating water from afar

Tue, 24 Jun 2025 00:00:00 +0000

Please do not take anything you read in any of my posts (but especially not this post) as engineering advice.

My parents’ house has an on-demand water heater for energy efficiency reasons. This has a small drawback: you have to press a button to prime the water heater and then wait 2-5 minutes before showering.

This turns into a somewhat bigger drawback for the one room for which it’s just Not Possible to wire up a button. The water heater company hypothetically sells a plug-and-play wireless solution for this sort of thing, but that is seemingly incompatible with my parents’ walls (???).

Thankfully, I have a have a playbook for this.

There’s just one problem: how do I make a Raspberry Pi talk to the water heater? I investigated a couple of different approaches:

analyze and replicate the water heater company’s proprietary Bluetooth protocol
install a physical button presser, inspired by the Recurse Center’s elevator-related shenanigans

but after a couple of discussions with my dad and a support tech from the company, we determined that we should instead emulate a button press. To find out what that means, let’s take a look at a very sketchy wiring diagram:

The heater supplies 12V at 0.2A on the red wire. When the button is pressed, it connects the red and black wires, which the heater interprets as a signal to start heating. The wire nuts are there because several buttons are already joined together this way.

This means we would have to add a new “button” and have it briefly connect two wires. Because the last time I actually touched some wires was over 10 years ago in robotics and I don’t want to start any fires, I reach out to the usual suspects: Tom and Logan. They inform me that the thing I am looking for is called a relay and that companies sell pre-built relay hats for the Pi. Super.

I ended up buying:

A Pi Zero 2W and accoutrements
A relay hat with headers and stabilization posts
Telephone wire from the hardware store, which already has bundled red/black wire

At some point I sit down to build the thing and realize that I don’t actually know how relays work. The relay I bought had three ports: NC, NO, COM. After some searching, I figure out that I want one wire in NO (“normally open”) and one in COM (“common”). This means that the relay, when activated, will close the circuit.

I downloaded the sample code from the company that sells the relay hats and realized that it is an extremely thin (~10 LOC) wrapper over the existing Python GPIO library provided and pre-installed by Raspberry Pi, so I just manually inlined it:

#!/usr/bin/env python3
import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BOARD)

class relay:
    relay_pins = {"R1":31,"R2":33,"R3":35,"R4":37}

    def __init__(self, pins):
        self.pin = self.relay_pins[pins]
        self.pins = pins
        GPIO.setup(self.pin,GPIO.OUT)
        GPIO.output(self.pin, GPIO.LOW)

    def on(self):
        GPIO.output(self.pin,GPIO.HIGH)

    def off(self):
        GPIO.output(self.pin,GPIO.LOW)

print("Starting water heater...")
r2 = relay("R2")
# time is in seconds, so turn on once for 4ms and then turn off again
TIME_IN_MS=4
MS_PER_SEC=1000
r2.on()
time.sleep(TIME_IN_MS/MS_PER_SEC)
r2.off()
print("...signal sent. Please wait 5 minutes before showering.")

If you read my previous post (linked above), you will know that is is, of course, a CGI script that is triggered on a website button press:

The tagline under #justshowerthings is randomly selected on page load from a medium-long list I wrote. All the items are carefully designed to toe the line.

All of the rest of the software is the same as in the previous post. Very boring stuff: httpd, systemd. Hopefully nothing goes wrong. But if it does and I need to administer this device from afar, I also set up Tailscale (no, this is not an ad; just happy).

The total bill for this came to ~$40 or so, which isn’t half bad. It could probably be done for 35 cents using an old microcontroller and a paperclip or something but I wanted an exceptionally boring (to me) approach. That’s all for now. Thanks for reading!

Travel notes: PLDI Seoul

Mon, 23 Jun 2025 00:00:00 +0000

I went to PLDI in Seoul, South Korea. I had a nice time. Here are some unstructured thoughts.

It was great to see old familiar faces, meet new people, and unexpected (though not unwelcome) to have multiple people approach and say they read my blog. Hello, dear readers!

Vegetarian food is tricky, but if you look for a “temple meal” or restaurants specifically called out as vegetarian friendly (or even full vegan), you will be okay. A lot of food that otherwise looks vegetarian probably contains fish products. If you are okay with a bit of a grey area, the Myeong-dong night market has a good fried potato cheese stick thing and a cream cheese garlic bread.

Go to a chicken and beer place with friends. We spent a good deal of time looking for a cute bar vibe (and did find one secret bar!) but the best bang for buck is definitely fries and 4000 Won Cass beer.

I had a nice time walking the Cheonggyecheon river (creek?). It’s separated from traffic, has fish and birds, and many people enjoy sitting alongside it. Don’t go on rainy day because it is also where rainwater flows to, but on an overcast or sunny day it’s great.

Not many people online suggest the city wall walk—in fact, I did not see it mentioned on any website I looked at—but I stumbled across it coming out of the Cheonggyecheon walk and in search of a 7-Eleven. It’s a nice 15 mile circular tour of old Seoul. I didn’t do the whole thing because it was already so late in the day but I will next time. It’s a lovely way to escape the huge buildings and constant traffic and see the small homes where people actually live. Bring a cold beverage (corn silk tea is nice). It’s hilly.

On the city wall walk by Naksan there is a cute cafe called cafe cogito. I had ice cream there and looked out over the city. It’s a lovely place to sit and cool off. The neighbors have a cat that goes adventuring around.

After dinner, I stumbled across yet another thing that I had somehow not seen mentioned anywhere: Jongno! I accidentally wandered into the Ikseon-dong Hanok village at night, where there are restaurants and shops in a reconstructed stone-and-timber village.

As I departed for the train, I also wandered into the night market on Donhwamun-ro 11-gil, which was packed and also looked excellent. It had tables and seating, giving it a very different feel than Myeong-dong. Pity I had already eaten… this looked like the place to be.

Getting to and from the airport (ICN) is irritating. You can do one straight shot from ICN to Seoul Station on the AREX train (which I recommend if it’s in budget for you and/or your employer; the slower train is fine too), but you if you need to transfer anywhere, it’s up, down, and around in circles in the train station. There is a lot of transfer friction.

Until next time, South Korea!

What I talk about when I talk about IRs

Thu, 12 Jun 2025 00:00:00 +0000

I have a lot of thoughts about the design of compiler intermediate representations (IRs). In this post I’m going to try and communicate some of those ideas and why I think they are important.

The overarching idea is being able to make decisions with only local information.

That comes in a couple of different flavors. We’ll assume that we’re compiling a method at a time, instead of a something more trace-like (tracing, tracelets, basic block versioning, etc).

Control-flow graphs

A function will normally have some control-flow: if, while, for, any amount of jumping around within a function. Let’s look at an example function in a language with advanced control-flow constructs:

int sumto(int n) {
  int result = 0;
  for (result = 0; n >= 0; n -= 1) {
    result += n;
  }
  return result;
}

Most compilers will deconstruct this for, with its many nested expressions, into simple comparisons and jumps. In order to resolve jump targets in your compiler, you may have some notion of labels (in this case, words ending with a colon):

int sumto(int n) {
  entry:
    int result = 0
  header:
    if n < 0, jumpto end
    result = result + n
    n = n - 1
    jumpto header
  end:
    return result
}

This looks kind of like a pseudo-assembly language. It has its high-level language features decomposed into many smaller instructions. It also has implicit fallthrough between labeled sections (for example, entry into header).

I mention these things because they, like the rest of the ideas in this post, are points in an IR design space. Representing code this way is an explicit choice, not a given. For example, one could make the jumps explicit by adding a jumpto header at the end of entry.

As soon as we add that instruction, the code becomes position-independent: as long as we start with entry, the chunks of code between labels could be ordered any which way: they are addressed by name and have no implicit ordering requirements.

This may seem arbitrary, but it gives the optimizer more flexibility. If some optimization rule decides, for example, that a branch to block4 may rarely be taken, it can freely re-order it toward the end of the function (or even on a different page!) so that more hot code can be on a single cache line.

Explicit jumps and labels turn the code from a strictly linear assembly into a control-flow graph (CFG). Each sequence of code without internal control-flow is a called basic block and is a vertex in this graph. The directed edges represent jumps between blocks. See for example this crude GraphViz representation:

Each basic block gets its own node in the graph, and directed edges between nodes represent jumps.

We’re actually kind of looking at extended basic blocks (EBBs), which allow for multiple control exits per block but only one control entry. A strict basic block representation of the above code would look, in text form, something like this:

int sumto(int n) {
  entry:
    int result = 0
    condbranch n < 0, iftrue: end, iffalse: header
  header:
    result = result + n
    n = n - 1
    condbranch n < 0, iftrue: end, iffalse: header
  end:
    return result
}

Notice how each block has exactly one terminator (control-flow instruction), with (in this case) 0 or 2 targets.

Opinions differ about the merits and issues of extended vs normal basic blocks. Most compilers I see use normal basic blocks. In either case, bringing the IR into a graph form gives us an advantage: thanks to Cousot and Cousot, our favorite power couple, we know how to do abstract interpretation on graphs and we can use this to build an advanced optimizer. See, for example, my intro to abstract interpretation post.

Register-based IRs

Some IRs are stack based. For concatenative languages or some newer JIT compilers, IRs are formatted in such a way that each opcode reads its operands from a stack and writes its outputs to a stack. This is reminiscent of a point-free coding style in languages such as Haskell or OCaml.

inc = (+ 1)

In this style, there is an implicit shared state: the stack. Dataflow is explicit (pushes and pops) and instructions can only be rearranged if the stack structure is preserved. This requires some non-local reasoning: to move an instruction, one must also rearrange the stack.

PUSH 1
PUSH 2
ADD

By contrast, in a register-based IR, things are more explicit. Instructions take named inputs (v0, v12, etc) and produce named outputs. Instructions can be slightly more easily moved around (modulo effects) as long as inputs remain defined. Local variables do not exist. The stack does not exist. Everything is IR “variables”.

v1 = CONST 1
v2 = CONST 2
v3 = ADD v1 v2

The constraints (names being defined) are part of the IR. This gets a little bit tricky if it’s possible to define a name multiple times.

v1 = CONST 1
v2 = CONST 2
x = ADD v1 v2
x = ADD x v2
v3 = MUL x 12

What does x mean in the instruction for v3? Which definition does it refer to? In order to reason about the instruction v3 = MUL x 12, we have to keep around some context. This is non-trivial: requiring compiler writers to constantly truck around side-tables and update them as they do analysis is slow and error-prone. Fortunately, if we enforce some interesting rules, we can push that analysis work into one pass up-front…

SSA

Static single assignment (SSA) was introduced by a bunch of folks at IBM (see my blog post about the different implementations). In SSA-based IRs, each variable can only be defined once. Put another way, a variable is its defining instruction; alternately, a variable and its defining instruction are addressed by the same name. The previous example is not valid SSA; x has two definitions.

If we turn the previous example into SSA, we can now use a different name for each instruction. This is related to the unique name assumption or the global names property: names do not depend on context.

v1 = CONST 1
v2 = CONST 2
x0 = ADD v1 v2
x1 = ADD x0 v2

Now we can identify each different ADD instruction by the variable it defines. This is useful in analysis…

CPS

I’d be remiss if I did not mention continuation-passing style (CPS) based IRs (and in fact, I had forgotten in the original draft of the post). As an IR, CPS is normally used in the analysis and optimization of functional programs, for example in the OCaml and Racket compilers. It is not required, however; MLton, for example, uses SSA in its compiler for Standard ML.

SSA and CPS can more or less represent the same programs, but they can each feel a natural fit for different languages (and different compiler authors). I don’t feel qualified to say much more here. For a more informed opinion, check out Andy Wingo’s approaching cps soup, especially the benefits and drawbacks near the end.

Type information

Speaking of CPS, I took a class with Olin Shivers and he described abstract interpretation as “automated theorem finding”. Unlike theorem provers such as Lean and Rocq, where you have to manually prove the properties you want, static analysis finds interesting facts that already exist in your program (and optimizers use them to make your program faster).

Your static analysis pass(es) can annotate your IR nodes with little bits of information such as:

has type int
has value 5
is between 2 and 5
is equivalent to the expression y*z
- See CF’s writeup of record_known_result: A Hint for Language-Specific Runtime Function Optimization in RPython’s Meta-JIT
does not have side effects
is loop invariant
does not escape
…

If your static analysis is over SSA, then generally the static analysis is easier and (potentially) storing facts is easier. This is due to this property called sparseness. Where a static analysis over non-SSA programs has to store facts about all variables at all program points, an analysis over SSA need only store facts about all variables, independent of context.

I sometimes describe this as “pushing time through the IR” but I don’t know that that makes a ton of sense.

Potentially more subtle here is that we could represent the above IR snippet as a list of tuples, where instructions are related via some other table (say, a “variable environment”):

code = [
  ("v1", "CONST", 1),
  ("v2", "CONST", 2),
  ("x0", "ADD", "v1", "v2"),
  ("x1", "ADD", "x0", "v2"),
]

Instead, though, we could allocate an object for each instruction and let them refer to one another by pointer (or index, if using Rust or something). Then they directly refer to one another (no need for a side-table), which might be faster and more compact. We can re-create nice names as needed for printing. Then, when optimizing, we look up the type information of an operand by directly reading a field (operands[N]->type or similar).

Another thing to note: when you start adding type information to your IR, you’re going to start asking type information questions in your analysis. Questions such as “what type is this instruction?”, where “type” could span a semilattice, and even refer to a specific run-time object by its pointer. In that case, it’s important to ask the right questions. For example:

wrong: Is instruction X a Const instruction?
right: Is the type of X known to be a specific object?

Const instructions are likely not the only opcodes that could produce specific objects; if you have an instruction like GuardIsObject, for example, that burns a specific expected pointer into the generated code, the type (and therefore the pointer) will come from the GuardIsObject instruction.

The big idea is that types represent a different slice of your IR than the opcodes and should be treated as such.

Anyway, SSA only stores type information about instructions and does not encode information that we might later learn in the IR. With basic SSA, there’s not a good way to encode refinements…

SSI

Static single information (SSI) form gives us new ways to encode metadata about instructions (variables). It was introduced by C. Scott Ananian in 1999 in his MS thesis (PDF). (I also discussed it briefly in the Scrapscript IR post.) Consider the following SSA program (represented as pseudo-Python):

# @0
x = int(...)
# @1
if x >= 0:
    # @2
    return x
else:
    # @4
    result = -x
    # @5
    return result

x is undefined at @0. x is defined and an integer at @1. But then we do something interesting: we split control flow based on the run-time value of x. We can take this split to add new and interesting information to x. For non-sparse analysis, we can record some fact on the side. That’s fine.

# @0
x = int(...)
# @1
if x >= 0:
    # @2
    LearnFact(x, nonnegative)
    # @3
    return x
else:
    # @4
    LearnFact(x, negative)
    # @5
    result = -x
    # @6
    return result

When doing a dataflow analysis, we can keep track of the fact that at @3, x is nonnegative, and at @5, x is negative. This is neat: we can then determine that all paths to this function return a positive integer.

Importantly, LearnFact(x, nonnegative) does not override the existing known type of x. Instead, it is a refinement: a set intersection. A lattice meet. The middle bit of a Venn diagram containing two overlapping circles, Integer and Nonnegative.

If we want to keep our information sparse, though, we have to add a new definition to the IR.

# @0
x = int(...)
# @1
if x >= 0:
    # @2
    newx0 = LearnFact(x, nonnegative)
    # @3
    return newx0
else:
    # @4
    newx1 = LearnFact(x, negative)
    # @5
    result = -newx1
    # @6
    return result

This is complicated (choose which variables to split, replace all uses, to maintain SSA, etc) but gives us new places to store information inside the IR. It means that every time we refer to newx0, we know that it is nonnegative and every time we refer to newx1, we know that it is negative. This information is independent of context!

I should note that you can get a lot of the benefits of SSI without going “full SSI”. There is no need to split every variable at every branch, nor add a special new merge instruction.

Okay, so we can encode a lot of information very sparsely in the IR. That’s neat. It’s powerful. But we should also be mindful that even in this very sparse representation, we are encoding information implicitly that we may not want to: execution order.

Sea of nodes

In a traditional CFG representation, the instructions are already scheduled, or ordered. Normally this comes from the programmer in the original source form and is faithfully maintained. We get data use edges in an IR like SSA, but the control information is left implicit. Some forms of IR, however, seek to reify both data and control dependencies into the IR itself. One such IR design is sea of nodes (SoN), which was originally designed by Cliff Click during his PhD.

In sea of nodes, every instruction gets its own vertex in the graph. Instructions have use edges to their operands, which can be either data or some other ordering property (control, effects, etc). The main idea is that IR nodes are by default unordered and are only ordered later, after effect analysis has removed a bunch of use edges.

Per Vognsen also notes that there is another motivating example of sea of nodes: in the previous SSI example, the LearnFact(x, nonnegative) cannot be validly hoisted above the n >= 0 check. In a “normal” IR, this is implicit in the ordering. In a sea of nodes world, this is explicitly marked with an edge from the LearnFact to the if n >= 0. I think Graal, for example, calls these LearnFact nodes “Pi nodes”.

I think I need to re-read the original paper, read a modern implementation (I get the feeling it’s not done exactly the same way anymore), and then go write more about it later. For now, see Simple, by Cliff Click and friends. It is an implementation in Java and a little book to go with it.

Design neighbors include value dependence graphs (VDG), value state dependence graphs (VSDG), region value state dependence graphs (RVSDG), and program dependence graphs (PDG).

Specification reification

Speaking of Cliff Click, I once heard/read something he said that sounded really interesting. Roughly, it was “elaborate the full semantics of the operation into the IR and let the optimizer sort it out”. That is, “open code” or “inline” your semantics.

For example, don’t emit code for a generic add operation that you later specialize:

v2 = GENERIC_ADD v0, v1

Instead, emit code that replicates the written semantics of the operation, whatever that is for your local language. This can include optimistic fast paths:

v1 = TYPE_OF v0
v2 = TYPE_OF v1
if (v1 == Fixnum && v2 == Fixnum) {
    v3 = FIXNUM_ADD v0, v1
}
else if (v1 == String && v2 == String) {
    v4 = STRING_CONCAT v0, v1
}
else {
    v5 = LOOKUP_METHOD v0, "add"
    v6 = CALL_METHOD v5, v0, v1
}
v7 = PHI v3, v4, v6

This has the advantage that you may end up with fewer specialized rewrite rules because constant propagation and branch folding take care of these specializations “for free”.

You can even attach probabilities to more or less likely branches to offer outlining hints in case all of this is never specialized.

Sure, the downside of this is that the generated IR might be bigger, so your optimizer might be slower—or worse, that your resulting generated code at the end might be bigger. But outlining, deduplication (functionalization?), and probably some other clever methods can help here.

Similarly, Maxime Chevalier-Boisvert and Marc Feeley write about this (PDF) in the context of basic block versioning. If the runtime’s generic add functions is written in IR, then callers to that function can specialize “through it” by calling it in different basic block contexts. That more or less gets you call-site specialization “for free”. See Figure 4 from their paper (lightly edited by me), where I think dollar-prefixed variable names indicate special functions known to the compiler:

function $rt_add(x, y) {
    if ($ir_is_i32(x)) { // If x is integer
        if ($ir_is_i32(y)) {
            var r = $ir_add_i32_ovf(x, y);
            if (r)
                return r;
            else // Handle the overflow case
                return $ir_add_f64($ir_i32_to_f64(x),
                                   $ir_i32_to_f64(y));
        } else if ($ir_is_f64(y))
            return $ir_add_f64($ir_i32_to_f64(x), y);
    } else if ($ir_is_f64(x)) { // If x is floating point
        if ($ir_is_i32(y))
            return $ir_add_f64(x, $ir_i32_to_f64(y));
        else if ($ir_is_f64(y))
            return $ir_add_f64(x, y);
    }
    // Evaluate arguments as strings and concatenate them
    return $rt_strcat($rt_toString(x), $rt_toString(y));
}

This is nice if you are starting a runtime from scratch or have resources to devote to re-writing chunks of the runtime in your IR. Then, even in a method JIT, you can get your inlined language semantics by function (partial) inlining.

More?

There’s probably more in this vein to be explored right now and probably more to invent in the future, too. Some other potentially interesting concepts to think about include:

union-find
e-graphs
ae-graphs and reification in IR (see also PLDI eqsat paper)
Webs, for operating on connected components of IR nodes that are all concerned about a thing (for example, call and function nodes are concerned about the signature of the callee)
- see also Benjamin Quiring paper
High-level IRs and strength reduction of language semantics before low-level stuff (no low-level concerns in my high-level IR, please)

Thank yous

Thank you to Chris Fallin, Hunter Goldstein, and Per Vognsen for valuable feedback on drafts of this post.

Zero-overhead checks with fake stack overflows

Wed, 28 May 2025 00:00:00 +0000

Imagine you have a virtual machine for a dynamic language. It supports starting multiple threads from within the guest language via some API. This is immediately challenging right out the gate because it requires keeping some key runtime-internal state in sync:

memory allocator / garbage collector
compiler
other bookkeeping

Instead of locking these components so that only one thread can use those shared resources at a time, runtime implementors tend to shard them: each thread gets its own slice to work with privately.

This means that each thread might have a separate mini-heap that it can use privately until exhaustion, and only then request a stop-the-world event. But how do we indicate that all threads should stop?

We could do something like check for a flag every N instructions, but this introduces significant overhead in the interpreter and does not play well with a compiler. Instead, ideally, there would be no overhead in the fast and normal case.

VM implementors have decided to solve this by folding the synchronization check into an existing check. Skybison, for example, folds this check into the stack overflow check.

On frame push, we have to check for a stack overflow anyway, so we can re-use the slow path. If the main thread wants to synchronize the other threads—because the programmer hit Control-C in the REPL, for example, causing an interrupt—it can artificially shrink the maximum stack size of the other threads. Then, the next time the other threads call a function, they will hit this overflow check and go to the slow path. In the slow path, they can disambiguate by checking another flag somewhere else.

Frame* Thread::pushNativeFrame(word nargs) {
  word locals_offset = Frame::kSize + nargs * kPointerSize;
  if (UNLIKELY(wouldStackOverflow(Frame::kSize))) {
    return handleInterruptPushNativeFrame(locals_offset);
  }
  return pushNativeFrameImpl(locals_offset);
}

NEVER_INLINE Frame* Thread::handleInterruptPushNativeFrame(word locals_offset) {
  if (handleInterrupt(Frame::kSize)) {
    return nullptr;
  }
  Frame* result = pushNativeFrameImpl(locals_offset);
  handleInterruptWithFrame();
  return result;
}

V8 does something very similar in Maglev on function entry.

But not all applications involve regular function calls. Sometimes it’s possible to have a hot loop in which no frames are pushed to the call stack. In this case, it might make sense to check for interrupts (or do other thread synchronization checks) in loop back-edges. CPython does this. PyPy does too. Ruby’s YARV does this too.

/* set PC to (PC + dst). */
DEFINE_INSN
jump
(OFFSET dst)
()
()
/* Same discussion as leave. */
// attr bool leaf = leafness_of_check_ints; /* has rb_threadptr_execute_interrupts() */
{
    RUBY_VM_CHECK_INTS(ec);
    JUMP(dst);
}

I’m writing this post because PyPy published a post about a new feature that uses a similar technique! For sampling profiling, they put their check (whether or not to sample) behind a check for exhaustion of the nursery (small heap).

Maybe next time I will write about guard pages, which I recently added to RBS…

Thank you to CF Bolz-Tereick for feedback on this post!

A simple search engine from scratch*

Tue, 20 May 2025 00:00:00 +0000

*if you include word2vec.

Chris and I spent a couple hours the other day creating a search engine for my blog from “scratch”. Mostly he walked me through it because I only vaguely knew what word2vec was before this experiment.

The search engine we made is built on word embeddings. This refers to some function that takes a word and maps it onto N-dimensional space (in this case, N=300) where each dimension vaguely corresponds to some axis of meaning. Word2vec from Scratch is a nice blog post that shows how to train your own mini word2vec and explains the internals.

The idea behind the search engine is to embed each of my posts into this domain by adding up the embeddings for the words in the post. For a given search, we’ll embed the search the same way. Then we can rank all posts by their cosine similarities to the query.

The equation below might look scary but it’s saying that the cosine similarity, which is the cosine of the angle between the two vectors cos(theta), is defined as the dot product divided by the product of the magnitudes of each vector. We’ll walk through it all in detail.

Equation from Wikimedia's Cosine similarity page.

Cosine distance is probably the simplest method for comparing a query embedding to document embeddings to rank documents. Another intuitive choice might be euclidean distance, which would measure how far apart two vectors are in space (rather than the angle between them).

We prefer cosine distance because it preserves our intuition that two vectors have similar meanings if they have the same proportion of each embedding dimension. If you have two vectors that point in the same direction, but one is very long and one very short, these should be considered the same meaning. (If two documents are about cats, but one says the word cat much more, they’re still just both about cats).

Let’s open up word2vec and embed our first words.

Embedding

We take for granted this database of the top 10,000 most popular word embeddings, which is a 12MB pickle file that vaguely looks like this:

couch  [0.23, 0.05, ..., 0.10]
banana [0.01, 0.80, ..., 0.20]
...

Chris sent it to me over the internet. If you unpickle it, it’s actually a NumPy data structure: a dictionary mapping strings to numpy.float32 arrays. I wrote a script to transform this pickle file into plain old Python floats and lists because I wanted to do this all by hand.

The loading code is straighforward: use the pickle library. The usual security caveats apply, but I trust Chris.

import pickle

def load_data(path):
    with open(path, "rb") as f:
        return pickle.load(f)

word2vec = load_data("word2vec.pkl")

You can print out word2vec if you like, but it’s going to be a lot of output. I learned that the hard way. Maybe print word2vec["cat"] instead. That will print out the embedding.

To embed a word, we need only look it up in the enormous dictionary. A nonsense or uncommon word might not be in there, though, so we return None in that case instead of raising an error.

def embed_word(word2vec, word):
    return word2vec.get(word)

To embed multiple words, we embed each one individually and then add up the embeddings pairwise. If a given word is not embeddable, ignore it. It’s only a problem if we can’t understand any of the words.

def vec_add(a, b):
    return [x + y for x, y in zip(a, b)]

def embed_words(word2vec, words):
    result = [0.0] * len(next(iter(word2vec.values())))
    num_known = 0
    for word in words:
        embedding = word2vec.get(word)
        if embedding is not None:
            result = vec_add(result, embedding)
            num_known += 1
    if not num_known:
        raise SyntaxError(f"I can't understand any of {words}")
    return result

That’s the basics of embedding: it’s a dictionary lookup and vector adds.

embed_words([a, b]) == vec_add(embed_word(a), embed_word(b))

Now let’s make our “search engine index”, or the embeddings for all of my posts.

Embedding all of the posts

Embedding all of the posts is a recursive directory traversal where we build up a dictionary mapping path name to embedding.

import os

def load_post(pathname):
    with open(pathname, "r") as f:
        contents = f.read()
    return normalize_text(contents).split()

def load_posts():
    # Walk _posts looking for *.md files
    posts = {}
    for root, dirs, files in os.walk("_posts"):
        for file in files:
            if file.endswith(".md"):
                pathname = os.path.join(root, file)
                posts[pathname] = load_post(pathname)
    return posts

post_embeddings = {pathname: embed_words(word2vec, words)
                   for pathname, words in posts.items()}

with open("post_embeddings.pkl", "wb") as f:
    pickle.dump(post_embeddings, f)

We do this other thing, though: normalize_text. This is because blog posts are messy and contain punctuation, capital letters, and all other kinds of nonsense. In order to get the best match, we want to put things like “CoMpIlEr” and “compiler” in the same bucket.

import re

def normalize_text(text):
    return re.sub(r"[^a-zA-Z]", r" ", text).lower()

We’ll do the same thing for each query, too. Speaking of, we should test this out. Let’s make a little search REPL.

A little search REPL

We’ll start off by using some Python’s built-in REPL creator library, code. We can make a subclass that defines a runsource method. All it really needs to do is process the source input and return a falsy value (otherwise it waits for more input).

import code

class SearchRepl(code.InteractiveConsole):
    def __init__(self, word2vec, post_embeddings):
        super().__init__()
        self.word2vec = word2vec
        self.post_embeddings = post_embeddings

    def runsource(self, source, filename="<input>", symbol="single"):
        for result in self.search(source):
            print(result)

Then we can define a search function that pulls together our existing functions. Just like that, we have a search:

class SearchRepl(code.InteractiveConsole):
    # ...
    def search(self, query_text, n=5):
        # Embed query
        words = normalize_text(query_text).split()
        try:
            query_embedding = embed_words(self.word2vec, words)
        except SyntaxError as e:
            print(e)
            return
        # Cosine similarity
        post_ranks = {pathname: vec_cosine_similarity(query_embedding,
                                                      embedding) for pathname,
                      embedding in self.post_embeddings.items()}
        posts_by_rank = sorted(post_ranks.items(),
                               reverse=True,
                               key=lambda entry: entry[1])
        top_n_posts_by_rank = posts_by_rank[:n]
        return [path for path, _ in top_n_posts_by_rank]

Yes, we have to do a cosine similarity. Thankfully, the Wikipedia math snippet translates almost 1:1 to Python code:

import math

def vec_norm(v):
    return math.sqrt(sum([x*x for x in v]))

def vec_cosine_similarity(a, b):
    assert len(a) == len(b)
    a_norm = vec_norm(a)
    b_norm = vec_norm(b)
    dot_product = sum([ax*bx for ax, bx in zip(a, b)])
    return dot_product/(a_norm*b_norm)

Finally, we can create and run the REPL.

sys.ps1 = "QUERY. "
sys.ps2 = "...... "

repl = SearchRepl(word2vec, post_embeddings)
repl.interact(banner="", exitmsg="")

This is what interacting with it looks like:

QUERY. type inference
_posts/2024-10-15-type-inference.md
_posts/2025-03-10-lattice-bitset.md
_posts/2025-02-24-sctp.md
_posts/2022-11-07-inline-caches-in-skybison.md
_posts/2021-01-14-inline-caching.md
QUERY.

This is a sample query from a very small dataset (my blog). It’s a pretty good search result, but it’s probably not representative of the overall search quality. Chris says that I should cherry-pick “because everyone in AI does”.

Okay, that’s really neat. But most people who want to look for something on my website do not run for their terminals. Though my site is expressly designed to work well in terminal browsers such as Lynx, most people are already in a graphical web browser. So let’s make a search front-end.

A little web search

So far we’ve been running from my local machine where I don’t mind having a 12MB file of weights sitting around. Now that we’re moving to web, I would rather not burden casual browsers with an unexpected big download. So we need to get clever.

Fortunately, Chris and I had both seen this really cool blog post that talks about hosting a SQLite database on GitHub Pages. The blog post details how the author:

compiled SQLite to Wasm so it could run on the client,
built a virtual filesystem so it could read database files from the web,
did some smart page fetching using the existing SQLite indexes,
built additional software to fetch only small chunks of the database using HTTP Range requests

That’s super cool, but again: SQLite, though small, is comparatively big for this project. We want to build things from scratch. Fortunately, we can emulate the main ideas.

We can give the word2vec dict a stable order and split it into two files. One file can just have the embeddings, no names. Another file, the index, can map every word to the byte start and byte length of the weights for that word (we figure start&length is probably smaller on the wire than start&end).

# vecs.jsonl
[0.23, 0.05, ..., 0.10]
[0.01, 0.80, ..., 0.20]
...

# index.json
{"couch": [0, 20], "banana": [20, 30], ...}

The cool thing about this is that index.json is dramatically smaller than the word2vec blob, weighing in at 244KB. Since that won’t change very often (how often does word2vec change?), I don’t feel so bad about users eagerly downloading the entire index. Similarly, the post_embeddings.json is only 388KB. They’re even cacheable. And automagically (de)compressed by the server and browser (to 84KB and 140KB, respectively). Both would be smaller if we chose a binary format, but we’re punting on that for the purposes of this post.

Then we can make HTTP Range requests to the server and only download the parts of the weights that we need. It’s even possible to bundle all of the ranges into one request (it’s called multipart range). Unfortunately, GitHub Pages does not appear to support multipart, so instead we download each word’s range in a separate request.

Here’s the pertinent JS code, with (short, very familiar) vector functions omitted:

(async function() {
  // Download stuff
  async function get_index() {
    const req = await fetch("index.json");
    return req.json();
  }
  async function get_post_embeddings() {
    const req = await fetch("post_embeddings.json");
    return req.json();
  }
  const index = new Map(Object.entries(await get_index()));
  const post_embeddings = new Map(Object.entries(await get_post_embeddings()));
  // Add search handler
  search.addEventListener("input", debounce(async function(value) {
    const query = search.value;
    // TODO(max): Normalize query
    const words = query.split(/\s+/);
    if (words.length === 0) {
      // No words
      return;
    }
    const requests = words.reduce((acc, word) => {
      const entry = index.get(word);
      if (entry === undefined) {
        // Word is not valid; skip it
        return acc;
      }
      const [start, length] = entry;
      const end = start+length-1;
      acc.push(fetch("vecs.jsonl", {
        headers: new Headers({
          "Range": `bytes=${start}-${end}`,
        }),
      }));
      return acc;
    }, []);
    if (requests.length === 0) {
      // None are valid words :(
      search_results.innerHTML = "No results :(";
      return;
    }
    const responses = await Promise.all(requests);
    const embeddings = await Promise.all(responses.map(r => r.json()));
    const query_embedding = embeddings.reduce((acc, e) => vec_add(acc, e));
    const post_ranks = {};
    for (const [path, embedding] of post_embeddings) {
      post_ranks[path] = vec_cosine_similarity(embedding, query_embedding);
    }
    const sorted_ranks = Object.entries(post_ranks).sort(function(a, b) {
      // Decreasing
      return b[1]-a[1];
    });
    // Fun fact: HTML elements with an `id` attribute are accessible as JS
    // globals by that same name.
    search_results.innerHTML = "";
    for (let i = 0; i < 5; i++) {
      search_results.innerHTML += `<li>${sorted_ranks[i][0]}</li>`;
    }
  }));
})();

You can take a look at the live search page. In particular, open up the network requests tab of your browser’s console. Marvel as it only downloads a couple 4KB chunks of embeddings.

So how well does our search technology work? Let’s try to build an objective-ish evaluation.

Evaluation

Let’s evaluate our search engine and measure how often it returns posts in the top few results when we query with hand-crafted search keywords.

We start by collecting an evaluation dataset of (document, query) pairs. Right from the start we’re going to bias this evaluation by collecting this dataset ourselves, but hopefully it’ll still help us get an intuition about the quality of the search. A query in this case is just a few search terms that we think should retrieve a document successfully.

sample_documents = {
  "_posts/2024-10-27-on-the-universal-relation.md": "database relation universal tuple function",
  "_posts/2024-08-25-precedence-printing.md": "operator precedence pretty print parenthesis",
  "_posts/2019-03-11-understanding-the-100-prisoners-problem.md": "probability strategy game visualization simulation",
  # ...
}

Now that we’ve collected our dataset, let’s implement a top-k accuracy metric. This metric measures the percentage of the time a document appears in the top k search results given its corresponding query.

def compute_top_k_accuracy(
    # Mapping of post to sample search query (already normalized)
    # See sample_documents above
    eval_set: dict[str, str],
    max_n_keywords: int,
    max_top_k: int,
    n_query_samples: int,
) -> list[list[float]]:
    counts = [[0] * max_top_k for _ in range(max_n_keywords)]
    for n_keywords in range(1, max_n_keywords + 1):
        for post_id, keywords_str in eval_set.items():
            for _ in range(n_query_samples):
                # Construct a search query by sampling keywords
                keywords = keywords_str.split(" ")
                sampled_keywords = random.choices(keywords, k=n_keywords)
                query = " ".join(sampled_keywords)

                # Determine the rank of the target post in the search results
                ids = search(query, n=max_top_k)
                rank = safe_index(ids, post_id)

                # Increment the count of the rank
                if rank is not None and rank < max_top_k:
                    counts[n_keywords - 1][rank] += 1

    accuracies = [[0.0] * max_top_k for _ in range(max_n_keywords)]
    for i in range(max_n_keywords):
        for j in range(max_top_k):
            # Divide by the number of samples to get the average across samples and
            # divide by the size of the eval set to get accuracy over all posts.
            accuracies[i][j] = counts[i][j] / n_query_samples / len(eval_set)

            # Accumulate accuracies because if a post is retrieved at rank i,
            # it was also successfully retrieved at all ranks j > i.
            if j > 0:
                accuracies[i][j] += accuracies[i][j - 1]

    return accuracies

Plotting top-k accuracy for various values of k gives us the following chart. Note that we get higher accuracy as we increase k – in the limit, as k approaches our number of documents we approach 100% accuracy. Also, accuracy increases as we increase the number of keywords. Finally, we can see the lines getting closer together as the number of keywords increases, which indicates there are diminishing marginal returns for each new keyword.

Do these megabytes of word embeddings actually do anything to improve our search? We would have to compare to a baseline. Maybe that baseline is adding up the counts of all keywords in each document to rank them. We leave this as an exercise to the reader because we ran out of time :)

It would also be interesting to see how a bigger word2vec helps accuracy. While sampling for top-k, there is a lot of error output (I can't understand any of ['prank', ...]). These unknown words get dropped from the search. A bigger word2vec (more than 10,000 words) might contain these less-common words and therefore search better.

Wrapping up

You can build a small search engine from “scratch” with only a hundred or so lines of code. See the full search.py, which includes some of the extras for evaluation and plotting.

Future ideas

We can get fancier than simple cosine similarity. Let’s imagine that all of our documents talk about computers, but only one of them talks about compilers (wouldn’t that be sad). If one of our search terms is “computer” that doesn’t really help narrow down the search and is noise in our embeddings. To reduce noise we can employ a technique called TF-IDF (term frequency inverse document frequency) where we factor out common words across documents and pay closer attention to words that are more unique to each document.

ZJIT has been merged into Ruby

Wed, 14 May 2025 00:00:00 +0000

Originally published on Rails At Scale.

Following Maxime’s presentation at RubyKaigi 2025, the Ruby developers meeting, and Matz-san’s approval, ZJIT has been merged into Ruby. Hurray! In this post, we will give a high-level overview of the project, which is very early in development.

ZJIT is a new just-in-time (JIT) Ruby compiler built into the reference Ruby implementation, YARV, by the same compiler group that brought you YJIT. We (Maxime Chevalier-Boisvert, Takashi Kokubun, Alan Wu, Max Bernstein, and Aiden Fox Ivey) have been working on ZJIT since the beginning of this year.

It’s different from YJIT in several ways:

Instead of compiling YARV bytecode directly to the low-level IR (LIR), it uses an high-level SSA-based intermediate representation (HIR)
Instead of compiling one basic block at a time, it compiles one entire method at a time
Instead of using lazy basic block versioning (LBBV) to profile types, it reads historical type information from the profiled interpreter
Instead of doing optimizations while lowering YARV to LIR, it has a high-level modular optimizer that works on HIR

The main difference is that the team is making the intentional choice to build a more traditional “textbook” compiler so it is easy for the community to contribute to.

There are some interesting tradeoffs. For example, while YJIT’s architecture allows for easy interprocedural type-based specialization, ZJIT’s architecture gives more code at once to the optimizer.

Let’s talk about the ZJIT architecture.

Current architecture

At a high level, ZJIT takes in YARV bytecode, builds an IR, does some optimizations, and emits machine code. Simplified, it looks like this:

How Ruby code flows through the compiler.

We’ll take the following sample Ruby program and bring it through the full compiler pipeline:

# add.rb
def add(left, right)
  left + right
end

p add(1, 2)
p add(3, 4)

Let’s start by getting a feel for YARV.

YARV

The Ruby VM has compiled two functions to YARV bytecode. First we see the top-level function (omitted for brevity with ...) and then the add instruction sequence (ISEQ). There’s a lot going on but it’s important to note that YARV is a stack machine with local variables. Most instructions pop their inputs from the stack and push their results onto the stack.

For example, in add, getlocal_WC_0 (offsets 0000 and 0002) is a specialized instruction that reads the left local from slot 0 (see the third column) and pushes it onto the stack. Same with right at slot 1. Then it calls into a specialized + handler, opt_plus (offset 0004), which reads the arguments off the stack and pushes the result back onto the stack.

$ ruby --dump=insns add.rb
...

== disasm: #<ISeq:add@add.rb:2 (2,0)-(4,3)>
local table (size: 2, argc: 2 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
[ 2] left@0<Arg>[ 1] right@1<Arg>
0000 getlocal_WC_0                          left@0                    (   3)[LiCa]
0002 getlocal_WC_0                          right@1
0004 opt_plus                               <calldata!mid:+, argc:1, ARGS_SIMPLE>[CcCr]
0006 leave
$

The opcode opt_plus is a generic method lookup and call operation but it also has several fast-path cases inlined into the VM’s instruction handler. It has code for handling the common case of adding two small integers (fixnums) with a fallback to the generic send.

static VALUE
vm_opt_plus(VALUE recv, VALUE obj)
{
    // fast path for fixnum + fixnum
    if (FIXNUM_2_P(recv, obj) &&
        BASIC_OP_UNREDEFINED_P(BOP_PLUS, INTEGER_REDEFINED_OP_FLAG)) {
        return rb_fix_plus_fix(recv, obj);
    }
    // ... some other common cases like float + float ...
    // ... fallback code for someone having redefined `Integer#+` ...
}

Importantly, it’s not enough for opt_plus to check the types of its arguments. The opcode handler also has to make sure (via a “bop check”¹) that Integer#+ has not been redefined. If the Integer#+ has been changed, it must fall back to the generic operation so the VM can call into the newly redefined method.

After running the bytecode function some number of times in the interpreter (a configurable number), ZJIT will change some opcodes to modified versions that profile their arguments. For example, opt_plus will get rewritten to zjit_opt_plus. This modified version records the types of the opcode’s input values on the stack into a special location that ZJIT knows about.

After a more calls to the function (some other configurable number), ZJIT will compile it. Let’s see what happens to opt_plus in HIR, the first part of the compiler pipeline. If you’re following along, from here on out, you’ll need to have a Ruby configured with --enable-zjit (see the docs).

HIR

In the bytecode, which is tersely encoded, jumps are offsets, some control-flow is implicit, and most dataflow is via the stack.

By contrast, HIR looks more like a graph. Jumps have pointers to their targets and there’s no stack: instructions that use data have pointers directly to the instructions that create the data.

Every function has a list of basic blocks. Every basic block has a list of instructions. Every instruction is addressable by its ID (InsnId, looks like v12), has a type (after the InsnId), has an opcode, and has some operands.

To show what I mean, here is the text representation of the HIR constructed directly from the bytecode (note how ZJIT has to be enabled with --zjit):

$ ruby --zjit --zjit-dump-hir-init add.rb
HIR:
fn add:
bb0(v0:BasicObject, v1:BasicObject):
  v4:BasicObject = SendWithoutBlock v0, :+, v1
  Return v4
$

The HIR text representation may on the surface look similar to the bytecode but that’s only really because they are both text. A more accurate depiction of HIR’s graph-like nature is this diagram:

Arrows indicate pointers from uses to the data used. Many instructions such as SendWithoutBlock and Send produce output data. We refer to this output data by the name of the instruction that produced it. This is why the Return instruction points to the Send.

(We also see that opt_plus has been turned back into a generic method send to :+. This is because (as mentioned earlier) under the hood, many opt_xyz instructions such as opt_plus are opt_send_without_block in disguise and we do our type optimization later in the compiler pipeline.)

Let’s pick apart this example:

v4:BasicObject = SendWithoutBlock v0, :+, v1 is an instruction
v4 is the both the ID of the instruction and name of its output data
The output has type BasicObject (or subclass)
It is a send operation, specifically opt_send_without_block
The self is v0
The method name is :+
The operands are both v0 and v1

Then the HIR goes through our optimization pipeline.

After some optimization, the HIR looks quite different. We no longer see a generic send operation but instead see type-specialized code:

$ ruby --zjit --zjit-dump-hir add.rb
HIR:
fn add:
bb0(v0:BasicObject, v1:BasicObject):
  PatchPoint BOPRedefined(INTEGER_REDEFINED_OP_FLAG, BOP_PLUS)
  v7:Fixnum = GuardType v0, Fixnum
  v8:Fixnum = GuardType v1, Fixnum
  v9:Fixnum = FixnumAdd v7, v8
  Return v9
$

The optimizer has inserted GuardType instructions that each check at run-time if their operands are Fixnum. If it is not a fixnum, the generated code will jump into the interpreter as a fallback. This way, we only need to generate specialized code—the FixnumAdd.

But this talk of GuardType and FixnumAdd is still pretty symbolic and high-level. Let’s go one step further in the compiler pipeline into LIR.

LIR

LIR is meant to be a multi-platform assembler. The only fancy feature it really provides is a register allocator. When we transform HIR into LIR, we mostly focus on transforming our high-level operations into an assembly-like language. To make this easier, we allocate as many virtual LIR registers as we like. Then the register allocator maps those onto physical registers and stack locations.

Here’s the add function in LIR:

$ ruby --zjit --zjit-dump-lir add.rb
LIR:
fn add:
Assembler
    000 Label() -> None
    001 FrameSetup() -> None
    002 LiveReg(A64Reg { num_bits: 64, reg_no: 0 }) -> Out64(0)
    003 LiveReg(A64Reg { num_bits: 64, reg_no: 1 }) -> Out64(1)
# The first GuardType
    004 Test(Out64(0), 1_u64) -> None
    005 Jz() target=SideExit(FrameState { iseq: 0x1049ca480, insn_idx: 4, pc: 0x6000002b2520, stack: [InsnId(0), InsnId(1)], locals: [InsnId(0), InsnId(1)] }) -> None
# The second GuardType
    006 Test(Out64(1), 1_u64) -> None
    007 Jz() target=SideExit(FrameState { iseq: 0x1049ca480, insn_idx: 4, pc: 0x6000002b2520, stack: [InsnId(0), InsnId(1)], locals: [InsnId(0), InsnId(1)] }) -> None
# The FixnumAdd; side-exit if it overflows Fixnum
    008 Sub(Out64(0), 1_i64) -> Out64(2)
    009 Add(Out64(2), Out64(1)) -> Out64(3)
    010 Jo() target=SideExit(FrameState { iseq: 0x1049ca480, insn_idx: 4, pc: 0x6000002b2520, stack: [InsnId(0), InsnId(1)], locals: [InsnId(0), InsnId(1)] }) -> None
    011 Add(A64Reg { num_bits: 64, reg_no: 19 }, 38_u64) -> Out64(4)
    012 Mov(A64Reg { num_bits: 64, reg_no: 19 }, Out64(4)) -> None
    013 Mov(Mem64[Reg(20) + 16], A64Reg { num_bits: 64, reg_no: 19 }) -> None
    014 FrameTeardown() -> None
    015 CRet(Out64(3)) -> None
$

It is much more explicit about what is going on than the HIR. In it we see some lower-level details such as:

FrameSetup and FrameTeardown, which correspond to native frame base pointer operations
Test, which is a single bit test instruction
Explicit conditional jumps to side-exit into the interpreter, such as Jz and Jo
Sub and Add, which are single math instructions

In LIR output for other functions, we might see some high-level HIR constructions turned into calls to C runtime (helper) functions.

It’s not noticeable in this example, but another difference between HIR and LIR is that LIR is one big linear block: unlike HIR, it does not have multiple basic blocks.

Last, from LIR, we go to assembly.

Assembly (ASM)

The assembly listing is a little bit long for the blog post but I will show an interesting snippet that illustrates the utility of GuardType and FixnumAdd:

$ ruby --zjit --zjit-dump-disasm add.rb
...
# Insn: v7 GuardType v0, Fixnum
0x6376b7ad400f: test dil, 1
0x6376b7ad4013: je 0x6376b7ad4000
# Insn: v8 GuardType v1, Fixnum
0x6376b7ad4019: test sil, 1
0x6376b7ad401d: je 0x6376b7ad4005
# Insn: v9 FixnumAdd v7, v8
0x6376b7ad4023: sub rdi, 1
0x6376b7ad4027: add rdi, rsi
0x6376b7ad402a: jo 0x6376b7ad400a
...
$

You can see that GuardType and FixnumAdd only require a couple of very fast machine instructions each. This is the value of type specialization!

This assembly snippet shows x86 instructions, but ZJIT also has an ARM backend. The generated code looks very similar in ARM.

Future plans and conclusion

It is still very early in the ZJIT project. While we encourage reading the source and trying local experiments, we are not yet running ZJIT in production and caution you to also avoid doing so. It is an exciting and bumpy road ahead!

For this reason, we will continue maintaining YJIT for now and Ruby 3.5 will ship with both YJIT and ZJIT. In parallel, we will improve ZJIT until it is on par (features and performance) with YJIT.

We’re currently working on a couple more features that will make the JIT run real code. For starters, we are implementing side-exits. Right now, if GuardType observes a type that it doesn’t expect, it aborts. Ideally we would be able to actually jump into the interpreter.

Side exits will enable us to do two exciting things:

Run the Ruby test suite, giving us a correctness baseline
Run yjit-bench and other production applications, giving us a performance baseline

Then we’ll get to work on profiling and seeing what optimizations are most impactful.

Thanks for reading this post! We’ll make more information and documentation available soon.

This is because it’s possible to redefine (almost?) any method in Ruby, including built-in ones such as Integer#+ (“basic operations”, hence “bop”).

The developers of the YARV VM want to include shortcuts for common operations such as adding small numbers (“fixnums”), but want to still support falling back to very dynamic behavior. ↩

Writing that changed how I think about PL

Tue, 13 May 2025 00:00:00 +0000

Every so often I come across a paper, blog post, or (occasionally) video that completely changes how I think about a topic in programming languages and compilers. For some of these posts, I can’t even remember how I thought about the idea before reading it—it was that impactful.

Here are some of those posts in no particular order:

a simple semi-space collector by Andy Wingo brought the concept of a Cheney/copying/compacting garbage collector from theory to practice for me. The garbage collector core¹ in the post is tiny, extensible, and can be understood in an afternoon.
Implementing a Toy Optimizer by CF Bolz-Tereick changed how I think about instruction rewrites in an optimizer. No more find-and-replace. Instead, use a forwarding pointer! I love union-find. The whole toy optimizer series is fantastic: each post brings something new and interesting to the table.
A Knownbits Abstract Domain for the Toy Optimizer, Correctly by CF Bolz-Tereick is a two-for-one. It both introduced me to a new abstract domain for optimizers and changed how I think about Z3. Before, I vaguely knew about Z3 as this thing that can check numeric operations, kind of. The post, however, uses Z3 as a proof engine: if Z3 can’t find a counterexample, the code is correct². Furthermore, it uses Z3 as a verifier for Python code by using the same Python code on both Z3 objects and normal Python objects.
Cranelift, Part 3: Correctness in Register Allocation by Chris Fallin also made proofs more accessible, but in a different way. Instead of proving your register allocator correct on all inputs, prove it correct on one input: the current code. This means that in production, you either get a correct register allocation or a meaningful crash. Additionally, it uses fuzzing as a state space exploration tool that can identify bugs by making the verifier fail.
Regular Expression Matching: the Virtual Machine Approach by Russ Cox made regular expression engines make sense to me. In the blog post is a small regular expression engine in under 50 lines of readable code. As a side effect, it also made coroutines/fibers/schedulers more understandable because its implementation of nondeterminism makes regex “threads” in user-space.
micrograd by Andrej Karpathy is a tiny implementation of neural networks with no libraries (not even NumPy). It’s how I came to understand machine learning and even write some blog posts about it.
How I implement SSA form by Fil Pizlo changed how I think about union-find. Instead of either storing an additional pointer inside every object or having a side-table, add an Identity tag that can store the pointer. Unlike the other two approaches, this is a destructive rewrite, but it saves space. It also introduces two new things that I am still mulling over: Phi/Upsilon form and TBAA-style heap effects.
Speculation in JavaScriptCore is another absolute banger by Fil Pizlo. I learn new things every time I re-read this post, which describes how JSC’s various optimizers work (and I guess how Fil thinks about optimizers).
Modernizing Compiler Design for Carbon Toolchain is a talk by Chandler Carruth on the design of the Carbon compiler. About 29 minutes into it Chandler sets incredibly aggressive compile-time budgets and explains how the compiler is architected to fit in that time budget. At around 40 minutes, he starts explaining per-layer how this works, starting at the lexer.
A Python Interpreter Written in Python by Allison Kaptur made bytecode interpreters (and, specifically, how CPython works internally) click for me.
Parsing expressions by precedence climbing by Eli Bendersky presented an understandable and easier-to-develop alternative to traditional hand-written recursive descent parsers. It both made parsers less scary to me and also at the same time illustrated how precedence climbing is the same algorithm as recursive descent but instead of having 10 different functions with similar structure, it uses one function and a table.
Ruby JIT Challenge by Takashi Kokubun is a great start to code generation and it is more general than JIT, too. In the post he also shows off a cool approach to register allocation that I had not seen before: fold your stack operations at compile-time to operate on a virtual stack of physical registers.
An Incremental Approach to Compiler Construction (PDF) by Abdulaziz Ghuloum brought compilers and code generation from a mystical multi-pass beast down to an understandable single-pass form. I like how it implements each feature end-to-end, one by one.
Lessons from Writing a Compiler by Fernando Borretti puts into words this stripey implementation strategy in the first section, which is really neat.
egg: Fast and extensible equality saturation by [multiple authors] changed how I think about optimizers and pass ordering. Why not just generate the compressed hypergraph of all possible versions of an expression and then pick the “best” one? I mean, there are a couple of reasons, but it was mind-expanding.
Cranelift: Using E-Graphs for Verified, Cooperating Middle-End Optimizations by Chris Fallin showed that e-graphs are workable and remarkably effective in a production compiler, even if you don’t do the “full shebang”.
Acyclic Egraphs and Smart Constructors by Phil Zucker further explores acyclic egraphs in the small. This one was a slow burn: I read it when it came out, didn’t really get it, and then months later it clicked more. I’m sure that in a few months it’ll click again.
This Reddit comment by Bob Nystrom and Flattening ASTs by Adrian Sampson are a double whammy that led to two things: 1) AST storage can be really compact, almost like bytecode 2) a really interesting discussion on Mastodon about massively-parallel lock-free abstract interpretation if IRs were stored like this. This (and some off-hand comments by Cliff Click about bump-allocation only taking “a few clocks”) changed how I think about allocating and referring to IR nodes.

I hope you enjoy the readings!

There’s just one small bug in that post: is_forwarded should check if the bit is 0, not 1. The correct version is:
```
int is_forwarded(struct gc_obj *obj) {
  return (obj->tag & NOT_FORWARDED_BIT) == 0;
}
```
“There is a contract between the GC and the user in which the user agrees to always set the NOT_FORWARDED_BIT in the first word of its objects.” ↩
For some definition of correct, anyway. What does it even mean for code to be correct? I have a whole unit on this in my course because it’s a tricky topic. ↩

Precedence climbing

Sun, 04 May 2025 00:00:00 +0000

I wrote a sample precedence-climbing expression parser. I wrote it for Patrick Dubroy’s excellent small website for small programs, which uses docco to pull out code comments into a side pane next to the code and make everything look pretty.

My implementation is a tokenizer and parser for simple math expressions and function calls in under 200 SLOC. It builds an “AST” in the form of nested Python lists that kind of look like S-expressions (but you could also make classes for AST nodes if you like).

>>> tokenize("1+2*3")
[1, '+', 2, '*', 3]
>>> parse(_)
['+', 1, ['*', 2, 3]]
>>>

It also comes with tests. It works standalone or can be used as the “simple expression parser” component of a larger recursive descent parser (that handles function definitions, variable binding, etc).

See also Parsing to IR and lvalues which takes this approach but builds an SSA based IR instead of an AST.

Nix derivations by hand, without guessing

Sat, 05 Apr 2025 00:00:00 +0000

I went on a little low-level Nix adventure yesterday and early this morning because of this excellent blog post. In it, Farid builds up the simplest possible Nix derivation—making a file that has the contents “hello world”. Here’s what I like about it:

No Nix language; just the really fiddly low-level bits of building a derivation by hand
It’s incremental, building from a little friendly JSON blob and incrementally adding scarier parts
It only uses two low-level Nix commands

One thing that I did not understand after reading, though, was where the hashes come from. Farid’s post does the same thing that I occasionally do with the C++ or Rust compiler, where we intionally cause an error to get information that the compiler already knows. That’s not very satisfying for someone (me) who is starting basically from zero. So I went down a bit of a rabbit hole trying to figure out how to manually generate these hashes.

But first, let’s get some terms out of the way. Please forgive (but help me correct) mistakes along the way, because this is my first time messing around with Nix.

What on earth is a derivation?

As far as I can tell, it’s a very precise recipe for building a file. It’s kind of like a Make recipe or a shell script except all of its inputs and outputs refer to well-known long paths in /nix/store.

Here’s an example from Farid’s blog that we will be trying to replicate in this post that has one of these path names:

{
  "name": "simple",
  "system": "x86_64-linux",
  "builder": "/bin/sh",
  "outputs": {
    "out": {
      "path": "/nix/store/5bkcqwq3qb6dxshcj44hr1jrf8k7qhxb-simple"
    }
  },
  "inputSrcs": [],
  "inputDrvs": {},
  "env": {
    "out": "/nix/store/5bkcqwq3qb6dxshcj44hr1jrf8k7qhxb-simple"
  },
  "args": [
    "-c",
    "echo 'hello world' > $out"
  ]
}

In it, we have a name field (your pick) and a system field (I am guessing there are a couple of well-known system/platform names. I am also running x86_64 Linux by happenstance. If you follow along, your hashes will be the same as my hashes only if you are on the same system, I think.).

We also have this thing called a builder, which is the singular command that gets run. In this case, we also pass it args (at the bottom) and environment variables from env. Don’t think too hard about inputSrcs and inputDrvs because they don’t come into play in this post.

Last, we have outputs, which, like env, also has this magical path name. It’s the output file (or directory, I think) that your Nix derivation is required to create. The name out in each of outputs and env is arbitrary and I think they don’t have to have the same name. Just convention.

This derivation roughly corresponds to the following Make recipe (remember that Make variables also start with $ so you have to escape your shell variables like $${var}):

/nix/store/5bkcqwq3qb6dxshcj44hr1jrf8k7qhxb-simple:
	export out=/nix/store/5bkcqwq3qb6dxshcj44hr1jrf8k7qhxb-simple; \
    echo 'hello world' > $${out}

But where does that huge path come from?

Getting set up

I didn’t want to install Nix, so I did all of my explorations inside Docker. If you’re not going to use Nix long-term, I recommend you do the same, since it otherwise makes some pretty invasive system changes.

Here’s all you need to get started:

FROM nixos/nix AS builder

Now you can build a Docker container:

$ docker build . -t notnix
...
$

and then start /bin/sh, the only shell in an easy-to-remember path:

$ docker run -i -t notnix /bin/sh
sh-5.2#
...

Any time I use a Nix command in the rest of the article, it’s either in a Docker RUN command or in /bin/sh inside a running Nix container.

Now let’s find some hashes.

Finding hashes

John Ott pointed out to me that Nix docs on store paths had a partial answer. He said that the path comes from hashing the ATerm representation of the derivation with the outPath set to an empty string.

…what?

So after some digging and re-reading Farid’s post, apparently ATerm is an old(er) configuration language that looks kind of like building OCaml variants. And I guess it makes sense that we shouldn’t need the path to calculate the path (otherwise we’d be in circular trouble). So I had Nix create the ATerm form of my JSON derivation without any paths:

{
  "name": "simple",
  "system": "x86_64-linux",
  "builder": "/bin/sh",
  "outputs": {
    "out": {
    }
  },
  "inputSrcs": [],
  "inputDrvs": {},
  "env": {
  },
  "args": [
    "-c",
    "echo 'hello world' > $out"
  ]
}

by running:

$ nix --extra-experimental-features nix-command derivation add < simple.json
/nix/store/1p6dixyqvjddfq5fmys3i55nl90ckjam-simple.drv
$

Running that command outputs the path of the ATerm form file, which we can check out:

$ cat /nix/store/1p6dixyqvjddfq5fmys3i55nl90ckjam-simple.drv
Derive([("out","","","")],[],[],"x86_64-linux","/bin/sh",["-c","echo 'hello world' > $out"],[("out","")])$

You can see they are making sure to remove all whitespace, even the trailing newline (hence the $ at the end, which is my shell prompt).

Okay, so we have an ATerm form of the derivation and it has no output path. I guess we hash it? I got a little lost at this point until Jamey Sharp chimed in with the even more detailed store path specification.

This clarified, after many reads, that we have to do the following steps. At some point I switched to using Python because it got a little text manipulation heavy:

Call the ATerm derivation sans output path the inner-fingerprint because we’re not doing anything with text or source types or NARs or something
Make a SHA256-hash of the inner-fingerprint and then base16-encode it. That’s called the inner-digest

Okay, not so bad:

import hashlib
import base64

with open(inner_fingerprint, "rb") as f:
    inner_fingerprint_hash = hashlib.file_digest(f, "sha256").digest()
inner_digest = (
    base64.b16encode(inner_fingerprint_hash).decode("utf-8").lower()
)

Then, once we have that, we do some more stuff to it:

Combine the inner-digest with some other fields like the derivation’s name and call that the fingerprint
Hash that, take the first 20 bits, and take the base32 representation of that

Alright, not so bad, Python can do all of this in the standard library:

name = deriv["name"]  # "simple"
# the "out" is the name we picked earlier and is arbitrary
fingerprint = f"output:out:sha256:{inner_digest}:/nix/store:{name}"
fingerprint_hash = hashlib.sha256(fingerprint.encode("utf-8")).digest()
fingerprint_digest = hashlib.b32encode(fingerprint_hash[:20])

However. The docs are misleading about two things and that sent me on a merry chase.

First of all, Nix does not use normal base32. They use a different character set. Also, they base32 in reverse. I didn’t figure either of these things out until tombl chimed in on Twitter.

Second of all, the store-path docs are outright lying when they say “the first 160 bits [20 bytes] of a SHA-256 hash”. Instead, what they should say is “do this weird XOR thing on the hash, folding it back onto itself kinda.”

I only got that second bit by digging through the Nix C++ codebase. So instead, what we really want is this:

def to_nix_base32(bytes_data):
    b32_alphabet = b"ABCDEFGHIJKLMNOPQRSTUVWXYZ234567"
    b32_nix = b"0123456789abcdfghijklmnpqrsvwxyz"
    trans = bytes.maketrans(b32_alphabet, b32_nix)
    return base64.b32encode(bytes_data[::-1]).translate(trans).decode("utf-8")

def compress_hash(h, newlen):
    result = bytearray(b"\0" * newlen)
    for i in range(len(h)):
        result[i % newlen] ^= h[i]
    return bytes(result)

fingerprint = (
    f"output:{output}:sha256:{inner_digest}:{STORE_DIR}:{name}"
)
fingerprint_hash = hashlib.sha256(fingerprint.encode("utf-8")).digest()
fingerprint_digest = to_nix_base32(compress_hash(fingerprint_hash, 20))

I’m using bytes.maketrans because Python makes it easy enough to convert between normal-base32 and nix-base32 but you should also take a look a standalone implementation of nix-base32. For example, here is a link to Tvix’s.

This magic number that pops out of the correct fingerprint hashing method is the same as the one from Farid’s post! 5bkcqwq3qb6dxshcj44hr1jrf8k7qhxb

Now we can add that back into the JSON as an output path and an environment variable to finally get the same JSON blob as before:

{
  "name": "simple",
  "system": "x86_64-linux",
  "builder": "/bin/sh",
  "outputs": {
    "out": {
      "path": "/nix/store/5bkcqwq3qb6dxshcj44hr1jrf8k7qhxb-simple"
    }
  },
  "inputSrcs": [],
  "inputDrvs": {},
  "env": {
    "out": "/nix/store/5bkcqwq3qb6dxshcj44hr1jrf8k7qhxb-simple"
  },
  "args": [
    "-c",
    "echo 'hello world' > $out"
  ]
}

And, just as Farid promised, Nix accepts our derivation JSON and gives us a new ATerm:

$ nix --extra-experimental-features nix-command derivation add < simple.json
/nix/store/vh5zww1mqbcshfcblrw3y92v7kkzamfx-simple.drv
$

It’s same derivation Path that Farid has in his post, too.

But having a derivation in hand doesn’t mean anything other than we have—finally—written a correct recipe to build a thing. Let’s run it and see the output!

$ nix-store --realize /nix/store/vh5zww1mqbcshfcblrw3y92v7kkzamfx-simple.drv
...
/nix/store/5bkcqwq3qb6dxshcj44hr1jrf8k7qhxb-simple
$ cat /nix/store/5bkcqwq3qb6dxshcj44hr1jrf8k7qhxb-simple
hello world
$

I’m calling that success. We build a derivation by hand without any guess-and-check!

Check out my Python code if you like.

Parsing to IR and lvalues

Thu, 27 Mar 2025 00:00:00 +0000

I don’t want an AST in my hobby compiler. I’m not going to use it for analysis—it would just be a stepping stone representation on the way to the thing I really want, which is an SSA CFG. So I decided to parse straight from tokens into an SSA IR. I ran into some issues, fixed them, and would like to share the implementation. Maybe it’s a normal strategy, maybe it’s not.

Precedence climbing

I’ve always been a little nervous around parsers so I decided to lean into it and get familiar with precedence climbing. I’d already written something similar for a little math AST (update: see also this one) so I figured I would be able to reasonably easily port that to my new project.

I was mostly right. For example, here is the (fairly direct) Rust port that turns expressions into IR instead of AST. The only difference is that instead of pointers, nodes hold onto IDs, and they are linearized in a big Vec¹ (hence push_op and push_insn).

impl Parser<'_> {
    // Recursive parse function for expressions (statements handled elsewhere)
    fn parse_(&mut self, mut env: &mut Env, prec: u32)
        -> Result<InsnId, ParseError> {

        let mut lhs = match self.tokens.next() {
            None => return Err(ParseError::UnexpectedEof),
            Some(Token::Bool(value)) => {
                self.push_op(Opcode::Const(Value::Bool(value)))
            }
            // ...
            Some(Token::LParen) => {
                let result = self.parse_(&mut env, 0)?;
                self.expect(Token::RParen)?;
                result
            }
            Some(token) => return Err(ParseError::UnexpectedToken(token)),
        };
        while let Some(token) = self.tokens.peek() {
            let (assoc, op_prec) = match token {
                // ...
                Token::Plus => (Assoc::Any, 4),
                Token::Minus => (Assoc::Left, 4),
                Token::Star => (Assoc::Any, 5),
                // ...
                _ => break,
            };
            let token = token.clone();
            if op_prec < prec { return lhs; }
            self.tokens.next();
            let next_prec =
                if assoc == Assoc::Left { op_prec + 1 } else { op_prec };
            // ...
            let opcode = match token {
                // ...
                Token::Plus => Opcode::Add,
                Token::Minus => Opcode::Sub,
                Token::Star => Opcode::Mul,
                _ => panic!("Unexpected token {token:?}"),
            };
            let rhs = self.parse_(&mut env, next_prec)?;
            lhs = self.push_insn(opcode, smallvec![lhs, rhs]);
        }
        lhs
    }
}

This started mostly fine (parsing 1 + 2 to an AST is very similar to parsing to the IR) but I quickly ran into a problem: what about names?

See, evaluating a name like abc in the expression 1 + abc is fine if you have all of the environment plumbing: look it up in the env. In this compiler, that means find the stack slot that we’ve assigned for it in this scope and emit a load from that stack slot.

impl Parser<'_> {
    fn parse_(&mut self, mut env: &mut Env, prec: u32)
        -> Result<InsnId, ParseError> {

        let mut lhs = match self.tokens.next() {
            None => return Err(ParseError::UnexpectedEof),
            // ...
            Some(Token::Ident(name)) => {
                let name = self.prog.intern(&name);
                let slot = env.lookup(name)?;
                self.push_insn(Opcode::Load(slot), smallvec![self.frame()])
            }
            // ...
            Some(token) => return Err(ParseError::UnexpectedToken(token)),
        };
        // ...
    }
}

But assignments to names are also expressions! a = 5 is a perfectly valid expression and we should not evaluate a. In fact, we should find the stack slot of a, wait until it’s assigned to, and then generate a store (or if it’s not assigned to, emit the load).

Names and assignment

Here’s the subtle thing (at least for me): parsing to an AST doesn’t have this problem because it implies two things:

Recursively building up expressions without committing them immediately to a buffer of IR
Leaving names unresolved, as strings (imagine a Name "foo" AST node or something)

In IR land, we’re trying to eagerly emit IR and resolve names to offsets at the same time.

The way I solved this first was by doing some kind of lookahead in the Ident case for lhs. If the next token is an equals sign, parse right and emit an assignment. That worked fine for a bit, but has two problems:

It doesn’t handle precedence well—ideally we would have all the precedence logic be in the existing precedence climbing loop instead
It breaks down for more complicated assignments like assignments to attributes

For attribute assignments like a.b.c = 5, we do want to evaluate a.b, but we do not want to evaluate a.b.c. We want to wait until we either see an equals sign (and store to a.b.c) or see this expression get used elsewhere and then do the load then.

After talking with Andy Chu of Oils fame (whose name is the first that comes to mind when I think of precedence climbing) and some faffing about, I think I figured out a neat enough API for this.

LValues

The key is to be able to defer evaluation just a little longer until we know more about the context. To do that, we add an enum that can hold one of three things (or more, if you allow array assignment):

pub enum LValue {
    Insn(InsnId),
    Name(NameId),
    Attr(InsnId, NameId),
}

The Insn case represents an evaluated expression. It can be used for its value immediately.

The Name case represents an un-evaluated name for use in either loads or stores.

The Attr case represents an un-evaluated attribute lookup for use in either loads or stores. The left hand side (the receiver, if you will) has already been evaluated—it’s an InsnId, and therefore also committed to the IR buffer—but the right hand side has not.

This LValue enum will only be used temporarily in the lhs variable. We add two new special cases to handle it:

If the token from the precedence climbing is an equals sign, generate assignment code (Store or StoreAttr in this case)
If the token from the precedence climbing is a dot, generate an Attr LValue

If we ever want to return this LValue from parse_ or use it as part of a larger expression (say, lhs + rhs), we will need to convert it to an InsnId by evaluating it. That function is called lvalue_as_rvalue.

impl Parser<'_> {
    fn parse_(&mut self, mut env: &mut Env, prec: u32)
        -> Result<InsnId, ParseError> {

        let mut lhs = match self.tokens.next() {
            None => return Err(ParseError::UnexpectedEof),
            // ...
            Some(Token::Bool(value)) => {
                LValue::Insn(self.push_op(Opcode::Const(Value::Bool(value))))
            }
            Some(Token::Ident(name)) => {
                LValue::Name(self.prog.intern(&name))
            }
            // ...
            Some(token) => return Err(ParseError::UnexpectedToken(token)),
        };
        while let Some(token) = self.tokens.peek() {
            let (assoc, op_prec) = match token {
                Token::Equal => (Assoc::Right, 0),  // NEW!
                // ...
                Token::Plus => (Assoc::Any, 4),
                // ...
                _ => break,
            };
            let token = token.clone();
            if op_prec < prec { return self.lvalue_as_rvalue(env, lhs); }
            self.tokens.next();
            let next_prec =
                if assoc == Assoc::Left { op_prec + 1 } else { op_prec };
            // Special case: we need to look up the frame slot and write to it
            // or assign to the attribute
            if token == Token::Equal {  // NEW!
                lhs = match lhs {
                    LValue::Insn(..) =>
                        return Err(ParseError::CannotAssignTo(lhs)),
                    LValue::Name(name) => {
                        let rhs = self.parse_(&mut env, next_prec)?;
                        self.store_local(&mut env, name, rhs)?;
                        lhs
                    }
                    LValue::Attr(obj, name) => {
                        let rhs = self.parse_(&mut env, next_prec)?;
                        let obj = self.push_insn(Opcode::GuardInstance, smallvec![obj]);
                        self.push_insn(Opcode::StoreAttr(name), smallvec![obj, rhs]);
                        LValue::Insn(rhs)
                    }
                };
                continue;
            }
            // Special case: we need to hold off evaluating the attribute until
            // it is read elsewhere or written
            if token == Token::Dot {  // NEW!
                let name = self.expect_ident()?;
                let obj = self.lvalue_as_rvalue(env, lhs)?;
                lhs = LValue::Attr(obj, name);
                continue;
            }
            // Evaluate the lvalue because we want to use it in a binary
            // operation
            let mut lhs_value = self.lvalue_as_rvalue(env, lhs)?;  // NEW!
            // ...
            lhs = LValue::Insn(self.push_insn(opcode, smallvec![lhs_value, rhs]));
        }
        self.lvalue_as_rvalue(env, lhs)  // NEW!
    }

    fn lvalue_as_rvalue(&mut self, env: &Env, lvalue: LValue)
        -> Result<InsnId, ParseError> {

        match lvalue {
            LValue::Insn(insn_id) => Ok(insn_id),
            LValue::Name(name) => self.load_local(env, name),
            LValue::Attr(obj, name) => {
                let obj = self.push_insn(Opcode::GuardInstance, smallvec![obj]);
                Ok(self.push_insn(Opcode::LoadAttr(name), smallvec![obj]))
            }
        }
    }
}

This seems to work pretty well for all of my parser tests and also seems extensible to array assignments (if I add them in the future).

In conclusion, I feel a little less nervous around parsers now.

For more reading on this, check out Flattening ASTs by Adrian Sampson and this Reddit comment by Bob Nystrom of Crafting Interpreters fame. The comment is reproduced here:

(quoting the OP)

I can imagine, while parsing, whenever you created a new node, rather than allocating space just for it and giving the parent node the pointer, you could push the node onto a vector or some other such growable contiguous array and give the parent a reference to that spot.

Yes, that will definitely help! Once you do that, you may consider other stepwise refinements.
- Since subexpressions are executed first, it makes sense to have them earlier in the array than their parents. So instead of walking from the root to the leaves, walk the leaves first and then have the parents follow them.
- At that point, parents no longer need references to their children. Instead, you just need some convenient place to store the results of the child evaluations. Maybe a stack.
- Now your array of instructions doesn’t need any actual links between them. It’s just a flat list of which operations to perform.
Ta-da, you just reinvented stack-based bytecode. ↩