Ian Henry

Periodic Spaces

Sun, 30 Nov 2025 00:00:00 +0000

One of my favorite SDF techniques is domain repetition:

(def eye-center [35 82 62])
(def eyes (ball 39 | move eye-center | mirror x :r 10))
(def eye-angle [0 0 0])
(defn pupils [target]
(def left-pupil
(ball 15 | shade black | move [39 0 0]
| align x target
| move eye-center
))
(def right-pupil
(ball 15 | shade black | move [39 0 0]
| align x target
| move [-1 1 1 * eye-center]
))
(union left-pupil right-pupil))
(defn get-target [seed i]
(hash3 seed i | remap- + [0 0 2] | normalize))
(defn eye-target [seed]
(def base (t + hash 30 seed * hash 20 seed))
(def frame (floor base + 100))
(mix
(get-target seed frame)
(get-target seed (frame + 1))
(ss (fract base) 0.49 0.51)))
(def anim (osc t 7 | ss 0.2 0.8))
(union :r 50
(ball [100 150 100] | move [0 7 3])
| shade sky
| union (expand eyes 20 | move [0 1 -43]) :r 10
| subtract :r 10 eyes
| union
(eyes | shade white | union-color (pupils (eye-target $i)))
| union (box :r 17 [(ss p.y -44 44 17 33) 44 17]
| morph (ball [29 44 17])
| rotate x 2.92
| move [0 12 100]
| shade orange)
| tile: $i [(anim * 400) (anim * 500) (anim * 500)] :limit [(anim * 10000) 8 (anim * 10000)]
| union (ground -150 | shade white)
| scale 0.25
| scale (ss anim 0 1 1 0.2)
| rotate y 0.45 x -0.33 z 0.08)

It’s a great party trick that lets you render an infinite number of shapes in real-time, with soft shadows and ambient occlusion and all the other nice things that SDFs give you. It seemed like magic to me when I first saw it, and I suppose it still does – albeit in a different way.

The trick that makes this possible is that you aren’t evaluating “an infinite number of shapes.” As each ray marches through the scene, it only evaluates one shape at a time.

Let’s look at a 2D slice of that scene:

(def eye-center [35 82 62])
(def eyes (ball 39 | move eye-center | mirror x :r 10))
(def eye-angle [0 0 0])
(defn pupils [target]
(def left-pupil
(ball 15 | color black | move [39 0 0]
| align x target
| move eye-center
))
(def right-pupil
(ball 15 | color black | move [39 0 0]
| align x target
| move [-1 1 1 * eye-center]
))
(union left-pupil right-pupil))
(defn get-target [seed i]
(hash3 seed i | remap- + [0 0 2] | normalize))
(defn eye-target [seed]
(def base (t + hash 30 seed * hash 20 seed))
(def frame (floor base + 100))
(mix
(get-target seed frame)
(get-target seed (frame + 1))
(ss (fract base) 0.49 0.51)))
(def zed (osc t 5 10 25))
(union :r 50
(ball [100 150 100] | move [0 7 3])
| color sky
| union (expand eyes 20 | move [0 1 -43]) :r 10
| subtract :r 10 eyes
| union
(eyes | color white | union-color (pupils (eye-target $i)))
| union (box :r 17 [(ss p.y -44 44 17 33) 44 17]
| morph (ball [29 44 17])
| rotate x 2.92
| move [0 12 100]
| color orange)
| tile: $i [300 300 300]
| scale 0.25
| slice z zed)

Actually, that’s freaking me out. Let’s look at a simpler 2D scene, and you can trust me that this generalizes to more complex shapes:

(star 20 10 :r 1
| color (hsv (hash $i) 0.5 1)
| tile: $i [80 80])

Pretend that that’s a 2D slice of a 3D scene. Like, imagine that we’re actually trying to render this:

(star 20 10 :r 1
| shade (hsv (hash $i) 0.5 1)
| tile: $i [80 80]
| extrude z 20)

In order to render a 3D scene, we shoot rays out from the camera at different angles until they hit something. It’s just much easier to visualize this process in 2D:

(def scene
(star 20 10 :r 1
| color (hsv (hash $i) 0.5 1)
| tile: $i [80 80]))
(def line-start [-200 28])
(def line-end [68 64])
(def progress (t / 3 | fract | ss 0 0.5))
(def indicator (union
(line line-start (mix line-start line-end progress) 2)
(circle (mix -5 5 (ss progress 0.95 1)) | move line-end)))
(union
scene
(indicator | expand 1 | color black)
(indicator | color white))

But when we’re rendering SDFs, we don’t cast rays continuously like this – they don’t smoothly advance until they hit a shape. Since we know the distance field at every point, we use that information to decide how far to advance. Actual ray marching looks more like this:

(def scene
(star 20 10 :r 1
| color (hsv (hash $i) 0.5 1)
| tile: $i [80 80] :limit 5))
(def direction [1 0.13 | normalize])
(def line-start [-200 28])
(gl/defn :vec3 march [:int iterations]
(var at line-start)
(var dist 0)
(for (var i 0:s) (<= i iterations) (++ i)
(+= at (dist * direction))
(set dist ,(gl/with [q at]
(shape/distance scene))))
(return [dist at]))
(def iterations 15)
(def line-end (. (march (int iterations)) yz))
(def frame-base (fract (t / 15) * iterations))
(def frame (frame-base | floor | int))
(def inframe (frame-base | fract | ss 0.5 0.6))
(gl/def highlight-circle (march frame))
(def i-dist highlight-circle.x)
(def i-progress highlight-circle.yz)
(defn ray-point [progress]
(line-end - line-start * progress + line-start))
(def next-point (i-dist * direction + i-progress))
(def indicator (union
(line line-start (mix i-progress next-point inframe) 2)
(circle i-dist | shell 2 | move i-progress)
))
(union
scene
(indicator | expand 1 | color black)
(indicator | color white))

…except, you know, faster.

The radius of that circle is the value of the distance field at each point of the ray’s journey. The distance value means “what’s the distance to the nearest shape,” and during ray marching you can interpret that as “how far can I move in any direction before I hit something.” When you’re ray marching in 3D, it gives you the radius of the largest empty sphere of space around the current position of the ray.

The trick that makes domain repetition possible is that, in order to compute the distance field of an infinite number of shapes, you actually only look at one shape at a time. It’s not actually “the distance to the nearest shape.” It’s “the distance to the shape in the same ‘cell’ as me.”

That is, when we repeat this, we’re really breaking it up into cells, and only checking “the current cell:”

(def scene
(star 20 10 :r 1
| color (hsv (hash $i) 0.5 1)
| tile: $i [80 80] :limit 5))
(def direction [1 0.13 | normalize])
(def line-start [-200 28])
(gl/defn :vec3 march [:int iterations]
(var at line-start)
(var dist 0)
(for (var i 0:s) (<= i iterations) (++ i)
(+= at (dist * direction))
(set dist ,(gl/with [q at]
(shape/distance scene))))
(return [dist at]))
(def iterations 15)
(def line-end (. (march (int iterations)) yz))
(def frame-base (fract (t / 15) * iterations))
(def frame (frame-base | floor | int))
(def inframe (frame-base | fract | ss 0.5 0.6))
(gl/def highlight-circle (march frame))
(def dist highlight-circle.x)
(def ray-at highlight-circle.yz)
(defn ray-point [progress]
(line-end - line-start * progress + line-start))
(def next-point (dist * direction + ray-at))
(def indicator (union
(line line-start (mix ray-at next-point inframe) 2)
(circle dist | shell 2 | move ray-at)
))
(def ray-cell (ray-at / 80 | round))
(def scene
(star 20 10 :r 1
| color (hsv (hash $i) 0.5 (gl/if (= ray-cell $i) 1 0.25))
| tile: $i [80 80] :limit 5))
(union
(rect 40 | shell 1 | color white | tile: $i [80 80] :limit 5)
scene
(indicator | expand 1 | color black)
(indicator | color white))

And this works. For fairly regular, symmetric shapes, this works.

But part of the fun of domain repetition is that you don’t have to use regular shapes. You can do this with arbitrarily asymmetric shapes, or shapes that vary depending on the cell they’re in.

In fact I’ve been doing that in all of these examples: the stars are colored differently based on their cell coordinates. But we can vary their shape as well:

(defn make-scene [f]
(star r (r / 2) :r 1
| gl/let [r (10 * hash $i 10 + 10)] _
| rotate (hash $i 130 | remap-)
| move (hash2 $i 140 | remap- * 20)
| f $i
| tile: $i [80 80] :limit 5))
(def scene (make-scene (fn [$ $i] $)))
(def direction [1 0.13 | normalize])
(def line-start [-200 28])
(gl/defn :vec3 march [:int iterations]
(var at line-start)
(var dist 0)
(for (var i 0:s) (<= i iterations) (++ i)
(+= at (dist * direction))
(set dist ,(gl/with [q at]
(shape/distance scene))))
(return [dist at]))
(def iterations 15)
(def line-end (. (march (int iterations)) yz))
(def frame-base (fract (t / 15) * iterations))
(def frame (frame-base | floor | int))
(def inframe (frame-base | fract | ss 0.5 0.6))
(gl/def highlight-circle (march frame))
(def dist highlight-circle.x)
(def ray-at highlight-circle.yz)
(defn ray-point [progress]
(line-end - line-start * progress + line-start))
(def next-point (dist * direction + ray-at))
(def indicator (union
(line line-start (mix ray-at next-point inframe) 2)
(circle (abs dist) | shell 2 | move ray-at)
))
(def ray-cell (ray-at / 80 | round))
(def scene (make-scene (fn [$ $i]
($ | color (hsv (hash $i) 0.5 (gl/if (= ray-cell $i) 1 0.25))))))
(union
(rect 40 | shell 1 | color white | tile: $i [80 80] :limit 5)
scene
(indicator | expand 1 | color black)
(indicator | color white))

Hey look! We have a problem. If you watch that march to its completion, you’ll see that the ray actually overshoots, and winds up in the middle of a star. Thanks to the power of signed distance functions, it’s able to realize that it’s inside a shape (the distance field is negative) and the ray marcher backs out until it finds the edge.

This demonstrates a neat fact about signed distance functions, but the real point is that our magical approximation of infinite distance isn’t so magical after all. We won’t always be able to back out like this: with a little worse luck, a bad distance field could cause us to overshoot completely.

We’ll get obvious visual artifacts when we use this technique to render a 3D scene:

(star 20 10 :r 1 | extrude z 3
| rotate (hash3 $i 10 | normalize) t
| move (hash3 $i 20 | remap- * 10)
| shade (hsv (hash $i) 0.6 1)
| tile: $i [50 50 50] :limit 5)

As only some pixels overshoot their destinations.

And we can mitigate this by evaluating not just the current cell, but the current cell and its immediate neighbors. Your distance function gets more expensive – you’re evaluating your shape N times for each march now – but it’s a small price to pay for infinity.

Here’s the exact same scene, but sampling the eight nearest shapes instead of the nearest one:

(star 20 10 :r 1 | extrude z 3
| rotate (hash3 $i 10 | normalize) t
| move (hash3 $i 20 | remap- * 10)
| shade (hsv (hash $i) 0.6 1)
| tile: $i [50 50 50] :limit 5 :oversample true)

The artifacts are gone, but my laptop is hot now.

Okay. So this is classic domain repetition: it’s a discrete operation. There is a concept of a discrete “cell” or “tile,” with integer coordinates, and evaluation is based around multiple instances of these discrete cells.

Great.

Background information: explained.

Now we can start the blog post.

Periodic Spaces

Here’s a star:

(star 50 25 :r 3 | rotate (t / 2) | color sky)

There is only one star there. It is a normal star.

Here is a plot of the x coordinate of our input (on the x axis) versus the x coordinate that we render at (on the y axis).

(union
(star 50 25 :r 3 | rotate (t / 2) | color sky)
(circle 1 | move [q.x q.x] | color yellow))

It is the identity function, because we aren’t doing anything weird yet.

Let’s do something weird.

For each pixel we’re rendering, instead of using the color of the star at that point, we can use the color of the star at a different point. A trivial example is to just scale the x coordinate:

(defn with-x [expr]
(gl/with [q [expr q.y]]
(union
(star 50 25 :r 3 | rotate (t / 2) | color sky)
(circle 1 | move [q.x q.x] | color yellow))))
(with-x (q.x / 2))

The effect that we see is a stretching effect: it looks like we scaled the star. But we didn’t really: we scaled the image of the star that we rendered. This is like if we put a star on a scanner and slid it across the glass as it scanned. We would get an image that appeared stretched – just like this.

Many SDF combinators work this way. If you want to move the star ten pixels to the right, you actually evaluate the star at a point ten pixels to the left.

(defn with-x [expr]
(gl/with [q [expr q.y]]
(union
(star 50 25 :r 3 | rotate (t / 2) | color sky)
(circle 1 | move [q.x q.x] | color yellow))))
(with-x (q.x + oss t 5 -50 50))

But notice: we can do anything we want to the input. It’s just an expression, and we can write whatever expression we want.

We could evaluate at a random x coordinate:

(defn with-x [expr]
(gl/with [q [expr q.y]]
(union
(star 50 25 :r 3 | rotate (t / 2) | color sky)
(circle 1 | move [q.x q.x] | color yellow))))
(with-x (q.x | hash | remap- * 100))

We could evaluate with a smoothly-varying, semi-random offset:

(defn with-x [expr]
(gl/with [q [expr q.y]]
(union
(star 50 25 :r 3 | rotate (t / 2) | color sky)
(circle 1 | move [q.x q.x] | color yellow))))
(with-x (perlin [q.x 0] 20 * 20 + q.x))

We can even go nuts and make x a function of x and y:

(defn with-x [expr]
(gl/with [q [expr q.y]]
(union
(star 50 25 :r 3 | rotate (t / 2) | color sky)
(circle 1 | move [q.x q.x] | color yellow))))
(with-x (perlin q [30 100] * 100))

Or we could get back to the point of this blog post, and evaluate x with a periodic function. What happens if we take x % 100?

(defn with-x [expr]
(gl/with [q [expr q.y]]
(union
(star 50 25 :r 3 | rotate (t / 2) | color sky)
(circle 1 | move [q.x q.x] | color yellow))))
(with-x (mod q.x 100))

Er, right. That doesn’t look very good, because the star is centered at zero. Let’s try a variant of modulo that centers its output around zero:

(defn with-x [expr]
(gl/with [q [expr q.y]]
(union
(star 50 25 :r 3 | rotate (t / 2) | color sky)
(circle 1 | move [q.x q.x] | color yellow))))
(defn mod- [a b] (a + (b / 2) | mod b - (b / 2)))
(with-x (mod- q.x 100))

Aha! We have re-created the tile function that we were using to repeat space at the beginning of this blog post. And the plot of our x coordinate now looks like a sawtooth wave. It’s discontinuous – as soon as we reach the end of one period, we jump right back to the next period.

This sharp discontinuity doesn’t matter in 2D. But let’s take a look at this exact same scene extruded into 3D:

(defn with-x [expr]
(gl/with [q [expr q.y]]
(star 50 25 :r 3 | rotate (t / 2) | shade sky)))
(defn mod- [a b] (a + (b / 2) | mod b - (b / 2)))
(with-x (mod- q.x 100) | extrude z 3)

There are visual artifacts around the tips of the stars: sometimes the ray overshoots, because this repetition does not produce a correct distance field. (You can also click the magnet icon in the top right of the image for an alternate visualization that will make these artifacts stand out more clearly.)

We can see that the distance field is not very good by looking at the gradient of the distance field for this scene:

(defn with-x [expr]
(gl/with [q [expr q.y]]
(union
(star 50 25 :r 3 | rotate (t / 2)))))
(defn mod- [a b] (a + (b / 2) | mod b - (b / 2)))
(with-x (mod- q.x 100))

Notice how, at the sharp lines dividing each period of the wave, the contour lines don’t match up (it’s easier to see if you click to pause the rotation). If we use Bauble’s built-in multisampled repetition operator, we can see what the distance field should look like:

(star 50 25 :r 3 | rotate (t / 2) | tile [100 0] :oversample true)

But our sawtooth-wave modulo-repetition doesn’t look like that. We don’t see the complex distance field between two stars, because we never evaluate two stars. We just evaluate our one lonely star at different coordinates.

So this sawtooth wave is not great – although this is an interesting way to interpret the (naive) tiling operator.

Let’s try another periodic function. What if we use a triangle wave?

(defn with-x [expr]
(gl/with [q [expr q.y]]
(union
(star 50 25 :r 3 | rotate (t / 2) | color sky)
(circle 1 | move [q.x q.x] | color yellow))))
(gl/defn :float triangle-wave [:float x :float period]
(var section (x / period))
(var dir (round section | mod 2 | remap- | -))
(return (fract (section + 0.5) - 0.5 * period * dir)))
(with-x (triangle-wave q.x 100))

An intuitive explanation of this is: when you reach the end of the scanner, instead of starting over, reverse and scan backwards. So we “scan” left and right over the star, and the effect is that every other star is mirrored.

And, surprisingly to me, this actually does produce a correct distance field:

(defn with-x [expr]
(gl/with [q [expr q.y]]
(star 50 25 :r 3 | rotate (t / 2))))
(gl/defn :float triangle-wave [:float x :float period]
(var section (x / period))
(var dir (round section | mod 2 | remap- | -))
(return (fract (section + 0.5) - 0.5 * period * dir)))
(with-x (triangle-wave q.x 100))

Notice how, whenever the period changes, the contour lines perfectly match up. This means that we really can ray march this perfectly in 3D, with no visual artifacts and no extra evaluation:

(defn with-x [expr]
(gl/with [q [expr q.y]]
(star 50 25 :r 3 | rotate (t / 2) | shade sky)))
(gl/defn :float triangle-wave [:float x :float period]
(var section (x / period))
(var dir (round section | mod 2 | remap- | -))
(return (fract (section + 0.5) - 0.5 * period * dir)))
(with-x (triangle-wave q.x 100) | extrude z 3)

Of course it’s not the same scene that we were rendering before – half the stars are flipped – but still. If you’re tiling in all three dimensions, this is an 8x speedup over multisampled evaluation.

Now, this effect is well-known, and you can apply this to classic instanced repetition as well. If you want to repeat an asymmetric shape, you can make it the whole distance field symmetric by mirroring every other instance. But it didn’t really click for me until I saw it as a periodic function of space like this.

What about a sine wave?

(defn with-x [expr]
(gl/with [q [expr q.y]]
(union
(star 50 25 :r 3 | rotate (t / 2) | color sky)
(circle 1 | move [q.x q.x] | color yellow))))
(gl/defn :float sine-wave [:float x :float period]
(return (sin (x * pi / period) * 0.5 * period)))
(with-x (sine-wave q.x 100))

This one is pretty weird. It’s not a correct distance field: the contour lines are not equal distances apart, because space is stretched nonlinearly near the edges of each tile.

(defn with-x [expr]
(gl/with [q [expr q.y]]
(star 50 25 :r 3 | rotate (t / 2))))
(gl/defn :float sine-wave [:float x :float period]
(return (sin (x * pi / period) * 0.5 * period)))
(with-x (sine-wave q.x 100))

Still, we can ray march it with minimal artifacts:

(defn with-x [expr]
(gl/with [q [expr q.y]]
(star 50 25 :r 3 | rotate (t / 2) | shade sky)))
(gl/defn :float sine-wave [:float x :float period]
(return (sin (x * pi / period) * 0.5 * period)))
(with-x (sine-wave q.x 100) | extrude z 3)

It’s nice to be able to achieve a “smooth tiling” effect like this. As we vary the period, the shapes smoothly join together instead of just getting chopped off:

(defn with-x [expr]
(gl/with [q [expr q.y]]
(star 50 25 :r 3 | rotate (t / 2) | shade sky)))
(gl/defn :float sine-wave [:float x :float period]
(return (sin (x * pi / period) * 0.5 * period)))
(with-x (sine-wave q.x (osc t 5 100 50)) | extrude z 3)

It’s a neat effect in full 3D, too:

(defn with-point [expr]
(gl/with [p expr]
(box 25 | rotate x (t / 2) z (t / 3) y (t / 5) | shade sky)))
(gl/defn :float sine-wave [:float x :float period]
(return (sin (x * pi / period) * 0.5 * period)))
(defn limit [f x period instances]
(gl/if (> (abs x) (period * instances))
(x - (period * instances * sign x))
(f x period)
))
(with-point
[(limit sine-wave p.x (osc t 3 50 75) 2)
(limit sine-wave p.y (osc t 3 50 75) 2)
(limit sine-wave p.z (osc t 3 50 75) 2)])

You can also use this trick with radial symmetry. Bauble’s built-in radial symmetry operation uses the same discrete approach as its tile operation:

(octahedron 100 :r 5
| rotate x (t / 2) z (t / 3)
| radial y 12 80
| shade sky)

Artifact city, and sharp lines at the edge of each period.

But if we write our own radial symmetry, we can swap that sawtooth for a sine wave:

(gl/defn :float sine-wave [:float x :float period]
(return (sin (x * pi / period) * 0.5 * period)))
(gl/def angle (sine-wave (atan2+ p.xz) (tau / 12)))
(gl/def dist (length p.xz))
(octahedron 100 :r 5
| rotate x (t / 2) z (t / 3)
| move x 80
| gl/with [p [(cos angle * dist) p.y (sin angle * dist)]] _
| shade sky)

Kind of interesting.

This is a useful spatial distortion, and it’s fun to explore periodic spaces like this. I think there’s some value especially in a cheap approximation of smoothly merged repeated surfaces: that’s another effect that you can achieve with multisampled repetition – you don’t have to take the min of all adjacent cells; you can apply a smooth union operation – but the speedup you get by just throwing a sine wave at it is significant.

But this technique is less powerful than classic instanced domain repetition. For one thing, we’re always repeating the same shape. And while we could invent our own notion of “cell coordinates” based on which period we’re in, and still vary the shape as we scanned it with a triangle wave, by doing so we would lose the symmetry that made the triangle wave appealing in the first place.

There’s another limitation that hasn’t mattered in any of our examples yet: using classic instanced repetition, you have the choice to sample more than just the immediately adjacent cells. This allows you to repeat shapes that aren’t bounded by a single cell.

For example, here’s a triangle radially tiled around the origin:

(triangle [25 100]
| color (hsv (hash $i) 0.5 1)
| rotate (-pi/2 + t)
| radial: $i 10 50)

Because we’re only evaluating one “slice” of space, the triangle is clipped where it crosses the boundary between two cells. By evaluating the nearest adjacent slice as well, we can improve this:

(triangle [25 100]
| color (hsv (hash $i) 0.5 1)
| rotate (-pi/2 + t)
| radial: $i 10 50 :oversample true
:sample-from 0
:sample-to 1)

But it’s still clipped. Let’s evaluate both adjacent slices:

(triangle [25 100]
| color (hsv (hash $i) 0.5 1)
| rotate (-pi/2 + t)
| radial: $i 10 50 :oversample true
:sample-from -1
:sample-to 1)

Better! But still not perfect. Let’s evaluate both adjacent slices, and the adjacent slices next to them:

(triangle [25 100]
| color (hsv (hash $i) 0.5 1)
| rotate (-pi/2 + t)
| radial: $i 10 50 :oversample true
:sample-from -2
:sample-to 2)

That’s not something you can do with a pure periodic function of space: there’s no way to have a slice that contains five different triangles.

But even so… what other periodic functions should we try?

Generalized Worley Noise

Wed, 26 Nov 2025 00:00:00 +0000

Worley noise is a type of noise used for procedural texturing in computer graphics. In its most basic form, it looks like this:

(color r2 (worley q 50 :oversample true))

That’s ugly and boring, but it’s a quick way to see what the effect looks like. If we use Worley noise to distort a 3D shape, we can get something like a hammered or cratered texture:

(def s (osc t 5 | ss 0.2 0.8 * 30 + 10))
(ball 100
| expound (worley p s) (sqrt s)
| slow 0.8
| shade sky
| rotate y (t / 10))

Like many procedural textures, it looks a lot better if you repeat the effect a few times with different frequencies:

(def s (osc t 5 | ss 0.2 0.8 * 30 + 10))
(ball 100
| expound (fbm 3 worley p s) (sqrt s)
| slow 0.8
| shade sky
| rotate y (t / 10))

There are some visual artifacts in these renderings, because they’re using a fast approximation of Worley noise that gives the wrong answer for some values.

That’s not very satisfying, but in order to explain where these artifacts are coming from, we first have to talk about how Worley noise works.

It’s pretty simple: you start with a grid of points.

(circle 0
| color (hsv (hash $i) 0.5 1)
| tile: $i [30 30]
| expand 3)

Then you move each point by some random offset:

(def animate (osc t 5 | ss 0.1 0.7))
(circle 0
| color (hsv (hash $i) 0.5 1)
| move (hash2 $i | remap- * animate * 10)
| tile: $i [30 30] :oversample true
| expand 3)

When you’re writing a shader, you can’t actually generate random numbers, so we’re using a hash function to produce random-looking offsets based on the logical position of each point (that is, $i = [0 0] for the center point, $i = [1 0] for the point to the right of that, etc).

Finally, once you have the points at random-looking positions, you compute the distance to the nearest point for every individual pixel in your input – and that’s Worley noise.

(def animate (osc t 5 | ss 0.1 0.7))
(def points
(circle 0
| color (hsv (hash $i) 0.5 1)
| move (hash2 $i | remap- * animate * 10)
| tile: $i [30 30] :oversample true
))
(set background-color
(vec3 (shape/distance points / 30)))
(expand points 3)

How do you compute the distance to the nearest point for any pixel you ask about? It’s actually pretty simple: you know that you started with a perfectly even square grid. For any pixel, you can compute the “grid cell” that that pixel falls into ([0 0], [0 1], etc). It’s just the pixel divided by the grid size, rounded to the nearest integer.

(def animate (osc t 5 | ss 0.1 0.7))
(union
(color r2 (hsv (hash $i) 0.5 1))
(circle 3 | color (hsv (hash $i) 0.5 0.1))
| move (hash2 $i | remap- * animate * 10)
| tile: $i [30 30] :oversample true :sample-from -1
)

And you know that the nearest point is either in this grid, or it’s in one of the immediately adjacent grids, because we only offset our points by at most half the grid size, so each randomly distributed point is still inside its original grid cell. Which means there’s no point inside any other cell that could be nearer than any point in one of the adjacent cells.

(def animate (osc t 5 | ss 0.1 0.7))
(def query-point (rotate [(osc t 5 50 100) 0] (t / 2)))
(def query-cell (query-point / 30 - 1 | round))
(union
(color r2 (hsv (hash $i) 0.5
(gl/if (<= (max ($i - query-cell | abs)) 1) 1 0.25)))
(circle 3 | color (hsv (hash $i) 0.5 0.1))
| move (hash2 $i | remap- * animate * 10)
| tile: $i [30 30] :oversample true :sample-from -1
| union
(union (circle 4 | color black) (circle 3 | color white)
| move query-point))

So that leaves you nine points to check, for every single pixel in your shader. Here’s the optimization that’s causing visual artifacts: instead of checking all eight adjacent points, only check the three nearest points. The nearest point to your sample position is probably in one of those cells, but it doesn’t have to be, so whenever you get unlucky you wind up with some visual artifacts.

(def animate (osc t 5 | ss 0.1 0.7))
(def query-point (rotate [(osc t 5 50 100) 0] (t / 2)))
(def query-cell (query-point / 30 - 1 | round))
(def query-bias (query-point / 30 | fract | round * 2 - 1 -))
(defn all [bvec] (and bvec.x bvec.y))
(defn or-bvec [a b] [(or a.x b.x) (or a.y b.y)])
(union
(color r2 (hsv (hash $i) 0.5
(gl/let [offset ($i - query-cell)]
(gl/if (and (<= (max (abs offset)) 1)
(all
(or-bvec (equal offset query-bias)
(equal offset [0 0]))))
1 0.25))))
(circle 3 | color (hsv (hash $i) 0.5 0.1))
| move (hash2 $i | remap- * animate * 10)
| tile: $i [30 30] :oversample true :sample-from -1
| union
(union (circle 4 | color black) (circle 3 | color white)
| move query-point))
# ahh that took so long

9 points to 4 points is a nice improvement in 2D, but in 3D this optimization takes you from 27 points to 8 points, which can be the difference between realtime and offline rendering.

But notice: this is getting a little bit complicated. And the original code snippet I showed you wasn’t very complicated at all:

(def animate (osc t 5 | ss 0.1 0.7))
(def points
(circle 0
| color (hsv (hash $i) 0.5 1)
| move (hash2 $i | remap- * animate * 10)
| tile: $i [30 30] :oversample true
))
(set background-color
(vec3 (shape/distance points / 30)))
(expand points 3)

Nowhere does that code compute cell coordinates or check for the nearest point. I just constructed this thing, said shape/distance, and somehow that just… gave me the distance to the nearest point.

I was able to do that because Bauble is a playground for making 3D graphics with signed distance functions. Bauble’s whole deal is computing distances to things! And Worley noise is just the signed distance function of a bunch of randomly distributed points. I’m used to thinking of signed distance functions as defining implicit surfaces of 3D shapes, but Worley noise uses the distance as a scalar in its own right.

So.

This is interesting.

What if… we took other signed distance functions, and used them as procedural noise distortions?

We’ll start simple. Instead of points, what if we randomly distribute a bunch of squares?

(def animate (osc t 5 | ss 0.1 0.7))
(def size 60)
(def points
(rect size
| color (hsv (hash $i) 0.5 1)
| move (hash2 $i | remap- * animate * 10)
| tile: $i (0.5 * size | vec2) :oversample true
))
(set background-color
(vec3 (shape/distance points + size / (0.5 * size))))
(expand points (- 3 size))

It’s not obvious that that will be interesting. Let’s look at it in action:

(defn squarley [input &opt period]
(default period 1)
(gl/with [q (input / period)]
(rect 1
| color (hsv (hash $i) 0.5 1)
| move (hash2 $i | remap- * 0.33)
| tile: $i [1 1] :oversample true
| shape/distance)))
(ball 100 | expound (squarley p.xy 20) 20
| rotate y (t / 10))

Since we only defined this noise function in 2D, we need a two-dimensional input. That’s a pretty boring 2D input. This is a little more interesting:

(defn squarley [input &opt period]
(default period 1)
(gl/with [q (input / period)]
(rect 1
| color (hsv (hash $i) 0.5 1)
| move (hash2 $i | remap- * 0.33)
| tile: $i [1 1] :oversample true
| shape/distance)))
(ball 100
| expound (squarley
[(atan2 p.xy | abs) (sqrt (150 - p.z | abs))]
[.5 1]) 20
| rotate y (t / 10))

We can apply multiple octaves of this, to get… something.

(defn squarley [input &opt period]
(default period 1)
(gl/with [q (input / period)]
(rect 1
| color (hsv (hash $i) 0.5 1)
| move (hash2 $i | remap- * 0.33)
| tile: $i [1 1] :oversample true
| shape/distance)))
(ball 100
| expound (fbm 3 squarley
[(atan2 p.xy | abs) (sqrt (150 - p.z | abs))]
[.5 1]) 20
| rotate y (t / 10)
)

But so far this is not a very interesting effect. What if we vary the orientation as well?

(def animate (osc t 5 | ss 0.1 0.7))
(def size 60)
(def points
(rect size
| color (hsv (hash $i) 0.5 1)
| rotate (hash $i 1000 * pi/2 * animate)
| move (hash2 $i | remap- * animate * 10)
| tile: $i (0.5 * size | vec2) :oversample true
))
(set background-color
(vec3 (shape/distance points + size / (0.5 * size))))
(expand points (- 3 size))

It’s a little bit more random-looking, I guess:

(defn squarley [input &opt period]
(default period 1)
(gl/with [q (input / period)]
(rect 1
| rotate (hash $i 1000 * pi/2)
| color (hsv (hash $i) 0.5 1)
| move (hash2 $i | remap- * 0.33)
| tile: $i [1 1] :oversample true
| shape/distance)))
(ball 100
| expound (fbm 3 squarley
[(atan2 p.xy | abs) (sqrt (150 - p.z | abs))]
[.5 1]) 20
| rotate y (t / 10)
)

But distorting 3D space with 2D noise is not… it doesn’t look great.

Let’s jump to 3D.

(def animate (osc t 5 | ss 0.1 0.7))
(def size 60)
(def points
(box size
| shade (hsv (hash $i) 0.5 1)
| rotate (hash3 $i 2000) (hash $i 1000 * pi/2 * animate)
| move (hash3 $i | remap- * animate * 10)
| tile: $i (0.5 * size | vec3) :limit 5 :oversample true
))
(union
(plane (- ray.direction)
| color (shape/distance points + size / (0.5 * size)))
(expand points (- 3 size)) | scale 2)

It’s a lot harder to visualize the distance field in 3D. What you’re seeing there is the distance field at the plane that passes through the origin and faces towards the camera – you can click and drag the camera around to take different slices of space. I know it’s not a great visualization, but the point is that this technique generalizes to 3D (even if it’s hard to imagine the distance field at every point in 3D space).

Let’s see how this looks when we use it to distort a 3D shape:

(defn cubeley [input &opt period]
(default period 1)
(gl/with [p (input / period)]
(box 1
| rotate (hash3 $i 2000) (hash $i 1000 * pi/2)
| move (hash3 $i | remap- * 0.33)
| tile: $i [1 1 1] :oversample true
| shape/distance)))
(torus y 100 50
| expound (cubeley p 30) 20
| rotate y (t / 10))

It’s kind of interesting? Definitely better than what we had before. Sort of a faceted gemstone effect.

Do you think our computers will catch on fire if we try multiple octaves of this?

(defn cubeley [input &opt period]
(default period 1)
(gl/with [p (input / period)]
(box 1
| rotate (hash3 $i 2000) (hash $i 1000 * pi/2)
| move (hash3 $i | remap- * 0.33)
| tile: $i [1 1 1] :oversample true
| shape/distance)))
(torus y 100 50
| expound (fbm 3 cubeley p 30) 20
| rotate y (t / 10))

I’m glad you’re still with me.

Let’s trade the boxes for cones:

(defn coneley [input &opt period]
(default period 1)
(gl/with [p (input / period)]
(cone y 1 1
| move (hash3 $i | remap- * 0.33)
| tile: $i [1 1 1] :oversample true
| shape/distance)))
(ball [100 150 100]
| expound (coneley p 30) 20
| rotate y (t / 10)
)

It’s kind of an interesting pinecone-y texture? I guess?

There are more primitives to try. But of course we don’t have to limit ourselves to primitive shapes.

This is a classic SDF example to demonstrate how easy it is to do constructive solid geometry stuff:

(box 100 :r 10
| subtract :r 10 (sphere 120))

What if… we used that as the basis for our Worley noise?

(def jitter (osc t 5 | ss 0.1 0.9))
(defn cubeley [input &opt period]
(default period 1)
(gl/with [p (input / period)]
(box 1 :r 0.1
| subtract :r 0.1 (sphere 1.2)
| move (hash3 $i | remap- * 0.33 * jitter)
| tile: $i [1 1 1] :oversample true
| shape/distance)))
(torus y 100 50
| expound (cubeley p 30) 20
| rotate y (t / 10))

I think it’s kind of more interesting without the randomization.

We’ve constructed an interesting 3D noise function, and we’re using it to distort 3D space. But of course, we can go back to considering this a “noise texture” in the original sense of the word:

(def jitter (osc t 5 | ss 0.1 0.9))
(defn cubeley [input &opt period]
(default period 1)
(gl/with [p (input / period)]
(box 1 :r 0.1
| subtract :r 0.1 (sphere 1.2)
| rotate (hash3 $i 1000) (hash $i 2000 * jitter)
| move (hash3 $i | remap- * 0.33 * jitter)
| tile: $i [1 1 1] :oversample true :sample-from -2 :sample-to 2
| shape/distance)))
(r2 | color (vec3 (fbm 4 cubeley [q 0] 128 | abs)))

Kinda neat.

The point of all of this is: Worley noise invites us to reconsider signed distance functions as more than implicit surfaces. And since Bauble makes it easy to construct signed distance functions, it’s a good playground for experimenting with textures like this.

Even if we never found anything particularly attractive, it’s fun to play around with space.

If this is your first time seeing Bauble, hey welcome! This post is an expansion of something I briefly talked about in a YouTube video once. The video has many more examples of the sorts of things that Bauble can do. Check it out if this piqued your interest!

Building Bauble

Fri, 10 Jan 2025 00:00:00 +0000

I made something that I think is pretty neat, and I want to tell you about it.

This is a little hot air balloon made out of alternating layers of brass and bronze that stack together with these angled facets:

It’s 3D printed, sort of, but it really is solid metal – it’s not a metallic filament. It’s made by “lost wax casting,” where you 3D print a model out of resin, then pack it in plaster, and then once the plaster dries you melt out the resin and fill the void with molten–

You know what? This is neat, but this actually isn’t what I wanted to tell you about.

Get cute with me

Neither is that, but we’re getting closer.

That’s the 3D model that the balloon is cast from. I didn’t actually cast the balloon – I paid someone else to do that for me – but I did make the 3D model.

And it’s an interesting 3D model. It’s not a triangle mesh, like most 3D shapes you encounter. It has no faces; it has no vertices. Instead, it’s made entirely out of math: this balloon is a pure function of 3D space.

Here, take a look:

(def thickness 25)
(def sections 12)
(def angle (pi * 0.25))
(def lobe-intensity 1)
(def bezel 1)
(def bronziness 1.5)
(def branzino false)
(ball [(100 / (lobe-intensity + 1)) 100 100]
| union :r 50 (cylinder y 25 50 | move [0 -100 0])
| scale y (ss p.y -100 100 1 0.8)
| intersect :r bezel
(plane y | shell thickness
| color (gl/if (mod $i.y 2 | = 0) (pow default-3d-color bronziness) default-3d-color)
| tile: $i [0 thickness 0]
| rotate z (remap- parity * angle)
| gl/let [parity (mod $i 2)] _)
| radial: $i y sections
| move y 40
)

There’s the source code to that hot air balloon. Mess around with it. Edit some constants. Pull up the autocomplete with ctrl-space, and see what else it can do.

This is called Bauble, and this is what I wanted to tell you about.

Bauble is a tool – toy? – that I wrote in 2022, because I wanted to make pictures with math on my computer. And not just simple geometrical things like that. I wanted to make pictures like this:

(defn fork [shape f1 f2]
(union (f1 shape) (f2 shape)))
(defn spoon [shape f1 f2]
(union shape (f1 shape) (f2 shape)))
(defn cel [shape rgb]
(shade shape :f (fn [light] (gl/do
(var cel-shadow (step 0.8 light.brightness + 1 / 2))
(var cel-shading (dot light.direction normal * light.color | quantize 2 * cel-shadow))
(var regular-shading (dot light.direction normal * light.color * cel-shadow))
(var b (mix cel-shading regular-shading 0.5 + 0.5))
(b * rgb)))))
(setdyn *lights* [(light/directional 1 [-2 -2 -1] 1024 :shadow 0.25)])
(def ear
(cone y 40 153 :r 12
| morph 0.15 (sphere 46 | move y 64)
| union :r 13
(cylinder y 26 30 | move y -10)
| scale z 0.5))
(def ears
(ear
| move x 134
| rotate z (tau * -0.01)
| mirror x))
(def body
(ball [1 0.75 0.5 * 100]
| union :r 72 (ball [0.58 (.84 * 0.75) 0.5 * 250] | move y -156)))
(defn body-color [$] (cel $ (hsv (4 / 6) 0.1 0.3)))
(def decoration (rect [32 10] :r 10 | rotate (q.x * 0.044 - pi) | rotate pi))
(def top-decorations
(fork decoration
(fn [$] ($ | rotate -0.21))
(fn [$] ($ | scale 0.9 | move x 71 | rotate -0.30))
| move x 40
| mirror x))
(def bottom-decorations
(spoon decoration
(fn [$] ($ | scale 0.95 | move x 76 y 0 | rotate -0.14))
(fn [$] ($ | scale 0.9 | rotate -0.59 | move x 140 y -37))
| mirror x))
(def decorations (union top-decorations (bottom-decorations | move y -80)))
(def tummy-patch
(box 110 :r 64
| morph 0.70 (sphere 118)
| move z 60 y -137
| cel (hsl (/ 69 255) 0.10 0.65)))
(def body-and-ears
(union :r 6
body
(ears | scale 0.42 | move y 81)
| body-color
| union-color (subtract tummy-patch (decorations | extrude z inf | scale 0.5 | move y -46))))
(def eyes
( sphere 14
| union :r 4 (box [14 0 1] | move z 4)
| cel 10 | union-color (sphere 5 | move z 14 | cel 0.05)
| scale z 0.5
| rotate x -0.37 y 0.34
| move [52 28 43]
| mirror x))
(def arms
(box [15 100 (ss p.y -100 50 25 40)] :r 15
| rotate z (p.y * 0.002)
| rotate z 0.30
| move x 131 y -122
| mirror x
| body-color))
(def whiskers
(union
(line [0 0 0] [85 0 0] 1.5 0.5 | rotate z 0.03 | move [0 6 0])
(line [0 0 0] [82 0 0] 1.5 0.5 | rotate z -0.03)
(line [0 0 0] [87 0 0] 1.5 0.5 | rotate z -0.10 | move [0 -6 3])
| move [60 0 41]
| mirror x
| color 0.1))
(def floor (ground -300 | cel (hsv 0.7 0.1 0.04)))
(def nostrils
(cylinder z 2.5 3
| move [7 1 4]
| mirror x
| color [0 0 0]))
(def nose
(ball 11
| subtract :r 2 (cylinder z 10 10 | move x 11 y -12 | mirror x :r 1)
| scale z 0.6 y 0.7 x 1.5
| rotate x -0.5
| cel (hsv 0 0.0 0.02)
| union-color nostrils
| scale 1.20
| move z 48 y 30))
(def lilypad
(cylinder y (5 * sin (theta + 1 * 3) * cos (theta - 1 * 2) + 40) 0.3
| move y (sin (theta * 4) * cos (theta * 2 + (length p.xz / 8 + 18)) * dot p.xz * 0.005 + (ss p.z 0 40 0 -10))
| slow 0.7
| union :r 5 (cylinder y 1 10 :r 1 | move y 10 | rotate x 0.2 z (sin (p.y / 4) * 0.1))
| move y 77
| cel (hsv (2 / 6) 0.9 0.5)
| gl/let [theta (atan2 p.xz)] _))
(union
eyes
(union body-and-ears
arms :r (10 - (distance [(abs p.x) p.y p.z] [120 -78 -1] / 10) | max 0)
| expound (perlin p 15) 0.05 2)
whiskers
floor
nose
lilypad
| tint white (fresnel 5 * 0.3))

I had just discovered signed distance functions, and I was enamored by the power that they give you to sculpt space with simple mathematical expressions.

Signed distance functions – SDFs – are amazing, and if this is the first time you’re hearing about them, you should probably drop everything you’re doing today and watch this twenty-five minute video of Inigo Quilez using signed distance functions to create an animation instead. Yes, twenty-five minutes is a lot of minutes. It’s worth it.

I know you didn’t actually watch the video, but the overall gist is that someone very smart and very good at math describes an animation he created out of pure functions of time and space. But the description is pretty high-level: he says things like “we’ll define three circles that we spin as we move down the parameterization of the curve,” which is a beautiful way to think about the effect he uses to create the braids in that video – but how do you actually do that?

Well, you write several hundred lines of something called GLSL, plumbing arguments around and looking up how to construct rotation matrices and forgetting that matrices are column-major in GLSL and trying to remember what you stuck in the w component of this vector and, well, you can do it, and lots of people have, but not without losing some of the mathematical elegance of the original, intuitive presentation.

Because I really just want to write “gimme three circles extruded along a bezier curve, and rotate them by an angle that varies with the current position along the curve,” you know?

(circle 10
| color (hsv ($i + 3 / 6) 0.6 1)
| radial: $i 3 5
| rotate ($t * tau * 4 + t)
| bezier: $t [-100 0 100] [0 100 0] [100 0 -100]
:to (osc t 5 0 1 | ss 0.1 1))

(You can click to pause any of the animations on this page.)

So: Bauble. I wrote Bauble to solve this impedance mismatch, so that I could play around with the SDFs the way that I wanted to play around with SDFs: in a functional, expression-oriented programming language. SDFs are signed distance functions, remember, and primitive operations on SDFs like rotation or translation are literal function composition. You can write a function that takes an SDF and an angle and returns a new SDF – a new function – for the rotated shape.

But you can’t write that in GLSL! GLSL doesn’t have first-class functions, so you actually have to do this composition by hand: if you want to rotate a shape, you have to rotate its input coordinate first, then pass the newly-rotated point in space to the SDF. Which, like, is fine, but it’s friction, and that’s the very simplest sort of operation – once you get into more interesting higher-order operations like instanced tiling, the friction stops feeling fine.

(gl/def apothem (osc t 15 5 10))
(circle (oss t 7 (apothem * 0.5) (apothem * 2 / sqrt 3))
| shade (hsv (hash $i + (t / 10)) 0.75 0.8)
| with-lights (light/point 1 (P + normal))
| move x (mod $i.y 2 * apothem)
| tile: $i [(apothem * 2) (apothem * sqrt 3)] :oversample true
| revolve z | rotate x (t / 20) z (t / 5)
| intersect (cylinder x 150 20 :r 20) :r 2)
(set camera (camera/perspective [403 0 0] :fov 45))

Bauble is not just a higher-level language, though. It’s more accurate to say that I started working on Bauble because I was frustrated with the speed at which I was able to write shaders using SDFs. Not just the verbose manual composition, but the verbose manual composition: it’s hard to compose a detailed scene in pure code! I didn’t only want to make abstract “shadery” looking things. I wanted to be able to make characters too, and that requires a degree of precise and subjective control: I wanted to be able to drag things around, edit shapes interactively, see my shader update live, look at it from different angles… but instead I was over here backspacing over a 1.4, typing 1.5, recompiling, and deciding if it looks better or worse.

# "manta raymarching"
(gl/def bg (ok/mix [0.1 0.1 0.25] sky (ray.direction.y | remap+)))
(triangle [(ss (q.y * gl/if (< q.x 0) 1 0) 0 65 65 40) 130] | rotate (q.y / 270)
| extrude y | expand (ss p.z 140 0 1 3)
| union :r 10 (cone x 18 -184 :r 10 | scale [1 1 2] | move x 49)
| shell 4
| subtract :r 5 (ball [30 30 30] | move x 75)
| union :r 10
(rect [11 5] :r [0 5 5 0]
| extrude z
| expand 1
| rotate x (p.y / -30)
| rotate x (p.x / 40) | pivot [-10 0 0]
| rotate y (p.x / 200 + (sin+ t * 0.5)) | pivot [-10 0 0]
| move x 70 z 28)
| scale [1 (ss p.x 0 -100 1 0.2) 1]
| union :r 3 (cone x 5 -100 | move x -45)
| shade (mix (blue * 0.03) [0.9 1 1]
(2000 / (distance (abs p.xz) [57 117] | pow 3) | clamp 0 1)
) :g 20
| mirror z
| union-color (
(shape/2d
(gl/if (> (hash ($i + 100) * (14 / length ($i * [2 1]))) 0.5)
(distance q (hash2 $i * 4) - (ss (hash $i) 0 1 1 1 * (ss normal.y 0.5 1)))
1000))
| tile: $i [4 4] :oversample true | shade white | extrude y inf)
| morph (shade r3 gray) :distance 0 :color (1 - occlusion :dist 40 | ss 0.4 0.5
+ (dot normal -y | max 0)
)
| union (ball 2 | move [61 -3 32] | mirror z | shade [0 0 0] :g 20 :s 1)
| rotate x (ss p.z 200 0 1 0 * osc t 3 pi/4 -pi/4) | pivot [0 0 10]
| rotate x (ss p.z -200 0 1 0 * osc (t + (sin t * 0.1)) 3 -pi/4 pi/4) | pivot [0 0 -10]
| rotate z (p.x / 800 * osc t 3 -1 1 + osc t 3 -0.1 0.1) | pivot [(oss t 6 -100 100) 0 0]
| rotate x (osc t 6 -0.1 0.1) y (osc t 12 -0.2 0.2)
| bound (ball 140 | move x -20) 20
| move (hash3 $i * 400 + [(osc t 3 -10 10) (oss t 6 -40 40) 0])
| gl/let [t (hash $i * 10 + t)] _
| tile: $i (vec3 700) :limit [50000 8 10] :oversample true
| move x (t * 150)
| map-color (fn [c] (mix c bg (depth / 5000 | pow 2 | clamp 0 1)))
| slow 0.8)
(set background-color bg)

And the camera, gosh – modeling in 3D with a fixed camera is hard. And – while I realize this sounds really dumb – I think the math to calculate a perspective matrix and position a camera where you want pointed in the direction that you want is actually much harder than any the math related to the actual SDFs that you’re trying to render.

So I whipped up a little hack that would basically just concatenate strings of GLSL for me, and put them in a little window with a moving camera.

It took me a few days to get it working: I decided that I wanted to use Janet as my “high level” language, because I’d had a positive experience with the language before, and I knew that it could at least in theory run inside the browser. I had never used WebAssembly before, and I had barely used Janet at this point, and even following existing examples I had quite a time getting it to work. Nevermind that it had been almost a decade since I’d done web development seriously, and my sole experience with WebGL at that point was making a few visuals for an old blog post.

But I got something working, eventually. Here’s the very first demo I ever recorded of the thing that would become Bauble:

Notice the dark, oversaturated colors. I didn’t know I had to do my own gamma correction! This was like my third ever shader. I had no idea what I was doing.

But even though this was extremely crude – it was literally GLSL string concatenation, of a few fixed primitive shapes, with no dynamic expressions of any kind – it was already so much better than writing GLSL by hand. Even just being able to type [1 2 3] instead of vec3(1.0, 2.0, 3.0) was worth the time I’d spent on it.

And it was fun. There’s something so viscerally satisfying about making something you can touch and play with and see in real time like this. I was having fun working on this little toy, so I kept working on it.

I implemented an orbital camera. I switched the editor to CodeMirror, and learned how to write a Janet grammar for it, so that I could directly manipulate the parsed AST to edit values with my mouse (ctrl-click and drag on any number!). With CodeMirror came TypeScript, which I had never used before, and some cruel prank called “rollup,” and I got to experience firsthand the hell of the modern JavaScript ecosystem. I wrote a UI, and decided to try something called SolidJS, which I’ve mildly regretted ever since.

(union
(revolve shape y radius
| move y (atan2+ p.xz / tau * sep + (round (p.y / sep) * sep))
)
(revolve shape y radius
| move y (atan2+ p.xz / tau * sep + (round (p.y / sep) - 1 * sep))
)
| let [shape (circle 2
| shade (hsv (hash $i + hash $j + (t * 0.1)) 0.7 1) :s 1 :g 10
| with-lights (light/ambient 1 normal)
| tile: $i [10 10] :limit 4
| radial: $j 5 50
| rotate (t / 3))] _
| gl/let [radius (osc p.y 1000 50 200) sep 146] _
| rotate y t)

Everything was very new and exciting, and I learned a lot about Wasm and Janet and OpenGL and SDFs and procedural art in general.

And I kept growing the capabilities of Bauble’s… compiler? Would we call it a compiler? It was still, at this point, a glorified string concatenator. But I taught it how to concatenate real fancy-like; I added support for custom dynamic expressions so that you could write things like “rotate space around the y-axis by an angle that varies with the current y coordinate:”

(star 100 50 | extrude y 100
| rotate y (osc t 3 | ss 0.1 0.9 * pi/2 * p.y / 100 + (0.5 * t))
| slow 0.5)

Eventually I even implemented animations, and complex surface-blending operations, and higher-order bounding operations to improve rendering performance, and domain repetition, and, and…

And finally my crowning achievement: custom dynamic lighting, with raymarched soft shadows, which you could specify on a shape-by-shape basis, and whose properties could vary over time and space to produce complex, interesting effects.

(def light-count 6)
(defn light [i]
(gl/def at (rotate [0 (osc t 5 20 200) 60] y (i / light-count * tau + t)))
(light/point (hsv (i / light-count) 1 1) at :shadow 0.25
:brightness (100 / (dot P at | abs | pow (osc t 3 0.7 1.2)) | min 2)))
(octahedron 20 :r 5 | rotate x (t + $i) y (t + $i) z (t + $i)
| shade (hsv $i 0.6 0.5)
| union (ground -40 | shade gray)
| with-lights ;(seq [i :range [0 light-count]] (light i))
| gl/let [$i (hash $i)] _
| tile: $i [80 0 80])

It was the most complicated feature of Bauble, one that stretched its string concatenator to the absolute limits, one that had to be special-cased in the typechecker in order to generate correct code, and one that would still occasionally generate invalid GLSL if you looked at it wrong.

It was also the last “must-have” feature. Once lighting was done, Bauble was “finished.” I wrote some token documentation, and a little tutorial, and I announced Bauble to the world. I forced myself to stop hacking on it for a little while, because I had more important things to do, and I went outside for the first time in two months.

(cylinder y 100 20 :r 20
| union (cone y 30 50 :r 5) :r 50
| expound (fbm 5 simplex [p.x p.z (distance p [0 50 0] - (t * 20) + (atan2+ p.xz / pi * -150) )] [50 50 20]) 3
| shade (mix blue sky 0.5) :g 30 :s 1
| slow 0.5
| tint sky (fresnel 5))
(set camera (camera/perspective [-180 100 0]))

I didn’t set it aside for long. But when I returned to it, when I looked back over what I had wrought in this furious coding binge, I found…

You know that scene in Raiders of the Lost Ark where they open up the roof of the Well of Souls, and they drop a torch down there, and the ground is just a solid mass of writhing snakes?

That was basically the codebase that I had produced.

(morph 0.88
(ball 40 | move x 10 | rotate y (t * 2) | move y (osc t 20 -150 150))
(hexagon :r 5 10
| revolve x (80 + (40 * hash [$i $j]))
| shade (hsv (hash ($i + $j) * 0.04 - 0.03) 1 0.8) :g 15 :s 0.2
| radial: $j y 20 :oversample true :sample-from -1)
| rotate y (p.y / 40 | sin * (mod $i 2 * 2 - 1))
| rotate y (mod $i 2 * tau)
| radial: $i y 2 :oversample true :sample-from -1
| rotate y (t / 10)
| with-lights (light/directional white [-1 -2 0 | normalize] 300 :shadow 0.1)
(light/ambient 0.25 normal)
| slow 0.5)

See, the string concatenation never just went away. The whole core “compiler” was still based on this fragile web of carefully crafted, hardcoded GLSL primitives. There was never, at any point, an abstract syntax tree. There was a sort of weird builder-like imperative “code printer” thing that sort of implicitly tracked an AST and like knew what was in scope at some times, but, like, if you ever wrote a function with a local variable called p you’d break everything, because p is, obviously, the name of a dynamic variable that–

You know what? I don’t need to explain it. I’m sure you can believe me when I say that it was awful code.

(color r2 (teal * 0.1 *
(fbm 8 :f (fn [q] (rotate (q * 2) (pi * sin (t / 100))))
(fn [q] (cos q.x + sin q.y /)) q (osc t 20 30 90))))
(set aa-grid-size 2)

But it wasn’t just the code. It was also a bad product. It was too limiting: Bauble was a tool for making shaders with SDFs, but it didn’t give you any way to actually write your own signed distance functions. You just couldn’t write arbitrary shader code. You could write some custom expressions, using a limited subset of the functions available to you in real GLSL, but there was no way that you, as a Bauble user, could have implemented any of the provided built-ins. There was no “escape hatch” to pure GLSL.

So you were limited to the built-ins, and there just wasn’t that much built-into it. It was missing so many things: I wanted 2D SDFs, and extrusions into 3D space. I wanted to be able to distort normals without altering distance fields. I wanted to be able to define custom material shaders that could use Bauble’s native shadow casting – you could define custom colors of course, but the only light-aware material in all of Bauble was a simple Blinn-Phong shader. And that fact was, of course, hardcoded.

(defn strip [axis q]
(revolve (trapezoid (mix lo hi h) (mix hi lo h) 20 :r 2 | rotate (h * pi) t
| gl/let [lo 0 hi 10 h (atan2+ q / tau)] _) axis 100))
(union
(strip y p.xz | move x -50 | shade sky)
(strip z p.xy | move x 50 | shade orange)
| rotate z (t / 3) y (t / 2)
| tint purple (fresnel 5 * 0.5))

Maybe more than anything else, I wanted to add 3D mesh export – I wanted to be able to export Bauble shapes into OBJ files or STL files or whatever the right one is today, because I wanted to 3D print my Baubles. But I also wanted to add custom camera support, and anti-aliasing, and video export…

But I had stretched my strings to the breaking point. Even I couldn’t understand what I’d written, and I knew that, if I wanted to keep growing Bauble, I would have to rewrite the core compiler from scratch.

(gl/let [t1 (t / 4)
t2 (ss (fract t1) 0.2 1 (floor t1) (ceil t1))]
(defn nudge [i]
(hash3 i - 0.5
| normalize
| rotate y (osc t2 4 -0.5 0.5) z (osc t2 3 -0.5 0.5) x (osc t2 2 -0.5 0.5)))
(intersect :r (s * 20 + 1)
(plane [+1 +1 +1 + nudge 0 | normalize] 80)
(plane [+1 +1 -1 + nudge 1 | normalize] 80)
(plane [+1 -1 +1 + nudge 2 | normalize] 80)
(plane [+1 -1 -1 + nudge 3 | normalize] 80)
(plane [-1 +1 +1 + nudge 4 | normalize] 80)
(plane [-1 +1 -1 + nudge 5 | normalize] 80)
(plane [-1 -1 +1 + nudge 6 | normalize] 80)
(plane [-1 -1 -1 + nudge 7 | normalize] 80)
| expound (perlin p (20 * s + 30)) (20 * s) 20
| shade (ok/hcl (t2 * 0.4) 0.4 0.6) | with-lights (light/ambient 1 normal)
| gl/let [s (osc t2 1 0 1)] _
| rotate [1 -1 -1 | normalize] (t / 10)
| tint normal+ (fresnel 3)
| tint white (fresnel 0.5 * 0.3)
| map-color (fn [c] (c * (mix 0.1 1 (dot normal [-1 1 1 | normalize] | max 0))))))

Fortunately though, over the course of Bauble’s development, I had produced a comprehensive suite of test scripts with reference images that demonstrated all of the edge cases and problems that I had faced and already fixed and…

No, of course not. I can’t even type that with straight fingers. There were no tests. Actually, worse: there was one test. And it was failing.

I tried to fix it, when I finally noticed it was broken. I tried to reverse engineer my own code, untangle my spaghetti mess to figure out how it had ever worked in the first place, but eventually I gave up. It just wasn’t worth it. There was nothing worth salvaging, and the thought of starting over from scratch after all of the work I’d already done was so discouraging that I just stopped working on Bauble altogether.

And that’s the story of Bauble. It’s a sad story, a story of a codebase collapsing under its own weight, of a prototype trying to grow into a product, and finding that the old aphorism still holds true.

(defn half-hour [offset]
(capsule y 300 (ss p.y 0 300 0 100) (ss p.y 0 300 80 100)
| expound (perlin [0 (t + 2 | log * 300 + offset) 0 + p] 50) 20 10
))
(defn cel [shape color1 color2]
(color shape (mix color1 color2 (fresnel 5 | quantize 3))
| tint (vec3 -0.3) (fresnel 1 | quantize 3)))
(union :r 10
(half-hour 0 | move y 4 | cel sky white)
(gl/with [p [1 -1 1 * p]] (half-hour 1000 | move y 0 | cel orange red))
| scale [1 0.5 1])
(set camera (camera/perspective [0 100 400]))

Two years passed.

(color r2
(ok/mix (ok/hcl (length q | ss 0 150 0.2 0.4 + (t / 30)) (length q / 200 | ss 0 1 0.3 0.1) 0.9)
(vec3 0.1)
(fbm 3 perlin (normalize [q 10] * (ss t 0 20 0 20 + t)) [(vec2 (length q | sqrt)) 6]
+ (length q / 150)))
| rotate (t / 20))

I used Bauble on and off, but found myself increasingly annoyed by its limitations. Occasionally I would even try adding new features, but I could barely type through my hazmat suit.

I kept meaning to write a blog post about Bauble, about everything that I’d learned – how to embed Janet into a website and make an interactive art project that doesn’t use JavaScript – but I never got around to it. Meanwhile I wrote a book about Janet, and dedicated a chapter to my embedding experience, but it never even mentions Bauble.

Despite being the most interesting side project that I’ve ever worked on, I haven’t written anything about it until now.

(gl/def sun-dir [0 -0.5 -1 | normalize])
(defn broad-height [xz] (perlin xz 10000 * 2000))
(gl/def sky-color (mix [1 0.5 1] [0 0 0] (ray.direction.y * 2) * 0.5
| mix (hsv (1 / 6) 0.5 1) (dot ray.direction (- sun-dir) | clamp 0 1)))
(plane y (perlin p.xz 600 * 200 + broad-height P.xz)
| expound (osc p.y 30) 10
| shade (hsv (0.6 / 6) 0.6 0.5 + (normal.yzx * 0.1)) :g (normal.y * 10) :s 0.1
| with-lights (light/directional 1 sun-dir 500 :shadow 0.5) (light/ambient (hsv 0.3 0.3 0.5) normal)
| map-color (fn [c] (mix c sky-color (1 - exp (* -0.0001 depth)) | pow [1.5 2 1.5]))
| slow 0.9)
(set background-color sky-color)
(gl/def camera-xz [(t * -500) 0])
(set camera (camera/perspective [camera-xz.x (broad-height camera-xz + 500) camera-xz.y] :dir [-1 -0.5 -0.10 | normalize]
| camera/tilt (osc t 20 -0.1 0.1)
| camera/pan (osc t 30 -0.1 0.1)))

I’m writing about it now because I recently took a few months off work, for the usual reason, and after some weeks of sleep deprivation and exhaustion and elation I found that I had a few cycles to spend on a side project.

But not a hard side project. Not something that required concentration, or prolonged stretches of focus – luxuries that I have temporarily foresworn. I needed something that I could, almost literally, do in my sleep.

(torus y 100 50
| expound (fbm 6 simplex p [100 50 100])
(osc t 10 | ss 0.25 0.75 | mix 1 (simplex+ (p + 1000 + (t * 10)) 300 | ss 0.2 1 * 15 + 1) _)
16
| shade 0.8 :g 40 :s 0.5
| rotate y (t / 2.5)
| slow 0.4
| tint [1 0.5 0.5] (fresnel 0.25 * 0.1)
| tint white (fresnel 5 * 0.1))
(set camera (camera/perspective [0 250 300 | rotate y (t / 2)]))

So I rewrote Bauble. Or rather, I didn’t rewrite Bauble – instead, I did all of the boring things that I never bothered with the first time around. I wrote a GLSL AST library, with a little pretty-printer. I wrote a typed expression-oriented language that adds “first-class” functions to GLSL. I wrote a Janet DSL for constructing programs in this high-level language, and I added Janet wrappers for (almost) all of GLSL’s built-in functions. I made a command-line interface to Bauble, using vanilla OpenGL instead of WebGL, so that I could finally write a real test suite.

I wrote a real test suite.

(def mouth (ball 50 | subtract (ball 200 | move y 200) | move z 100 | shade black))
(defn fork [shape f1 f2]
(union (f1 shape) (f2 shape)))
(def teeth
(box :r 2 [5 10 2]
| rotate x 0.30
| move y -40
| fork
(fn [$] ($ | move x 6))
(fn [$] ($ | rotate x -0.04 | move x -6 z -1))
| shade white :g 10 :s 1 )) # :ambient 0.5
(def eye-center [39 265 41])
(def eyelid
(ball 40
| move eye-center
| subtract :r 10
(ball 30 | move eye-center | move [0 -10 16])))
(def eye-color
(shade r3 white :g 10 :s 1 # :ambient 0.3
| union (ball 10 | move z 30 | shade black :g 10 :s 1)))
(def eyeball
(ball 35
| shade white :g 10 :s 1
| union-color (ball 10 | move [0 0 30] | shade black :g 10 :s 1)
| rotate x 0.34 y (sin t | ss 0 0.1 * 0.2 - 0.1)
| move eye-center))
(def neck
(cylinder y 30 100
| move y 100
| rotate z (p.y * -0.001)
| rotate x (p.y * 0.0010)
| move y 85
| union :r 10 eyelid))
(def head (ball 100 | scale y 0.9))
(def feet (box [20 30 20] :r 15
| union :r 15 (box :r 10 [10 20 30]
| fork
(fn [$] ($ | rotate y 0.1 | move x 8))
(fn [$] ($ | rotate y -0.1 | move x -11))
| rotate x 0.015 y 0.37
| move [15 -31 22])
| move [50 -82 0]
| rotate y 0.12
| subtract :r 5 (plane y -110)
| mirror x))
(union
(union :r 60 head neck | union :r 10 feet | shade green :s 0.5 :g 6 | tint white (fresnel 15 * 0.5) | subtract :s 2 mouth)
eyeball
(teeth | move z 80)
(ground -110 | shade gray)
| scale 0.5)

I worked bottom-up this time, building one boring primitive at a time and stacking them on top of each other. It was not the joyful exploratory everything-is-new interactive process of building Bauble for the first time, but it was still rewarding: I could see where it was going, and how to get there. It was delayed gratification this time, knowing that if I just got through the slog of the rewrite, I would be rewarded with something that I could be proud of.

(union :r 10
(cone y 120 200)
(cone y 100 150 | move [80 0 -44])
(cone y 89 137 | move [-110 0 -9])
| expound (fbm :f 2.2 5 perlin [2 1 2 * p] 80) 20 40
| slow 0.5
| shade (normal+ | rotate y 1.34 | pow 2)
| move y (osc t 10 | ss 0 0.8 -230 -50)
| union :s 20 (plane y (osc (perlin [1 2 * p.xz] 200 + (t * 0.5)) 1 0 10) | shade sky :g 20 :s 1 | tint (fresnel 3))
)

And I am, now. This is a new Bauble, and it is, across every axis, a better Bauble.

You’ve already seen it of course, but let me give you a quick tour of what you can do with it now.

You can edit complicated shaders without lag: Bauble uses “web workers” now, so that all of the Janet evaluation and compilation and rendering takes place off of the UI thread. This… this doesn’t actually seem to work very well in Chrome or Safari, at least on my “Apple Silicon” MacBook – recompilation is pretty stuttery, taking around 100ms with both the OpenGL and Metal backends. But it’s buttery smooth in Firefox. Weird. WebGL rendering performance is also just universally better in Firefox – if any of the examples on this page are dipping below 60fps, maybe try switching?

You can export 3D models, and you can 3D print them:

That shape on the left is called a “gyroid,” and it was the first Bauble that I ever 3D printed (or, well, had someone cast in bronze for me). There’s no gyroid primitive in Bauble, but you can create custom shapes by writing out an implicit function directly:

(def gyroid (shape/3d (gl/with [p (p / 15)]
(dot (cos p) (sin p.yzx)) + 1 * 10)))
(intersect :r 2.5 gyroid (ball 145))
# Er okay so this is not a "real" gyroid;
# it's more like a half-filled gyroid,
# because it's hard to print thin walls.
# The real deal looks like this:
# (def gyroid (shape/3d (gl/with [p (p / 15)]
# (dot (cos p) (sin p.yzx)) * 10)))
#
# (intersect :r 2.5 (gyroid | shell 1) (ball 145))

I don’t have a 3D printer, and this feature is pretty new, so I haven’t really explored this very much yet. Also Bauble’s mesh export is… primitive, to say the least. It’s just marching cubes, which means you have to generate pretty large models if you want to preserve fine details. I realize that there are many better algorithms for triangulating SDFs, and Bauble should probably use one of them. But… I can only code for a few minutes a week now, and I spent my whole budget on the next feature.

You can embed Bauble on other pages. Not the way that I’ve been doing – the crimes I committed to Bauble’s build system in order to embed the editor here are not really replicable. But you can export your shaders to GLSL, and embed them on any page to add interactive 3D examples in a few lines of code. No one even needs to know you’re using Bauble:

How do planet work?

Innard

Since Bauble pre-compiles the shader, the actual “bauble.js” that you have to embed is just a single 8kb pure-JS file.¹ You don’t need to include the Janet compiler or WebAssembly or anything fancy like that – in fact, you don’t even have to use the Bauble library at all. You can construct the graphics context and compile the shader and draw it yourself, if you’d like.

There’s a biannual event called the “lisp game jam,” and I think it would be fun to use Bauble to render the graphics for a game. Janet has pretty good bindings to Raylib, and you could use that to handle the input and sounds, but render all the graphics with Bauble.

Here, click on this, and then move around with WASD:

Obviously that’s not… a game, exactly. There’s no hit detection, and fire probably shouldn’t cast shadows. But, you know, that’s 30 lines of Bauble code plus 40 lines of JS for the event handling? Imagine what you could do if you weren’t furiously trying to finish the blog post you started writing months ago.

One of my favorite new features of Bauble is that you can edit vectors interactively. Not just the ctrl-click-and-drag on scalars that I mentioned already, but actual dragging vectors around in 3D. Here: put your cursor inside the [50 100 150], and then open quad view with alt-q. You should see crosshairs, and then you can cmd- or ctrl-click and drag one of the orthographic viewports to edit the vector with your mouse. Try it on the [0 0 0] too, to move the box around!

(box [50 100 150]
| move [0 0 0])

So that’s everything you can do with Bauble.

Except… it’s not, is it? I just listed all of the things that I can do with Bauble. Because I know some things about SDFs, and I know Janet, and I understand this weird DSL that I’ve created. But you don’t – yet. How could you?

Which brings me to the last, biggest, and most important new feature of Bauble:

This is https://bauble.studio/help. I wrote a giant reference page with hundreds of interactive examples of every primitive and operation and thing that you can possibly do with Bauble. And it’s available right in the editor, any time you trigger autocomplete: the reference page and completions are both generated from the docstrings of the actual Janet functions. And in case the docstrings aren’t sufficient, there is a little source link next to every single definition that will take you straight to the code.

The documentation isn’t perfect: some small helpers are missing examples; the “escape hatch” to writing raw GLSL isn’t really described at all, and it doesn’t include any of the functions that Bauble lifts directly from GLSL. And I fully realize that a reference like this is no substitute for a decent tutorial.

Bauble still needs a proper tutorial, and one day I’ll write it.

One day I’ll write the Book of Bauble, and explain SDFs and procedural noise and periodic distortions of space and all the tricks that I’ve learned, and how you can apply them to Bauble.

One day.

Let’s say… eighteen years from now, just to be on the safe side.

This is a little misleading, because the compiled shaders themselves are like 5-15kb each. I could minify their source, which would help a bit, but even without doing that, a single Bauble and the player library clocks in around 2% the size of embedding p5.js. (Comparing minified, uncompressed sizes, which is, again, misleading.) ↩︎

Quote-unquote "macros"

Mon, 12 Aug 2024 00:00:00 +0000

You’ve probably seen this Python 101 thing before:

@memoized
def fib(n):
if n <= 1:
return n
return fib(n - 1) + fib(n - 2)

Leaving aside the absurdity of computing Fibonacci numbers recursively, it’s a common first introduction to Python decorators and higher-order functions. fib is just a function, and memoized takes that function and returns a new function (or something with a __call__ method) that, you know, memoizes the result.

Python’s decorators give us a nice notation for writing this, but we could write the same thing in any dynamic language with first-class functions. Like JavaScript:

const memoized = (f) => {
const results = new Map();
return (x) => {
if (results.has(x)) {
return results.get(x);
} else {
const result = f(x);
results.set(x, result);
return result;
}
}
}
let fib = (n) => {
if (n <= 1) {
return n;
}
return fib(n - 1) + fib(n - 2);
}
fib = memoized(fib);

And we’ll just gloss over, like, value equality and variadic functions and the memory leak this will cause.

Is this useful? Probably not, but that’s not the point. The point is that it’s a simple concrete example of an abstract idea – higher-order functions, or decorators, or basic metaprogramming. It’s pedantically useful, even if it’s not practically useful.

So now let’s talk about macros.

We could use macros to implement something like Python’s decorators – a better notation for doing something that we can already do without them. And this is great! Within limits, this is great. But that’s not what this blog post is about.

Instead, I want to talk about something that you can’t really do without macros. A new expressive power that macros enable, something more than just notational freedom.

So bear with me for a moment, because I know this is a little silly. But let’s consider the following alternative memoization scheme:

(defn dumb-example [x]
(while (some-complicated-stuff-happens)
(pretend-like-this-function-is-big))
(def result (memoize (do-something-very-expensive x)))
(do-more-interesting-work))

Instead of memoizing a function to return a new memoized function, I want to memoize an expression. The memoized examples we saw earlier took place on the callee side – the function declaration – but here I want the memoization to happen on the caller side – at the function invocation.

I want it to be the case that this function only actually calls do-something-very-expensive once per unique value of x, even across separate invocations of dumb-example. And if you can stop wondering why you’d want to do this, think for a moment about how you would do this. How could you even write this in JavaScript?

We’ll simplify it a little: instead of worrying about arbitrary expressions, let’s assume that this is always a function call with a single argument, like we did before:

const dumbExample = (x) => {
while (someComplicatedStuffHappens()) {
pretendLikeThisFunctionIsBig();
}
const result = memoize(doSomethingVeryExpensive, x);
doMoreInterestingWork();
}

How do you implement memoize?

I think that you basically can’t, in JavaScript. Or, more accurately: I can’t think of a way to do it.¹

You can easily implement something very similar:

const resultMap = new Map();
const dumbExample = (x) => {
while (someComplicatedStuffHappens()) {
pretendLikeThisFunctionIsBig();
}
const result = memoize(resultMap, doSomethingVeryExpensive, x);
doMoreInterestingWork();
}

If you can manually find a place to hoist the memoization dictionary. And, you know, in practice this would be completely fine. But it wouldn’t teach us anything new.

The problem is that we need some place to store the results that’s somehow tied to this callsite. Not the function, not the program containing the function, but this particular invocation of memoize.

And macros let us do exactly that: a macro will let us allocate, at compile time, a dictionary of results which we can then reference at runtime.

Which would have been a very surprising statement to me, a few years ago. Before I started writing Janet, I thought that macros were syntactic transformations. And in some languages – Rust or OCaml, say – that’s exactly what they are. Syntactic transformations, AST twisters, token shufflers. Still very useful! But you can’t write a purely syntactic macro that admits per-callsite memoization.²

But in Janet, and some other lisp-family languages, macros are so much more than syntactic transformations. They’re ways to execute code – any code – and dynamically generate new functions – any functions – which will become our eventual program. Let’s take a look:

(defmacro memoize [[f & args]]
(def results @{})
~(let [f ,f
args [,;args]
memoization-key [f args]]
(if (has-key? ,results memoization-key)
(in ,results memoization-key)
(let [result (f ;args)]
(put ,results memoization-key result)
result))))
(memoize (+ 1 2))

Set aside the horrific hygiene crimes in this macro definition, and just think about results. We allocated that at compile time, during macro expansion. Just one table! And then we unquoted multiple references to it into our macro expansion.

And this absolutely does not work.

It doesn’t work because unquote puts this table into the abstract syntax tree of our function, and tables, when they’re interpreted as abstract syntax trees and evaluated by Janet, create new tables. The empty table @{} is just how Janet represents the abstract syntax tree for an empty table literal.

So if we look at the macro expansion for our call:

(let [f +
args [1 2]
memoization-key [f args]]
(if (has-key? @{} memoization-key)
(in @{} memoization-key)
(let [result (f (splice args))]
(put @{} memoization-key result)
result)))

We’re allocating a brand new table every time we reference it! We didn’t put “a reference to the table we allocated at compile time” into the function. We put “the abstract syntax tree of a table literal” into our function. It doesn’t matter that every one of those tables in the abstract syntax tree is actually the same table. When Janet’s compile function sees a table – any table – it interprets that as “allocate a new table.”

So we need some way to say “no no no, this value isn’t an abstract syntax tree; it’s just a value. I just want a literal reference to this exact table that I allocated at compile time; I don’t want you to interpret it as an abstract syntax tree at all.”

And you can do that, by quoting the table:

(defmacro memoize [[f & args]]
(def results @{})
~(let [f ,f
args [,;args]
memoization-key [f args]]
(if (has-key? ',results memoization-key)
(in ',results memoization-key)
(let [result (f ;args)]
(put ',results memoization-key result)
result))))
(memoize (+ 1 2))

We still unquote results, but we unquote it inside a quote form. We quote-unquote the table.

And this works! Every time this macro invocation appears, it will expand to an abstract syntax tree like this:

(let [f +
args [1 2]
memoization-key [f args]]
(if (has-key? (quote @{}) memoization-key)
(in (quote @{}) memoization-key)
(let [result (f (splice args))]
(put (quote @{}) memoization-key result)
result)))

And, well, you can’t see this when we print it out like this, but every one of those tables is still the same table. Maybe this is an easier way to visualize what’s happening:

(let [f +
args [1 2]
memoization-key [f args]]
(if (has-key? (quote <pointer-to-our-one-table>) memoization-key)
(in (quote <pointer-to-our-one-table>) memoization-key)
(let [result (f (splice args))]
(put (quote <pointer-to-our-one-table>) memoization-key result)
result)))

And any time Janet evaluates (quote _), it doesn’t look at the thing being quoted. It just returns it. Even if it’s a reference-style, mutable value like it is here.

This was a very long way to point out a pretty simple thing, but I wanted to write this post because I wish that I’d understood this sooner. It took a long time before quote “clicked” for me. I was using quote to write macros long before I actually understood how it worked, but I was not using it to its fullest.

If you’d asked me two years ago, I probably would have said that quote “returns the abstract syntax tree representing its argument.” (+ 1 2) is 3, and '(+ 1 2) is the abstract syntax tree ['+ 1 2] that represents that function call.

And while this seems kinda right, it’s just not. quote doesn’t “return the abstract syntax tree” of anything. quote returns whatever you hand it. The abstract syntax tree ['+ 1 2] came from the Janet parser. The parser created that; quote had nothing to do with it. quote just forwarded it along.

Even the fact that I’m using '+ to mean “the symbol that is just the plus sign” is a little funny. Maybe it’s easier to think of this as the abstract syntax tree [(symbol "+") 1 2]. The symbol, once again, came from the parser, not the quote. But I’m just so used to thinking of '+ as “the way you write a symbol literal.”

Anyway, we can observe this property of quote in a much simpler – albeit stranger – scenario. What does this function do?

(defn foo []
(def x @{})
(put x (length x) true)
x)

It’s not a trick question; it does what you’d expect:

repl:1:> (foo)
@{0 true}
repl:2:> (foo)
@{0 true}
repl:2:> (foo)
@{0 true}

But what about this one?

(defn foo []
(def x '@{})
(put x (length x) true)
x)

It’s… well, it’s not a trick question either, but this definitely would have confused me earlier in my Janet career:

repl:1:> (foo)
@{0 true}
repl:2:> (foo)
@{0 true 1 true}
repl:3:> (foo)
@{0 true 1 true 2 true}
repl:4:> (foo)
@{0 true 1 true 2 true 3 true}

It’s the same table every time, because there was only ever one table in the first place: the table that the Janet parser allocated to represent the abstract syntax tree of the characters @{}.

And we’re mutating it.

Neat.

Is this useful? Well… it’s sort of like a static storage qualifier…? But… but no. It’s not really useful. I struggle to even think of a situation where you would want a per-callsite (memoize), and in any case it would be easier to write that particular macro without using quote-unquote at all:

(defmacro memoize [form]
(def results @{})
(defn memoize-call [f & args]
(def memoization-key [f args])
(if (has-key? results memoization-key)
(in results memoization-key)
(let [result (f ;args)]
(put results memoization-key result)
result)))
~(,memoize-call ,;form))

In this version we allocate a single closure at compile time, and insert a reference to that into the expanded form. The closure refers to the shared table, and we no longer have to think about gensym, or argument evaluation order, or quote-unquoting, or anything else. We’re back in the safe land of writing normal code with values and closures, instead of the weird upside-down world of writing quasiquoted code with abstract syntax trees and evaluation rules.

But. Using quote-unquote to pass values from compile time to runtime is still a generally useful technique. You can’t always write a closure like this – sometimes you really do need to produce an AST with a reference to a mutable value you allocated at compile time.

The problem is, it’s useful in situations that don’t really fit into a blog post. I’ve been spending some time recently rewriting the Bauble compiler, now that I actually know Janet, and this quote-unquote pattern comes up in just about every nontrivial macro I write. These are macros that produce Janet code that you can evaluate to construct a GLSL AST, but some amount of evaluation happens at macro expansion time, and the result of that ahead-of-time evaluation gets quote-unquoted into the intermediate Janet code.

I’ve also used quote-unquote in conjunction with mutable values allocated at compile-time (like what we saw here) in Judge, my inline snapshot testing framework, to ensure that every assertion you write actually runs by the time a test completes.

But, well, those are big and complicated and hard to explain and talk about, and I just wanted to point out how I underestimated quote for so long, in case you’ve been balancing your parentheses under the same misconception.

You could create a global memoization map keyed on the function that you’re calling, but this would actually have different semantics than I’m imagining. If I said memoize(f, 1) + memoize(f, 1) I would expect those to each invoke f, because instances of memoize shouldn’t share results. Why not? Because this is a fake example, and a global memoization is a different (easier!) thing than per-call-site memoization.

Update: someone on Hacker News pointed out that you actually can do this if you write it as a template application instead of a normal function call, because applications of constant templates pass a unique reference per callsite – even when the template function itself varies. Amusingly, the intended purpose of this feature is memoization of template expansions… ↩︎
If macros could rewrite the syntax around themselves, instead of just rewriting themselves, then you could imagine hoisting a const resultMap = new Map(); outside of the function definition and doing this as a purely syntactic transformation. But as far as I know, no one has actually implemented a macro system powerful enough to do that. ↩︎

How to Learn Nix, Part 49: nix-direnv is a huge quality of life improvement

Sun, 28 Jan 2024 00:00:00 +0000

The reason I discovered an ancient blog post the other day was that I had something new to say about Nix for the first time in over two years.

The thing I want to say is this: nix-direnv is great. It fixes roughly every problem that I’ve had with nix-shell, and does so in a much nicer way than my previous ad-hoc solutions.

This is important because I mostly just use Nix to document and install per-project native dependencies. I do use it to install “global” tools as well, but that is rarely very interesting, and most of my interaction with Nix these days consists of editing small shell.nix files.

But it took a bit of doing to get to the point that I felt good about using Nix for this. For one thing, shells don’t register GC roots, which means that every time you collect garbage you have to re-download all the dependencies for the project you were working on. We overcame that hurdle in part 37, by making a custom wrapper around nix-shell that sets up GC roots correctly, but it was surprisingly difficult.

For another thing, Nix is pretty insistent that you use bash as your interactive shell. I figured out a workaround for that in Nix classic, but essentially failed to make nix develop similarly usable.

nix-direnv solves both of these problems. Instead of spawning a new shell, it just adds environment variables to your existing shell. And when it evaluates shell.nix, it automatically registers the result as a GC root.

It also only re-evaluates shell.nix when it actually changes, which means that it in the typical case there’s no startup time. In contrast, my GC-root-installing wrapper takes about 750ms to open a typical shell (raw nix-shell, without the GC root evaluation dance, takes only 400ms). This doesn’t sound very long, because it’s not – I’m running Nix on what I can only characterize as a supercomputer. But I originally installed Nix on a laptop that pre-dated germ theory, and its startup latency was a lot more annoying.¹

nix-direnv also automatically updates the environment when shell.nix changes, so you don’t have to close and re-open your nix-shell whenever you add a dependency. Not only is this ergonomically better, but it also means that you don’t mess up your shell history every time you add a dependency or exit a project.

I had never used direnv before, and to this date the only thing I’ve used it for is managing my Nix shells. But it’s a general tool for managing per-directory environment variables, which is essentially all that nix-shell is. nix-shell can also register bash functions – if you’re using bash – which is useful if you want to use it to debug a derivation. But for my purposes, environment variables are all I really need.

direnv has some built-in support for Nix, but it isn’t great; direnv publishes a table outlining some of the advantages of using nix-direnv. nix-direnv is some sort of plugin(?) that replaces the native Nix support with something much better. And it’s great. It makes the “reproducible developer environment” aspect of Nix just work™. And it’s pretty easy to use:

First off, install nixpkgs.direnv and nixpkgs.nix-direnv.

I installed them with nix-env, using the same declarative wrapper that I wrote in part 22. If you install nix-direnv in a different way, the following will be different.

Installing nix-direnv doesn’t “enable” the plugin; you have to separately tell direnv about it:²

mkdir -p ~/.config/direnv
echo 'source ~/.nix-profile/share/nix-direnv/direnvrc' > ~/.config/direnv/direnvrc
echo 'eval "$(direnv hook zsh)"' >> ~/.zshrc

Once you do that, you have to run the following commands in every directory that you want to nix-shellify:

echo 'use nix' > .envrc
direnv allow

And you’re done. That’s it! Now every time you navigate to that directory, you’ll have…

$ cd ~src/project
direnv: loading ~/src/project/.envrc
direnv: using nix
direnv: nix-direnv: using cached dev shell
direnv: export +CONFIG_SHELL +HOST_PATH +IN_NIX_SHELL +MACOSX_DEPLOYMENT_TARGET +NIX_BUILD_CORES +NIX_CFLAGS_COMPILE +NIX_COREFOUNDATION_RPATH +NIX_DONT_SET_RPATH +NIX_DONT_SET_RPATH_FOR_BUILD +NIX_ENFORCE_NO_NATIVE +NIX_IGNORE_LD_THROUGH_GCC +NIX_INDENT_MAKE +NIX_NO_SELF_RPATH +NIX_STORE +PATH_LOCALE +SOURCE_DATE_EPOCH +XDG_DATA_DIRS +__darwinAllowLocalNetworking +__impureHostDeps +__propagatedImpureHostDeps +__propagatedSandboxProfile +__sandboxProfile +buildInputs +builder +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +dontAddDisableDepTrack +gl_cv_func_getcwd_abort_bug +name +nativeBuildInputs +nobuildPhase +out +outputs +patches +phases +propagatedBuildInputs +propagatedNativeBuildInputs +shell +shellHook +stdenv +strictDeps +system -PS2 ~PATH

Oh. Well that’s not great.

By default direnv prints every environment variable that it adds, removes, or changes. Which makes sense if you’re using it for, like, credentials or something, but for Nix shells it’s just a waste of scrollback.

There’s not really a simple way to suppress printing that giant export line, but you can hack it away by adding something like this to your .zshrc:

export DIRENV_LOG_FORMAT="$(printf "\033[2mdirenv: %%s\033[0m")"
eval "$(direnv hook zsh)"
_direnv_hook() {
eval "$(direnv export zsh 2> >(egrep -v -e '^....direnv: export' >&2))"
};

(The .s in the regex exclude the “dim text” control characters at the beginning of the line.)

That removes the giant export line without removing the rest of the input. And now:

$ cd ~src/project
direnv: loading ~/src/project/.envrc
direnv: using nix
direnv: nix-direnv: using cached dev shell

Ahhh. That’s better.

I’ve been using nix-direnv for a few months now, and I must say: I wish that I had installed it sooner. It’s a much nicer experience than the default nix-shell, and I’m happy that I can get rid of the bespoke hacks that I’ve accrued over the years.

…almost. The one thing this does not help with is nix-shell -p. nix-shell -p is a useful way to “temporarily” install packages without actually putting them on your PATH, and I still use my zsh hack so that nix-shell -p doesn’t drop me into a bash session. Although I do this rarely enough that I could probably just suffer through it.

To be fair I use tmux and just always have sessions open for the projects I’m working on, so it’s not like it was annoying very often. ↩︎
I think this is the sort of thing that home-manager does for you automatically, but I don’t use home-manager. ↩︎

How to Learn Nix, Part 48: Installing (single-user) Nix on macOS

Fri, 26 Jan 2024 00:00:00 +0000

Note: I wrote the vast majority of this post on August 24, 2022, but I decided that it was a whiny rant that I didn’t want to publish. I came across the draft in January 2024, remembered that this entire series is a whiny rant, and decided to publish it anyway.

I’ve been a happy Nix user for about 18 months now, and– well, not happy happy, but satisfied… no… not really satisfied either; perhaps it’s more of a resigned disgruntlement; a feeling that despite its many flaws, it’s still better than anything else out there, and I’ve invested so much time into it already that it would be a shame to give up now, so… am I describing Stockholm syndrome?

I’ve been imprisoned in Nix’s castle for about 18 months now, and I recently came into a new laptop – one of those shiny blue numbers – so naturally one of the first things I did was install Nix on it.

It would be nice if this blog post were only like a paragraph long, wouldn’t it? Step one: run the installer; there is no step two.

But I am afraid to report: there is very much a step two.

a tale of two nixen

There are two flavors of Nix, and you must choose at installation time which variety you will run.

You can install “single-user” Nix, in which Nix is a binary that you run as an interactive command that does things and forks subprocesses and works like any other package manager you’ve ever used. Or you can install “multi-user” Nix, in which Nix is a daemon, running as root, and the nix command merely makes requests to that daemon, which in turn coordinates a set of special nixbld users that exist so that you don’t build all of your software as root.

If you’re using NixOS, you have to use multi-user Nix, which makes sense to me: Nix controls the entire operating system, and there needs to be some central puppeteer to coordinate changes to the kernel or whatever.

But if you don’t use NixOS, then you have a choice. You can choose to set up a persistent daemon, create a dozen nixbld users, and have every command you run proxy through this centralized service. Or you can choose… not to do that.

I don’t know why you’d choose a multi-user installation when the single-user option is available. There might be a very good reason to prefer multi-user Nix, even when you are installing Nix on a laptop with a single user, but I do not know what it is. Obviously there is no documentation to explain the distinction or persuade the user one way or the other, so I can only speculate.

And needless to say, when I first tasted the forbidden fruit of the Nix tree all those year ago, I chose a single-user install, and I have never regretted that.

But then something upsetting happened.

macOS has always been an unloved corner of Nix, and installing Nix on macOS has been rather fraught with peril ever since macOS Catalina came out.

The problem seems so trivial: Nix requires a /nix directory to exist, but macOS does not let you create new top-level root directories, so you have to actually create a separate volume and mount it as /nix. Which might not sound difficult, but getting that to work with FileVault and the various hardware encryption configurations of different generations of Apple hardware and… I dunno, probably some other complicating factors, I don’t know how computers work… there was a heroic effort to keep macOS Nix working at all, at a time when some Nix maintainers were arguing that it would be easier to deprecate support for macOS entirely.

But there was an unfortunate casualty of this struggle: the single-user installation option was removed.

If you want to install Nix on macOS today, you must use a multi-user installation. That means running nix-daemon, that means messing with launchctl, creating a bunch of build users…

Which brings things back to me.

I had a new laptop. I wanted to install Nix on it. And the only option facing me…

Was to get in there and hack up an unofficial, unsupported single-user install, because I just can’t abide the idea of a multi-user Nix install.

And it’s not only a philosophical objection! Although I admit that I have a visceral aversion to the idea of installing a daemon running as root for something that absolutely does not need to be a daemon or running as root, I can understand that many people do not care about that.

But there is a real, practical reason to avoid the multi-user configuration: if you are using a multi-user Nix install on macOS, then every time you upgrade macOS, Nix will break, and you will have to go in and do some surgery to fix it.

This was true when I wrote this, in 2022. I don’t know if this is still true, but an alternate Nix installer – which also only supports multi-user installs on macOS – puts “survives macOS upgrades” at the top of its advantages over the official installer. Which implies to me that this is still a problem.

This is because the multi-user installation edits some of the system-provided config files – like /etc/bashrc – to insert some Nix setup things. And every time (every time?) you upgrade macOS, it will overwrite these files and restore them to the defaults. Is it hard to fix? No. Is it annoying? Yes. Is it sufficiently annoying to try to figure out a single-user install? To each their own.

In any case: the only difficult thing about installing Nix on macOS is creating that darn /nix directory. And clearly the multi-user install step knows how to do that. So I should be able to steal that part of it, create the directory, and then manually perform a regular single-user install. Right?

Right.

right

Well it worked great, actually! It wasn’t hard at all. The steps are:

First, download the Nix install script.

The official Nix installation instructions are of the curl | bash variety. But if you actually look at the script that you’re curling, it just turns around and curls another script¹ and runs that.

Actually a directory full of them. So the first thing to do is to get that directory of actual install instructions.

You can find them on GitHub, although I just modified the original script to download the installer, extract it, and then quit before doing anything else (er, and also to not delete the extracted artifacts on EXIT). The really interesting one is create-darwin-volume.sh, which, as you can probably guess, creates /nix.

But! You cannot just invoke this script by itself. This script is invoked by the larger installation script, which sets some environment variables that the create-darwin-volume expects to be able to read. Specifically it expects to be able to read…

You know what? I don’t think you care. Or rather, if you do care, I’m sure that you can figure it out from reading the script. It wasn’t very hard, and by the time you’re reading this the precise interplay between the two scripts probably changed anyway, and you would have to dive into the code to figure out what the modern version requires anyway.

Once you’ve created your /nix, modify the install script to remove the lines that force a multi-user install on macOS:

case "$(uname -s)" in
"Darwin")
INSTALL_MODE=daemon;;
*)
INSTALL_MODE=no-daemon;;
esac

Then run the install script, and you’re done.

or just, like, just make a multi-user install

It’s the future now – I wrote this section in 2024, which is part of why this segue is so abrupt.

So, I didn’t do that. I am still running the single-user nix install that I hacked together a year and a half ago. I was kind of worried initially that I was overlooking something that made this more complicated and it would break the moment I upgraded Nix, but no: it has held up perfectly. I don’t know why the official install script forbids single-user installs on macOS, because they seem to work better than multi-user installs – I have never once had to think about Nix during a macOS upgrade.

But I feel that I should correct something that I wrote previously: there are actual reasons why you might prefer a multi-user install, even on macOS.

It’s actually kind of nice to run Nix builds as special nixbld users. Those users only have permission to read and write to the Nix store; they can’t accidentally stick config files in your home directory during install time, for example, or reference a source file that you didn’t actually copy to the store. Whereas, with a single-user install, it’s possible to write Nix expressions that – accidentally or not – depend on files anywhere on your computer.

So the multi-user install has two advantages:

You get some level of sandboxing/protection from malicious packages
You get an error when you write a non-hermetic nix expression, instead of silently succeeding

Those advantage don’t seem worth any added complexity to me, let alone having to deal with upgrade problems, but not everyone uses Nix exactly the way that I do.

(1) is interesting from a security perspective in theory, but since I usually download pre-built derivations from the binary cache and then run it as myself, I am already vulnerable to malicious nixpkgs. Not as vulnerable, perhaps, if the binary cache did some kind of fancy virus scanning of uploaded executables… but we’re in the same ballpark.

I don’t really care about (2) at all, although I could see this being useful to people who regularly share Nix expressions with other human people. The cost of making a mistake here just seems so low to me, and the fix is so simple, but I guess if you make this mistake very often it would be nice to hear about it right away.

There might be even more advantages than these! But we have once again reached the limits of my knowledge.

why do you care so much

It’s 2022 again.

I can’t really explain why I’m so averse to a multi-user Nix installation. But I am.

And I know that if I had met Nix just a little bit later in life – after 2.4 was released, when the single-user install option was deprecated – I never would have come back to Nix in the first place.

Which is a shame, because I really do like Nix, and as much as I like to razz it, I still prefer it – philosophically and practically – to Homebrew.

So I am writing this post to say: you can still install Nix on macOS like any other package manager. You don’t need to take one look at it and go “whoa what the heck I’m not setting up some root-user daemon just to download packages from the internet” and close the tab. You can still use it like you would use any other package manager. You just… have to get your hands dirty first.

Which might actually be the perfect introduction to Nix, come to think of it.

Amusingly, even though the first step included no checksum (though you can find one if you care), this script does checksum the downloaded file. Maybe because it’s served from a different subdomain? I don’t know. ↩︎

The Fibonacci Matrix

Sun, 30 Jul 2023 00:00:00 +0000

When you think about the Fibonacci sequence, you probably imagine a swirling vortex of oscillating points stretching outwards to infinity:

Okay, no, obviously you don’t. Yet.

When you think about the Fibonacci sequence, you probably flush with a latent rage when you remember that it is, more often than not, the way that we introduce the concept of “recursive functions” to new programmers, in some sort of cruel hazing intended to make it harder for them to ever appreciate how recursion can help them write better programs. Sometimes we even add memoization, and call it “dynamic programming,” in order to impress upon them that even the most trivial problems deserve complex, inefficient solutions.

Er, okay, you probably don’t think about the Fibonacci sequence much at all. It doesn’t, you know, come up very often.

But I hope that you will spend some time thinking about it with me today, because I think that the Fibonacci sequence – despite being a terrible showcase for recursion – is a really interesting vector for discussing some techniques from linear algebra.

how to fibonacci	space complexity	time complexity
insane recursion	linear	exponential
memoized insane recursion	linear	linear
trivial iteration	constant	linear
exponentiation-by-squaring	constant	logarithmic
eigendecomposition	let's talk

We will spend no time on the recursive Fibonaccis; I’m sure that you’ve seen them before. Instead, let’s skip right to the “obvious” way to calculate Fibonacci numbers:

function fib(n) {
if (n == 0) {
return 0;
}
let current = 1;
let previous = 0;
for (let i = 0; i < n - 1; i++) {
const next = current + previous;
previous = current;
current = next;
}
return current;
}

No recursion, no memoization. We have two pieces of state: the “current number” and the “previous” number, and at every step of the iteration we advance both of these to new values.

But there’s something very interesting about this function: the new values for our state are a linear combination of the old values.

current' = current + previous
previous' = current

Using x' to mean “the next value for x.”

And you might recognize this as a “system of linear equations.” I think it’s more obvious when we write it like this:

current' = 1 * current + 1 * previous
previous' = 1 * current + 0 * previous

And you might remember that there’s another, more cryptic way to write down a system of linear equations:


current'
previous'


1	1
1	0


current
previous

This is exactly the same thing! This is just another way of writing the equation – it’s just a shorthand notation.

Here, let’s test it out to make sure of that:


1	1
1	0


8
5


1 • 8 + 1 • 5
1 • 8 + 0 • 5


13
8

Well that’s exactly what we expected – 13 is the next Fibonacci number in the sequence, and 8 was the previous one.

We can, of course, repeat this process, by applying the system of linear equations again:


1	1
1	0


13
8


21
13

Or, to put that another way:


1	1
1	0


1	1
1	0


8
5


21
13

And here’s why we care: matrix multiplication is associative, so we can actually think of that like this:


1	1
1	0


1	1
1	0


8
5


21
13

Or:


2	1
1	1


8
5


21
13

In other words: given a system of linear equations to find the next state of our iteration, we can square the matrix-of-coefficients of the system to find a new system of linear equations that represents “two states from now.”

Of course we don’t need matrices to do this. We can compute a formula for “two steps” of our iteration using term substitution:

current' = current + previous
previous' = current
current'' = current' + previous'
previous'' = current'
current'' = (current + previous) + current
previous'' = (current + previous)
current'' = 2 * current + previous
previous'' = current + previous

Which is a new system of linear equations – which we can represent as a matrix as well.


current''
previous''


2	1
1	1


current
previous

We got the same result, because of course we did: multiplying by this matrix really means “advance to the next state.” Multiplying twice means “advance to the next state and then advance to the next state after that.”

And we can keep going. What’s the state three steps from now?


1	1
1	0


1	1
1	0


1	1
1	0


3	2
2	1

Or, more concisely:


1	1
1	0


3	2
2	1

If we do this repeatedly, you might notice a familiar pattern start to emerge:


1	1
1	0


5	3
3	2


1	1
1	0


8	5
5	3


1	1
1	0


13	8
8	5

Which makes sense, doesn’t it? Because if we multiply this matrix with the matrix [1 0] – our starting values – then it’s going to advance forward through six steps of the Fibonacci sequence in a single leap. So naturally we have to be encoding something about the sequence itself in the matrix – otherwise we wouldn’t be able to advance by N steps in constant time.

Now, the insight that takes this from linear to logarithmic is that we don’t have to do this multiplication one step at a time. We can multiply in leaps and bounds.

Let’s call our original starting matrix F, for Fibonacci.


1	1
1	0

We’ve already calculated F²:

F²


2	1
1	1

And now it’s only one more matrix multiplication to calculate F⁴:

F⁴

F²F²


2	1
1	1


2	1
1	1

We can use this fact to calculate arbitrary matrix powers, by breaking the problem up into products of powers of two:

F²¹ = F¹⁶F⁴F¹

And by doing that, we can calculate the nth Fibonacci number in only log₂(n) steps.

Okay, so that’s fun and all, but that’s not really what this blog post is about.

I don’t know about you, but if I came across this matrix in the wild, I would not think “Oh, that’s the Fibonacci sequence”:


1	1
1	0

I would probably think “huh, I dunno, it’s like, a reflection, sort of, or maybe a shear; what’s a shear again, hang on, I need to see a picture.”

That is, I am used to thinking of matrices as transformations of points in space – scales and rotations and things like that. I’m not really used to thinking of matrices as “state machines.”

But this duality is the beauty of linear algebra! Matrices are transformations of points in space and graphs and state machines all at the same time.

So let’s take a look at the Fibonacci transformation, applied to arbitrary points in R²:

That animation is progressively applying and removing the transformation, so we can get some intuition for how it deforms a square. But we’re really more interested in repeated applications of the transformation. So let’s start with the same points, but multiply by that same matrix over and over:

Interesting. Over time, they have a tendency to stretch out along the long diagonals of this rhombus. Let’s zoom out:

Every time a point reflects over that diagonal, it reflects at a slightly different angle, slowly converging towards this straight line.

You might already have an idea of what that straight line means. You might know that, if you look at the ratio between subsequent Fibonacci numbers, they approximate the golden ratio:¹

1 / 1 = 1
2 / 1 = 2
3 / 2 = 1.5
5 / 3 = 1.666...
8 / 5 = 1.6
13 / 8 = 1.625
21 / 13 = 1.61538462
34 / 21 = 1.61904762

The golden ratio is irrational, but every subsequent Fibonacci number is a better and better rational approximation. (The golden ratio is around 1.618033988749 – so we’re already pretty close.)

It’s interesting to see that these estimations don’t “sneak up” on the golden ratio. In fact they alternate between over- and under-estimating it. Which is exactly what we saw in our visualization!

If you return to the “state machine” interpretation of our matrix, remember that the value we’re plotting as x is really “the current Fibonacci number,” and the value we’re plotting as y is “the previous Fibonacci number.” So the ratio between successive numbers – x/y – is just the slope of the lines that our points are traveling along. And we could see points reflecting over that diagonal, over- and under-shooting it, slowly converging… towards the line whose slope is the golden ratio.

Which is, in fact, the “long diagonal” of our rhombus.

And this makes sense, I think – this isn’t some weird coincidence. The golden ratio is all about the ratio between parts and wholes being the same as ratio between parts. And the Fibonacci sequence is all about adding together parts to become wholes that become parts in the next number of the sequence.

Here, our two parts are the “current” and “previous” values, and the whole that they make is the “next” Fibonacci number. Even if we start with two numbers that are completely unrelated to the Fibonacci sequence – say, 8 and 41 – the simple way that we pick the next number will cause us to approximate the golden ratio after only a few iterations:

8 / 41 = 0.1951219
(8 + 41 = 49) / 8 = 6.125
(49 + 8 = 57) / 49 = 1.16326531
(57 + 49 = 106) / 57 = 1.85964912
(106 + 57 = 163) / 106 = 1.53773585

Why is that? Well, because of the definition of the golden ratio.

(A + B) / A = A / B = φ

This is extremely unrigorous, but I can try to sketch out a very informal argument for why this is:

Let’s say the ratio between A and B is some unknown quantity S. It’s not the golden ratio, it might not be anywhere near the golden ratio; we have no idea what it is. In my 8 and 41 example, it wasn’t even in the right ballpark.

A/B = S

(A + B) / A = (1 + B / A) = 1 + (1/S)

So the ratio between the next element in our series and A will be (1 + (1/S)).

We still don’t know what S is! But if we do this again…

A' / B' = 1 + (1/S)

(A' + B') / A' =

(1 + (B' / A')) =

1 + (1 / (1 + (1 / S)))

After each iteration, the original S will become a smaller and smaller component in the final answer, until eventually we’ll just have an expression that looks like this:

1 + (1 / (1 + (1 / (1 + (1 / (1 + (1 / ...)))))))

Whatever our original S was, its contribution to the final result will eventually be negligible. Even after just a few iterations, we can see that the choice of S doesn’t make a huge difference in the outcome:

1 + (1 / (1 + (1 / (1 + (1 / (1 + (1 / -5000))))))) = 1.6667

1 + (1 / (1 + (1 / (1 + (1 / (1 + (1 / 0.001))))))) = 1.5002

And of course even that will fade away after a few more steps.

In fact the version of that expression with an infinite number of steps – where there is no S at all, but just an infinite sequence of divisions – is the “continued fraction” expression of the golden ratio.

Except, well, I’m lying here.

That residue will not fade away for all values of S. First of all, if S is zero, it doesn’t matter how small that term gets – you’re not going to squeeze a number out of it.

But there is another, more interesting value of S that breaks this rule. There is one other number that will not tend towards 1.618 when you repeatedly take its reciprocal and add one. It is the number that is already one plus its own reciprocal:

1 + (1 / 1.61803399) = 1.61803399

Oh, gosh, yes, the golden ratio is one plus its own reciprocal. But I was talking about the other number with that property:

1 + (1 / -0.61803399) = -0.61803399

This number is (1 - φ), and it is also -φ^-1. The golden ratio is weird like that.

That number is a weird number, because if we have two numbers with that ratio – say, -1.236 and 2 – and we applied our transformation, those points would not spread their wings towards the diagonal. What would they do instead?

Aha. Well, that makes sense.

Some points tend towards the top right, some points tend towards the bottom left, but some points get stuck. Sucked into the origin, cursed to forever travel along this one straight line.

Points along the long diagonal also travel in a straight line – they don’t bounce over the diagonal, because they’re already on it. Let’s just focus on these perfectly straight lines:

Not all matrices will produce straight lines like this when you apply them repeatedly. A rotation matrix, for example, will always change the direction of every single line each time you multiply a point by it.

These straight lines are called eigenvectors, which is German² for something like “intrinsic vector” or “characteristic vector."³

Well, to be more precise, any particular point on those straight lines is an “eigenvector.” The vector [φ 1] is an eigenvector, and so is [-2.1φ -2.1]. And the vector [-1/φ 1] is an eigenvector, and so is [-2/φ 2].

But all of the eigenvectors on each line are “similar,” so I’m just going to pick [φ 1] and [(1-φ) 1] as our two representative eigenvectors.

When you multiply an eigenvector of a matrix by the matrix itself, you get back a new eigenvector on “the same line.” That is to say, you get back another eigenvector that is just some scalar multiple of the original eigenvector.

For example, when we multiply our first eigenvector by the Fibonacci matrix:


1	1
1	0


φ
1


φ + 1
φ

Well… it’s not obvious that this is the case, but we actually just scaled the vector by φ. Because φ² = φ + 1. The golden ratio is weird.

Similarly:


1	1
1	0


1 - φ
1


2 - φ
1 - φ

We scaled it by (1 - φ), again somewhat cryptically:

(1 - φ)(1 - φ) =

(1 - 2φ + φ²) =

(1 - 2φ + φ + 1) =

(2 - φ)

So when we multiply our Fibonacci matrix with its eigenvectors, we scale those numbers by φ and (1 - φ). These scaling factors are called “eigenvalues,” and it’s weird that they look so much like the eigenvectors. That’s… that’s a weird Fibonacci coincidence, a weird golden ratio thing, and not a general pattern that holds for eigenvectors and eigenvalues in general.

Okay, so why do we care about this?

Well, once we know the eigenvectors and eigenvalues of the matrix, we can actually perform repeated matrix multiplication in constant time.

…Sort of. You have to imagine a big asterisk after that sentence, which I will explain below.

To explain how, we’re going to need to do a little bit of linear algebra. But first, I just want to restate everything I’ve said so far in explicit notation:

Multiplying F with each eigenvector is the same as multiplying that eigenvector by its corresponding eigenvalue. So:


1	1
1	0


φ
1


φ
1

And:


1	1
1	0


1 - φ
1

(1 - φ)


1 - φ
1

Right. But there’s actually a way to write those two equalities as a single equality:


1	1
1	0


φ	1 - φ
1	1


φ	1 - φ
1	1


φ	0
0	1 - φ

Instead of writing out each eigenvector as a separate column vector, I stuck them into a matrix. And instead of scaling each one by a scalar, I multiplied that matrix by a diagonal matrix.

This is the same statement, though: right-multiplication by a diagonal matrix just means “scale the columns of the left matrix by the corresponding diagonal value.” We can gut check this by performing the multiplication, and seeing that we’re making the exact same statements as before:


1	1
1	0


φ	1 - φ
1	1


φ + 1	2 - φ
φ	1 - φ


φ	1 - φ
1	1


φ	0
0	1 - φ


φ + 1	2 - φ
φ	1 - φ

But now we’re making these statement about both eigenvectors in parallel.

This equality – this statement about how multiplication by the Fibonacci matrix scales eigenvectors – is the secret to computing Fibonacci numbers in “constant time”:


1	1
1	0


φ	1 - φ
1	1


φ	1 - φ
1	1


φ	0
0	1 - φ

The trick here is that we’re going to right-multiply both sides of the equation by the inverse of our eigenvector matrix. This will eliminate it from the left-hand side entirely:


1	1
1	0


φ	1 - φ
1	1


φ	0
0	1 - φ


φ	1 - φ
1	1

-1

And now we have a new way to calculate the “next Fibonacci number.” Previously we knew how to do it by multiplying with the matrix F. Now we can do it by multiplying with, uhh, this inverse eigenvector matrix thing, and then the diagonal matrix of eigenvalues, and then the non-inverse matrix-of-eigenvectors.

Much simpler, right?

This is getting really long and complicated and I’m going to run out of space soon, so let’s give these things names:


1	1
1	0


φ	1 - φ
1	1


φ	0
0	1 - φ

That’s an upper-case lambda, and look, it’s just the convention for the eigenvalue matrix. Eigenvalues are called λ, and when you put them in a diagonal matrix you call it Λ. I don’t make the rules here.

Now that we have some abbreviations, we can write that as the much more palatable:

F = Q Λ Q^-1

Now, the whole reason that we’re doing this is to take advantage of another trick of associativity:

F² = (Q Λ Q^-1)(Q Λ Q^-1)

F² = Q Λ (Q^-1Q) Λ Q^-1

F² = Q Λ Λ Q^-1

F² = Q Λ² Q^-1

That was very abstract, so take a second to think about what this means. F² is the matrix that calculates two steps of our Fibonacci state machine. And we can use this same trick to calculate any power of F, just by calculating powers of Λ.

Fⁿ = Q Λⁿ Q^-1

And this is good, because Λ is a diagonal matrix. And it’s really easy to exponentiate a diagonal matrix! You just exponentiate each element of its diagonal. We don’t even need to use repeated squaring.

This means that we can actually calculate arbitrary powers of F in constant time… if we pretend that exponentiation of a scalar is a constant time operation.

It’s not, though. I mean, yes, exponentiation of an IEEE 754 64-bit floating-point number is constant time, but that’s not what we said. We’re talking about exponentiating an irrational number, and my computer can only represent approximations of that number, and that floating-point error adds up fast. So in order to actually use this to compute large Fibonacci numbers, we would need to use arbitrary-precision floating point, and exponentiating arbitrary precision values is not constant time. It’s… I don’t know, probably logarithmic? But like both to the exponent and the size of the result, and the size of the result is increasing exponentially, so it nets out to linear? I don’t actually know.⁴

But I don’t want to spoil the fun. This is still a very interesting trick, and it’s worth understanding how it works, even if it doesn’t actually give us a way to compute arbitrarily large Fibonacci numbers in constant time.

So: what are we doing.

We moved a bunch of symbols around, and we wound up with this expression:

Fⁿ = Q Λⁿ Q^-1

But I don’t really know what Q^-1 means, and it’s not really clear to me why I should care. Why is multiplying by these three weird matrices the same as multiplying by F? What, intuitively, are we doing here?

At a high level, we’re translating points into a different coordinate system, then doing something to it, and then translating them back into our original coordinate system.

You already know that we can write any point in space as a vector – X and Y coordinates. That’s what we’ve been doing this whole time.

But we can also write a point in space as the sum of two other vectors. Like, [5 3]. We could write that as [1 2] + [4 1] instead. Which, okay, sure. That’s not very interesting.

One “interesting” way to write [5 3] is as the sum of these two vectors: [5 0] + [0 3]. Or, to say that another way:


1
0


0
1


5
3

This is interesting because [1 0] and [0 1] are basically the “X axis” and “Y axis.” And we can think of the point [5 3] as a (trivial!) linear combination of these two axes.

But we could pick different axes. We can pick any vectors we want as our axes,⁵ so let’s pretend for a moment that our axes are [1 1] and [1 -1] instead. Which means that we would write [5 3] as:


1
1


1
-1


5
3

Or, to write that another way:


1	1
1	-1


4
1


5
3

Now we can think of this vector-of-coefficients, [4 1], as another way to identify the point in space x=5 y=3 when we we’re pretending that our axes are [1 1] and [1 -1]. Except in linear algebra we’d call these “basis vectors” instead of “axes.”

But how did we find the coefficients [4 1]? Well, I just found that one by hand; it was pretty easy. But in general, if we want to express some other point using these basis vectors – let’s say [63 -40] – we’ll need to solve an equation that looks like this:


1	1
1	-1


?
?


63
-40

And we can do that by, you know, regular algebra. We “divide” both sides by our matrix-of-basis-vectors, by left-multiplying with the inverse matrix:


1	1
1	-1

-1


1	1
1	-1


?
?


1	1
1	-1

-1


63
-40

And after the inverses cancel, we’re left with the following formula:


?
?


1	1
1	-1

-1


63
-40

And the problem reduces to matrix inversion.

Now, I don’t know about you, but I don’t remember how to invert a matrix. I know there’s a formula in two dimensions, but the only thing I remember about it is that it involves calculating the determinant, and I forgot how to do that too. So let’s just ask a computer to invert it for us:


?
?


0.5	0.5
0.5	-0.5


63
-40

Hmm. I feel like I probably could’ve worked that out myself.

But that lets us solve the equation, and figure out how to write the point [63 -40] as a combination of the vectors [1 1] and [1 -1]:


0.5	0.5
0.5	-0.5


63
-40


51.5
11.5

Great! We did it.

And here’s why we care:

We can use this exact same trick to write down the points in our Fibonacci sequence as a linear combination of our two eigenvectors. Like this:

Click or tap to add points there, to see how we can write each point in space as a combination of the “short diagonal” and “long diagonal” eigenvectors of our matrix.

Normally to identify a point in space we would give its XY coordinates: go this far along the X-axis, then this far along the Y-axis. But here we’re representing points in “φ” and “1 - φ” coordinates: go this far along the short diagonal, then this far along the long diagonal.

But how do we know how far to go along these diagonals? Well, we “divide by” the eigenvectors. In other words, we have to compute the inverse of this matrix:


φ	1 - φ
1	1

-1

√5


1	φ - 1
-1	φ

Now, matrix inversion is boring, so I’m just presenting the answer here. This inverse matrix is how we can convert from “XY coordinates” into “eigenvector coordinates.”

Let’s work through a concrete example to make sure this works.

[8 5] is a point on the Fibonacci sequence. We can express that as a combination of eigenvectors instead:

√5


1	φ - 1
-1	φ


8
5

≈


4.96
0.04

4.96 and 0.04 are the coefficients we will pair with our eigenvectors: we have to travel 4.96 units down the long diagonal, and 0.04 units along the short diagonal to arrive at the point [8 5].

4.96


Φ
1

0.04


1 - Φ
1

≈


8.025
4.96


-0.025
0.04


8
5

Great. It worked!

But that wasn’t very interesting – we just converted our point into the eigenvector basis and then right back into the normal XY basis. It was kind of a pointless transformation.

But we don’t have to do the unconversion immediately. We can keep the point in this “eigenbasis” for a little while, and do stuff to the vector-of-coefficients, and then convert it back.

Specifically, we can scale the coefficients by the eigenvalues of our Fibonacci matrix. We can multiply the “long diagonal” component by Φ², and multiply the short diagonal component by (1 - Φ)², and we’ll have a new point: something close to [12.985 0.015]. And if we convert that back into XY coordinates:

12.985


Φ
1

0.015


1 - Φ
1


21
13

We just advanced our point two more steps along the Fibonacci sequence, with nothing more than scalar exponentiation and a constant number of vector operations.

This is exactly the same as the expression:

F² = Q Λ² Q^-1

But as someone with no background in linear algebra, I find it easy to get lost in the notation, so it’s easier for me to think about this as operations on separate column vectors rather than as operations on matrices. Even though they are the same thing.

Of course, calculating two steps of the Fibonacci sequence in constant time isn’t that impressive. But we can do the same with Φ¹⁰⁰⁰, and use that to calculate the thousandth Fibonacci number in constant time.

…Assuming we could calculate Φ¹⁰⁰⁰ in constant time. Which we can’t, in real life.

Alright.

The post is over; you saw the trick. “Eigendecomposition,” this is called.

I glossed over a few steps – I spent absolutely no time explaining how I knew the eigenvalues and eigenvectors of this matrix, for example. I just asserted that they were related to the golden ratio. But in reality you can solve for them, or ask a computer to do it for you. It’s pretty mechanical, like matrix inversion – it seems linear algebra is best explored with a repl nearby.

In any case, I think that the why of eigendecomposition is more interesting than the how.

As for the Fibonacci sequence… well, this is a pretty terrible way to actually calculate Fibonacci numbers. Even if we pretend that we only care about numbers that can fit in IEEE 754 double-precision floats, we still can’t use this technique to calculate very many Fibonacci numbers, because the floating-point error adds up too quickly.

But if we only care about double-precision floats… well, there is one more Fibonacci implementation to consider. It’s an algorithm that that runs in constant time, and constant space, and covers the full gamut of floating-point numbers without accumulating any error at all…

const fibs = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765, 10946, 17711, 28657, 46368, 75025, 121393, 196418, 317811, 514229, 832040, 1346269, 2178309, 3524578, 5702887, 9227465, 14930352, 24157817, 39088169, 63245986, 102334155, 165580141, 267914296, 433494437, 701408733, 1134903170, 1836311903, 2971215073, 4807526976, 7778742049, 12586269025, 20365011074, 32951280099, 53316291173, 86267571272, 139583862445, 225851433717, 365435296162, 591286729879, 956722026041, 1548008755920, 2504730781961, 4052739537881, 6557470319842, 10610209857723, 17167680177565, 27777890035288, 44945570212853, 72723460248141, 117669030460994, 190392490709135, 308061521170129, 498454011879264, 806515533049393, 1304969544928657, 2111485077978050, 3416454622906707, 5527939700884757, 8944394323791464];
const fib = (n) => fibs[n];

But it’s more fun to overthink it.

In case you are one of today’s lucky 10,000: the golden ratio is also very close to the conversion rate between miles and kilometers, so you can use Fibonacci numbers to approximate conversions between miles and kilometers in your head. For example, 80km ≈ 50mi. This is a weirdly good conversion – the exact answer is 49.7097mi.

It even works when you don’t have a round Fibonacci number to work with. 120kph is probably around 90% of 130kph, and 90% of 80mph is 72mph… the correct answer would be 74.6mph, but we got a decent ballpark with nothing but eyeball math. ↩︎
Er, the German word is Eigenvektor, so it’s like… half a loan word? A loan sub-word? Whatever, the “eigen” part is the relevant bit here. ↩︎
Straight lines are eigenvectors, but eigenvectors are not necessarily straight lines. Rotation matrices also have eigenvectors, but they have complex eigenvectors. Straight lines like this are real eigenvectors. ↩︎
The exact same argument applies to the “logarithmic” exponentiation-by-squaring algorithm as well – squaring arbitrarily large numbers requires arbitrary precision multiplication. It feels different to me, though, because of floating point error: when you’re exponentiating eigenvalues, you need to use arbitrary precision arithmetic even when your final answer could fit into a double. But the integer squaring approach only needs bigints when the Fibonacci numbers themselves become too large to fit into words. ↩︎
As long as the vectors we pick are “linearly independent” – there’s no way to express [5 3] as a combination of [1 0] and [2 0], for example. ↩︎

My Kind of REPL

Wed, 05 Jul 2023 00:00:00 +0000

I want to tell you about an idea that has had a huge influence on the way that I write software. And I mean that in the literal sense: it’s changed the way that I write software; it’s re-shaped my development workflow.

The idea is this: you can write programs that modify themselves.

And I don’t mean macros or metaprogramming or anything fancy like that. I mean that you can write programs that edit their own source code. Like, the files themselves. The actual text files on disk that have your source code in them.

That’s not the whole idea, though. There’s more to it: you write programs that can edit themselves, and then you use that as your REPL.

Instead of typing something into a prompt and hitting enter and seeing the output on stdout, you type something into a file and hit some editor keybinding and the result gets inserted into the file itself. Patched on disk, right next to the original expression, ready to be committed to source control.

This read-eval-patch loop is already a little useful – it’s a REPL with all of the conveniences of your favorite editor – but we’re still not done. There’s one more part to the idea.

Once you have your expression and your “REPL output” saved into the same file, you can repeat that “REPL session” in the future, and you can see if the output ever changes.

And then you use this technique to write all of your tests.

That’s the whole idea. You use your source files as persistent REPLs, and those REPL sessions become living, breathing test cases for the code you were inspecting. They let you know if anything changes about any of the expressions that you’ve run in the past. This kind of REPL can’t tell you if the new results are right or wrong, of course – just that they’re different. It’s still up to you to separate “test failures” from expected divergences, and to curate the expressions-under-observation into a useful set.

This is pretty much the only way that I have written automated tests for the last six years. Except that it isn’t really – this is the way that I haven’t written automated tests. This is the way that I have caused automated tests to appear for free around me, by doing exactly what I have always done: writing code and then running it, and seeing if it did what I wanted.

I’m exaggerating, but not as much as you might think. Of course these tests aren’t free: it takes work to come up with interesting expressions, and I have to write setup code to actually test the thing I care about, and there is judgment and curation and naming-things involved. And of course there is some effort in verifying that the code under test actually does the thing that I wanted it to do.

But the reason that it feels free is that nearly all of that is work that I would be doing already.

Like, forget about automated tests for a second – when you write some non-trivial function, don’t you usually run it a couple times to see if it actually does the thing that you wanted it to do? I do. Usually by typing in interesting expressions at a REPL, and checking the output, and only moving on once I’m convinced that it works.

And those expressions that I run to convince myself that the code works are exactly the expressions that I want to run again in the future to convince myself that the code still works after a refactor. They are exactly the expressions that I want to share with someone else to convince them that the code works. They are, in other words, exactly the test cases that I care about.

And yes, I mean, I’m still exaggerating a bit. There is a difference between expressions that you run once and expressions that you run a thousand times. In particular, you want your “persistent REPL” outputs to be relatively stable – you don’t want tests to fail because the order of keys in a hash table changed. And usually you want to document these cases in some way, to say why a particular case is interesting, so that it’s easier to diagnose a failure in the future.

But we’re getting a little into the weeds there. Let’s take a step back.

You’ve heard the idea, now let’s see how it feels. I want to demonstrate the workflow to you, but it’s going to require a little bit of suspension of disbelief on your part: this is a blog post, and I don’t really have time to show you any particularly interesting tests. I also have to confess that I don’t write very many interesting tests outside of work, so this will be a combination of completely fake contrived examples and trivial unit tests lifted from random side projects.

With that in mind, let’s start with one from the “extremely contrived” camp:

Yes, I know that you haven’t actually implemented a sorting algorithm since the third grade, but I hope that it gives you the gist of the workflow: you write expressions, press a button, and the result appears directly in your source code. It’s like working in a Jupyter notebook, except that you don’t have to work in a Jupyter notebook. You don’t have to leave the comfort of Emacs, or Vim, or VSCode, or BBEdit – as long as your editor can load files off a file system, it’s compatible with this workflow.

In case you skipped the video, the end result looks something like this:

(use judge)
(defn slow-sort [list]
(case (length list)
0 list
1 list
2 (let [[x y] list] [(min x y) (max x y)])
(do
(def pivot (in list (math/floor (/ (length list) 2))))
(def bigs (filter |(> $ pivot) list))
(def smalls (filter |(< $ pivot) list))
(def equals (filter |(= $ pivot) list))
[;(slow-sort smalls) ;equals ;(slow-sort bigs)])))
(test (slow-sort [3 1 2]) [1 2 3])
(test (slow-sort []) [])
(test (slow-sort [1 1 1 1 1]) [1 1 1 1 1])

Those tests at the end might look like regular equality assertions – and, in a way, they are. But I only had to write the “first half” of them – I only wrote the expressions to run in my “REPL” – and my test framework filled in right-hand sides for me.

Now, as you probably noticed, there were a lot of parentheses in that demo. Don’t be alarmed: the parentheses are entirely incidental. I just happen to be spending a lot of time with Janet lately, and I’ve written a test framework that’s heavily based around this workflow.

But there’s nothing parentheses-specific about this technique. I first met this type of programming in OCaml, via the “expect test” framework. The specifics of OCaml’s expect tests are a little bit different, but the overall workflow is the same:

let%expect_test "formatting board coordinates" =
print_endline (to_string { row = 0; col = 0 });
[%expect "A1"];
print_endline (to_string { row = 10; col = 4 });
[%expect "E11"]
;;

Instead of the expression-oriented API we saw in Janet, an OCaml expect test is a series of statements that print to stdout, followed by automatically-updating assertions about what exactly was printed. This makes sense because OCaml doesn’t really have a canonical way to print a representation of any value,¹ so you have to decide exactly what you want your REPL to Print.

And this workflow works in normal languages too! Here’s an example in Rust, using the K9 test framework:²

#[test]
fn indentation() {
 k9::snapshot!(
 tokenize(
 "
a
 b
 c
d
 e
 f
g
"
 ),
 "a ␤ → b ␤ c ␤ ← d ␤ → e ␤ → f ␤ ← ← g ␤"
 );
}

We’re pulling from the “trivial side project” category now. That’s a test I wrote for an indentation-sensitive tokenizer, but again, I just wrote an expression that I wanted to check and then checked that the result looked right. I didn’t have to spend any time crafting the token stream that I expected to see ahead of time – I only had to write down an expression that seemed interesting to me.

The fact that you don’t actually have to write the expected value might seem like a trivial detail – it’s not that hard to write your expectations down up front; people do it all the time – but it’s actually an essential part of this workflow. It’s an implausibly big deal. It is, in fact, the first beautiful thing I want to highlight about this technique:

1. When it’s this easy to write tests, you write more of them

The fact that I don’t have to type the expectation for my tests matters a lot more than it seems like it should. Like, I’m an adult, right? I can walk through that code and say “a, newline, indent, b, newline…” and I can write down the expected result. How hard can it be?

I actually just timed myself doing that: it took almost exactly 30 seconds to read this input and write out the expected output by hand (sans Unicode). 30 seconds doesn’t sound very long, but it’s friction. It was a boring 30 seconds. It’s something that I want to minimize. And what if I have 10 of these tests? That 30 seconds turns into, like, I don’t know; math has never been my strong suit. But I bet it’s a lot.

And that was an easy example! Let’s look at something more complicated: here’s one of dozens of tests for the parser that that tokenizer feeds into:

#[test]
fn test_nested_blocks()
 k9::snapshot!(
 parse(
 "
foo =
 x =
 y = 10
 z = 20
 y + z
 x
"
 ),
 "foo (n) = (let ((x (let ((y 10) (z 20)) (+ y z)))) x)"
 );
}

Of course I could have typed out the exact syntax tree that I expected to see ahead of time, but I know that I won’t do that – I don’t even want to time myself trying it. If I actually had to type the input and output for every single test, I’d just write fewer of them.

And yes, you still have to verify that the output is correct. But verification is almost always easier than constructing the output by hand, and in the rare cases where it isn’t, you still have the option to write down your expectations explicitly.

And sometimes you don’t have to verify at all! Often when I’m very confident that my code already works correctly, but I’m about to make some risky changes to it, I’ll use this technique to generate a bunch of test cases, and just trust that the output is correct. Then after I’m done refactoring or optimizing, those tests will tell me if the new code ever diverges from my reference implementation.

But I don’t just write more tests because it’s easy. I usually write more tests because it makes my job easier. Which is the next beautiful thing that I want to highlight:

2. This workflow makes tests useful immediately

We all know that tests are valuable, but they’re not particularly valuable right now. The point of automated testing isn’t to make sure that your code works, it’s to make sure that your code still works in five years, right?

But you won’t be working on this same codebase in five years. And there’s no way that you would make a change that breaks one of these subtle invariants you’ve come to rely on. The time you spend writing automated tests might help someone, eventually, but that’s time that you could instead be spending on “features” that your “company” needs to “make payroll this month.” Which means that – if you’re already confident that your code is correct – it can be tempting to not spend any time hardening it against the cold ravages of time.

But when tests actually provide immediate value to you – when writing tests becomes a useful part of your workflow, when it becomes a part of how you add new features – suddenly the equation changes. You don’t write tests for some nebulous future benefit, you write tests because it’s the easiest way to run your code. The fact that this happens to make your codebase more robust to future changes is just icing.

But in order to really benefit from this kind of testing, you might have to rethink what a “good test” looks like. Which brings me to…

3. Good tests are good observations of your program’s behavior

There are good tests, and there are bad tests. This is true whether or not you’re writing this particular style of test, but the ceiling for what a “good test” can look like is much, much higher when you’re using this technique.

I’ll try to show you what I mean. Let’s look at a test for a function that finds the winner in a game of Hex:

let%expect_test "find winning path" =
let board = make_board ~size:5 "A4 A2 B4 B3 C3 C4 D2 D3" in
print_s [%sexp (find_winning_path board : Winning_path.t)];
[%expect "None"];
play_at board "E2";
print_s [%sexp (find_winning_path board : Winning_path.t)];
[%expect "(Black (A4 B4 C3 D2 E2))"];

Okay. Sure. I can sort of think for a minute and see that there was no winning path until black played at E2, and then there was one, and it’s probably reporting the correct path. But let’s compare that with an alternative:

let%expect_test "find winning path" =
let board = make_board ~size:5 "A4 A2 B4 B3 C3 C4 D2 D3" in
print_board board (find_winning_path board);
[%expect {|
There is no winning path:
1 2 3 4 5
A . w . b .
B . . w b .
C . . b w .
D . b w . .
E . . . . .
|}];
play_at board "E2";
print_board board (find_winning_path board);
[%expect {|
Black has a winning path:
1 2 3 4 5
A . w . * .
B . . w * .
C . . * w .
D . * w . .
E . * . . .
|}];

Isn’t that better? This test has convinced me in a single glance that the code is working correctly. And if, at some point in the future, the code stops working correctly – if I refactor the find_winning_path function and break something – I claim that this test will give me a decent head start on figuring out where I went wrong.

Of course it took some work to write this visualization in the first place. Not a lot of work – it’s just a nested for loop – but still, it did take longer to write this test than the (Black (A4 B4 C3 D2 E2)) version.

This is some of my favorite kind of work, though. It’s work that will speed up all of my future development, by making it easier for me to understand how my code works – and, even better, by making it easier for other people to understand how my code works. I only have to write the print_board helper once, but I can use it in dozens of tests – and so can my collaborators.

Here’s another test that took a little work to produce, but that I found extremely useful during development:

You can imagine this as taking each of the top shapes and tracing them around the perimeter of the bottom shapes, maintaining contact at all times. Sort of a tricky function to write correctly, and one where visualization really helps me see that I got it right – and, of course, to see what I did wrong when I inevitably break it.

But that’s kind of cheating: that visualization uses Emacs’s iimage-mode to render a literal image inline with my code, which is not something that you can do everywhere. And since part of the appeal of this technique is the universal accessibility, I should really stick to text.

Here, here’s a better example. This is a function that, given a source of uniformly distributed random numbers, gives you a source of normally distributed random numbers:

(defn marsaglia-sample [rng]
(var x 0)
(var y 0)
(var r2 0)
(while (or (= r2 0) (>= r2 1))
(set x (math/rng-uniform rng))
(set y (math/rng-uniform rng))
(set r2 (+ (* x x) (* y y))))
(def mag (math/sqrt (/ (* -2 (math/log r2)) r2)))
[(* x mag) (* y mag)])

Except… does it work? I just copied this from Wikipedia; I don’t really have very good intuition for the algorithm yet. Did I implement it correctly?

It’s not immediately obvious how I would write a test for this. But if I weren’t trying to “test” it – if I just wanted to convince myself that it worked – then I’d probably plot a few thousand points from the distribution and check if they look normally distributed. And if I had any doubts after that, maybe I’d calculate the mean and standard deviation as well and check that they’re close to the values I expect.

So that’s exactly what I want my automated tests to do:

(deftest "marsaglia sample produces a normal distribution"
(def rng (math/rng 1234))
(test-stdout (plot-histogram |(marsaglia-sample rng) 100000) `
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣷⣾⣦⣆⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⣿⣷⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⣿⣿⣿⣷⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣼⣿⣿⣿⣿⣿⣿⣿⣿⣿⣆⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣶⣦⣄⡀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣶⣤⣤⣄
μ = 0.796406
σ = 0.601565
`))

And hark! That’s clearly wrong. And the test tells me exactly how it’s wrong: it’s only producing positive numbers (the histogram is centered around the mean). This helps me pinpoint my mistake, and to fix it:

(defn symmetric-random [rng x]
(- (* 2 x (math/rng-uniform rng)) x))
(defn marsaglia-sample [rng]
(var x 0)
(var y 0)
(var r2 0)
(while (or (= r2 0) (>= r2 1))
(set x (symmetric-random rng 1))
(set y (symmetric-random rng 1))
(set r2 (+ (* x x) (* y y))))
(def mag (math/sqrt (/ (* -2 (math/log r2)) r2)))
[(* x mag) (* y mag)])
(deftest "marsaglia sample produces a normal distribution"
(def rng (math/rng 1234))
(test-stdout (plot-histogram |(marsaglia-sample rng) 100000) `
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣦⣾⣿⣦⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣴⣿⣿⣿⣿⣿⣿⣆⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣴⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⣠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⡀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⢀⣴⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣶⡀⠀⠀⠀⠀⠀⠀
⢀⣀⣀⣤⣶⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣶⣤⣀⣀⡀
μ = 0.001965
σ = 0.998886
`))

And, like, yes: writing this test required me to write a Unicode histogram plotter thingy, like a crazy person. But I’ve already written a Unicode histogram plotter thingy, like a crazy person, so I was able to re-use it here – just as I will be able to re-use it again in the future, the next time I feel like it will be useful.

In fact I’ve accumulated a lot of little visualization helpers like this over the years – helpers that allow me to better observe the behavior of my code. These helpers are useful when I’m writing automated tests, because they help me understand when the behavior of my code changes, but they’re also useful for regular ad-hoc development. It’s useful to be able to whip up a histogram, or a nicely formatted table,³ or a graph of a function, when I’m doing regular old printf debugging.

Which reminds me…

4. You can use printf debugging right in your tests

I’ve shown you two types of these “REPL tests” so far.

There’s a simple expression-oriented API:

(test (+ 1 2) 3)

And there’s the imperative stdout-based API:

let%expect_test "board visualization" =
print_board (make_board ~size:5 "A4 A2 B4 B3 C3 C4 D2 D3");
[%expect {|
1 2 3 4 5
A . w . b .
B . . w b .
C . . b w .
D . b w . .
E . . . . .
|}];

When I first starting writing tests like this, the stdout-based API seemed a little weird to me. Most of my tests just printed simple little expressions, and the fact that I had to spread that out across multiple lines was annoying. And if I did want to create a more interesting visualization, like the one above, couldn’t I just write an expression to return a formatted string?

Yes, but: print is quick, easy, and familiar. It’s nice that rendering that board was just a nested for loop where I printed out one character at a time. Of course I could have allocated some kind of string builder, and appended characters to that, but that’s a little bit more friction than just calling print. I’d rather let the test framework do it for me automatically: the easier it is to write visualizations like this, the more likely I am to do it.

But when I actually spent time writing tests using this API, I came to really appreciate a subtle point that I had initially overlooked: stdout is dynamically scoped.

Which means that if I call print in a helper of a helper of a helper, it’s going to appear in the output of my test. I don’t have to thread a string builder through the call stack; I can just printf, and get printf-debugging right in my tests.

For example: our function to make normally distributed random numbers relies on “rejection sampling” – we have to generate uniformly distributed random numbers until we find a pair that lies inside the unit circle.

I’m curious how often we actually have to re-roll the dice. It’s not an observable property of our function, but we can just print it out in the implementation:

(defn marsaglia-sample [rng]
(var x 0)
(var y 0)
(var r2 0)
(var iterations 0)
(while (or (= r2 0) (>= r2 1))
(set x (symmetric-random rng 1))
(set y (symmetric-random rng 1))
(set r2 (+ (* x x) (* y y)))
(++ iterations))
(def rejections (- iterations 1))
(when (> rejections 0)
(printf "rejected %d values" rejections))
(def mag (math/sqrt (/ (* -2 (math/log r2)) r2)))
[(* x mag) (* y mag)])

And then re-run our test, exactly like we did before:

(deftest "marsaglia sample produces a normal distribution"
(def rng (math/rng 1234))
(test-stdout (plot-histogram |(marsaglia-sample rng) 100) `
rejected 1 values
rejected 1 values
rejected 1 values
rejected 1 values
rejected 1 values
rejected 2 values
rejected 1 values
rejected 1 values
rejected 2 values
rejected 1 values
rejected 1 values
rejected 1 values
rejected 2 values
rejected 1 values
rejected 1 values
rejected 1 values
rejected 2 values
rejected 1 values
rejected 1 values
rejected 2 values
rejected 1 values
rejected 1 values
rejected 1 values
rejected 2 values
rejected 1 values
rejected 1 values
rejected 1 values
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⡇⠀⠀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⡇⠀⡀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⡇⠀⡇⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡄⢠⠀⢸⣤⡄⡇⢠⣧⡇⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⢠⡇⢸⡄⣼⣿⡇⡇⢸⣿⡇⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡆⡆⠀⢸⢸⡇⢸⡇⣿⣿⡇⡇⣾⣿⡇⡇⠀⠀⠀⢰⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⡆⠀⠀⠀⡇⣷⣶⢸⣾⣷⣾⣷⣿⣿⣷⣷⣿⣿⣷⡇⢰⡆⠀⣾⡇⡆⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⢰⠀⠀⡇⠀⡆⡆⣷⣿⣿⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡇⢸⣷⡆⣿⡇⣷⢰⡆⠀⠀⠀⠀⠀⠀
⠀⠀⠀⢸⢸⡇⡇⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡇⣿⣿⣿⣿⡇⣿⠀⠀⠀⠀⠀
μ = -0.054361
σ = 1.088199
`))

Alright, so we rejected 33 points, which means that we generated 133 – a rejection rate of around 25%. The area of the unit circle is π, and the area of its bounding square is 4, so I’d expect us to reject around (1 - π/4) ≈ 21.5% of points. Pretty close!

Now, this is the sort of thing that I wouldn’t keep around in an automated test. This was exploratory; this was a gut check. That output is not in a very useful format, and I don’t think it’s really worth ensuring that this rate remains stable over time – I can’t imagine that I’ll screw that up without breaking the function entirely. But the important takeaway is that it was really, really easy to sprinkle printfs into code, and to see their output right next to the code itself, using the exact same workflow that I’m already using to test my code.

And while this output was temporary, I actually do wind up committing many of the exploratory expressions that I run as I’m developing. Because…

5. Good tests make good documentation

Compare:

sep-when is a function that takes a list and a predicate, and breaks the input list into a list of sub-lists every time the predicate returns true for an element.

With:

(test (sep-when [1 2 3 5 3 6 7 8 10] even?)
[[1] [2 3 5 3] [6 7] [8] [10]])

My brain can intuitively understand the behavior of this function from the example code more easily than it can parse and understand the (ambiguous!) English description. The description is still helpful, sure, but the example conveys more information more quickly. To me, anyway.

I think that this is mostly because my brain is very comfortable reading REPL sessions, which might not be true for everyone. And I’m not saying that good English-language documentation isn’t important – ideally, of course, you have both. But there are two important advantages to the second version:

Tests don’t lie; they can’t become stale.
I didn’t have to write it.

I spent a couple minutes trying to come up with a pithy English description of this function, and I don’t even think I did a very good job. Writing good documentation is hard, but generating examples is easy – especially when my test framework is doing half the work.

And since it’s easy to generate examples, I do it in cases where I wouldn’t have bothered to write documentation in the first place. This sep-when example was a real actual helper function that I pulled from a random side project. A private, trivial helper, not part of any public API, that I just happened to write as a standalone function – in other words, not the sort of function that I would ever think to document.

It’s also a trivial function, far from the actual domain of the problem I was working on, so it’s not the sort of thing I would usually bother to test either. And I might regret that – bugs in “trivial” helpers can turn into bugs in actual domain logic that can be difficult to track down, and I probably only have a 50/50 shot of getting a trivial function right on the first try in a dynamically-typed language.

But when the ergonomics of writing tests is this nice, I actually do bother to test trivial helpers like this. Testing just means writing down an expression to try in my “REPL,” and I can do that right next to the implementation of the function itself:

(defn sep-when [list f]
(var current-chunk nil)
(def chunks @[])
(eachp [i el] list
(when (or (= i 0) (f el))
(set current-chunk @[])
(array/push chunks current-chunk))
(array/push current-chunk el))
chunks)
(test (sep-when [1 2 3 5 3 6 7 8 10] even?)
[[1] [2 3 5 3] [6 7] [8] [10]])

Which means that, not only will I catch dumb mistakes early, but I also automatically generate examples for myself as I code. If I come back to this code a year from now and can’t remember what this function was supposed to do, I only have to glance at the test to remind myself.

And on, and on…

I could keep going, but I think that my returns are starting to diminish here. If you still aren’t convinced, James Somers wrote an excellent article about this workflow that I would recommend if you want to see more examples of what good tests can look like in a production setting.

But I think that it’s difficult to convey just how good this workflow actually is in any piece of writing. I think that it’s something you have to try for yourself.

And you probably can! This technique is usually called “inline snapshot testing,” or sometimes “golden master testing,” and if you google those phrases you will probably find a library that lets you try it out in your language of choice. I have personally only used this technique in OCaml, Janet, and Rust, but in the spirit of inclusivity, I also managed to get this working working in JavaScript, in VSCode, using a test framework called Jest:

I will be the first to admit that that workflow is not ideal; Jest is absolutely not designed for this “read-eval-patch-loop” style of development. Inline snapshots are a bolted-on extra feature on top of a traditional “fluent” assertion-based testing API, and it might take some work to twist Jest into something useful if you actually wanted to adopt this workflow in JavaScript. But it’s possible! The hard part is done at least; the source rewriter is written. The rest is just ergonomics.

There is a cheat code, though, if you cannot find a test framework that works in your language: Cram.

Cram is basically “inline snapshot tests for bash,” but since you can write a bash command to run a program written in any other language, you can use Cram as a language-agnostic snapshot testing tool. It isn’t as nice as a native testing framework, but it’s still far, far nicer than writing assertion-based tests. And it works anywhere – I use Cram to test the test framework that I wrote in Janet, for example, since I can’t exactly use it to test itself.

Okay. I really am almost done, but I feel that I should say one last thing about this type of testing, in case you are still hesitant: this is a mechanic. This is not a philosophy or a dogma. This is just an ergonomic way to write tests – any tests.

You can still argue as much as you want about unit tests vs. integration tests and mocks vs. stubs. You can test for narrow, individually significant properties, or just broadly observe entire data structures. You can still debate the merits of code coverage, and you can even combine this technique with quickcheck-style automatic property tests. Here, look:

module Vec2 = struct
type t =
{ x : float
; y : float
}
[@@deriving quickcheck, sexp_of]
let length { x; y } =
sqrt (x *. x +. y *. y)
;;
let normalize point =
let length = length point in
{ x = point.x /. length
; y = point.y /. length
}
;;
end

Pretty simple code, but does it work for all inputs? Well, let’s try a property test:

let assert_nearly_equal here actual expected ~epsilon =
[%test_pred: float]
~here:[here]
(fun actual -> Float.( < ) (Float.abs (actual -. expected)) epsilon)
actual
let%expect_test "normalize returns a vector of length 1" =
Quickcheck.test [%quickcheck.generator: Vec2.t]
~sexp_of:[%sexp_of: Vec2.t]
~f:(fun point ->
assert_nearly_equal [%here] (Vec2.length (Vec2.normalize point)) 1.0 ~epsilon:0.0001)
[@@expect.uncaught_exn {|
(* CR expect_test_collector: This test expectation appears to contain a backtrace.
 This is strongly discouraged as backtraces are fragile.
 Please change this test to not include a backtrace. *)
("Base_quickcheck.Test.run: test failed"
(input ((x -3.3810849992682576E+272) (y 440.10383674821787)))
(error
((runtime-lib/runtime.ml.E "predicate failed"
((Value 0) (Loc test/demo.ml:20:15) (Stack (test/demo.ml:29:26))))
"Raised at Ppx_assert_lib__Runtime.test_pred in file \"runtime-lib/runtime.ml\", line 58, characters 4-58\
 \nCalled from Base__Or_error.try_with in file \"src/or_error.ml\", line 84, characters 9-15\
 \n")))
Raised at Base__Error.raise in file "src/error.ml" (inlined), line 9, characters 14-30
Called from Base__Or_error.ok_exn in file "src/or_error.ml", line 92, characters 17-32
Called from Expect_test_collector.Make.Instance_io.exec in file "collector/expect_test_collector.ml", line 262, characters 12-19 |}]
;;

This is not, you know, a good property test, but just focus on the workflow – even though it’s just an assertion failure, it’s nice to see that error in context. I don’t need to manually correlate line numbers with my source files, even if I’m working in a text editor that can’t jump to errors automatically.

So the mechanic of source patching is still somewhat useful regardless of what sort of tests you’re writing. This post has focused on pretty simple unit tests because, you know, it’s a blog post, and we just don’t have time to page in anything complicated together. And I’ve especially emphasized testing-as-observation because that happens to be my favorite way to write most tests, and it’s a unique superpower that you can’t really replicate in a traditional test framework.

But that doesn’t mean that read-eval-patch loops are only useful in these cases, or that you can only adopt this technique with a radical change to the way you think about testing.

Although it might not hurt…

The convention where I work is to render all data structures as s-expressions, but I didn’t want to upset you any further. ↩︎
Rust also has expect_test, which I have not tried, and Insta, but I had trouble getting it to work when I tried it a few years ago. ↩︎
I couldn’t find room for this in the post – it isn’t very flashy – but in real life I think that tables are the most useful tool in my testing lunchbox. You can see a few examples here of tests with tables in them, although the tables are not really the stars there. ↩︎

Generalized Macros

Tue, 18 Apr 2023 00:00:00 +0000

I’ve been writing a lot of Janet lately, and I’ve been especially enjoying my time with the macro system.

Janet macros are Common Lisp-flavored unhygienic gensym-style macros. They are extremely powerful, and very easy to write, but they can be pretty tricky to get right. It’s easy to make mistakes that lead to unwanted variable capture, or to write macros that only work if they’re expanded in particular contexts, and it can be pretty difficult to detect these problems ahead of time.

So people have spent a lot of time thinking about ways to write macros more safely – sometimes at the cost of expressiveness or simplicity – and almost all recent languages use some sort of hygienic macro system that defaults to doing the right thing.

But as far as I know, no one has approached macro systems from the other direction. No one looked at Common Lisp’s macros and said “What if these macros aren’t dangerous enough? What if we could make them even harder to write correctly, in order to marginally increase their power and expressiveness?”

So welcome to my blog post.

I want to show you an idea for a new kind of macro. A macro that can not only rewrite itself, but actually rewrite any form in your entire program.

I think that “what, no, why on earth” is an entirely reasonable response to that statement, so let’s take a look at a motivating example together.

Lots of languages have something like defer that will run an expression at the end of a block, whether or not the rest of the code raises an exception. defer is just like wrapping the rest of the function in a try-finally block, except that, well, you don’t actually have to do any wrapping. Which means no indentation increase, and no extra nested parentheses.

Here’s an example of what defer might look in Janet:

(do
(def f (file/open "foo.txt"))
(defer (file/close f))
(def contents (file/read f))
(do-something-dangerous-with contents))

Now, we can’t implement defer as a traditional macro, because a traditional macro can only rewrite itself. But what we really want to do is rewrite the parent form that the defer appears in, to wrap its siblings in a finally expression:

(do
(def f (file/open "foo.txt"))
(finally
(do
(def contents (file/read f))
(do-something-dangerous-with contents))
(file/close f)))

So generalized macros let us write exactly this.¹

I want to start at the punchline and work backwards, because my favorite part is how simple it is to write this. So although I don’t actually expect this to make any sense yet, let’s go ahead and look at the implementation of this defer macro:

(defmacro defer [] [expr]
(macro lefts rights
~(,;lefts (finally (do ,;rights) ,expr))))

Pretty tame, right?

So in order to do understand how this works, we’ll have to change three things about macros:

Macros no longer have to appear at the head of the form; they can appear anywhere within a form.
Macros now have two argument lists: the forms “to the left” of the macro, and the forms “to the right” of the macro.
Macros can either return new abstract syntax trees – like a traditional macro – or they can return new, anonymous macros as first-class values.

Let’s go through these ideas one at a time in slightly more detail, and then we’ll circle back to the definition of defer.

Macros can appear anywhere

Scheme already has something called “identifier macros,” which can appear anywhere within a form. You can use them to say that foo is a macro, and it can appear in any expression context, and then make (+ 1 foo) expand to something like (+ 1 (some complicated expression)).

But identifier macros can still only rewrite themselves. In order to do anything interesting with this, we need to add…

Macros can see forms “to the left” and “to the right”

On the face of it this might sound like I’m trying to introduce “infix macros,” so that you could write something like (1 + 2) and rewrite that to the traditional (+ 1 2) syntax. And, to be clear, that is a thing you can do:

(defmacro + [left] [right]
[+ left right])

I think that infix macros could be very useful – we’ll talk more about that in a bit – even if infix math is not particularly compelling to someone practiced in the prefixual arts.

But the real reason to support “infix” macros is for cases like defer, where the (defer ...) expression occurs in the middle of a form, and acts as sort of an “infix” expansion point. But in order for that to work, we need to add…

First-class macros

I think this is the trickiest part to wrap your head around, but it’s the most important. This is the trick that allows macros to rewrite not only themselves, but also the forms around themselves – their parents, their grandparents, their… cousins? I guess? Any other form in your program, actually.

So the idea is that we can create first-class anonymous macros, and return them from our macro implementations. And then those macros will get expanded in the context of the parent form that they now appear in.

This is a lot like returning an anonymous function, except that functions are perfectly reasonable values to put in your abstract syntax trees… so it’s like returning a special function, a function with a little tag attached that says “hey, I’m not a real runtime value, I’m a macro, so you should call me before you finish macro expansion.”

And just to be super explicit: this is different from a macro returning a syntax tree that contains another macro invocation. You can already write “recursive macros,” or macros that return invocations of other macros. But by creating actual new first-class macros at expansion time, you can close over macro arguments and reference them during the next phase of expansion.

So with these changes in mind, let’s come back to the implementation of defer:

(defmacro defer [] [expr]
(macro lefts rights
~(,;lefts (finally (do ,;rights) ,expr))))

Our defer macro takes two binding forms: [] and [expr]. So it expects no arguments to the left – the word defer has to appear at the beginning of its form – and it expects exactly one argument to its right. In other words, it looks like a normal, traditional prefix macro of one argument.

But then it returns an anonymous macro that closes over its expr argument. So if we just look at one step of the expansion, we’ll see an abstract syntax tree that looks like this:

(do
(def f (file/open "foo.txt"))
<macro>
(def contents (file/read f))
(do-something-dangerous-with contents))

But macro expansion isn’t over. After expanding the (defer ...) form, the macro expander will notice that it expanded to another macro, so it will expand that. Which winds up invoking our anonymous macro, passing it ['do '(def f ...)] as its “left” arguments and ['(def contents ...) '(do-something...)] as its “right” arguments.

And then that will return a replacement for the entire (do ...) form, giving us our final result:

(do
(def f (file/open "foo.txt"))
(finally
(do
(def contents (file/read f))
(do-something-dangerous-with contents))
(file/close f)))

Neat, right?

This type of macro gives us a lot more freedom to decide how we want our code to look. I’m honestly not really sure how much more freedom, because I haven’t spent very much time with the idea yet. But I’ve been thinking about it for a while, and I’ve come up with a few examples of things that we can do with this – some much dumber than others.

Let’s take a look at a few of them.

Nest less

I think that reducing the number of nested parentheses and general indentation might be the most compelling use case for this sort of macro. I mean, really this is all defer does: it lets you write finally with a little less nesting, and with the expressions in a slightly different order.

I spend most of my programming time writing OCaml. OCaml doesn’t have “block scope” or “function scope” like most languages – it has expression scope. You introduce new “variables” (they can’t actually vary; all OCaml bindings are “const") using let ... in, and the binding only exists on the right-hand side of that particular expression.

If you think about the nesting of the OCaml abstract syntax tree, it looks something like this:

(let x = 10 in
(let y = 20 in
(x + y)))

But of course you don’t write OCaml like that. For one thing, the parentheses are redundant:

let x = 10 in
let y = 20 in
x + y

For another thing, this triangular indentation is really annoying. So you actually write it like this:

let x = 10 in
let y = 20 in
x + y

The parse tree for that expression is still nested, but it doesn’t look nested – you always format your code linearly.

So lisps also have let, but lisps don’t have the luxury of leaving off the parentheses, so we’re back to the first example:

(let [x 10]
(let [y 20]
(+ x y)))

And if we tried to write that without indentation, then…

(let [x 10]
(let [y 20]
(+ x y)))

Immediate aneurysm.

Fortunately, every lisp dialect that I know of mitigates this problem substantially by allowing let to introduce multiple bindings in a single form:

(let [x 10
y 20]
(+ x y))

Which means that we only have to increase the nesting by one level in the very common case that we have a series of let expressions. But by using generalized macros instead, we can write a version of let that doesn’t increase indentation at all. I’ll call it def:

(defmacro def [] [name value]
(macro lefts rights
~(,;lefts (let [,name ,value] ,;rights))))

def lets us write code like this:

(def x 10)
(def y 20)
(+ x y)

Which gets transformed into code like this:

(let [x 10]
(let [y 20]
(+ x y)))

Of course Janet – and most lisps – have a “linear assignment” form like this built into the language. In Janet it’s called – coincidentally enough – def.

In fact in Janet, def is actually the primitive way to create new bindings, and let is a macro that just desugars to do + def, which is very reasonable and pragmatic, but feels weird to me.

It feels weird to me because, in my mind, let should be syntax sugar for fn – Janet’s word for lambda. Because, after all, these two expressions are equivalent:²

(let [x 10] (+ x 1))
((fn [x] (+ x 1)) 10)

let allows us to write the expression in a much more natural order, but we can introduce new bindings without any lets at all.

This might sound like weird mathematical lambda calculus trivia, but it’s not: it’s important to understand introducing new variables as a special-case of function application, even if this particular function application happens to be trivial.

Because we can apply the same technique that we just used – rewriting def to let, and rewriting let to fn – to do something much more interesting.

Generalized function application

So Haskell has something called do notation. You’ve probably seen something like this before:

addAll :: [Int] -> [Int] -> [Int]
addAll xs ys = do
x <- xs
y <- ys
return (x + y)

ghci> addAll [1, 2] [10, 20]
[11,21,12,22]

This is equivalent to the following Janet code:

(defn add-all [xs ys]
(mapcat (fn [x]
(mapcat (fn [y]
[(+ x y)])
ys))
xs))

But I think the Haskell code is easier to read. Partly that’s because the argument order to Janet’s mapcat function makes the values we’re traversing appear in reverse order in our source code, and we could fix this by redefining mapcat with a argument different order:

(defn add-all [xs ys]
(mapcat xs (fn [x]
(mapcat ys (fn [y]
[(+ x y)])))))

This reminds me of the transformation we did when we changed ((fn [x] (+ x 1)) 10) into (let [x 10] (+ x 1)). So what if we take it one step further, and do the same thing we did to get def?

(defn add-all [xs ys]
(as x mapcat xs)
(as y mapcat ys)
[(+ x y)])

It’s not quite as concise as Haskell’s do notation: Haskell is able to use the type of the expression to determine what <- means, so there’s no need to specify the mapcat bit: it’s implied from the fact that we gave it a list.

Janet doesn’t have an analog for type classes, so we have to be a little more explicit, but this means that we can do more than just “bind” with the as macro. We can also map:

(defn add-all [xs ys]
(as x mapcat xs)
(as y map ys)
(+ x y))

Implementing as is just as easy as implementing def:

(defmacro as [] [name f arg]
(macro lefts rights
~(,;lefts (,f (fn [,name] ,;rights) ,arg))))

If you haven’t programmed in a language like Haskell, this particular syntax sugar might seem a little odd at first. But a specialized notation for generalized function application is extremely useful – we have it in OCaml too, through a syntax extension called ppx_let:

let%bind x = xs in
let%bind y = ys in
return (x + y)

I think OCaml’s notation is actually more clear than Haskell’s – it highlights the symmetry between “ordinary” let bindings and “fancy” let bindings like these. And because it can do more than just bind, we can also avoid the explicit return in OCaml:

let%bind x = xs in
let%map y = ys in
x + y

map and bind aren’t the only functions of this variety, either. Although monads are ubiquitous in OCaml, I spend a lot of my time working with arrows as well. And arrows have yet another notation: let%sub and let%arr.

All of these are generalizations of regular function application. Without worrying about what any of this means, just look at how similar the shape of these different type signatures are:

val (@@) : 'a -> ('a -> 'b) -> 'b
val map : 'a f -> ('a -> 'b) -> 'b f
val bind : 'a f -> ('a -> 'b f) -> 'b f
val sub : 'a s -> ('a r -> 'b s) -> 'b s
val arr : 'a r -> ('a -> 'b) -> 'b s

Okay, I know; this isn’t supposed to be a blog post about monads or arrows. Let’s get back to macros.

Infix operators

So Janet has some very useful “threading” macros that allow you to write code in a more “linear” fashion than you could without them. They’re useful when you’re performing a series of transformations to a value:

(filter |(string/has-prefix? "a" $)
(map string/ascii-lower
(map |($ :name)
people)))

With the power of threading macros, you could write that like this instead:

(->> people
(map |($ :name))
(map string/ascii-lower)
(filter |(string/has-prefix? "a" $)))

This is a lot like “method chaining” in other languages:

people
.map(person => person.name)
.map(name => name.toLowerCase())
.filter(name => name.startsWith("a"))

It’s easier for me to read the linear notation that uses the threading macro. But it’s not actually any easier to write it.

I like that method chaining allows me to write the code in the order of the operations: “Start with people, get the name, lowercase it, filter it to names that start with ‘a.'”

When writing the threading macro, though, the way you type this is “start with people, okay wait, go back, surround it in parentheses, add a ->> at the beginning, now move the cursor to the end of the form, and then get the name…”

I don’t like that. And I know that there are fancy editors that allow you to easily wrap expressions in threading macros or any other without repositioning the cursor, but I’d rather use a syntax that doesn’t require a structural editor to work with comfortably.

So here’s another way to write this:

(people
@ (map |($ :name))
@ (map string/ascii-lower)
@ (filter |(string/has-prefix? "a" name)))

This uses @ as an infix function application macro. I prefer | myself, and that’s the notation that I chose for Bauble, but | is how you create short anonymous functions in Janet, so I don’t want to step on that.

The main reason I prefer this notation is that it’s easier for me to type. I don’t know that it’s any easier to read than ->, but it allows me to write code in the order that I think it, and I like being able to choose a syntax that maps neatly onto my brain.

Another infix macro that I like is .. . is a convenient macro for looking up a keyword in a struct or table, so that you can write struct.key instead of (get struct :key).

In Janet struct.key parses as a single symbol, so we can’t actually implement this as a generalized macro without a separate preprocessing step to split it into three symbols. But we can use it as struct . key, which parses as three separate symbols:

(defmacro . lefts [key & rights]
~(,;(drop-last lefts)
(get ,(last lefts) ,(keyword key))
,;rights))

Which we can then use like this:

(print foo . bar)

(print (get foo :bar))

This is very goofy looking, but you could imagine a language where . always parsed as its own symbol, so that we could just write foo.bar and have that expand to (get foo :bar) automatically.

Another infix macro that I think could be interesting is :, to create a pair.

Janet already uses : as a leader character for declaring keywords, so this won’t work in Janet. But you could imagine, again, a different language where : is used as a short way to create a pair of two elements. So:

foo:bar

Would become:

(foo bar)

This might be useful in languages that use wrapped lets, where you have to write:

(let ((x 10)
(y 20))
(+ x y))

Instead, you could write that as:

(let (x:10 y:20)
(+ x y))

But have it parse in exactly the same way.

You could imagine it in cond as well:

(cond
(> x 0): "positive"
(< x 0): "negative"
(= x 0): "zero"
true: "nan")

Of course Janet doesn’t require wrapping each entry in a cond expression in parentheses, so this isn’t as compelling in Janet.

For a slight variation on this, imagine a macro called ::. It’s just like : – it creates a pair – but the pair appears in reverse order from how you write it. We’re well off the Janet path now, but we could use this as a concise notation for adding type annotations without an explosion of parentheses.

Let’s say we have a – function? macro? – something called Int that provides a type annotation to our compiler. We’d normally write the type of an expression like this:

(def x (Int (compute-thing)))

But look all those close parens! I don’t want to balance those. So instead, we could write:

(def x (compute-thing) :: Int)

Which is equivalent to the less intuitive (to me):

(def x Int:(compute-thing))

:: doesn’t mean “type annotation,” though, it just means “wrap in parentheses.” We could use it to do dumber things:

"hello" :: print

Which would expand, of course, to (print "hello"). But… I don’t know why you would want to do that.

Comment

Something that I occasionally wish for is a (comment ...) macro that lets me ignore code.

You can’t actually write such a macro in Janet. Janet has a macro called comment in the standard library, but comment always expands to nil, and nil is not nothing. This means there are lots of places you can’t use (comment ...):

(defn foo [a (comment this is a comment)]
(print a))

If you tried to compile that, you’d get an error:

compile error: unexpected type in destruction, got nil

Because after macro expansion, the compiler actually sees:

(defn foo [a nil]
(print a))

Which is not valid.

With generalized macros, though, you can write a comment that actually disappears:

(defmacro comment [] [&]
(macro lefts rights [;lefts ;rights]))
(defn foo [a (comment this is a comment)]
(print a))

After expansion, that will just be:

(defn foo [a]
(print a))

Like it never happened.

`if`/`else`

One thing that sometimes trips me up when I’m writing Janet is if.

if takes three forms: a boolean expression, an expression to evaluate if it’s truthy, and an expression to evaluate if it’s falsy. Which is nice and concise, but it’s different enough from other languages that I use – languages with explicit elses – that sometimes I’ll write code like this by mistake:

(if should-do-something-important
(print "okay, performing important work:")
(perform-important-work))

That actually doesn’t perform important work – (perform-important-work) is the “else” section of that conditional. In order to do more than one thing in the “then” branch, we have to wrap all of the statements in do.

And of course Janet has a when macro that does exactly what I want:

(when should-do-something-important
(print "okay, performing important work:")
(perform-important-work))

Which doesn’t have an else branch, and usually when I’m writing an if without an else I should just use when in the first place.

But.

Generalized macros actually let us write if with an explicit else. I’m not saying this is a good idea, but they let us write something like:

(if (empty? name)
(print "you must provide a valid name"))
(else
(print "okay i think it checks out"))

This example is a little weird because, in order for this to work nicely, we’ll have to rename the built-in if. I’ll call the ternary version if-then-else for this example, and say that if now means the same thing as Janet’s when.

(defmacro else [] [else-exprs]
(macro lefts rights
(match (last lefts)
['if & then-exprs]
~(,;(drop-last lefts)
(if-then-else (do ,;then-exprs) (do ,;else-exprs))
,;rights)
(error "else must come immediately after if"))))

This is interesting because the else macro actually rewrites the form before itself, changing the if to an if-then-else and then fussing with its arguments.

Weirder, more exotic things

I could keep going, but, as you have probably noticed, this is already a very long blog post. There’s a lot more that you can do with generalized macros – some of it useful, some of it unhinged, and we don’t have time to talk about all of it.

So far we’ve only seen macros that rewrite their parents or immediate siblings, but you can write macros that return macros that return macros, and use them to rewrite arbitrary forms anywhere in your program. You could write a macro that rewrites “the nearest enclosing function definition,” recursively accumulating first-class macros until finally expanding all of them.

You can write actual left- and right-associative infix operators, and I think that if you tried hard enough, you could even use dynamic variables and controlled macro expansion to implement infix operator precedence (although I don’t think you should).

You could implement (a weaker version of) Janet’s splice built-in as a generalized macro. You could implement “identifier macros” that look around themselves and expand to something different when they appear as the first argument to a (set ...) form. You could, you could…

You could do a lot of things, but I’m going to have to leave these as exercises to the reader, because it’s time to switch gears and talk about why you shouldn’t do this.

Problem one: the repl

The main reason that this seems like a bad idea is that macros like this don’t work at the top level.

If you’re just using the repl, and you type (defer (file/close f)), what happens? One of the arguments to that macro is “everything to the right.” But there isn’t anything to the right! At least, not yet. And it won’t be able to supply everything to the right until you stop typing altogether.

This might not seem like a big deal for defer – just don’t use defer at the repl – but it is a big deal for, say, def. And I don’t know an elegant way to solve this problem: in the general case, macros could look arbitrarily far ahead, so we’d have to wait until we closed the repl session to be able to expand them. And that’s kinda gross.

Problem two: generalized macros don’t always compose

All of the examples that we’ve seen so far play nicely together, but it’s possible to write generalized macros that don’t compose with one another.

The problem is that macro behavior can depend on expansion order. Regular macros always get a chance to run before their arguments get expanded, which is very convenient. Generalized macros don’t have that luxury – because macros can see the forms around them, the order that you expand those forms matters.

In my implementation I chose to expand macros in a depth-first, left-to-right order. So macros always see the arguments “to the left” of themselves fully expanded, and the arguments “to the right” completely unexpanded.

And this can be problematic. For example, let’s say we make an infix alias for set, called :=:

(var x 0)
(x := 1)

Which expands to:

(var x 0)
(set x 1)

This is a trivial generalized macro to write.

Now let’s say we have another macro, which looks at the form to its left to see if it immediately follows set. When it does, it rewrites that set to something else. We could use this to implement some kind of custom associative data structure:

(defmacro at [] [dict key]
(macro lefts rights
(if (= lefts ['set])
~(assign ,dict ,key ,;rights)
~(,;lefts (lookup ,dict ,key) ,;rights))))

So that macro lets us write:

(set (at dict key) 10)

And have that expand to:

(assign dict key 10)

Meanwhile, if we write (print (at dict key)), that will expand to (print (lookup dict key)).

Each of these generalized macros make sense on their own. But if we try to use them together, they just don’t work:

((at dict key) := 1)

Because (at dict key) expands first. It looks at the forms to its left, sees that there aren’t any, so after one step of expansion we have:

((lookup dict key) := 1)

Then we expand :=, and finish with:

(set (lookup dict key) 1)

Which of course was not what we wanted.

I think that using generalized macros safely requires really understanding the effect that they have on the syntax tree of your program. They’re more like ->> and friends – explicit syntax re-arrangers – than they are like other kinds of macros.

Problem three: you have created in your code a work of madness that no other human being can possibly hope to understand

ugh not again

Prior art

I feel like this approach is so simple that it must have been done before, but I can’t find any references to it. That said, I have no idea how to search for it effectively! So if you’ve seen this technique before, or if you’ve heard of it being used in the past, I’d love to hear about it.

Proof of concept

I implemented this macro system in Janet, in order to play around with it and test out my macro implementations.

It was pretty easy to write! It really is a very modest generalization of a traditional macro system. I didn’t actually write a custom module loader that would let you use this as your default macro system in Janet code, but I wrote (the equivalent of) macex, and adding the custom module loader would be pretty easy if you wanted to use it “for real.”

You can look at the code here: https://github.com/ianthehenry/macaroni

Or take a peek at some of the tests, to see the examples in this post in action, as well as some weirder things that didn’t make the cut.

Of course we could teach do to scan through all of its arguments and look for defers, and implement this that way, but then this would only work in do expressions. What if we want to do this inside a (fn [] ...) block? Or an if? Or a while? By making defer itself do the transformation, we can use it anywhere – and make it easy to add new macros that behave like defer, without having to teach do about them. ↩︎
Back in the Old Days, JavaScript only had function-scoped variables, so the only way to create new bindings in JavaScript code was to create and invoke an anonymous function like this. The pattern was so common that we called them “immediately invoked function expressions,” and it was a real thing that you had to do in order to, for example, close over distinct values in a loop. ↩︎

Why Janet?

Wed, 12 Apr 2023 00:00:00 +0000

I never thought it could happen to me. I mean, parentheses? In this day and age? But for the past couple years, my go-to programming language for fun side projects has been a little Lisp dialect called Janet.

(print "hey janet")

I like Janet so much that I wrote an entire book about it, and put it on The Internet for free, in the hopes of attracting more Janetors to the language.

I think you should read it, but I know that you don’t believe me, so I’m going to try to convince you. Here’s my attempt at a sales pitch: here is why you – you of all people – should give Janet a chance.

Janet is simple

Janet is an imperative language with first-class functions, a single namespace for identifiers, and lexical block scoping. The core of the language is very small, consisting of only eight instructions: do, def, var, set, if, while, break, fn. But thanks to macros, there are lots of high-level wrappers that give you more powerful or convenient control flow.

You can “learn” Janet in an afternoon, because the runtime semantics are extremely familiar: think JavaScript, plus value types, minus all the wats. And the rest of the language is small: the entire standard library fits on one page. It was this ease of getting started that got me hooked in the first place.

Janet is distributable

It’s easy to compile Janet programs into native executables that statically link the Janet runtime. And you can share those programs with other people, without asking them to install Janet first – or your project’s dependencies, or anything else for that matter. You don’t even have to tell them it’s written in Janet!

The way that Janet pulls this off is very elegant: Janet compiles itself to bytecode, and then writes that bytecode into a .c file that also starts up the Janet runtime. Then it compiles that C file with your system’s C compiler. Since Janet is designed to be easy to embed, this makes perfect sense: it is, essentially, embedding itself into a trivial C executable.

A simple Janet “hello world” compiled to a native binary weighs under a megabyte (784K for Janet 1.27.0 on aarch64 macOS, but your mileage may vary). This includes the full Janet runtime, garbage collector, and even the bytecode compiler – so you can write programs that evaluate Janet code at runtime, if you want to.

This makes Janet an excellent choice for writing little command-line apps. Which is especially true when you consider that…

Janet is unrealistically good at parsing text

Instead of regular expressions, Janet’s text wrangling is based around parsing expression grammars. Parsing expression grammars are simpler, more powerful, and more predictable than regular expressions. They aren’t line-oriented, so they can parse multi-line text without a problem. They can also parse HTML, or JSON, or any other non-regular language. They can also parse binary file formats – they have no problems with arbitrary null bytes.

They really are parsers: structured, composable, first-class parsers. And they’re pretty easy to learn!

Janet has the best subprocess DSL of any high-level language

There is a third-party library called sh that provides a shell scripting DSL that allows you to express pipes and redirects directly in Janet. Like this:

($ find . -name *.janet | say)

It’s pretty incredible. It’s such a nice DSL that I dedicated a whole chapter of Janet for Mortals to it – and the things that you can do with it. It elevates Janet from a reasonable alternative to Perl to a reasonable alternative to Bash for a surprisingly large range of programs.

Janet is embeddable

Lua has become the de facto “embedded language,” which is a shame, because… well, this isn’t a post about Lua. You might not care about this very much, but there’s a chance that it’s just because you haven’t tried it yet: being able to write progams that expose scripting interfaces is a pretty fun superpower.

Embedding Janet is very easy: the Janet runtime is a small C library, and all you have to do is link it in and then call regular C functions to manipulate Janet values. You can even embed it into websites, and write static sites with custom programmable DSLs!

Janet has mutable and immutable collections

Janet’s collection types come in mutable and immutable flavors. Immutable collections have value semantics: the immutable vector [1 2] is indistinguishable from (take 2 [1 2 3]), despite the fact that they have different memory addresses. Mutable collections, on the other hand, have reference semantics: the hash table @{:x 1 :y 2} is only equal to itself. Another hash table with the same keys and values is a distinct object.

Not every language has immutable composite values built right into the standard library!

Macros, macros, macros

I think this is the real reason you should learn Janet, but I didn’t want to lead with it because I didn’t want to scare you off.

You can write Janet just fine without ever learning how to write macros. But you should learn how, because writing macros is fun. It feels different than any sort of programming that I’ve done before.

Writing macros requires thinking twice at once: you’re writing code to write code, so you have to keep two threads of execution straight in your mind: the code that is running now, at compile time, manipulating values and abstract syntax trees, and the code that you are manipulating, the application code that you produce, the code that will run in the future.

Janet’s macros are not hygienic, and Janet does not have a separate namespace for functions. But by allowing you to unquote literal functions, Janet makes it possible to write macros that are completely referentially transparent. It’s an incredibly simple and elegant solution to an otherwise very delicate problem. And the fact that it is possible to do this in Janet highlights my next favorite feature…

Janet lets you pass values from compile-time to run-time

This is the most interesting thing about Janet, in my opinion. But it might not sound very interesting at first – really all it means is that any Janet value can be serialized to disk and read back in later.

But this serialization is implicit: when you compile a Janet program, it runs all of the top-level instructions – regular statements, function declarations, whatever – and then, once it’s executed all of the top-level values, Janet writes down a snapshot of your program’s state to disk.

And it’s a full snapshot of your program’s state: shared references are preserved, so mutable values can still be mutated after you “resume” the snapshot. Generators remember exactly what instruction they need to run the next time you resume them. Closures gonna close.

Macros are a special-case of compile-time code execution – manipulating abstract syntax trees to create new functions – but this is a superpower that you can enjoy without any macros at all. Making a game? Reticulate your splines ahead of time! Or embed assets in your final binary by reading files at compile time – you can perform arbitrary side effects!

Janet for Mortals has an example of using this to autogenerate database bindings based on a SQL schema file – a bit of a silly example, but something that would be quite difficult to do in most languages.

Janet feels good in the hand

This is completely subjective, but I love Janet’s syntax. It strikes a perfect balance of simplicity, uniformity, and variety.

It uses pervasive parentheses, but breaks them up with [] for lists and {} for tables.

Mutable literals are always prefixed with @: @"mutable string", {:immutable hash-table}, etc.

Anonymous functions are written (fn [x] (+ 1 x)), but there’s a shorthand notation for lifting any expression into a function with |: |(+ 1 $).

Janet supports “splats” or “spreads” with ;: (+ ;args).

String literals can be written with any number of backticks, and closed with the same number of backticks. Escape sequences like \n don’t apply in backtick-quoted strings, so you can create strings with any contents without ever thinking about how to escape them – all you have to do is wrap them in a sufficient number of backticks.

Rest parameters use & instead of .: (defn foo [first & rest] ...).

Janet doesn’t support reader macros, so the syntax itself is fixed. If you know how to read Janet, you can read all Janet programs. Which is not to say you can make sense of them…

Janet prefers comfort to tradition

Janet does not adhere to the ancient customs. CAR is called first. PROGN is called do. LAMBDA is fn, and SETQ is def. nil is not the empty list; it is its own type, and there are first-class Booleans in the language. It eschews EQ, EQL, EQUAL, and EQUALP. There is nary a linked list in sight.

This isn’t really good or bad, but I thought it was worth calling out: if you saw the parentheses and assumed FORMAT was not far behind, maybe give Janet a second look.

Ian Henry

Periodic Spaces

Periodic Spaces

Generalized Worley Noise

Building Bauble

How do planet work?

Quote-unquote "macros"

How to Learn Nix, Part 49: nix-direnv is a huge quality of life improvement

How to Learn Nix, Part 48: Installing (single-user) Nix on macOS

a tale of two nixen

right

or just, like, just make a multi-user install

why do you care so much

The Fibonacci Matrix

My Kind of REPL

1. When it’s this easy to write tests, you write more of them

2. This workflow makes tests useful immediately

3. Good tests are good observations of your program’s behavior

4. You can use printf debugging right in your tests

5. Good tests make good documentation

And on, and on…

Generalized Macros

Macros can appear anywhere

Macros can see forms “to the left” and “to the right”

First-class macros

Nest less

Generalized function application

Infix operators

Comment

if/else

Weirder, more exotic things

Problem one: the repl

Problem two: generalized macros don’t always compose

Problem three: you have created in your code a work of madness that no other human being can possibly hope to understand

Prior art

Proof of concept

Why Janet?

Janet is simple

Janet is distributable

Janet is unrealistically good at parsing text

Janet has the best subprocess DSL of any high-level language

Janet is embeddable

Janet has mutable and immutable collections

Macros, macros, macros

Janet lets you pass values from compile-time to run-time

Janet feels good in the hand

Janet prefers comfort to tradition

`if`/`else`