Chris Arderne

An open source story

Fri, 01 May 2026 00:00:00 +0000

A fun thing happened over the last few years. I contributed in a small way to part of the open source geospatial ecosystem.

First, some background on geo

Zarr is a format for storing large multidimensional arrays on cloud storage (like S3). It plays well with xarray and NumPy, and lets you efficiently query small slices of multi-TB datasets. For many people and workflows, it’s the spiritual successor to GeoTiff and Cloud Optimizted GeoTiff. But one big downside with Zarr is that it doesn’t have “pyramids” built-in (pre-created zoomed-out views), since it’s designed much more for analysis than visualisation.

The Holy Grail is to have a single Zarr array covering the entire globe, where you can index into not only latitude and longitude, but also time, date, altitude etc. Instead of thousands of Tiffs and chips and images, just one beautiful global array. And the wine in the grail would then be the ability to natively visualise this data in the browser at any zoom level. Ideally without a server — that’s the whole point of cloud-native after all. Someone wrote a whole blog post about this, but it has some spoilers for the story below.

My small contribution

For several years, the nice people at CarbonPlan had a project called carbonplan/maps. It sort of let you do this, but was more a proof-of-concept than a library that could be used.

I riffed on it a little bit and created zarr-gl (rhymes with gargle). Much the same thing, slightly fewer dependencies and simpler WebGL code that I could at least understand (although still not very well). The main difference is that I made it a library so that anyone could use it to drop a Zarr layer into a Mapbox/MapLibre map. Still quite clunky, couldn’t reproject or use non-Mercator projections. I had a itch, went as far as the itch took me. Made a little demo site. But then hit some hard problems and didn’t have the pressing need to push through them.

It wasn’t great, but it was timely and a step in the right direction. A few people noticed, and then the folks at CarbonPlan picked up the reins again and made something much better. Now it’s actually useful and workable!

And then shortly after that, Development Seed added Zarr support to deck.gl, which should make it possible to do interesting client stuff (animation, band math).

And that’s it. Everyone else involved was much more competent and dedicated than me, and ultimately no one would or should use my little thing because it’s been superseded. But it was fun to be part of a tiny little shift in the Zeitgeist in a specific niche of the open source world. People started realising that a thing should be possible, and over the course of a few years it went from hacky to glorious through vague cooperation and tinkering of different groups of people.

Maintainerships

On the flipside to the above, some (not many) of my projects have been more successful than I’d like. Mostly the ones that I’m not particularly excited in working on.

signal-export is a thing I created (well forked actually, though the original code is all gone) in 2019 when my family moved from WhatsApp to Signal and I needed a way to continue to do backups. It has slowly racked up 732 GitHub stars, and receives a steady stream of bug reports, pull requests, and also emails to my inbox filled with gratitude or questions needing answers. I use it myself once a year, and so long as it vaguely works I don’t want to think about it more than that. The code is a mess.

But the maintenance burden is small, I write virtually none of the code now, and the community keeps it up-to-date with Single’s data model changes. And there’s a fun give and take: I’m happy I don’t have to do the work, and contributors seem generally excited to contribute some Open Source Code! And it’s one area where LLMs have actually made the contributions better: it’s no where near famous or important enough to attract slop, and now many people are able to fix small bugs or make updates. My job is just to make sure it remains safe to use. I don’t really care about the codebase otherwise, so I’m not particularly opinionated.

I have a similar story with pi-sandbox. It’s a little plugin I made for the pi coding agent that sandboxes network and filesystem access. I made it for myself, fixed a bunch of issues, got some stuff merged in Anthropic’s sandbox-runtime library. Then I forgot about it and when I looked back a month or two later I had 10+ issues and a similar number of PRs that I hadn’t noticed! Almost all were good quality and I’m working my way through them. I’m glad people find it useful and want to improve it, but it’s still undeniably work.

Much like signal-export, it’s a sensitive bit of software. People are relying on it (within reason) to prevent AI agents from doing bad stuff. I really don’t want to ship security vulnerabilities. But I also want to keep it good and up-to-date and honour the effort that community contributors are putting into issues and pull requests. And I also want to do my job and play outside and all that.

It’s a tiny view into what maintainers of really popular projects have to deal with. Certainly a tricky one to balance. Soon I’ll be working somewhere with a very popular open source library. I expect this balance to be slightly easier (and more enjoyable) when it’s your job. Let’s see.

The Parable of the Journey

Fri, 13 Mar 2026 00:00:00 +0000

Two carpenters were going on a journey to build a temple. As they left their homes, lo, a spirit appeared in the road and offered them a ride.

The first man harkened to the spirit, and was delivered quickly to Jerusalem. But the second man did not, for he shied not from an arduous journey. He toiled against bandits and biting winds, shared his bread, and learned much from the scribes and the Pharisees that he met on his way. He grew strong and wise, and regretted not his decision to walk.

But then he arrived in Jerusalem and alas: the temple was already complete. He gazed in wonder and tried to share his thoughts, but they had moved on to the next project.

Practice and product

This is partially because I’ve been too-online for the last few weeks (recovering from surgery, I’m fine thank you) but I’ve noticed a divide between two very different systems of motivation in programmers.

Firstly, the desire to build and produce, focused on outcomes and destinations. This tribe feels ascendant at the moment, and everyone is spending a lot of time talking about how productive they’ve become. And they’re right: we’re producing more temples than ever, the software supply curve is hurtling to the right. Amazing things are happening.

And then there’s the quieter incentive of crafting neat solutions to tricky problems. By typing text into an editor, in this case. You don’t hear about this as much right now, but lots of people really like to do this.

Text in editors is obviously just one part of being a software engineer. All the rest is much the same, and all the higher-level stuff is still there. But the joy of solving problems with elegant bits of code has changed dramatically. On its way to being a hobby, to be done when productivity doesn’t matter. Like walking when you could drive.

And much like driving instead of walking, there are risks: weak muscles, missing mental maps. Go clone a codebase you’ve never seen and try to figure out how it works without asking an LLM. I bet you’ll have some mental blocks, maybe check your phone. We’ve survived all sorts of new technologies. We’ll probably survive this… but there is something disconcerting about outsourcing our thinking to a GPU^[1] .

Practising less, producing more

So the producers are now producing more than ever. Random widgets, fun websites, data analyses. All sorts of cool things that would have taken ages to do before, but are now a snap of the finger. It’s mesmerising how easily you can conjure them.

But something has been lost in this ease. I used to do loads of random little data analyses and explorations and apps. Not because I thought the outcome would be interesting to anyone: just because I thought the process of trial-and-error, comparing datasets, tweaking algorithms… I thought that journey would be interesting to me.

There’s an idea → execution → result loop. AI deletes the execution step, I foresee the inevitable result, and move on. Maybe I even press the magic AI button and see the result. I still do random projects. They’re bigger and faster, plausibly better. But I don’t learn as much, and seeing the result doesn’t hit like it used to. That flow-state zen magic isn’t there.

Not about success

I want to be clear that I’m not talking about who’s going to succeed in the new world. Success used to be a function of brains, now it’s a slightly different function of brains. Being good at writing code is probably correlated with being good at writing prompts. People talk about agency and distribution. Execution. Business-y stuff.

What I’m talking about is just joy. The joy in writing some elegant code. The small joys of small human things. The joy in a great chess move, without the let-down of an engine telling you it was actually suboptimal. Organic, human, meaty stuff.

We’ve got LLMs that are already better than most at coding (narrowly defined). It won’t be long before they’re better than most people at most things. I think it’s going to be quite a hit to Homo sapiens’ ego. I think it’s going to be a weird time for humanity.

Coding was uniquely human, until it wasn’t. I hope the agents enjoy the journey.

[1] ^{go back} There’s a common idea that LLMs are just another layer of abstraction, like the transition from transistors to assembly to C to TypeScript. But I think it’s quite different: all the previous jumps involved solving the same kinds of problems in the same kinds of ways. I’ve programmed with magnetic relays, and it was joyous in just the same way as using MATLAB to solve Newton’s method or Python to make shapes with arrays. Writing in English to an LLM is not the same.

I made an app for my cycling club

Sun, 15 Feb 2026 00:00:00 +0000

And you should too!?

I’m a member of Oxford’s Friendliest Cycling Club, also apparently the “Club of the Year” of 2023.

About a year ago the club had a small administrative crisis around the club ride organisation process. From what I understand (quite little), most clubs are smaller and homogeneous-er. They have set ride times, everyone shows up, they split up and go for rides.

The Condors are different. It’s a big club, the kinds of rides people want to do are quite diverse (slow trundles through the countryside, speedy chaingangs, gravel and mtb) and so during Covid a system of posting things on Facebook evolved. This annoyed everyone who doesn’t like Facebook, so another system using Google Sheets evolved. This annoyed everyone^[1] .

I was at home visiting family at the time and one morning I sat down and vibed out a simple app to take on the role of official club ride organising app. I say “vibed”, but this was before the singularity^{[citation needed]}, so I mostly made it by hand with some components from v0.

It’s a simple CRUD app, club members create rides, others “join” them, leave comments etc. There’s a map and some details about the distance, expected speed, planned coffee stops.

And it’s been a huge success! It’s hard to compare pre- and post-app, but there seem to be more rides happening, more people leading rides, more people trying out new types of rides. It’s a simple thing, but it takes basically no effort from me, costs the club around £10 a month, and seems to be doing a lot of good. I recently added a map of all the routes the club has used recently, hoping to make it easier for aspiring ride leaders to get inspiration.

Condors typical migratory pattern

More than anything, it’s been a success for me. I’d been playing footsie with different clubs in Oxford. I was a member of the Oxford Mountaineering Club, briefly the Cowley Chess Club, and still the Headington Road Runners, although I rarely make it. The app gave me a reason to get more stuck in with one club, along with a bit of free notoriety. And it’s paid back such dividends in really becoming part of a community, building it and getting the joy of being part of something.

We’re in a funny moment where the kind of app available for next to free is shifting from a static Wordpress site, to a fully-featured CRUD app. The marginal cost to build another one is basically 0, but there are still thousands of companies and people and clubs and things out there for whom they’re inaccessible… Most problems are better solved in other ways, but in the right places a bit of software can do wonders, it turns out.

PS: If you’re a member of a cycling club that is also complicated, I’d be happy to hear from you. The app isn’t currently open source (for no reason), but I’d be happy to open it up and help get your club going with it.

[1] ^{go back} An app called Spond was also considered, but it requires downloads and logins and horrible stuff like that. Having our own app lets us carefully titrate the level of friction to new joiners.

The Babysense baby monitor sucks

Tue, 20 Jan 2026 00:00:00 +0000

At the time of writing, the New York Times slash “Wirecutter” recommend the Babysense MaxView monitor as their Top Pick. Many other review sites do the same.

I’m just pissing a small voice into the SEO winds to say that it actually sucks.

The battery barely lasts a few hours, even with the video off.
The build quality is super flimsy.
It’s huge and bulky.
The screen will burn out your eyeballs if you look at it at night, even with the brightness at a minimum.

I have no recommendations for better ones, but at least the HelloBaby we had before (it got chewed up, don’t ask) was lighter, better build quality, longer battery life. It was also much too bright, but at least the screen was smaller so the neighbours didn’t also get retinal blasted.

Charging up for another night of eyeball scalding

Sports in 2024/2025

Fri, 07 Nov 2025 00:00:00 +0000

I’ve been into trail running and mountainy stuff for a long time. I did my first big trail running event in 2013 (the Hout Bay Trail Challenge) and have spent most of the years since then loving running and hiking in the mountains. Because I mostly spent time in places with amazing mountains, I never had a Garmin or used Strava - it just seemed like something that would detract from the pure enjoyment of being outdoors. I also didn’t do many events: I generally preferred just going solo or with friends. So I never had any idea whether I was getting faster or fitter or anything like that.

That all changed when we moved to Oxford. Table Mountain or the Pyrenees are no longer on my doorstep. My mountain mojo is diminished, I even sold my ice axe and crampons. So I finally caved and bought a Garmin Forerunner 255s, and now virtually every run, cycle and swim lands on Strava. Sometimes I’m sad that I don’t more kudo-worthy stats about previous runs and adventures, massive climbs in the Canadian Rockies, 14 hour slogs in the Pyrenees and getting lost in Jonkershoek. But I’m still really attracted to the more intrinsic motivation of not owning a Garmin, and envious of the people that have that.

I’ve also started allowing myself the financial indulgence of entering more events for the motivation and community, and seeing new parts of Oxfordshire and the UK. So I thought I’d note some of those down and try do an update once a year or so. This isn’t every official thing I’ve done, but it’s some of the more interesting (to me, anyway) ones.

March 2024 - Teddy Hall Relay

My first outing with my running club, the Headington Road Runners. A team relay of 7km each, starting and ending at Iffley Sports track. Probably my first time ever running “competitively”, super fun! Ran the loop in 4:00 min/km, which I think was fine. Cute very-studenty prize ceremony afterwards in one of the university halls… Teddy Hall I guess?

March 2024 - Salisbury Plain Easter Epic

A gravel cycling “sportive” organised by Glorious Gravel. We stayed in a cute little inn in Hurstbourne Tarrant called the George and Dragon, which has sadly since closed. They have the start in rolling waves, I had a late morning and started about last. And then spent the day trying to catch everyone else. I’m probably a bit too competitive for this kind of thing; most people are just there to have a good day out… But loads of fun nonetheless, a great route around the high plains (ha) of Salisbury.

May 2024 - Oxford Town and Gown 10k

I trained really hard for this: lots of intervals, 5k at-pace runs, 10k runs with a mix, all the stuff. And then it was crazy hot on the day, I went out at a 3:40 min/km pace and just got slower and slower. Heart rate pegged at Zone 5, I didn’t do very well.

Feeling good. Before the heat hit.

May 2024 - RideLondon 100

A cycling sportive, 100 miles through the scenery north of London. Essex I think? A closed-road event, of which there are only four in the UK! Except they closed the roads on us once and we had to wait behind a bunch of police motorbikes for 20 minutes. After that I got into an amazing paceline and held on to 40-odd kph for as long as I could. On my gravel bike and knobbly 45mm tires. Got funny looks but wasn’t the funniest one there.

June 2024 - OxDuro

It’s like an anarchist’s idea of a bike race. Just a gpx file and some rules. 170km of road/gravel around Oxford, but the “race” only happens across a bunch of timed Strava segments. A big day out. I was in the top 1-2-3 for most of the segments, but messed one of them up by not realising it had started until half-way in… I’d go back and re-do it if I could. Probably the closest I’ve ever come to winning something, it was a pretty small field!

June 2024 - Ridgeway Relay

Taking in a lot of the same scenery as OxDuro. An off-road running relay covering the full extent of the Ridgeway, again with the Headington Road Runners. I was leg 8, peak heat of the day. Pretty tough, I held on to our 1st/2nd position but definitely lost some time. Super complicated logistically getting the team spread out all over the Ridgeway and then home again.

September 2024 - Oxford SwimRun

3.8km swim, 16.5km run. You haven’t heard about swimrun? It’s a very silly sport, but kinda suits me. It’s from the Swedish archipelagos, where it makes sense. You swim and run from island to island, keeping your shoes on in the water and then running in your wetsuit. In Oxford it’s a bit contrived but still fun. The water was only 12 °C! I was the only one to swim in a full length wetsuit, horrible to run in, but at least I didn’t abandon on the swim. I came 12th out of 50 odd!

October 2024 - Oxford Half Marathon

Training for this with a two-month-old baby was tough. But just doing 5-7km runs and the occasional 10-12 km was enough. It was a perfect cold morning, about 5 °C for most of the run. I finished in 1h28min without pushing very hard. Hoping to leave some alpha for a future PR! Super festive and fun, got me so pumped to do more running races.

Feeling great, 100m from the end!

November 2024 - Chiltern Ridge Winter 50k

I think I entered this the night after the Oxford Half. Baby-life continued, the longest runs I got in to train for this were a 30km and a 17km run. It was a lovely run, perfect weather despite the time of year. I managed to sneak in at 4h59min for a pace of 5:58 min/km and comfortably in the top quartile or so. Quite a slog, the last 10km I was getting very slow. Realised I need to get more cushiony shoes for long runs (have since bought some Topo Mtn Racers).

December 2024 - Watlington Winter 10k

Fun small 10k trail running race with two brutal climbs. I started out too far back, as I always do: everyone always seems so serious and I haven’t done enough races to know where I stand. But the first 4km were flat and wide, so I got ahead. And ended up 8th out of 150 or so!

February 2025 - Zwift Racing League

What do you do deep in winter with a baby and little time? Put the bike on a turbo trainer and compare wattage output I guess… Zwift is brutally honest about what counts on mostly flat routes on a perfect surface where you never have to stop or turn: watts. My watts/kg didn’t count for much until one very hilly race, which I won, and then get disqualified from and forcibly promoted to a higher league. I’m not bitter.

May 2025 - Shotover Trail Race

A new event started by some champions from Headington Road Runners. I did quite well I think, pushed very hard around the tough hilly course. I then joined a 120km group cycle a few days later and had to bail after 40km (I had to still get home, note). Body was still wrecked.

June 2025 - Blenheim Triathlon

A “sprint” triathlon: 0.7 km swim, 20 km cycle and 5 km run. I was 87th/1200 on the swim, then 410th on transition. Then 140th on the ride, 500th on the next transition. Then 47th on the run and 71st overall! Triathlon is a bit boring so you gotta talk numbers I guess. I had fun, it was a lovely enclosed route with no cars and thousands of spectators. Very expensive for 1h25 of fun.

I'm in this picture, I think.

June 2025 - OxDuro (again)

Despite hardly cycling in the spring, I thought I could have a shot at really winning it this year. Sadly some pros turned up and smoked the field, but it was still a great day out. I rode with a group of Condors for the first half.

July 2025 - Thames Valley Orienteering

I think “real” orienteering is running around forests in the dark and being very last. But the TVOC kindly organised a bunch of “urban” races for normal people: navigate quite easily around various bits of Oxfordshire and tick off (automatically with an app running on your phone) checkpoints as you. For more or less points. I tried to catch-‘em-all once, turned off my brain and just ran really hard. I didn’t manage.

August 2025 - Oxford Olympic Tri

A cheaper triathlon, and double the distance! The lake was 20 °C or something, which meant we weren’t allowed to used wetsuits. We’re tough! 60th/300 on swim, 280th on T1, 50th on ride, 270th on T2, 29th for run, 48th overall. I made a spreadsheet and I woulda been in the 20s maybe in theory if I wasn’t so damn slow at the transitions. I’m going to work on that before I do another triathlon. And use slick tires on my bike rather than big gravel tires.

September 2025 - Wheatley Cyclocross

My first ever cyclocross race, what a blast! I slightly stupidly started right at the back, so I spend 10 laps passing half the field to come middle-th out of 40 odd people. Easily the hardest 20km I’ve done! I have a new bike with electronic shifting and it tells me I shifted 645 times.

Representing the Condors. There were loads of other cyclists I swear.

Neovim config for 2025

Thu, 15 May 2025 00:00:00 +0000

I wrote about setting up a “modern” Neovim config about 2.5 years ago. In that post, I put a ton of effort into figuring out all the then-new Neovim features, Treesitter, LSP, and converted my config to Lua.

And now, everything has changed again and that post is completely irrelevant. Great. But thankfully since Neovim 0.11 it’s all much easier, half the plugins aren’t needed any more, and the remaining ones are quite simple to setup. But all the configs you see shared online are huge and complicated. So I’m sharing my simple one.

TLDR: Here’s the 180 line Neovim config as a Github Gist.

Getting started

You need to install Neovim v0.11+. On macOs this is brew install neovim. On Ubuntu, the default apt install neovim is too old, but the snap install neovim is up-to-date.
Then place your config in ~/.config/nvim/init.lua.
Then run nvim! There will be lots of messages. Give it a beat, quit, run it again.

Basic settings

This hasn’t changed much. Feel free to ask your neighbourhood LLM what all of these do if you have any questions. Most of these are pretty unambiguously useful. Some are worth familiarising yourself with. The persistent undo history is probably the only clever thing here: killer feature to be able to quit and re-open Neovim and continue undoing/redoing.

-- Basic settings
vim.opt.hlsearch = true
vim.opt.number = true
vim.opt.relativenumber = true
vim.opt.mouse = "a"
vim.opt.showmode = false
vim.opt.spelllang = "en_gb"

-- Leader (this is here so plugins etc pick it up)
vim.g.mapleader = ","  -- anywhere you see <leader> means hit ,

-- use nvim-tree instead
vim.g.loaded_netrw = 1
vim.g.loaded_netrwPlugin = 1

-- Use system clipboard
vim.opt.clipboard:append({ "unnamed", "unnamedplus" })

-- Display settings
vim.opt.termguicolors = true
vim.o.background = "light" -- set to "dark" for dark theme

-- Scrolling and UI settings
vim.opt.cursorline = true
vim.opt.cursorcolumn = true
vim.opt.signcolumn = 'yes'
vim.opt.wrap = false
vim.opt.sidescrolloff = 8
vim.opt.scrolloff = 8

And it keeps going…

-- Title
vim.opt.title = true
vim.opt.titlestring = "nvim"

-- Persist undo (persists your undo history between sessions)
vim.opt.undodir = vim.fn.stdpath("cache") .. "/undo"
vim.opt.undofile = true

-- Tab stuff
vim.opt.tabstop = 2
vim.opt.shiftwidth = 2
vim.opt.expandtab = true
vim.opt.autoindent = true

-- Search configuration
vim.opt.ignorecase = true
vim.opt.smartcase = true
vim.opt.gdefault = true

-- open new split panes to right and below (as you probably expect)
vim.opt.splitright = true
vim.opt.splitbelow = true

-- LSP
vim.lsp.inlay_hint.enable(true)

Plugins

This is a relatively constrained list of useful plugins. In short:

A theme (I like gruvbox, you probably prefer solarized)
A basic status line with lualine
A tree-based file browser panel with nvim-tree
Language Server Protocol (LSP) using mason, mason-lspconfig and nvim-lspconfig.
Code autocomplete with blink.cmp
TreeSitter (syntax highlighting)
A modal command menu with Telescope

This is just enough to get “IDE” functionality and some niceties that you’re probably used to from VSCode/similar.

local plugins = {
  { "nvim-lua/plenary.nvim" },       -- used by other plugins
  { "nvim-tree/nvim-web-devicons" }, -- used by other plugins

  -- Gruvbox theme (feel free to choose another!)
  { "ellisonleao/gruvbox.nvim" },
  
  { "nvim-lualine/lualine.nvim" },  -- status line
  { "nvim-tree/nvim-tree.lua" },    -- file browser

  -- Telescope command menu
  { "nvim-telescope/telescope.nvim" },
  { "nvim-telescope/telescope-fzf-native.nvim", build = "make" },

  -- TreeSitter
  { "nvim-treesitter/nvim-treesitter", build = ":TSUpdate" },

  -- LSP stuff
  { 'mason-org/mason.nvim' },          -- installs LSP servers
  { 'neovim/nvim-lspconfig' },         -- configures LSPs
  { 'mason-org/mason-lspconfig.nvim' },-- links the two above

  -- Some LSPs don't support formatting, this fills the gaps
  { 'stevearc/conform.nvim' },

  -- Autocomplete engine (LSP, snippets etc)
  -- see keymap:
  -- https://cmp.saghen.dev/configuration/keymap.html#default
  {
    'saghen/blink.cmp',
    version = '1.*',
    opts_extend = { "sources.default" }
  },
}

Then you need to bootstrap lazy (the plugin manager) and install the plugins:

local lazypath = vim.fn.stdpath("data") .. "/lazy/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
  vim.fn.system({
    "git",
    "clone",
    "--filter=blob:none",
    "https://github.com/folke/lazy.nvim.git",
    "--branch=stable",
    lazypath,
  })
end
vim.opt.rtp:prepend(lazypath)
require("lazy").setup(plugins)

Some plugin configuration

Now that the plugins are installed, you need to configure them. Google these plugins if you want to see how to customise them, but the defaults are good enough to get started with!

vim.cmd.colorscheme("gruvbox")  -- activate the theme
require("lualine").setup()      -- the status line
require("nvim-tree").setup()    -- the tree file browser panel
require("telescope").setup()    -- command menu

TreeSitter

The first slightly complicated one! But much simpler than it was a few years ago. You can see the full list of available TreeSitter parsers here. TreeSitter most obviously improves syntax highlighting. But it also does other more subtle stuff like improve code folding, and enables various other plugins (like treesitter-context) to do their thing. I’ve included a few parsers in the config below, but go to the link above and add as many as you like!

require("nvim-treesitter.configs").setup({
  ensure_installed = {
    "typescript",
    "python",
    "rust",
    "go",
    -- etc!
  },
  sync_install = false,
  auto_install = true,
  highlight = { enable = true, },
})
-- some stuff so code folding uses treesitter instead of older methods
vim.opt.foldmethod = "expr"
vim.opt.foldexpr = "nvim_treesitter#foldexpr()"
vim.opt.foldlevel = 99

LSP

This is the bit that was basically impossible in Neovim 4 years ago, a huge pain to set up 2 years ago (see my previous blog post), and now pretty straight-forward. All you need is the plugins installed above and these few lines of config. You can see a full list of available servers here. It’s really worth scrolling through and adding whichever ones you like the look of, you might see some you don’t expect!

Note that these require associated language toolchains to be installed, so don’t add eslint if you don’t have npm installed! The installation will fail and you’ll get lots of angry messages in a very small confusing status bar.

require("mason").setup()
require("mason-lspconfig").setup({
  ensure_installed = {
    "gopls",
    "basedpyright",
    "eslint",
    "ruff",
    "rust_analyzer",
    -- etc!
  },
})

You can find the default keybindings for LSP stuff here. Some of the main ones:

]d and [d to jump between type errors, lint problems etc
K to give docs for the symbol under the cursor
Ctrl-] to goto to the definition of the symbol under the cursor
Ctrl-w d to show detailed diagnostics for errors, lints
grr goes to uses/references
grn does a smart rename
gra opens code actions (fixing imports, stuff like that)

Code formatting

Many languages can be formatted directly by their LSP server (eg ruff for Python), but others still just need an old-school CLI formatter.

For these cases we have conform.nvim. It will try to use the formatters specified, falling back to an LSP if there isn’t one. So you can set up a command to do a conform format (see further down) and it will just work. If you want any linting that your LSP doesn’t provide, there’s also nvim-lint, which does a similar thing but for linting.

require("conform").setup({
  default_format_opts = { lsp_format = "fallback" },
  formatters_by_ft = {
    typescript = { "prettier" },
    typescriptreact = { "prettier" },
    json = { "prettier" },
    -- etc
  },
})

Autocomplete

We already set up blink.cmp in the plugins, and left it with its default key bindings. You can pretty much just use it, but some useful keys are:

Ctrl-space open the autocomplete menu, or show docs/signature for the current option
Ctrl-y accept the current option
Ctrl-e hide the autocomplete

Key bindings

Everything until this point is pretty universal. If you don’t already have mega-strong vim opinions, just use what I’ve shared above and you’ll probably be quite happy.

Key bindings are obviously quite personal… I’ll share my setup and you can pick and choose what you like.

I already set the leader key to , up at the top. Repeating it here. This is basically a prefix for many commands. Eg you’ll see below that my formatting command is <leader>fo, which means I hit ,fo (one after the other, not at the same time!) to run my formatting command. , is quite a popular choice. \ is the default.

vim.g.mapleader = ","

Then two more customisations that are probably less common. The first lets you hit <space> instead of entering : to enter a command. So eg to open the Lazy dialog to check your installed plugins, you can enter <space>Lazy<Enter>, which is slightly easier than a colon… The second is more particular. The default key for undo in neovim is u, but for redo it’s Ctrl-R, which is horrible. So I map q to redo. So I can hit u q in quick succession to go back and forth.

vim.keymap.set("n", "<space>", ":")
vim.keymap.set("n", "q", "<C-r>")

These basically just set n to always be next search result down the page, and N always up. Same for ' forward when character seeking and ; backwards. The default has these operating relative to the direction you started searching in, which can be hard to keep track of.

vim.keymap.set("n", "n", "v:searchforward ? 'n' : 'N'", { expr = true })
vim.keymap.set("n", "N", "v:searchforward ? 'N' : 'n'", { expr = true })

vim.keymap.set({ "n", "v" }, ";", "getcharsearch().forward ? ',' : ';'", { expr = true })
vim.keymap.set({ "n", "v" }, "'", "getcharsearch().forward ? ';' : ','", { expr = true })

Two little commands for toggling line numbers and word wrapping:

vim.keymap.set("n", "<leader>n", ":set nonumber! relativenumber!<CR>")
vim.keymap.set("n", "<leader>w", ":set wrap! wrap?<CR>")

Moving between and resizing splits. The normal command to e.g. move to the split below is quite tedious. This makes it just Ctrl-j. Similar for the other directions. You can do something similar for resizing if you want… or just use the mouse 😉.

vim.keymap.set("n", "<C-j>", "<C-W><C-J>")
vim.keymap.set("n", "<C-k>", "<C-W><C-K>")
vim.keymap.set("n", "<C-l>", "<C-W><C-L>")
vim.keymap.set("n", "<C-H>", "<C-W><C-H>")

You’ve already installed nvim-tree, now you need some commands to make it work. I find myself using Ctrl-f to open the file browser on the current file repeatedly. Then I hit enter, choose which split to open the file in, and hit Ctrl-c to close the file browser again.

vim.keymap.set("n", "<C-t>", ":NvimTreeFocus<CR>")
vim.keymap.set("n", "<C-f>", ":NvimTreeFindFile<CR>")
vim.keymap.set("n", "<C-c>", ":NvimTreeClose<CR>")

Formatting. This does the conform stuff, falling back to LSP stuff.

vim.keymap.set("n", "<leader>fo", require('conform').format)

Telescope. Don’t skip this one! These are super useful commands that will change how you navigate a codebase. You don’t even need the file browser with these. Hit ,ff then start typing the name of a file. If the file you want isn’t checked in to a repo, ,fa does the same for other files. ,fg gives you instant ripgrep across your files. ,fb lets you quickly switch between recently open buffers (this is what vim people call files). And ,fh is probably the only way you’ll ever get comfortable finding help from within Neovim…

local tele_builtin = require("telescope.builtin")
vim.keymap.set("n", "<leader>ff", tele_builtin.git_files, {})
vim.keymap.set("n", "<leader>fa", tele_builtin.find_files, {})
vim.keymap.set("n", "<leader>fg", tele_builtin.live_grep, {})
vim.keymap.set("n", "<leader>fb", tele_builtin.buffers, {})
vim.keymap.set("n", "<leader>fh", tele_builtin.help_tags, {})

The full config

Here’s the 180 line Neovim config as a Github Gist.

And here’s my actual current Neovim config, which is a bit longer with one or two extra bits that you probably don’t need.

A boring latency investigation

Sat, 14 Dec 2024 00:00:00 +0000

I built a toy Next.js app as an excuse to play around with some stuff (like React 19 forms). I had a form that would create a new object and then redirect to it. Mostly submit-to-loaded was <200ms, but occasionally it would spike to over one second, which is very much loading spinner territory. It seemed to mostly happen after not using the app for a few minutes. So I decided to investigate a bit.

TLDR: unsurprisingly, the culprit was my mediocre internet slash DNS server. But it made for a fun investigation anyway.

Traces

First… well first I bumped the server and database specs to at least have a single core each (logical or physical? we’ll never know). To at least reduce the likelihood that it’s all just resource contention with some other persons’s app.

Then I setup Sentry as an easy way to get some distributed full-stack traces. These suggested that node-postgres’ connect was a potential culprit. So I set up a connection pool, configured it to never terminate connections, and swapped in pg-native. There was some marginal improvement but I was still seeing the occasional spike in latency. I set up a simple API with some manual traces to remove React rendering and the data transfer as possible culprits. An example trace is shown below, and I never saw the time for the core logic of this route go above 35 ms. I wasn’t being at all careful with queries (using Drizzle, fwiw) and this proves that’s the right decision: each one is ~2 ms, so I could add 20 more and it would hardly matter.

I think this is also shows the advantage of colocating your server and database (on Render, in this case). There are plenty of fancy modern database/backend tools (Supabase, Turso, Neon and 20 more) but if you just use them like a normal database from your backend, and aren’t careful about where they’re physically located, you might just add a bunch of milliseconds to every single query.

Synthetic tests

So I wondered if the actual navigation or React-y stuff was slowing things down and set Checkly to work filling out and submitting my form every 5 minutes. Checkly doesn’t seem very well set up for performance monitoring (unless I also ship my traces to them), but I never saw the full submit-reload go much above 250 ms.

const { expect, test } = require('@playwright/test')

test.setTimeout(5000)
test('form-submit', async ({ page }) => {
  await page.context().addCookies([{ ... }])
  await page.goto('https://.../new')
  await page.locator('button[aria-label="Add"]').click()
  await page.locator('div[data-value="Foo"]').click()
  await page.locator('textarea[name="content"]').fill('Hello')

  const start = Date.now()
  const [submitResponse] = await Promise.all([
    page.waitForResponse(response =>
      response.url().includes('/new') &&
      response.request().method() === 'POST'),
    page.locator('form').evaluate(form => form.submit())
  ])
  expect(submitResponse.status()).toBe(303)
  await page.waitForLoadState('networkidle')
  
  const end = Date.now()
  console.log({ duration: end - start })
})

So clearly the form is fine… and my internet is wonky. But I wanted to see what was actually going on, and neither Sentry nor Checkly provided enough detail on what was sucking up the time when there was a slow response.

Curl

I wrote (well, Claude wrote) a little script to GET the API route, then immediately again, then sleep for 5 minutes and repeat. Wondering whether it would be slower after 5 minutes of inactivity. This is easy in my case, because no one else is using my toy app 😄.

while true; do
    curl 'https://.../api' -w '%{json}' \
      | jq -s 'add + {clock: now}' \
      >> data.json
    # repeat
    sleep 300
done

I ran this for an hour or two on my laptop over WiFi and my mediocre internet. I also ran it on a Hetzner box in Falkenstein, about a four-hour drive from my app server in Frankfurt. See the results below, showing the histogram of time taken (in milliseconds) for each operation. “Core logic” is measured and returned by the app, the rest are recorded by curl.

A couple of interesting things (to me at least):

The Core logic slowed down by about 10ms because I removed pg-native. So the eight queries are each about a millisecond slower.
The initial TCP connection was always <10ms on Hetzner, but all over the place on my laptop. Presumably because of the lower latency connection for the back-and-forward (thanks zoomie for pointing this out).
If you look closely at the DNS lookup for my laptop, you can see the tail lifting again at 200 ms. I think this is ultimately what was causing my latency: I’m currently experimenting with Eero’s ad-blocking, and presumably it has a low cache TTL, so it’s looking up the domain again after a few minutes. But I’m surprised Hetzner also had a tail at almost 100ms.
The SSL handshake is about four times faster on my laptop: presumably my Apple M2 running circles around a little Hetzner single-core box (again, thanks zoomie).
The time to Send first byte I think are much the same, the spread is probably just wider for Laptop because I got far more data points (sorry, Science). Note that this excludes the DNS/TCP/SSL steps. Median of 100 ms in either case, which isn’t bad for a little Node backend doing a bunch of database queries.
And then the Total time is much the same.

All times in milliseconds

Progressive Forms with React 19

Wed, 11 Dec 2024 00:00:00 +0000

So, React 19 is here! And Server Components and Forms are now the blessed way. It’s like old-school backend-first web-dev all over again but with two great advantages:

Full-stack type-safety
You can inject client-side interactivity when needed

In the beginning, the only way with React was single-page-apps (SPAs) with mountains of clunky state and AJAX. Things improved and Routers were invented, and useQuery made data fetching and management easier. But state is hard! Every read or mutation has several layers where state can persist, and subtle interdependencies. Not to mention there’s no graceful downgrade if JavaScript isn’t available, and you have to ship loads of the stuff to make it all work on the client.

Remix pushed hard on forms and backend routing. Hit a form, reload. Next.js saw that this was good, so they created the App router and a nightmare transition for their users, but the dust is now settling (at least for greenfield projects). And now all this stuff has found its way into React itself, in the form React Server Components and new Form tooling.

But they’re new and slightly weird and the best patterns for some basic things still aren’t obvious. Specifically:

Data validation
Error handling
Maintaining state
Optimistic loading
Client validation

So I’m sharing what I think is a pretty good setup for fancy React 19 forms. I’m going to assume you’re already familiar with with Server Components and "use server" and forms in general. If not, it’s worth reading the Next.js docs on the topic and following the interesting links where they lead. As always, you can just skip to the repo carderne/react-forms (or scroll to the bottom) if you prefer.

This post is structured according to the five points above, with the idea that you can step off at any point: each step adds more goodies, but also more complexity and more client-side stuff.

Data validation

A basic server action with some zod validation. You can also use zod-form-data to make some of this more ergonomic.

// actions.ts

"use server";

import { redirect } from "next/navigation";
import { z } from "zod";

const schema = z.object({
  todo: z.string().min(3, { message: "Please write more!" }),
});

export async function addItemAction(
  formData: FormData
) {
  const obj = Object.fromEntries(formData);
  const data = schema.parse(obj);
  console.log(data); // or, you know, persist to DB
  redirect("/");
}

And a form to use it.

// form.tsx

import Form from "next/form";
import { addItemAction } from "./actions";

export function ItemForm() {
  return (
    <Form action={addItemAction}>
      <input name="todo" />
      <button type="submit">
        Submit
      </button>
    </Form>
  )
}

That works pretty well. And it’s so simple compared to “modern” ways of mutating data. You persist the data, redirect as needed, and wherever the user lands will load the data. This means all the complex state stuff stays in the database, where it belongs.

Error handling

The only problem with the code above is that it does nothing for errors. For example, the todo field requires a minimum of 3 characters. If the user enters only two, it will throw and the user will get an error page.

So of course you use schema.safeParse(...) but then what do you do with the error? This is where React 19 comes in, with the new useActionState hook. It wraps your action and gives you a state object where your server code can return errors and messages for the client.

On the backend, we return an object with an errors field (you’re obviously free to call this whatever you want). And we can use some handy zod methods to create error messages keyed to the schema fields.

// actions.ts

"use server";

import { redirect } from "next/navigation";
import { z } from "zod";

const schema = z.object({
  todo: z.string().min(3, { message: "Please write more!" }),
});

export interface AddItemState {//+
  errors: { todo: string[] }//+
}//+

export async function addItemAction(
  _state: AddItemState,//+
  formData: FormData
): Promise<AddItemState> {//+
  const obj = Object.fromEntries(formData);
  const { data, error } = schema.safeParse(obj);
  if (error) {//+
    return {//+
      errors: error.flatten().fieldErrors//+
    }//+
  }//+
  console.log(data);
  redirect("/");
}

And the form. When there’s a validation error from entering a too-short string, it’ll display the “Please write more!” message above the input. We also get a nice Loading state with the pending value from the hook.

// form.tsx

"use client"; // this must be client side now//+

import Form from "next/form";
import { useActionState } from "react";//+
import { addItemAction, type AddItemState } from "./actions";//+

export function ItemForm() {
  const [state, formAction, pending] = useActionState<AddItemState, FormData>(//+
    addItemAction,  // our action//+
    {},             // and default state//+
  );//+
  return (
    <Form action={formAction}>//+
      <div>{state.errors?.todo?.map((e) => e)}</div>//+
      <input name="todo" />
      <button type="submit">
        {pending ? "Loading" : "Submit"}//+
      </button>
    </Form>
  );
}

Maintaining state

That’s a big improvement but there’s still one problem: every time you hit a validation error, the form will reset. This is to maintain parity with native forms, which reset as soon as they’re submitted. There’s a massive thread at facebook/react#29034 discussing this^[1] , with two main approaches shared for getting around this:

Submit the form manually using onSubmit
Return all the original form data as part of the AddItemState return value.

I’m going to show the second option, mostly because it lets us get further without resorting to imperative logic, and it seems a bit more elegant. But with very complex (multi-step) forms, some combination of the two could be required.

Here’s our code again. I added some type helpers that you can re-use wherever you have a form. These take any zod schema and create a neat return type with the FormData plus an array of error messages for each field in the schema.

// types.ts

type InferFieldErrors<T extends z.ZodType> = {
  [K in keyof z.infer<T>]?: string[] | undefined;
};
export type ActionState<T extends z.ZodType> = {
  formData?: FormData;
  errors?: InferFieldErrors<T>;
};

And here’s the action. The key differences being the fact that we now return formData in the error path (and the new AddItemState created using the helpers above).

// actions.ts

"use server";

import { redirect } from "next/navigation";
import { z } from "zod";
import { type ActionState } from "./types";//+

const addItemSchema = z.object({
  todo: z.string().min(3, { message: "Text must be longer" }),
});
export type AddItemState = ActionState<typeof addItemSchema>;//+

export async function addItemAction(
  _state: AddItemState,
  formData: FormData,
): Promise<AddItemState> {
  const formDataObj = Object.fromEntries(formData);
  const { data, error } = addItemSchema.safeParse(formDataObj);
  if (error) {
    return {
      formData, // we return the formData as-is//+
      errors: error.flatten().fieldErrors,
    };
  }
  console.log(data);
  redirect("/");
}

And the form. The type cast on the defaultValue isn’t wonderful, and there are cases with <select> (not to mention file uploads) that will need more careful consideration (and probably just being controlled components client side).

// form.tsx

"use client";

import Form from "next/form";
import { useActionState } from "react";
import { addItemAction, type AddItemState } from "./actions";

// imagine we have some `item` being passed to the component
export function ItemForm({ item }: { item: string }) {//+
  const [state, formAction, pending] = useActionState<AddItemState, FormData>(
    addItemAction,
    {},
  );
  return (
    <Form action={formAction}>
      <div>{state.errors?.todo?.map((e) => e)}</div>
      <input
        name="todo"
        // and we show the formData if we have it, otherwise `item`//+
        defaultValue={(state.formData?.get("todo") as string) ?? item}//+
      />
      <button type="submit">
        {pending ? "Loading" : "Submit"}
      </button>
    </Form>
  );
}

Now when the action returns an error, it will also return the data we sent it. It’s a bit silly flinging data back-and-forth like that, but so long as it’s relatively small it should be fine. Also the returned formData doesn’t need to be re-serialised, so that’s nice.

Optimistic loading

Some Forms redirect somewhere (e.g. create a new thing, then redirect to it). But many are adding stuff to some form of table or list, and you just want to see your new item added to that. In client-side React, you’d typically have some useState and then push items into an array. Or with Tanstack, have a useMutation that invalidates a useQuery cache.

And with our approach, instead of redirecting as we’ve been doing, we’ll rather revalidate the path (in Next.js-speak). We’d also like to add an optimistic update to the Form: as soon as you hit submit, optimistically add the item to the table/list, then forward it to the backend action, and then reload the component with the new data. Note that this can force you down the path of validating your inputs on the frontend (as well as the backend), because otherwise you’ll see things appearing and then disappearing if there’s a validation error on the backend.

The way we do this is with React 19’s new useOptimistic hook. If you try to use it with a Server Component-heavy approach, you’ll immediately notice a problem: because useOptimistic is about sharing state between your Form and your Table, you’ll have to bring it higher in the component tree and force the containing component to be rendered client side. Not ideal. But fortunately there’s an elegant solution: Context. I’m not going to show all the code, but following this handy blog post, we can do some clever stuff.

Firstly, some pretty standard context code. This is almost 100% boilerplate, so you can use the same pattern in various places.

// optimistic.tsx

"use client";

import { createContext, useContext, useOptimistic } from "react";

// The type here will need to include everything you show in the
// table, which may be a superset or subset of the zod schema
type Item = { ... }

type ContextType = {
  optimistic: Item[];
  addOptimistic: (_: Item) => void;
};

const Context = createContext<ContextType | undefined>(undefined);

export function OptimisticProvider({
  children,
  items,
}: {
  children: React.ReactNode;
  items: Item[];
}) {
  const [optimistic, addOptimistic] = useOptimistic(
    items,
    (state: Item[], newItem: Item) => {
      return [...state, newItem];
    }
  );

  return <Context value={{ optimistic, addOptimistic }}>{children}
  </Context>;
}

export function useOptimisticContext() {
  return useContext(Context);
}

Then in our containing page, which can remain a server component.

// page.tsx

import { OptimisticProvider } from "./optimistic";
import { ItemForm } from "./form";
import { ItemTable } from "./table"; // we'll see this shortly
import { getItems } from "./db"; // pretend this exists

export default function Home() {
  const items = getItems();
  return (
    <OptimisticProvider items={items}>
      <ItemForm />
      <ItemTable />
    </OptimisticProvider>
  );
}

Now our Form component is quite different. If you want to pull out the Submit menu into a separate component (quite likely), you can use the other new useFormStatus hook to access the containing form’s pending state without having to pass props around.

// form.tsx

"use client";

import Form from "next/form";
import { useActionState, useRef } from "react";//+
import { useOptimisticContext } from "./optimistic";//+
import { addItemAction, type AddItemState } from "./actions";

export function ItemForm() {
  const ref = useRef<HTMLFormElement>(null);//+
  const { addOptimistic } = useOptimisticContext();//+
  const [state, formAction, pending] = useActionState<AddItemState, FormData>(
    addItemAction,
    {},
  );
  const optimisticAction = (formData: FormData) => {//+
    addOptimistic({//+
      todo: formData.get("todo") as string,//+
    });//+
    ref.current?.reset();//+
    formAction(formData);//+
  };
  return (
    <Form ref={ref} action={optimisticAction}>//+
      <div>{state.errors?.todo?.map((e) => e)}</div>
      <input
        name="todo"
        defaultValue={(state.formData?.get("todo") as string) ?? ""}
      />
      <button type="submit">
        {pending ? "Loading" : "Submit"}
      </button>
    </Form>
  );
}

The only change we make to the server action is to revalidate instead of redirect.

// actions.ts

<snip>
  revalidatePath("/");//+
  return { };//+
}

And finally we can use our optimistically-updated data in some kind of table or list component:

// table.tsx

"use client";

import { useOptimisticContext } from "./optimistic";

export function ItemTable() {
  const { optimistic } = useOptimisticContext();
  return (
    <div>
      {optimistic.map((item, idx) => (
        <div key={idx}>{item.todo}</div>
      ))}
    </div>
  );
}

Client validation

I initially stopped there, preferring to keep data validation to the server to keep the client lightweight and (relatively) simple. But I got an elegant suggestion on how to add client-side data validation, and since most developers will probably end up needing this anyway, we might as well try to do it nicely.

First we pull out the validation logic from the action to somewhere where it can be used on the backend and frontend.

// validate.ts

import { z } from "zod";
import { type ActionState } from "./types";

const addItemSchema = z.object({
  todo: z.string().min(3, { message: "Text must be longer" }),
});
export type AddItemState = ActionState<typeof addItemSchema>;

export function validateItem(
  formData: FormData,
): AddItemState {
  const formDataObj = Object.fromEntries(formData);
  const { data, error } = addItemSchema.safeParse(formDataObj);
  if (error) {
    return {
      formData,
      errors: error.flatten().fieldErrors,
    };
  }
  return { data };//+
}

Since we’re now returning { data } in the success path, we’ll need to update the ActionState to include that (previously it was only returning bad news).

// types.ts

type InferFieldErrors<T extends z.ZodType> = {
  [K in keyof z.infer<T>]?: string[] | undefined;
};
export type ActionStateError<T extends z.ZodType> = {//+
  data?: never;//+
  formData?: FormData;
  errors: InferFieldErrors<T>;
};

export type ActionStateSuccess<T extends z.ZodType> = {//+
  data: z.infer<T>;//+
  formData?: never;//+
  errors?: never;//+
};//+

export type ActionState<T extends z.ZodType> =//+
  | ActionStateSuccess<T>//+
  | ActionStateError<T>;//+

Now we update our action to use this new validation function:

// actions.ts

"use server";

import { redirect } from "next/navigation";
import { validateItem, type AddItemState } from "./validate";//+

export async function addItemAction(
  _state: AddItemState,
  formData: FormData,
): Promise<AddItemState> {
  const res = validateItem(formData);//+
  if (res.errors) return res;//+
  console.log(res.data);//+
  revalidatePath("/");
  return { errors: {} };//+
}

And finally, we now use that exact same validation function in the form. Note that we apply it before we do anything optimistic, so we avoid embarrassing flashes of bad data!

// form.tsx

"use client";

import Form from "next/form";
import { useActionState, useRef } from "react";
import { validateItem } from "./validate";//+
import { useOptimisticContext } from "./optimistic";
import { addItemAction, type AddItemState } from "./actions";

export function ItemForm() {
  const ref = useRef<HTMLFormElement>(null);
  const { addOptimistic } = useOptimisticContext();
  const [state, formAction, pending] = useActionState<AddItemState, FormData>(
    (prev, formData) => {//+
      const res = validateItem(formData);//+
      if (res.errors) return res;//+
      addOptimistic(res.data);//+
      ref.current?.reset();//+
      return addItemAction(prev, formData);//+
    },//+
    { errors: {} },//+
  );
  return (
    // JSX is unchanged
  );
}

Aaaaand that’s it! You now have a fully-featured form that degrades gracefully in the absence of JavaScript, handles validation and errors with aplomb, and gives nice snappy SPA-esque optimistic loading of entered data. That’s a lot of code to submit a form, but you’ll notice most of it can be squirrelled away into a library and re-used.

You can find the full code at carderne/react-forms. I think the ActionState types and approach are relatively elegant, but I’m curious to see what other patterns emerge. I’m really enjoying Server Components: it seems like after all the misadventures of needless SPAs and Redux, there’s a happy path for lightly stateful multi-page apps, that can gracefully upgrade and downgrade as required.

And hopefully the fact that it’s been upstreamed into React will make it easier for Remix (slash React Router) and other to continue to offer compelling alternatives to Next.js, who are worrying dominant.

Bonus: Generic optimistic hook

I’ve found that the above optimistic.tsx context code is so boilerplate that we can actually just write it once and use it all over the place. Basically just take the code above and put it in a generic function that returns the <Context> provider and the useOptimisticContext.

// lib.tsx
import { createContext, useContext, useOptimistic } from "react";

export function createOptimisticContext<T>() {//+
  type ContextType = {
    optimistic: T[];//+
    addOptimistic: (_: T) => void;//+
  };

  const Context = createContext<ContextType | undefined>(undefined);

  function OptimisticProvider({
    children,
    items,
  }: {
    children: React.ReactNode;
    items: T[];//+
  }) {
    const [optimistic, addOptimistic] = useOptimistic(
      items,
      (state: T[], newItem: T) => {//+
        return [...state, newItem];
      }
    );

    return <Context value={{ optimistic, addOptimistic }}>{children}</Context>;
  }

  function useOptimisticContext() {
    const context = useContext(Context);
    if (typeof context === "undefined") {
      throw new Error("Context must be used within provider");
    }
    return context;
  }

  return { OptimisticProvider, useOptimisticContext };//+
}

Then the optimistic provider code for your page just becomes:

// optimistic.tsx
"use client";

import { createOptimisticContext } from "./lib";

type Item = { ... }

export const { OptimisticProvider, useOptimisticContext } =
  createOptimisticContext<Item>();

[1] ^{go back} Enjoyed seeing some Next.js docs writers chiming in that they were also scratching their heads a bit.

September 2024

Wed, 25 Sep 2024 00:00:00 +0000

I don’t normally share much personal stuff on this blog because… well the internet is a gross place and an airgap feels healthy. But we became parents around two months ago, so that’s pretty important context for what’s been going on here. I left my job at Translucent, am enjoying (slash surviving) a few months of paternity leave, and starting a new job (more on that another time) in November.

Books

Not too surprisingly, I’ve read very few books recently. It probably takes 30 uninterrupted minutes to to get into a (decent) book, and those just don’t exist right now. Well they do, but I use them to get outside and do exercise. The only novel-length thing I’ve read is Bill Bryson’s The Road to Little Dribbling, which was easily the worst book of his that I’ve read.

This book-shaped void has been filled by two things. The first is an absolute mountain of… books. Books of the “how to make your baby sleep well and be happy” variety. Most of these are pretty terrible. Some (like Emily Oster) make an effort to stick to things that are supported by evidence. Others (Brain Rules for Baby, Wonder Weeks) pretend to do that, but then just say whatever they want and cite widely refuted studies. Still others (Weissbluth, Gina Ford, Suzy Giordano) are more vibes-based, but will get you kicked out of certain social groups for even admitting you’ve read them. Finally there are a few (Harvey Karp, Your Baby Week by Week) which seem just about sane and relatively socially acceptable. I must have learned something going through all of these, but I can’t say what it is exactly.

The second thing is listening to podcasts.

Podcasts

I somehow never got into podcasts. But now I spend a lot of time walking slowly and jiggling a baby, so I’ve suddenly got on board. I’ve also been thinking about what I do and don’t enjoy about the podcast format. Because it occurred to me that I love talk radio. When I lived in the US, my favourite thing on road trips was to tune into various local AM radio stations and hear about the crazy stuff bothering small-town America. Even in the UK I’ve often enjoyed the super parochial stuff on offer: should the military be allowed to have horses; does using cash make it easier to budget.

Part of it is the same reason Netflix sucks: the curse of choice. You turn on the radio, and listen a bit, change channels, do your chores, listen a bit more. On the podcasts app, I must first choose from hostile-looking list of channels, then try to judge from the title whether I want to listen to that thing for 1-3 hours. Stressful. Part of it is also just the format: with written words (like these ones) you can effortlessly pause, skim (what you’re doing now, probably), re-read. Slow down. Speed up. Anyway, I got some recommendations so at least that part was resolved.

First I accidentally listened to The Ancients: Moses and the Exodus, which was short and interesting. I’ve read most of the Bible and recently watched The Prince of Egypt, so I feel like I know my stuff here. I’ve generally taken the Wikipedia-approved view that the Exodus probably never happened, but I enjoyed how the podcast looked quite deeply into figuring out what did happen to inspire the story, and to whom. No clear answer, but definitely enjoyed this. Pretty short, I know the topic well so it was easy to follow. 3 stars.

What I was actually looking for was Fall of Civilizations: The Assyrians. Partially my brain isn’t running on full steam right now. But it’s a three hour slog through some pretty detailed stories. And even though I’ve read quite a bit about these people and this time period, I just struggle to keep track of what the hell is going on. With a book, I’d pause; re-read that paragraph; flip back to the map or timeline; quickly Google something for a reference. Basically, repeatedly zoom in and out to contextualise and situate. With the podcast… if the voice in my ear doesn’t do this for me, and I suddenly can’t remember why I must care about Esarhaddon, then I’m at a bit of a loss. But I think maybe I just need to lower my expectations: treat it more like entertainment, like a chat with Ezra Klein or whoever, and just enjoy the story. I haven’t finished it yet, so, no stars (yet).

Then I picked up a nice theme around children with a recommendation for Ezra Klein: On Children, Meaning, Media and Psychedelics. The psychedelics were just a marketing ploy, they hardly discussed that. Ever since I read How to Change Your Mind I’ve been fascinated by the idea that babies are basically on a permanent, slowly decaying psychedelic trip. It gives me a lot more empathy for them. They mostly talked about hyper-attention-sucking YouTube shows designed for young kids. This is several years in the future for us, but I really enjoyed the distinction between pleasurable (even in a purely hedonistic way) and attention-demanding. Scrolling Twitter is the latter, but rarely the former. I also really liked that they discussed this without reference to whether or not watching YouTube shows was bad for your kid’s outcomes. Just, like, is it actually as pleasurable as playing with a stick or whatever? Ditto adults: what is it exactly that makes you feel wholesome after reading a book, or a magazine, or whatever it is for you, which you don’t get from scrolling. 4 stars (out of 5).

Next up was Conversations with Coleman and I spotted an episode called Is Therapy Bad for You, which seemed to mostly focus on childrent. I think Coleman’s schtick is being centrist and not afraid to ask the big questions etc. This is fine, but it really raises my expectations of rigour. I enjoyed their discussion, which discussed the risks of too much therapy in children, even though it was mostly them patting each other on the back about how right they are. They mentioned the over-pathologising of mental conditions, but instead of looking into the cultural taxonomy of mental disorders, they just happily agreed that the ones they thought were dumb were fake, while the important scary ones are real. If their vibe wasn’t so self-serious I’d forgive it, but it isn’t so I won’t. 2/5.

To round it out I listened to Conversations with Tyler: Paul Bloom on the Psychology of Children, which was much better! Tyler Cowen’s interviewing style is absolutely bizarre, like a gatling gun firing questions from a random sentence generator. But it seems to work and Paul Bloom (and presumably his other guests) was either smart enough or prepped enough to respond well. Unlike Coleman Hughes, the conversation was more focussed on the guest and what he thought, and less on being “right”. So it doesn’t invite nitpicking in the same way and you can just enjoy the conversation. I can’t remember anything, but I enjoyed it a lot, and I guess that’s the point. 4/5!

This is totally unrelated, but whenever I listen to Tyler Cowen, in my head I’m picturing Vizzini (Wallace Shawn) from The Princess Bride.

I think I’m figuring out podcasts. Let’s see what next month brings.

Why Britain has stagnated

I also read some things online. The Substack website is terrible (at least on iOS Safari), so I finally caved and downloaded their app. It’s not much better: the number of times it completely forgets what I was reading 5 minutes ago and provides no obvious way to find it again (EDIT: I found the swipe-left thing to see history!). But it does do some social-media-esque stuff which makes it easy to find other things in your network of interests.

Mostly I’ve been slowly reading through Foundations by Ben Southwood and co, because it seems like required reading this month. It’s full of depressing stats like this one:

The planning documentation for the Lower Thames Crossing, a proposed tunnel under the Thames connecting Kent and Essex, runs to 360,000 pages, and the application process alone has cost £297 million. That is more than twice as much as it cost in Norway to actually build the longest road tunnel in the world.

I liked David Hugh-Jones’ brief commentary. Also some interesting pushback on the energy aspects from James Murray. Waiting for Tom Forth to weigh in. This discussion has me curious about the political make up of the UK, where I’ve lived for four years now. Labour ostensibly supports increasing investment in housing, energy and transport. But the paper in general is probably further to the right than Labour. The Tories have demonstrated that they’re not interested in sensible policy, the Greens are against building things, and I’m not sure the Lib Dems have any policy positions just yet.

I’m still learning about this country, clearly, especially when it comes to small-L labour and how the unions are getting on. In that vein, I read an interesting US-focused piece arguing that unions are not the way to help workers. I don’t have a good enough grasp of the history and current situation to weigh in, but I’m enjoying some of the discussion around this.

It’s also a bit topical suddenly, as US east-coast dockworkers are currently threatening one of the biggest dock strikes in US history. The same guy who (maybe?) helped solve the Los Angeles port gridlock in 2021 has an interesting thread about the potential impacts of an east-coast strike. The best bit is this video from the president of the International Longshoremen’s Association.

Legendary.

Most interesting to me was the list of demands (2 minutes into the video) in the screenshot above. I had a naive idea that the rejection of automation was either an historical thing, or somehow less explicit of a demand. When people claim that unions hold back investment and productivity (what everyone seems to agree the UK needs), I thought this was based on something more nuanced than… it’s the union’s core demand.

Of course, in the UK, the most publicised union action is nurses, doctors, teachers, rail workers. From the little I know, their demands have little to do with automation and lots to do with reasonable wages and working conditions. Something which I think most people in the UK have a lot of sympathy for.

UPID is as UPID does

Fri, 16 Aug 2024 00:00:00 +0000

Context

I developed a new type of universally-unique ID, and it was pretty fun. I was inspired to do so when I read this blog post a month or two ago, and it mentioned Stripe’s pretty prefixed IDs. They look like this:

cus_MJA953cFzEuO1z
└─┘ └────────────┘
 └─ prefix    └─ random

I’ve used Stripe’s API before and thought these IDs were great. Any time I dumped out some JSON I didn’t have to wonder what each bunch of random characters stood for, it told me!

The downside is that you have to make a choice when you implement them. Either

Store them as TEXT in the database, which means you’ll have slower lookups and waste MBs, or
Store just the random part as a 128bit UUID in the database, which means your APIs have to strip/prepend the prefix at some boundary each time.

If your ORM is clever or extensible, you can solve option 2. For example, TypeID has an Elixir implementation that will handle this back-and-forthing for you. But TypeID is a pretty general spec and not able to make this universal. There’s no way to make this work with Prisma, for example.

UPID

But the rest of use are stuck with one of these imperfect solutions. So I came up with UPID. It looks like this in its text encoding:

user_2accvpp5guht4dts56je5a
└──┘ └──────┘└───────────┘└─ version
 └─ prefix └─ time   └─ random

And something like this, in binary:

00000001 10010001 01011100 10110100
01111110 11101101 11100011 00011110
01001111 10100010 00010101 00101011
10101011 11010110 00010101 01110110

That’s right, it’s just 128 bits! That means anywhere you have a 128 bit datatype (like UUID), you can drop in a UPID with no issue. So you can use UPID in your server code and store them as UUIDs in (for example) Postgres. You don’t need to strip/prepend the prefix, because it’s always there.

It is most similar to a ULID, but with bits taken away from the randomness and time components to make space for the prefix. Like ULID, it uses a 32 character alphabet (making 5 bits per character), but unlike Crockford’s base32, it keeps the whole alpha part, so you can write anything with a-z in the prefix.

Also like ULID, it is “lexicographically sortable”, which basically just means it can be ordered by date. This is useful on its own, but is also valuable for things like ensuring index locality, WAL efficiency and easy sharding. However, UPID has lower time precision (256ms vs 1ms), which is arguably better for two reasons: some people are concerned you can leak information with time-based IDs, and 256ms leaks a lot less; and many system clocks have worse than 1ms precision, so you won’t have a false sense of monotonicity that might not exist.

Referring back to the two “problems” from further up:

Stored as a 128 bit value in the database, so no inefficient TEXT values.
Always knows what its prefix is, so you don’t need to remember to add/remove anything at any boundary.

Data layout

So, how does it work? Here’s that same ID from further up, with the separate components broken up.

    user   _   2accvpp5      guht4dts56je5       a
   └────┘     └────────┘    └─────────────┘   └─────┘
   prefix       time            random        version     total
   4 chars      8 chars         13 chars      1 char      26 chars
       └────────│────────────────│───────────┐  │
                │                │           │  │
                │                │           │  │
             40 bits            64 bits      24 bits     128 bits
             5 bytes            8 bytes      3 bytes      16 bytes
             time               random       prefix+version

Notice that in the binary format, the time bits are stored first, so that if you sort a bunch of UPIDs, they will be neatly ordered by time.

Any library that implements the UPID spec just needs to be able to shuffle back and forth between the 128 bit binary representation and the text representation. There are already Python, Rust, Postgres (those are all links to the same repo btw) implementations, along with one in TypeScript that powers a little demo website that looks like this:

Cool

Cool?

This was the rare case where writing it in Rust was actually easier than in Python, because it’s more explicit about what’s a u8 or u64 or u128, whereas in Python it’s all just bytes and good luck figuring out how many or what the precision is.

This matters when you’re bit-shifting like a madman and need to keep track of which bits came from where:

let res = (time_bits << 88)
	| (random << 24)
	| ((prefix[0] as u128) << 16)
	| ((prefix[1] as u128) << 8)
	| prefix[2] as u128;

You can also do stuff like this, and Rust will shout at you if you accidentally make your alphabet too long:

pub const ENCODE: &[u8; 32] = b"234567abcdefghijklmnopqrstuvwxyz";

That’s the base32 alphabet by the way. The numbers come first so that, if for some reason you want to sort the text-encoded strings, they’ll still be sorted in the right order!

Using it

In Python:

from upid import upid
upid("prod")            # prod_2acptrhhfi7asnb5iessba

Rust:

use upid::Upid;
UPID::new("cust")      // cust_2acptrk7ypgl2hl45g3vca

Or TypeScript:

import { upid } from "upid-ts";
upid("acct")           // acct_2acptrqnipixsl2xugym7a

In all of this cases, the actual data is just a 128 bit binary blob. And if you store it somewhere as a UUID or u128 or something, you can just load it right back into a UPID.

If anyone wants to write an implementation in another language (ChatGPT should be able to do most of the work), let me know and I’ll add it to the list.

Wait a second, what about Postgres!?

Postgres

Thanks to pgrx, it’s now dead-easy to write extensions for Postgres in Rust. And this is the best part. If you install the extension, then you get the binary-text back-and-forthing directly in Postgres.

So you can do stuff like this:

CREATE EXTENSION upid_pg;

CREATE TABLE members (
    id   upid NOT NULL DEFAULT gen_upid('memb') PRIMARY KEY,
    name text NOT NULL
);

INSERT INTO members (name) VALUES('Bob');

SELECT * FROM members;
--              id              | name
-- -----------------------------+------
--  memb_2acptt2ytgmpf5cmj6iy5a | Bob

That means your library code doesn’t even need to care about UPID, or install any extra libraries. It will be transmitted over the wire as a pretty string, and stored in the DB as an efficient 128 bit blob.

The only downside is that, as easy as it is to write Postgres extensions, it’s not yet very easy to install them. I’ve created .deb packages (get yours at the Releases) so you can install it quite easily on Debian-based distros. Alternatively, you can try out the Docker image that has Postgres 16 with the UPID extension pre-installed:

docker run -e POSTGRES_HOST_AUTH_METHOD=trust \
    -p 5432:5432 carderne/postgres-upid:16

Go on…

So that’s UPID! I hope someone finds it useful, or at least interesting. I’ll be using it in some of my personal projects, and trying to talk whoever I can into using it in their critical production projects. There’s not much risk — at the end of the day it’s just a 128 bit ID, and the full implementations are a few hundreds lines of code at most. Go on…