Home

Reading a zip file from a Cloudflare Worker

Tue, 30 Sep 2025 00:00:00 GMT

tl;dr - Example repo: https://github.com/jmorrell/cloudflare-read-zip-file-example

I needed to add some import / export functionality to a project, and ran into the problem of working with zip files on Cloudflare workers.

The standard library for working with Zip files seems to be zip.js but this is quite a large dependency. While that might not matter for a long-running Node app, on Workers your code must be downloaded and parsed on every cold start, so avoiding big dependencies where possible is usually worth it.

For creating the zip files I found littlezipper, which has worked well, however I did not find a similarly small alternative for reading zip files.

However this is exactly the kind of small, well-defined problem that LLMs are actually really good at solving. We have a well-defined widely-used file format, and a set of JavaScript APIs for doing stream manipulation, and both should be well-represented in the training data of a modern LLM.

With some prompting Claude came up with ZipReader implementation and a set of tests that should satisfy my use-case.

const arrayBuffer = await file.arrayBuffer();
const zipReader = new ZipReader(arrayBuffer);

console.log(`Processing zip file: ${file.name}`);
console.log(`Total entries: ${zipReader.getFiles().length}`);

for (const fileInfo of zipReader.getFiles()) {
  console.log(fileInfo.filename);
}

Both my implementation and littlezipper are pretty limited in that:

They only support the "Deflate" compression method, which seems to be the most common
Both require holding all of the file contents in memory, which will be a problem for larger files with Worker's 128MB memory constraint

A better approach might be to leverage R2 (or the new node:fs support) for storage and HTTP Ranged Reads to only ever hold part of the file in memory at any given time, but that will have to wait for another day. rinsuki/async-zip-reader seems to implement this approach, but it brings in its own dependencies, and I have not tried it out.

Resolving "FOREIGN KEY constraint failed" with Cloudflare SQLite

Sun, 21 Sep 2025 00:00:00 GMT

tl;dr - You likely need PRAGMA defer_foreign_keys = on; and these docs.

While writing a SQL migration for a Durable Object I kept hitting this frustrating error:

FOREIGN KEY constraint failed: SQLITE_CONSTRAINT

I had something similar to the following:

CREATE TABLE A (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  name TEXT
);

CREATE TABLE B (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  id2 INTEGER,
  book TEXT,
  FOREIGN KEY(id2) REFERENCES A(id)
);

I wanted to add a CHECK constraint to a column in table A which can be done by renaming and copying:

CREATE TABLE new_A (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  name TEXT NOT NULL CHECK (type IN ('image', 'html', 'pdf'))
);

INSERT INTO new_A
SELECT
  id,
  name
FROM A;

DROP TABLE A;

ALTER TABLE new_A RENAME TO A;

Note that this invariant was already enforced by the app. This would not be safe to do in general.

However this won't work, and you'll get the FOREIGN KEY constraint failed error.

The SQLite docs will lead you to:

PRAGMA foreign_keys = false

-- Do the migration

PRAGMA foreign_keys = true

However this will not work either. Though frustratingly it will if you dig into the .wrangler directory, find the sqlite db file, and try applying it directly.

The solution is in the Cloudflare docs. The doc is for D1 but this applies to SQLite in Durable Objects as well.

D1's foreign key enforcement is equivalent to SQLite's PRAGMA foreign_keys = on directive. Because D1 runs every query inside an implicit transaction, user queries cannot change this during a query or migration.

D1 allows you to call PRAGMA defer_foreign_keys = on or off, which allows you to violate foreign key constraints temporarily (until the end of the current transaction).

We can now fix our migration!

PRAGMA defer_foreign_keys = on;

CREATE TABLE new_A (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL CHECK (type IN ('image', 'html', 'pdf'))
);

INSERT INTO new_A
SELECT
  id,
  name
FROM A;

DROP TABLE A;
ALTER TABLE new_A RENAME TO A;

PRAGMA defer_foreign_keys = off;

Automatically documenting a Durable Object's SQLite schema

Sat, 20 Sep 2025 00:00:00 GMT

One of the first things you run into trying to build with SQLite in Durable Objects is handling SQL migrations.

I've been using Drizzle to manage my Durable Object schemas, which has worked well, but felt that it was a bit heavier than what I needed for my current project. Cloudflare has recently released the @cloudflare/actors library which has a much simpler approach (originally based on Lambros' durable-utils).

With this library you can define your migrations in-line and run them yourself before you access your SQL tables.

import { Storage } from "@cloudflare/actors/storage";

export class ChatRoom extends DurableObject<Env> {
  storage: Storage;

  constructor(ctx: DurableObjectState, env: Env) {
    super(ctx, env);
    this.storage = new Storage(ctx.storage);
    this.storage.migrations = [
      {
        idMonotonicInc: 1,
        description: "Create users table",
        sql: "CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY)",
      },
    ];
  }
  async fetch(request: Request): Promise<Response> {
    // Run migrations before executing SQL query
    await this.storage.runMigrations();

    // Query with SQL template
    let userId = new URL(request.url).searchParams.get("userId");
    const query = this.storage.sql`SELECT * FROM users WHERE id = ${userId};`;
    return new Response(`${JSON.stringify(query)}`);
  }
}

I appreciate the simplicity of this approach, however once you have more than a few migrations, it can be difficult to keep track of what your current schema even is. I don't want to re-play SQL statements in my head to understand how to query what I need.

I worked around this by extracting the migrations to a separate file, which I can import in the Durable Object directly:

export default [
  {
    idMonotonicInc: 1,
    description: "Create exports table",
    sql: `
      CREATE TABLE IF NOT EXISTS example (
        id TEXT PRIMARY KEY,
        created_at INTEGER NOT NULL DEFAULT (unixepoch()),
        name TEXT NOT NULL
      );
    `,
  },
  {
    idMonotonicInc: 2,
    description: "Add description column to example table",
    sql: `ALTER TABLE example ADD COLUMN description TEXT NOT NULL DEFAULT 'default';`,
  },
  {
    idMonotonicInc: 3,
    description: "Add age column to example table",
    sql: `ALTER TABLE example ADD COLUMN age INTEGER NOT NULL DEFAULT 0;`,
  },
];

import { Storage } from "@cloudflare/actors/storage";
import migrations from "./example-migrations";

export class Example extends DurableObject {
  storage: Storage;
  env: Env;

  constructor(ctx: DurableObjectState, env: Env) {
    super(ctx, env);
    this.env = env;
    this.storage = new Storage(ctx.storage);
    this.storage.migrations = migrations;
  }

However, since the database in the Durable Object is "just" SQLite, and our migrations module contains a plain JavaScript object we can write a script to run them against an empty SQLite database and dump the resulting schema. Any time we add a new migration we can re-run this and check the output in as documentation.

import sqlite3 from "sqlite3";
import { open } from "sqlite";
import fs from "node:fs";
import sqlFormatter from "@sqltools/formatter";

async function getSchema(migrationsPath, schemaWritePath) {
  let migrations = await import(migrationsPath).then((m) => m.default);

  let dbPath = `/tmp/test-${Math.random().toString(36).slice(2)}.db`;

  let db = await open({
    filename: dbPath,
    driver: sqlite3.Database,
  });

  for (const migration of migrations) {
    await db.exec(migration.sql);
  }

  let schema = await db.all(`
    SELECT sql 
    FROM sqlite_master 
    ORDER BY type DESC, name
  `);

  let out = `-- This file was generated by a script
-- DO NOT EDIT THIS FILE MANUALLY
`;

  for (let item of schema) {
    if (item.sql) {
      out += "\n" + sqlFormatter.format(item.sql) + ";\n";
    }
  }

  await db.close();

  fs.rmSync(dbPath);

  fs.writeFileSync(schemaWritePath, out);
}

// A list of each of the Durable Objects with migrations to document
let DurableObjects = [
  {
    migrations: "path/to/example-migrations.ts",
    schema: "path/to/example-schema.sql",
  },
];

for (let durableObject of DurableObjects) {
  await getSchema(durableObject.migrations, durableObject.schema);
}

Now instead of trying to play back all of the migration SQL commands in my head, I have a nice up-to-date schema to reference while I'm working on my logic.

-- This file was generated by a script
-- DO NOT EDIT THIS FILE MANUALLY

CREATE TABLE example (
  id TEXT PRIMARY KEY,
  created_at INTEGER NOT NULL DEFAULT (unixepoch()),
  name TEXT NOT NULL,
  description TEXT NOT NULL DEFAULT 'default',
  age INTEGER NOT NULL DEFAULT 0
);

Generating Image Placeholders on Cloudflare Workers

Thu, 24 Jul 2025 00:00:00 GMT

I was recently blown away by Lean Rada's excellent blog post on CSS-only blurry image placeholders (seriously, it's some very creative work!) and I wanted to try generating image placeholders for a Cloudflare Workers app that I'm working on.

If you're in a hurry, the code can be found in this GitHub repo along with a little demo app.

A lot of the existing docs on generating these image placeholders rely on modules like Sharp which don't run on Cloudflare Workers. You could also compile a native module into WASM and use that, but that's quite a lot of work to sort out compilers and config!

Luckily you can ditch all of these and just leverage the platform's Image bindings directly.

The Image bindings let you transform, resize, filter, rotate, add watermarks, etc.

const response = (
  await env.IMAGES.input(stream)
    .transform({ rotate: 90 })
    .transform({ width: 128 })
    .transform({ blur: 20 })
    .output({ format: "image/png" })
).response();

There were two things missing from the docs <a href="#footnote-1" id="footnote-ref-1" data-footnote-ref="true" aria-describedby="footnote-label"><code>[1]</code></a> that I discovered while looking at the type definitions that help us out here:

You can request an array of rgb or rgba pixel values from the Workers binding directly. No need to write code to parse an image file format directly
There is a "fit" for resizing images that ignores the existing aspect ratio called "squeeze" that is currently missing from the docs

Another issue was an inability to get the resized image dimensions from the output. When requesting an rgb output you receive an array with a single dimension with 3 values per pixel. Unlike a photo format this does not encode any information about the width or height of the image. I work around this by calculating the expected dimensions from the original's dimensions.

I implemented 4 different image placeholder algorithms:

Dominant Color (easy)

The simplest approach I could think of was to squish the whole image down into one pixel, and let the image software decide what the color should be. This works surprisingly well given the lack of sophistication! The whole implementation fits in just a few lines:

async function getDominantColor(image: ReadableStream): Promise<string> {
  let rgbImage = await env.IMAGES.input(image)
    .transform({ width: 1, height: 1, fit: "cover" })
    .output({ format: "rgb" });
  let rgbImageBuffer = await rgbImage.response().arrayBuffer();
  let pixelData = new Uint8Array(rgbImageBuffer);

  let r = pixelData[0];
  let g = pixelData[1];
  let b = pixelData[2];

  return `#${r.toString(16).padStart(2, "0")}${g.toString(16).padStart(2, "0")}${b.toString(16).padStart(2, "0")}`;
}

Dominant Color from a Palette

Averaging all of the colors together sometimes creates a muddy looking output. This resizes the image to be easy to work with in the workers environment and uses the underlying Modified Median Cut Quantization (MMCQ) algorithm from the color-thief library to group colors together. If the image has a lot of a single color, such as text on a white background, it will tend to pick out this main color instead of averaging it with the rest of the colors in the image.

async function getDominantColorFromPalette(image: ReadableStream): Promise<string> {
  let rbgImage = await env.IMAGES.input(image)
    .transform({ width: 200, height: 200, fit: "cover" })
    .output({ format: "rgb" });
  let rgbImageBuffer = await rbgImage.response().arrayBuffer();
  let pixelData = new Uint8Array(rgbImageBuffer);

  // get a representative color palette from the image
  let palette = quantize(pixelData, 5);
  // get the most prominent color
  let dominantColor = palette[0];
  let r = dominantColor[0];
  let g = dominantColor[1];
  let b = dominantColor[2];

  return `#${r.toString(16).padStart(2, "0")}${g.toString(16).padStart(2, "0")}${b.toString(16).padStart(2, "0")}`;
}

Blurhash

https://blurha.sh/ is a clever approach that compresses an image into blurry gradients and packs that data into a small string like LEHV6nWB2yk8pyo0adR*.7kCMdnj. The TypeScript library works great on Workers, though note that it expects an rgba array as input.

I did have to jump through some hoops to get the correct dimensions of the new image. See the full code here.

async function getBlurhash(image: ReadableStream, aspectRatioInfo: AspectRatioInfo): Promise<string> {
  let resizedImage = await env.IMAGES.input(image)
    .transform({ width: RESIZE_DIMENSION, height: RESIZE_DIMENSION, fit: "contain" })
    .output({ format: "rgba" });
  let resizedImageBuffer = await resizedImage.response().arrayBuffer();
  let pixelDataClamped = new Uint8ClampedArray(resizedImageBuffer);

  let { width: resizedWidth, height: resizedHeight } = getResizedDimensions(
    aspectRatioInfo,
    RESIZE_DIMENSION,
    pixelDataClamped.length / 4
  );
  return encodeBlurhash(pixelDataClamped, resizedWidth, resizedHeight, 4, 4);
}

CSS Blobhash

I cribbed from Lean's implementation and lightly converted it for the Images binding. See the full implementation here.

async function getCSSBlobHash(image: ReadableStream): Promise<number> {
  let resizedImage = await env.IMAGES.input(image)
    .transform({ width: 3, height: 2, fit: "squeeze" })
    .output({ format: "rgb" });
  let resizedImageBuffer = await resizedImage.response().arrayBuffer();
  let pixelDataClamped = new Uint8ClampedArray(resizedImageBuffer);

  return encodeCSSBlobHash(pixelDataClamped);
}

Results

Playing around with the output I was quite surprised to find that picking out the dominant color from a palette produced what I thought was consistently the best balance. The blurry approaches are often very pleasing for photographs, but my use-case will have a lot of images designed for showing up in a social media feed, and these feel a little off when blurred. Sometimes keeping it simple is the best approach.

I'll close out with some screenshots of examples. Feel free to play with the generator with your own images: https://github.com/jmorrell/low-quality-image-placeholders-on-cloudflare-workers

<a href="#footnote-ref-1" id="footnote-1"><code>[1]</code></a> I've filed tickets internally to get these issues fixed

On OpenTelemetry and the value of Standards

Sun, 15 Dec 2024 00:00:00 GMT

Physical standards are amazingly foundational things that we frequently take for granted. I can buy a 1/4-20 bolt today and fit it into machines built many decades ago. There is a variety of tooling readily available to work with threaded bolts: wrenches, impact drivers, torque wrenches of all kinds, thread gauges, taps and dies, threaded inserts.

Importantly, this is all astoundingly cheap. An individual bolt might just be a few pennies, barely more than the cost of raw materials. Wright's Law states that for every cumulative doubling of units produced, costs tend to fall by a constant percentage. The more we make of something, the cheaper it is per-unit.

The volumes and competition generated by a shared standard push us further up the experience curve than would be possible if each company produced their own bespoke product. We can build bigger, faster, more specialized tools and amortize their cost over more usage.

Standards enable investment.

While we operate under different constraints, the same is broadly true in software. One-off solutions can be eye-wateringly expensive, but standards make for cheap and plentiful tooling.

Today's tools

When it comes to understanding and introspecting our systems, our industry has been a bit of a wild west for a long time.

Mostly we rely on vendors. Vendors build SDKs that bend over backward to automagically wrap libraries and generate mystery payloads that get shipped off to proprietary services. This usually exists as an opaque layer on top of our software. We get trained that instrumentation is someone else's responsibility, too complicated for us mere mortals.

This can be great for the handful of vendors that manage to win a majority of the market, but it's a pretty bad experience for everyone else. Additionally it draws hard limits on what's possible.

You're never going to get Django to build in Datadog support. AWS is never going to integrate New Relic into its platform. And if they did, this would only entrench that handful of vendors even more.

Without standards in this area all investment in the tooling comes from vendors who all end up re-building very similar things. It's hard or economically treacherous for others to build on top of their proprietary stacks. The lack of standards prevents broader investment.

We're trapped in a local maximum. Open Standards provide a way out and, hopefully, a better experience.

Today's experience

When I deploy a web app built on a common stack today (say: Rails and Postgres) whether on a cloud, PaaS, or my own hardware I get precious little information about what my software is doing. I may get some basic HTTP, memory, CPU metrics, but if I want more fine-grained data I'll need to go elsewhere and start from scratch.

This is in sharp contrast to the detailed information about each HTTP request or database query we get in an APM tool like New Relic. Want to see the slowest requests on the /checkout endpoint filtered by region and device? New Relic likely has you covered. Need to see your slowest database queries? It's only a couple clicks away!

Fair play to New Relic, it took a lot of exceptional engineering to make this look seamless! What if the next company didn't have to build all of this from scratch? What if we could expect more from our tools in the first place? Rails can easily capture all of this information. Why do we need a vendor to write fancy code to wrap it and surface it to us? Wouldn't the instrumentation likely be better if the Rails developers designed the instrumentation for their own code?

Prior to OpenTelemetry there hasn't been a clear mechanism for doing this. Imagine I'm writing a database client library and want to communicate to my user's system an observation like query X took 1.2ms, returning 152kb of data across 200 rows. Without a standard way of emitting this data, I don't have many good options. I could write log messages to stdout, but they might not be formatted the way the user needs. I might build a plugin system so the user could bring their own logic, requiring more work from the user and making my own library more complicated in the process.

A vision for the future

What if I could build instrumentation directly into my code and the user could configure what they wanted to do with it? It might look like:

class Database
  def execute(sql)
    in_span("sql_query") do |span|
      # execute the query
      span.set_attributes(
        "query" => obfuscated_query,
        "results_size_kb" => results_size,
        "results_row_count" => results.length,
      )
    end
    return results
  end
end

That already exists! It's the OpenTelemetry API. It's not even directly tied to the OpenTelemetry OTLP format, and you could use this instrumentation to generate a completely different trace format or even just log lines if you wanted.

Let's go a bit farther and imagine a better future where all of our dependencies come with re-usable instrumentation, where profileration of a single standard mean our IDE, platforms, languages, and frameworks call all work seamlessly together.

When building my service locally, I have local tools to view the telemetry my service is emitting, perhaps they're even built into my IDE. When I deploy to a PaaS service, it automatically collects and surfaces traces, metrics, and logs right in the dashboard. Maybe the platform automatically pulls in more metadata for me so I can easily view how my requests are performing as a new versions roll out, or how my latest changes affected response times in Germany, or notice a spike in errors on the /account/:id/details endpoint and jump to an exemplary trace.

A few button clicks later and now this data is flowing to my preferred Observability vendor as well as our own data warehouse for deeper analysis and long-term storage.

I haven't had to write a single line of code to enable this, or stand up any custom infrastructure, and I don't even really need to know how it's happening. Like my IDE debugger or the Chrome Dev Tools, it's just another tool that I expect to work. However once I discover that I do need some additional data, there's a standard and easy way of extending the data my frameowrk and libraries are creating for me.

OpenTelemetry is usually sold as a way of avoiding lock-in for Observability vendors <a href="#footnote-1" id="footnote-ref-1" data-footnote-ref="true" aria-describedby="footnote-label"><code>[1]</code></a>, but that's just a start. If the standard is successful it'll also be built into libraries, frameworks, tooling, IDEs, languages, and platforms. For many users it's likely the standard will mostly disappear into the background and become an expected part of the system.

We're still very early on this journey as an industry with OpenTelemetry.

<blockquote class="bluesky-embed" data-bluesky-uri="at://did:plc:gttrfs4hfmrclyxvwkwcgpj7/app.bsky.feed.post/3ldca564b3s2f" data-bluesky-cid="bafyreibfxbrfdfhvmdc2shhsxgpjcf3i2tltigcqaeimox3igzrkvk7ua4"><p lang="en">what I really want to see are framework-level “batteries included” sdk reimplementations on top of the OTel APIs.

spiritually similar to how rust has coalesced around the tracing crate with pluggable exporters</p>— austin 🎄 (<a href="https://bsky.app/profile/did:plc:gttrfs4hfmrclyxvwkwcgpj7?ref_src=embed">@aparker.io</a>) <a href="https://bsky.app/profile/did:plc:gttrfs4hfmrclyxvwkwcgpj7/post/3ldca564b3s2f?ref_src=embed">December 14, 2024 at 1:21 PM</a></blockquote><script async src="https://embed.bsky.app/static/embed.js" charset="utf-8"></script>

But is OpenTelemetry the right standard?

How do we know that OpenTelemetry is the right standard to bet on? Don't some people have problems with it? Isn't it "design by committee"?

David Cramer had a popular post about his issues with OpenTelemetry, and Ivan Burmistrov brought up a number of points in All you need is Wide Events, not “Metrics, Logs and Traces”. Hazel Weakly recently dove into one of OpenTelemetry's biggest flaws: lack of a standard for long-running spans. I generally agree with most of their points! <a href="#footnote-2" id="footnote-ref-2" data-footnote-ref="true" aria-describedby="footnote-label"><code>[2]</code></a>

If your main experience with OpenTelemetry today looks like a more tedious, worse version of adding an APM vendor I don't fault you for taking this view. I've even cautioned people away from adoption today if they have a small team or are under a lot of pressure. (My wide events guide can help though!)

<blockquote class="bluesky-embed" data-bluesky-uri="at://did:plc:qbim5usypxqjsxb27zjxn733/app.bsky.feed.post/3lasw3fary62j" data-bluesky-cid="bafyreigvnfzc5fhetkp7pagqyjxq6jkhes3grbxdud5enaeilux7t5jl3u"><p lang="">OpenTelemetry is a really important project, but if you’re in a place where you are struggling to understand your systems, trying to adopt OTel as a way out is:

likely not going to solve your problem
going to give you heaps more complexity to navigate</p>— Jeremy Morrell (<a href="https://bsky.app/profile/did:plc:qbim5usypxqjsxb27zjxn733?ref_src=embed">@jeremymorrell.dev</a>) <a href="https://bsky.app/profile/did:plc:qbim5usypxqjsxb27zjxn733/post/3lasw3fary62j?ref_src=embed">April 20, 2024</a></blockquote><script async src="https://embed.bsky.app/static/embed.js" charset="utf-8"></script>

OpenTelemetry is complicated and endlessly extensible. That comes as a necessary byproduct of supporting so many different stakeholders. What looks like unnecessary bloat to you frequently turns out to be core to someone else's adoption. That complexity enables everyone to get on the same page.

It's not perfect, but it is succeeding at the main thing a standard needs: wide adoption. OpenTelemetry has shown major uptake by vendors and is starting to be adopted by end-users in enterprise. OpenTelemetry is the 2nd most active CNCF project after Kubernetes. There is clear momentum and no obvious competitors.

The benefits from having a single standard, and what we could build on top of that foundation, far outweigh the drawbacks in complexity. We should be very thankful that the creators of OpenCensus and OpenTracing decided to collaborate rather than compete.

OpenTelemetry is not a monolith

OpenTelemetry can be the foundation of this better future, but it's not a certainty yet. It's a big, sprawling specification, and it can be helpful to break down how each part contributes to the whole.

Caveat: Most of my experience is with the Ruby, Go, and JavaScript ecosystems. Things may look different in other languages.

OTLP

If a vendor supports receiving OpenTelemetry, usually that means they support receiving OTLP formatted data. All of my interactions with the OTLP format so far (mostly tracing and metrics) have been good ones. It's not very hard to generate from scratch, and there is good library support for both generating and parsing this data.

There are a growing number of tools to visualize and work with this data locally, and I hope this trend continues. No one should be debugging traces and metrics by squinting at serialized data structures dumped to their terminal.<a href="#footnote-3" id="footnote-ref-3" data-footnote-ref="true" aria-describedby="footnote-label"><code>[3]</code></a>

My most pressing request for OTLP is an official way of representing unfinished spans similar to the span snapshots that Embrace is using in their mobile tooling.

API

The split between the API and SDK in OpenTelemetry is not well-understood, but is one of the most interesting ideas in the project.

The API is a small set of interfaces to create traces, metrics, logs that have no implementation, just a hook that the end user can use to register their desired behavior. The OpenTelemetry SDKs build on these interfaces, but nothing is stopping others from doing the same thing. If library authors embed them into their code, then we can avoid needing fancy Instrumentation libraries.

I mainly have experience with the tracing and context APIs, and find them fairly straight-forward to use, if not always the most idiomatic.

Ultimately I think that these APIs are good candidates for getting folded into the language itself. Rust is leading the way.

SDK

If you have added OpenTelemetry to your service, the library you installed was the SDK. It's a full-featured, configurable toolkit for instrumenting your code and emitting telemetry. You can think of it like the Sentry or New Relic libraries, though a better analogy might be a toolkit you could use to build those libraries.

I've had very mixed experiences with the OpenTelemetry SDKs.

They are amazingly extensible and battle-hardened bits of software. Even when handed really weird requirements, I was always able to find a way to get them to do what my company needed them to do. I've only experienced one or two production issues even after years of using them at scale. The maintainers should be incredibly proud of their work.

However it's not all sunshine and rainbows. The documentation is not always the best, and I frequently find myself digging into their code to figure out how something works. The SDKs introduce many concepts to the user and require lots of configuration even on the simplest happy path. When compared to a polished, opinionated vendor experience it's clear that the SDKs don't measure up.

I'd like to see a better default experience. When I printf to stdout I see the data immediately and without configuration in my terminal. I can easily pipe this to a file. It shows up in my IDE. My deployment platform likely has first-class support for collecting and sending this data to a vendor of my choice. I want that same ease-of-use for OTLP streams.

I fear that we are moving away from this vision and towards even more layers of configuration.

Collector

The Collector is configurable glue, a swiss army knife that can morph into whatever your organization needs. There's probably a plugin that converts from whatever you have to OTLP and back again. If there isn't, writing a custom one isn't too difficult. It allows an organziation to gradually adopt OpenTelemetry in their existing systems and makes building centralized tooling to handle telemetry streams more tractable.

I think the Collector is a large reason for the success organizations are having in adoption of OpenTelemetry. People love deploying collectors! Perhaps too much. It's easy to turn around and realize that you have more than a dozen in production, all somehow justified.

In a future where OpenTelemetry has become a standard I think there will be less need of the Collector-as-glue, but I still see it living on at the heart of many toolchains.

Contrib

The SDK provides core functionality, and the contrib collections provide plugins and adapters. This is where you find OpenTelemetry's Rails instrumentation or Node's Postgres client instrumentation or plugins for the collector to receive legacy formats.

The contrib instrumentations represent gobsmacking amounts of engineering effort. This code can be really difficult to write and maintain, especially as the underlying library evolves and changes. Navigating breaking changes and which version is supported by which contrib version can be a real pain. Despite that I've mostly had good experiences with these!

However I would like to see a future where they largely aren't necessary because vendor-neutral instrumentation has been added directly to the libraries or other systems themselves.

Semantic Conventions

Semantic Conventions is the idea that every time your are instrumenting a similar thing, such as an HTTP service or a job queue, the telemetry emitted should look the same, have the same shape as telemetry from other HTTP services or job queues. The status code for an HTTP response should always be http.response.status_code, not http.status_code, not http_status_code, not statusCode or status-code.

While technically the simplest, in my opinion Semantic Conventions is one of the most ambitious OpenTelemetry specs. If broadly adopted it will allow tooling to recognize patterns and surface much better information for users. However trying to name so many things across so many domains is an incredibly difficult thing to achieve, let alone getting buy in across so many implementations.

<a href="#footnote-ref-1" id="footnote-1"><code>[1]</code></a> I suspect vendor neutrality is over-hyped relative to its actual merits. From experience migrating an org with a non-trivial number of systems, alerts, dashboards, and integrations with other systems, and hundreds of engineers who need to learn how to use the new vendor, double-writing telemetry into the new system is not the hard part.

While it does provide some insurance against a vendor charging you extortionate rates, for most organizations migrating vendors could cost on the order of millions of dollars in salary and months of time in opportunity cost that could be have been spent improving the core product. And that's if the organization has the engineering maturity to organize this kind of migration in the first place, many do not.

Choose your observability vendors with the same care with which you choose your cloud provider, whether you use OpenTelemetry or not.

<a href="#footnote-ref-2" id="footnote-2"><code>[2]</code></a> In particular, I generally agree with Ivan Burmistrov's points in All you need is Wide Events, not “Metrics, Logs and Traces”. A span is an event. A set of metrics can be an event. A log is an event. An event attached to a span is... another event.

OpenTelemetry creates a lot of complexity that seems unnecessary from this perspective, but I suspect a simpler event-based model would never have been adopted by any traditional vendor. Without wide adoption, it would very likely fail as a standard.

<a href="#footnote-ref-3" id="footnote-3"><code>[3]</code></a> I'd like if the OpenTelemetry project itself provided more tooling here though those discussions seem to have fizzled out.

And please do not make me run a heavy-weight production tool in docker to visualize my telemetry. Give me native apps, CLI tools, IDE plugins. Treat this as a first-class concern because it shapes so much of the users' experience.

A Practitioner's Guide to Wide Events

Tue, 22 Oct 2024 00:00:00 GMT

Adopting Wide Event-style instrumentation has been one of the highest-leverage changes I've made in my engineering career. The feedback loop on all my changes tightened and debugging systems became so much easier. Systems that were scary to work on suddenly seemed a lot more manageable.

Lately there have been a lot of good blog posts on what "Wide Events" mean and why they are important. Here are some of my recent favorites:

The tl;dr is that for each unit-of-work in your system (usually, but not always an HTTP request / response) you emit one "event" with all of the information you can collect about that work. "Event" is an over-loaded term in telemetry so replace that with "log line" or "span" if you like. They are all effectively the same thing.

Charity Majors has been promoting this approach lately under the name "Observability 2.0", creating some new momentum around the concept, however, it is not a new idea. Brandur Leach wrote about "Canonical Log Lines" both on his own blog in 2016 and as used by Stripe in 2019. And AWS has recommended it as a best-practice for ages.

Okay... I think I get the idea... but how do I do "wide events"?

This is where I find a lot of developers get tripped up. The idea sounds good in theory, and we should totally try that one day! But I have this stack of features to ship, that bug that's been keeping me up at night, and 30 new AI tools that came out yesterday to learn about. And like... where do you even start? What data should I add?

Like anything in software, there are a lot of options for how to approach this, but I'll talk through one approach that has worked for me.

We'll cover how to approach this in tooling and code, an extensive list of attributes to add, and I'll respond to some frequent objections that come up when discussing this approach.

For this post we'll focus on web services, but you would apply a similar approach to any workload.

Choose your tools

We will need some way to instrument your code (traces or structured log lines) and somewhere to send the instrumentation to in order to query and visualize it.

This approach is best paired with a tool that lets you query your data in quick iterations. I like Honeycomb for this, but any Observability tool backed by a modern OLAP database is likely going to work in a pinch.

Honeycomb, New Relic, and DataDog built their own columnar OLAP data stores, though now with the availability of ClickHouse, InfluxDB IOx, Apache Pinot, and DuckDB there are new Observability tools popping up all the time.

If you aren't constrained, I highly recommend defaulting to using OpenTelemetry and Honeycomb. Your life will be easier.

However even if you are stuck in a corporate environment with a strong allergy to technology built after 2010 you can leverage log search tools like ElasticSearch in a pinch. Stripe's blog post goes over how to use Splunk for this.

In any tool you want to focus on getting proficient at 3 core techniques in order to sift through your events. The faster you are able to apply these, iterate, and ask questions of your data, the better you'll be able to debug issues and see what your system is really doing. When observability folks refer to "slicing and dicing" data, this is what they are generally referring to. I'll represent queries using a made-up SQL dialect, but you should be able to find equivalents in your tool's query language.

Visualizing

Existing in a human body comes with its fair share of downsides, but the human visual cortex is really, really good at recognizing patterns. Give it a fighting chance by getting really good at summoning visualizations of the data your system is emitting. COUNT, COUNT_DISTINCT, HEATMAP, P90, MAX, MIN, Histogram. Learn to leverage whatever graphs your tool makes available to you. Practice it. Get fast.

Grouping

With each new annotation that we add to our wide events, we create another dimension along which we can slice our data. GROUP BY allows us to look along that dimension and see if the values along that dimension match our expectations.

GROUP BY instance.id

GROUP BY client.OS, client.version

Filtering

Once we've narrowed in one dimension that is interesting, we usually want to dig further into that data. Filtering down so that we're only looking at data from one endpoint, or from one IP address, or sent by the iOS app, or only from users with a specific feature flag turned on allows us to narrow our focus to a very specific segment of traffic.

WHERE http.route = "/user/account"

WHERE http.route != "/health"

WHERE http.user_agent_header contains "Android"

Write a middleware to help you

If you are using an OpenTelemetry SDK it is already creating a wrapping span around the request and response. You can access it by asking for the active span at any point during the processing of the request.

let span = opentelemetry.trace.getActiveSpan();
span.setAttributes({
  "user_agent.original": c.req.header("User-Agent"),
});

However if anyone wraps any of your code in a child span the "active span" will change to be that new wrapping span! There is no first-class way of addressing this original "main" span in OpenTelemetry. However, we can work around this by saving a reference to this specific span in the context so we can always have access to the "main" wrapping span.

// create a reference to store the span on the opentelemetry context object
const MAIN_SPAN_CONTEXT_KEY = createContextKey("main_span_context_key");

function mainSpanMiddleware(req, res, next) {
  // pull the active span created by the http instrumentation
  let span = trace.getActiveSpan();

  // get the current context
  let ctx = context.active();

  // set any attributes we always want on the main span
  span.setAttribute("main", true);

  // OpenTelemetry context is immutable, so to modify it we create
  // a new version with our span added
  let newCtx = ctx.setValue(MAIN_SPAN_CONTEXT_KEY, span);

  // set that new context as active for the duration of the request
  context.with(newCtx, () => {
    next();
  });
}

// create another function that allows you to annotate this saved span easily
function setMainSpanAttributes(attributes) {
  let mainSpan = context.active().getValue(MAIN_SPAN_CONTEXT_KEY);
  if (mainSpan) {
    mainSpan.setAttributes(attributes);
  }
}

Now our annotation code can look a little simpler, and we can always know that we're setting these attributes on the wrapping span.

setMainSpanAttributes({
  "user.id": "123",
  "user.type": "enterprise",
  "user.auth_method": "oauth",
});

You can play around with a minimal running example here.

At Heroku we had internal OpenTelemetry Distributions that set this up for you automatically and added as many automatic annotations as possible to these spans.

If you are not using OpenTelemetry here's a gist that might help you get started. My previous post may help you put this logic together.

What do I add to this "main" span?

<blockquote class="twitter-tweet" data-conversation="none" data-dnt="true" data-theme="light"><p lang="en" dir="ltr">And how many dimensions do you plan to emit and pack into your wide events?<br><br>MANY. Hundreds! The more you have, the better you can detect and correlate rare conditions with precision.<br><br>As you adjust to the joys of debugging with rich context, you will itch for it everywhere. ☺️</p>— Charity Majors (@mipsytipsy) <a href="https://twitter.com/mipsytipsy/status/1744579558962336138">January 9, 2024</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

We need to add attributes about the request, and there are likely far more of these than you would expect. It's easy to come up with a dozen or so, but in a well-instrumented code base there will be hundreds of attributes.

Note that while this is a long list, it is definitely not exhaustive. OpenTelemetry defines sets of attribute names as Semantic Conventions that can also be used for inspiration. I have tried to follow these in my naming where possible.

A convention to filter out everything else

Traces contain lots of spans, so it's helpful to have a convention for identifying and searching for these "wide events". root and canon were floated as options, but I've landed on calling them main spans.

Attribute	Examples	Description
`main`	`true`	Present only for spans designated as a "wide event", usually wrapping a request / response, or a background job

This convention allows you to quickly figure out "what does the traffic to this service look like?" with a single query:

SELECT
  COUNT(*)
WHERE
  main = true
GROUP BY http.route

Service metadata

Of course we need to add some information about the service we're running. Consider adding additional metadata about which team owns the system, or which Slack channel the owning team hangs out in, though note that this can be tedious to update if your workplace experiences frequent re-orgs. Tying these to a service catalog like Backstage is left as an exercise to the reader.

Attribute	Examples	Description
`service.name`	`api` <br> `shoppingcart`	What is the name of this service?
`service.environment`	`production` <br> `staging` <br> `development`	Where is this service running?
`service.team`	`web-services` <br> `dev-ex`	Which team owns this service. Useful for knowing who to page in during incidents.
`service.slack_channel`	`web-services` <br> `dev-ex`	If I discover an issue with this service, where should I reach out?

How many services does each team run?

SELECT
  COUNT_DISTINCT(service.name)
WHERE
  service.environment = "production"
GROUP BY service.team

Ever look at the load on a system and then wonder "Is that appropriate for the machine this is running on?", and now you have to look through other tools or config files to get that information. Throw that context on the wide event so that it's available when you need it.

Attribute	Examples	Description
`instance.id`	`656993bd-40e1-4c76-baff-0e50e158c6eb`	An ID that maps to this one instance of the service
`instance.memory_mb`	`12336`	How much RAM is available to this service?
`instance.cpu_count`	`4` <br> `8` <br> `196`	How many cores are available to this service?
`instance.type`	`m6i.xlarge`	Does your vendor have a name for this type of instance?

What are the services with the most memory that we run? What instance types do they use?

SELECT
  service.name,
  instance.memory_mb,
  instance.type
ORDER BY instance.memory_mb DESC
GROUP BY service.name, instance.type
LIMIT 10

However you're orchestrating your systems make sure that all of the relevant information is added. I've included some examples from the Kubernetes semantic conventions for inspiration.

Attribute	Examples	Description
`container.id`	`a3bf90e006b2`	An ID used to identify Docker containers
`container.name`	`nginx-proxy` <br> `wordpress-app`	Container name used by container runtime
`k8s.cluster.name`	`api-cluster`	Name of the kubernetes cluster your service is running in
`k8s.pod.name`	`nginx-2723453542-065rx`	Name of the kubernetes pod your service is running in
`cloud.availability_zone`	`us-east-1c`	AZ where you're running your service
`cloud.region`	`us-east-1`	Region where you're running your service

But even if you're using a Platform-as-a-Service you can still pull out a lot of useful information!

Attribute	Examples	Description
`heroku.dyno`	`web.1` <br> `worker.3`	The env var `DYNO` that is set on your app at runtime
`heroku.dyno_type`	`web` <br> `worker`	The first part of the `DYNO` env var before the `.`. Separating this makes it easier to query
`heroku.dyno_index`	`1` <br> `3`	The second part of the `DYNO` env var after the `.`. Separating this makes it easier to query
`heroku.dyno_size`	`performance-m`	The selected dyno size
`heroku.space`	`my-private-space`	The name of the private space that your are deployed into
`heroku.region`	`virginia` <br> `oregon`	Which region is this app located in?

How many dynos are we running? What dyno types are they? For which services?

SELECT
  COUNT_DISTINCT(heroku.dyno_index)
GROUP BY service.name, heroku.dyno_type, instance.type

Build info

Inevitably some of the first questions asked in any incident are "Did something just go out?" or "What changed?". Instead of jumping to your deployment tool or looking through GitHub repositories, add that data to your telemetry.

Threading this data from your build system through to your production system so that it's available at runtime can be a non-trivial amount of glue code, but having this information easily available during incidents is invaluable.

Attribute	Examples	Description
`service.version`	`v123` <br> `9731945429d3d083eb78666c565c61bcef39a48f`	However you track your version, ex: a version string or a hash of the built image
`service.build.id`	`acd8bb57-fb9f-4b2d-a750-4315e99dac64`	If your build system gives you an ID, this context allows you to audit the build if something goes wrong
`service.build.git_hash`	`6f6466b0e693470729b669f3745358df29f97e8d`	The git SHA of the deployed commit so you can know exactly which code was running
`service.build.pull_request_url`	`https://github.com/your-company/api-service/pull/121`	The url of the pull request that was merged that triggered the deploy
`service.build.diff_url`	`https://github.com/your-company/api-service/compare/c9d9380..05e5736`	A url that compares the previously deployed commit against the newly deployed commit
`service.build.deployment.at`	`2024-10-14T19:47:38Z`	Timestamp when the deployment process started
`service.build.deployment.user`	`keanu.reeves@your-company.com`	Which authenticated user kicked off the build? Could be a bot
`service.build.deployment.trigger`	`merge-to-main` <br> `slack-bot` <br> `api-request` <br> `config-change`	What triggered the deploy? Extremely valuable context during an deploy-triggered incident
`service.build.deployment.age_minutes`	`1` <br> `10230`	How old is this deploy? Shortcuts the frequent incident question "Did something just go out?"

Won't this be a lot of repetitive data? These values do not change except between deploys! See Frequent Objections

What systems have recently been deployed?

SELECT
  service.name,
  MIN(service.build.deployment.age_minutes) as age
WHERE
  service.build.deployment.age_minutes < 20
GROUP BY service.name
ORDER BY age ASC
LIMIT 10

What's up with the spike of 500s when we did the last deploy?

SELECT
  COUNT(*)
WHERE
  service.name = "api-service" AND
  main = true
GROUP BY http.status_code, service.version

HTTP

You should get most of these from your tracing library instrumentation, but there are usually more you can add if, for example, your organization uses non-standard headers. Don't settle for only what OpenTelemetry gives you by default!

Attribute	Examples	Description
`server.address`	`example.com` <br> `localhost`	Name of the HTTP server that received the request
`url.path`	`/checkout` <br> `/account/123/features`	URI path after the domain
`url.scheme`	`http`, `https`	URI scheme
`url.query`	`q=test`, `ref=####`	URI query component
`http.request.id`	`79104EXAMPLEB723`	Platform request id: ex: `x-request-id`, `x-amz-request-id`
`http.request.method`	`GET` <br> `PUT` <br> `POST` <br> `OPTIONS`	HTTP request method
`http.request.body_size`	`3495`	Size of the request payload body in bytes
`http.request.header.content-type`	`application/json`	Value of a specific request header, "content-type" in this case, but there are many more. Pick out any that are important for your service
`http.response.status_code`	`200` <br> `404` <br> `500`	HTTP response status code
`http.response.body_size`	`1284` <br> `2202009`	Size of the response payload body in bytes
`http.request.header.content-type`	`text/html`	Value of a specific response header, "content-type" in this case, but there are many more. Pick out any that are important for your service

SELECT
  HEATMAP(http.response.body_size),
WHERE
  main = true AND
  service.name = "api-service"

User-Agent headers contain a wealth of info. Don't rely on regex queries to try and make sense of them down the road. Parse them into structured data from the beginning.

Attribute	Examples	Description
`user_agent.original`	`Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.3`	The value of the HTTP `User-Agent` header
`user_agent.device`	`computer` <br> `tablet` <br> `phone`	Device type derived from the `User-Agent` header
`user_agent.OS`	`Windows` <br> `MacOS`	OS derived from the `User-Agent` header
`user_agent.browser`	`Chrome` <br> `Safari` <br> `Firefox`	Browser derived from the `User-Agent` header
`user_agent.browser_version`	`129` <br> `18.0`	Browser version derived from the `User-Agent` header

What browsers are my users using?

SELECT
  COUNT(*)
GROUP BY user_agent.browser, user_agent.browser_version

If you have any custom user agents or headers used as a convention within your org parse that out too.

Attribute	Examples	Description
`user_agent.service`	`api-gateway` <br> `auth-service`	If you have a distributed architecture, have each service send a custom `User-Agent` header with its name and version
`user_agent.service_version`	`v123` <br> `6f6466b0e693470729b669f3745358df29f97e8d`	If you have a distributed architecture, have each service send a custom `User-Agent` header with its name and version
`user_agent.app`	`iOS` <br> `android`	If a request is coming from a mobile app, make sure it includes which app and its version
`user_agent.app_version`	`v123` <br> `6f6466b0e693470729b669f3745358df29f97e8d`	If a request is coming from a mobile app, make sure it includes which app and its version

Route info

We're not done with HTTP attributes yet! One of the most important bits is the API endpoint that the request matched. OpenTelemetry SDKs will usually give this to you automagically but not always. Consider extracting the route parameters and query parameters as additional attributes.

Attribute	Examples	Description
`http.route`	`/team/{team_id}/user/{user_id}`	The route pattern that the url path is matched against
`http.route.param.team_id`	`14739` <br> `team-name-slug`	The extracted segment of the url path as it is parsed for each parameter
`http.route.query.sort_dir`	`asc`	The query parameters that are relevant to the response of your service. Ex: `?sort_dir=asc&...`

SELECT
  P99(duration_ms)
WHERE
  main = true AND
  service.name = "api-service"
GROUP BY http.route

User and customer info

Once you get the basics down, this is the most important piece of metadata that you can add. No automagic SDK will be able to encode the particulars of your user model.

It's common for a single user or account to be responsible for a 10%+ of a business' revenue, and frequently their usage patterns look significantly different than the average user. They probably have more users, store more data, and hit limits and edge-cases that will never show up for the user paying $10 / month. Be sure you can separate their traffic from others.

Attribute	Examples	Description
`user.id`	`2147483647` <br> `user@example.com`	The primary ID for a user. If this is an email and you're using a vendor, consider your org's policy on putting PII in external services.
`user.type`	`free` <br> `premium` <br> `enterprise` <br> `vip`	How does the business see this type of user? Individual accounts are sometimes responsible for 10%+ of a business' income. Make sure you can separate their traffic from others!
`user.auth_method`	`token` <br> `basic-auth` <br> `jwt` <br> `sso-github`	How did this user authenticate into your system?
`user.team.id`	`5387` <br> `web-services`	If you have a team construct, which one does this user belong to?
`user.org.id`	`278` <br> `enterprise-name`	If this user is part of an organization with an enterprise contract, track that!
`user.age_days`	`0` <br> `637`	Not the user's literal age, but how long ago was this account created? Is this an issue experienced by someone new to your app, or only once they've saved a lot of data?
`user.assumed`	`true`	Have an internal way of assuming a user's identity for debugging? Be sure to track this
`user.assumed_by`	`engineer-3@your-company.com`	And track which actual user is assuming the user's identity

SELECT
  P99(duration_ms)
WHERE
  main = true AND
  service.name = "api-service"
GROUP BY user.type

Rate limits

Whatever your rate limiting strategy, make sure the current rate limit info gets added too. Can you quickly find examples of users that are being rate-limited by your service?

Attribute	Examples	Description
`ratelimit.limit`	`200000`	You might not now, but you will likely have users with different rate limits in the future, note down what the actual limit is for this request
`ratelimit.remaining`	`130000`	What is the budget remaining for this user?
`ratelimit.used`	`70000`	How many requests have been used in the current rate window
`ratelimit.reset_at`	`2024-10-14T19:47:38Z`	When will the rate limit be reset next? if applicable

This user has a support ticket open about being rate-limited. Let's see what they were doing

SELECT
  COUNT(*)
WHERE
  main = true AND
  service.name = "api-service" AND
  user.id = 5838
GROUP BY http.route

What routes are users who have burned most of their rate limit hitting? Does this activity look suspicious?

SELECT
  COUNT(*)
WHERE
  main = true AND
  service.name = "api-service" AND
  ratelimit.remaining < 100
GROUP BY http.route

Caching

For every code path where we could shortcut with a cache response, add whether or not it was successful

Attribute	Examples	Description
`cache.session_info`	`true` <br> `false`	Was the session info cached or did it need to be re-fetched?
`cache.feature_flags`	`true` <br> `false`	Were the feature flags cached for this user or did they need to be re-fetched?

Localization info

What localization options has the user chosen? This can be a frequent source of bugs

Attribute	Examples	Description
`localization.language_dir`	`rtl`, `ltr`	Which direction is text laid out in their language?
`localization.country`	`mexico`, `uk`	Which country are they from?
`localization.currency`	`USD`, `CAD`	Which currency have they chosen to work with?

Uptime

Tracking how long the service has been running when it serves a request can help you visualize several classes of bugs:

Issues that show up on a reboot
Memory leaks that only start to show up when the service has been running for a long time
Frequent crashes / restarts if you have automatically restart the service on failure

I recommend also either adding the log10 of the uptime or having some way of visualizing this. When graphed this emphasizes the important first few minutes of a service without being squished into the bottom of the graph by instances with several days or more of uptime.

Attribute	Examples	Description
`uptime_sec`	`1533`	How long has this instance of your app been running? Can be useful to visualize to see restarts
`uptime_sec_log_10`	`3.185`	Grows sub-linearly which allows you to visualize long-running services and brand new ones on the same graph

SELECT
  HEATMAP(uptime_sec),
  HEATMAP(uptime_sec_log_10)
WHERE
  main = true AND
  service.name = "api-service"

Metrics

This one might be a bit controversial, but I've found it helpful to tag spans with context about what the system was experiencing while processing the request. We fetch this information every ~10 seconds, cache it, and add it to every main span produced during that time.

Capturing metrics in this way is not mathematically sound. Since you only get data when traffic is flowing, you can't calculate a P90 for cpu load that would stand up to any rigorous scrutiny, but that's actually fine in practice. It's close enough to get some quick signal while you're debugging without switching to a different tool, especially if you can avoid calculations and visualize with a heatmap.

I wouldn't recommend setting alerts on this data though. Plain ol' metrics are great for that.

Jessica Kerr recently wrote about this approach on the Honeycomb Blog.

Attribute	Examples	Description
`metrics.memory_mb`	`153` <br> `2593`	How much memory is being used by the system at the time its service this request
`metrics.cpu_load`	`0.57` <br> `5.89`	CPU load of the system service this request. Given as # of active cores
`metrics.gc_count`	`5390`	Last observed number of garbage collections. Could be cumulative (total since service started) or delta (ex: number in the last minute)
`metrics.gc_pause_time_ms`	`14` <br> `325`	Time spent in garbage collections. Could also be cumulative or delta. Pick one and document which
`metrics.go_routines_count`	`3` <br> `3000`	Number of go routines running
`metrics.event_loop_latency_ms`	`0` <br> `340`	Cumulative time spent waiting on the next event loop tick. An important metric for Node apps

Are these requests getting slow because we're running out of memory or CPU?

SELECT
  HEATMAP(duration_ms),
  HEATMAP(metrics.memory_mb),
  HEATMAP(metrics.cpu_load)
WHERE
  main = true AND
  service.name = "api-service"
GROUP BY instance.id

Async request summaries

When using a tracing system async requests should get their own spans, but it can still be useful to roll up some stats to identify outliers and quickly find interesting traces.

Attribute	Examples	Description
`stats.http_requests_count`	`1` <br> `140`	How many http requests were triggered during the processing of this request?
`stats.http_requests_duration_ms`	`849`	Cumulative time spent in these http requests
`stats.postgres_query_count`	`7` <br> `742`	How many Postgres queries were triggered during the processing of this request?
`stats.postgres_query_duration_ms`	`1254`	Cumulative time spent in these Postgres queries
`stats.redis_query_count`	`3` <br> `240`	How many redis queries were triggered during the processing of this request?
`stats.redis_query_duration_ms`	`43`	Cumulative time spent in these redis queries
`stats.twilio_calls_count`	`1` <br> `4`	How many calls to this vendors api were triggered during the processing of this request?
`stats.twilio_calls_duration_ms`	`2153`	Cumulative time spent in these vendor calls

Surely my service makes a reasonable number of calls to the database... right?

SELECT
  HEATMAP(stats.postgres_query_count)
WHERE
  main = true AND
  service.name = "api-service"

Instead of adding this explicitly, couldn't we aggregate this by querying the whole trace? See Frequent Objections

Sampling

Once you start collecting fine-grained telemetry from your systems at a significant scale you run head-on into the problem of sampling. Running systems can produce a lot of data! Engineers frequently want to store and query all of it. Exact answers always! Make it fast! Also cheap! But it's trade-offs all the way down. Telemetry data is fundamentally different from the transaction data you're storing for your users, and you should think about it differently.

Luckily you only really need a statistically significant subset of the full dataset. Even sampling 1 out of every 1000 requests can provide a suprisingly detailed picture of the overall traffic patterns in a system.

Sampling is a suprisingly deep topic. Keep it simple if you're starting and do uniform random head sampling, but track your sample rate per-span so you can be ready for more sophisticated approaches down-the-line.

Good tooling will weight your calculations with a per-span, so you don't have to mentally multiple the COUNT call by the sample_rate to get an accurate answer. Here are some relevant articles:

Attribute	Examples	Description
`sample_rate`	`1` <br> `500`	`N` where 1 in `N` events will be sampled and stored and the rest dropped. If you're sampling `1%` of requests, the `sample_rate` would be `100`

Timings

I find it super useful to break up the work that gets done to respond to a request into a handful of important chunks and track how long each segment took on the main span.

Wait, isn't that what child spans are for?

Wrapping absolutely everything in its own span is the most common failure mode I see when engineers first get access to tracing tools. You have to design the structure of your data for the way you want to query it.

Child spans are helpful for waterfall visualization for a single request, but can be difficult to query and visualize across all of your requests. Putting that information on a single span makes it easier to query and also helps with tools like Honeycomb's BubbleUp which can then immediately tell you that that group of requests was slow because authentication took 10 seconds for some reason.

Attribute	Examples	Description
`auth.duration_ms`	`52.2` <br> `0.2`	How long did we spend performing authentication during this request?
`payload_parse.duration_ms`	`22.1` <br> `0.1`	Identify the core workloads of the service and add timings for them

Errors

If you encounter an error and need to fail the operation, tag the span with the error information: type, stacktrace, etc.

One approach that I have found super-valuable is tagging each location where we throw an error with a unique slug describing the error. If this string is unique within your codebase, it is easily found with a quick search. This allows someone to jump straight from a spike in errors on a dashboard to the exact line of code that throwing the error. It also provides a convenient low-cardinality field to GROUP BY.

You're unlikely to be able to wrap all possible errors, but any time a failed request doesn't have an exception.slug that is a good sign that you have places in your code where your error handling could be improved. It's now really easy to find examples of requests that failed in ways you didn't anticipate.

if isNotRecoverable(err) {
  // note the use of a plain string, not a variable, not dynamically generated
  // consider enforcing this with custom lint rules
  setErrorAttributes(err, "err-stripe-call-failed-exhausted-retries");
  throw err;
}

Attribute	Examples	Description
`error`	`true` <br> `false`	Special field for whether the request failed or not
`exception.message`	`Can't convert 'int' object to str` <br> `undefined is not a function`	The exception message encoded in the exception
`exception.type`	`IOError` <br> `java.net.ConnectException`	The programmatic type of the exception
`exception.stacktrace`	`ReferenceError: user is not defined`<br>`at myFunction (/path/to/file.js:12:2)`<br>`...`	Capture the stack trace if its available to help pin-point where the error is being thrown
`exception.expected`	`true`, `false`	Is this an expected exception like a bot trying to hit a url that doesn't exist? Allows filtering out of exceptions we can't prevent but don't need to worry about
`exception.slug`	`auth-error` <br> `invalid-route` <br> `github-api-unavailable`	Create a unique grepp-able slug-value to identify the code location of an error if its predictable during development time

Which of our enterprise users hit the most errors last week? And which one?

SELECT
  COUNT_DISTINCT(user.id)
WHERE
  main = true AND
  service.name = "api-service" AND
  user.type = "enterprise"
GROUP BY exception.slug

Show me traces where we likely need to improve our error handling

SELECT
  trace.trace_id
WHERE
  main = true AND
  service.name = "api-service" AND
  error = true AND
  exception.slug = NULL
GROUP BY trace.trace_id

Feature flags

Fine-grained feature flags are a developer super power that allows you to test code changes in production with only a fraction of your users or traffic. Adding the flag information per-request allows you to compare how the new code is working as you opt more of your traffic into the new code path. Coupled with the broad visibility you can get with wide events, and this can make even tricky migrations vastly more manageable and allow you to ship code with confidence.

Note that semantic conventions differ here and suggest adding feature flag information as events on the span. I would suggest following that standard since it will ultimately have the best support from vendors if it's moved to stable, but especially in the mean time, I'm also putting this info on the main span.

Attribute	Examples	Description
`feature_flag.auth_v2`	`true` <br> `false`	The value of a particular feature flag for this request
`feature_flag.double_write_to_new_db`	`true` <br> `false`	The value of a particular feature flag for this request

What errors are the users in the new authentication flows hitting? How does it compare to the control group?

SELECT
  COUNT
WHERE
  main = true AND
  service.name = "api-service" AND
GROUP BY feature_flag.auth_v2, exception.slug

Versions of important things

Runtimes, frameworks, and any major libraries you are using can be really helpful context.

Attribute	Examples	Description
`go.version`	`go1.23.2`	What version of your language runtime are you using?
`rails.version`	`7.2.1.1`	Pick out any core libraries like web frameworks and track their version too
`postgres.version`	`16.4`	If you can add the versions of any datastores you're using, even better

A security issue with Rails just got announced. What versions of the framework are our services using?

SELECT
  COUNT_DISTINCT(service.name)
WHERE
  service.environment = "production"
GROUP BY rails.version

Our memory usage seems higher that it used to be. Didn't we upgrade the runtime recently? Does that correlate?

SELECT
  HEATMAP(metrics.memory_mb)
WHERE
  main = true AND
  service.name = "api-service"
GROUP BY go.version

Your specific application

Now we go off the map and get to the really valuable stuff. Your app likely does something unique or works in a particular domain. You might need to really care about which professional credentials a Dentist using your app has, or which particular storage warehouse a package is in, or which chip is in the embedded tracking device installed in the cat that your app exists to track.

No framework is going to be able to understand what parts of your domain are important to track and automate this for you, you have to do that.

Attribute	Examples	Description
`asset_upload.s3_bucket_path`	`s3://bucket-name/path/to/asset.jpg`	If you upload something, add context about where
`email_vendor.transaction_id`	`62449c60-b51e-4d5c-8464-49217d91c441`	If you interact with a vendor, track whatever transaction ID they give you in case you need to follow up with them
`vcs_integration.vendor`	`github` <br> `gitlab` <br> `bitbucket`	If there are 3-4 types that something might fall into, be sure to add that context. Ex: If 2% of requests start failing because bitbucket is experiencing issues, this will help identify the source of the issue immediately.
`process_submission.queue_length`	`153` <br> `1`	Any time you interact with a queue, see if you can get the current length during submission

Things to note

You should probably add the thing

If you find yourself asking "Am I ever really going to need this bit of data?", default to throwing the attribute on. The marginal cost of each extra attribute is very small. If the data volume does start to grow, prefer wider, more context-rich events and a higher sample rate vs smaller events with a lower sample rate.

Heatmaps are your friend

Honeycomb's heatmaps are amazing at helping you find outliers, seeing multi-modal distributions, and getting a feel for your data. I wish more tooling supported them. I am not sure I can build software without them any more.

Embrace the feedback loop

When you are modifying code, make a change to the telemetry so that you can see the impact of the new code running. Once the code is released, check to make sure that you see the outcome you expected. Don't hesitate to add specific fields for one release and them remove them after.

Tighter feedback loops are like going faster on a bicycle. They make for more stable systems and let you move faster with confidence.

Semantic conventions and naming consistency

I've tried to embrace semantic conventions in my naming, but would not be surprised if I've made multiple errors. Naming is hard!

It's also hard to get consistency right within an organization or even across multiple systems owned by the same team. I would recommend trying to use semantic conventions as a guide, but do prioritize getting data out of your system in some form and getting some early wins over exacting adherence to an evolving specification. Once this data has proven its value within your organization, then you will have the leverage to spend engineering cycles on making things consistent.

In the long run semantic conventions should allow Observability vendors to build new value and understanding on top of the telemetry you emit, but this effort is only just getting started.

Frequent Objections

Does this really work??

I have done this for dozens of production systems. Every single time the data has been invaluable for digging in and understanding what the system is actually doing, and we've found something surprising, even for the engineers who had worked on the system for many years.

Things like:

Oh, actually 90% of the traffic of this system comes from one user
Wait, one of our worker processes is actually running a month old version of the code somehow?
This API endpoint usually has payloads of 1-2kb, but there is an edge case affecting one user where it's 40+MB. This causes their page loads to be several minutes longer than the p99.
After instrumenting the authentication middleware, around 20% of requests still didn't have user info. There was a whole second authentication system for a different class of users that hadn't been touched in years.
This endpoint that we'd like to deprecate accepts data in the form of A, B, and C, but none of our traffic ever even uses C. We can just drop support for that now.

I don't like it. This feels wrong

For anyone feeling that way now, I ask you to give it five minutes.

I find that when a log line wraps around your terminal window multiple times, most developers have a pretty visceral negative reaction.

This feels right:

[2024-09-18 22:48:32.990] Request started http_path=/v1/charges request_id=req_123
[2024-09-18 22:48:32.991] User authenticated auth_type=api_key key_id=mk_123 user_id=usr_123
[2024-09-18 22:48:32.992] Rate limiting ran rate_allowed=true rate_quota=100 rate_remaining=99
[2024-09-18 22:48:32.998] Charge created charge_id=ch_123 permissions_used=account_write request_id=req_123
[2024-09-18 22:48:32.999] Request finished http_status=200 request_id=req_123

But this feels wrong:

[2024-10-20T14:43:36.851Z] duration_ms=1266.1819686777117 main=true http.ip_address=92.21.101.252 instance.id=api-1 instance.memory_mb=12336
instance.cpu_count=4 instance.type=t3.small http.request.method=GET http.request.path=/api/categories/substantia-trado
http.route=/api/categories/:slug http.request.body.size=293364 http.request.header.content_type=application/xml
user_agent.original="Mozilla/5.0 (X11; Linux i686 AppleWebKit/535.1.2 (KHTML, like Gecko) Chrome/39.0.826.0 Safari/535.1.2" user_agent.device=phone
user_agent.os=Windows user_agent.browser=Edge user_agent.browser_version=3.0 url.scheme=https url.host=api-service.com service.name=api-service
service.version=1.0.0 build.id=1234567890 go.version=go1.23.2 rails.version=7.2.1.1 service.environment=production service.team=api-team
service.slack_channel=#api-alerts service.build.deployment.at=2024-10-14T19:47:38Z
service.build.diff_url=https://github.com/your-company/api-service/compare/c9d9380..05e5736
service.build.pull_request_url=https://github.com/your-company/api-service/pull/123
service.build.git_hash=05e5736 service.build.deployment.user=keanu.reeves@your-company.com
service.build.deployment.trigger=manual container.id=1234567890 container.name=api-service-1234567890 cloud.availability_zone=us-east-1
cloud.region=us-east-1 k8s.pod.name=api-service-1234567890 k8s.cluster.name=api-service-cluster feature_flag.auth_v2=true
http.response.status_code=401 user.id=Samanta27@gmail.com user.type=vip user.auth_method=sso-google user.team_id=team-1

You are structuring data so that it can be read efficiently by machines, not humans. Our systems emit too much data to waste precious human lifetimes using our eyeballs to scan lines of text looking for patterns to jump out. Let the robots help.

This seems like a lot of work

If you want to implement everything I've talked about in this post that would be a ton of work. However, even implementing the easiest subset is going to provide a lot of value. Not doing this results in so much more work building a mental model of your system, trying to debug by thinking through the code and hoping your mental model matches reality.

A lot of this logic can be put into shared libraries within your org, though getting them adopted, keeping them updated and in-sync, and getting engineers used to these tools presents a whole different set of challenges.

Many of these things could be surfaced to you by opinionated platforms or frameworks. I would love to see things move in this direction.

Isn't this a lot of data? Won't it cost a lot??

First, you should compare this to your current log volume per request. I have seen many systems where this approach would reduce overall log volume.

However storing this data for every request against your system could be too expensive at scale. That's where sampling comes in. Sampling gives you the controls to determine what you want to spend vs the value you receive from storing and making that data available to query.

Realtime OLAP systems are also getting cheaper all the time. Once upon a time Scuba held all data in memory to make these types of questions quick to answer. Now most OLAP systems are evolving to columnar files stored on cloud object storage with queries handled by ephemeral compute which is many orders of magnitude cheaper.

In the next section I'll show just how much cheaper.

Repeated data

Many of these fields will be the same for every request. Isn't that really inefficient?

This is where our intuitions can lie to us. Let's look at a concrete example.

I wrote a script <a href="#footnote-1" id="footnote-ref-1" data-footnote-ref="true" aria-describedby="footnote-label"><code>[1]</code></a> to generate a newline-delimited JSON file with a lot of the above fields and at least somewhat reasonable fake values.

Let's say our service is serving 1000 req/s all day and sampling 1% of that traffic. Rounding to a whole number, that's about a million events. Generating a million example wide events results in a 1.6GB file.

http_logs.ndjson     1607.61 MB

But we repeat the keys on every single line. Even just turning it into a CSV cuts the size by more than 50%.

http_logs.csv         674.72 MB

Gzipping the file shows an amazing amount of compression, hinting that this isn't as much data as we might think.

http_logs.ndjson.gz   101.67 MB

Column store formats like parquet and Duckdb's native format can do even better.

http_logs.parquet      88.83 MB
http_logs.duckdb       80.01 MB

They store all of the data for a specific column contiguously, which lends itself to different compression approachs. In the simplest case, if the column is always the same value, it can store that fact only once. Values that are the same across an entire row group are incredibly cheap.

If there are 2-3 different values, it can use dictionary-encoding to bit-pack these values really tightly. This also speeds up queries against this column.

DuckDB has a great writeup on this which goes into much more detail. All of the data remains available and is easily (and quickly!) queryable.

This is hardly "big data". Storing this on Cloudflare's R2 for a month would cost US$ 0.0012. You could keep 60 days of retention for US$ 0.072 / month.

❯ duckdb http_logs.duckdb
D SELECT COUNT(*) FROM http_logs;
┌──────────────┐
│ count_star() │
│    int64     │
├──────────────┤
│      1000000 │
└──────────────┘
Run Time (s): real 0.002 user 0.002350 sys 0.000946
D SELECT SUM(duration_ms) FROM http_logs;
┌───────────────────┐
│ sum(duration_ms)  │
│      double       │
├───────────────────┤
│ 999938387.7714149 │
└───────────────────┘
Run Time (s): real 0.003 user 0.008020 sys 0.000415

There are even in-memory and transport formats to help reduce size in memory and on the wire. OpenTelemetry is adopting arrow for its payloads for this reason.

I found this podcast on the FDAP stack particularly helpful in understanding this space.

Couldn't we `JOIN` data from multiple spans together to get this information? Query the whole trace at once?

This is certainly possible. Honeycomb has started allowing you to filter on fields on other spans in the same trace. However I'd qualify this as very advanced. You want to make the right thing the easiest thing, and if you make it harder to ask questions, people will simply ask fewer questions. There are already a million things competing for our attention. Keep it simple. Make it fast.

Does this mean I don't need metrics?

You should probably still generate high-level metrics, though I bet you will need far fewer.

Metrics are great when you know you want an exact answer to a very specific question that you know ahead of time. Questions like "How many requests did a serve yesterday?" or "What was my CPU usage like last month?"

<a href="#footnote-ref-1" id="footnote-1"><code>[1]</code></a> Well... mostly Cursor wrote it

OpenTelemetry Tracing in 200 lines of code

Wed, 11 Sep 2024 00:00:00 GMT

Developers tend to treat tracing as deep magic, and OpenTelemetry is no exception. OpenTelemetry may be even more mysterious given how many concepts your are exposed to even with beginning examples. <a href="#footnote-1" id="footnote-ref-1" data-footnote-ref="true" aria-describedby="footnote-label"><code>[1]</code></a>

It also doesn't help that as part of building a mature, battle-tested tracing library, the code itself tends to become more and more cryptic over time, contorting itself to avoid edge-cases, work across many environments, and optimize code paths to avoid impacting performance of its host applications. Auto-instrumentation is particularly prone to inscrutibility as it seeks to auto-magically wrap code that may never have been intended to be wrapped or extended.

It's no wonder then that most developers approach tracing libraries as unknowable black boxes. We add them to our applications, cross our fingers, and hope they give us useful information when the pager goes off at 2am.

They are likely a lot simpler than you expect! Once you peel back the layers, I find a useful mental model of tracing looks like "fancy logging" combined with "context propagation" a.k.a "passing some IDs around".

Logs

Developers tend to be very comfortable with logs. We start off with "Hello world!", and they stay with us forever after. We reach for them to add console.log("potato") to see if our code is even being run (even though the debugger is like right there).

Logs are useful! Though hopefully someone comes along and convinces you that your logs should always be structured as sets of key / value pairs.

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">If you make one New Year’s resolution related to modernizing your infrastructure and preparing for the brave new world of distributed systems, I suggest this:<br><br>Structure your logs.<br><br>You won’t be sorry. You will eventually be so, so glad you did.<br><br>Just do it.</p>— Charity Majors (@mipsytipsy) <a href="https://twitter.com/mipsytipsy/status/951272678345687040?ref_src=twsrc%5Etfw">January 11, 2018</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

It's nice to have some consistency in your logs: make sure each one has a consistently-formatted timestamp or includes a field like "name" so you can easily find them when searching. You probably have found yourself writing a helper function like this in your projects:

log("user-authenticated", { userId, remaingingRateLimit });

// ...

function log(name, attributes = {}) {
  console.log(
    JSON.format({
      name,
      timestamp: new Date().getTime(),
      ...attributes,
    })
  );
}

{ "timestamp": 1725845013447, "name": "user-authenticated", "userId": "1234", "remaingingRateLimit": 100 }

You might also have written something to help track how long some sub-task takes:

logTiming("query-user-info", () => {
  db.fetchUserInfo();
});

// ....

function logTiming(name, attributes = {}, lambda) {
  let startTime = new Date().getTime();

  // run some subtask
  lambda();

  let durationMs = new Date().getTime() - startTime;
  log(name, { durationMs, ...attributes });
}

{ "timestamp": 1725845013447, "name": "query-user-info", "durationMs": 12 }

If so, congrats! You're halfway to re-inventing trace spans.

Spans are ✨fancy✨ log lines

A trace is built-up of spans. The example below shows a single trace with 4 different spans:

You can think of a span as a collection of key / value pairs, much like a log line, with a few required fields. Spans must have:

a name
a timestamp
a duration
a set of IDs: trace ID, span ID, and a parent span ID. More about these later

All other fields can be added as additional keys and values in an attributes map.

In code this might look something like this:

let span = new Span("api-req");
let resp = await api("get-user-limits");
span.setAttributes({ responseCode: resp.code });
span.End();
console.log(span);

// ...

class Span {
  constructor(name, context = {}, attributes = new Map()) {
    this.startTime = new Date().getTime();
    this.traceID = context.traceID ?? crypto.randomBytes(16).toString("hex");
    this.parentSpanID = context.spanID ?? undefined;
    this.spanID = crypto.randomBytes(8).toString("hex");
    this.name = name;
    this.attributes = attributes;
  }

  setAttributes(keyValues) {
    for (let [key, value] of Object.entries(keyValues)) {
      this.attributes.set(key, value);
    }
  }

  end() {
    this.durationMs = new Date().getTime() - this.startTime;
  }
}

This output would be:

Span {
  startTime: 1722436476271,
  traceID: 'cfd3fd1ad40f008fea72e06901ff722b',
  parentSpanID: undefined,
  spanID: '6b65f0c5db08556d',
  name: 'api-req',
  attributes: Map(0) {
    "responseCode": 200
  },
  durationMs: 3903
}

which we could format as an equivalent log line:

{
  "startTime": 1722436476271,
  "traceID": "cfd3fd1ad40f008fea72e06901ff722b",
  "spanID": "6b65f0c5db08556d",
  "name": "api-req",
  "responseCode": 200,
  "durationMs": 3903
}

Traces are collections of spans

If you want to see all of the logs from a particular request, you have likely done something like:

// generate a request id and inherit one from your platform
let requestID = req.headers["X-REQUEST-ID"];
// ...
log("api-request-start", { requestID });
let resp = await apiRequest();
log("api-request-end", { requestID });

which would allow you to search for a particular request id to see what happened while that particular request was executed:

{ "timestamp": 1722436476271, "requestID": "1234", "name": "fetch-user-permissions" }
{ "timestamp": 1722436476321, "requestID": "1234", "name": "api-request-start" }
{ "timestamp": 1722436476345, "requestID": "1234", "name": "api-request-end" }
{ "timestamp": 1722436476431, "requestID": "1234", "name": "update-db-record" }
{ "timestamp": 1722436476462, "requestID": "1234", "name": "create-email-job" }

Tracing execution like this can get you surprisingly far! But we can do better.

Trace spans have 3 different IDs that make up the Trace Context. The first two are really simple:

Span ID: a random ID for each span
Trace ID: a random ID that multiple spans can share

The last one is a little more complicated:

Parent Span ID: the Span ID that was the active span when this span was created

The Parent Span ID is what allows a system to create a DAG out of each trace once it has received each individual span. When rendered as a tree this results in the waterfall view we all know and love, but it's important to remember that this is only one possible visualization of the data.

Existing in the context

Our context only really needs two values: the trace id, and the id of the current span. When we create a new span, we can inherit the spanID if one exists, generate a new spanID, and set the new context.

let context = {
  traceID: "cfd3fd1ad40f008fea72e06901ff722b",
  spanID: "6b65f0c5db08556d",
};

We need some way of passing this context around our application. We could do this manually:

let { span, newContext } = new Span("api-req", oldContext);

Indeed, this is how it is done in the official Go SDK however in most languages this is done implicitly and handled automatically by the library. In Ruby or Python you can use a thread local variable, but in Node you would use AsyncLocalStorage.

At this point it helps to wrap our span creation in a helper function:

import { AsyncLocalStorage } from "node:async_hooks";

let asyncLocalStorage = new AsyncLocalStorage();
// start with an empty context
asyncLocalStorage.enterWith({ traceID: undefined, spanID: undefined });

async function startSpan(name, lambda) {
  let ctx = asyncLocalStorage.getStore();
  let span = new Span(name, ctx, new Map());
  await asyncLocalStorage.run(span.getContext(), lambda, span);
  span.end();
  console.log(span);
}

startSpan("parent", async (parentSpan) => {
  parentSpan.setAttributes({ outerSpan: true });
  startSpan("child", async (childSpan) => {
    childSpan.setAttributes({ outerSpan: false });
  });
});

And with that we have the core of our tracing library done! 🎉

Span {
  startTime: 1725850276756,
  traceID: 'b8d002c2f6ae1291e0bd29c9791c9756',
  parentSpanID: '50f527cbf40230c3',
  name: 'child',
  attributes: Map(1) { 'outerSpan' => false },
  spanID: '8037a93b6ed25c3a',
  durationMs: 11.087375000000002
}
Span {
  startTime: 1725850276744,
  traceID: 'b8d002c2f6ae1291e0bd29c9791c9756',
  parentSpanID: undefined,
  name: 'parent',
  attributes: Map(1) { 'outerSpan' => true },
  spanID: '50f527cbf40230c3',
  durationMs: 26.076625
}

Our tracing library is perfectly usable even at under 60 LoC but it has 2 big drawbacks:

We have to manually add it everywhere
It also is restricted to a single system. We do not have a mechanism for passing the trace context between any two systems in our larger service.

Let's fix that!

Going distributed

Distributed tracing sounds scary and intimidating, but it generally means that tracing context can be passed around between systems so that you can track what operation kicked off what other operation, and that all of this data gets reported to the same place at the end.

Whenever we make an HTTP call to another system, we need to pass along the Trace ID and the Current Span ID. We could add these two fields to all of our HTTP payloads manually but there's a W3C standard for how to encode this into an HTTP header so that it gets sent as metadata for every request. The traceparent header looks like:

00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

There is a full specification to digest and implement here, but for our purposes today we can ignore most of this and think of it as conforming to this format:

00-{ Trace ID}-{ Parent Span ID }-01

which allows us to parse and serialize our trace context with some simple functions:

let getTraceParent = (ctx) => `00-${ctx.traceID}-${ctx.spanID}-01`;

let parseTraceParent = (header) => ({
  traceID: header.split("-")[1],
  spanID: header.split("-")[2],
});

We need to be sure to add it to our outgoing requests, and parse it on any incoming requests. Instrumentation helps with that.

Wrap stuff in other stuff

Instrumentation is a fancy term for "wrap some other code in our code so we can track stuff". Real tracing libraries sometimes go to extreme lengths to make wrapping built-in or popular libraries happen behind the scenes when you configure the library. We're not going to be doing that.

Instead we'll provide some middleware for the Hono web framework that the user can manually add.

async function honoMiddleware(c, next) {
  // check the incoming request for the traceparent header
  // if it exists parse and inherit the trace context
  let context = EMPTY_CONTEXT;
  if (c.req.header("traceparent")) {
    context = parseTraceParent(c.req.header("traceparent"));
  }

  // set the context and wrap the whole req / res operation in a span
  await setContext(context, async () => {
    await startSpan(`${c.req.method} ${c.req.path}`, async (span) => {
      // Before we let our app handle the request, let's pull some info about
      // it off and add it to our trace.
      span.setAttributes({
        "http.request.method": c.req.method,
        "http.request.path": c.req.path,
      });

      // let our app handle the request
      await next();

      // Pull information about how our app responded
      span.setAttributes({
        "http.response.status_code": c.res.status,
      });
    });
  });
}

We also need to handle making outgoing HTTP calls and making sure we attach the traceparent header. There is no built-in concept of middleware for the built-in fetch command, so instead we're going to wrap it like a javascript burrito. We'll have to do this ourselves, but it's not so bad, especially compared to what doing it for real looks like.

// pass the original function into our wrapping logic
function patchFetch(originalFetch) {
  // return a function with the same signature, but that executes our logic too
  return async function patchedFetch(resource, options = {}) {
    // generate and add the traceparent header
    let ctx = getContext();

    if (!options.headers) {
      options.headers = {};
    }
    options.headers["traceparent"] = getTraceParent(ctx);

    // run the underlying fetch function, but wrap it in a span and
    // pull out some info while we're at it
    let resp;
    await startSpan("fetch", async (span) => {
      span.setAttributes({ "http.url": resource });
      resp = await originalFetch(resource, options);
      span.setAttributes({ "http.response.status_code": resp.status });
    });

    // pass along fetch's response. It's like we were never here
    return resp;
  };
}

Here's a quick little app to show off our tracer in action.

Let's send this to Honeycomb

We can log out our spans to the terminal, but that's not a great user experience. Before we look at OpenTelemetry, I think it's instructive to look at Honeycomb's Event's API. Before Honeycomb went all-in on OpenTelemetry they had a much simpler just-send-us-JSON approach. They no longer recommend it, but we can still use this API today for our toy project.

You can see the full exporter code here, but the logic for building the payload is the interesting bit:

// literally put all of the data together in one big json blob... like a log line!
// and then POST it to their API
function spanToHoneycombJSON(span) {
  return {
    ...Object.fromEntries(globalAttributes),
    ...Object.fromEntries(span.attributes),
    name: span.name,
    trace_id: span.traceID,
    span_id: span.spanID,
    parent_span_id: span.parentSpanID,
    start_time: span.startTime,
    duration_ms: span.durationMs,
  };
}

Since we're not using a standard format you do have to tell Honeycomb which field maps to Trace ID, Span ID, etc in the dataset configuration, but this is all the data needed to build up the trace waterfall!

Where's the OpenTelemetry?

So we have our ~~fancy logs~~ tracing set up, and instrumentation and context propagation are actually pretty simple, but OpenTelemetry is a big, complicated standard! Have you read through the specification!?

And... that's not wrong. OpenTelemetry is a big project, however we only need one small part of it for our purposes. When you install an OpenTelemetry SDK for your langauge it emits data in the OpenTelemetry Protocol (OTLP). Every OpenTelemetry SDK in every language emits OTLP. The OpenTelemetry Collector is a collection of tools for receiving OTLP, transforming OTLP, translating other formats to OTLP. You could say OTLP is kind of a big deal.

OTLP has its own protobuf specification so you can efficiently compress telemetry data into a binary message that will be the same across any platform, OS, or CPU architecture. We could generate a JavaScript module to parse and emit these protobuf messages from the .proto files, but that sounds like too much work.

Protobuf also defines a JSON mapping as part of the spec, and the specification repo has a handy example for us to start from, so let's keep it simple instead.

Generating this payload is a bit more complicated than Honeycomb's old events format. There are some new terms like "resource" and "scope", and we have to massage the attributes a bit. However if you squint, you can see that it's all the same data. Full version here

function spanToOTLP(span) {
  return {
    resourceSpans: [
      {
        resource: {
          attributes: toAttributes(Object.fromEntries(globalAttributes)),
        },
        scopeSpans: [
          {
            scope: {
              name: "minimal-tracer",
              version: "0.0.1",
              attributes: [],
            },
            spans: [
              {
                traceId: span.traceID,
                spanId: span.spanID,
                parentSpanId: span.parentSpanID,
                name: span.name,
                startTimeUnixNano: span.startTime * Math.pow(10, 6),
                endTimeUnixNano:
                  (span.startTime + span.elapsedMs) * Math.pow(10, 6),
                kind: 2,
                attributes: toAttributes(Object.fromEntries(span.attributes)),
              },
            ],
          },
        ],
      },
    ],
  };
}

However the power of standards means that we're no longer limited to just one vendor. We can now send this to any service that accepts OpenTelemetry!

Honeycomb still works, of course.

Baselime too!

We can visualize our telemetry locally using otel-desktop-viewer!

We can even render our data in the terminal with otel-tui

That's it!?

That's it! In 181 lines of code we've implemented tracing, trace context propagation, instrumentation, and exported it in a standard format that any vendor or tool should be able to accept. Thanks to the magic of 🌈standards🌈.

Is this... you know... legal?

If you were paying attention, there are a lot of OpenTelemetry specifications around how an OpenTelemetry SDK should be structured and behave. Our little library doesn't do almost any of that. This is just for learning but one could imagine a simplified non-compliant SDK such as this one for Cloudflare Workers. It emits OTLP but doesn't conform to all of the SDK specifications. How should we think about this?

At the OTel Community Day last June, OpenTelemetry cofounder Ted Young was asked a similar question. I wrote down his answer as best as I could. Lightly paraphrased:

We don’t care about the spec. We care about that the black box participates in tracing and emits OTLP and semantic conventions. The true spec is the data.

I take that to mean, while official SDKs are expected to follow the spec, it's reasonable for other implementations to deviate as long as an outside observer cannot tell the difference from the behavior. Our little library does't quite pass due to the shortcuts taken in parsing the traceparent header, but it would not take much more code to fix this.

If OpenTelemetry continues to be successful I expect OTLP, the ability to emit and receive it, will get built into everything: your IDE tooling, the platform you deploy on, the framework on which you are building, even hardware. Some day your internet-connected dryer will almost certainly speak OTLP, if it doesn't already.

Hold on... why is opentelemetry-js so much bigger?

If we can build a functional tracer in under 200 lines, why does the official JavaScript SDK have so much more code?

It might help to go over a non-exhaustive list of things the offical SDK handles that our little learning library doesn't:

Buffer and batch outgoing telemetry data in a more efficient format. Don't send one-span-per-http request in production. Your vendor will want to have words.
Work in both the browser and in Node environments
Gracefully handle errors, wrap this library around your core functionality at your own peril
Be incredibly configurable. Need to do something that isn't bog standard? You can probably make it happen
Automagically wrap common libraries in robust, battle-tested instrumentation
Optimized for performance when used in your code's tight loops
Conform to semantic conventions
Support two whole other types of telemetry: metrics and logging

And more! I hope this post has given you a better mental model for the work happening under all of the production hardening and extensibility that tends to build up on these libraries when they have to be used in the real world. A library that can reliably work across most places we deploy JavaScript and meet a very broad range of user needs is highly non-trivial.

But tracing? We know that's just ✨fancy logging✨ right?

All code for this post can be found at jmorrell/minimal-nodejs-otel-tracer on Github.

<a href="#footnote-ref-1" id="footnote-1"><code>[1]</code></a> Count the concepts in this simple example

Tracer
TracerProvider
SpanExporter
SpanProcessor
Resource
Attribute
Event
Span
Context
Both an API and and SDK
Semantic Conventions

This is intimidating to even senior developers. OpenTelemetry knows this learning curve is a problem and has spun up a new SIG to work on improving the developer experience.