OpenAPI is hard

About three years ago, in early 2022, I was sitting at my desk at 6 a.m. with my head in my hands.

I was absolutely sure it was going to work this time—I was sure I had nailed it. But alas, it turned out to be just another way to fail.

The problem staring me in the face was the same one every engineer encounters early on when pursuing this mission:

“How hard can it be?”

It turns out it can be really hard. Way harder than I ever expected.

What exactly is so hard?

I’m talking about building an engine, a framework, and a platform that can fully understand OpenAPI. It seems simple on the surface, right? It’s just a description language in YAML or JSON—how tough could that be?

Little Languages

Recently, I’ve been reading and enjoying Crafting Interpreters by Robert Nystrom. In the introduction, he notes:

‘For every successful general-purpose language, there are a thousand successful niche ones. We used to call them ‘little languages’, but inflation in the jargon economy led to the name ‘domain-specific languages’. These are pidgins tailor-built to a specific task. Think application scripting languages, template engines, markup formats, and configuration files.’

He then lists examples like Make, XSLT, SQL, JSON, BASH, YAML, HTML, and more.

This analogy really hit home. It exposes the root issue of why OpenAPI stumps so many engineers once they go beyond the simplest use cases.

OpenAPI isn’t just one language, it’s a language composed of multiple other languages. That complexity is where things begin to spiral out of control. For instance:

YAML supports anchors and aliases, and ordered lists
JSON supports neither anchors nor aliases, and ordering is not guaranteed
OpenAPI 3.0 uses an incompatible variation of JSON Schema.
OpenAPI 3.1 uses standardized JSON Schema.
JSON Schema itself is a complex standard.
JSON Pointers can be local, remote, absolute, or relative.

Surely this is just JSON

The first thought is often: “I can just parse this JSON into some data structures.”

It starts off fine, until duplication crops up. Then references (JSON Pointers) appear in the spec, and suddenly your code breaks. The model no longer works.

Now the engineer needs a lookup or hash table to index references and stitch them together.

Circular references cause infinite loops or stack overflows when pointers form direct or indirect cycles.

The realization hits that we’re basically writing an interpreter and compiler for a programming language, not a simple parser. And building a compiler is hard. It’s at this juncture that most folks give up and look for a library to handle it all.

After trying a bunch, they settle on one that mostly works—but still doesn’t feel complete.

Measure twice, cut once

About 18 months ago, I set out to build a visual navigation system to explore these vast graphs of references across a network of files.

I needed a super-powerful, low-level API to organize, navigate, and dig into references, files, directories, and relationships.

My existing mechanisms were too limited. So I came up with the idea of the ‘rolodex’: an over-engineered, complex system that can locate pointers to anything, anywhere across a massive network of files.

Looking up individual nodes across a graph is not simple

The rolodex holds a ‘spinning’ reel of indexes containing every pointer to everything in one place.

It looked like a huge, heavy lump of machinery inside libopenapi for a while, and took around three months to build it into the existing codebase and hook it up without breaking anything.

In the end, it had… more or less the same functionality as before. So much work for so little immediate gain?

This is the way

To showcase the power and value of this work, I’d like to draw your attention to the OpenAPI Doctor and one of its latest, greatest features.

When you load the OpenAPI Doctor, you’ll see the example Train Travel API from bump.sh.

Clickable references, jump about the document with ease

Now, all the references are lit up as green (regular) or yellow (polymorphic). Hovering over a reference highlights it and shows some metadata. If you hold down the Command/Control key and click, it’ll jump straight to that reference in the spec.

Navigating a spec is so much easier now. I’ve wanted this feature for years.

Let’s go deep

Let’s download Redocly’s OpenAPI starter example.

Head over to GitHub, click the green ‘Code’ button, then ‘Download ZIP.

Next, open the OpenAPI Doctor and click the blue ‘Import’ button.

There are two options: import by URL or upload a ZIP. Choose Upload .zip or .tar and select the ZIP file you just downloaded.

Upload the Redocly ZIP file downloaded from github

The doctor reads the ZIP, detects the OpenAPI file inside, and if there are multiple entry documents, it lets you pick which one to load.

The rolodex panel then opens, displaying the entire file system that makes up your OpenAPI spec. Folders can be expanded, files can be clicked, and problems or references are immediately rendered.

The Rolodex pane opens up for multi-file specifications

This new experience lets you see violations in context inside each file, even though everything is part of a single large graph.

Not only can you jump to external references (to JSON pointers in other files or direct fragments), but you can also see and diagnose them in real-time.

Even Markdown files or sample code referenced by the graph are integrated seamlessly.

Exploring exploded and distributed specs is so much easier

Only the files that are referenced are loaded into the rolodex, notice nothing except the specification files are rendered.

Nuke it

When you’re ready to start fresh, hit the Nuke Workspace option (the nuclear icon next to the disabled import button). It wipes everything clean—no take-backs.

Then re-import the same thing, but this time use the URL to the Redocly OpenAPI spec. The experience remains the same, except the directory structure is now the full remote path to the spec, instead of the ZIP file structure.

Let’s go hard

The Redocly’s OpenAPI starter example is a simple, practical example.

But what if we want to see a massive spec? A monster. Let’s ask the doctor to read the DigitalOcean OpenAPI spec.

It’s over 1,600 files with hundreds of thousands of references. It destroys most tools, but not the Doctor!

Paste the Digital Ocean OpenAPI Spec URL into the Doctor, and it’ll chomp through the entire graph.

Sucking down the monster from the internet

Give it a couple of seconds to load, and a second or two for the browser to render—and you’re in.

Explore the monster, now defeated by the doctor

The Grand Challenge

Now, try importing the DigitalOcean spec into any other OpenAPI editor or tool.

If you can find one that:

Doesn’t blow up
Doesn’t fail or error out
Actually shows the remote file system
Allows contextual exploring and debugging

…I’ll buy you and your family dinner.