Vladimír Gorej’s Site

From packets to pixels: how the internet actually works

2026-08-05T08:00:00+00:00

From packets to pixels · Part 1

This article is part of a hands-on series that rebuilds the web from first principles: the network first, then HTML, CSS, and JavaScript.

All parts

I have spent my whole career close to the web's machinery: five years maintaining Swagger tools, these days SpecLynx, UseArazzo and SwaggerExpert, plus daily work building tools that help agents talk to APIs. HTTP is where I live. And yet, when I recently stopped to examine my mental model of the internet itself, I realized how much of it I had never actually verified.

Most of what an experienced engineer knows about the layers beneath their own code is inherited rather than learned: absorbed from documentation skimmed at 2 a.m., from a colleague’s offhand explanation, from a bug that got fixed without ever being fully understood. That model works. It’s mostly right. But “mostly right” stays invisible until the day a DNS change stubbornly refuses to take effect, or a retried request duplicates an order, and you discover the gap had been sitting there the whole time.

So I went back and checked. Not the exotic corners: the basics. What a packet actually is. Why a bare domain traditionally can’t be a CNAME. What “DNS propagation” really means (nothing propagates anywhere). Where a browser spends its time between receiving bytes and showing you pixels. Most of it confirmed what I already believed. A few things corrected me.

This article is the write-up of that pass: the explanation I wish I’d been handed years ago, and the one I’d now give to anyone who builds on the web without having looked underneath it. It walks the whole path once, end to end: each section builds on the previous one, and each ends with hands-on exercises that need nothing more than a terminal and a browser.

Run them. That part isn’t decoration. Verifying something yourself is the difference between knowing it and having heard it, a distinction worth defending now that a confident, plausible explanation is always one prompt away. A language model will happily tell you how DNS works. dig +trace will show you.

A network of networks: packets, IPs, and ports

The internet is not a single thing you connect to. It is a network of networks: your home network, your ISP’s network, university networks, data-center networks, all agreeing to pass traffic between each other using the same set of rules (protocols). Nobody owns or runs the whole thing.

So where do separate networks physically touch? Mostly at Internet Exchange Points (IXPs). An IXP is, quite literally, a building (or a few racks in a data center) containing a large shared network switch. ISPs, cloud providers, and content companies each rent a port on that switch and agree to exchange traffic with each other directly, an arrangement called peering. Without an IXP, your ISP would have to pay intermediate carriers to haul your video stream across the country and back; with one, both networks plug into the same room and hand packets straight to each other. That keeps traffic local (a stream between two Prague networks never needs to leave Prague), cuts costs, and shortens the path your packets travel. The largest exchanges, like DE-CIX in Frankfurt or AMS-IX in Amsterdam, move tens of terabits per second at peak. Cooperation through shared protocols and peering agreements, not central control, is what holds the internet together.

One distinction worth fixing early: the internet is not the web. The internet is the plumbing: cables, routers, addresses, and the protocols that move data between machines. The web (websites, browsers, HTML, links) is just one application running on top of that plumbing, alongside email, video calls, online games, and messaging apps. This article covers the plumbing first, then climbs up to the web.

When you open a website, two machines are involved:

Client: the machine that asks for something (your laptop, your phone, a script).
Server: the machine that answers (a computer somewhere running software that listens for requests).

“Client” and “server” describe roles, not hardware. The same machine can be a client in one exchange and a server in another.

IP addresses: how machines find each other

Every device directly reachable on the internet has an IP address, a numeric label that works like a postal address for computers.

IPv4: four numbers 0–255, e.g. 142.250.185.78. Only about 4.3 billion possible addresses exist, and we’ve effectively run out.
IPv6: much longer hex addresses, e.g. 2a00:1450:4001:82f::200e. Designed to never run out.

Because IPv4 addresses are scarce, your home devices usually share one public IP address. Your router hands out private addresses (like 192.168.1.x) internally and translates between the two, a trick called NAT (Network Address Translation).

Packets: how data actually travels

Data is never sent as one continuous stream. It’s chopped into small chunks called packets (typically ~1,500 bytes). Each packet carries the destination IP, the source IP, a sequence number, and a slice of the actual data. The sequence number records the chunk’s position in the original message: packet 1, packet 2, packet 3, and so on. It’s what lets the receiving machine put the chunks back together in the right order, and notice when one never arrived so it can be re-sent.

Packets travel independently. Two packets from the same message may take different routes through the network and arrive out of order; the receiving machine uses the sequence numbers to reassemble them. This design makes the internet resilient: if one route fails, packets simply flow around it. Between your machine and a server, packets hop through a chain of routers, each forwarding the packet closer to its destination.

Protocols: the shared rules

None of this works unless every machine speaks the same language. That’s what protocols are: agreed-upon formats and procedures, stacked in layers:

Layer	Protocol	What it does
Addressing & routing	IP	Gets packets from one machine to another
Transport	TCP	Reliable, ordered delivery; retransmits lost packets
Transport	UDP	Fast, no guarantees; good for video calls, games
Application	HTTP, SMTP, DNS…	Meaning of the data (web pages, email, name lookups)

TCP is worth understanding: before sending data, client and server perform a handshake (SYN → SYN-ACK → ACK) to establish a connection. TCP then numbers every byte, acknowledges receipt, and re-sends anything lost. The web runs mostly on TCP (though HTTP/3 now runs on QUIC, explained in the HTTP section).

Ports: many conversations, one machine

A server has one IP address but runs many services. Ports (numbers 0–65535) distinguish them, like apartment numbers within one building:

Port	Service
80	HTTP (web, unencrypted)
443	HTTPS (web, encrypted)
22	SSH (remote shell)
25 / 587	SMTP (email)
53	DNS

So a full “address” for a conversation is really IP + port: 142.250.185.78:443 means “the service listening on port 443 at that machine.”

You already use this daily as a developer without leaving your desk. localhost is a reserved name (resolving to 127.0.0.1) that means “this very machine”: client and server both on your own computer. When you run a dev server and open http://localhost:3000, your browser is the client, your project is the server, and 3000 picks it out among everything else running locally. Dev tools grab high, unclaimed ports by convention (3000, 5173, 8000, 8080) because the low, well-known ones require admin rights and are spoken for.

The whole trip, previewed

When you visit a website, roughly this happens; the rest of this article unpacks each step:

Your machine looks up the site’s IP address (DNS).
It opens a TCP connection to that IP on port 443.
Client and server negotiate encryption (TLS).
Your browser sends an HTTP request.
The server, wherever the site is hosted, answers; the response is split into packets and routed hop by hop.
Your browser reassembles and renders it.

🧪 Try it yourself

The one Arazzo condition no tool can evaluate safely

2026-07-08T08:00:00+00:00

I'm building an Arazzo runner: a program that executes API workflows step by step. That means every step needs an answer to a deceptively simple question: did this step succeed?

In the case I was implementing, that answer came from a Criterion Object with a simple condition, such as $statusCode == 200. It looks obvious. It reads like plain English. But if Arazzo workflows are meant to be executable and portable, then every runner has to read that condition the same way.

So I wrote a quick evaluator. It handled the easy cases, then quietly fell apart on corner case after corner case I hadn’t anticipated. That was enough to make me stop, put the runner down, and go into research mode.

This is the story of what that turned up, and of how the detour became a standalone library, @swaggerexpert/arazzo-criterion: a small evaluator that reads each condition as faithfully as the spec permits, and is honest about the one case where no tool can know for sure.

What a condition looks like

Arazzo is the OpenAPI Initiative’s specification for describing sequences of API calls: workflows. After each step, you check that it worked by evaluating its success criteria. Each criterion is a Criterion Object, and one common form is a simple condition:

condition: $statusCode == 200

Read it out loud and it’s obvious: did the server return a 200 status code? That readability is the whole appeal. You can write conditions that sound like what you mean:

condition: $response.body.status == 'available' && $response.body.pets[0].id > 0

Is the status “available”, and does the first pet have a real id? A person can read that. My job was to write software that reads it the same way a person does, and, just as importantly, the same way every other runner does.

That portability requirement is the part that matters. Arazzo isn’t prose documentation for humans; it’s a machine-readable workflow description. If one runner treats a condition as true and another treats the same condition as false, the workflow has stopped being portable. At that point the problem is no longer just parsing. It’s interoperability.

My first instinct: just run it

My first idea was the lazy one: a condition looks like code. $statusCode == 200 is almost a line you could paste into a program and let the language run. Fill in the $... parts (like $statusCode) with their real values, hand the rest to the language’s eval, and read back true or false. An afternoon’s work.

I’m glad I didn’t ship that. The obvious objection is safety: feeding arbitrary text into eval is the classic way to run code you never meant to run.

So I reached for the safer version: parse the text with a proper JavaScript parser first (a well-tested component that turns code into a structured tree without executing anything), then walk that tree myself and decide what each piece means. No blind execution, no injection hole. This felt right for about a day.

Then the deeper problem surfaced, and it’s the reason this whole project exists. A JavaScript parser reads JavaScript’s rules. A Python runner would reach for a Python parser, a Go one for Go’s, each with its own subtly different semantics. I’d have traded “whatever eval does” for “whatever this one language’s parser does”, still shackled to a single language, with no guarantee that two conformant runners reach the same answer. Arazzo even layers on comparison rules that no language reproduces out of the box, but I’ll come back to those.

That was the click: I didn’t need JavaScript semantics. I needed Arazzo semantics. A definition that sits above any single language, that I could write down once and that every implementation, in any language, could follow to the exact same result. That’s what pushed me toward describing the condition language with its own precise, portable rulebook. And that’s where the real problem revealed itself.

Two languages, one line

Writing that rulebook down sounds simple. It isn’t, and here is why: that one innocent line is secretly two different mini-languages mixed together.

The first is for pointing at data: $statusCode, $response.body, $request.query.limit. Arazzo calls these runtime expressions, and each one just points at a value somewhere in the request or the response, saying where to look. They already have a precise definition of their own. I’d formalized that definition earlier in a separate library, @swaggerexpert/arazzo-runtime-expression, and building it out carefully even fed back into the standard: the work helped tighten the runtime-expression grammar in Arazzo 1.1.0 (PR #454). So my criterion library doesn’t reinvent any of that. It just reuses that parser.

The second is for comparing things: ==, !=, <, >, the word-joiners && and ||, and so on. This is the part that says what to check once you’ve looked.

In the common case I cared about, a condition is a runtime expression plus a comparison: ⟨where to look⟩ ⟨what to check⟩. Simple enough, until you notice that the two languages share some of the same symbols.

That’s the whole problem in one sentence. Let me show you why it bites.

Why the obvious reading breaks

Runtime expressions in Arazzo are allowed to be surprisingly loose. A query-parameter name, for instance, can legally contain almost any character, including =, <, >, even spaces. On its own that’s fine; a runtime expression is usually the only thing on the line, so there’s nothing after it to get confused with.

Now put a runtime expression next to a comparison and watch it go wrong:

condition: $request.query.limit == 10

You and I read that instantly: the query parameter limit equals 10. But the runtime-expression reader, following its own loose rules, sees no reason to stop at limit. Spaces are allowed in a name. So is =. So it happily swallows the entire line, == 10 included, treating the whole thing as if the parameter were literally named "limit == 10". The comparison never gets its turn, and the reading collapses.

The same line, read two ways. The whole problem is deciding where the runtime expression stops.

It’s a bit like reading the sentence “Call Mr. Green house painters today” and not knowing whether someone is named “Mr. Green” or “Mr. Green house.” Without a clear signal for where the name ends, a literal-minded reader keeps going and gets it wrong.

The part I could fix, and the part I couldn’t

Some of these collisions have a clean answer. Take the dot, ., which shows up both inside runtime expressions and as a way to reach into data. It sounds ambiguous, but it usually isn’t, because the runtime-expression rulebook itself knows where a runtime expression like $response.body naturally ends. So I can simply ask it, and let the rulebook draw the line rather than guessing. That collision, I could solve.

The unsolvable one is the shared =. Consider a query parameter that genuinely has an = in its name:

condition: $request.query.a=b == 1

Is that = part of the parameter’s name (a=b), or the start of the comparison ==? Both readings are legal. There is no rule, none exists in the Arazzo spec, that says “this = is part of the name” versus “this = is an operator.” A human can’t reliably tell either. Nobody can. The string is simply ambiguous, and no amount of clever programming invents an answer that isn’t there.

This is the honest heart of it: the ambiguity isn’t a flaw in my code. It’s a property of what was written. The best any implementation can do is be predictable about it and say so out loud.

Reading a condition the way it was meant

So how do you read a condition the way its author meant it, while staying upfront about the cases you can’t? Here’s the approach I shipped.

Instead of teaching my comparison-reader every rule of the runtime-expression language (copying someone else’s rulebook and letting it drift out of date), I do something simpler. When I hit a runtime expression, I grab the whole suspicious chunk and hand it to the one component that already knows every rule: @swaggerexpert/arazzo-runtime-expression, the parser from earlier, used unmodified. I ask it a single question: how much of this is a valid runtime expression?

The parser reads as far as it legitimately can and hands back the leftover. Whatever’s left is the “what to check” part. So for $response.body.data[0].id, it reports that $response.body is the runtime expression, and I treat the rest, .data[0].id, as the criterion language’s navigation into the resolved value: reach into the first data item’s id. The boundary is drawn by the component that actually knows the rules, not by me guessing.

Two things fall out of this:

The upside: I never fork or modify the runtime-expression rulebook. It stays the single source of truth, and my library just asks it. When those rules improve upstream, I get the improvement for free.
The catch: my library first does a rough pass (“this looks like a runtime expression, roughly”) and only then does the parser confirm it for real. Passing the rough pass isn’t enough; something can look like a runtime expression and still be nonsense. Think of a bouncer glancing at you versus someone actually checking your ID. That two-look approach, it turns out, is exactly how big specifications embed one language inside another: match the rough shape first, verify it for real second.

The one case that stays broken (on purpose)

In this simple grammar, because I stop the runtime-expression operand at the comparison symbols, that operand can never contain those symbols. That’s the flip side of making the common case work:

import { test } from '@swaggerexpert/arazzo-criterion';

test('$statusCode == 200');        // true:  reads exactly as intended
test('$request.query.a=b == 1');   // false: the '=' in the name can't be told apart
test('$response.body#/a=b == 1');  // false: a pointer can't fence the '=' either

That middle line is the unavoidable casualty from earlier, and it fails cleanly. It doesn’t silently guess wrong, which is the important part.

The last line is the one that looks like a way out. It is tempting to think the #/… pointer form acts as a fence, with the = sitting safely inside the pointer. It doesn’t. A JSON Pointer has no closing delimiter, and its tokens may contain = and even spaces. So the whole string $response.body#/a=b == 1 is also one valid runtime expression, with the pointer /a=b == 1. Same two readings, same refusal.

If you really need to reach a key like that, leave the simple type behind and use a jsonpath criterion. JSONPath quotes its member names, so $.pets[?@['a=b'] == 1] means exactly one thing.

Condition	How it reads	Result
`$statusCode == 200`	compare the status code to 200	works
`$request.query.limit == 10`	compare query param `limit` to 10	works
`$request.query.a=b == 1`	ambiguous: is `=` in the name, or the operator?	refused: ambiguous
`$response.body#/a=b == 1`	no fence either: a pointer token may contain `=` and spaces	refused: ambiguous

I didn’t bury this. It lives as a documented limitation, issue #3, with the reasoning spelled out. If a future version of the Arazzo spec adds a way to “escape” these symbols, the limitation lifts, so I’ve asked it to. Until then, being honest about it is the feature.

Comparisons play by Arazzo’s rules

So far I’ve shown test, which only answers “is this a valid condition?”. The point, though, is to evaluate one and get an actual true or false. The library never fetches live data itself, so you hand it a small resolve function that turns a runtime expression into its current value:

import { evaluate } from '@swaggerexpert/arazzo-criterion';

const context = {
  $statusCode: 200,
  '$response.body': { status: 'Available', data: [{ id: 42 }] },
};
const resolve = (expression) => context[expression];

evaluate('$statusCode == 200', { resolve });                    // true
evaluate("$response.body.status == 'available'", { resolve });  // true, even though the data says "Available"
evaluate('$response.body.data[0].id > 10', { resolve });        // true

That second line hides the last decision worth mentioning, and it’s the loose end from the eval idea I promised to come back to. When you compare things, the spec has its own gentle rules: text comparisons ignore capitalization ('Available' equals 'available'), the number 200 and the text '200' are treated as equal, and a check passes on anything truthy and fails on false, nothing, or a missing value. Those aren’t any programming language’s native rules, which is the second reason eval would have quietly given wrong answers, even in a world with only one language. I implemented them faithfully instead.

The tempting mistake is to import a rule from a similar feature next door. People often assume an empty list should count as “false”, but that instinct comes from other condition types (the ones built on JSONPath or XPath, where an empty match means failure). simple has no such rule. The library follows the plain reading of the spec: a condition passes on any truthy value and fails only on false, null, or a missing value. An empty list is none of those, so this evaluator treats it as truthy and passes it.

And that word “truthy” hides a real gap. The spec never defines whether an empty array is truthy, so the answer falls to the host language, and languages disagree: [] is truthy in JavaScript but falsy in Python (bool([]) is False). This library runs on JavaScript, so [] passes; a Python runner could fail the very same condition. It’s the ambiguous = all over again, just quieter: where the spec stays silent, two conformant runners can still diverge. I’ve since opened an Arazzo spec issue to pin these truthiness rules down.

Why this matters beyond Arazzo

Strip away the specifics and you’re left with something you’ll recognize once you’ve seen it:

A web address sitting inside a sentence: where does the link end and the prose resume?
A file name that’s allowed to contain spaces, listed next to other file names.
Any place where you paste one kind of expression into the middle of another.

The general truth is this: whenever you embed one little language inside another and they share the same symbols, someone has to decide where one ends and the other begins, and if those symbols overlap with no way to escape it, some inputs are simply undecidable. That’s not a bug you failed to fix. It’s a fact about the input. Good engineering isn’t pretending the fact away; it’s choosing where to make the tradeoff, making the common case feel effortless, and naming the rare case that can’t work.

Arazzo’s simple condition is just a crisp, real example: a very permissive runtime-expression language, a comparison language on top, a shared set of symbols, and a spec that offers no escape. So I made the everyday conditions read exactly as intended, and I wrote down the one they can’t.

For the curious: the grammar

Feel free to skip this part. But I promised a definition that lives above any single language, so here it is. The Arazzo spec describes simple only in prose and never gives it a formal grammar, so this is an original formalization: an ABNF grammar, the same portable notation the internet’s specs are written in, that any implementation in any language can follow to the same result.

Show the full ABNF grammar

‘Off Spec’ № 01: On OpenAPI, agents, and what specs are actually for

2026-07-03T08:00:00+00:00

'Off Spec', brought to you by APIwiz, is a conversation series about how APIs actually get built, maintained, governed, and consumed at scale — what the specs don't capture. For № 01, Shane O'Connor put fifteen questions to me. This is a re-post of the original interview.

I spent five years as a core maintainer of Swagger tools. Before that I was principal engineer at Apiary, home of API Blueprint. I’m a maintainer of the AsyncAPI specification. And today I co-found SpecLynx and work as an AI engineer at Jentic, where the two ends of my career — specifications and agents — are finally colliding.

1. The series is called Off Spec. So let's start there: what's something about how APIs actually get built and consumed that no specification will ever capture?

No specification fully captures the original author’s intent.

A spec can describe paths, schemas, status codes, and examples, but it rarely explains why the API was designed that way: the trade-offs, constraints, assumptions, and consumer needs behind it.

That intent matters. Without it, people can follow the contract and still miss the design.

The real API is not just the spec. It is the spec plus the documentation, SDKs, examples, error messages, and production behavior people come to rely on.

2. You were principal engineer at Apiary, home of API Blueprint, a spec that genuinely competed with Swagger. What actually killed Blueprint, and what did OpenAPI get right that it didn't?

Blueprint was not killed by one thing. It was a mix of tooling economics, industry convergence, and probably corporate gravity after Apiary.

API Blueprint had a beautiful authoring experience. It was readable, elegant, and very human-friendly. But OpenAPI was more explicit and easier for a broad tooling ecosystem to build around: generators, validators, gateways, documentation tools, and enterprise platforms.

Once the industry started converging on OpenAPI, even Apiary had to support it. At that point Blueprint was no longer just competing on specification quality; it was competing against a network effect.

So I would not say Blueprint lost because it was bad. It lost because OpenAPI became the safer ecosystem bet, and after the Apiary/Oracle chapter there was not enough independent momentum left to reverse that.

3. Five years maintaining Swagger tools that millions of developers depend on. What's the most consequential decision you made that almost nobody noticed?

Probably Swagger ApiDOM.

We built Swagger ApiDOM as a semantic parser for API specifications, a unified way to parse, process, transform, and reason about specs across the Swagger ecosystem. The important part is that we integrated it in a completely backward-compatible way.

For most users, the only visible effect was that OpenAPI 3.1 support arrived. But underneath, it changed the foundation: Swagger tooling no longer had to treat every spec version and format as a separate world. It also opened the door to supporting AsyncAPI, Arazzo, and Overlays through the same underlying model.

Good infrastructure becomes invisible when it works well.

4. OpenAPI 3.1 finally aligned with JSON Schema 2020-12 after years of divergence, and you built the renderer for it. Was that alignment worth the pain, or did it fix a problem most users never actually had?

Yes, it was worth the pain.

Most real-world specifications were still OpenAPI 3.0.x, but there was clear interest in OpenAPI 3.1 support. I do not think most users cared about JSON Schema 2020-12 alignment as a standards debate. They cared that newer specifications worked, validation was more consistent, and tooling could support OpenAPI 3.1 properly.

For tooling authors, the old divergence between OpenAPI Schema Objects and JSON Schema created years of special cases across renderers, validators, parsers, and generators. OpenAPI 3.1 did not remove all complexity, but it did move the ecosystem closer to a shared foundation.

So yes, it fixed a real problem, just not always at the layer where most users described it.

5. What's a piece of received wisdom in the API world that you think is just wrong? "Design-first always," "REST is dead," "spec-driven everything." Get as spicy as you want.

The received wisdom I think is wrong is that specs are mainly for documentation.

That view is too small. In today’s world, an API specification should be treated more like infrastructure-as-code: a living description of the system that is versioned, tested, checked against reality, and kept up to date.

A spec should participate in the lifecycle of the API: design, review, validation, testing, governance, SDK generation, agent consumption, and runtime checks.

The future is not "write a spec to produce nice documentation." The future is "use the spec to prove the API, the tooling, and the consumers still agree."

6. Most teams treat the OpenAPI spec as a documentation output, something you generate at the end. What are they fundamentally missing about what a spec is for?

They are missing the feedback loop.

If the spec is generated only at the end, it can tell consumers what the API does, but it cannot help the team catch problems while the API is still being shaped.

A good spec should be part of the pull request, the CI pipeline, and the release process. It should catch breaking changes, validate examples, drive contract tests, generate SDKs, and give consumers something stable to review and build against as the API evolves.

Documentation is one output of a spec. The bigger job is keeping design, implementation, tooling, and consumers aligned.

7. If you could delete one feature from the OpenAPI spec, and one thing from your own past code, what goes, and who'd be mad about it?

From OpenAPI 3.1, I would probably delete the jsonSchemaDialect fixed field.

Most people think of OpenAPI 3.1 as “the version that aligned with JSON Schema 2020-12.” But technically, through jsonSchemaDialect, an OpenAPI document can declare a different JSON Schema dialect, including previous drafts or even a custom dialect.

In theory, that flexibility makes sense. In practice, I have not seen many OpenAPI documents that actually use it. The complexity it brings to tooling implementation feels disproportionate. Parsers, validators, renderers, and generators suddenly have to care not just about OpenAPI 3.1, but about which JSON Schema dialect is active and what behavior that implies.

From my own past code, I would happily delete the code that tried to provide full support for jsonSchemaDialect. It was the kind of implementation work where the theoretical correctness was real, but the practical value for most users was hard to justify.

Who would be mad? Probably the very small number of people who really need custom dialect support. But I suspect most users would happily trade that flexibility for simpler, more predictable tooling.

8. You also maintain AsyncAPI. Why does event-driven still feel like a second-class citizen in the API tooling world, and is that finally changing?

Event-driven still feels second-class because it is harder to see.

With HTTP APIs, the interaction model is familiar: request, response, status code, payload. It maps naturally to documentation, testing, gateways, SDKs, and mental models. Event-driven systems are different. The important behavior is spread across producers, consumers, brokers, topics, retries, ordering, delivery guarantees, and time.

That makes the tooling problem harder. You are not just describing an endpoint; you are describing a system of conversations that may happen asynchronously and indirectly.

I do think it is changing. AsyncAPI gives the industry a shared language for describing those systems, and as workflows, agents, and event-driven architectures become more common, that language becomes more important.

9. You're now an AI engineer at Jentic. LLMs and agents have quietly become the biggest consumers of API specs. Was OpenAPI ever written for machines, or for humans pretending to be machines?

OpenAPI was written for machines, but not for agents.

It was designed so tooling could parse an API contract and generate documentation, clients, servers, validators, tests, and governance checks. In that sense, it was absolutely machine-readable by design.

But agents are a different kind of consumer. They do not just need a valid contract or a generated client. They need to decide whether an operation is relevant, what the side effects might be, how to combine calls, how to recover from errors, and whether the API is safe to use in a given context.

That is where work like Jentic API Scorecard comes in: measuring whether an OpenAPI document is actually aligned with how agents use specs.

The question is no longer just "is this valid OpenAPI?" It is "does this spec make the API understandable, usable, and safe for an autonomous consumer?"

10. Honest take: are we about to watch a generation of lovingly hand-crafted API specs get bypassed, because an agent would rather read the code or just hit the endpoint and see what happens?

I think agents will bypass bad specs, not good ones.

If a specification is outdated, incomplete, or just a thin documentation artifact, then yes, an agent may get more value from reading code, examples, traffic, or trying the endpoint directly. In that world, the lovingly hand-crafted spec risks becoming decorative.

But a good OpenAPI document is still one of the best ways to give an agent a structured, intentional view of an API. Code tells you what is implemented. Runtime behavior tells you what happens. A spec should tell you what is intended, supported, stable, and safe to rely on.

So I do not think agents make specs irrelevant. I think they raise the bar.

Specs that are stale will be ignored faster. Specs that are accurate, rich, and aligned with reality become even more important.

11. MCP and tool schemas are suddenly everywhere. Is that the API world reinventing OpenAPI badly, or solving something OpenAPI genuinely couldn't?

I do not think MCP and tool schemas are just OpenAPI reinvented badly. They are solving a different problem.

OpenAPI describes an API surface in detail: operations, parameters, schemas, responses, auth, errors, and documentation. MCP is more focused on exposing capabilities to agents as callable tools.

That narrower focus is useful. An agent does not always need the full API description; it needs to know what action it can take, what inputs are required, what result to expect, and whether the action is safe in context.

But I do think there is a risk of relearning old API lessons. Versioning, authentication, error handling, pagination, compatibility, and governance all come back eventually.

So my take is: MCP is not a replacement for OpenAPI. It is an agent-facing runtime layer. The interesting future is connecting them: using OpenAPI for rich API context, and MCP/tool schemas for the operational interface agents actually call.

12. You founded SpecLynx and re-built ApiDOM. Serious bets. What's something you poured yourself into that the world just didn't want, and were they right not to want it?

SpecLynx is probably the clearest example.

We built a VS Code extension and editor around API specifications, and I still believe the problem was real. At the time, a better human editing experience for specs made sense.

But the world has evolved. The more important surfaces now are MCP, LSP, CLIs, CI pipelines, and agent workflows, places where specifications are not just written and viewed, but actively used, checked, transformed, and connected to other systems.

Were people right not to want it? In that form, probably yes. Not because the idea was wrong, but because the interface the world needs has changed. The future is less about people lovingly editing specs by hand, and more about tools and agents continuously working with them.

13. What's the least glamorous part of API tooling, the unsexy plumbing, that actually matters more than all the design-philosophy debates people love to have on Twitter?

Proper parsing and reference resolution.

They are not glamorous, but they are where API tooling either becomes trustworthy or quietly falls apart. People often think parsing means “read some JSON or YAML,” and reference resolution means “follow a $ref.” In reality, both are much harder.

Good tooling has to understand the specification version, preserve semantic meaning, track source locations, handle malformed input, and resolve references across files, URLs, anchors, bundles, dialects, and all the edge cases real users create.

Everything else depends on that foundation: validation, rendering, editing, linting, code generation, governance, testing, and now agent consumption.

If parsing or resolution is wrong, every layer above it becomes unreliable.

14. Open source at Swagger's scale: what's the honest story about who actually pays for the tooling the entire API industry quietly runs on?

The honest story is that open source at this scale is often paid for indirectly, but it is paid for.

In the Swagger ecosystem, SmartBear played a central role. It owned the Swagger brand, built commercial products around it, and funded engineering work on open source tooling that mattered both to those products and to the wider API community.

That model has real benefits. The ecosystem gets dedicated engineering time, and the company has a strategic reason to invest. But it also creates tension: the tooling is used far beyond one company, while the responsibility for maintaining it is not shared equally.

So the honest answer is that some of the work is funded by companies with a reason to invest, and some of it is still carried by maintainers because the ecosystem depends on it.

15. Lightning round. One word or one line each.

Design-first or code-first?: Contract-first.
OpenAPI or GraphQL schema?: OpenAPI for action-oriented APIs; GraphQL schema when clients need to query related data flexibly.
Most overrated API tool?: Static API portals.
Most underrated?: A good parser.
Will REST still dominate in 2030?: Yes, but agents will care more about clear operations, good descriptions, predictable errors, and safety than whether the API is perfectly RESTful.

This interview originally appeared on LinkedIn as part of APIwiz's 'Off Spec' series. Read the original article.

Advanced memoization for JavaScript functions with lodash.memoize

2022-02-05T18:00:00+00:00

"In computing, memoization or memoisation is an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again."

Wikipedia on Memoization

Lodash is a modern JavaScript utility library delivering modularity, performance & extras. It’s a ubiquitous library, and your project probably uses it, even if you’re not aware of it. Lodash contains an implementation of memoization in the form of memoize function.

By default, memoize function only uses the first argument provided to the memoized function as the cache key. Here is the simplest possible example of memoization:

1
2
3
4
5
6
7
8
const { memoize } = require('lodash');

const func = (arg1) => `My name is ${arg1}`
const funcM = memoize(func);

console.dir(funcM('John')); // => 'My name is John'
console.dir(funcM('John')); // => 'My name is John' (cache hit)
console.dir(funcM.cache.size); // => 1

It’s important to realize that the cache key of the memoized function is determined using SameValueZero comparison:

1
2
3
4
5
6
7
8
const { memoize } = require('lodash');

const func = (arg1) => JSON.stringify(arg1)
const funcM = memoize(func);

console.dir(funcM({a: 1})); // => '{"a":1}'
console.dir(funcM({a: 1})); // => '{"a":1}'
console.dir(funcM.cache.size); // => 2

Now, what if we introduce more arguments to our func?

1
2
3
4
5
6
7
8
const { memoize } = require('lodash');

const func = (arg1, arg2) => `${arg1} ${arg2}`;
const funcM = memoize(func);

console.dir(funcM('John', 'Doe')); // => "John Doe"
console.dir(funcM('John', 'FooBar')); // => "John Doe" (cache hit)
console.dir(funcM.cache.size); // => 2

We can see that there is an obvious problem. As stated above, lodash only uses the first argument as the cache key. To fix this, we have to use resolver function:

1
2
3
4
5
6
7
8
9
const { memoize } = require('lodash');

const func = (arg1, arg2) => `${arg1} ${arg2}`;
const resolver = (arg1, arg2) => JSON.stringify([arg1, arg2]);
const funcM = memoize(func, resolver);

console.dir(funcM('John', 'Doe')); // => "John Doe"
console.dir(funcM('John', 'FooBar')); // => "John FooBar"
console.dir(funcM.cache.size); // => 2

This resolver solution obviously wouldn’t scale. What if our func needs to consume complex objects as arguments? That’s fine unless the object arguments are huge or contain a cyclic reference.

When arguments are huge objects, we’ll spend too much CPU time serializing those objects into JSON strings to determine the cache key. And instead of the SameValueZero comparison for objects, we’ll be comparing huge strings. When object arguments contain a cyclic reference, we’ll not be able to serialize them into JSON string and determine the cache key.

Lodash has one remaining trick up its sleeve to overcome these two problems - being able to influence how cache keys are maintained. To achieve our goal, we’ll have to create a specialization of lodash.memoize; the function called memoizeN:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
const { memoize } from "lodash";

const sameValueZero = (x, y) => x === y || (Number.isNaN(x) && Number.isNaN(y));
const shallowArrayEquals = (a) => (b) => {
  return Array.isArray(a) && Array.isArray(b)
    && a.length === b.length
    && a.every((val, index) => sameValueZero(val, b[index]));
};
const list = (...args) => args;

class Cache extends Map {
  delete(key) {
    const keys = Array.from(this.keys());
    const foundKey = keys.find(shallowArrayEquals(key));
    return super.delete(foundKey);
  }
  get(key) {
    const keys = Array.from(this.keys());
    const foundKey = keys.find(shallowArrayEquals(key));
    return super.get(foundKey);
  }
  has(key) {
    const keys = Array.from(this.keys());
    return keys.findIndex(shallowArrayEquals(key)) !== -1;
  }
}

const memoizeN = (fn, { resolver = list } = {}) => {
  const { Cache: OriginalCache } = memoize;
  memoize.Cache = Cache;
  const memoized = memoize(fn, resolver);
  memoize.Cache = OriginalCache;
  return memoized;
};

The real trick here is that the memoized function parameters are transformed into a list, and this list forms a cache key. We override the lodash.memoize.Cache class on an ad-hoc basis with our implementation capable of doing cache key lookups for our list cache keys.

Observe the following example:

1
2
3
4
5
6
7
8
9
10
11
const complexObj = {a: 1};
const simpleObj = {b: 2};

const func = (arg1, arg2) => arg2;
const resolver = (arg1, arg2) => [arg1, JSON.stringify(arg2)];
const funcM = memoizeN(func, { resolver });

console.dir(funcM(complexObj, simpleObj)); // => {b: 2}
console.dir(funcM(complexObj, {b: 2})); // => {b: 2} (cache hit)
console.dir(funcM({a: 1}, {b: 2})); // => {b: 2}
console.dir(funcM.cache.size); // => 2

We intend to memoize func using both arguments to form the cache key. For the first argument, use the SameValueZero comparison on the original value. For the second one, use SameValueZero comparison on JSON stringified value.

We can go even further with the specialization to make memoizeN consume a list of comparator functions applied to appropriate arguments instead of abusing a resolver to serialize some arguments.

The implementation will look as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
const memoize = require('lodash/memoize');
const isEqual = require('lodash/isEqual');

const sameValueZero = (x, y) => x === y || (Number.isNaN(x) && Number.isNaN(y));
const makeShallowArrayEquals = (comparators) => (a) => (b) => {
  return Array.isArray(a) && Array.isArray(b)
    && a.length === b.length
    && a.every((val, index) => {
      const comparator = typeof comparators === 'function' ? comparators : comparators[index] || sameValueZero;
      return comparator(val, b[index]);
    });
};
const list = (...args) => args;

class Cache extends Map {
  shallowArrayEquals = makeShallowArrayEquals(sameValueZero);

  delete(key) {
    const keys = Array.from(this.keys());
    const foundKey = keys.find(this.shallowArrayEquals(key));
    return super.delete(foundKey);
  }
  get(key) {
    const keys = Array.from(this.keys());
    const foundKey = keys.find(this.shallowArrayEquals(key));
    return super.get(foundKey);
  }
  has(key) {
    const keys = Array.from(this.keys());
    return keys.findIndex(this.shallowArrayEquals(key)) !== -1;
  }
}

const memoizeN = (fn, { resolver = list, comparators = sameValueZero }) => {
  const { Cache: OriginalCache } = memoize;
  memoize.Cache = Cache;
  const memoized = memoize(fn, resolver || list);
  memoized.cache.shallowArrayEquals = makeShallowArrayEquals(comparators);
  memoize.Cache = OriginalCache;
  return memoized;
};

module.exports = memoizeN;

This version of memoizeN applies a specific comparator algorithm for each argument:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
const obj1 = {a: 1};
const obj2 = {b: 2};
const obj3 = {c: 3};

const func = (arg1, arg2, arg3) => arg3;
const strictEquality = (x, y) => x === y;
const sameValueZero = (x, y) => x === y || (Number.isNaN(x) && Number.isNaN(y));
const deepEquality = lodash.isEqual;
const comparators = [strictEquality, sameValueZero, deepEquality];
const funcM = memoizeN(func, { comparators });

console.dir(funcM(obj1, obj2, obj3)); // => {c: 3}
console.dir(funcM(obj1, obj2, {c: 3})); // => {c: 3} (cache hit)
console.dir(funcM(obj1, {b: 2}, obj3); // => {c: 3}
console.dir(funcM.cache.size); // => 2

If we define comparators option as a function, the memoized function will use it to compare every argument. When comparators are not provided, SameValueZero comparison is used for all arguments.

1
2
3
4
5
6
7
8
9
10
11
const obj1 = {a: 1};
const obj2 = {b: 2};
const obj3 = {c: 3};

const func = (arg1, arg2, arg3) => arg3;
const sameValueZero = (x, y) => x === y || (Number.isNaN(x) && Number.isNaN(y));
const funcM = memoizeN(func, { comparators: sameValueZero }); // equivalen with memoizeN(func)

console.dir(funcM(obj1, obj2, obj3)); // => {c: 3}
console.dir(funcM(obj1, obj2, obj3)); // => {c: 3} // cache hit
console.dir(funcM.cache.size); // => 1

I’ve extracted memoizeN into GitHub Gist installable via npm. If you prefer to use memoizeN as npm package, here is how to achieve that:

 $ npm install gist:2e6c77d38cf00faacceaf37e46b76a32 --save

This command will create the following entry inside your dependencies:

"dependencies": {
  "@char0n/memoizeN": "gist:2e6c77d38cf00faacceaf37e46b76a32"
}

Then use it via CommonJS or ESM:

const memoizeN = require('@char0n/memoizeN');
// or
import memoizeN from '@char0n/memoizeN';

memoizeN

Advanced memoization for JavaScript functions with lodash.memoize.

If you need even more advanced functionality like LRU cache, TTL, reference counting, and others, I suggest you look at memoizee. memizee is probably the most feature-rich implementation of memoization for JavaScript.

How to handle errors in SwaggerUI

2022-01-24T08:00:00+00:00

Error handling is an essential aspect of today's modern Single Page Applications. Error handling refers to the anticipation, detection, and resolution of different kinds of errors. As of version v4.3.0, SwaggerUIs error handling capabilities have considerably improved and allowed SwaggerUI integrators to easily integrate their custom errors handlers.

Original SwaggerUI error handling consisted of wrapping every render method of every class component with an imperative try/catch statement and displaying the Fallback component if the error was thrown. Along with displaying the Fallback component, console.error was used to display the caught error in the browser console. This solution required wrapping every function component into class component to make sure that the component has the render method.

React 16 introduces a new concept of an error boundary. It’s no longer necessary to create a custom solution for catching errors in React apps, as we have a standardized way of doing it.

In order to fully utilize new error boundaries, I had to rewrite SwaggerUIs view plugin, which all other plugins build on. An additional new plugin called safe-render was introduced to allow configurable application of error boundaries to SwaggerUI components.

View plugin

View plugins public API didn’t change, I’ve just added one additional util function called getDisplayName which is used for accessing React component names.

1
2
3
4
5
6
7
8
9
10
{
  rootInjects: {
    getComponent: memGetComponent,
    makeMappedContainer: memMakeMappedContainer,
    render: render(getSystem, getStore, getComponent, getComponents),
  },
  fn: {
    getDisplayName,
  },
}

view plugin is no longer responsible for error handling logic. Rewriting the plugin had several positive side effects like making the SwaggerUI rendering faster and simplifying the React component tree in React developer tools:

Before refactor

After refactor

Safe-render plugin

This plugin is solely responsible for error handling logic in SwaggerUI. Accepts a list of component names that should be protected by error boundaries.

Its public API looks like this:

1
2
3
4
5
6
7
8
9
10
{
  fn: {
    componentDidCatch,
    withErrorBoundary: withErrorBoundary(getSystem),
  },
  components: {
    ErrorBoundary,
    Fallback,
  },
}

safe-render plugin is automatically utilized by base and standalone SwaggerUI presets and should always be used as the last plugin, after all the components are already known to the SwaggerUI. The plugin defines a default list of components that should be protected by error boundaries:

[
  "App",
  "BaseLayout",
  "VersionPragmaFilter",
  "InfoContainer",
  "ServersContainer",
  "SchemesContainer",
  "AuthorizeBtnContainer",
  "FilterContainer",
  "Operations",
  "OperationContainer",
  "parameters",
  "responses",
  "OperationServers",
  "Models",
  "ModelWrapper",
  "Topbar",
  "StandaloneLayout",
  "onlineValidatorBadge"
]

As demonstrated below, additional components can be protected by utilizing the safe-render plugin with configuration options. This gets really handy if you are a SwaggerUI integrator and you maintain a number of plugins with additional custom components.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
const swaggerUI = SwaggerUI({
  url: "https://petstore.swagger.io/v2/swagger.json",
  dom_id: '#swagger-ui',
  plugins: [
    () => ({
      components: {
        MyCustomComponent1: () => 'my custom component',
      },
    }),
    SwaggerUI.plugins.SafeRender({
      fullOverride: true, // only the component list defined here will apply (not the default list)
      componentList: [
        "MyCustomComponent1",
      ],
    }),
  ],
});

componentDidCatch

This static function is invoked after a component has thrown an error.
It receives two parameters:

error - The error that was thrown.
info - An object with a componentStack key containing information about which component threw the error.

It has precisely the same signature as error boundaries componentDidCatch lifecycle method, except it’s a static function and not a class method.

Default implement of componentDidCatch uses console.error to display the received error:

export const componentDidCatch = console.error;

To utilize your own error handling logic (e.g. bugsnag), create new SwaggerUI plugin that overrides componentDidCatch:

1
2
3
4
5
6
7
8
9
10
11
12
const BugsnagErrorHandlerPlugin = () => {
  // init bugsnag

  return {
    fn: {
      componentDidCatch = (error, info) => {
        Bugsnag.notify(error);
        Bugsnag.notify(info);
      },
    },
  };
};

withErrorBoundary

This function is HOC (Higher Order Component). It wraps a particular component into the ErrorBoundary component. It can be overridden via a plugin system to control how components are wrapped by the ErrorBoundary component. In 99.9% of situations, you won’t need to override this function, but if you do, please read the source code of this function first.

Fallback

The component is displayed when the error boundary catches an error. It can be overridden via a plugin system. Its default implementation is trivial:

1
2
3
4
5
6
7
8
9
10
11
12
import React from "react"
import PropTypes from "prop-types"

const Fallback = ({ name }) => (
  <div className="fallback">
    😱 <i>Could not render { name === "t" ? "this component" : name }, see the console.</i>
  </div>
)
Fallback.propTypes = {
  name: PropTypes.string.isRequired,
}
export default Fallback

Feel free to override it to match your look & feel:

1
2
3
4
5
6
7
8
9
10
11
12
13
const CustomFallbackPlugin = () => ({
  components: {
    Fallback: ({ name } ) => `This is my custom fallback. ${name} failed to render`,
  },
});

const swaggerUI = SwaggerUI({
  url: "https://petstore.swagger.io/v2/swagger.json",
  dom_id: '#swagger-ui',
  plugins: [
    CustomFallbackPlugin,
  ]  
});

ErrorBoundary

This is the component that implements React error boundaries. Uses componentDidCatch and Fallback under the hood. In 99.9% of situations, you won’t need to override this component, but if you do, please read the source code of this component first.

Change in behavior

In prior releases of SwaggerUI, almost all components have been protected, and when thrown error, Fallback component was displayed. This changes with SwaggerUI v4.3.0. Only components defined by the safe-render plugin are now protected and display fallback. If a small component somewhere within SwaggerUI React component tree fails to render and throws an error, the error bubbles up to the closest error boundary, and that error boundary displays the Fallback component and invokes componentDidCatch.

If you’re interested in more technical details, here is the PR that introduced the new error handling into SwaggerUI.

How to migrate from querystring to URLSearchParams in Node.js

2021-10-14T07:00:00+00:00

Node.js marked the querystring as legacy API in version 14.x, and recommends using URLSearchParams. But doesn't give us any clue how to actually migrate. This article fills this void and provides a migration guide.

querystring

querystring API is quite simple and straightforward:

querystring.parse(): Parses a URL query string into a collection of key and value pairs. querystring.decode() is an alias.
querystring.stringify(): Produces a URL query string from a given obj by iterating through the object's "own properties". querystring.encode() is an alias.
querystring.escape(): Performs URL percent-encoding on the given string in a manner that is optimized for the specific requirements of URL query strings. Is used by querystring.stringify().
querystring.unescape(): Performs decoding of URL percent-encoded characters on the given string. Is used by querystring.parse().

Parse a URL query string

const { parse } = require('querystring');

parse('foo=bar&abc=xyz&abc=123');

// returns
// { 
//   foo: 'bar',
//   abc: ['xyz', '123']
// }

Produce a URL query string

const { stringify } = require('querystring');

stringify({ foo: 'bar', baz: ['qux', 'quux'], corge: '' });

// returns 'foo=bar&baz=qux&baz=quux&corge='

Perform URL percent-encoding on the given string

const { escape } = require('querystring');

escape('str1 str2');

// returns 'str1%20str2'

Perform decoding of URL percent-encoded characters on the given string

const { unescape } = require('querystring');

escape('str1%20str2');

// returns 'str1 str2'

URLSearchParams

What’s the actual difference between querystring vs. URLSearchParams?

"The WHATWG URLSearchParams interface and the querystring module have similar purpose, but the purpose of the querystring module is more general, as it allows the customization of delimiter characters."

Node.js 14.x documentation

Working with URIs/URLs in JavaScript was always confusing. WHATWG URL specification finally standardizes the term URL and provides a standardized way how we work with URLs. URLSearchParams is part of this specification. Now let’s migrate our previous querystring examples to URLSearchParams API.

Parse a URL query string

const params = new URLSearchParams('foo=bar&abc=xyz&abc=123');
// URLSearchParams { 'foo' => 'bar', 'abc' => 'xyz', 'abc' => '123' }

params.get('foo'); // 'bar'
params.getAll('abc') // ['xyz', '123']

Produce a URL query string

new URLSearchParams({ foo: 'bar', baz: ['qux', 'quux'], corge: '' }).toString();
// returns 'foo=bar&baz=qux%2Cquux&corge='

As we can see, what we got is a different result compared to querystring.

API	Result
querystring	foo=bar&baz=qux&baz=quux&corge=
URLSearchParams	foo=bar&baz=qux%2Cquux&corge=

"Unlike querystring module, duplicate keys in the form of array values are not allowed. Arrays are stringified using array.toString(), which simply joins all array elements with commas."

Node.js 14.x documentation

To get the result compatible with querystring, we have to initialize URLSearchParams with a list of tuples instead of object:

new URLSearchParams([
  ['foo', 'bar'],
  ['baz', 'qux'],
  ['baz', 'quux'],
  ['corge', ''],
]).toString();
// returns 'foo=bar&baz=qux&baz=quux&corge='

Perform URL percent-encoding on the given string

There is no low-level API for encoding simple strings in URLSearchParams. We have to be a little creative to achieve URL encoding. Below are 2 different yet functionally equivalent ways how to achieve the same result.

new URLSearchParams([['', 'str1 str2']]).toString().slice(1);
new URLSearchParams({'': 'str1 str2'}).toString().slice(1);
// returns 'str1+str2'

Again, we can see what we got is a different result compared to querystring. WHATWG URL specification is now precise about how various segments of URL gets encoded. Spaces in query parameters are now plus-sign (+) encoded, compared to percent-encoded spaces in the path segment of the URL. This currently accepted behavior is backward incompatible with querystring, and there is nothing you can do about it except to take it.

API	Result
querystring	str1%20str2
URLSearchParams	str1+str2

Perform decoding of URL percent-encoded characters on the given string

There is no low-level API for decoding a simple string in URLSearchParams. Again we have to be creative to achieve URL decoding on the given string.

const urlPlusEncodedString = 'str1+str2';

new URLSearchParams(`=${urlPlusEncodedString}`).get('');
// returns 'str1 str2'

URLSearchParams will handle percent-encoded spaces as well:

const urlPercentEncodedString = 'str1%20str2';

new URLSearchParams(`=${urlPercentEncodedString}`).get('');
// returns 'str1 str2'

Other differences

Handling of question mark character

URLSearchParams removes question mark character from the start of the URL query string. querystring keeps it and makes it part of the key name.

const { parse } = require('querystring');

parse('?foo=bar'); // returns { '?foo': 'bar' }
new URLSearchParams('?foo=bar'); // returns URLSearchParams { 'foo' => 'bar' }

Converting URLSearchParams object to Plain JavaScript Object

URLSearchParams object is a class instance. It is not a classical Plain JavaScript Object (POJO). To transform URLSearchParam instance into a POJO, use the following recipe.

const params = new URLSearchParams('foo=bar');
const paramsPOJO = Object.fromEntries(params.entries()); // { 'foo': 'bar' }

Have you found other differences? I’d be happy to add them to this list.

Closing words

querystring is a legacy Node.js API that shouldn’t be used anymore. New code should only use URLSearchParams, which is part of WHATWG URL specification. Old code using querystring can easily be migrated to URLSearchParams using this guide. The only backward incompatibility between the two APIs is that URLSearchParams use plus-encoded spaces instead of percent-encoded spaces in querystring.

How to apply the Apache 2.0 License to your Open Source software project

2021-10-02T12:15:00+00:00

Apache 2.0 License is one of the best OSS licenses to use in your Open Source software project, but do you actually know how to properly apply it? We'll go through all the common mistakes and show how to use the license correctly.

I’ve always been a 3-Clause BSD License (AKA New BSD License or Modified BSD License) guy. As I’ve matured, my understanding of Open Source licensing has grown as well, and so have my requirements. When I start a new Open Source project now, I just use Apache 2.0 license. I want to point out that I am not a lawyer, and I am certainly not your lawyer. My legal opinions are based on research and lengthy conversations with legal departments of companies that I worked for.

Why Apache 2.0 license?

There are various reasons why I prefer Apache 2.0 license over the others (specifically the 3-Clause BSD License). This WhiteSource article covers the differences nicely, but for completeness, let me mentioned them here explicitly as well:

Explicit grant of patent rights: Apache License 2.0 explicitly lays down the grant of patent rights while using, modifying, or distributing Apache licensed software; it also lists the circumstances when such grant gets withdrawn.
Precise definitions of the used concepts: Apache License 2.0 explicitly defines all the terms and concepts that it uses. This leaves little scope for ambiguity.
Reusable without rewording: Apache License 2.0 can be easily used by other projects without any rewording in the license document itself. It also means that it should be sufficient to provide the canonical URL https://www.apache.org/licenses/LICENSE-2.0.txt, where the copy of the license can be obtained, instead of the license text.

These differences are most important to me, but other differences like “prominent notices stating that You changed the files” might be more critical for you. Here is the full text of the Apache 2.0 license:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
                                Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

1. Definitions.

   "License" shall mean the terms and conditions for use, reproduction,
   and distribution as defined by Sections 1 through 9 of this document.

   "Licensor" shall mean the copyright owner or entity authorized by
   the copyright owner that is granting the License.

   "Legal Entity" shall mean the union of the acting entity and all
   other entities that control, are controlled by, or are under common
   control with that entity. For the purposes of this definition,
   "control" means (i) the power, direct or indirect, to cause the
   direction or management of such entity, whether by contract or
   otherwise, or (ii) ownership of fifty percent (50%) or more of the
   outstanding shares, or (iii) beneficial ownership of such entity.

   "You" (or "Your") shall mean an individual or Legal Entity
   exercising permissions granted by this License.

   "Source" form shall mean the preferred form for making modifications,
   including but not limited to software source code, documentation
   source, and configuration files.

   "Object" form shall mean any form resulting from mechanical
   transformation or translation of a Source form, including but
   not limited to compiled object code, generated documentation,
   and conversions to other media types.

   "Work" shall mean the work of authorship, whether in Source or
   Object form, made available under the License, as indicated by a
   copyright notice that is included in or attached to the work
   (an example is provided in the Appendix below).

   "Derivative Works" shall mean any work, whether in Source or Object
   form, that is based on (or derived from) the Work and for which the
   editorial revisions, annotations, elaborations, or other modifications
   represent, as a whole, an original work of authorship. For the purposes
   of this License, Derivative Works shall not include works that remain
   separable from, or merely link (or bind by name) to the interfaces of,
   the Work and Derivative Works thereof.

   "Contribution" shall mean any work of authorship, including
   the original version of the Work and any modifications or additions
   to that Work or Derivative Works thereof, that is intentionally
   submitted to Licensor for inclusion in the Work by the copyright owner
   or by an individual or Legal Entity authorized to submit on behalf of
   the copyright owner. For the purposes of this definition, "submitted"
   means any form of electronic, verbal, or written communication sent
   to the Licensor or its representatives, including but not limited to
   communication on electronic mailing lists, source code control systems,
   and issue tracking systems that are managed by, or on behalf of, the
   Licensor for the purpose of discussing and improving the Work, but
   excluding communication that is conspicuously marked or otherwise
   designated in writing by the copyright owner as "Not a Contribution."

   "Contributor" shall mean Licensor and any individual or Legal Entity
   on behalf of whom a Contribution has been received by Licensor and
   subsequently incorporated within the Work.

2. Grant of Copyright License. Subject to the terms and conditions of
   this License, each Contributor hereby grants to You a perpetual,
   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
   copyright license to reproduce, prepare Derivative Works of,
   publicly display, publicly perform, sublicense, and distribute the
   Work and such Derivative Works in Source or Object form.

3. Grant of Patent License. Subject to the terms and conditions of
   this License, each Contributor hereby grants to You a perpetual,
   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
   (except as stated in this section) patent license to make, have made,
   use, offer to sell, sell, import, and otherwise transfer the Work,
   where such license applies only to those patent claims licensable
   by such Contributor that are necessarily infringed by their
   Contribution(s) alone or by combination of their Contribution(s)
   with the Work to which such Contribution(s) was submitted. If You
   institute patent litigation against any entity (including a
   cross-claim or counterclaim in a lawsuit) alleging that the Work
   or a Contribution incorporated within the Work constitutes direct
   or contributory patent infringement, then any patent licenses
   granted to You under this License for that Work shall terminate
   as of the date such litigation is filed.

4. Redistribution. You may reproduce and distribute copies of the
   Work or Derivative Works thereof in any medium, with or without
   modifications, and in Source or Object form, provided that You
   meet the following conditions:

   (a) You must give any other recipients of the Work or
   Derivative Works a copy of this License; and

   (b) You must cause any modified files to carry prominent notices
   stating that You changed the files; and

   (c) You must retain, in the Source form of any Derivative Works
   that You distribute, all copyright, patent, trademark, and
   attribution notices from the Source form of the Work,
   excluding those notices that do not pertain to any part of
   the Derivative Works; and

   (d) If the Work includes a "NOTICE" text file as part of its
   distribution, then any Derivative Works that You distribute must
   include a readable copy of the attribution notices contained
   within such NOTICE file, excluding those notices that do not
   pertain to any part of the Derivative Works, in at least one
   of the following places: within a NOTICE text file distributed
   as part of the Derivative Works; within the Source form or
   documentation, if provided along with the Derivative Works; or,
   within a display generated by the Derivative Works, if and
   wherever such third-party notices normally appear. The contents
   of the NOTICE file are for informational purposes only and
   do not modify the License. You may add Your own attribution
   notices within Derivative Works that You distribute, alongside
   or as an addendum to the NOTICE text from the Work, provided
   that such additional attribution notices cannot be construed
   as modifying the License.

   You may add Your own copyright statement to Your modifications and
   may provide additional or different license terms and conditions
   for use, reproduction, or distribution of Your modifications, or
   for any such Derivative Works as a whole, provided Your use,
   reproduction, and distribution of the Work otherwise complies with
   the conditions stated in this License.

5. Submission of Contributions. Unless You explicitly state otherwise,
   any Contribution intentionally submitted for inclusion in the Work
   by You to the Licensor shall be under the terms and conditions of
   this License, without any additional terms or conditions.
   Notwithstanding the above, nothing herein shall supersede or modify
   the terms of any separate license agreement you may have executed
   with Licensor regarding such Contributions.

6. Trademarks. This License does not grant permission to use the trade
   names, trademarks, service marks, or product names of the Licensor,
   except as required for reasonable and customary use in describing the
   origin of the Work and reproducing the content of the NOTICE file.

7. Disclaimer of Warranty. Unless required by applicable law or
   agreed to in writing, Licensor provides the Work (and each
   Contributor provides its Contributions) on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
   implied, including, without limitation, any warranties or conditions
   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
   PARTICULAR PURPOSE. You are solely responsible for determining the
   appropriateness of using or redistributing the Work and assume any
   risks associated with Your exercise of permissions under this License.

8. Limitation of Liability. In no event and under no legal theory,
   whether in tort (including negligence), contract, or otherwise,
   unless required by applicable law (such as deliberate and grossly
   negligent acts) or agreed to in writing, shall any Contributor be
   liable to You for damages, including any direct, indirect, special,
   incidental, or consequential damages of any character arising as a
   result of this License or out of the use or inability to use the
   Work (including but not limited to damages for loss of goodwill,
   work stoppage, computer failure or malfunction, or any and all
   other commercial damages or losses), even if such Contributor
   has been advised of the possibility of such damages.

9. Accepting Warranty or Additional Liability. While redistributing
   the Work or Derivative Works thereof, You may choose to offer,
   and charge a fee for, acceptance of support, warranty, indemnity,
   or other liability obligations and/or rights consistent with this
   License. However, in accepting such obligations, You may act only
   on Your own behalf and on Your sole responsibility, not on behalf
   of any other Contributor, and only if You agree to indemnify,
   defend, and hold each Contributor harmless for any liability
   incurred by, or claims asserted against, such Contributor by reason
   of your accepting any such warranty or additional liability.

END OF TERMS AND CONDITIONS

APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "[]"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

Copyright [yyyy] [name of copyright owner]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

There are two variants of the Apache 2.0 license: justified version and version with text aligned to the left. The justified version is the one you get by fetching https://www.apache.org/licenses/LICENSE-2.0.txt - I call it the canonical form. The license has 201 lines of text and is composed of the header (lines 1-3), the terms and conditions (lines 5-176) and the appendix (lines 178-201).

License

To apply the license to your software project, create a LICENSE file in the source tree’s top level. Now copy the full Apache 2.0 License text from https://www.apache.org/licenses/LICENSE-2.0.txt into a LICENSE file. Notice that the LICENSE file doesn’t have any extension, but you can optionally name the file LICENSE.txt as well.

Do not modify the license text. If you used a 3-Clause BSD License in the past, you usually downloaded the license template, changed it, and saved it as a LICENSE file.

Copyright  

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

You may be tempted to do the same with Apache 2.0 License. The license text contains what looks like a copyright notice on line 189:

Copyright [yyyy] [name of copyright owner]

Actually, this is a copyright notice template and an additional boilerplate that are part of the Apache 2.0 License appendix, which starts at line 178 and explains how to apply the Apache License to individual source files instead of having one single LICENSE file in the top level of the source tree. The text in the appendix is very explicit about what the copyright notice template and additional boilerplate are for (use in source files). Changing this copyright notice template directly in license text is IMHO an improper application of the license. The whole appendix (lines 178-201) can technically be omitted from the license, as it’s outside the scope of “Section 1 through 9”, which forms the terms and conditions of the license.

My opinion is backed by the “Reusable without rewording” attribute of the Apache 2.0 License I mentioned before. It’s even backed by my experience in a big corporation - two years ago, I was working for one of the biggest tech companies in the world. The company had established an Open Source office with a Legal Department dealing with use of Open Source within the company. This Open Source office stipulated that for an Open Source project to be licensed under Apache 2.0 License, it is not required to have an explicit Apache 2.0 License text as part of the project. It is absolutely sufficient if the Open Source project README file contains the name of the license, either “Apache 2.0 License” or its SPDX identifier. Ideally, (but not required) canonical URL where the copy of the license can be obtained should be mentioned as well: https://www.apache.org/licenses/LICENSE-2.0.txt. This is just a legal opinion of one big tech company. Don’t take this legal opinion as implied truth.

Copyright notice

In the previous section, we’ve stipulated that the Apache 2.0 License text doesn’t actually have a place where to put an explicit copyright notice. Since 1989, Berne Convention established that the copyright is automatic, and the use of copyright notices to claim the copyright has become optional.

This is how an explicit copyright notice looks like:

Copyright 2021 Vladimír Gorej

When an explicit copyright notice is missing in Open Source software that you want to use in your project, you’re theoretically still OK, as it is possible to compile one using the following assumptions:

The name of the copyright holder is the name of the first committer to the project
The copyright holder contact is established by using the email address of the first committer
The year of the copyright notice is established as the year of the first commit to the project

If you cannot somehow obtain the above information, you’re kind of screwed, as there is no way to establish the copyright notice for compliance purposes in big tech companies.

It’s always a good idea to provide an explicit copyright notice when applying the Apache 2.0 License. But, where do we put one? Well, if we would read the Apache 2.0 Licence terms and conditions carefully, section 4.d) (line 106) has an answer for that: a “NOTICE” text file. Just create a new file called NOTICE and put your copyright notice there.

Today there are three competing formats of copyright notices:

Historical format: Copyright 2017-2019, Vladimír Gorej
Copyright 1998, Linus Torvalds
Newer format: Copyright The Ramda Adjunct contributors
Copyright The Kubernetes Authors
Hybrid format: Copyright 2017-2019 Vladimír Gorej and the Ramda Adjunct contributors

Given above, your NOTICE file could look like this:

Ramda Adjunct
Copyright 2017, Vladimír Gorej

Closing words

Choose the appropriate Open Source license for the needs of your project. If you’re deciding between BSD, MIT, and Apache 2.0, go for Apache 2.0 License. The license automatically grants patent rights. It’s clear, explicit, and reusable without rewording. To apply the license to your Open Source software project, create two files: LICENSE and NOTICE in the top level of the source tree. The copy of the license text that you obtained from https://www.apache.org/licenses/LICENSE-2.0.txt goes into the LICENSE file. Your copyright notice goes into the NOTICE file. Doing these two things makes the application of the Apache 2.0 License as explicit and clear as it can be. Sounds pretty easy, right?

Another good source of information about applying Apache 2.0 License to a software project can be official “How to” from Apache Software Foundation directed to Apache Committers.

Things I have learned while maintaining JavaScript monorepo with Lerna

2021-09-07T12:15:00+00:00

This is the second part of the article series dealing with monorepo management using Lerna. The first part dealt with monorepo management in depth using Lerna and npm local paths long before npm@7 workspaces were introduced. This article will try to describe monorepo management practices in the world of npm@7 workspaces. Nevertheless, I highly recommend you read the first part of this series to be familiar with how things used to be.

What is monorepo?

Let’s first define what monorepo is:

Monorepo is a software development strategy where code for many projects is stored in the same repository.

Monorepo code is usually divided into separate contexts, called packages. Every package contains code specific to its context and can depend on zero or more other packages within monorepo. Monorepo must have a mechanism to interconnect packages and allow us to use one package code within another.

In the following texts, I’m assuming you have your monorepo setup as described in the first part of this series.

Dependency management

Dependency managed has been simplified significantly with npm workspaces. npm workspaces require npm@7. This version of npm uses the new format of a package-lock.json file. You’ll need to convert your package-lock.json file to the new format for workspaces to work correctly.

A word on expressing dependencies between packages

We now use asterisk symbol to express dependencies between different monorepo packages. Given that following are dependencies of package-c which, requires package-a and package-b, this is how we express it in package.json:

1
2
3
4
5
6
"dependencies": {
  "ramda": "=0.27.0",
  "ramda-adjunct": "=2.27.0",
  "package-a": "*",
  "package-b": "*"
}

Notice that we no longer use explicit npm local paths but rather an asterisk symbol. Asterisk symbol is translated to npm local paths under the hood in a package-lock.json file.

Before npm workspaces, we had to list all packages in the dependencies field of the main monorepo package.json file.

1
2
3
4
5
"dependencies": {
  "package-a": "file:packages/package-a",
  "package-b": "file:packages/package-b",
  "package-c": "file:packages/package-c"
}

This is no longer required. npm workspaces support recognizing monorepo packages by defining a workspaces field in the main monorepo package.json file:

1
2
3
"workspaces": [
  "./packages/*"
]

We don’t need to add a new package explicitly in the dependency field whenever we want to add one. This saves time and prevents cryptic errors when we forget to add the package explicitly as a dependency.

A word on development dependencies

It is no longer required to keep all development dependencies in main the monorepo package.json file. Every monorepo package can now have its own development dependencies in a particular version, given the specific needs.

Before npm workspaces, the devDependencies field of every package package.json file was ignored. Now, npm@7 recognizes it and installs the npm packages listed in devDependencies.

IMHO it still makes sense to keep certain devDepedendies (like testing frameworks) in the main monorepo package.json file.

A word on topology

npm workspaces are aware of monorepo package topology. Workspaces know that package-c uses package-a and package-b as its dependencies. Let’s run the following command, which runs build npm script in all the monorepo packages:

 $ npm run build --workspaces

This command will run npm run build for all the monorepo packages. But it seems that npm workspaces don’t run the monorepo packages command in order as expected. Let’s say that package-a depends on package-b and package-c depends both on package-a and package-b. The order of execution you get from running the command is:

package-a
package-b
package-c

But the correct order of build should be:

package-b
package-a
package-c

Proper order of execution can be achieved by defining each package explicitly and in the correct order in the workspaces array of package.json. This array then needs to be maintained manually to reflect the monorepo topology.

{
  "workspaces": ["./packages/package-b", "./package/package-a", "./packages/package-c"]
}

Because of this, Lerna still plays a significant role here. Lerna will determine which packages have build npm script defined, and then it determines the proper execution order.

 $ lerna run build

Order of execution:

package-b
package-a
package-c

A word on installing a new dependency

npm workspaces completely simplified the process of adding a new production dependency. Let’s say we have a package-a, and we want to add a dependency of ramda-adjunct@2.27.0. The only thing we need to do is run the following command:

  $ npm install ramda-adjunct@2.27.0 --workspace=package-a --save

A word on installing a new development dependency

As already stipulated, development dependencies can be installed either in top-level monorepo (suitable for testing frameworks)

 $ npm install mocha --save-dev

Or installed as a specific development dependency of a particular monorepo package

 $ npm install rimraf --workspace=package-a --save-dev

A word on npm binaries

Before npm workspaces, to run a binary from one of the monorepo packages, we had to use the link-parent-bin npm package to link top-level npm binaries to all monorepo packages. This is no longer required as npm supports this natively.

Closing words

Given all that has been written here, I think npm workspaces still doesn’t replace Lerna. Instead, they just significantly simplified how we work with Lerna. Lerna still plays a significant part in change detection, releases, and running npm scripts in the proper order (given the package topology).

All the examples mentioned in the article can be found in the following Lerna monorepo repository I’ve setup-ed for you. Clone it, install it, play with it and have a lot of fun with it!

lerna-monorepo-taming

This monorepo implements current practices and conventions used while building monorepos using Lerna.

Maintaining Lerna monorepo series:

Part 1: Things I wish I had known when I started JavaScript monorepo with Lerna
Part 2: Things I have learned while maintaining JavaScript monorepo with Lerna

How to validate OpenAPI definitions in Swagger Editor using GitHub Actions

2021-03-07T08:39:04+00:00

Building a GitHub Action workflow that uses Swagger Editor to validate an OpenAPI definition.

The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to HTTP APIs, allowing both humans and computers to discover and understand the service’s capabilities without access to source code, documentation, or network traffic inspection. When properly defined, a consumer can understand and interact with the remote service with minimal implementation logic.

An OpenAPI definition can then be used by documentation generation tools to display the API, code generation tools to generate servers and clients in various programming languages, testing tools, and many other use cases.

I took the above description of what OpenAPI is directly from its specification. If you’re one of the people who write OpenAPI definition by hand in Swagger Editor, this article was written just for you.

A couple of years ago, I worked with Ubiquiti Inc. We were producing great wireless hardware solutions with embedded UIs. I was fortunate to have joined the company when a cross hardware configuration system called UNMS was being architected and developed. The backend part of the system consisted of a vast REST API layer that the Frontend layer was consuming. We used a design-first approach to producing the API. The workflow of this design-first approach consisted of the following steps:

create changes in OpenApi 2.0 definition using Swagger Editor
issuer: issue a GitHub Pull Request for the team to review how new API endpoints could look like
reviewer: paste the OpenAPI definition from Pull Request to Swagger Editor to see if no errors were introduced
merge Pull Request
implement the actual REST API defined in OpenAPI definition

Today I’d immediately think about how to automate this workflow and make Continuous Integration do the work for us. Unfortunately, at that time, my knowledge about CI/CD pipelines was limited, and I wasn’t aware of the full benefits and values CI/CD provides.

Some time has passed since then. I did my homework and studied CI/CD topic thoroughly. When GitHub Actions were born I immediately became a massive fan. Let’s try to automate the workflow described above using GitHub Actions.

Technical design

We need to produce a GitHub Action that uses Swagger Editor to validate the OpenAPI definition provided as a parameter to that action. Using a Swagger Editor in GitHub Action can be achieved in two ways: running it in a docker container using swagger-api/swagger-editor image or using https://editor.swagger.io/ directly. Now that we have Swagger Editor running, we use puppeteer to open a headless version of Chromium Browser. Then we paste the OpenAPI definition into the Swagger Editor and wait for it to render errors (if the definition is valid, no errors will be generated). Read errors from the Swagger Editor using puppeteer and represent them via GitHub Actions API. GitHub Action will report failure on any error or success if the document is valid and contains no errors.

Why use Swagger Editor and not just JSON Schema validation? Swagger Editor adds its own layer of error recognition on top of JSON Schema validation. Unfortunately, this additional layer is tightly coupled to Swagger Editor code, and the easiest way to use it is to use it via running the Swagger Editor in a browser.

Implementation

The implementation wasn’t that straightforward as I expected, but I still managed to implement this GitHub Action against the technical design mentioned earlier with all its requirements. I called it swagger-editor-validate.

swagger-editor-validate

This GitHub Actions validates OpenAPI (OAS) definition file using Swagger Editor.

Usage

There are two significant use-cases of using this GitHub Action, but both of them use the fact that your definition file lives in your GitHub repository. You just need to provide a file system path to it.

Public use-case

If you have access to the internet and don’t mind that this GitHub Action sends your OpenAPI definition to https://editor.swagger.io/ for validation, then use this workflow.

1
2
3
4
5
6
7
8
9
10
11
12
13
on: [push]

jobs:
  test_swagger_editor_validator_remote:
    runs-on: ubuntu-latest
    name: Swagger Editor Validator Remote

    steps:
      - uses: actions/checkout@v2
      - name: Validate OpenAPI definition
        uses: char0n/swagger-editor-validate@v1
        with:
          definition-file: examples/openapi-2-0.yaml

Private use-case

If you want to maintain complete privacy and your OpenAPI definition may contain sensitive information use the following workflow. The workflow uses a swagger-editor docker image that runs as a service of the workflow.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
on: [push]


jobs:
  test_swagger_editor_validator_service:
    runs-on: ubuntu-latest
    name: Swagger Editor Validator Service

    # Service containers to run with `runner-job`
    services:
      # Label used to access the service container
      swagger-editor:
        # Docker Hub image
        image: swaggerapi/swagger-editor
        ports:
          # Maps port 8080 on service container to the host 80
          - 80:8080


    steps:
      - uses: actions/checkout@v2
      - name: Validate OpenAPI definition
        uses: char0n/swagger-editor-validate@v1
        with:
          swagger-editor-url: http://localhost/
          definition-file: examples/openapi-2-0.yaml

This is what you’ll see if your OpenAPI definition validates successfully.

Successful validation

This is what you’ll see if your OpenAPI definition contains errors.

Failed validation

Future

During implementations, I identified other features that this action would benefit from. To not extend the implementation scope of the first version, I decided not to implement these features but instead document them and implement them later. These features include:

I think the most interesting one is - Mechanism for ignoring errors. Commonly, OpenAPI definitions contain errors when validated in https://editor.swagger.io/. There are cases when there is nothing authors can do with these errors, which are known and ignored. This will open a door for such a use-case, which I’m sure is quite common.

Hope that swagger-editor-validate will become a handy tool for anybody using Swagger Editor to edit OpenAPI definitions. I do know it will become convenient for my old team in Ubiquiti Inc ;] Please leave any feedback in the comments. I’d highly appreciate it!

How Swagger Adjust can help you build extensible React+Redux apps

2021-02-25T08:39:04+00:00

Step by step guide to building React-Redux apps using pluggable architecture. Features are produced as plugins and are later composed to create the app. Plugins can be removed, added, or amended by other plugins.

More than four years ago, I worked on a backend project where I needed to render HTML documentation from OpenApi 2.0 JSON document defining our REST API written in hapi.js. The obvious choice was, of course, SwaggerUI. hapi.js ecosystem came with already implemented integrations for OpenApi 2.0 and SwaggerUI. We just needed to use them, and we did. It all worked like a charm. At that time, I successfully got out of the complex Angular 1.2/2 world and got into the simple and more predictable world of React+Redux. I figured out that SwaggerUI was built in React, so I navigated to its GitHub repo and read the code to learn how large-scale React apps are written. The code was different from the code I was used to writing. I built React applications using classical patterns like Presentational and Container Components (smart & dump components) and ducks pattern. It took me some time to understand that SwaggerUI was primarily built for extensibility in mind. Everything inside the code was revolves around this primary idea. The codebase was composed of several plugins. These plugins were either adding new behaviors to SwaggerUI or enhancing other behaviors defined in different plugins. There were almost no static imports involved because it had its own flavor of DI-container called System. Although the architecture looked interesting, it seemed overkill to use something like that in my current project.

In 2017, I was employed by a company called Apiary. I joined it a couple of months after the acquisition by Oracle, and my primary task was to architect and develop a new version of Apiary Documentation Renderer named: ApiaryUI. When I started to work on it, I remembered my study on SwaggerUI and its unique architecture. I built ApiaryUI around the same ideas of external extensibility. Of course, the code was completely different, but the architecture was very similar to SwaggerUI. The core of ApiaryUI was based on a DI-container called Kernel. I managed to finish the new ApiaryUI on time. Unfortunately, it has never been released as OpenSource due to different political and organizational reasons. Still, it’s been utilized by Apiary in its web app and used throughout various Oracle organizations.

Today I work for SmartBear, the company that’s behind SwaggerUI and the entire Swagger ecosystem. My primary focus in SmartBear is not on SwaggerUI, but from the moment I joined the company, I started to think about improving and modernizing its DI-container (System). I’ve been actually thinking about this for a long time. I started writing this article almost 3 years ago, when I decoupled its DI-container for the first time as I needed it for a small project with external extensibility requirements. Unfortunately, I never finished the article and never published the decoupled SwaggerUI DI-container as an OpenSource.

Swagger Adjust

But it’s all changing now. Let me introduce Swagger Adjust - a plugin-able framework for building extensible React+Redux applications. This framework was born by decoupling ideas of the plugin-able core of SwaggerUI into a separate reusable component. It allows you to produce React+Redux applications where every component, being a React Component, Redux state slice composed of actions, selector, reducers, or general business logic, is override-able and extensible.

Swagger Adjust is a wholly rewritten SwaggerUI’s DI-container (System) built on the Redux Toolkit. The name for the DI-container is the same as SwaggerUI’s – System. Generally, the API of Swagger Adjust is backward compatible with SwaggerUI. When I say “generally”, I mean it in a way that the default configuration of Swagger Adjust is incompatible with SwaggerUI but can be adapted by configuration to be fully compatible. Swagger Adjust is a free and Open Source software. It comes with a dual license – Apache License Version 2.0 is used for original code fragments, documentation fragments, and original ideas from SwaggerUI, and BSD 3-Clause License is used for everything else.

To demonstrate Swagger Adjust power, I’ve built a simple TodoList app composed of two plugins: todoList and todoListEnhancer. todoList plugin has the basic functionality of displaying a list of TodoItems and adding a new one. todoListEnhancer adds additional functionality like completion, deletion, and also batch operations on the entire TodoList.

todoList plugin

todoList plugin looks like this when it renders. It contains AppBar, input for creating new todo items, and a list of already added todo items.

The directory structure of todoList plugin looks pretty standard. There is a components directory containing all the React Components. Redux-related code is concentrated in: actions.js, selectors.js, and reducers.js. Source code of these Redux files is unaware that it's part of a plugging-able DI-container and are built in the same way as they'd be built using Redux Toolkit. Please take a moment here and browse these files yourself to see the actual code.

The only non-standard file is plugin.js. This file imports everything that todoPlugin consists of and exports a function returning an object with a specialized shape that our DI-container recognizes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import TodoListLayout from './components/TodoListLayout';
import TodoListAppBar from './components/TodoListAppBar';
import TodoListInput from './components/TodoListInput';
import TodoList from './components/TodoList';
import TodoListItem from './components/TodoListItem';
import { addItem } from './actions';
import { selectTodoList, selectTodoListItems } from './selectors';
import reducers, { initialState } from './reducers';

const TodoListPlugin = () => {
  return {
    components: {
      TodoListLayout,
      TodoListAppBar,
      TodoListInput,
      TodoList,
      TodoListItem,
    },
    fn: {
      todoList: {},
    },
    statePlugins: {
      todoList: {
        initialState,
        actions: { addItem },
        selectors: { selectTodoList, selectTodoListItems },
        reducers,
      },
    },
  };
};

export default TodoListPlugin;

Now, this is where the real magic happens. Instead of using static imports in React components, we use specialized hooks to get everything we need from DI-container. For the TodoList component to correctly render, we need to select all todo items from the state and then use the TodoListItem component to render them.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import React from 'react';
import List from '@material-ui/core/List';
import Divider from '@material-ui/core/Divider';
import { makeStyles } from '@material-ui/core/styles';
import { useSystemComponent, useSystemSelector } from 'swagger-adjust';

const useStyles = makeStyles(() => ({
  root: {
    paddingTop: 0,
    paddingBottom: 0,
  },
}));

const TodoList = () => {
  const classes = useStyles();
  const items = useSystemSelector('todoList', 'selectTodoListItems');
  const TodoListItem = useSystemComponent('TodoListItem');

  return (
    <List className={classes.root}>
      {items.map((item) => (
        <React.Fragment key={item.id}>
          <TodoListItem item={item} />
          <Divider />
        React.Fragment>
      ))}
    List>
  );
};

export default TodoList;

DI-container has its own React Context called SystemContext, which accepts its instance. Tne first thing we do is – compose a list of plugins and pass it to the System class constructor (represents DI-container). DI-container automatically compiles the plugins under the hood and creates a store instance for us. Then we wrap our main App component into react-redux and SystemContext providers. This allows hooks mentioned above to access anything from the DI-container within React components.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
/**
 * This component is just responsible to render the main TodoList component.
 * We need to this to allow other plugins to override TodoListLayout.
 */

const App = () => {
  const TodoListLayout = useSystemComponent('TodoListLayout');

  return <TodoListLayout />;
};

ReactDOM.render(
  <React.StrictMode>
    <Provider store={store}>
      <SystemContext.Provider value={system.getSystem}>
        <App />
      SystemContext.Provider>
    Provider>
  React.StrictMode>,
  document.getElementById('root')
);

Let’s look into our second plugin – todoListEnhancer. As mentioned before, this plugin enhances the original todoList plugin and adds additional features. Enhancement doesn’t happen via todoList plugin code modification but rather by its configuration. And this is where Swagger Adjust really shines.

todoListEnhancer plugin

Every todo item in todoListEnhancer is now associated with creation time, the checkbox for completion, the bin icon for deletion, and at the bottom of TodoList, there are two buttons for batch operations.

Here, we’re presented with todoListEnhancer directory structure and the source code of plugin.js. Again, please take a moment here and browse these files yourself to see the actual code. Inside plugin.js we’re replacing the TodoListItem component with the new one. We’re adding the new TodoListBatchOperations component and wrapping the TodoList component with a new one that composes the original TodoList component with TodoListBatchOperations. Additionally, we’re exposing the time formatting function to the plugin so that additional plugins could easily override how time is formatted.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
import { assocPath } from 'ramda';

import { completeItem, uncompleteItem, deleteItem, completeAll, deleteAll } from './actions';
import { formatTimestamp } from './fn';
import reducers from './reducers';
import TodoList from './components/TodoList';
import TodoListItem from './components/TodoListItem';
import TodoListBatchOperations from './components/TodoListBatchOperations';

const TodoListEnhancerPlugin = (system) => {
  const todoListFn = assocPath(['formatTimestamp'], formatTimestamp, system.fn.todoList);

  return {
    components: {
      TodoListItem,
      TodoListBatchOperations,
    },
    wrapComponents: {
      TodoList,
    },
    fn: {
      todoList: todoListFn,
    },
    statePlugins: {
      todoList: {
        actions: {
          completeItem,
          uncompleteItem,
          deleteItem,
          completeAll,
          deleteAll,
        },
        wrapActions: {
          addItem: (oriAction) => (payload) => {
            oriAction({ ...payload, completed: false, createdAt: Date.now() });
          },
        },
        reducers,
      },
    },
  };
};

export default TodoListEnhancerPlugin;

The last thing we need to do is to register the todoListEnhancer plugin with the DI-Container.

1
2
3
const plugins = [TodoListPlugin, TodoListEnhancerPlugin];
const system = new System({ plugins });
const store = system.getStore();

When building React+Redux features in this architecture, one always has to think hard about exposing React components, Redux code, and other business or presentational logic to the plugin so that other plugins can easily enhance it. This also works in reverse – when building features that need to be “protected” from enhancements, instead use classical static ES6 imports, which prohibits anybody from enhancing your plugins.

Advanced usage patterns

During writing Swagger Adjust and using it on different projects, I’ve identified multiple advanced usage patterns that need to be mentioned. These advanced usage patterns include:

Swagger Adjust consumers need to understand these usage patterns properly as they set conventions on how Swagger Adjust was designed to be used.

Word on conventions

Conventions are essential when producing any code. When utilized properly, it makes code produced by different people look like it was produced by a single person. Redux Toolkit has done a great job setting up conventions and standards for React+Redux applications. Swagger Adjust adopts all these conventions and adds a couple of its own:

use createAction helper to create action creators
use following convention for action type: “namespace/action”
use createAsyncThunk to create async actions or handle side effects
structure Files as Feature Folders as shown in this article
name your selectors in the following naming scheme: select
create memoized selectors using createSelector utility

Consult Redux Style Guide for additional conventions.

Closing words

Swagger Adjust it’s an ideal framework for creating User Interfaces (UI) with external extensibility requirement. It should work pretty well for creating renderers for different specifications like OpenApi, AsyncApi, or ApiBlueprint (as SwaggerUI demonstrated). It allows external extensibility via configuration and plugins, bringing much more features than demonstrated in this article. I welcome you to consult our documentation to learn more.

You can find all code demonstrated in this article in the Swagger Adjust GitHub repository.

swagger-adjust

Pluggable framework for creating extendable React+Redux applications.