Event-Driven by Oskar Dudycz — articles

Open Source, a relic, a charity or still the thing?

A friend asked me recently whether there is any sense in doing Open Source now. My answer was the same as always. I see three options.

1. You get money out of it.
2. You’re dogfooding it in your work projects.
3. You have fun doing it.

If none of that is true, it makes no sense and never did. Best if there’s a combination of 1 or 2 with 3. And at least one of those options is a bare minimum.

Doing OSS is mostly for fun; success stories of single people who succeeded financially through it are rare. Personally, even though some of my projects got wide adoption, I always struggled to make money from them to justify the effort. Unfortunately, OSS creators don’t get enough respect for their effort; with GenAI, that’s even lower. And I don’t think that’s good for the industry long-term, but that’s, in my opinion, the reality.

So my advice would be to think about whether:

1. Your project has commercial potential.
2. It can be turned into a product.
3. There are people willing to pay for that.

If yes, go closed-source and productise it. You can always open it later or make the source available to customers.

This kind of goes back to the old days of shareware, etc. I don’t like that, but this is how it goes. I also need to learn from my own advice.

„I’m just standing on the shoulders of giants”

is a common phrase to show that you’re humble.

In the Open Source, it is the opposite. In our industry, giant companies stand on the shoulders of a small group of passionate, regular folks.

Many people believe that “someone is sponsoring those projects”, but when you think about “ok, but did I sponsor some?” then the answer is usually no.

I have the luck to get some sponsorship. Not huge, but still. I’m calling that “Whiskey money”. I don’t drink a lot, but I could buy a bottle with that amount from time to time and could salute those good folks.

It’s important to think of supporting OSS as due diligence, not gratitude. It’s ensuring that critical parts of your architecture will be maintained and continued. Essentially, that lowers your cost and risk.

Yes, we can vibe code a lot these days, but do we want to take responsibility and get late-night calls if that part doesn’t work?

So folks, check whether the OSS you’re using has support contracts, consulting or workshops, sponsorship, or paid add-ons. Don’t think of them as charity, but as a way to outsource work someone else can do cheaper and more reliably.

We all wanted to be Netflix, who do we want to be next?

I recently realised that there’s an aspect of GenAI tooling that’s not spoken about much, and I think that’s important to where we are going as an industry. Before I get into the details, let me start with some background.

For a long time, our industry was (over) focused on following big tech’s practices. “Let’s be like Netflix!” - that was a common phrase. Even if we weren’t saying it out loud, our discussions, directions, design, and tooling choices said it.

Of course, there was always a group of people saying: “well, it depends; we’re not Netflix, aye? So maybe we do stuff simpler?”. Still, even they were saying stuff like “I don’t like microservices, but there are valid scenarios for it”. The soft power of big tech was strong.

To some degree, blind copying of their practices led to overengineering, over-staffing, and overly complex operational stuff (hey, Kubernetes!). It even impacted recruitment and staffing with weird practices like LeetCode or System Design interviews.

Yet, undeniably, it also had a good impact. Cloud computing commoditization enabled companies without enough internal skills to build better, more secure products. It also enabled models like pay-as-you-go. Plus, those edgy problems that big tech had ended up with useful, mature open-source projects.

Of course, those “insightful blog articles” included a lot of marketing. They shaped our industry narratives. Companies wrote about the problems they had, which were mostly problems of scale, and those became the problems everyone talked about. They were success stories. Even when they were doing post mortems, or other fuck ups, they ofc presented it as “hey, but we eventually to solve it, so we’re good right?”. Also, some practices attributed to them weren’t actually used. Google didn’t use Angular; Spotify didn’t use the Spotify Model, etc. Still they carried a set of good intentions: explaining how stuff can be done bigger, faster, or just a bit of bragging on what they do. On its own, I consider that a good thing.

It was also a good thing because of the form it took. A blog post is an argument. You can read it, disagree with it, take the idea and drop the implementation, or tell your team it solves a problem you don’t have. It caused discussions and new ideas. Sometimes backslashes, sometimes maturity. Those ideas had a positive impact, because there was something on the table to point to. We could say, that’s what we’re going to try, or we could say: no, it’s not for our scale. That led to answering questions of why. Of course, many didn’t ask why and just blindly followed, causing more issues than benefits.

Now, here’s what’s bothering me.

We all wanted to be like Netflix. Do we want all of our development processes and practices to be like OpenAI’s or Anthropic’s?

Some would say “yeah, why not, they’re big and successful” (whether they are is a different story), but…

But what you get is not an OpenAI or Anthropic way. It’s the way they think engineers like us should, or “could”, do our work.

I don’t have insider info from those companies, so this is more my bet: knowing they’re the “Ivy League” of devs, my bet is they don’t think we’re capable of doing it all right. They may optimise the solutions and the engineering workflow for us at the “average Joe engineer” level.

This is a much different dynamic than we had with “Let’s be like Netflix”.

Copying Netflix meant copying someone who had solved (or at least claimed to solve) a problem in a specific context. Of course, Netflix and likes were selling something too. Those posts were recruitment, reputation, sometimes a push for their own open source. But their revenue didn’t depend on whether we adopted any of it, and the practice travelled together with the reasoning, because the reasoning was most of what they published.

With the LLM tooling, the practice comes embedded in a product, the reasoning stays inside the vendor, and that vendor’s revenue depends on how much we use it. We can’t read the thinking the way we could read a blog post. We can’t measure thre consequences.

I’m not even talking about coding. Many people let LLM tooling to break down problem, define what counts as done. And those decisions were always the slow part, and they are where development cost, quality and delivery time actually come from. Nobody outside our industry cares how we code; they care about those three. So when the deciding moves into the tool, what stays on our side? If we’re betting that all can be outsourced, then what’s our role?

With all this generated code, specs, and whatever else you get from Genie, you’re commoditising the knowledge harvested (or, um, stolen) by OpenAI, Anthropic and Google. That can be fine if you’re below average in your engineering practices or you’re lacking certain skills. I’ve seen teams that, thanks to that, were able to survive with an inherited maze created by other people. For plenty of teams and plenty of tasks, the average answer is better than what would have been shipped otherwise. We already see obvious choices like React for frontend and PostgreSQL for the database.

I’m all for choosing a boring tech stack. As a default choice, the issue is whether that’s the only choice or if we’re making our solution simplistic instead of simple. Or complicated instead of complex.

LLM training is as biased as the people who trained it, through weights, training data and whatever tuning sits on top. Same with the harness, or even more, as it’s the code someone put there. I wrote about it some time ago in the pre-LLM AI era.

GenAI tools are statistical parrots; they repeat the most probable answer, which most of the time means either mediocre or good, decided by the person who set the weights during training. Training is always subjective. If you ship one default to everybody, you tune it for the widest audience, which lands in roughly the same spot.

Some can say, that we could use open source models instead. Personally, I don’t consider open models or custom workflows per se as something changing this dynamic. I think that’s better, but not a solution. They change who holds the defaults, not the fact that defaults are now doing the arguing for us. Plus the cost of servers is still super high compared to the value.

With GenAI tooling we’re not only getting data and ideas on how to solve our design challenge or fix a bug. We’re also getting the workflow we’re forced to work with. This is impacting us from both angles. We ask and work on the design with this tooling and also let it do the final work. Harness is not only for getting and interpreting your needs through a model. It actually also harnesses your code and the design you get.

And now the question is: who’s putting the leash? Are we on our chosen design to ensure it’s as we want. Or is the GenAI harness putting the leash on us to provide the design it’s capable of or fine-tuned for?

I wrote about my vision of the harness. I see it more as a way to ensure our process is reproducible and follows our intention: tests, automation, traces, a setup. We design it to be able to iterate fast, but not just for going fast, but to get quickly feedback loop and validate our assumptions. Going fast, in my opinion, shouldn’t be ever a goal on it’s own.

The vendor harness is a different animal. It’s created by vendor with the masses in mind. It has its own system prompt and tool definitions; it decides what gets pulled into context and what doesn’t, when the agent decides it’s done, and whether it asks you something or just guesses. It reflects the vision of authors on how the process look like. It’s not optimised for our approach; we need to fit into it if we want to get the benefit. Just like with application frameworks: we can benefit as long as our approach aligns with the framework’s author’s vision; otherwise, we’ll struggle by not doing things idiomatically. The challenge with LLMs is that we don’t necessarily know the idioms, since big vendors aren’t sharing them. They only provide guides as they see us working with their tools. And those guides don’t have to be optimised for doing things right, but for using those tools more. And we never know, as internals are hidden.

So, we don’t know the weights or vendors training data; we don’t know how those harnesses work internally, and even if we did, they could change at any time. Between one release and the next, the tool making decisions inside our workflow can start behaving differently, and you won’t necessarily be told.

I’m not saying to drop using them. On my own I’m using those tools everyday, with mixed results. Till this day what works best for me is the process I described in Interactive Rubber Ducking with GenAI:

I’m using GenAI tool to ask me questions about my idea,
Then letting me answer and make decision without making them for me,
Then summarising into spec, that I work on,
Doing the work on my direction and review.

None of the workflow, automated multiagents etc. didn’t work for me in a proper, reliable way.

I also believe that’s also more aligned with how LLMs are build, they’re great in text processing. They’re not thinking, they’re just processing.

What to do about it? The first step is to realise it we have a problem, but frankly, I don’t have clear solution.

I don’t know.

I think that’s also why we have blogs, to share, put a seed, have a discussion. It’s fine to just don’t know all answers.

What I know is that we shouldn’t be outsourcing our thinking, as that’s the role of engineer, if we lose it, then indeed our profession is doomed. Do we want to truly bet on that?

I also believe that’s a more important discussion than whether we should code or not, or what’s the latest and greatest model today at 12:37 AM.

Thoughts?

Cheers!

Oskar

p.s. You may be also interested on my other articles on GenAI

Avoid Duke Nukem Forever Mode

How old are you? I’m Duke Nukem 3D old.

Not in the sense that I was born in 1996, but that it came out when I was the right age. Being 11 years old was the right age to laugh hard at Duke’s jokes. Duke Nukem 3D was the first shooter game that asked the question: Why so serious? It was a unique game, as almost none of the earlier (and future) blockbusters won the audience with a lame sense of humour.

It was a great hit. Even though it wasn’t actually 3D, more like 2.5D. It doesn’t have the best graphics, but it’s good enough, especially when paired with the creativity of the plot, scenarios, and gameplay. It sold 3.5 million copies.

It was such a success that the year after, the 3D Realms game studio co-founder George Broussard announced the sequel: Duke Nukem Forever. He expected to deliver it as a Christmas gift in 1998. It appeared to be quite a bold statement, since the game was released in June 2011. Yes, 13 years later than promised. What happened?

It’s a long story; look here: an almost hour-long video on that:

So let me be shorter and give you an even shorter version than the Wikipedia Page:

They realised that the 2.5D engine was already obsolete, and in 1.5 years it’ll be even more so, so they bought a license for the Quake 2 engine. For 0.5 million bucks. A lot, keeping in mind that they got only 0.6 million bucks in advance from their publisher for the new game.
They showed the nicely looking trailer; critics were happy, but creators wanted more. They wanted something better, and after 14months of investment, they bought the latest-the-greatest at that time Unreal Engine, because why not. Just half a year till the promised Christmas.
Not surprisingly, changing the engine while building the ~~car~~ game wasn’t the best move for the deadlines, and the game was still unfinished and in moving parts, even more so than it was. So, in 1999, they decided to change the engine again: to the new Unreal Engine, which had multiplayer capabilities. Yes, they decided to add multiplayer to an initially single-player game. A bit of scope creep.
They worked on it and even showed something that looked nice in 2001, which, as Broussard said, was actually smoke & mirrors. Take-Two Interactive bought them and got into a fight with the new owners, claiming it would be done when it was done.
In 2004, they switched engines again, this time to Doom 3.
In 2006, nothing was working yet; the team was “basically pulling it all together and trying to make it fun”
In 2009, they laid off the staff, downsizing the company. Take-Two filed a lawsuit against 3D Realms over their failure to complete Duke Nukem Forever, citing $12 million paid to Infogrames in 2000 for the publishing rights
Nine ex-employees continued development throughout 2009 from their homes. Eventually, they created Triptych Games, an independent studio.
Somehow, they managed to complete something and release it in 2011.
The outcome? The game was a total flop, nowhere near the success of Duke Nukem 3D.

Fun fact: until 2024, Duke Nukem Forever held the Guinness World Record for the longest development of a video game, at 14 years and 44 days.

Ok, why am I writing about it in the Architecture Weekly?

Because I’m also the one to blame. Not for the Duke Nukem delay, but for falling into the same trap. “Do as we tell, not as we do”.

When I started Emmett, I went with the lean approach. I started by delivering the API for building Event-Driven applications. The first version didn’t even have any event store implementation.

I showed that to the rest of the world, and I quickly got feedback from folks who wanted to use it and said they were happy with what they saw. So I added the storage I already had in my samples: EventStoreDB storage.

Then I added, for fun, Pongo - a MongoDB-compliant library that allows you to use PostgreSQL as a document database.

And I decided to add Dumbo, a shared package that will be responsible for connection management, SQL queries, etc. and use it to build the PostgreSQL event store.

That worked well; Pongo made it to Hacker News’ front page. Emmett, PostgreSQL is probably the best Node.js PostgreSQL-based event store (not much competition, to be fair). People are using it in production. I haven’t had much success making income from it yet, but at least adoption is growing steadily.

Then I decided it was time to make it production-ready.

It was 1.5 years ago.

And yes, it’s production-ready, but the new official versions are still unreleased: 0.43.0 of Emmett and 0.17.0 of Pongo.

Why? For various reasons, some personal ones, but focusing on the development stuff:

I entirely rewrote connection management and implemented my own connection pooling.
Decided to add more database types for Emmett and Pongo: SQLite, Cloudflare D1. Fun fact: I used the sqlite3 Node.js driver, which has since become unmaintained.
That required making Dumbo drivers pluggable.
I decided to add OpenTelemetry.
I added resiliency and batching for processor implementations. Essentially rewrote the whole async processing.
I added Workflows for business process coordination.
And many, many other stuff.

Essentially, I locked myself into the big, great bang release. I also fell into:

Which is fine, as those tools are great: Emmett 0.43.0 and Pongo 0.17.0 betas are totally different products: richer in features, more performant, resilient, etc.

Still, if you fell into that, then it’s easy to justify yourself and persuade yourself that there’s “one more thing needed” and “if I waited so long, then we could wait a bit more”.

Luckily, I didn’t stop the feedback loop. I was publishing alphas and betas quite often. I treated this as something I could release as-is if I had to. So didn’t turn upside down. Some people even started to use those betas in prod. Some couldn’t, of course, because of compliance yada yada.

Still, I should have known that better. In past, I either fell into the same miserable place and promised myself not to fall into that again. Well, I lied.

The worst thing is that this is mostly sitting in my own head. People would prefer to get more frequent updates. Of course, one reason was that I wanted to group breaking changes to make this easier for folks, but if you group too many, people won’t migrate. There are other options, and I already smoothed the migration story.

That’s also a warning for you. With GenAI tools, it’s super easy to fall into analysis paralysis or refucktoring. It’s also easy to fall into the waterfall-style plan. Having too many options, too many ideas, even if they’re great doesn’t help.

Yes, if you’re motivated enough, you can achieve anything, but…

…but you cannot achieve everything. The time and focus we have isn’t infinite.

It’s much better to stay focused, have one streamlined workflow, deliver continuously, and handle breaking changes. And most importantly…

Focusing on the feedback loop.

So don’t be like George Broussard; don’t be like me and avoid Duke Nukem Forever mode. It’s oh-too-easy to fall into it. And consequences can be severe.

Just because you can do something doesn’t mean you have to. We need to pick wisely.

Cheers!

Oskar

Start Small, Grow Big: how to pick the first feature for Event Sourcing

Let’s say we did the hard part. We read up on Event Sourcing, even here on my blog; we did a modelling workshop, we got the nods around the table, and someone finally said:

“Fine, let’s try it somewhere”.

Then, after the cheerful atmosphere, the follow-up arrives:

“OK, but where?”

We do some head-scratching, pick something from the top of our backlog without further consideration. Some people say we should event-source everything so we can select anything, right?

Yes, you can use Event Sourcing everywhere, but you don’t have to, and it won’t add enough value over the familiar approach to justify the learning curve. It’s not that hard; it really doesn’t, but well, if you need to learn both the new approach and new tooling, and persuade colleagues, then it’s better to pick wisely.

In When not to use Event Sourcing? I ended with a suggestion: Start Small, Grow Big. Start with functionality that won’t hurt you if it fails, deploy it to production, learn from it, then rinse and repeat.

That sounds like nice advice, doesn’t it? But it’s easy to write and harder to apply.

Picking the right problem

“Small” is a word everyone agrees with and nobody defines it the same way. It’s in the same category as the “good enough” term. How small is good enough? How small is too small or too big? If we pick something so small, we can get a wrong, simplistic impression. If we pick something too complex (like our core domain), we may get into trouble, as…

Starting in the core domain is the most tempting mistake

Our core domain usually sounds like the best fit for Event Sourcing. That’s where the most complex logic happens. That’s where the most sophisticated business processes are. In theory, that’s where extended business observability pays off. Choosing based on the business domain fit would sound reasonable, aye?

Starting a new pattern in the core domain means doing three hard things at once:

fighting the complexity of a domain we don’t understand yet,
learning the practice and the patterns,
and learning the tooling.

Each is a real burden on its own. If we join them together, then we get the triple-size burden (or more). When we start getting issues (and we will), we can’t tell which of the three is currently biting us. Something breaks, and we can’t say whether we modelled the process badly, applied the pattern badly, or hit a library limitation.

We make the most important decisions when we’re the dumbest, and here we’d manage to be dumb in three directions at the same time. Then the blame game starts, and we choose the easy excuse:

Event Sourcing didn’t work for us.

Or worse:

Event Sourcing is too hard.

When we onboard a project with a new approach, we’re betting our credibility. And that’s a risky bet. If we do it wrongly, then even if next time we come up with the right choice, our colleagues can say:

Oskar, do you remember, how did it end up the last time you wanted to introduce Event Sourcing? Yeah, then how about no?

I’ve been on the other side of this. I rejected using Event Sourcing in a project where the domain fit well, as I was the only person familiar with it and we had teams spread across three countries. Saying no was cheaper than saying yes and becoming the single point of knowledge for everyone.

That’s also what I advised my consulting clients several times when they asked me to review their idea of applying Event Sourcing to their projects. Rarely was their business use case the biggest issue. More often the selected business case complexity compared to their stage in the learning journey.

Small doesn’t mean unimportant

In the ”When not to use Event Sourcing?” article, I suggested starting with non-important business functionality. That can be interpreted differently than I meant, so let me correct myself.

What we want is functionality that’s low-consequence-if-wrong, which is different from noone-cares-about-it. Unimportant features usually have no business decisions in them, and a feature with no decisions is not a great fit for Event Sourcing.

If it’s crude, we may come up with events like SomethingCreated, SomethingUpdated, and SomethingDeleted, produce a lot of boilerplate, and conclude that Event Sourcing brings useless ceremony. When picking a trivial use case, we typically end up with State Obsession or Property Sourcing.

I was a victim of that. I worked on a project with a dedicated configuration service. Mostly classical crudish dictionaries, but all integration ran through events. The amount of repetitive code was enormous. If you pick such a choice, you’ll only learn that choosing a bad fit for Event Sourcing is a bad fit for Event Sourcing. Or conclude that Event Sourcing is stupid. Or both.

The feature we pick has to be real. Simple, but real. A colleague depends on it, it has a business name, it holds real data, and someone complains when it breaks. A demo shopping cart persuades nobody, including ourselves, because everyone knows demos work. It just needs to be a feature where being wrong for a day means somebody fixes it by hand and no money is lost.

One thing to rule out before we go looking. Collecting events from several sources and assembling them into a read model isn’t Event Sourcing. It’s useful, and I’ve written about doing it well when events arrive out of order in Dealing with Race Conditions in Event-Driven Architecture, but if all we do is gather data and display it, we’re doing event-driven integration.

I wrote a whole article on why the distinction matters: Event Streaming is not Event Sourcing!. What makes it Event Sourcing is making decisions. We treat our events as the source of truth by: reading them, building state from them, and making a decision that produces a new event. In that race conditions article, the interesting moment is when the projection stops being a projection and becomes a workflow, because it started deciding something.

Audit needs are the same story. Event stores make auditing easier, and a proper audit log is more work than appending events, which I covered in Is the audit log a proper architecture driver for Event Sourcing?. If the only reason for our pilot is “we’d like history”, we’ll build history and learn nothing about write models.

Selection checklist

OK, let me give a rough checklist to make my rambling more actionable.

Let’s start with the business use case selection:

Who owns the process lifecycle? If the lifecycle lives in another system and we watch it, we’re building an anti-corruption layer. Events from outside are rumours until we interpret them.
Do we produce our own events, or only consume other people’s? If nothing new comes out of the module, we’re building a read model and calling it Event Sourcing.
Can a command be rejected for a business reason? Database timeouts don’t count. I mean rules like “this was already approved, so it can’t be changed anymore”. If nothing can be refused, we have no business logic, no decision-making, and no reason for Event Sourcing.
Is the decision a business rule or a retry policy? Retry, backoff and dead-lettering belong to our messaging tool. They’re technical concepts with business consequences, but not a real business decision.
Could we run an EventStorming session on it with a non-technical colleague? It’s much easier if we can collaborate with business people to get the domain knowledge from them. We can still do it on our own, but then we risk inventing the business scenario. If the events have no real business names, there’s no domain to model, and we lose the shared understanding that makes events valuable in the first place.
Does the stream end? Short streams keep event stores fast and versioning manageable. A pilot on an unbounded stream teaches that lesson the painful way.

Then, can we afford to be wrong?

Is it off the critical path? Nobody should be waiting on our learning curve to hit a release date.
What happens if it’s wrong for a day? If the answer involves money, safety or a regulator, it’s the wrong pilot.
Could we walk it back in a week? If the retreat is a project of its own, we picked something too big.
Is it small enough to do slightly rogue? If we need a formal decision before we can start, we’re back to arguing instead of showing.
Can the event store we choose run on storage we already operate? A new database means a conversation with the platform team, and that’s where most pilots die.

Then, will we actually learn something?

Does it fit inside a single module? We want eventual consistency and idempotency, and we don’t want clock skew and cross-queue ordering in the same week.
Is any part of it asynchronous? Waiting on an external system counts, and so does a timeout that fires on its own. Testing the async approach isn’t entirely about Event Sourcing as a pattern, but about checking how our tooling behaves and how we can reason about the challenges that will eventually land.
Is it real? Someone depends on it, it has a business name, it holds real data, and somebody complains when it breaks. Of course, we should limit the risk, but we shouldn’t also pick features that no one cares about, because then we won’t run it for real and get proper learning from production usage.

Bad picks

What are the examples of bad picks? No problem, let me walk you through three examples.

Email notification dispatch looks promising. Sent, delivered, bounced, opened is a state machine with a decent vocabulary, except the state machine belongs to the mail provider and we’re only watching it. The decisions we’d implement are retry and escalation, which our broker already does, so it’s a technical concept. We’re not making decisions; we’re just handling side effects.

Import and batch job processing looks promising too. Started, chunk processed, failed, retried, completed. No domain expert will attend a meeting about ChunkProcessed, and retry-versus-abort is infrastructure policy. Imports deserve explicit events when we migrate relational data, as I described in The end is near for CRUD data, but they’re not a pilot. So same issue as before.

A payment verification module looks the most promising of all. Before a card payment goes through, several checks run at once: is this likely to be fraud, how risky is this customer, is the shop still within its daily limit. Each check runs in a different system and answers at its own pace, sometimes in the wrong order, so the fraud answer can land before we’ve even heard that the payment exists. Somebody has to collect those answers in one place and show whether verification is finished. Real vocabulary: a lifecycle that ends, out-of-order events on day one.

But look at who decided what. The fraud system decided the score. The risk system decided the rating. The shop’s limit was checked by the service that owns limits. Our module matches the answers up by payment number, fills in the blanks as they arrive, and displays how far along we are. Everything we hold was decided somewhere else; nothing can be refused, and nothing new comes out of us.

It becomes a candidate when we’re asked to rule on the answers rather than collect them: fraud beats an automatic approval, and once the fraud score and the limit check are both in, the payment is settled one way or the other. Now we hold state, apply a rule the business will argue about, and produce our own event saying verification is complete. That’s the moment I described in Dealing with Race Conditions, where a projection had quietly turned into a workflow.

So they all fail for the same reason: events flow through the module, and the module decides nothing.

As a rule of thumb: someone is waiting for an answer; the answer can be no, and something happens on its own while they wait.

A decent pick

A customer wants to send back a pair of shoes.

They ask for a return; we check whether it’s still within the return window and whether we take it back at all. If it is, we send them a label and wait. The parcel may arrive next week, or never. When it arrives, warehouse staff scan it, inspect the item, and decide whether the shoes are in the state the customer claimed. Then the return is granted, and finance pays out, or it’s refused, and we ship the shoes back.

The events in a warehouse could look like that (pardon me if you’re a Warehousing expert; honestly, I’m not):

ReturnRequested
ReturnRefused
ReturnLabelSent
ParcelArrivedAtWarehouse
ItemInspected
ItemFoundWorn
ReturnGranted
ReturnAbandoned

We decide two things here: whether to accept the return at all, which we can refuse when the customer asks on day 45 of a 30-day window. And whether the shoes came back in the state they described, which we can refuse after inspection. The courier and the scanner tell us where the parcel is.

Then there’s ReturnAbandoned. It fires thirty days after the label was sent, when the parcel never showed up. Nobody triggers it manually. If we talk to warehouse managers, they will happily argue about whether thirty days is the right number. And hey, this is what a basic handling policy looks like.

And if we get any of it wrong for a day, support fixes it by hand, and the customer waits an extra day for their money.

The CRUD version is a returns table with a status column: requested, label_sent, received, granted. Each value overwrites the previous one, and so does updated_at.

Six months later, finance wants to know whether shortening the window to 30 days made customers send things back sooner or just made us refuse more of them. The table holds current statuses and one timestamp per row, so it can’t answer either half. If our support team finds that some customers got refunded twice and asks whether the same parcel was inspected twice, the table shows just one “granted” and never records the first inspection at all.

Returns also make the business case without hand-waving. With a CRUD version and a status column, we know how many returns are open. With Event Sourcing events, we know how many arrive after the window closed, how long parcels sit in the warehouse before inspection, how often an inspection overturns what the customer told us, and which products come back worn.

Someone in operations asks all four of those within a month, and a status column has already erased the answers. That’s the never lose data argument, small enough to fit in a pilot.

Returns aren’t the only feature like this; for instance, budget approval works the same way: someone submits a budget, the financial director approves or rejects it, and an approved one can no longer be changed. So does a reservation that we hold for a while and release when nobody confirms it.

The waiting is the part we want in a pilot. The courier answers when it answers, and the thirty-day clock runs on its own. The read model showing open returns will be behind the events for a while, and the same courier callback will arrive twice sooner or later. We have to handle both of those rather than read about them.

We also don’t want to learn everything at once. Inside one module we control the order in which our own events land, and we can read what we just wrote. Once returns starts talking to the finance module over a queue, we get out-of-order delivery, timestamps from machines whose clocks disagree, and the question of what to do when the payout confirmation arrives before we recorded that we asked for the payout. I wrote about handling that in Dealing with Race Conditions. It’s a fourth learning curve on top of the three we already have.

Don’t forget to audit the tooling

The pilot is also a chance to check the tooling. If we found a proper business concept that makes decisions, we should also check if we got strong consistency guarantees, for instance, ACID, optimistic concurrency. We can also check what options it gives for building read models, processing asynchronous subscriptions, etc. This will tell us a lot about how mature the tooling we selected is. And no, vibe-coded tools are not a good choice for critical infrastructure like an event store.

When I wrote Let’s build event store in one hour!, I listed what an event store has to do:

appending an event at the end of the stream,
reading all events from the stream,
a guarantee of ordering within the stream,
being able to read our writes,
strong consistency, atomic writes and optimistic concurrency.

And the second tier, good to have rather than mandatory:

subscribing to notifications about newly appended events, best if push-based,
global ordering,
built-in projections,
streams archiving.

There’s a growing number of vibe-coded libraries that look like event stores and skip parts of the first tier. Optimistic concurrency is the usual omission, since it’s the hardest to implement. That means: you’ll have to deal with that on your own. Without such features, you’ll be vulnerable to race conditions, data loss, and eventual inconsistencies.

So it’s worth running these against our pilot:

append to the same stream from two places at once and see whether one of them is rejected,
kill the process while a projection is being updated, and check what state we come back to,
rebuild a projection from scratch,
add a field to an event type that already has data stored,
write a test for a single decision, and see how much setup it needs.

The last one is about developer experience rather than correctness, and it’s still worth measuring, because that’s what the team does all day.

It’s also worth preferring an event store that runs on storage we already have. If we can put it on the Postgres our organisation already operates, nobody has to agree to run a new database for an experiment, and we skip the conversation with the platform team. A lot of pilots stop right there. It doesn’t get the library out of the checks above, though. Running on familiar storage says nothing about whether it handles concurrent appends.

Showcasing the pilot outcome

If we argue and try to persuade others during a meeting to do Event Sourcing, it’s not easy. We don’t have proof that this can work; we just have rumours and strong belief. We may immediately put ourselves in a defensive position, as others may fear the risk of change. And it’s fair to be afraid.

That’s why picking the scope is so critical. The pilot has to be small enough that we can do it slightly rogue. Not asking for a formal decision is reasonable when nobody is harmed by us spending a short time working on it.

Rogue means not asking permission, and being prepared to ask for forgiveness. We should still have justification for it and authority to make such a decision (read more in Business Won’t Let Me and other lies we tell to ourselves). It shouldn’t mean hiding the work from our team or writing something only we can maintain, because that destroys the persuasion we were going for. We also shouldn’t hide from ownership of our decision.

That’s why I recommend working on a feature with removability in mind. It helps if we fail, because then we at least minimise the consequences of a bad decision.

If we fail, we drop our pilot; if we succeed, we can showcase what we did. Showing a returns process that has been running for two months, next to the chart of how many parcels never arrive, and mentioning that we’re using Event Sourcing, sparks much more proactive discussion than an imaginary scenario. The same inertia that blocks new ideas starts working for us once the code exists, because it’s hard to throw away something that already works.

Once one feature works, the next steps get earned by evidence rather than enthusiasm. A second stream type when we find another process with real decisions. A separate read model store when the queries start to hurt. Cross-module messaging when two modules genuinely need to talk. Each of those should answer something we observed in production.

None of this is specific to Event Sourcing. Any pattern new to the team deserves the same treatment: a real but forgiving feature, one learning curve at a time, an exit we’ve thought about, and something to show at the end. There’s nothing magical about it, we just forget the pragmatism when we’re excited.

Cheers!

Oskar

Fractal Architecture, Cognitive Load, Vertical Slices and other terms that do(n't) fit your head

I wrote on LinkedIn that it seems that there are never enough articles on Vertical Slices. Each time I write a follow-up, there’s another follow-up needed. And guess what? After last week, I need another one!

Adam asked me in the comments:

How do you see Fractal Architecture fitting into this approach? To me, it makes sense to place shared code in the smallest logical grouping of slices that use it, keeping it close to its usage while isolated from the rest of the application.

And to give you a proper answer, I need to start by telling you about the Aussie guy called John Sweller.

In 1988, he coined an important term: Cognitive Load. He said:

“Our working memory is only able to hold a small amount of information at any one time and that instructional methods should avoid overloading it in order to maximise learning”

When I was younger, I believed that I could multitask; I also bragged that I had such a good memory that when I started managing my calendar and taking notes, I got old. And yeah, it seems that I’m old now.

When our industry evolved from the machines responsible for calculations and highly specialised machines and their operators, and became a big, valuable industry, we started to have issues with scaling. Not only technical scaling of distributed systems, but most importantly, scaling the work delivery.

Our teams grew bigger, and more and more industries started to benefit from the automation that computers brought. Developers slowly stopped working in a narrow vertical structure with business folks, but became jacks of all trades and masters of none. Or actually, some had to become masters of the one: technology.

The pendulum swung, and we landed in the ~~Helios~~ Technology-centric world:

And yes, I know, in theory, the entities are “Enterprise Business Rules”, which means that business logic should be at the heart of our application, but effectively what you see, and where the emphasis is, are the layers. Even if you measure the surface of those layers, those technical layers are big enough that we can lose sight of what we are here for. And we’re here to deliver business value through software.

Of course, for organisations it was easier to tell:

hey, you don’t need to care what you do, just deliver this nicely looking UI,
just take care of those distributed services integration, ensure they work, and you don’t need to care what they do,
hey, you’re the chosen one, smarty pants, right? Yeah, you’ll work on the domain.

It was justifiable to some degree, since all of that required specific technical skills that took time to master. It was also easier to just throw people who were supposed to be “good enough” to solve cases. Learning domains takes time. For me, it was usually about a year before I felt comfortable enough to understand the business. So people hoped that breaking down the work by technical aspects would cut the onboarding time.

But well, it ended up as something explained by a similar diagram:

Yup, collapsed like Atlantis,

Why did this fail? I wrote about it in:

In short? It skyrocketed the cognitive load. Our architecture should be optimised to keep together things that change together. That way, we can work within a certain scope without jumping from one place to another. And that’s what working with layered architecture looks like.

To add one web api endpoint, you need to change presentation, application, domain, and storage layers, each kept entirely separately just for the sake of subjective cleanliness. Also, if you had to understand what you have to do to complete a feature that touches multiple other features, you need to have a PhD.

In general, the investigations looked like wanting to check the weather on your phone, and then realising 2h later that you just bought a skateboard after going through the path of reminiscences brought by doom scrolling on social media. And you still don’t know if it’s going to rain or not.

So the pendulum swung again. We like to bounce back and forth; just like a pendulum, we can’t stop somewhere in the middle. Many people are now claiming that we don’t need any layers, that we should be vertically aligned. So we should cut our applications down to only the business features. We don’t need any grouping; what we care about is being autonomous and delivering features separately.

The initial idea was nice, something like “let’s get back to the roots and think differently, keeping the focus on business”. Yet, we got semantic diffusion of the original idea, and what was heard the most was: no layers, just vertical features.

I explained that in detail in My thoughts on Vertical Slices, CQRS, Semantic Diffusion and other fancy words.

Why is it an issue?

Have you seen a huge backlog in Jira? Just a huge to-do list filled with tasks for some features.

How easy is it to understand what actually has to be done?

How do you manage the priority?

How do you understand dependencies, and why are those features actually added?

Typically, that’s where the Jirology starts, and that’s why we hate it. But the issue originates somewhere else.

Cynically, or using Occam’s razor, the answer is simple. More or less how W. Edwards Deming framed it:

A bad system will beat a good person every time.

If our goal is the same: scaling the development process and its output, then no matter what we change internally, which architecture we choose, it’ll diffuse into what we optimise for.

So, as always, breaking the business capability into a smaller set of features was how many architects and CEOs distributed the work and made it parallel. Faster rarely means better. Unless we’re in the race car industry.

Also, for many software engineers, feature-based delivery was a way to simplify their lives. They could write a vague ADR, with just a few sentences describing the idea. It’s easy to write something that sounds good, but without enough detail, it’s hard to tell the implications and the exact delivery plan.

Now, when you have the vague plan accepted, you can slowly, feature by feature, deliver lower-quality, simplistic solutions, backing this up with the claim that

“Well, it’s done according to the specification we agreed on. We don’t have enough time to go again through this process. We need to deliver the feature.”

Sounds familiar? For me, too familiar. I have seen numerous cases where requirements drift one after another, slice by slice. And teams backing their next change with the previous one.

I’ve also seen numerous cases where teams felt really comfortable narrowly focusing on the exact features without even asking what the bigger picture is or why it was added. That led to, obviously, a maze, duplication of code and responsibility and lots of features that have had a hard time talking to each other.

Ok, so should our pendulum swing again, and should we get back to what, not my uncle, was praising?

Again, maybe the answer is to finally stop somewhere in the middle and focus on the outcome instead of the output? So, the business outcome and the software product.

Indeed, we should invest in decreasing the cognitive load and making our software easier to fit in our heads. We already learned that:

technical split is not great,
feature split is better but still far from great.

Some time ago, two people I respect a lot started talking about Fractal Architecture.

I’m afraid to write about it, not to start counting where it becomes semantically diffused. But that’s important.

Mark Seemann also wrote a great book, Code That Fits in Your Head. Check it out; it’s a simple book teaching on why code is important and design activity, and why we need to invest in fast feedback loops and simulations. Highly relevant nowadays. Much better than reading next-gen AI bros’ hot takes.

Nevertheless, the idea is that, indeed, if we want our software to do what it does, to deliver business value, we should focus on the business. Yet, to avoid ending up with the overfocus and feature factory (as we discussed), we should note that we could compose it. With the idea that big things are composed of small things.

Ok, enough philosophy; let’s get back to the original example from the previous article. We discussed the following e-commerce order features:

📁 orders
    📁 verifying-order
    📁 confirming-order
    📁 registering-order
    📁 pending-orders
    📁 order-storage // 👈 look ma, technical concept!

Now, if we added a shopping cart, then it could look like:

📁 shopping-carts
    📁 adding-product-item
    📁 removing-product-item
    📁 confirming-cart
    📁 clearing-cart
    📁 cart-storage // 👈 look ma, technical concept!

We could group that into the e-commerce module

📁 e-commerce
    📁 shopping-carts
        📁 adding-product-item
        📁 removing-product-item
        📁 confirming-cart
        📁 clearing-cart
        📁 cart-storage
    📁 orders
        📁 verifying-order
        📁 confirming-order
        📁 registering-order
        📁 pending-orders
        📁 order-storage

If we zip that into the e-commerce system, we’d see:

📁 e-commerce
    📁 shopping-carts
    📁 orders

We could continue to elevate that into, e.g. the whole ERP system with other modules like marketing, sales, etc. Each of them would have its own sub-modules.

📁 erp
    📁 e-commerce
    📁 marketing
    📁 sales

Looks familiar?

Yes, the layout is the same as the order module. What’s more, this can also go down, so:

📁 orders
    📁 verifying-order
        📁 anti-fraud-detection
            (...)
        📁 high-value-custome-verification
        📁 external-order-verification
    📁 (...)

It’s the same repeated pattern. Just like fractals, they’re chaotic in the sense that each fractal is different, just like each software product is different, but if we look carefully enough, then those chaotic patterns start to repeat themselves.

So why isn’t that a standard if it’s so great and simple?

Things get hard when you start thinking about cross-boundary splits and their dependencies.

We already discussed with the features factory that no-boundaries is not a solution. Even if we remove them all and make all things separate, they won’t magically become autonomous. They’ll still have dependencies, just hidden, which is the worst type of coupling you can have.

Let’s look again at this example

import { type OrderStorage } from '../order-storage;

// Dependencies
export type CheckDriver = (id: DriverId) => Promise<DriverStatus>;

export type VerifyOrderDependencies = {
    storage: Pick<OrderStorage, 'getOrder' | 'saveOrder'>;
    drivers: {       
        checkDriver: CheckDriver;
    }
}

// Feature
export const verifyOrderHandler = async (
  {
    orderStorage: {
      getOrder,
      saveOrder
    },
    drivers: {
      checkDriver
    }
    checkContractor,
    checkDriver,
  }: VerifyOrderDependencies,
  command: VerifyOrder,
): Promise<void> => {
  const order = await getOrder(command.orderId);

  const driver = checkDriver(order.driverId),

  await saveOrder(verifyOrder({ ...command, driver }, order));
};

We declare explicitly what our feature needs.

Why couldn’t we do the same for the whole module?

import { verifyOrderHandler, type CheckDriver } from ‘verifying-order’;

// Dependencies
export type OrdersDependencies = {
    database: {
       connectionString: string
    },
    drivers: {       
        checkDriver: CheckDriver;
    }
}

// Feature
export const orders = async (
  props: OrdersDependencies,
  command: VerifyOrder,
): Promise<void> => {
  const db = drizzle(props.database.connectionString);
  const storage = ordersStroage(db);

  const verifyOrder = (command) => verifyOrderHandler(
    {
        // DEPENDENCIES
        drivers: deps.drivers, // 👈 EXTERNAL
        storage // 👈 INTERNAL
    },
    command
  );

  return {
     verifyOrder, // 👈 EXPOSES
  }  
};

See, we define:

Dependencies: what do we need from the outside world, e.g. where our data is stored (connection string) and how drivers are checked (from the outside world). Then we can feed that to child components using both external dependencies (e.g. drivers) and other internal components (e.g. storage).
Capabilities: what do we expose to the outside world (e.g. order verification, which may be used by other modules, or a dedicated API component wrapping orders).

We could do the same at the higher level and the lower level. Just like Fractals, or like Hive (as nicely put by Julien Topçu & Thomas Pierrain).

Read also the article Onion, Hexagonal, Clean or Fractal Architecture? All of them, and more!! where Urs Enzler explains how he mixes all of that together.

And that’s also what the C4 model is doing: embracing different levels. Although it has a hard limit of 4, so it’s not a true fractal, which makes sense, since we should also be cautious about setting up paths that are too deep.

Of course, that doesn’t mean we need to go crazy and define all dependencies this way. If those are calls to inner module methods or some shared code, we can call it explicitly. Then we’re coupling those components together, but if it’s explicit and helps us tame complexity and make things more explicit, then why not?

Nevertheless, that’s why I don’t like jargon like “bounded context”, as it’s vague, but it’s presented as precise. I prefer to be “explicitly implicit” and just say that we’re composing the application from a component tree (or building a fractal).

Whether it’s a system, component, module, or slice, we can tackle that the same way. Of course, each will differ in responsibility, scope, etc., but as long as we have clear dependencies expressed as: here’s what we need, and a public API as: here is what you can get from us. We can freely compose our relationship.

All of that allows us to keep our cognitive load at a level where what we see fits into our heads, and focus on the business features, not technicalities. This helps us fight the forces we discussed today that push us to optimise for output rather than outcome.

Thoughts?

Cheers!

Oskar

Vertical slices, their ownership and external dependencies

A few questions about Vertical Slices come up again and again, and they’re good ones.

If a slice cuts through all the layers, do we get a table per slice?
If I’ve split the application into seven areas, are those seven bounded contexts, or seven slices of one? And does the answer change what they’re allowed to know about each other?
How can a “verify order” slice check what a “register order” slice wrote, if the two aren’t supposed to know about each other?
Where does the fetching code live when one screen needs data owned by another module, and the dependency rules block every place we could put it?
Should a UI component know about the API call a neighbouring component makes?

For me, these are all versions of one question: what does a slice do when it needs something from the outside world?

Calling every area of an application a bounded context sets the bar for separation as high as it can go, because contexts are genuinely meant to be autonomous. Once you’ve done that, connecting any two of them looks like a violation, and the other questions have no legal answer.

The assumption doing the damage rarely gets stated because it feels too obvious: a slice ought to be self-contained, so needing something from elsewhere means the cut was wrong. I held it myself for a while.

I’ve written before about how to slice the codebase effectively and shown a worked module, and I listed “you can’t share code between slices” as one of the myths that grew around the pattern. What I didn’t do was show the positive answer. Minimise isn’t zero, so what does the non-zero look like once you type it out?

Vocabulary is where this gets tangled, so let me start there.

Slice, module, context

Three words that get used interchangeably, and I think a lot of the pain comes from that.

A vertical slice is one piece of functionality, cut through the whole application. For me, a slice is more a function than an entity. “Verify a transport order” is a slice. It has a way in, some business logic, and whatever it reads and writes. If you’re thinking of it as a thing with a lifecycle, you’re probably thinking of an entity, which is a different concept that lives within the slice’s reach rather than being the slice.

A module is a logical grouping of slices. Orders is a module. It holds registering an order, verifying it, confirming it, and listing what’s pending. The grouping is a judgement call, and the criterion I use is what changes together.

A bounded context, in DDD terms, is a linguistic barrier. It’s a set of functionality that the business uses the same vocabulary for, distinct from other contexts. In a given context, a word always means the same thing. Across contexts, the same word can mean something else entirely.

A frontend and a backend aren’t two bounded contexts; they’re two deployment targets. You can have one deployment for multiple contexts, and a single bounded context with multiple deployments. If a frontend feature is part of the bounded context and has backing WebAPI, then the business should use the same words across both. Two product listings that differ in which fields they show aren’t two contexts either, since nothing about the vocabulary changes between them. Draw the line in either place, and you get the terminology without the boundary (or just a pure technical boundary), and then wonder why the “contexts” need constant coordination.

Seven features of one application are almost always slices, or at most modules, sitting inside a single context. That’s good news, because it means they were never obliged to be autonomous.

I’ll be honest: I don’t love the term “bounded context”. It’s imprecise in the original books and even more misleading in practice, because most people encounter it third-hand and take it to mean “a big folder we agreed to keep apart”. If the word causes arguments on your team, drop it and talk about which functionalities share a vocabulary. That usually works better.

Labels aside, what I do in practice is start from the functionalities we have to deliver, group them logically into components, and check that grouping against the UX map. Technological splits (a layer for data access, a layer for services, one for the frontend and one for the backend) have always ended badly in my projects. Business grouping usually held up longer.

Slices boundaries in practice

Let’s use the transport order. It gets registered, then verified, then confirmed, then settled. Verification means checking that the contractor and the driver actually exist in the carrier’s own system, which is an external API we don’t control.

It’s tempting to model this as one orders row with a status column that moves registered → verified → confirmed, updated through a generic endpoint. That’s how it usually starts, and the trouble begins there, because the endpoint that changes status also changes the pickup address and the contact phone. The system records that the order is verified. It doesn’t record that anybody verified it.

Greg Young made this argument better than I will in Task-Based UI: when the client posts data-centric structures back and forth, the domain has no verbs, and the user’s intent is lost on the way in. His point is that the client should tell the server to do something, so that the intent becomes the exact task for the backend expressed by the message we sent (rather than being inferred from a combination of fields mixed with the current state).

Naming the operation is also what gives you something to slice along. If every operation is “update the order”, you have one feature and nothing to divide. Once you have VerifyOrder, ConfirmOrder, RejectOrder, you have folders, and each folder is named the way the business names the operation. This holds for plain CRUD systems too. The name is the value, and the underlying implementation can be a single UPDATE.

So we get:

📁 orders
    📁 verifying-order
    📁 confirming-order
    📁 registering-order
    📁 pending-orders

Which leaves the question I started with. Verification needs the carrier’s system, and confirming needs to know what verification established. Where does that go?

The shape I use is a handler taking two arguments: its dependencies, then the message.

// orders/verifying-order/verifyOrder.ts

export type ContractorStatus = 'Active' | 'Unknown' | 'Suspended';
export type DriverStatus = 'Licensed' | 'Unknown' | 'Expired';

// Dependencies
export type CheckContractor = (id: ContractorId) => Promise<ContractorStatus>;
export type CheckDriver = (id: DriverId) => Promise<DriverStatus>;

// Handler
export const verifyOrderHandler = async (
  {
    getOrder,
    saveOrder,
    checkContractor,
    checkDriver,
  }: {
    getOrder: GetOrder;
    saveOrder: SaveOrder;
    checkContractor: CheckContractor;
    checkDriver: CheckDriver;
  },
  command: VerifyOrder,
): Promise<void> => {
  const order = await getOrder(command.orderId);

  const [contractor, driver] = await Promise.all([
    checkContractor(order.contractorId),
    checkDriver(order.driverId),
  ]);

  await saveOrder(verifyOrder({ ...command, contractor, driver }, order));
};

The first parameter is dependencies, and I think about it the same way I think about React props. The component declares what it expects; whoever renders it decides what to pass. Same here: the handler declares what it expects, and it has no opinion about where those functions come from: the same module, a different module, an HTTP client, or a stub.

The business logic underneath takes everything as data and returns a decision:

// orders/verifying-order/verifyOrder.ts

// Business logic
export const verifyOrder = (
  command: VerifyOrder & {
    contractor: ContractorStatus;
    driver: DriverStatus;
  },
  order: Order,
): Order => {
  if (order.status !== 'Registered')
    throw new Error(`Cannot verify an order in ${order.status} state`);

  if (command.contractor !== 'Active')
    throw new Error(`Contractor is ${command.contractor}`);

  if (command.driver !== 'Licensed')
    throw new Error(`Driver licence is ${command.driver}`);

  return {
    ...order,
    status: 'Verified',
    verifiedBy: command.verifiedBy,
    verifiedAt: command.now,
  };
};

Nothing here is fetched, so testing it means passing values in and checking what comes out. No mocks, no container, no database.

There’s no ITransportManagementSystem interface here with fifteen methods. Pragmatically, you can pull the whole thing in, and plenty of codebases do, but I’ve come to prefer narrowing it: two functions, three cases each, defined in this folder, in this slice’s language.

The carrier’s system almost certainly has a richer model of a contractor than the three cases, including credit terms, insurance validity, certificate expiry dates, and territorial permissions. Verification only needs to know whether this order can proceed. Declaring the narrow type keeps the slice in its own vocabulary rather than importing someone else’s, which is the bounded-context idea applied on a much smaller scale.

Duck typing and dependencies

Nothing in the codebase declares that it implements CheckContractor. There’s no implements clause anywhere. Any function of a compatible shape satisfies it. That means the type can live with the consumer, the slice that needs it, rather than with whoever ends up providing it.

So you can write the need, the handler, and the tests before anyone has decided who will serve it. Whether checkContractor becomes an HTTP call, a query against a table we replicate nightly, or a function returning 'Active' for the first three weeks stays open. When an external shape doesn’t match, you fit it or remap it at the point of supply, and neither side changes.

Golang bets its whole dependency story on this. Interfaces are satisfied implicitly, and the convention is that the consumer declares the interface it needs, sized to its use. Hence the proverb about the bigger interface being the weaker abstraction. io.Reader is one method, and half the standard library composes through it.

Structural typing gives us that in TypeScript, and I think it’s underused by people arriving from C# and Java, where the habit is to define a nominal interface and hand it around.

So where do the dependencies come from?

Somewhere near the entry point, you create the real things once, build the small functions the handlers asked for, and pass them in:

// apps/api/composition.ts
const db = drizzle(process.env.DATABASE_URL!);
const carrier = carrierApiClient(process.env.CARRIER_API_URL!);

const contractorStatus =
  (client: CarrierApiClient): CheckContractor =>
  async (id) => {
    const contractor = await client.getContractor(id);

    if (!contractor) return 'Unknown';

    return contractor.suspendedAt ? 'Suspended' : 'Active';
  };

export const orders = {
  getOrder: getOrderFrom(db),
  saveOrder: saveOrderTo(db),
  checkContractor: contractorStatus(carrier),
  checkDriver: driverStatus(carrier),
};

That mapping function is where two vocabularies meet, and I like having exactly one place where that happens. Everything the carrier knows about a contractor collapses into three cases, in a file you can read in ten seconds.

The routes are then thin:

// apps/api/routes.ts
app.post('/orders/:id/verification', async (req, res) => {
  await verifyOrderHandler(orders, {
    orderId: req.params.id,
    verifiedBy: req.user.id,
    now: new Date(),
  });

  res.status(204).end();
});

That’s the mechanism: partial application, done by hand, in a file whose job is to know about everything so that nothing else has to. No container, no registration, no lifetime scopes. In a monorepo, this file lives in apps/, and packages/ holds slices that declare but never resolve. If you’re using Nx, this is the composition point its boundary rules exist to protect. In other environments, where Dependency Injection Containers are out of the box, you can still use the same way; in .NET, you can use the same pattern by injecting dependencies explicitly even to methods.

The tests use the same shape:

const contractor = (status: ContractorStatus): CheckContractor => () =>
  Promise.resolve(status);

test('refuses to verify an order for a suspended contractor', async () => {
  const store = inMemoryOrderStore([registeredOrder]);

  await expect(
    verifyOrderHandler(
      { ...store, ...carrierStubs, checkContractor: contractor('Suspended') },
      verifyCommand,
    ),
  ).rejects.toThrow('Contractor is Suspended');
});

There’s no mocking framework here, because there’s nothing to mock. You can pass an in-memory implementation, a stub, or a real client pointed at a sandbox. The handler can’t tell the difference.

Everything outside the slice is external

From inside a slice, there’s one category of thing: external. Another slice next door, the parent module, a different module, a third-party API, the database. All the same. The slice states a function type and doesn’t ask where the implementation comes from.

That keeps the slice movable. Because it defines its own dependencies, changing the logical grouping later is a matter of what you inject, not a rewrite. If verification turns out to belong in its own module, or gets extracted into a service, the handler stays as it is. Testing in isolation comes out of the same property, as the test above shows, and I get it as a side effect rather than designing for it.

None of this is about hiding the coupling. For me, independence isn’t a value in itself. I’d rather know the connections I need to have and be able to look at the code to see what it depends on and what it does. Hidden dependencies are still there, only harder to find. What I’m optimising for is cohesion, and explicit dependencies serve that.

What about the module’s public API?

If you’ve read my Architecture Weekly piece on VSA, you’ll have seen me recommend an api.ts per module, exposing what’s public and hiding the rest. That may seem to contradict what I’ve just described.

They point in opposite directions across the same boundary. api.ts is what a module offers: the surface it’s willing to support and that the module owns. CheckContractor is what a slice asks for: a need, owned by the consumer, expressed in the consumer’s terms.

You want both, because they protect against different things. Without the module API, everything is reachable, and any internal change can break a caller you didn’t know you had. Without consumer-declared needs, the consumer is coupled to the shape of whatever the provider decided to expose, including the parts it never calls.

The composition root is where the two meet. It takes what the module offers and adapts it to what the slice asked for. That adapter is a few lines, it lives in one place, and it’s the only code that knows both vocabularies.

Cycles stop being a problem. Orders needs contractor standing. Pricing needs order history to work out volume discounts for the same contractor. That’s a genuine mutual need, and if both modules import each other, you have a cycle. The compiler may tolerate it. Your boundary rules probably won’t. This is the situation where the tooling forbids two modules at the same layer from referencing each other, and it’s easy to conclude from that the design must be wrong.

The usual escape is to extract the shared parts into a common module. That works twice; then the common module becomes the place everything ambiguous lands, and changing it means changing everything.

But there’s no cycle if neither module imports the other. Orders declares CheckContractor. Pricing declares its own:

// pricing/volume-discounts/orderHistory.ts
export type CompletedOrder = Readonly<{
  completedOn: Date;
  netValue: Money;
}>;

export type GetCompletedOrders = (
  contractorId: ContractorId,
  since: Date,
) => Promise<CompletedOrder[]>;

Two fields, because that’s what a volume discount calculation uses. The composition root supplies a function that queries orders and maps the result down to that shape.

Most module-level cycles I’ve run into were a shared concept whose owner hadn’t been decided yet. Declaring narrow needs lets you defer that decision rather than resolve it early and wrong. If it later turns out that contractor standing genuinely belongs in one place with a single definition, you’ll know more by then, and moving it is cheap because only the adapters point to it.

If your dependency rules forbid the arrangement your domain wants, it’s worth checking whether they’re describing your design or your import graph. Rules that block a module from reaching another module are useful. Rules that block it because two files sit at the same “layer” are enforcing a layering you may have already outgrown.

Two entry points, one feature

What if the same operation can be triggered two ways?

An order gets confirmed by a dispatcher through the UI. It also gets confirmed automatically when the carrier’s system reports the assignment accepted. Same rule, different trigger, different surrounding logic.

Both files go in confirming-order/, side by side. I showed this pattern in Vertical Slices in practice using an API endpoint and an external booking event that share a single command handler. With dependencies declared per handler, the event-triggered one looks like this:

// orders/confirming-order/confirmOrderOnAssignmentAccepted.ts

export const confirmOrderOnAssignmentAccepted = async (
  { getOrder, saveOrder }: { getOrder: GetOrder; saveOrder: SaveOrder },
  event: AssignmentAccepted,
): Promise<void> => {
  const order = await getOrder(event.orderId);

  await saveOrder(
    confirmOrder(
      {
        orderId: event.orderId,
        confirmedBy: SystemUser,
        now: event.acceptedAt,
      },
      order,
    ),
  );
};

The two handlers don’t have the same dependencies. This one has no carrier check at all, since the event came from the carrier and we already have the answer. The dispatcher-triggered version loads more, because it starts with less. The business logic is shared; the application logic isn’t, and each entry point declares what it actually uses.

The same applies to what you reuse across slices. I’d reuse a policy or a calculator, say a pure function like settlementFor(order, tariff) used by both confirmation and settlement, before I’d reuse a whole handler. The pure function is a function of its arguments and cheap to share. Handlers differ in what they load, check and record, and that’s the part that tends to change.

Vertical Slices and Database

Back to the table per slice. Three separate things get conflated in that question.

Business logic goes per entity or aggregate. The rules about what states an order can be in and which transitions are legal belong to the order. One place, and every slice that decides about an order goes through it. Slices don’t each get a private notion of what an order is.

Read models go per query. This is where a table per feature is right. The dispatcher’s pending-orders board needs order number, contractor, pickup window and verification state, sorted by pickup time. Build a table that answers that. The settlement report needs something else and gets its own. Resist making a single query serve five screens by expanding columns.

Database schemas go per module. One for orders, one for pricing. A slice is a feature, not a persistence boundary. A schema per slice gives you migrations that correspond to nothing, and joins across five schemas to draw a single screen.

You can do all of this with one status column and an ORM. If you want the four distinct operations that currently collapse into status = 'cancelled' to stay distinguishable, appending them as facts is what Event Sourcing offers, and it composes well with this. Nothing above depends on it.

Vertical Slices and Frontend

Don’t force a 1:1 mapping between the UI and the backend. One screen is routinely composed of data gathered from several modules, and one user action can trigger operations in several. That mismatch is normal. GraphQL exists partly as an attempt to solve it, whatever you make of that as a solution.

A dispatcher board showing pending orders, contractor status and a confirm button touches three slices across two modules. Two reasonable arrangements:

Compose at the page. The page fetches from several endpoints and arranges the results, holding no business logic of its own. Components receive what they need as props and know nothing about where they came from, for the same reason that handlers take dependencies as arguments. A component that takes props is easy to move and easy to feed with a different implementation.

Or write a backend-for-frontend. If the screen needs it in one request, make a slice whose job is to serve that screen, depending on the others and stitching them together. It’s still a slice; it’s named after a screen because that’s honestly what it is.

Which one depends on how much the round-trip costs you and how stable the screen is. Both beat growing one endpoint until it serves every screen you have.

The backend usually knows which operations are currently available for a given order, and the frontend is often re-deriving that from status fields. Returning the available actions with the data keeps that decision in one place. That’s the part of HATEOAS I find useful, without the rest of the ceremony.

Splitting the frontend by kind of thing tends to look suspicious to me. Two listings that differ only in which fields they show are usually one feature with a parameter. Splitting by market or country is more often real, because it genuinely is a different application with different rules. It’s hard to judge from a description; what I’d do is map the functionalities first and see which ones actually change together.

TLDR

The question I started with was what a slice does when it needs something it doesn’t own. My answer is that it declares the need and leaves it to something else to decide where it comes from.

Which comes down to:

Everything outside the slice is external, whether it’s the next folder or another system.
Declare narrow function types where they’re used, in your own vocabulary, rather than importing a wide interface.
Compose by passing functions in, in one file, near the entry point.
A module’s public API and a slice’s declared need are different directions of the same boundary, and the composition root adapts between them.
Reuse policies and calculators; let handlers duplicate.
Business logic per entity, read model per query, schema per module.
Don’t expect the frontend to mirror the backend.

None of this needs a framework, a container or a particular database. It’s more about a convention applied consistently.

And your first grouping will still be wrong somewhere. Mine usually is. That’s fine, as long as being wrong stays cheap, which is the argument for removability over maintainability and for keeping the couplings visible rather than tucked away. A slice that states its dependencies is one you can move.

And the cherry on top: this approach helps LLM agents too. Everything a feature needs sits in one folder, with a handful of function types crossing the boundary. To change how verification works, you open verifying-order. The command, the rules, both entry points and the tests are in it.

That was always the argument for grouping changes together: it reduces how much a person has to hold in their head. The same property determines how much can fit in a context window and how much of a codebase an agent has to read before it can safely change one behaviour. A layered structure that needs five folders touched for one feature costs an agent the same way it costs us, faster.

So it’s an old argument that happens to have got more valuable.

Cheers!

Oskar

p.s. If you’re dealing with such issues, I’m happy to help you through consulting, training or mentoring. Contact me and we’ll find a way to unblock you!

Fixing bugs in Event Sourcing is hard, for real?

Every system ends up with bad data. Some of it comes from integrations, some from users doing things nobody predicted, and a good part of it comes from us, because we shipped a change that looked fine in review.

This is about what we can do on Tuesday morning, when someone from support writes: “the price on this reservation looks wrong”.

The bug

Imagine we’re working on a hotel reservations system. Pricing there is fiddly: rate plans get revised, extras are added after booking, and the city sets the tourist tax on its own schedule.

Let’s say that the tourist tax used to be a single value in configuration, 4.50 per person per night, multiplied by nights and guests. That was enough for years, because the rate never moved.

Then the city announced an increase to 7.00 from 1 April. It appeared that one value can’t express that: a stay from 30 March to 2 April has nights on both sides of the boundary. So we added rates with validity periods and a lookup to resolve them, and shipped it on 12 March.

The lookup resolves the rate once per reservation, from the check-out date, and applies it to every night. For a stay inside a single period that gives the right answer, which is every case the tests covered. For a stay crossing 1 April, it charges 7.00 for the March nights too. Crazy, but well, that’s how life is sometimes.

A rate change has people checking numbers they’d normally trust, and on 19 March a front office manager reported that the projected tax for early-April arrivals doesn’t match what she gets counting by hand. We confirmed it and deployed a fix at 16:02 the same day.

Then the actual work starts. Which reservations are wrong, by how much, and what do we do about them?

What we have, with state

Take the reservation support asked about. Seven nights, two guests, 27 March to 3 April, booked on 14 March, breakfast added two days later. Here’s the row:

| id    | check_in   | check_out  | rate_plan_id | total_amount | updated_at       |
|-------|------------|------------|--------------|--------------|------------------|
| 8f21  | 2026-03-27 | 2026-04-03 | RP-STD-2026  | 1358.00      | 2026-03-16 14:02 |

The total is wrong. What should it be? The row doesn’t say how the number was produced: no breakdown of rooms, extras and tax, no record of which nights got which tax rate, and no version on rate_plan_id, so we can’t tell which prices were in force on 14 March. updated_at says 16 March, which is the breakfast, not the booking.

So we recompute and write the migration:

It matches 1,240 rows. We run it at 17:30 on 19 March.

What that migration does

Around 900 of those rows genuinely came from the broken build. The other 340 were booked before 12 March, had correct totals, and were caught because someone changed a room or added a note during the window. updated_at moves for all of that, and there’s nothing in the row that separates “this changed” from “this changed because of the bug”. We recalculate them along with the rest.

recalculate_total reads the rate plan as it stands today, and the plans were revised on 15 March for the season. RP-STD-2026 went from 180.00 a night to 205.00. For reservation 8f21 that means:

|         | before      | after migration   | correct  |
|---------|-------------|-------------------|----------|
| rooms   | 1260.00     | 1435.00           | 1260.00  |
| extras  | 168.00      | 168.00            | 168.00   |
| cityTax | 98.00       | 73.00             | 73.00    |
| total   | 1526.00     | 1676.00           | 1501.00  |

The migration fixed the tax and repriced the rooms. It did that to every reservation booked before 15 March, roughly 400 of them, and each guest gets a confirmation with a total higher than the one they agreed to.

And then migration number two

Complaints reach us around 2 April, and we work out what happened. Now we need to put the room prices back. To do that, we need two things per reservation: the date it was booked, so we know which rate plan version applied, and what the total was before we touched it.

We have neither. updated_at reads 2026-03-19 17:30 for all 1,240 rows we updated, so it no longer distinguishes the ones we broke from the ones we fixed from the ones we shouldn’t have touched. total_amount holds the value our migration wrote. The booking date was never stored separately, and the only thing that used to approximate it was updated_at.

So migration two is written against data that migration one produced. We reconstruct booking dates from confirmation emails where we still have them, guess at the rest, and pick a WHERE clause that we hope covers the right subset. If we get that wrong, the input to migration three is the output of migration two. I’ve been on both ends of this, and it doesn’t get more pleasant.

We can prepare for this in a state-based system, of course. Keep the breakdown in a JSON column, store the rate plan version on the reservation, add a price_history table, add a corrected_by marker. I’ve done all of those. The catch is that we add each one after the incident that taught us to, on the table where that incident happened, and today’s incident is on a different table. What we’re building at that point is a partial event log, decided per column, under time pressure.

The same bug, event-sourced

Here’s the stream for the same reservation:

| when         | event                        | data                                                       |
|--------------|------------------------------|------------------------------------------------------------|
| 14 Mar 09:40 | `ReservationInitiated`       | 27 Mar – 3 Apr, DBL, 2 guests                              |
| 14 Mar 09:41 | `RatePlanApplied`            | RP-STD-2026 v4, 180.00 / night                             |
| 14 Mar 09:41 | `ReservationPriceCalculated` | rooms 1260.00, cityTax 98.00, total 1358.00                |
| 14 Mar 09:41 | `ReservationConfirmed`       |                                                            |
| 16 Mar 14:02 | `ExtraAdded`                 | breakfast, 2 guests, 168.00                                |
| 16 Mar 14:02 | `ReservationPriceCalculated` | rooms 1260.00, extras 168.00, cityTax 98.00, total 1526.00 |

The corrupted value is cityTax, and it’s sitting there in plain sight. With two guests, a March night costs 9.00 in tax and an April night 14.00, so five March nights and two April nights come to 45.00 + 28.00 = 73.00. The event says 98.00, which is all seven nights at 14.00. Everything else in that event is right, and RatePlanApplied still says the booking was made at 180.00 a night under version 4, so we can calculate the correct total without reconstructing anything:

| What    | recorded | correct |
|---------|----------|---------|
| cityTax | 98.00    | 73.00   |
| total   | 1526.00  | 1501.00 |

That’s the difference from the table. The migration above went wrong because the inputs to the original calculation were gone, and here they aren’t. The 14 March event still says what it said on 14 March, whatever was appended later.

Knowing which events to look at

The other half is metadata. Here’s the full event:

{
  "type": "ReservationPriceCalculated",
  "data": {
    "reservationId": "8f21",
    "nights": [
      { "date": "2026-03-27", "room": 180.00, "cityTax": 14.00 },
      { "date": "2026-03-28", "room": 180.00, "cityTax": 14.00 },
      "...",
      { "date": "2026-04-02", "room": 180.00, "cityTax": 14.00 }
    ],
    "cityTax": 98.00,
    "total": 1358.00
  },
  "metadata": {
    "timestamp": "2026-03-14T09:41:12Z",
    "correlationId": "b31e...",
    "causationId": "a07c...",
    "userId": "booking-engine",
    "buildSha": "a4f9c2e",
    "schemaVersion": 3
  }
}

There’s one interesting metadata: buildSha. What is it? It’s the commit the service was running when it appended the event. It’s an environment variable and a few lines in whatever builds our metadata. On 19 March, it lets us query by the thing we actually care about:

SELECT stream_id, data, metadata
FROM events
WHERE type = 'ReservationPriceCalculated'
  AND metadata->>'buildSha' = 'a4f9c2e';

That returns 900 events, and they’re the 900 the broken binary produced. The 340 reservations that were merely touched during the same week don’t appear, because their price events came from a different commit.

The same metadata helps with the diagnosis. correlationId groups everything that came from one user action, and causationId chains back to the command that triggered this event. When the bad value comes out of a handler three hops from the request, that chain is how we find which request it was.

Without the SHA, we’re still ahead, because the position and timestamp of an appended event don’t move once it’s written. Still, I recommend to add the sha.

Fix forward, don’t rewrite

The instinct is to go into the store and edit those events. I’d steer you away from that. Events are immutable, and, surprising as it sounds, having precise history, bugs included, is a valid scenario. It’s often the only way to see later what really went wrong. I put it this way in the migration section of Simple patterns for events schema versioning: “you should not change the past… even including bugs, is a valid scenario.”

Instead, we could append a corrective event:

type ReservationPriceCorrected = {
  type: 'ReservationPriceCorrected';
  data: {
    reservationId: string;
    previousTotal: number;
    correctedTotal: number;
    reason: string;
  };
};

It’s the accountant’s move: we don’t rub out a ledger line; we post a correcting entry next to it. Then we rebuild the read models so the downstream views show the corrected total, and fix the bug so it stops producing new ones.

There’s a side effect that’s easy to miss. The correction is an event, so the next person to open this stream sees that a bug happened, when it was corrected and why. Nobody has to remember, and if we surface it in the UI, the front desk sees it too.

If we get the correction wrong too

We might. Say our correction script has its own bug and writes 1501.00 as the room total rather than the reservation total.

The recovery is the same operation as before, because the 14 March events are still there. We read the original calculation, the rate plan version that applied, our correction and its buildSha, and we append a second correction computed from the same inputs we had the first time. Nothing we did on 19 March destroyed the material we need on 2 April.

That’s the loop from the state-based version broken. Each attempt adds a fact rather than replacing the evidence, so attempt three is calculated from the booking, not from attempt two.

The customer already fixed it

Another reservation, four nights, two guests, 29 March to 2 April, at 210.00 a night. Rooms 840.00, tax charged 56.00, correct tax 41.00.

| when         | what                            |                                            |
|--------------|---------------------------------|--------------------------------------------|
| 14 Mar 09:41 | `ReservationPriceCalculated`    | cityTax 56.00, total 896.00, build a4f9c2e |
| 15 Mar 11:20 | guest calls the hotel           | "that's not what I was quoted"             |
| 15 Mar 11:26 | `ReservationPriceCorrected`     | total 839.00, by anna.k                    |
| 17 Mar 08:03 | `ReservationConfirmationResent` |                                            |
| 19 Mar 16:02 | we deploy the fix               |                                            |
| 19 Mar 17:30 | we run the bulk correction      |                                            |

Anna’s reason for that correction reads: city tax recalculated to 41.00, 5% goodwill on rooms agreed with FO manager. Our recalculation gives 881.00, and it’s correct by the pricing rules. Her 839.00 is also correct, because she took 42.00 off the rooms to settle an argument she was having on the phone.

If our bulk correction is a blind recalculation, at 17:30 it sets the reservation back to 881.00 and undoes her. The guest gets a third number in a third email and calls again. Support opens the reservation, sees the current total, and has no idea why it changed or that it was us. Anna’s note now describes a value that no longer exists.

With the stream, logically the check could look like in pseudo code (hey TS, no pun intended):

const needsCorrection = (events: ReservationEvent[]) => {
  const bad = events.findIndex(
    (e) =>
      e.type === 'ReservationPriceCalculated' &&
      e.metadata.buildSha === BROKEN_BUILD,
  );

  if (bad < 0) return false;

  // someone already dealt with it, leave it alone
  return !events
    .slice(bad + 1)
    .some((e) => e.type === 'ReservationPriceCorrected');
};

Everything it skips goes on a list for a person to look at. If we’d rather model the goodwill as its own DiscountGranted event, the rule becomes “skip if any price-affecting event follows the bad one”, which is the safer version anyway.

We can write that check because the events are there to check. In the mutable model, a bug and a deliberate correction look identical: a number in a column. There’s nothing to branch on, unless we can reconstruct intent from an audit table, assuming the support tool writes to one.

We could do such a check and fix as part of the one-time async job; we could even generalise it and run it as the SQL migration, running it only if it wasn’t run. We could optimise it by loading multiple reservations at once, then doing bulk append, if our store allows that, but the logical work remains the same. The rest is mostly performance optimisation.

Make the correction a feature

Better still, treat the corrective operation as a requirement and build it before we need it.

Ask the business “how do you fix this today?”. There’s almost always an existing answer: a form, a manager’s approval, a note in the folder. Model that. CorrectReservationPrice as a command, with permissions, validation and a mandatory reason, producing ReservationPriceCorrected.

Our bulk fix then goes through that command, with the same rules and the same events as when Anna runs it from the front desk. Nobody connects to the production database at 23:00.

And when the business says something can never happen, the follow-up is still: fine, how often?

There’s a harder version of this. I once watched a hotel checkout get stuck on a blocked financial account, which then jammed the entire night audit, with no way out except a migration or a hotfix. That’s the worst place to end up, telling a customer they can’t operate until we ship code. Bugs, bad data and stuck processes aren’t edge cases we can design away, which is the argument in No, it can never happen!: “We cannot assume that we won’t have a bug, or our hardware or network won’t have any random failure.” And if there’s no way to correct something inside the application, someone will do it with an UPDATE at 23:00. Read the horror story described there.

And check What texting your Ex has to do with Event-Driven Design.

“But now the history contains a bug”

Yes. The wrong price was real. It was shown to the guest, it went into a confirmation, it may have reached the invoice and the tax report. A log that shows only the corrected price can’t explain why the guest called.

If we need a clean stream later, for a regulator or a migration, still don’t edit in place. Copy and transform into a new stream, and keep the original.

None of this stops us shipping the bug, makes repricing 900 reservations free, or spares us the conversations with guests whose total changed twice.

What we get is better material to work with. We can name the events the broken build produced. We can read the value it wrote, with the inputs beside it. And we can see which reservations someone has already handled, and what they agreed to.

In a state-based system, fixing data overwrites the evidence we’d need to check whether the fix worked, which is how one migration ends up repairing the last one. Keeping the record of the mistake is what lets us check our own work afterwards.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

Throw, Result, or neither?

There is one question I keep getting about my event-sourced code: why don’t I use Result?

It usually comes after someone sees a decision throwing an exception. They would put the failure in a Result, leave try/catch at the application boundary, and make the distinction explicit in the return type. That is a fair question, but the return type is only part of the answer.

I don’t mind using exceptions. I also return business failures as events. Which one I choose depends on what the failure means and what the application needs to do with it. What’s more, I don’t mind Result either, as long as I’m using a language where it’s a native and idiomatic way to use it. Stil…

Some failures are worth retaining as business data. An out-of-stock request can contribute to an unmet-demand report. A declined payment can start another process or be grouped by its reason. An item limit can explain why customers abandon an operation.

Gojko Adzic calls the systematic use of unexpected or marginal usage patterns to improve a product lizard optimization. Repeated out-of-stock requests can reveal demand we do not serve or a user experience that encourages impossible quantities. Turning every attempt into only a response or an exception log makes that pattern harder to find. An event gives the application data it can query, project, and use in another process.

Event Sourcing can help take advantage of unwanted and unexpected failures. We can model negative outcomes as events, just like regular ones. Our business logic informs us of what happened, not whether it’s a success or an error. It’s just an outcome of our decision.

So essentially we’re getting three options to deal with an unexpected scenario.

throw an error (e.g. OutOfStockError) and catch it at the boundary;
return a Result with the failure on its error track;
return an event (e.g. ProductItemOutOfStock) and decide separately what we do with it.

Having that, my answer is rarely “just throw” or “just return Result.” Before choosing either, I need to know what should be persisted, what the caller should receive, and whether the failure belongs in the application’s exception path.

An error as a business fact

Let’s say that the customer asked for a quantity we cannot supply. Recording that outcome can be useful even though the cart did not change. The cart may also reach its item limit or fail payment authorisation. These are expected business outcomes, and the decision can describe each one with a distinct event:

const addProductItem = (
  command: AddProductItem,
  state: ShoppingCart,
): ProductItemAdded | ProductItemOutOfStock | ShoppingCartItemLimitReached => {
  if (state.status === "Confirmed")
    throw new IllegalStateError(
      "Cannot add a product to a confirmed shopping cart",
    );

  const { shoppingCartId, productItem, availableQuantity, maximumItems, now } =
    command.data;

  if (productItem.quantity > availableQuantity)
    return {
      type: "ProductItemOutOfStock",
      data: {
        shoppingCartId,
        productId: productItem.productId,
        requestedQuantity: productItem.quantity,
        availableQuantity,
        attemptedAt: now,
      },
    };

  if (state.productItems.length >= maximumItems)
    return {
      type: "ShoppingCartItemLimitReached",
      data: {
        maximumItems,
        requestedProductId: productItem.productId,
      },
    };

  return {
    type: "ProductItemAdded",
    data: { shoppingCartId, productItem },
  };
};

ProductItemOutOfStock carries the requested and available quantities. ShoppingCartItemLimitReached carries the configured limit and the product that crossed it.

We could also model a single ProductItemAddingFailed event instead. It could have a reason string, but then we’d use the granularity and precision of the information. With distinct event types, a projection, workflow, endpoint, or test can check the relevant outcome directly.

What’s more, we could also add a FailedToAddProductToClosedShoppingCart and get rid of throwing entirely. We’ll get to that, but for now…

Broken rules can throw

Not every failure needs a business event. A confirmed cart can no longer be modified, so AddProductItem is not a valid operation for that state. A command without required data is invalid as well. An unavailable event store, a stock-service timeout, or a serialisation error means the operation could not be completed reliably. I use exceptions for these cases.

The throw happens before an event is returned, so the handler has nothing to append. The exception retains its stack and reaches the application’s error boundary.

Some people prefer to use Result and hope for deterministic decision-making. Still, in most popular dev environments (TypeScript, Java, C#, Python, etc.), we can get exceptions. Even in strongly-typed Rust, we can get a panic or other cases. In the presented code loadShoppingCart can reject because of data unavailability, appendToStream can report a concurrency or connection error, and base code library code can throw on a random, stupid null pointer. The application still needs an error boundary somewhere:

try {
  const result = await handleAddProductItem(command);
  return mapResultToResponse(result);
} catch (error) {
  return mapToErrorResponse(error);
}

That boundary might be inside WebAPI route, a message handler callback, or wrapped with a conventional error middleware. The remaining choice is whether expected business outcomes should use the same path.

What `Result` would add

Result puts the success value and the expected failures on separate branches, visible in the type like:

type Result<Success, Failure> =
  | { success: true; value: Success }
  | { success: false; error: Failure };

The cart decision becomes:

type AddProductItemFailure =
  | ProductItemOutOfStock
  | ShoppingCartItemLimitReached;

const addProductItem = (
  command: AddProductItem,
  state: ShoppingCart,
): Result<ProductItemAdded, AddProductItemFailure> => {
  const { shoppingCartId, productItem, availableQuantity, maximumItems, now } =
    command.data;

  if (productItem.quantity > availableQuantity)
    return {
      success: false,
      error: {
        type: "ProductItemOutOfStock",
        data: {
          shoppingCartId,
          productId: productItem.productId,
          requestedQuantity: productItem.quantity,
          availableQuantity,
          attemptedAt: now,
        },
      },
    };

  if (state.productItems.length >= maximumItems)
    return {
      success: false,
      error: {
        type: "ShoppingCartItemLimitReached",
        data: {
          maximumItems,
          requestedProductId: productItem.productId,
        },
      },
    };

  return {
    success: true,
    value: {
      type: "ProductItemAdded",
      data: { shoppingCartId, productItem },
    },
  };
};

Technically it looks more or less the same: ProductItemAdded is returned on the success branch, while the other two errors are returned on the failure branch. The caller checks the branch before returning the event. In this application, adding the product changes the cart and produces a 204 response. Running out of stock or exceeding the limit leaves the cart unchanged and produces a 409 response. The handler appends only the success value:

const handleAddProductItem = async (
  command: AddProductItem,
): Promise<Result<ProductItemAdded, AddProductItemFailure>> => {
  const state = await loadShoppingCart(command.data.shoppingCartId);
  const result = addProductItem(command, state);

  if (result.success === false)
    return PreconditionFailed();

  await eventStore.appendToStream(
    shoppingCartStreamName(command.data.shoppingCartId),
    [result.value],
  );

  return NoContent();
};

If we used event, it could look like that:

const handleAddProductItem = async (
  command: AddProductItem,
): Promise<Result<ProductItemAdded, AddProductItemFailure>> => {
  const state = await loadShoppingCart(command.data.shoppingCartId);
  const event = addProductItem(command, state);

  if (event.type === "ProductItemOutOfStock" || event.type === "ShoppingCartItemLimitReached")
    return PreconditionFailed();

  await eventStore.appendToStream(
    shoppingCartStreamName(command.data.shoppingCartId),
    [result.value],
  );

  return NoContent();
};

Both versions preserve the same three event types. Result adds a success-or-failure classification around them. Which can be fine, but if you look again at the sample, it’s still pretty easy to ignore the different consequences of the events. We’re just one check on success further from it.

I saw many codebases where people were just blindly muting all errors by default, getting no benefit from it, but just polluting the codebase. Also keeping in mind that they still need to wrap the codebase with try/catch and have a conventional mapping layer.

We can also see that the Result wrapper in our case doesn’t add much besides success/error classification. If we’d like to return different response status based on the failure scenario, we’d need to add the same switch. So we’re not removing the decision; we’re just making it easier to ignore.

Of course, many languages made it streamlined by adding pipe operator etc. Yet, again, many popular environments don’t have it. TypeScript has no pipe operator; the TC39 proposal is not (yet?) part of the language. Repeating the branch adds checks throughout the call chain; introducing a helper library adds vocabulary that is not native to the language. Those libraries are extremely noisy and require tribal knowledge and onboarding.

Scott Wlaschin, who popularised railway-oriented programming, makes the same qualification: Result models expected alternatives, but it is not intended to wrap every function or replace exceptions.

Same for events; we should always discuss with the business if this information is important for them and if they want to keep it. We should also ask what should happen and if we can and should recover in our business application when this scenario happens.

ProductItemAdded, ProductItemOutOfStock, and ShoppingCartItemLimitReached just tell what has happened. Domain doesn’t need to care how the application handles it. It can mean failure for the current request and still be worth recording because a projection or workflow consumes it. Adding those rules to Result would move persistence concerns into the domain decision and add premature classification. When we change the behaviour in the application and record one of those scenarios, we would need to change domain code, even if business rules are the same. Just to formally move one case from failure to success.

Adding Result changes every calling layer without replacing the persistence rules, batch handling, or exception boundary. I do not see much of a gain here. Of course, a codebase that already uses Result as its error channel may judge it differently, especially when the alternatives are not events. Thus…

Returning an event doesn’t mean persisting it

ProductItemOutOfStock might be needed after the request. Measuring unmet demand, suggesting alternatives, or showing failed attempts to support staff requires a durable event. Still, if our business doesn’t care about the fact and we just want to log it, or forward to the HTTP Response, then we may not need to record it. Appending it would retain data the application does not need. Omitting it from the decision would instead force the endpoint to reread the outcome.

For this example, the decision returns ProductItemOutOfStock to the caller and the stream remains unchanged. I call this selective persistence: the decision describes the outcome, while the application decides whether it should be durable.

Command Handling can be described by the following steps:

Read events from the stream and build the state from them (in other words aggregate stream).
Run the business logic using the command and the state. Use the default (initial) state if the stream does not exist.
Append the result of the business logic (so events) at the end of the stream from which you’ve read events. Use the read version (or the one provided by the user) for an optimistic concurrency check.

In a nutshell, we could generalise it as:

function handleCommand<State, Command, Event>(
  eventStore: EventStore,
  streamId: string,
  decide: (state: State) => Event[],
  evolve: (event: Event, state: State) => State,
  initialState: () => State,
  // 👇 See what I did here
  skipEvent?: (event: Event) => boolean
  ) {
  // 1. Get the state
  const { state, currentStreamVersion } = await eventStore.aggregateStream(streamName, {
    evolve,
    initialState,
  });

  // 2. Run business logic
  const events = decide(state);

  // 3. Filter out "failure" events
  const eventsToAppend = skipEvent ? events.filter(skipEvent) : events;

  // 4. Append events
  const appendResult = await eventStore.appendToStream(streamName, [event], {
    expectedStreamVersion: currentStreamVersion,
  });

  // 5. Return result
  return {
    ...appendResult,
    events, // 👈 all events
    appendedEvents, // 👈 only those that were appended
    newState: evolve(aggregation.state, event),
  }
}

A command handler reads a stream, rebuilds its state, runs a decision, applies the produced events, and appends them with an optimistic concurrency check. Emmett takes care of that flow. The application provides evolve to apply an event to state and initialState for a new stream.

We can pass skipEvent function that inspects the decision’s outcome before anything is appended. The decision still returns the same events; the function only changes how the handler treats them.

In Emmett, I allow it to encapsulate the Command Handler setup by:

// 1. Command handler setup
const handle = CommandHandler<ShoppingCart, ShoppingCartEvent>({
  evolve,
  initialState,
  middleware: [
    skipOn(
      (event) => event.type === "ProductItemOutOfStock"
        || event.type === "ShoppingCartItemLimitReached"
    ),
  ],
});

// 2. Usage
const result = handle(eventStore, shoppingCartId, addProductItem);

I think that this setup clearly shows that we’re separating the business logic from how the business logic handles that, by explicitly using skipOn middleware, we’re telling that when such an event happens, we don’t want to store it, but we’ll still return it in the result.

This gives us the option to still do e.g. error mapping to HTTP status as:

import {
  Conflict,
  NoContent,
  on,
  ResponseFromEvents,
  toWeakETag,
} from "@event-driven-io/emmett-expressjs";

const addProductItemApi = (router: Router) =>
  router.post(
    "/shopping-carts/:shoppingCartId/product-items",
    on(async (request: AddProductItemRequest) => {
      const shoppingCartId = request.params.shoppingCartId;
      const productId = String(request.body.productId);
      const availableQuantity = await inventory.getAvailableQuantity(productId);

      const command = {
        type: "AddProductItem",
        data: {
          shoppingCartId,
          productItem: {
            productId,
            quantity: Number(request.body.quantity),
          },
          availableQuantity,
          maximumItems: shoppingCartPolicy.maximumItems,
          now: new Date(),
      },

      const { events, nextExpectedStreamVersion } = await handle(
        eventStore,
        shoppingCartId,
        (state) => addProductItem(state, command),
      );

      if(events.some(event => event.type !== "ProductItemAdded"))
        return PreconditionFailed();

      return NoContent({
        eTag: toWeakETag(result.nextExpectedStreamVersion),
      }),
  });

We could, of course, do more advanced error or status mapping by returning exact data, etc. But this example proves that we’re not losing anything compared to the Result scenario. The code is still pretty simple, as with throwing, but more explicit and straightforward.

Several decisions on one stream

You may be wondering why I added middleware, which sounds posh for simple filtering. But let’s discuss one more scenario: a batch import from the external e-commerce system.

A batch import often contains the contents of multiple shopping carts. Passing all of them to one command handler would be the wrong boundary here: each cart has its own stream and its own concurrency check.

Even a single imported shopping cart can still require several operations. Especially if we’re trying to deduce what has happened to a shopping cart in the external system, or we should add some specific operations on top (e.g., discounts, etc.).

The common scenario for imports is translating state changes (e.g., products in the shopping cart) into commands. This gives at least a chance to make our domain context meaningful. Not perfect, but we’ve got to save ourselves sometimes.

After translation, we may end up with a sequence of granular business operations. The mapping could look like that:

const importCartCommands: AddProductItem[] = importedCart.productItems.map(
  (productItem) => {
    type: "AddProductItem",
    data: {
      shoppingCartId: importedCart.shoppingCartId,
      productItem,
      availableQuantity: stockByProductId[productItem.productId],
      maximumItems,
      now,
    },
  }),
);

And feed it into our command handler:

const { events, nextExpectedStreamVersion } = await handle(
  eventStore,
  shoppingCartId,
  importCartCommands.map(command => (state) => addProductItem(state, command)),
);

Every command uses the existing addProductItem decision and sees the provisional state produced by the previous accepted products. That allows us to apply our own rules and also handle idempotence correctly. Persistence waits until every command has been accepted.

After batch processing, we can gather the result events. What’s more, each operation should use the state with the applied event from the previous command. We store the result in the shopping carts as a single atomic batch once the last command has been handled.

Technically it’s the same command logic, just wrapped with foreach, as we now have a sequence of decisions instead of the single one.

What should happen if we encounter unwanted scenarios? So e.g. ProductItemOutOfStock or ShoppingCartItemLimitReached? Should we stop processing or continue? Let me be a consultant for a moment: It depends! Fine, but on what? On your business logic.

In our case, we probably wouldn’t want to stop adding product items if one of them was out of stock. Tho, we may decide that there’s no point in processing it further if we already reached the items limit.

How to do it? We’d need to define a mapping function with different possible outcomes.

type MapDecision = <Event>(event: Event) => void | 'APPEND' | 'SKIP' | 'STOP' | 'REJECT';

Where:

APPEND or no result - success, append this event,
SKIP - ignore this outcome, don’t append it, but continue processing,
STOP - ignore this outcome, stop processing and append only those that were already accepted,
REJECT - reject the whole batch, even if there were some successful events.

The example could be defined for all shopping cart events not to need repeat it in all functions:

const mapShoppingCartDecision = (event: ShoppingCartEvent) => {
  switch(event.type) {
    case "ProductItemOutOfStock":
      return "SKIP";
    case "ShoppingCartItemLimitReached":
      return "STOP";
    case "ShoppingCartWasAlreadyConfirmed":
      return "REJECT";    
  }
}

We could pass it to our command handler and decide what to do with our processing.

In Emmett, I wrapped it as:

const handle = CommandHandler<ShoppingCart, ShoppingCartEvent>({
  evolve,
  initialState,
  middleware: [
    skipOn(
      (event) => event.type === "ProductItemOutOfStock",
    ),
    stopOn(
      (event) => event.type === "ShoppingCartItemLimitReached"
    ),
    rejectOn(
      (event) => event.type === "ShoppingCartWasAlreadyConfirmed"
    ),
  ],
});

Still, as you see, we’re building on top of our business logic. It remains the same; we’re just handling persistence, statuses, etc.

Returned events show which command handling passed before the rejection and which event stopped the import. The importer can report that outcome against the source record and continue with the next cart, which has a different stream.

For one product, rejecting meant leaving the stream unchanged. For an imported cart, it may also mean discarding the accepted events produced earlier for the same record.

If every command is accepted, the handler appends their events together to that cart stream. If one decision returns ProductItemOutOfStock or ShoppingCartItemLimitReached, result.events contains the outcomes produced up to that point, the imported record makes no change to the cart, and later commands are not considered.

The import job can map those events to a record-level outcome. It can report the exact business issue to the source system and continue with the next cart:

const validatedProductIds = result.events.flatMap((event) =>
  event.type === "ProductItemAdded" ? [event.data.productItem.productId] : [],
);

const failure = result.events.find(
  (event) =>
    event.type === "ProductItemOutOfStock" ||
    event.type === "ShoppingCartItemLimitReached",
);

const importOutcome = (() => {
  switch (failure?.type) {
    case "ProductItemOutOfStock":
      return {
        status: "WaitingForStock" as const,
        validatedProductIds,
        productId: failure.data.productId,
        requestedQuantity: failure.data.requestedQuantity,
        availableQuantity: failure.data.availableQuantity,
      };

    case "ShoppingCartItemLimitReached":
      return {
        status: "Rejected" as const,
        validatedProductIds,
        productId: failure.data.requestedProductId,
        maximumItems: failure.data.maximumItems,
      };

    default:
      return {
        status: "Imported" as const,
        importedProductIds: validatedProductIds,
        streamVersion: result.nextExpectedStreamVersion,
      };
  }
})();

await importStatusStore.record(importedCart.externalId, importOutcome);

switch (importOutcome.status) {
  case "Imported":
    await salesChannel.confirmImport(
      importedCart.externalId,
      importOutcome.streamVersion,
    );
    break;

  case "WaitingForStock":
    await pendingImports.waitForStock(
      importedCart.externalId,
      importOutcome.productId,
    );
    break;

  case "Rejected":
    await salesChannel.rejectImport(importedCart.externalId, {
      productId: importOutcome.productId,
      maximumItems: importOutcome.maximumItems,
    });
    break;
}

A saved-list restore uses different rules. Available products return to the cart, unavailable ones are reported, and reaching the cart limit ends the restore. Products added before either outcome remain valid, so this operation is not atomic.

The additions run in one call because every decision needs the cart state produced by the accepted products before it. Otherwise, each decision would evaluate the item limit against an outdated cart.

The loop stages and applies available products. It returns an unavailable product without staging it and continues. It also returns the item-limit event without staging it, then stops because the cart cannot accept another distinct product

And here we’re getting back to the essential question.

So should we throw or not?

If we’re getting the request from our UI, we’re the authority, so throwing could be fine; we return the status information telling the computer says no. (Read more in How to validate business logic).

Still, should we throw on an import scenario? Who’s the authority here? What we’re doing is internalising decisions that someone else already made and is just informing us about, or telling us to do our job based on this information.

So throwing is not a valid option here. The external system told us that its cart contains these products. That is true about the source system, but it does not mean our cart can accept them. The importer translates the external record into commands, and the returned events describe how our local model understood each products.

For an accepted record, the ProductItemAdded events identify the imported products and the resulting stream version is recorded. If stock is missing, the importer records the checked products and the missing quantity. It can continue with other records, then rebuild this cart’s commands with current availability after an inventory update. Retrying the unchanged commands immediately would only reproduce the same outcome. If the cart exceeds its item limit, waiting for stock cannot help; the source record needs correcting, so the importer marks it as rejected.

Of course we could still throw if this operation is synchronous and we want to give quick feedback to the caller. Still, typically, we run batch operations as a background process; also, to have proper resilience, you already saw above how much integration could happen during such an import. Services we integrate with can be unavailable, have transient issues, time out, etc.

Those failures enter the import job’s retry or dead-letter policy because there is no reliable business outcome to report yet. The returned events let the application choose between continuing, waiting for a relevant business change, and rejecting one record. Exceptions would just stop processing and not allow recovery and continue with next steps (Read more in No, it can never happen!)

There can be various error cases also on our side.

If another request changes the cart before our append, Emmett handles the concurrency conflict by loading the stream again and rerunning the decisions against its new state. Decision middleware runs again as well.

An out-of-stock import is not a concurrency conflict. The cart state did not become stale; the input said that only two items were available. Running the same command again still returns ProductItemOutOfStock. The importer waits for an inventory change and then builds a new command with the current availability.

A custom retry policy can inspect a completed attempt, but it runs after that attempt’s append phase. Retrying a result is safe only when nothing was appended, and useful only when state or input can change.

A decision and its middleware can therefore run several times during one handler call. Code that only reads the command and state can be rerun. Sending an email or charging a card inside that code would repeat the external effect. In these examples, required external data is fetched before invoking the handler and included in the command. Effects caused by appended events run later in processors, after persistence.

Middleware could also be useful for authorisation, logging, and measurements. Some of that work belongs around each decision; some should run only once for the complete handler call. Its position relative to the retry boundary determines whether it sees aggregate state and whether a retry repeats it:

await beforeAll(input);

const result = await retry(async () => {
  const state = await aggregate(streamName);

  for (const decision of decisions) {
    await runWithDecisionMiddleware(decision, state);
  }

  return appendSelectedEvents();
});

await afterAll(result);

return result;

The callback before the retry boundary receives the complete input once. It can authorise the command batch without repeating that work after a concurrency conflict. State is not available yet because the stream has not been loaded.

Middleware inside the boundary receives each decision, and the state is rebuilt for the current attempt. A concurrency retry runs it again.

The callback after the boundary receives the final result once, after persistence. It can record measurements based on appended events or the new stream version. If it throws, already appended events remain committed. Validation belongs earlier; notifications that must be delivered need a durable handler or outbox.

Never Throw in Asynchronous Handlers

As you see, message processing can end up being pretty complex. That’s also why we use Event-Driven Architecture, as it helps to make that explicit and reason about it.

The import we discussed could be (and probably should be) triggered by an event (e.g. through a webhook), as we’re informed that those items were already processed and we should just accept or ignore it. What has been seen cannot be unseen.

Whether throwing is safe depends on what sits above the code to catch it. As discussed, a command handler behind an HTTP endpoint, its usual home, has the request: a thrown error unwinds into a response.

Asynchronous handlers, the projections, reactors, and workflows that react to events, have nothing above them. A throw propagates up, stops the processor, and leaves later events unhandled. The endpoint is only the usual home, not a rule: a command handler run from a reactor loses the same safety net. Instead of throwing, we need to turn the problem into data, so return an event.

A message handler can turn a declined shopping cart operation into a compensating workflow. For instance, for ProductItemOutOfStock order more products from the producer. It can, of course, just skip a message that should not block processing, or stop without advancing further.

The same for read models. A projection cannot prevent the source event from being recorded because that event is already in the stream. Throwing only prevents the read model from advancing. If an event carries a value the read model did not expect, throwing does not undo it. The check that should have stopped it belongs in the business logic, before the event was ever recorded; by the time the projection runs, it is too late.

It builds a read model from events that are already recorded, and a recorded event is a fact: the projection only interprets it, so it has nothing to reject. If an event carries a value the read model did not expect, throwing does not undo it. The check that should have stopped it belongs in the business logic, before the event was ever recorded; by the time the projection runs, it is too late.

So accept the event and build the best read model you can from it. Skip a duplicate, fall back to a default, clamp a value back into range, whatever keeps the read model sensible and moving. Its error handling must account for delivery and recovery rather than reuse the rules of a command decision.

Conclusion

The decision returns ProductItemAdded, ProductItemOutOfStock, or ShoppingCartItemLimitReached because the application needs to know which one happened. Out of stock is an unfavourable outcome, but that alone says nothing about storage. During an import it cancels the current source record. During a saved-list restore it is skipped while earlier additions remain. At the HTTP endpoint it becomes a conflict response. The event stays the same; its handling depends on the use case.

I do not try to remove exceptions from this design. Adding a product to a confirmed cart is an invalid operation. An unavailable event store prevents the handler from producing a reliable result. Both go to the application’s error boundary.

Result could label some of the returned events as failures, but persistence, continuation, and retry still need their own decisions. The exception path remains as well. The event types already distinguish the business outcomes, so I prefer to leave them as they are.

So when someone asks me whether they should throw an error or use a Result type, I sometimes tell: actually, you can do neither.

Cheers!

Oskar

p.s. if you liked this one, check also:

Idempotent Command Handling
Against Railway-Oriented Programming by Scott Wlaschin
Emmett: Error Handling
Emmett: Command Handling

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

On swimming, learning and my journey to being the last one

Work impacts how we live, and our lives impact how we do our work. My name is Oskar Dudycz, not Paulo Coelho, and today I’m not going to talk about software architecture or the event-driven world, but about the world in general, or to be precise, my world. I’ll let you say hello to it, but if you’re looking for a technical piece, that’s not happening today.

If this story were written in Fakt, Bild, or Sun, it could have been titled:

Trained for 6 years. Finished as the last one.

That’d be almost true about what has happened to me. Almost would already be a lot for Fakt, Bild or Sun. Now, let’s go with the full story.

I’m a weird type of guy, I don’t have a driving license. And neither did I have it. I learned to ride a bike when I was 9 years old and got it for my First Communion. And I didn’t learn to swim till I was 35. Yes, that happened 6 years ago.

Did I have some trauma, like drowning? Nah, just didn’t happen. Maybe because my father couldn’t swim, maybe because I didn’t have motivation. Dunno, just didn’t happen.

Of course, not having certain skills can be fine; you can learn to live without them. You can even be considered a fashionable hipster. Some folks consider my lack of a driver’s license as a personal worldview stand. The fact that I like to cycle or ride a train suggests to some that I’m an ecologist. And don’t get me wrong, I like nature, I try to be a good citizen, but that’s not my main motivation.

This is just what I do, not who I am.

So am I a swimmer now? Well, it seems that I can swim now. I took classes from 2018 to 2019, then stopped for about 3 years, and got back into it when I signed my daughter up for swimming classes.

Now, when I say I’m practising swimming, everyone assumes I’m a triathlete preparing for a competition. And well, I’m not.

I don’t even like to swim much. I mean, it’s fine, I don’t mind, but it’s not that it’s my passion. I don’t have thoughts like:

Hell, I’d like to swim now.

And I still have thoughts like:

Hell, I’d play football now.

Which I don’t do, as I don’t play football anymore. Even though it was my passion for most of my life, I wasn’t talented at it, but I liked doing it. I even played as a junior for my hometown club (Miedź Legnica) and later in amateur leagues. And in those amateur leagues, I tore something called the lateral patellar retinacula. Actually, two of them, in both knees. Which means that my knees are unstable, and if I continued to play, it’d end up with some challenging surgery.

That also means that I had to decide to be unsuccessful somewhere else.

So, I’m an unsuccessful footballer and now an unsuccessful swimmer in the making.

I tried running and going to the gym, but it was always boring for me. I could do it for a few months, but then I quit. And off we go, for not training.

And I tried swimming.

Why? We don’t need to be capable of everything; it’s often a fine choice not to learn everything. At some point, a lack of certain skills can be limiting. You already know that it wasn’t my worldview to always have my feet on solid ground, but the main motivations were:

It’s nice to do some swimming on holidays,
It’s NOT nice to be a lame dad who can’t swim with his daughter,
I kindaish always wanted to learn that, just didn’t happen,
It’d be good to do some exercises, keeping in mind my sitting style of work,
I had a swimming pool 10m by walk,
I had and still have a great teacher. Thanks, Justyna!

So I started learning to swim again in 2024. It’s an interesting exercise when you learn something new as an adult. It’s more challenging because you’re overthinking rather than acting on your instincts. Or maybe your old habits are making your instincts unintuitional. It also shows how everyone has a different way of learning.

Take Breaststroke style (in Polish called “Żabka” - frog). Most people claim that it’s the easiest style to learn. For me? Madness, how much stuff you need to remember and coordinate: hands, back, legs.

It also shows how everything is connected. It appeared that I couldn’t do it properly because I wasn’t stretched enough. Only after I started stretching was I able to move my back and twist my legs correctly. Well almost. Almost in the Fakt, Bild and Sun way.

Ok, so I learned to swim kindaish. But the best progress I made was last year.

When we started the season in September, my coach told our group that we’d start competing. I said hell yeah, and Justyna took the irony for a real agreement. So it started.

It appeared that the shortest possible distance is 475 meters, which is around 19 swimming pool lengths. Ah, and it’s on the lake, deep for 22 meters. I didn’t know about that last part when I signed for it. I also didn’t know if I could swim for so long with the crawl style. I had probably a record of around 6-8 swimming pools.

And well, I made it.

Was I stupid or just dumb? I hope that neither of those. I used the best way to learn: practical and deadline-driven.

Instead of one training per week, I started doing two. I took some private lessons with Justyna and also started doing longer swimming sessions. Slowly but surely, I beat my record to 10, 15, 20, then 40.

Yes, it was stressful; yes, it required a patient, slow process, but it gave me at least the trust in myself that if I can swim 40, then I’ll probably make 19.

It also helped to do a real dry run in the lake and the sea. If you only swim in the swimming pool, you’ll be surprised how different swimming in a lake is.

The water is dirty; it’s super easy to lose your bearings and go in a longer circle than you should.

And guess what? The real competition brings even more surprises. The crowd of other swimmers is a challenge. Constant pushing on the start, distracting from catching the flow. Then stress, and moments of doubt about yourself. You can prepare for that, but only to some degree; the real learning comes through the next stage, which starts in competition.

Were I actually the last one? No, I was 53 out of 59. Seems I wasn’t all bad. Just half bad. I had 14m 20s time. My best on practice was 12m.

Still, not too bad for a guy who learned to swim at 35, if you ask me. I’m super proud of myself.

My friend Damian (jokingly) told me that I should write this as

“10 things I learned about EDA from practicing swimming and preparing for a competition”

And that’s not going to happen, but let me give an inspirational quote from the man, the legend, Michael Jordan:

I don’t compete with other people. I compete with what I’m capable of.

And that’s what I did. Of course, Michael wasn’t competing with others, as he was the best, GOAT. I wasn’t competing, as I was one of the worst.

Yet, I was still much better than my past self. And I also achieved more than those who haven’t even tried.

Interestingly, I was sometimes perceived as humble, and sometimes as too self-assured. Maybe it comes from a common answer when someone asks me if I can do something:

I don’t know; I haven’t tried it yet.

But I think that stems from my belief that we are all human. We’re not worse than others, but we’re also not better than others, which can be seen as bold or humble, depending on the focus of this belief.

We won’t know what we’re capable of until we try it. And when we do, we’ll already be much further than your past selves that haven’t tried.

I’m not trying to say that we can achieve everything; I probably won’t ever be a good swimmer. If I could achieve that, then anyone can. I’m not saying that’s going to be easy. It’s not. It requires time. And time is a precious resource. Not infinite. The best way to find it is to find it. But to do it, you need to set a priority, which means something else will become less important.

Pick wisely.

Should you also swim? Well, maybe. You definitely should do some exercises if you’re a developer like me. But precisely swimming? Up to you. Should you get a hobby other than work? Definitely.

Why did I write this article? A bit to brag, as I’m happy I made it.

Brag about being 53 out of 59 after 6 years of training? Seems so! You need to cherish small successes that, from the right perspective, appear to be big ones.

So, are there 10 things you can learn about software engineering from practising swimming and preparing for a competition?

I don’t know; you tell me.

Cheers!

Oskar

p.s. if you liked this one, check also:

Addition by subtraction in Software Design

I want it all, and I want it now. This is not only a Freddie Mercury quote but also, too often, a general plan for our software product. The picture tells more than words, so here it is, the typical product we designed:

And, well, that’s probably not what our user wanted. They might just want a sharper steak knife, so something more like:

I’m also to blame; I’m falling into precisely the same trap too often. Like recently, while working on Strictland, the contract testing library I announced in the previous article.

The idea behind it is to detect message schema drift easily. You write such a test:

@Test
void orderPlacedSchema_DoesNotChange() {
    // Strictland specification
    MessageContract.specification(Json.Jackson.defaults())
        .given(new OrderPlaced(orderId, "Alice", placedAt))
        .whenSerialized()
        .thenContractIsUnchanged();
}

It serialises the message and stores the payload inside the same git repository as the test. Thanks to that, you can keep it close and detect contract changes during the review by looking at the changed files.

Initially, files were kept in the same folder as the test file. That made it accessible, as you don’t need to jump from one folder to another and see related test files. The file was named by message type class name (so in this case it’d be OrderPlaced.approved.text). Simple, but what if someone adds:

Multiple tests, trying to store different snapshots for the same message type?
Multiple test files dispersed across different folders that test the same message type (e.g., in the features that consume those messages)?
Multiple message types across different modules/packages with the same name?
Multiple tests for different message-type versions and variants (e.g., with all fields, with only required fields, etc.)

All of that is solvable by some conventions, but what if you’re an open-source library creator like me? Then you can’t assume the same type of thinking, level of tidiness or knowledge of your users. You should be the one to take on the pain and make your users’ lives easier. And guess what? That applies not only to open source, but also to other products.

“Taking the pain” and “helping users” can be interpreted as an attempt to please everyone, which translates to the mentioned earlier: “I want it all, and I want it now.”

That’s what I tried to do. I decided to add “selectable storage layouts strategies”:

Near the test file - so similar to the one we had,
In the nested folder near the test file - similar, but separating the snapshot files from tests, thanks to that, you could look in the nested folder, keeping the tests not polluted by the numerous tests,
In the root folder - putting it close to the root folder, keeping snapshots together- still the same repo but giving potentially better accessibility.

And ”selectable storage layouts strategies” were not enough for me. I also wanted to add “various grouping strategies” where one could select whether they want to group message snapshots by:

Message type name,
Test files and their names.
Or just apply no grouping.

It sounded smart to me, and hell, with GenAI, I wouldn’t even need to fight with it, as my friend Claude would do it for me, right? RIGHT?

Now, ”selectable storage layouts strategies” should sound enough as a warning to me and “various grouping strategies” should ring a loud alarm bell. But it didn’t. I planned how to do it and started the implementation.

A few iterations later, the permutations of those various strategies started to pile up. Some of them were just hard to do. Some were conflicting. And most of them weren’t even useful.

Examples? I decided to make all files follow the unified format:

{messageTypeName}.{version}.{variant}.snap.approved.{extension}

And that’s fine, but then what to use as the message type name? Class name? Fine, but what about a flat layout, or grouping by message type name, with multiple messages of the same name?

Let’s use the full class path as the name? Fine, but then what if you have a root folder and a nested folder strategy? Would you nest folders by the class path, and then repeat it in the name?

Or maybe you’d differentiate the file name format depending on the strategy one chose?

Ok, but what if the user decides to change the strategy at some point?

I decided to run a sanity check I should have done earlier. Better later than too late. The discussion inside my head looked more or less like this:

Me: What’s the actual goal of this library?
I: To detect message schema drift.
Me: Ok, then do I need to build the snapshot library?
I: To some degree, I’m using snapshot comparison to detect that drift.
Me: Sure, but I probably don’t need to cover all the cases that the generic snapshot library has to, right?
I: Right.
Me: Ok, so what’s our bet for users? Are they advanced or just beginning the journey?
I: At the beginning.
Me: Then I should help them, set a sane default, and guide them. Especially since this is the early phase of this library, I haven’t yet received much feedback.
I: Agreed, maybe we could start then with the recommended layout and grouping, and then gather feedback. I can add more strategies as needed.

That sounds like an obvious thing to do, but it requires pausing to find the time to evaluate our assumptions. We shouldn’t always be mad at ourselves for not predicting everything earlier. Some stuff is just hard to predict. Personally, I usually see some steps after I try to implement them. Those are also the steps I explained in the article about the OODA loop. We should:

Observe - This is the intake of raw, unfiltered information. In our world, this means looking at the state of the system.
Orient - This is the most critical and difficult stage. It’s where we filter our observations through your experience, culture, and technical knowledge.
Decide - Based on our orientation, formulate a hypothesis.
Act - We execute.

Rinse repeat.

If we think about it this way, the time spent considering those various cases is not wasted. It’s similar to the anti-requirements exercise described by Andreas Öhlund and David Boike or Barry O’Reilly’s Residuality Theory. We gather numerous needs and ideas about our product evolution and see what sticks and what remains. Not all ideas will be great; some will be bad, but that doesn’t matter; it gives us food for thought.

Still, we can’t forget about the most important part: subtraction.

We need to subtract all the stuff that’s not needed, leaving only the useful residues, the one that solves the main goal.

For my case, I realised that if I chose the single strategy, it would organise my contract into the root test folder like this:

📁 src/test/resources/
  📁 contract-registry/
    📁 com/acme/orders/OrderPlaced/
      📄 OrderPlaced.1.default.snap.approved.json
      📄 OrderPlaced.2.default.snap.approved.json
    📁 com/acme/orders/OrderInitiated/
      📄 OrderInitiated.1.AllData.snap.approved.json
      📄 OrderInitiated.1.OnlyRequired.snap.approved.json
      📄 OrderInitiated.2.AllData.snap.approved.json
      📄 OrderInitiated.2.OnlyRequired.snap.approved.json
    📁 InvoiceIssuedEvent/
      📄 InvoiceIssuedEvent.1.default.snap.approved.json

By nesting by message type and full path and grouping by contract name, I’m not only keeping snapshots but also creating a Contract Registry.

So this subtraction of some features not only gave me focus on the specific scenario but also enabled me to create scenarios that I would actually like to have eventually, for instance:

keep the message schema in the same folder as snapshots,
markdown docs of the message,
Having defined the folder structure, generate documentation,
check dependency, and who uses that,
etc.

That led me to conclude the following two rules:

The addition of imaginary features subtracts the actual value.
The subtraction of imaginary features adds the actual value.

It’s also aligned with my motto: Removability over Maintainability.

What’s the lesson then?

We’re not wasting time on building too slowly; we’re wasting time on building stuff that doesn’t matter. All those useless features we imagine without evaluating them with people are a wasted opportunity to deliver better for them.

Nowadays, with LLMs, it’s tempting to rush into implementation and add every feature we can imagine. That’s never free. We’re increasing the cognitive load for our users and also for us; we waste time on stuff that doesn’t matter and are constantly distracted.

Yet, if we don’t do sanity checks, trying things in a proof-of-concept way can be a big multiplier. If we stop after that and rethink.

I’d still advocate doing a sanity check earlier than I did in the described scenario and thinking first, but not all can be forseen like that. So:

Generate as many ideas as you can.
Try to think about how they match together, think about the tradeoffs, the cost of adding them, the cost of removing them and the cost of changing your mind.
Define potential solutions.
Stop the addition phase and rethink what you came up with. Ask yourself what you’re building and what the actual goal is.
Start the subtraction phase and cut as much as you can on the stuff that’s actually not needed. Of course, don’t drop ideas, remember that quantification is subjective. Deliver a simple solution that will give you enough optionality to change your mind.
Send it wild and gather feedback.

And that’s precisely what I did: I just released Strictrland 0.4.0 and will plan the next steps based on user feedback!

Cheers!

Oskar

Announcing Strictland - new contract testing library for message compatibility

I just released something. It’s called Strictland. And it’s a contract testing library. Why did I do it?

Before I go further, if you can’t wait, you can check it on:

GitHub - https://github.com/event-driven-io/strictland
Maven Central - https://github.com/event-driven-io/strictland

So now you already know that it’s a JVM (Java, Scala, Kotlin, etc.) Open Source library.

Let me first show you a sneak peek. Stop for a moment and have a look on this:

@Test
void ensureOrderPlacedCompatibilityWithNewerVersion() {
    // Strictland specification
    MessageContract.specification(Json.Jackson.of(yourObjectMapper))
        .given(new OrderPlaced(orderId, "Alice"))
        .whenDeserializedAs(OrderPlacedWithCoupon.class)
        .thenBackwardCompatible();
}

What would you say if you saw such a test? Think about it, we’ll get back to it.

What’s Strictland?

But again, why did I do it if there are mature solutions like Pact, Spring Cloud Contracts, or Confluent Schema Registry?

If you’ve used consumer-driven contract testing, the usual approach is to run both the provider and the consumer, record the consumer’s expectations against a mock, verify the provider against those expectations, and share those contracts through a broker.

In Strictland, I took a smaller, simpler approach. It serialises the message in a standard unit test and saves the output as a snapshot file that you commit alongside your code.

The test fails when the serialised shape changes, or when it contains a breaking change - up to you to specify expectations. A check confirms that an older and a newer version of the message can still read each other’s data (or the other way round).

Because it’s only serialisation and a file, the setup stays small:

The checks are ordinary unit tests in your existing suite, so there’s no broker, schema registry, or mock service to run, and nothing to start in Docker.
The contract is the serialised JSON committed alongside the test, so any format change appears in a normal diff and is reviewed like any other code.
You write the check beside the message it covers and get the answer in the same fast feedback loop as the rest of your tests. The check uses your application’s own serializer, so the snapshot is the exact bytes you ship.
Strictland checks the serialised shape of a message and whether its versions stay compatible. It doesn’t exercise a live exchange between running services, so it complements that kind of tooling rather than replacing it.

Strictland checks the serialised shape of a message and whether its versions stay compatible. It doesn’t exercise a live exchange between running services, so it complements that kind of tooling rather than replacing it.

It’s not as powerful as popular tooling, but it’s also much simpler to start catching our mistakes. Traditional solutions allow you to mock protocols, put a man in the middle, and even generate client code. All of that is great if you need it and have experience with it.

Most of the customers I’m helping through consultancy and training aren’t there yet. Setting up those tools is a lot of heavy lifting for them and adds additional complexity. And well, maybe they don’t need to be, as this approach served me well in my past projects. I always handcrafted such a tool in my projects, but finally decided to make it properly.

Why such a name?

I’m putting (too?) much effort into my projects’ names. It’s a word game. Contract testing rewards a strict approach to your message shapes, and Mr. Strickland was strict enforcer in Back to the Future. That puts it in good company next to its sibling Emmett, named after Doc Emmett Brown. I also didn’t want to collide with the older JS validation package.

How to use it?

Getting back to the essence… How to use it?

You write a small unit test that locks down a message’s format. Later, you rename a field, change a type, or adjust how a value serialises; the code still compiles and your other tests pass, but that one fails and points at what changed. You fix it in your build before a consumer or a stored event has hit the old format in production.

When a message changes by accident, a snapshot check shows you exactly what moved. When you evolve a message on purpose, a compatibility check confirms an old and a new version can still read each other’s data.

You can start by installing it from Maven Central, by adding dependency.

<dependency>
  <groupId>io.event-driven</groupId>
  <artifactId>strictland</artifactId>
  <version>0.3.0</version>
  <scope>test</scope>
</dependency>

Then adding such test:

MessageContract.specification(Json.Jackson.defaults())
    .given(new OrderPlaced(orderId, "Alice", placedAt))
    .whenSerialized()
    .thenContractIsUnchanged();

The first run serializes the message and writes the result to an approved file named after the class, OrderPlaced.approved.txt, saved next to the test:

{"orderId":"00000000-0000-0000-0000-000000000001","customer":"Alice","placedAt":"2024-01-01T12:00:00Z"}

You review that file and commit it. From then on the check compares against it and fails if the format drifts, so a later change to the format shows up in the same pull request as the code that caused it.

A message under contract goes through one of two checks.

A snapshot check confirms the message still serializes exactly as it did when you last approved it, so nothing reading it downstream breaks. A failure means the format changed: a field renamed, a date format switched, a value newly dropped or added.

A compatibility check is for the version you evolve on purpose, so changing a message doesn’t leave the ones already in your store or on the wire stranded. Use thenBackwardCompatible() to confirm the newer version still reads a message the older one wrote, the events you stored last year, or a request already sent. Use thenForwardCompatible() to confirm a reader that hasn’t upgraded yet still reads a message the newer version writes, so you can ship the new shape before everyone reading it has caught up. Both compare the fields the two versions share and fail if a required one is missing or a shared value has changed.

@Test
void ensureOrderPlacedCompatibilityWithNewerVersion() {
    // Strictland specification
    MessageContract.specification(Json.Jackson.of(yourObjectMapper))
        .given(new OrderPlaced(orderId, “Alice”))
        .whenDeserializedAs(OrderPlacedWithCoupon.class)
        .thenBackwardCompatible();
}

As you see, in the specification, you pass your serializer.

var spec = MessageContract.specification(Json.Jackson.of(yourObjectMapper))

I encourage you to use your application’s object mapper and pass the same ObjectMapper it uses, so the test checks the exact bytes you ship. That’ll help you avoid nasty surprises, where tests are using a different format than your framework or tooling.

Strictland provides an implementation of a sensible Jackson setup: ISO-8601 dates, nulls kept, unknown properties ignored on read. You can use it with Json.Jackson.defaults(). But it’s more of an accessible thing to get you started quickly.

You can also define your own serializer if you’re using an unsupported (yet?) format or serializer type. See basic examples in:

CsvMessageSerializer and its tests or,
SimpleBinaryMessageSerializer and its tests.

Should you use it?

Strictland is young and pre-1.0, so the API can still move between versions. The checks themselves are small and well covered, and the snapshots they produce are just files in your repository, so trying it out costs little and commits you to nothing.

I’d genuinely like your feedback. If something is missing or awkward, tell me through comments, on Discord or open an issue.

Currently, I’m working on making snapshot location more organised and configurable, not to end up with a bloated snapshots tests folder.

I also need to rethink the given name; it’s from Behaviour-Driven Design, but it appears to be a reserved word in Scala 3. Cross-language and cross-stack libs are much fun!

For now, for JVM, as the Java-based customers I’m helping also needed that, but soon want to add .NET and TypeScript/Javascript (and maybe more).

Tell me your honest thoughts, give it a try and a star if you liked it. Play with it, and pass me your feedback. It’ll motivate me to continue working on it!

Would you like to learn more about consumer-driven tests and how Strictland can be used for them?

And read in the event versioning series:

Cheers!

Oskar

You can fork a package, but can you own it?

Fork your dependencies, trim them to only your use case, never update unless it breaks for your users. I’ve been vocal about this for 10+ years. I’ve always said that updating is way riskier than latent bugs (which can be tracked and CVEs monitored).

If you are updating a dependency, it’s on you to analyze every single commit in the full transitive set of dependencies. If you dont see anything compelling, dont update!

I remember at HashiCorp once in awhile an engineer would try to update a dep or replace a DIY lib with an external one and id always ask “show me the commit we need.” Dont update for the sake of it.

Feeling pretty swell about this mentality with all the supply chain attacks happening.

That’s from Mitchell Hashimoto. A friend sent it to me, and the first word he reached for was bold. Mine too. I mostly agree with Mitchell, honestly. But the second I read it, my head jumped to a caveat, and the caveat turned out to be the thing I actually wanted to write about.

We can fork a small library and trim it to what we use. I’ve done it, it’s fine. But could we fork React and maintain it? I couldn’t. I don’t think most teams could either. So “fork your dependencies” is wonderful advice right up to the point where the dependency is too big to hold in our hands, and then it quietly stops being advice at all.

And that caveat says something about the advice itself. I think that it is, actually, not about forking. I see it more about knowing exactly what we’ve taken on and being willing to own it, and that’s the bit most of us skip. We don’t decide to take a dependency. We just install one. And almost everything that goes wrong with dependencies:

the supply chain attacks,
the licence dramas,
the half-finished SBOMs,
the things we find out about only after they break

comes back to that one move: we took dependence on someone else’s product without ever deciding to. Everything else here is a variation on it.

To fork or not?

I was reminded of this some time ago on a live Q&A about my TypeScript port of some code, and a whole chunk of it was one question asked five different ways:

Why did you write your own deepEquals instead of just pulling a package?

I tried to explain that it wasn’t overachieving, and it wasn’t not-invented-here pride either. I explained that maybe the code wasn’t trivial, but straightforward enough for me, and is the type of code I write once and forget. The reception was, let’s say, mixed. I get it, it’s much easier to just install a package and outsource maintenance costs. Especially when we’re in a hurry, we don’t make any decisions; we just do. We reach. And this reach can be a decent interim solution, but we should reflect on our next steps.

Why do I reach for my own code on the small stuff? Look at which packages actually get hit in all these supply chain attacks we keep reading about. It’s almost always the little ones. The one-liners. Remember the Left-pad Incident? It’s usually some tiny or low-level helper nobody thinks about. They’re everywhere, they’re trivial, and precisely because they’re trivial, nobody is watching them. Which is also exactly why they’re so easy to write ourselves. So for me, handwriting a small helper doesn’t always mean that we’re paranoid. It may be an option that will buy us some calm. It gives us knowledge of exactly what is in our own system.

That’s also why, in Emmett, my Event Sourcing library,, I’m stubborn about keeping the core package free of dependencies and limiting whatever I inject elsewhere. Probably too stubborn, if I’m honest. But I’d rather err on the side of caution, because when it goes wrong on the user’s side, it goes far worse than a bit of code I maintain myself. It’s about responsibility for the outcome of our actions.

If I had to compress what I believe into one line, it wouldn’t be “fork everything”, and it wouldn’t be “write everything ourselves”. It would be something much less sexy than those two:

Be precise about which dependencies we take on, look at how many dependencies pull in, and treat that as part of the decision. They’re now our dependencies.

The number of dependencies itself isn’t the point. It just tells us how much we’re agreeing to own without ever seeing it.

Because our direct dependencies were never the worst part, we chose those more or less cautiously. Most of the time, we can follow what they’re doing. The scary part is the dependencies of our dependencies, and theirs, all the way down, the part we didn’t choose, can’t see, and have no say over.

And I want to be careful here, because it’s easy to let this slide into another round of JavaScript-bashing, and that’s not what I mean. Every ecosystem has this. JS and TypeScript just sit at one far end of it, where there’s a package for absolutely everything, which is, in general, good, as we don’t need to reinvent the wheel every other week, unlike in some other places. But it’s also how we end up with a node_modules we couldn’t fully explain if someone put a gun to our head.

At the other extreme, there’s Microsoft and .NET, where the instinct runs so hard the opposite way that it tips into Not Invented Here Syndrome. Neither end is the “right” one. They’re both defaults people drift into without ever making a decision.

For me, it’s not about reaching zero dependencies. But having dependencies that we cautiously agreed upon.

Which takes me to the part that, in my experience, almost nobody does. We can’t make a call on what we can’t see, and if we don’t even have the basic knowledge (e.g. some list) of what we depend on, then every conversation about supply chain risk is a bit of theatre.

Dependency Inventory

In most of the environments, there are tools to generate the Software Bill of Materials - the inventory of our dependency tree. In some, they’re even built in. It’s easy to dunk on NPM, but it’d be better to do due diligence before doing so. Not many people seem to know this, but recent versions of npm ship an npm sbom. So the tooling exists, even in NPM. That isn’t the problem.

The problem is that most organisations have never generated one in their life. No SBOM, no inventory, nothing written down anywhere. So the day the next Log4Shell lands, and there will be a next one, they can’t answer the very first question anyone will ask them: do we run this, and if so, where?

On the other hand, tools often don’t help here, even those built to do so. NPM audit mostly does the opposite. I honestly can’t remember the last time I installed something, and the audit didn’t immediately tell me to bump a stack of packages. Most of it is false positives, with no real attempt to say how dangerous any of it is. And that lands us in the oldest trap going: if it’s always red, we stop looking at red. So the one signal that was supposed to make us stop and decide ends up training everyone to decide nothing.

Bus factor and rug pulls

There’s a related thing I can’t quite leave alone, so let me wander into it. I think a lot of teams spend their energy on the symptom and never once look at the source.

Watch what happens in the .NET world whenever a popular package changes the deal. Fluent Assertions went commercial. Moq shipped a thing that quietly hashed git email and phoned it home. MassTransit and AutoMapper announced commercial licenses within the same stretch. And nearly every time, the reaction across .NET shops is identical. It’s a mixture of:

let’s rip it out and write their own,
search for a free alternative,
Cry to Microsoft to buy the lib or provide a replacement,

Essentially: a throw-the-baby-out-with-the-bathwater strategy.

And for me, that’s solving the wrong thing entirely. The source isn’t that the package started charging money, or pulled a rug. The source is that we took on a critical dependency without ever admitting to ourselves that it was critical, and never once thought about what we’d do if the terms changed. We didn’t consider the bus factor, and we didn’t do due diligence to ensure the work on it was sustainable and could continue. Pulling it out and hand-rolling a replacement fixes none of that. It just resets the same trap, this time with only the code we maintain.

The IdentityServer episode was the clearest version of it I’ve seen. People were upset that they had to pay suddenly. Then, in the next sentence, it is called a critical security component. Then, in the sentence after that, it asks what the free alternatives were. A critical security component that we want for free and are ready to swap out overnight is, to my mind, asking for a security incident.

And there’s a bit of maths that quietly settles most of these arguments, if anyone bothers to do it. Take what the licence costs us per year. Then take into account what it would cost to have an engineer build and maintain our own version. Put the two side by side.

Almost every time, paying the maintainer comes out cheaper, and on top of that, we’ve lowered the bus factor on something we already lean on, which is its own kind of supply chain security. “We’d write it ourselves, but then we’d have to maintain it” is true. I just read it as the argument for paying the person who already does, not against it. If we depend on something, its survival is our problem too. That’s part of owning the decision.

I know this case too well.

LLM as a fork

Getting back to Mitchell’s thought. The part I find most interesting is because of the moment we’re in. I keep hearing that LLMs change all of this, that writing our own small things is suddenly trivial, so the whole dependency question softens. I don’t buy it. It’s never that easy. Writing the small thing was never the hard part anyway. Owning it, understanding it, maintaining it, being the one on the hook when it breaks at 2 am, that’s the hard part, and no model takes that off our plate.

I don’t see how LLMs can change the cost of owning code. They can (maybe) change the cost of producing it. That doesn’t fix the “install without deciding”. The old move was install and move on. The new move is “vibe it” and move on. Same missing decision, new flavour. The same lack of responsibility and ownership.

This trend isn’t new. It’s a classic Shadow IT. If you haven’t been around long enough to run into the term, Shadow IT refers to the tools and systems people build or adopt within a company without going through whoever is officially meant to approve them. The spreadsheet that quietly runs a whole department. The little script someone wrote on a Friday that half the team now depends on. Nobody in the platform group has ever heard of the integration. It has always existed because people route around slow governance to get their job done, and most of the time, nobody notices until it breaks.

With LLMs, it’s more tempting than it has ever been. Someone in sales promises a customer a feature the team supposedly needs. The team has no time, so they cobble a tool together from the API and ship it. It doesn’t work. The customer says they’re not paying for this. It escalates. The thing was unowned from the moment it was conceived; nobody decided to take it on, it just appeared, and the blame game is starting.

And here’s where I think it all settles, because the corporate steamroller flattens everything in the end. Companies will dictate the allowed list, the way they always have. The cautious majority will stick to what’s known and popular: React, TypeScript, Python, Spring Boot. That’s what they did last time, and the time before. And the people who want to move faster will do it off the books, with an LLM, as Shadow IT. The declarative, standards-based frameworks that hide their complexity will do well in that world, because that style suits how these tools work, but it’s the same ceiling as before. We bet on React. We don’t own it. The small stuff we can hold; the big stuff stays as bet.

What to do then?

We cannot fix the entire software industry, but we can fix how our own engineering teams operate. Instead of waiting for automated audits to scream at us, or ripping out packages in an emotional panic, I suggest a simple, regular exercise for our organisation.

Sit down and explicitly define your dependency posture:

Inventory: List the dependencies you use (even without peer dependencies). Use tools npm-sbom to actually see what you are pulling in.
Criticality: Identify which of these packages are absolutely critical to your system.
Lifecycle: Define a clear strategy for upgrading and versioning them. Are you updating just for the sake of it, or are you looking for specific commits like Mitchell suggests?
The Bus Factor: Ask yourself: what happens if the author of a critical package gets hit by a bus, burns out, or the tool becomes paid?
Mitigation: Decide on a concrete backup plan for that exact scenario. Do you fork it? Do you pay the license fee? Maybe pay earlier for support or help in another way to maintain it.
Response Time: Estimate how quickly you can upgrade and deploy the application if a major security breach occurs in a dependency. Also, if the strategy is to use replacement, then how fast will you be able to replace this dependency?

Building reliable software requires intent. We don’t have to write everything from scratch, but we must be precise about what we bring into our software. Architecture is not just about writing code; it is about choosing which liabilities we are willing to own.

Cheers!

Oskar

How soon is now in PostgreSQL?

How soon is now? In PostgreSQL, it’s not always as soon as you’d think. I learned that the hard way recently, so you don’t have to.

It took me hours and wasn’t easy to reproduce, even though the fix is one line. I found it in a Cybertec post, as I quite often do when I’m staring at something odd in PostgreSQL. I’m supposed to know my way around the database, but I missed it, which is another reason I want to write this down.

I was working on distributed locking in Emmett. When you scale a service horizontally, you can easily end up with two instances of the same message processor running at once. That’s bad. Both instances would pull the same events, both would write to the same projection storage, and we’d get duplicated side effects, overwritten state and broken checkpoints. So we need to guarantee that exactly one instance of each processor is active at any time. Emmett does that using two things working hand in glove: PostgreSQL advisory locks and a row in the emt_processors table. The row keeps the durable side of ownership: which instance currently holds the processor (processor_instance_id), when it last checked in (last_updated), and what state it’s in (status). I described the full design in Rebuilding Event-Driven Read Models in a safe and resilient way, so I won’t bore you with the whole picture here.

For this story, the part that matters is what happens when an instance crashes. The crashed processor’s connection is gone, so its advisory lock has already been released. A new instance can grab the advisory lock without resistance. But the row in emt_processors still says status = 'running' and still points to the previous owner, because the crash didn’t give anyone a chance to clean it up.

From the outside, we can’t tell whether the previous owner has crashed or is just between heartbeats. So we wait. If the row’s last_updated is older than a configurable timeout, the new instance is allowed to claim ownership anyway. Anyone quiet for that long is treated as gone. To make this graceful, the lock acquisition runs inside a retry policy. A fresh instance starting just after a crash doesn’t fail straight away; it retries until the timeout window expires.

The bug

The takeover decision lives in the upsert against emt_processors. In the real function, that upsert sits inside a Common Table Expression (CTE) alongside a pg_try_advisory_xact_lock call.

For the record: the snippets below skip that wrapping (and trim a couple of unused parameters) to keep the focus on the upsert, where the bug lives. The full version is in the source.

CREATE OR REPLACE FUNCTION emt_try_acquire_processor_lock(
    p_processor_id           TEXT,
    p_processor_instance_id  TEXT,
    p_lock_timeout_seconds   INT
)
RETURNS BOOLEAN
LANGUAGE plpgsql
AS $$
BEGIN
  INSERT INTO emt_processors (processor_id, processor_instance_id, status, last_updated)
  VALUES (p_processor_id, p_processor_instance_id, 'running', now())
  ON CONFLICT (processor_id) DO UPDATE
  SET processor_instance_id = p_processor_instance_id,
      status                = 'running',
      last_updated          = now()
  WHERE   
     -- same instance reconnecting
     emt_processors.processor_instance_id = p_processor_instance_id                      
     -- previous owner stopped cleanly
     OR emt_processors.status = 'stopped'     
     -- previous owner timed out    
     OR emt_processors.last_updated
        < now() - (p_lock_timeout_seconds || ' seconds')::interval;
  RETURN FOUND;
END;
$$;

The last branch is the takeover. It reads naturally: if the previous owner hasn’t checked in for longer than the timeout, the new instance can replace them. All tests were green. Stop me if you think you’ve heard this one before. The problem surfaced through user feedback (thanks, Martin!), and it took me a long time to reproduce; none of the existing tests covered the scenario that triggered it. Once I had a new end-to-end test that pinpointed the symptom, the rest was the usual, long, boring, debugging loop.

To see why, open psql and run this:

BEGIN;
SELECT now() AS tx_now, clock_timestamp() AS wall_clock;
SELECT pg_sleep(2);
SELECT now() AS tx_now, clock_timestamp() AS wall_clock;
COMMIT;

You’ll get something like:

            tx_now             |          wall_clock
-------------------------------+-------------------------------
 2026-05-25 10:00:00.123456+00 | 2026-05-25 10:00:00.124012+00

            tx_now             |          wall_clock
-------------------------------+-------------------------------
 2026-05-25 10:00:00.123456+00 | 2026-05-25 10:00:02.131845+00

The first column is the same in both rows. The second one is two seconds apart. As it turns out, now() is a synonym for transaction_timestamp(): it returns the time the transaction began, and keeps returning that value for every statement inside the same transaction. A light that never goes out, in other words. clock_timestamp() reads the wall clock each time it’s called, so it advances as time does. Cybertec wrote a good walkthrough of the whole family of timestamp functions if you want the full picture.

What difference does it make? For a column like last_updated, the constancy of now() is usually what you want: every row touched in the same transaction shares a single timestamp, which keeps audit logs and write batches coherent. For asking “has enough time passed?” inside the same transaction, the same constancy works against us.

Now back to the retry. The consumer that calls tryAcquire looks roughly like this:

pool.withTransaction((tx) =>
  asyncRetry(
    () => tryAcquireProcessorLock(tx.execute, options),
    { retries: 10, minTimeout: 200, maxTimeout: 1000 },
  ),
);

pool.withTransaction opens a database transaction and passes its executor to the body. asyncRetry then repeatedly calls the stored procedure on that same executor, with a backoff between attempts. So even though the retries are spread out in real time, every call runs inside the same database transaction:

withTransaction        (transaction starts at T)
  └── asyncRetry
       ├── call lock function    → now() = T
       ├── call lock function    → now() = T   (200 ms later)
       ├── call lock function    → now() = T   (400 ms later)
       └── ...

last_updated < now() - timeout evaluates the same way every iteration. The predicate is effectively constant for the lifetime of that transaction. From the database’s perspective, no time was passing between attempts, even though the retries were spread across real seconds. (of course, the valid question is whether retries should happen inside a transaction, but let’s say that this is out of scope of today’s article, deal?).

So what was the fix? Change the time source inside the function. PL/pgSQL lets you declare local variables, so I added one at the top, initialised from clock_timestamp(), and used it everywhere the function previously called now():

CREATE OR REPLACE FUNCTION emt_try_acquire_processor_lock(
    p_processor_id           TEXT,
    p_processor_instance_id  TEXT,
    p_lock_timeout_seconds   INT
)
RETURNS BOOLEAN
LANGUAGE plpgsql
AS $$
DECLARE
  v_current_time TIMESTAMPTZ := clock_timestamp();
BEGIN
  INSERT INTO emt_processors (processor_id, processor_instance_id, status, last_updated)
  VALUES (p_processor_id, p_processor_instance_id, 'running', v_current_time)
  ON CONFLICT (processor_id) DO UPDATE
  SET processor_instance_id = p_processor_instance_id,
      status                = 'running',
      last_updated          = v_current_time
  WHERE emt_processors.processor_instance_id = p_processor_instance_id
     OR emt_processors.status = 'stopped'
     OR emt_processors.last_updated
        < v_current_time - (p_lock_timeout_seconds || ' seconds')::interval;
  RETURN FOUND;
END;
$$;

clock_timestamp() ignores transaction boundaries. Every call to the stored procedure reads the wall clock fresh, so each retry sees a slightly later value than the last. After enough retries inside the wrapping transaction, the takeover predicate flips, and the new instance wins.

The function uses the timestamp in two places: when setting last_updated on the new owner, and when comparing the previous owner’s last_updated to the timeout. I switched both to v_current_time, so the write and the check read from the same wall clock. Mixing clock_timestamp() on one side and now() on the other would leave a subtler version of the same bug.

A cleaner option for the future is to move the retry one layer up, so each attempt opens its own transaction. That would remove the trap entirely and let the function go back to plain now(). For now, the local variable does the job.

Why my tests didn’t catch it

Now to the testing side. I had a careful suite of integration tests for the stored procedure. Two instances racing for the lock. The same instance reconnects after a crash. Takeover after a custom timeout. They all passed, and they could not have caught this bug. Here’s the shape of a typical one:

await pool.withTransaction((connection) =>
  lock.tryAcquire({ execute: connection.execute }),
);

That’s the shape: set up some state, call tryAcquire once, check the result. A single call to the stored procedure works fine with now() in the WHERE clause. There is only one timestamp involved per call, and the predicate evaluates correctly against it. The bug only shows up when several calls share one transaction, which happens when the consumer’s withTransaction wraps a retry policy. The stored-procedure tests never put those two together.

The end-to-end consumer tests do go through the consumer’s withTransaction wrapper, but they covered the happy paths: clean start, clean stop, two consumers competing, and an instance reclaiming its own stale lock. None of them combined the three conditions that together expose the bug:

a previous owner whose row still says status = 'running' (a crash, not a graceful stop),
a new instance arriving with a different instance ID,
a retry acquisition policy with a timeout short enough for the retries to outlast it inside the test’s deadline.

Any one of those missing and the takeover predicate either succeeded on the first attempt (so the retry never fired), or the test finished before the retry’s failure to make progress was visible.

So why did this slip through both layers? The stored-procedure tests never combined a retry policy with the stale-row state, so they never produced multiple calls inside one transaction. The end-to-end tests did exercise the retry path, but none of them happened to combine all three conditions above at once. Both layers had blind spots, and the bug lived exactly where they overlapped.

The Pull Request that fixes the bug also adds a new family of tests that mount tryAcquire under the same transactional wrapper the real consumer uses, with the crash + new-instance + retry combination wired up on purpose. That’s the kind of test I should have had from the start.

What I’m taking away

Two things about now():

now() is the right tool when you want every row touched in the same transaction to share a timestamp. created_at, last_updated, audit columns. That stability is a feature.
now() is the wrong tool when you want to ask “has time moved on?” from inside a transaction. Use clock_timestamp() when you genuinely mean the wall clock.

And one harder thing about tests. Inner tests for stored procedures give me a tight feedback loop and pinpoint failures, but only inside the scaffolding I build for them. End-to-end tests run the real wiring, but I can’t enumerate every combination of timeouts, instance IDs and crash states without the suite collapsing under its own weight. The combination where this bug lived wasn’t reachable from either side by accident; it needed a deliberate setup.

I don’t have a clean rule for where to draw that line, and honestly, I don’t think there is one. My takeaway is to look at the seam: the spot where the inner test invokes the code differently from how the production caller invokes it. Here, it was a single-call test against a retry loop sharing one transaction. Wherever that gap sits, write a test there. Not at the unit level, not at the full end-to-end level, but in a setup that mirrors how the real caller actually drives the code and exercises the path you care about.

TLDR

now() returns the start of the current transaction, not the current moment. Inside a transaction it doesn’t change between statements. If you wrap a retry loop around a function that uses now() in a WHERE clause, and the retry loop runs inside one transaction, the predicate is frozen and the retries do nothing. Use clock_timestamp() when you mean “right now”. And pay extra attention to the seam between inner and end-to-end tests, because that’s where mismatches between how tests drive the code and how production drives it tend to hide.

The fix and the new tests live in Emmett Pull Request #339.

Uff. That bug was nasty.

On mashing up modelling techniques for fun and profit

Many people believe there should be one, and only one, way to model software. I think differently, I like to mix different techniques.

For instance, I believe there’s a strong synergy between C4 Model, Context Maps, and EventStorming. They all allow us to look at the system from a different angle and act as simulations, providing different feedback on whether our model will fly.

Look below for the diagram I prepared for my upcoming workshop.

It’s a C4 diagram showing containers in a hospitality system (so-called Property Management, where property means hotel).

A container gives quick feedback on how many pieces I’ll potentially need to manage and deploy, and it also shows me boundaries.

Context Maps help in understanding how they relate to each other, which module or team has bigger leverage, and can force more on others. It can also show me the information flow and simulate which module will expose the API.

I can then take some business process and have a look at how the message flow will look. This can happen as an early simulation or after EventStorming sessions.

Then C4 allow me to also look inside the container, show components if I need to understand more details. I can group containers if my bounded context has more of them.

How to start with it? Let me show you an example.

The starting point could be understanding the system’s overall view; integrations with external systems usually introduce the most complexity. C4 System Diagram works well for that.

We see actors representing our system users and interactions with external systems. We can decide what integration is out of scope for now, and may come as the second step, but not for now. We can also show the current state as is, then in the second diagram show our vision of what we want it to be. Be creative, this is not a relational database, we don’t need to normalise it.

Then we may decide to dive into what we need to build to facilitate that, make a bet based on our current knowledge, zoom in and check the C4 Container view.

As mentioned, this can already be used both for static documentation and for discussions during a workshop, simulating the potential complexity and boundaries. Important to note: until it’s settled, it’s more than fine to show different versions, discuss them, and add notes and questions.

We may have concluded that C4 is fine, but it doesn’t capture all the important reasons we modelled it this way. Or during the workshop, it may not be clear how to discuss precisely how to break it down. Context Maps can help in that.

They can give us information on which module is generic (Open Host), which module (or team) is more important (upstream), and which module or team has less leverage (Downstream). Which provides api or data (Supplier) and which one is using it (Consumer).

We could have a dedicated model only focusing, on that, but we could also capture it in the same one. It could look as follows:

We can even use colours, e.g., upstream as green and downstream as red. This can already give us feedback, e.g., that Cashiering is an important module for financial processing and that it is downstream of multiple modules. Maybe we can change it and redefine boundaries, or maybe live with it, but take some corrective actions.

We can also put more data and use a specific example of a process and have a look at the specific process, as shown in the original diagram:

This can also pinpoint some issues, e.g., a generic module, which is more likely to expose commands (see more on why in my article on Passive-Aggressive Events). We can see if we have all the needed ACLs defined, or which module acts as a coordinator.

Of course, keeping it all on the same diagram is not perfect; it’d be great if we had a tool that would enable zoom-in/zoom-out, link and enable/disable some of the details and notation. Then we can even better play with what we have. Maybe I should vibe such?

Nevertheless, I encourage you to try different techniques, experiment, and combine them. For instance, Example Mapping would, imho, play great here as an extension. I know that some use Wardley Maps or Domain Storytelling.

Start small, use existing techniques and tools, and try to see how you could benefit from mixing them. Show it to your friends and try to collaborate and have fun together.

I even learned from Richard Gross that this has an even name: Model Storming. So, crunching your design with different modelling techniques and coming up with some useful variations.

I have even more mashups like that. Tell me if you’d like me to expand on it or this part in the follow-up articles!

Also, if you did such experiments, please share them with me and others. Happy to see what you came up with.

Cheers!

Oskar

Don't overestimate domain expertise

I was toying with LLM-based domain research. It reminded me of the common mistake we make when we try to practice DDD: overreliance on what domain experts are telling us.

I wanted to remind myself of the domain I used to know: hospitality management, to adjust my upcoming workshop. I selected LLM (Claude Opus) as a sparing partner. And boy, I got a loooot of details. I was swamped by them. The LLM modelled everything: marketing consent, loyalty timing, inventory management, revenue posting, regulatory submissions, data retention policies, all at the same level of detail, which buried the core checkout flow under noise, the thing I asked about.

That’s not much different from initial work with domain experts. When we’re starting discovery, people tend to start by explaining everything they do and feel is important. Quite often, that means you’ll get more domain-expert pet peeves in the surroundings than in the process descriptions.

You also get a lot of jargon, words that sound familiar but mean something different.

That’s also what I got from LLM, I asked it to review reference sources like:

popular tools documentation,
open specifications from the hospitality organisations,
some laws like GDPR related stuff,

I researched how different tools handle the guest checkout process. I knew it, and implemented it in past; it’s a surprisingly complex process, as you need to:

verify if the stay is fully paid and close the financial account,
generate invoice
mark guest’s stay as completed,
schedule full room cleanup,
mark in the inventory that the room will be available soon (unless the maid finds that it’s broken),
cleanup GDPR-related data that’s not needed to keep,
adjust loyalty points and update CRM,
etc.

I remembered how it works, but not all the details, and I would like to double-check whether anything in the industry has changed.

I wanted to get the overall vision first, then gradually dive deeper. As mentioned, I was overwhelmed with naming like:

Folio,
Drain Pending postings,
Property,
Account Receivables,

etc.

Don’t get me wrong, those were valid names in these domains. Sounds plausible if you’ve never worked in hospitality. Sounds plausible if you have, too. So what’s wrong with it?

They’re valid terms, as people actually use them, but weird, as they don’t tell “how the process works” but “how people do stuff,” and that’s not the same thing.

For instance, Property is a hotel, kind of makes sense. But Folio?

This name comes from the Oracle Opera. Yes, Oracle has a system in this domain, a dominating one. At some point, authors decided to name things this way, and now 30 years later, that vocabulary is baked into how thousands of hoteliers talk about their work. “Drain pending postings” is a phrase real cashiers say.

“Settle the folio” is a thing real cashiers do.

The problem: none of it explains how our system should work. It only tells what current systems do.

Also, when I asked about the business rules and policies, I got stuff like:

The system immediately posts room and tax charges for day-use reservations upon opening the Billing screen.

Why are transactions such as night stay charges or taxes added when we open the billing screen? That sounds counterintuitive, but maybe it was a fair tradeoff 30 or 20 years ago for an on-premises system installed in the specific hotel (ekhm “property|). Yes, Opera had (and maybe still had) consultants, similar to SAP. They’d go to the hotel, go to the back office, log in to the server and hack stored procedures in the Oracle database to fine-tune Opera behaviour.

Also, is there really a “Draining Pending Postings” option before doing checkout nowadays? Maybe it is, but in modern systems, we shouldn’t manually check all bills, etc., and explicitly pull them from integrated payment solutions. Nowadays, all the accommodation charges should already be recorded on the bill. The financial module continuously collects charges from payment gateways throughout the stay. There’s no separate moment of “draining”. The LLM invented a coordination command to match a phrase (“drain the interfaces”) that real cashiers use as shorthand for “let me check that nothing’s outstanding.”

As I asked, LLMs researched systems in this space: OPERA, Mews, Apaleo, and Cloudbeds. Each has its own vocabulary and mechanics, and the training data heavily favours OPERA because its documentation is everywhere. When I pushed back on terminology, the model would just swap in different OPERA jargon instead of actually thinking about what’s modern. The blending happened invisibly, mixing vocabulary from one system with mechanics from another, all delivered with equal confidence.

To be fair, the same happens quite often when talking to domain experts. They explain how it works now and what people do. Quite often, they bring us solutions instead of problems. Usually, solutions are based on their experience and how they see the updated version. This can be fine as a brain dump, but it’s not enough to translate it directly into the software design.

In Domain-Driven Design, finding and understanding the ubiquitous language is considered the most important aspect. Yet, Ubiquitous Language is not the source of truth. It’s the way to keep our heads from exploding due to the constant split-brain situation. It’s a tool to reduce cognitive load and the need for additional translation. That’s why we separate domain contexts and bind them to specific departments, people, and the language they use.

It’s fine to start with the current state of the art. Understanding how people do their job. We need to understand, though, that what we get is a mixture of habits (both good and bad), tribal knowledge, jargon, etc. If you ask different tribes, each will tell you something different.

Domain language is a cognitive tool, not gospel. LLMs compound this problem by reiterating competitor vocabulary without understanding the reasoning behind those systems, and they tend to align with whatever the prompter already believes.

Domain experts also struggle to define what they want and how it should work. That’s also why they hire us. It’s our job to help them and to transfer those sometimes contradicting visions into working software. That’s what we’re learning and what we do when modelling and step-by-step shaping the working software. That’s our work as engineers. We should work together to have a proper outcome. Collaborate.

And I’m not making the bold statement here that we’re smarter or we know better. We’re not; we have different expertise and different roles in the software development process.

Contrary to what many people believe in the DDD community, in my opinion, we’re not here to become domain experts; we’re here to build software.

The model we built doesn’t have to reflect the whole universe; it’s a way to take part of the business, understand, and automate it in the form of a software system. A software system is a tool for the business make more money. So software needs to be useful in a certain, defined way. Not all possible ways.

We need to learn how to communicate with business people, for instance:

Don’t use jargon or acronyms or assume someone should know something.
Understand that what someone wants to tell us might not be what they hear.
Do not take others’ behaviour personally.
Be assertive, critical and sceptical. Also, to our own judgments.
Be curious about the business domain. Don’t assume too much.
Don’t use “business won’t let me” as an easy excuse.

For some people, that’s too much. That’s probably why they try to ask LLM instead of Domain Experts, hoping that we won’t need to learn that, and we’ll get a solution for free.

We’re sometimes annoyed by being pushed hard by business people, I get that, but if we want to build something useful, that’s also where LLM will fail, as they will just agree eventually to what we believe. And that’ll probably be even worse than the skewed reality shown by the domain expert, since this will be Artificial Reality.

I keep hearing blank statements that “LLMs are great for research”. Blank because they usually are not followed up with what the author means by “great”. LLMs have many inherent limitations here. We’re anthropomorphising them too much. They don’t reason; they’re statistical machines. We should constantly remind ourselves of that.

I think people who claim that “LLMs are great at research” are just conflating their own skills and projecting them onto LLMs.

Hence, in my opinion, that’s why we’re getting those hot takes. Might be that they just have the skills to drill down, organise research, evaluate it, and model system design.

And I think this narrative is dangerous.

Because I, too, could sit down and say that I iteratively arrived at a solution thanks to LLMs. Write this article in a much different narrative, praising LLMs. I could say that if someone couldn’t get the same result, then that’s a skill issue.

But that wouldn’t be true, because someone without my experience in the domain and in modelling probably wouldn’t pick up on it.

Furthermore, it’s not necessarily true that I did it well; that would just be my perspective.

In practice, in this context, “LLM does it well” means “LLM does it the way I wanted it to.” And if the outcome is right, it is more likely that the “operator” had the necessary skill and used an LLM as a tool to speed it up.

The research I did looked at what our competition does and how they name things, but without internal knowledge of why those systems got to where they are. If our goal is to provide additional value to users, then just doing a blind copy won’t take us far. Instead of doing Lift and Shift, building something that has the same features but does better, we’ll get Lift and Shift with silent f.

Yes, LLMS can gather and compile multiple sources into a summary. That’s actually impressive and can spare us a lot of time. They’re also taught in the public documents, so some of their knowledge is built in. They look impressive on the surface, but not so if you look deeper. In the mentioned research, I got disconnected pieces of information, the techniques were randomly assembled with muddled technical and business language, and the whole thing didn’t tell a coherent story. Because of the randomness, you never know where the knowledge came from, what was omitted and what was skewed. With domain experts, you can at least understand the origin of their biases.

LLM also has limitations. The biggest is the context size, and being a yes-man. If we ask for the big picture, usually, we’ll end up with a swamp of information instead. If we don’t know how to sort this knowledge, what we’re looking for, and do a proper drill-down, then we won’t be able to untangle it. If we don’t have domain experts and don’t know the domain, how will we challenge issues like the ones I gave above?

I also keep hearing that LLMs are really useful in modelling. Sure, I tried that. I also asked LLM to try to model, and well, even after numerous iterations and using modelling tools, the results were mediocre. Mostly cliches and bad modelling practices.

The output muddied what we had before. The context was lost, and the process was oversimplified. When asked to apply specific tools like the C4 model, EventStorming, and Context Maps, it was forcing DDD patterns instead of describing actual processes. Even with precise instructions, it was building models with broken notation where rules floated outside the command-event chains, leaning on clichéd solutions, losing context between iterations. The domain was also presented as a big ball of mud. Concepts from room reservation bleed into the cashiering module. It was a recurring pattern across different versions of the same mistakes.

If you give too much context, they’ll mix and blend multiple conflicting vocabularies from different operational tribes trying to satisfy you. If you give it too little, it will come with simplistic hallucinations.

Is it hopeless then? Not at all, we can get help from LLM, but when we use it as a tool, not a replacement. We should not outsource thinking.

LLMs are useful for grasping the big picture and identifying known unknowns that we may miss as we get into the domain. We can learn about the language specifics, as mentioned in the Folio, Postings, Account Receivables, etc. LLMs can help us drill down into interesting aspects. We can get a brief understanding of domains we don’t know at all. But then we need to dive deeper and collaborate with our domain experts, real stakeholders. We need to understand what we want to build, organise our findings and focus on a certain context.

If we want to model our system, then LLMs can help us do boring work. It can organise our findings and create rapid text-based transitions with a defined format (including modelling practices). This can work. Yet it’s still our job to do domain discovery, refine the knowledge, evaluate and test it. We still need to focus on the feedback loop with real world.

We won’t hide from gathering modelling skills, we won’t skip the process of translating the domain into technical design, and we’ll still need to drive how and when to drill down. Those are engineering skills that were needed and will still be needed. And that’s fine, as if they wouldn’t, why would someone want to hire us?

I see that (for unknown reasons) collaboration is not discussed anymore. Working collaboratively with fellow humans has become something people dread rather than embrace. Maybe some people hope that they won’t need to talk to “domain experts” and “fight them”, because they “have everything here in LLM”.

And yes, they have all.

All the same issues they would have with domain experts.

As much as we shouldn’t trust domain experts blindly, but work with them, we shouldn’t blindly trust LLMs. We shouldn’t drop our engineering and design skills and outsource them.

If we want our software products to be better than the competition, simply blending their experience isn’t enough. We should focus on finding our special sauce, and I don’t see any other way to make it right than to work together and collaborate. We can use LLMs to help us do it faster, but as tools, not as solutions or replacements.

Check also:

Cheers!

Oskar

Vibing, Harness and OODA loop

Hey, have a look at what I made during the weekend. I had some time, grabbed a beer, turned on the computer and tried to code this feature. If I could do so much during the weekend, how much could you and your team do with it in 2 weeks?

It’s almost a 1:1 quote of what I heard from the startup founder I worked with over 10 years ago. I’m sure that you’ve heard similar phrases from people you worked with. We all know the annoying type of person who doesn’t code anymore but thinks, “I still got it!”. Then they threw a piece of stuff at you to “just fine-tune it a bit and do final touches”. Then they’re the first ones to ask “Why so long?“.

Nowadays, the Internet is full of such people. They shout about what they did with Claude or how much progress LLM tools have made. Some even predict the end of coding. I already wrote that this is wrong perspective. I won’t repeat that, but I want to say that…

Vibing isn’t new and isn’t always an issue.

I’m saying that LLM tools are an appraisal for ignorance. The more ignorant we are of the topic we’re working with, the better we see the outcomes. And that, by itself, is not always bad, as there’s power in ignorance if we focus on getting it done with the simplest tools we have.

Still, this can be terrible if we fall in love too much with what we’ve vibed.

To understand why that “weekend beer” energy is both a superpower and a liability, we need to look at the OODA Loop.

Disclaimer, it’s not a competition for Ralph Wiggum Loop. It’s much older and generic.

Military strategist John Boyd developed the OODA loop (Observe, Orient, Decide, Act) for fighter pilots. In a dogfight, the pilot who cycles through these four stages the fastest and most accurately survives.

In software, the “dogfight” is the gap between your intent and the production-ready feature.

OODA loop is built from four steps:

Observe - This is the intake of raw, unfiltered information. In our world, this means looking at the state of the system.
Orient - This is the most critical and difficult stage. It’s where you filter your observations through your experience, culture, and technical knowledge.
Decide - Based on your orientation, you formulate a hypothesis.
Act - You execute.

Getting back to my favourite founder and LLM-based tools.

The reason founder could build a PoC in a weekend while the team needed more than two weeks is that he bypassed the Observe and Orient phases. He went straight from a vague idea to Act.

If we skip or brush past the observation step, it feels like lightning speed. If the fancy UI grid is there and it does something we wanted, we move on. We’ve outsourced Orientation to our own ego. It’s too easy to assume that because we wrote it, it works.

Observation is the intake of raw data. In a professional environment, our eyes aren’t enough. We need a Harness. If we don’t have automations, tests, integration tests, and pristine traces, we aren’t observing the system; we’re just looking at it. If the inputs are messy, our observation is clouded.

But real engineering, the kind that takes those “two weeks”, is about closing the loop properly. That’s also where we need different perspectives and knowledge sharing.

Orientation is where you process those observations. This is the part where LLMs make us feel smarter than we are. If we don’t understand how a database handles concurrent connections, our “orientation” of a generated script will be shallow. We’ll see code that “looks” right, decide it’s fine, and act by deploying it.

The “I still got it” crowd loves the Decide and Act phases because that’s where the visible progress happens. LLM tools have made these phases nearly instantaneous. We can decide to build a feature and have the code for it in ten seconds.

The problem is that the faster we Act, the faster we need to Observe. If our “Act” phase takes seconds but our “Observe” phase requires a manual weekend of clicking around and drinking beer, our OODA loop is broken. We’re just generating a pile of stuff that we haven’t actually verified.

That’s why the team usually needs more than an imaginary “two weeks”. They are not “fine-tuning” the single-brilliant-dude masterpiece. They are building the infrastructure required to make the OODA loop sustainable.

And to make that possible, they need to run the full loop: Observe, Orient, Decide, Act. And do it multiple times. That takes time, but it’s required to assess the direction, automate what needs to be automated, and ensure they can iterate further and run this loop sustainably. That’s critical for delivering the outcome at the expected pace.

Of course, there’s a danger here, overfocusing on the Orient and Decide can lead to overengineering, building stuff we don’t need. That’s where ignorance can be blissful, especially when we connect it with humility. Being humble about what we don’t know and trying things the easiest way, then learning and making enhancements. Still, humility fails under deadline pressure. The harness doesn’t.

Let me give you…

The example

I’m adding proper Observability and Open Telemetry to Emmett right now. I spent some time working on it and instrumented the first component: Command Handling.

Of course, I had tests to prove it works, but I don’t trust them enough, and I wanted to try it on a real sample, since you never know until you run it. Even the best test suite won’t tell you all.

So I decided to plug it into the sample. See if it works, how ergonomic the API is and how it fits conventions in this area.

To do it, I decided to use Grafana stack and set it up with Docker Compose. So, stable, boring stack. Not going to lie, I vibed the config. Not that there are no docs, but I intentionally wanted to see the typical config people use.

If someone says LLM-based tools are great at proof of concepts, they don’t run the stuff they vibed. If I made the observation based on the initial config, then an oriented decision would be that it won’t work. Of course, then I did the typical back-and-forth, with the LLM tool doing some Linux command Voodoo to make it work. Once. Then, if you try to repeat it, you won’t know how to do it without doing Voodoo again.

Again, that’s not much different from the other stuff we do. I’m sure that you had multiple cases, when someone didn’t use Continuous Deployment tools, but clicked through Azure, AWS, GCP portal, deployed the stack, and then there was no trace on how to set it up again (e.g. to have a different environment for testing or demos for customers).

So, we need a harness, we need a leash to keep our process on track.

How to do the harness? My advice is to start simple. We may ask LLMs to give us shell scripts, and we may ask them to run them multiple times. We also need experience and knowledge of what we want to achieve and the tools we use. It’s fine not to remember all the YAML config to set up the Grafana stack, but it’s not fine not to understand why you even use it, how it relates, and how to set it up.

Still, our first loop can close on the first working solution, even a manually vibed one. But that’s not even a PoC. We need to automate them.

I asked LLM to take notes on what issues it had, and it solved them. Then, based on that, I asked to research how to code it in TypeScript. And to use tools I know, used in past, validating if there are no new more modern ones. For instance, I was a big fan of Gulp.js and Bullseye in the past, but they’re mostly dead. I wanted to have something in the same spirit, using native, maintained tooling.

I ended up with the following tools:

execa for running shell scripts,
native fetch for calling http endpoints,
native Node.js test tools for checking if the stack works as expected.

Then I asked it to create the script to automate the shell Voodoo they did to make Grafana stack and Docker Compose work.

Essentially, it should:

Run Docker Compose script starting up services (Grafana, Prometheus, Loki, Tempo, PostgreSQL, etc.).
Wait for them to check when they’re ready (it usually takes some time).
Start the application and make a request.
Check if the predefined dashboard with Emmett metrics appears, and shows expected traces and metrics.

Initial diagnostic tools looked like that

async function fetchWithDiag(label: string, url: string, init?: RequestInit) {
  const res = await fetch(url, init);
  if (!res.ok) {
    const body = await res.text().catch(() => '(could not read body)');
    console.error(`\n  ✗ ${label} → HTTP ${res.status}\n  body: ${body}\n`);
  }
  return res;
}

async function diagnoseCollector() {
  const text = await fetch(URLS.otelCollectorMetrics)
    .then((r) => r.text())
    .catch(() => 'unreachable');
  const emmett = text
    .split('\n')
    .filter((l) => l.startsWith('emmett_') && !l.startsWith('#'))
    .slice(0, 5);
  console.log(
    emmett.length
      ? `\n  collector /metrics (emmett lines):\n  ${emmett.join('\n  ')}`
      : '\n  collector /metrics: no emmett_* lines found',
  );
}

async function diagnosePrometheus() {
  const json = await fetch(
    `${URLS.prometheus}/api/v1/label/__name__/values`,
  )
    .then((r) => r.json() as Promise<{ data: string[] }>)
    .catch(() => ({ data: [] as string[] }));
  const emmett = json.data.filter((n) => n.startsWith('emmett_'));
  console.log(
    emmett.length
      ? `\n  Prometheus emmett_* metrics: ${emmett.join(', ')}`
      : '\n  Prometheus: no emmett_* metrics found yet',
  );
}

async function diagnoseLoki() {
  const labels = await fetch(`${URLS.loki}/loki/api/v1/labels`)
    .then((r) => r.json() as Promise<{ data?: string[] }>)
    .catch(() => ({ data: [] as string[] }));
  console.log(`\n  Loki labels: ${(labels.data ?? []).join(', ') || '(none)'}`);
}

async function diagnoseDockerLogs(service: string, lines = 10) {
  const { stdout } = await execa('docker', [
    ...COMPOSE,
    'logs',
    '--tail',
    String(lines),
    service,
  ]).catch(() => ({ stdout: '(could not get logs)' }));
  console.log(`\n  docker logs ${service} (last ${lines}):\n  ${stdout.split('\n').join('\n  ')}`);
}

Are they pretty? No. Can they be improved? Yes. Do they have to be improved at this specific moment? No.

The setup uses test infrastructure


const CLEANUP = process.env['CLEANUP'] === '1' || process.env['CLEANUP'] === 'true';
const CLEANUP_AFTER = process.env['CLEANUP_AFTER'] === '1' || process.env['CLEANUP_AFTER'] === 'true';
const NO_START = process.env['NO_START'] === '1' || process.env['NO_START'] === 'true';

// ─── configuration ───────────────────────────────────────────────────────────

const COMPOSE = ['compose', '-f', 'docker-compose.yml', '--profile', 'observability'];

const URLS = {
  app: 'http://localhost:3000',
  prometheus: 'http://localhost:9090',
  tempo: 'http://localhost:3200',
  loki: 'http://localhost:3100',
  grafana: 'http://localhost:3001',
  otelCollectorMetrics: 'http://localhost:8889/metrics',
};

// Fresh client per run — avoids stale cart state from previous runs.
const SERVICE_NAME = 'expressjs-with-postgresql';
const CLIENT_ID = randomUUID();
const CART_ENDPOINT = `${URLS.app}/clients/${CLIENT_ID}/shopping-carts/current/product-items`;
const CONFIRM_ENDPOINT = `${URLS.app}/clients/${CLIENT_ID}/shopping-carts/current/confirm`;

// Matches the .http file — unitPrice is resolved server-side.
const ADD_PRODUCT_BODY = JSON.stringify({ productId: randomUUID(), quantity: 10 });

before(async () => {
  console.log(`\n▶ client ID for this run: ${CLIENT_ID}\n`);

  if (NO_START) {
    console.log('▶ --no-start: skipping docker compose and app startup');
    return;
  }

  if (CLEANUP) {
    console.log('▶ --cleanup: killing port 3000 and tearing down stack (down -v)…');
    await execa('bash', ['-c', 'fuser -k 3000/tcp 2>/dev/null || true']).catch(() => {});
    await new Promise((r) => setTimeout(r, 500));
    await execa('docker', [...COMPOSE, 'down', '-v', '--remove-orphans'], {
      stdio: 'inherit',
    });
  }

  const stackReady = await fetch(`${URLS.prometheus}/-/ready`)
    .then((r) => r.ok)
    .catch(() => false);

  if (stackReady) {
    console.log('▶ observability stack already up — skipping docker compose up');
  } else {
    console.log('▶ starting observability stack…');
    await execa('docker', [...COMPOSE, 'up', '-d'], { stdio: 'inherit' });
  }

  console.log('▶ waiting for backends…');
  await waitFor(() => checkUrl('Prometheus', `${URLS.prometheus}/-/ready`), {
    timeout: 90_000, label: 'Prometheus',
  });
  await waitFor(() => checkUrl('Grafana', `${URLS.grafana}/api/health`), {
    timeout: 90_000, label: 'Grafana',
  });
  await waitFor(() => checkUrl('Tempo', `${URLS.tempo}/ready`), {
    timeout: 90_000, label: 'Tempo',
  });
  await waitFor(() => checkUrl('Loki', `${URLS.loki}/ready`), {
    timeout: 90_000, label: 'Loki',
  });

  // /health returns { status: 'ok', service: 'expressjs-with-postgresql' } —
  // checking service name lets us distinguish our app from other processes on :3000.
  const checkOurApp = () =>
    checkUrl('app /health', `${URLS.app}/health`, async (res) => {
      const json = (await res.json().catch(() => ({}))) as { service?: string };
      if (json.service !== SERVICE_NAME) {
        console.log(
          `    app /health: service="${json.service ?? '(missing)'}", expected="${SERVICE_NAME}"`,
        );
        return false;
      }
      return true;
    });

  const appIsOurs = stackReady && (await checkOurApp());

  if (appIsOurs) {
    console.log('▶ app already running and healthy — skipping npm start');
  } else {
    const portTaken = await fetch(URLS.app).then(() => true).catch(() => false);
    if (portTaken) {
      // Port is occupied but not by our app — stale process or unrelated service.
      console.error(
        '\n  ✗ Port 3000 is occupied by a process that is not this app.\n' +
          '  It may be a stale version of this app (connected to a wiped database)\n' +
          '  or a completely different service.\n' +
          '  Fix: run  npm run verify:observability:cleanup  to kill it and restart,\n' +
          '  or manually free port 3000.\n',
      );
      process.exit(1);
    }

    console.log('▶ starting app…');
    app = execa('npm', ['start'], { stdio: 'inherit' });

    await waitFor(checkOurApp, { timeout: 60_000, label: 'app /health' });
  }

  console.log('▶ setup complete\n');
});

As you see, nothing fancy, the cleanup is even simpler

after(async () => {
  if (app) {
    console.log('\n▶ stopping app…');
    app.kill('SIGTERM');
    await app.catch(() => {});
    console.log('▶ app stopped');
  }

  if (CLEANUP_AFTER) {
    console.log('▶ tearing down stack (down -v)…');
    await execa('docker', [...COMPOSE, 'down', '-v', '--remove-orphans'], {
      stdio: 'inherit',
    });
    console.log('▶ stack torn down');
  } else {
    console.log('▶ stack is still running');
    console.log('▶ to clean up: npm run verify:observability:cleanup');
  }
});

Having that we can run tests:


test('successful command returns x-trace-id header', async () => {
  const res = await fetchWithDiag('POST add product', CART_ENDPOINT, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: ADD_PRODUCT_BODY,
  });

  assert.equal(res.status, 204, `Expected 204 — body logged above`);

  const header = res.headers.get('x-trace-id');
  if (!header) {
    console.error(
      '  ✗ x-trace-id missing — verify the wrapper app in src/index.ts ' +
        'adds it via @opentelemetry/api before mounting the emmett app',
    );
  }
  assert.ok(header, 'x-trace-id header missing');
  assert.match(header, /^[0-9a-f]{32}$/, `"${header}" is not a 32-hex trace ID`);

  traceId = header;
  console.log(`  trace ID: ${traceId}`);
});

test('OTel collector exposes Emmett metrics on port 8889', async () => {
  // Send a few more requests so metrics are definitely recorded.
  for (let i = 0; i < 5; i++) {
    await fetch(CART_ENDPOINT, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: ADD_PRODUCT_BODY,
    });
  }

  try {
    await waitFor(
      async () => {
        let text: string;
        try {
          const res = await fetch(URLS.otelCollectorMetrics);
          text = await res.text();
        } catch {
          console.log('    collector :8889: connection refused');
          return false;
        }
        const emmettLines = text.split('\n').filter((l) => l.startsWith('emmett_') && !l.startsWith('#'));
        if (emmettLines.length === 0) {
          const allFamilies = [...new Set(text.split('\n').filter((l) => !l.startsWith('#') && l).map((l) => l.split('{')[0]))].slice(0, 5);
          console.log(`    collector :8889: no emmett_* metrics yet. Present: ${allFamilies.join(', ') || '(none)'}`);
          return false;
        }
        return true;
      },
      { timeout: 90_000, interval: 5_000, label: 'emmett metrics on collector :8889' },
    );
  } catch (err) {
    await diagnoseCollector();
    await diagnoseDockerLogs('otel-collector');
    throw err;
  }
});

I put it into a single file that can be run as a regular Node.js script.

It already showed me (and Claude) that what they initially did wasn’t working if you try to run it multiple times. It also showed that doing a full cleanup and rebuild, and making it reproducible, needs more work.

Is it done? Not yet; it takes too much time and resources to run it continuously throughout the pipeline. The code is a bit messy, so it needs to be organised. It’s segmented into blocks, includes basic automation and tests, and has already gone through some failures to get it done.

Could I do it better? Sure, but that’s not the point. I wanted to show you my findings during weekend vibing (without beer tho),the real, not polished iteration.

The main idea behind OODA loops is not to be perfect, but to iterate quickly, gather feedback as soon as possible, learn from it, develop another theory, and verify it through action.

It’s not about vibing, but it’s also not about analysis paralysis.

I hope you’re now better equipped to think about when vibing, with beer or without, with LLMs or without, actually helps, and when it doesn’t.

Vibe coding is just high-frequency steering. It only works if you have a Harness: a mechanical way to observe and orient, so you don’t steer the whole project into a wall.

Act takes seconds now. Observe takes as long as it always did. Without a harness, you’re not going faster; you’re just making more stuff you haven’t checked.

Harness is not magic, a new discipline, or the next buzzword; I hope I showed you that a bit in this article on what it may look like.

So iterate fast, but wisely remembering to do the full loop. It’s great that LLMs can help us make Acting faster, but we should not skip other steps. We should aim for a fast feedback loop to iterate in the right direction and achieve continuous improvement, to deliver proper value.

Just like Vibing isn’t new, we shouldn’t abandon ”old” engineering practices. We should also not replace collaboration with solitary self-high fives.

Check also:

Cheers!

Oskar

Yoda Principle for better integrations

Try not. Do. Or do not. There is no try!

I’m calling this the Yoda Principle.

Master Yoda said that to Luke Skywalker a long time ago in a galaxy far, far away. He was teaching Luke how to name commands properly while trying to untangle some legacy enterprise mess.

I’m sure you’ve also seen a death star of weirdly-named stuff. Some of them have already tripped you hours of thinking whether someone named this thing badly, or there’s some hidden truth behind it.

Let’s discuss that by the example: E-Commerce order fulfilment.

The order is placed automatically once the customer confirms the items in the shopping cart. We’re not making any product reservations before the shopping cart is confirmed, as this would lock it for other customers, and, as you know, they tend to drop items from their carts.

Once we receive the event notification that the cart has been confirmed, we’ll start the order fulfilment process. It starts (as mentioned) with the order initiation, which acknowledges and initiates the multi-step fulfilment process.

The first step is checking product availability before confirming the order. We need to determine whether we can proceed with completing the shipment and initiating product payment. If the product is unavailable, we need to either ask the customer to wait until we have it again or cancel the order.

We have dedicated modules for order fulfilment and for inventory. Fulfilment is the orchestrator, and inventory is responsible for tracking the state in warehouses.

The ordering module would need to call the inventory module to verify product availability. We could send a command (through the messaging system or web api). How would we name it?

What about VerifyProductExists? We’d send the product id and quantity from the order information, and return true if we have enough products, false otherwise. Sounds fair, right?

Well, it may seem nice at first glance, but what happens if more than one order verifies the same product availability, and we’re running short?

Then we’re vulnerable to race conditions I described in Tell, don’t ask! Or, how to keep an eye on boiling milk. The information we get is only valid at the time of querying. If we don’t lock the product quantity, it can change before we get a response (think: Black Friday-like demand).

Naming our command like VerifyProductExists is a mistake.

VerifyProductExists is not even a command; it’s a query. Command is a request (intention) to run business logic. Query is a request to return data.

Of course, pragmatically, our command can return status information about the result of our operation. But the intention is different.

What’s our real intention here?

The real intention is that we’d like to reserve products so we can ship them and get payment for them, not to verify if they exist. It’d be better to name our command as ReserveProducts or LockProducts.

Why does it matter?

If we’re naming our commands with Verify/Validate/Check prefixes, we’re putting ourselves into the wrong mindset. We’re not focused on actions and integrations, but just brief checking. If we’re in such a mode, it’s easy to handwave the integration complexity.

Locking for shipment may require sending someone to double-check that the product is in the warehouse and hasn’t been stolen, damaged, or other steps. It may be an async process on its own. Still from the ordering module, we should not care, as we’re telling what our intention is and expect to get events informing us whether the reservation succeeded, failed, or timed out (we don’t want to lock products forever in case of order fulfilment issues, but only for some time).

Prefixes like Verify/Validate/Check, etc., are just synonyms for trying. And well, commands are always a form of trying. The handling module can reject the command, as its business rules and state are the source of truth.

We should always assume that the command processing can fail. We should not be discouraged by that, and we should double-check everything. We should not be intimidated by the potential failure, but prepared for it.

We should try not. Do or do not. There is no try.

What if we have both? So VerifyProductExists and LockProducts? It can work if the first one is a query used by the Shopping Cart module, without any guarantee that the data isn’t stale, on a best-effort basis.

If we’re always requiring VerifyProductExists from the handling module before LockProducts, we’re making our communication chatty. I described that in What does Mr Bean opening the car have to do with programming? that this is not only a bad developer experience, but also just redundancy. Locking should already verify whether the product exists, so why require someone to memorise those scenarios instead of checking it internally?

The same goes for cases like:

verify payment has been made,
check if the order wasn’t already fulfilled,
validate if the shipment has been completed,
etc.

All of them either hide a missing business concept or should be a business rule verified within the specific action (e.g., confirming an order).

I recognise this may seem nitpicky, but big things are built on small details.

If we don’t think about such things, we’ll not only end up with misnamed integrations but also fight race conditions and incorrect boundaries.

Then we’ll be doomed.

So better think twice and do or do not.

May the force be with you!

Oskar

Anti-patterns in event modelling - Passive-Aggressive Events

Have you ever heard phrases like.

Just an update, the milk ran out. Someone finished it and put the empty carton back.

So everyone is aware, the meeting started 15 minutes ago.

Heads up: the coffee machine is empty again.

It’s fine, I already walked the dog.

I’m sure you either heard or used such phrases.

We all know that there’s some intention behind it.

The intention is not to inform, but to trigger some action.

Formally, we’re reporting on events to announce the facts, but in practice, they’re passive-aggressive words. The real intention is to command someone.

We don’t want to inform you that the trash bin is full, but we want someone to take it out. We don’t want to inform you that the coffee machine requires a coffee bean refill, but we want someone to do it.

Passive-aggressive tone is the worst. It’s toxic for both sides of the communication. Usually, it’s just better to ask someone to do it.

The same rule also works in Event-Driven Modelling. We should avoid passive-aggressive communication at all costs.

We should watch out for Passive-Agressive Events. So events that should be commands.

I already warned you in past: don’t let Event-Driven Architecture buzzwords fool you.

Event-Driven Architecture is an integration architecture style. We’re trying to model our business processes so they run smoothly. For that, we prefer a non-blocking communication flow, with things happening in parallel at their own pace. The goal is to achieve autonomous components that reduce the time needed to understand them. That helps maintain, or even replace them as your business evolves.

And events are enablers for that. They notify of what has happened, allowing other components to interpret facts and take the next steps.

But… Let me show one more photo.

It’s parliament, per the official definition: a room full of angry, shouting people.

If we model our communication only in terms of events, our system will look just like that. We’d just announce new facts in a passive-aggressive style and not be interested in what happens next. Oh, wait, are we really not interested? Actually, we are. If someone won’t do what we expect with our information, we’ll be even angrier.

What’s the difference between a command and an event? Both are messages. They convey specific information: a command indicating intent to do something, or an event describing what happened. From the computer’s point of view, they are no different. Only the business logic and the interpretation of the message can distinguish between an event and a command.

And that’s the main difference: commands can be rejected by the command handler. Events can only be ignored.

If we send an event, we expect someone to be interested, but we don’t know who or how many components will. We just inform.

This can easily change into passive-aggressive:

I did my work, now it’s your turn.

And here’s the crucial moment. If we’ll always have a single consumer for an event that needs to run the specific logic and expect to get the particular event back, then it should be a command. It’s not an event, we don’t inform. We just want someone to take the next specific step and let us know when they finish.

But hey, aren’t we making our communication synchronous?

What does it even mean, synchronous or asynchronous? That’s what Sam Newman discussed in his great talk. The main conclusion is that synchronous vs asynchronous discussion is actually about blocking or non-blocking. And that’s much broader than the technicalities of whether we call something in-process via an HTTP endpoint or a messaging system.

It’s a common misconception that events are published through a messaging system (e.g. Kafka, RabbitMQ, SQS, WhateverQueue) and commands are sent through WebAPI.

Those are technicalities. As said, both events and commands are messages; we can send them through a messaging system or via HTTP (e.g., events via webhooks).

This misleading split stemmed from our expectation about handling, so we expected the command to wait for the result. For events, we don’t expect a specific result, at least in theory.

If we publish a specific event to the messaging system and expect a specific critical path of follow-up events, then we’re not making our communication non-blocking. It’s still sequential. We cannot proceed until the expected sequence occurs.

Whether something is blocking or not is not established by the tools we use, but by how our business process looks.

Speaking about it.

Let’s get back to our favourite E-Commerce Order scenario (read more in Predictable Identifiers: Enabling True Module Autonomy in Distributed Systems).

We could model it so we just publish the OrderConfirmed event and passively-aggressively expect that others will take it from there. So:

The payment module will initiate the payment.
Inventory will start completing shipments.
The notification module will send a confirmation e-mail.
Fraud detection module will check if the order is not rigged.

Once we receive information about a successful shipment or payment registration, we can complete the order.

You may notice two paths for order processing:

Blocking - We need to wait for information about payments and shipments. This is our critical path.
Non-blocking- Order process shouldn’t stop if the notification wasn’t sent or the data warehouse wasn’t able to process events. We’d like that to happen, but it’s expected rather than critical.

Now, both payments can fail (if our customer doesn’t have enough money), and the shipment may not be completed (if it’s Black Friday, and multiple people are competing for the same product).

If that happens, as the ordering module, we also need to take action, for instance, do reimbursement if the shipment wasn’t completed, and eventually cancel the order.

If we’re just focused on events, we tend to forget about “negative” scenarios. If all communication is through events, then it’s too easy to stay in I-Alread-Did-My-Job mode.

We may not notice that another module can actually say no:

Payment module can say: Man, that’s not going to happen, you’ve already run out of money.
Shipment module can say: Man, I’m sorry, but you weren’t fast enough and we’ve run out of product.

And both of those scenarios will block successful order completion.

If we don’t foresee that and stay in passive-aggressive mode, this will have severe consequences: blocked orders, missed communication, and dissatisfied customers.

How to find such scenarios? Doing Example Mapping during modelling can be a good option for that.

Most importantly, we need to embrace the fact that some scenarios require direct, blocking communication, and others don’t. Just like in real life, sometimes it’s just more effective to tell someone to do something. We should avoid micromanagement and aim for autonomy, but not end up with anarchy.

In our case, it’d be better to have a coordinator (workflow, saga, process manager, etc.) that publishes the OrderConfirmed event for modules not on the critical path and sends commands like RecordPayment and InitiateShipment.

By that, we’re separating responsibilities and making explicit what should be explicit. This also helps in understanding the business process, as you have a central place to see the critical flow and get proper observability.

Lacking tracing and observability of the business process is one of the most common issues I see in my clients’ projects. As said, if we don’t want to end up with parliament instead of proper communication in our system, we need to be explicit about our intention.

Is that all? Not quite, there’s one more message type we model as events that should not be events.

Gregor Hohpe, in “Enterprise Integration Patterns”, besides Event and Command defines one more message type: Document.

What’s the Document? It’s a state. Or to be precise: self-contained data we have at a certain point in time. We can store it, but we can also publish information about its new value.

That’s probably why Martin Fowler frames it as Event-Carried State Transfer, and I don’t like that term. For me, it’s extremely misleading; it’s not an event, as it doesn’t tell of what has happened, but what has changed.

It’s a common anti-pattern, a variation of State Obsession. Too many people believe it’s fine to connect the messaging system to the database, use tools like Change Data Capture, and publish it automatically to others.

Still, we’re ending up again in passive-aggressive communication:

You have all you need. The whole state is in the events, just interpret it.

But how can you reason about what has happened if instead of OrderConfirmed you get OrderCreated, OrderUpdated, OrderDeleted?

You not only deal with Clickbait Events but also have a leaking business abstraction. All consumers need to understand the internals of your processing to detect a specific type of change. I wrote about it in detail in Internal and external events, or how to design event-driven API.

Again, the loose coupling of the event-driven processing is only loose for producers; consumers need to adapt. This can lead to hidden coupling, where a change in the producer breaks consumer flows. And that’s the worst type of coupling you can get.

If we’re making commands explicit, we’re also making an explicit relationship between components. It’s no longer flattened to producer <=> consumer, where the producer always shapes the communication. Now, if the other component exposes a command, that’s the driving force behind its behaviour. This helps to shape autonomy. In our case, we could make the Payment Module a generic module with a stable public API for registering payments, and an ordering module that requests them, in accordance with the Shipment Module. Fraud Detection could continue subscribing to events, as it already does. Context Mapping can greatly help in finding those relationships.

TLDR

We tend to be all about events these days, but they’re not the only message types. In our systems, messages take various forms: Events, Commands, and Documents, each serving distinct purposes:

Documents are all about state transitions, which are essential for syncing data across services but missing deeper business insights.
Commands represent a clear intent to act, directed with an expectation of execution, and can be accepted or rejected.
Events are immutable facts, announced without waiting for a response. They’re like broadcasting news, hoping it catches the right ears.

Event-Driven Architectures enable loose coupling, but only for producers. To make consumers loosely coupled, we need to take extra steps, embrace different message types, and have them participate in modelling business processes.

If we go too far with an event-all-the-things communication style, we’ll make our system a room full of shouting people, with a passive-aggressive communication style. Or just aggressive.

In consequence, we won’t know what’s happening in our system, will see only noise, and will have a hard time making it reliable, observable and predictable. We should treat our messages as a communication contract, API and model their flow in a way that shapes our regular communication.

So next time, ask yourself if your event shouldn’t be a command. If it has always had a single consumer and you expect a specific event back, then it’s probably so. It’s all about being clear about the intention, not lying to yourself and others.

I hope this article will equip you with the knowledge to fix that.

If you’re dealing with such issues, I’m happy to help you through consulting, training or mentoring. Contact me and we’ll find a way to unblock you!

See also more in series about event modelling anti-patterns:

Check also more general considerations:

Cheers!

Oskar

The one where Oskar explains Example Mapping

One of the first indications of getting old(er) is when people stop getting your movie or music references. Of course, based on this rule, some people are always old. That happens.

Recently, I realised during my workshops that referencing Friends is not so cool anymore. It started to happen when I was explaining the Example Mapping technique.

It always starts with “The one where”.

Just like in Friends.

I started to notice a bit slower head nodding and a bit more awkward smiles from the attendees. I repeated

You know, like in titles of Friends episodes.

And head-nodding stopped; only totally awkwardly polite smiles remained. Definitely, it wasn’t The One Where Everybody Finds Out. So I finally asked:

You weren’t watching Friends, didn’t you?

Obviously, the answer was:

Emmm. No…

Okay, then, if you don’t know Friends or the Example Mapping technique, this will be The One Where You Find Out.

Let’s say that we’re working on the guest checkout feature for a hotel management system.

We could start by asking the business how it works. We could get an answer that:

The guest approaches the desk and requests checkout. The clerk inquires about the quality of the products and services, and after receiving an answer, requests the room key. After gathering, the key clerk checks whether the balance is settled. If it’s settled, then proceed with the checkout. Marking the stay as completed.

Sounds straightforward, but we should already have several questions popping up, e.g. what does it mean that “balance is settled”? We could get quick feedback that:

This means that the difference between the sums of all charges and payments is equal to zero.

Then we could try to come up with an example:

Ah, so for instance, when guests haven’t paid upfront for their stay, right?

Right.

Oh, then we need to charge them, right?

Right.

We could visualise what we discovered in the following way:

Now, this generated another flow for us. We have a new feature we weren’t aware of: the guest’s stay payment registration. Let’s try to start this time from the visualisation.

It’s the one Oskar pays for his stay, because he wants to check out but didn’t pay upfront. The payment is registered, and we can try checking out again. Sounds fine, but we should ask whether there are any rules for payments. It may appear that:

Yes, there are some, for instance:

Only guests with a valid credit card can pay with it for their stay,

Guests paying in cash need to hand it over before accepting the payment.

And hey, we just found out new business rules, let’s put them on the board and update our flow to be more precise and reflect our scenario by adding a note that this scenario represents a guest paying with a credit card.

Now, what if the payment fails? Can it fail? Let’s ask the business!

Payment may fail if the payment gateway is unavailable or the issuer rejects it.

What if our internet connection is down for the whole day?

The clerk should ask the guest for cash.

What if the guest doesn’t have cash? 🤔

Then the shift manager can authorise unsettled balance checkout and register a charge with a delayed due date.

Here’s the updated flow. The one where Oskar pays for his entire stay with a credit card, but the Internet is down, and he doesn’t have cash.

Now, we found out:

A new outcome, failed payment,
A new rule, that we need Internet access to authorise credit card payment,
A new feature, the shift manager can authorise unsettled balance checkout and register a charge with a delayed due date.

How would the authorisation look? How should we register a delayed charge?

We should follow double-entry bookkeeping and register the authorisation (type of payment) with the unbalanced amount to settle the balance for today, and register an additional delayed charge for the same amount

The flow will look like:

And that’s precisely how the Example Mapping session looks like. It’s a structured conversation format created by Matt Wynne. You take a user story, gather a small group (usually a developer, tester, and someone from the business side), and spend around 25-30 minutes breaking it down together.

You don’t need a big setup, a huge ceremon, you don’t need sticky notes, you can just use plain text like:

Given: Example

When: We use specific features

Then: Based on business rules, we get a specific outcome.

Business people don’t need to give them to you in such form. You can use the interview as I showed above and note it on your own, while you’re discussing stuff. It’s also a nice way to collaborate and visualise your discussions.

You don’t even need to start with an interview; you can use the Example Mapping as a brainstorming tool to generate as many examples of your (part of the) system. Then, try to model it as you see fit and ask the business for clarifications in the preferred form. It can help facilitate discussion with your team, not only with business stakeholders.

It’s super helpful, as misunderstandings are expensive. When multiple people try to describe the same rule using real examples, you’ll quickly see where assumptions don’t match. Better to find that out in a 30-minute chat than after two weeks of coding the wrong thing.

It also works as a readiness check. Too many red cards? The story isn’t ready. Too many blue cards? The story is probably too big. If examples come easily and everyone nods along - you’re good to go.

There’s also more, as you can try to distil business rules based on the examples and outcomes the business describes.

It’s worth noting that I cheated you in colours. The original looks like this:

Why did I change them?

Example Mapping plays nicely with other collaboration techniques like Event Storming. And if you’re familiar with the Event Storming colour scheme, that’s also the reason why I used it. I aligned them. See:

I’m typically using it during modelling sessions to:

brainstorm (read more in Start Alone, Then Together: Why Software Modelling Needs Solitary Brainstorming),
challenging existing models with real-world examples,
expanding the model with uncovered (through examples) use cases,
finding business rules,
helping with facilitation by looking at the model from a different perspective.

And hot spots and notes, as known from EventStorming, are super helpful here. Read also more in The Underestimated Power of Hot Spots and Notes in EventStorming.

What’s more, if you look at the Given/When/Then pattern, you may notice that it works nicely with Behaviour-Driven Design. I already wrote that Behaviour-Driven Design is more than tests. How to do it? Check here.

I’ll also expand on it in the next articles. I’m doing the extreme Example Mapping with events, so stay tuned, the more will come.

For now, check also those materials:

And most importantly, try it. Take one feature from your system, try to crunch it, or start your design session with this technique.

Most of the teams I’m working with are enjoying this technique, as it’s a fun way to get quick, actionable outcomes.

Cheers!

Oskar

Interactive Rubber Ducking with GenAI

You may already know that I’m a GenAI sceptic. And a general sceptic.

Do you know that scepticism comes from the Greek σκέπτομαι (skeptomai), meaning ‘to search, to think about, or look for’? So my intention is not to say no to everything new, but more to think about it first, and understand before I say yes.

There’s a lot of stuff about GenAI that makes me smile, but I still understand that my way is my way, and I won’t stop the world. I won’t even try. Thus, I want to research and consider how those tools can help me. I already wrote that I don’t feel like 10x Dev, but I’m finding more ways to get help from it.

One of the ways that helped me is something I call “Interactive Rubber-Ducking”.

Initially, I called it just brainstorming, but that wouldn’t be precise, as I’m not using it to brainstorm ideas, more to challenge and clarify them.

Most of the code I write nowadays is done in my OSS projects. I’m grateful to have a great community with people actively contributing in different ways; still, the canonical design and code work is on my side. As I work in an event-driven niche, I’m often alone with my own thoughts. I try to use the RFC process and discuss it with other fellow humans, but they are not always available. Even if they do, to avoid wasting their time, I need to know what to tell or ask them. I need to give some proposals (with alternatives) to have an effective discussion. I may seem organised, but that’s not always the thing. Sitting in your own head is not a great place to be in general. If you’re a technical leader or an architect, I’m sure that you know that solitude too well.

GenAI tools are not great sparing partners. They’re Yes men. If they read this article, they’d for sure confirm it. They’d probably do it even without reading it. Of course, you can ask them not to be sycophant. You can ask numerous MUSTS with capital letters and bolded NEVER here and there, and it can help, but it won’t fully beat the way they were trained.

And talking to yourself is a similar experience: you’ll find numerous ways to justify your own decisions, and looking at the same place for too long will make you miss obvious blind spots.

Ok, so why would we take those two blind “people” and try to make them help each other?

That’s kinda what “Interactive Rubber-Ducking” is. It takes a blind human with an idea, and another blind not-so-human asking questions. It starts with such a prompt:

Ask me one question at a time so we can develop a thorough, step-by-step spec for this idea. Each question should build on my previous answers, and our end goal is to have a detailed specification I can hand off to a developer. Let’s do this iteratively and dig into every relevant detail. Remember, only one question at a time.

Once we are done, save the spec as spec.md

Before asking another question, store the previous one with the answer in qa.md. Write literally the question and answer, not just a summary.

Here’s the idea:

Don’t treat it as “one magical prompt that will change your life”. Most important is why we’re doing it, what happens next, and who’s actually doing the work. Spoiler alert: it’s not LLM.

I’m using it as a command in Claude Code, and, most importantly, with beefier models like Opus, which can better reason and ask better questions. Doing it with lower-level models always gave me much worse results.

I’m using it with Claude Code, not Claude Chat, because I want the model to scan my codebase. I can ask it to look in certain areas or to reference my answers. I can even ask to search the web or MCPs like Context 7 to check documentation and APIs for popular libraries. Then it’s getting more into brainstorming sometimes, than rubber-ducking, but that’s fine.

As a result, we’ll get two artefacts:

qa.md - with the log of the back-and-forth discussion,
spec.md - in theory spec built by LLM, but imho it’s more of a concise summary.

It may look like Specification-Driven Design, but it’s not.

My goal for this exercise is not to get an actionable specification.

The goal is to get our LLM-based Rubber Duck to ask us hard questions and make us think, not to make the LLM think for us. Find blind spots, and challenge our thinking.

But we’re drivers, we need to know what we want to do, we need to know all the WHYs, and we also need to know HOW. LLM is here to help, but not to do creative work for us. It just pulls it out from our heads.

It also helps to see how our design may be seen by others, especially such mediocre thinkers as LLMs.

I don’t expect the Agent to be able to start implementing the spec. I expect it to reflect all considerations and summarise findings. I’m always double-checking to make sure it includes all the important points. If not, I’ll keep doing Q&A until I’m satisfied.

Having both of those files will allow us to keep a full discussion without losing important details, and a shorter version. We can feed that to another model for review, or try to work on tasks and develop a more detailed plan. Sometimes I plan on my own; for simpler tasks, I may ask the LLM to do it fully. Usually, I’m driving the LLM step by step, passing just specific asks.

The example? Why not.

I recently did such an exercise, trying to narrow down how to introduce the Second-level cache to Pongo and Emmett. What’s Second-level cache? A Second-level cache is a local store of data managed by the persistence provider to improve application performance.

Why do I want to introduce it? Because I got an issue from the user that rebuilding projections with a lot events can take too long. One of the reasons is that applying an event on a projection takes:

loading the current state,
updating it,
storing the result.

Those are two operations per event. If we have a batch of 100 events, that’d mean 200 operations; for 1000 events, this would be 2000, so the classical N+1 problem. We could do it differently, and within a batch:

group events by the target documents,
load all of them in one operation by finding all documents within an array of ids (taken from events),
caching them,
applying events in memory,
storing updated documents in one operation.

Then we’ll get at worst 102 operations for 100 events and 1002 for 1000 events, so a linear increase.

Still, I decided to add batching and introduce 2nd-level cache as a solution. I and my experience were a driving force behind that. I could, of course, rubber duck it, but I had that already planned, not just detailed. I also did some initial research before doing the session.

Let me show you the result below. Not too bad if you ask me. Of course, the task wasn’t very innovative; it’s a standard way for mature solutions. Still questions were surprisingly good and helped me.

I hope this log will show you (besides how bad I’m at typos) some insight into how I think, talk, and consider the design.

Tell me your thoughts, if that matches your way, if you haven’t done it yet, try it and tell me how it went!

Interactive Rubber Ducking example

My idea

I’d like to provide 2nd-level cache to Pongo. I want to either use keyv (https://www.npmjs.com/package/keyv) or lru-cache (https://www.npmjs.com/package/lru-cache) packages.

For now, I’d like to do it by id.

I probably need to either extend pongoCollection src/packages/pongo/src/core/collection/pongoCollection.ts with findeOneById and findManyByIds methods, or check if _id is passed in and then try to access the cache.

Cache should be set up either on the pongo collection, pongo db or pongoClient.

Optimistic Concurrency errors should invalidate cache (as that means someone else updated the record in db, e.g. through a different node).

InsertMany and insertOne should put it into the cache.

I’d also like to expand the handle method to allow passing either an id or an array of IDs to support batch loading of entities and improve performance, and to support batch update/insert/delete.

Thoughts?

Which defaults should I set? (Of course, allowing users to override it).

Check the related code I pasted. I want to keep it aligned and as non-invasive as possible.

Q1: Cache library choice — lru-cache, keyv, or custom interface?

Question: You mentioned both keyv and lru-cache. They pull in different directions. Which trade-off do you prefer?

A) Start with lru-cache, optionally let users swap adapters later.
B) Start with keyv so cross-node sharing works out of the box.
C) Define a thin cache interface (get/set/delete/clear) and ship lru-cache as the default adapter.

Answer: Custom cache interface (option C). No dependency on external APIs — keep it pluggable. Start with an in-memory provider only. Considered writing a custom LRU with me, but wants to ship fast. Key point: don’t stick to any third-party API surface, allow pluggability.

Q2: Where should the cache live in Pongo’s hierarchy?

Question: Three attachment points: A) PongoClient level (shared across all), B) PongoDb level (per database), C) PongoCollection level (per collection). Or configurable at multiple levels?

Answer: Allow defining it at different levels — cascading configuration. Set a default higher up, override per collection if needed.

Q3: Inheritance/override behavior for cascading cache config?

Question: For the cascading config — what should the inheritance/override behavior be?

Two sub-questions:

Should a lower level be able to disable caching even if a parent enables it?
Should cache settings (like max size, TTL) also cascade, or only the cache provider instance itself?

Answer:

Yes. Cache should be an object with type, general params and specific for this type, or 'disabled'. That’s a proper devex.
Yes, as above. But if one selects a different cache type then general params should be inherited but type-specific should not be inherited and should fallback to defaults.

Claude’s thoughts: The CacheConfig | 'disabled' pattern is clean — discriminated unions play nicely with TypeScript. The inheritance rule makes sense: general params (TTL, max size) cascade, but type-specific params reset to defaults when you switch providers, avoiding leaking nonsensical config (e.g., a Redis connection string bleeding into an in-memory cache). Also worth distinguishing between “I didn’t specify anything” (inherit from parent via undefined) and “I explicitly want defaults” (reset) — undefined vs 'disabled' vs CacheConfig gives a clean three-state: inherit / off / explicit.

Q4: Cache interface shape — async, batch ops, scoping?

Question: Settling the cache interface details:

Async or sync? Making the interface Promise-based from the start keeps it pluggable. Sound right?
Batch methods (getMany, setMany, deleteMany) — should the cache interface support batch ops natively, or loop internally?
Should clear() be scoped per collection, or always clear the entire cache?

Answer:

Maybe we could return either promise or no promise, then if we await it, it would support both? Also ensure that settings are close to what lru-cache and keyv are having.
Having them (even as syntactic sugar) would be a good devex.
We could pass down not only settings, but also the cache provider instance, then if settings are the same we can reuse cache without recreating it. If set had ttl provided, then we could do it by overriding those TTLs on item level. ClearAll probably would be something we should expose only for advanced, typically user should not be aware of cache.

Claude’s thoughts: MaybePromise<T> (i.e. T | PromiseLike<T>) works — await on a non-Promise just resolves immediately. No overhead for sync providers, no breaking change for async ones. Passing a cache instance is smart: config becomes either settings (Pongo creates the cache) or a pre-built instance (user brings their own). Multiple collections can explicitly share one cache instance. clear() should be internal/advanced — scope by collection prefix internally so a collection-level clear doesn’t nuke another collection’s entries when sharing an instance. For aligning with lru-cache/keyv settings: both support max (max entries) and ttl (TTL in ms). Start with max and ttl as the general params.

Q5: Cache key strategy

Question: Since cache instances might be shared across collections, how to avoid key collisions?

A) Pongo manages key prefixing internally ({dbName}:{collectionName}:{id}). The cache interface works with plain string keys, namespacing is Pongo’s concern.
B) Key is just the document _id. Simpler but only safe with per-collection instances.

Answer: A, definitely.

Q6: Cache invalidation and consistency with optimistic concurrency

Question: How should cache interact with different operations?

Successful writes → update/evict cache. Clear.
Concurrency errors → evict stale entry. Clear.
Reads with _id filter → cache hit or miss-then-populate. Clear.
handle method — should the read inside handle use the cache (risking more concurrency errors on stale data) or always go to DB?

Answer: Definitely use cache. We could consider disabling cache through handle options.

Claude’s thoughts: Use cache by default — optimistic concurrency is the safety net. If cached version is stale, write fails, we evict, caller can retry. Adding skipCache?: boolean to HandleOptions is a clean escape hatch for high-contention cases.

Q7: Should `skipCache` be available on read methods too?

Question: Should skipCache also be available on findOne and other read methods for symmetry, or keep it handle-only?

Answer: Good point, having skipCache on find methods (also findMany etc.) would be good.

Claude’s thoughts: Consistent API surface. Add skipCache?: boolean on the shared options type (e.g. CollectionOperationOptions) so it’s available everywhere without repeating yourself.

Q8: Batch operations on `handle` — signature and handler shape

Question: You want handle to accept an id or array of ids. Two design choices:

Overloaded signature (handle(id: string | string[], ...)) vs separate method (handleMany)?
Handler shape for batch: A) still receives/returns one document at a time (Pongo batches DB ops internally) or B) receives/returns full array?

Also noted: since PongoDocuments always have _version, if someone provides expectedVersion we could compare it against the cached version and decide if it’s stale without hitting the DB.

Answer: Option A for handler shape. handleMany would be consistent with other APIs, but single id or array (string | string[]) is clean enough.

Claude’s thoughts: Version comparison in cache is a cheap short-circuit — if cached _version doesn’t match expectedVersion, we know it’s stale without a round trip. On signature: TypeScript overloads can narrow the return type (string → single result, string[] → array). handle is a higher-level orchestration method, not direct CRUD, so a union input feels natural. Go with handle(id: string | string[], handler, options?).

Q9: Defaults — max entries, TTL, enabled by default?

Question: What defaults to set? Proposed: max: 1000, ttl: 60_000 (1 min), and should cache be enabled by default or opt-in?

Answer: Stick to recommended lru-cache defaults for now. Enable cache by default.

Claude’s thoughts: Enabling by default works because: (1) optimistic concurrency is always the safety net, (2) it’s document-by-id only so invalidation is straightforward, (3) users can set 'disabled'. lru-cache requires max to be set (no default) and has no default TTL. Suggested: max: 1000, no TTL (rely on LRU eviction + invalidation on writes/concurrency errors) — most lru-cache-aligned.

Q10: TTL or no TTL as staleness backstop?

Question: No TTL means cached entries live until LRU-evicted or invalidated by a Pongo write. If something updates the DB outside this Pongo instance (another service, direct SQL), the cache won’t know. Options:

A) No TTL, accept the trade-off — OC catches it on next write.
B) Set a conservative default TTL (e.g., 5 minutes) as a staleness bound.

Answer: Fine with using TTL and sometimes hitting OC issue. OC assumes rare conflicts.

Q11: Where does cache integration hook into existing code?

Question: Two approaches for minimal invasiveness:

Wrap at the pongoCollection level — intercept operations inside the factory function, driver/SQL layers untouched.
Decorator/wrapper pattern — cachedPongoCollection wraps PongoCollection, adds cache logic around delegation.

Answer: Leaning more to option 1. Thinks it’ll be easier to make observability precise eventually.

Claude’s assessment: Agreed with option 1. Cache decisions depend on internal state (whether filter targets _id, document _version, write success vs concurrency error). A wrapper would need to re-inspect all of that from the outside. For observability, cache hit/miss logic next to the actual DB call enables precise metrics in context. Pongo’s factory functions already compose behavior internally (transactions, schema migration) — cache is another concern at the same level. A decorator would duplicate filter-inspection logic and add indirection that doesn’t carry its weight.

Q12: Cache interaction with transactions

Question: How should cache interact with transactions?

Should writes within a transaction update the cache immediately, or only after commit?
Should reads within a transaction check the cache?

Answer:

After commit.
We should use cache, but allow skipping it on transaction or session options. We can also allow setting cache settings per session.

Claude’s thoughts: Reads within a transaction benefiting from cache is more aggressive but logical — most transactions are short-lived. The narrow case: insertOne then findOne same _id within a transaction — cache won’t have the uncommitted insert, falls through to DB, gets the right answer anyway. No correctness issue, just no perf benefit for that case. Per-session cache settings fit the cascading pattern (e.g., startSession({ cache: 'disabled' }) for batch migrations). Proposed cascade priority: client → db → collection → session → per-operation skipCache.

Q13: Cascade priority — does session override collection?

Question: Does the cascade priority client → db → collection → session → per-operation skipCache feel right? Session overriding collection is the non-obvious part — the argument being that session is the most immediate runtime context, while collection is structural config. Or would you prefer collection always wins over session?

Answer: That’s fine. Session should precede collection, as normally you create session to e.g. share connection, and join the number of operations logically, so it’d make sense actually to also do it to e.g. override settings like caching.

Claude’s thoughts: Session as logical grouping of operations with shared context (connection, transaction) — overriding cache settings at that level is extending the same concept. Bulk import in a session? Set cache disabled once on the session instead of skipCache on every call. Final cascade: client → db → collection → session → per-operation.

Q14: Cache provider error handling

Question: If the cache provider throws (e.g., Redis connection drops), should Pongo:

A) Treat it as a cache miss — fall through to DB silently, maybe log a warning. Cache failures never break the application.
B) Propagate the error — let the caller deal with it.

Answer: A — yes, when we add observability to Pongo, we need to make it also a metric and trace event/attribute/log entry.

Claude’s thoughts: Cache is an optimization, not a correctness requirement. Silent degradation now, observable degradation when the observability layer lands.

Q15: Cache warming, events/hooks, and delete operations

Question: Three remaining areas:

Cache warming — should Pongo support pre-populating the cache on startup?
Cache events/hooks — should users be able to listen to cache hits/misses/evictions?
Delete operations — deleteOne/deleteMany should evict from cache. Anything non-obvious there?

Answer:

For now keep it as a future concept, or define pluggable strategy for users.
You can add the basic hook to the Pongo Cache wrapper. Shouldn’t be that hard to just call some function.
Yes, they should evict if they can, we can delete (for now) when deleteMany deletes by ids.

Claude’s thoughts: Cache warming deferred — can add a strategy hook later. Basic event callbacks (onHit, onMiss, onEvict, onError) on the cache wrapper are low-cost and useful for debugging even before full observability. Delete eviction scoped to id-based deletes for now — filter-based deleteMany doesn’t evict (no way to know which ids were affected without querying first).

Interactive Rubber Ducking result

Pongo 2nd Level Cache — Specification

Overview

Add a document-level (by _id) 2nd level cache to Pongo with a pluggable provider interface. Ships with an in-memory provider backed by lru-cache. Enabled by default. Designed to be minimally invasive to existing code.

Cache Interface

`PongoCacheProvider<T>`

A thin, Pongo-owned interface. No coupling to any third-party API.

type MaybePromise<T> = T | PromiseLike<T>;

interface PongoCacheProvider {
  get(key: string): MaybePromise<PongoDocument | null | undefined>;
  set(key: string, value: PongoDocument, options?: { ttl?: number }): MaybePromise<void>;
  delete(key: string): MaybePromise<void>;
  getMany(keys: string[]): MaybePromise<(PongoDocument | null | undefined)[]>;
  setMany(entries: { key: string; value: PongoDocument; ttl?: number }[]): MaybePromise<void>;
  deleteMany(keys: string[]): MaybePromise<void>;
  clear(): MaybePromise<void>;
}

MaybePromise return types: sync providers (in-memory) return values directly, async providers (Redis) return Promises. await handles both transparently.
Batch methods (getMany, setMany, deleteMany) are first-class. Default in-memory implementation may loop internally, but the interface allows optimized batch ops for external providers.
clear() is internal/advanced — not exposed to typical users. When sharing a cache instance across collections, scoping is handled via key prefixing by Pongo, not by the provider.

Cache key strategy

Pongo manages key prefixing internally: {dbName}:{collectionName}:{documentId}.

The cache provider works with plain string keys — namespacing is Pongo’s concern, not the provider’s.

Event hooks

The Pongo cache wrapper supports basic callbacks:

onHit?(key: string): void
onMiss?(key: string): void
onEvict?(key: string): void
onError?(error: unknown, operation: string): void

These are optional and intended for debugging and future observability integration.

Configuration

`CacheConfig`

type CacheConfig = {
 type: string;               // e.g., 'in-memory', 'redis', etc.
 max?: number;               // max entries (general param, cascades)
 ttl?: number;               // TTL in ms (general param, cascades)
  // type-specific options live here too, keyed by type
 [key: string]: unknown;
} | 'disabled';

Three states:

undefined — inherit from parent level
'disabled' — explicitly turn off caching at this level
CacheConfig object — explicit configuration

Cascading configuration

Cache config can be set at multiple levels. Each level inherits from its parent unless explicitly overridden:

client → db → collection → session → per-operation

Inheritance rules:

General params (max, ttl) cascade down.
If a lower level switches type, type-specific params reset to defaults (not inherited from parent).
Session overrides collection — session is a logical grouping of operations, natural place to override runtime behavior (e.g., disable cache for a bulk import).
Per-operation skipCache?: boolean is the most granular escape hatch.

Passing a cache instance

Users can provide either:

Settings — Pongo creates and manages the cache provider.
A pre-built cache provider instance — Pongo uses it directly.

If settings are the same across multiple collections, Pongo can reuse the same provider instance internally. When a user passes an instance, multiple collections can explicitly share one cache.

Defaults

Enabled by default
max: follow lru-cache recommended defaults (1000)
ttl: follow lru-cache recommended defaults
Default provider: in-memory (lru-cache)

Integration points

Where: `pongoCollection` factory function

Cache logic is added directly inside pongoCollection, not as an external decorator/wrapper. This gives cache operations access to internal state (filter inspection, _version, write outcomes) and keeps observability precise.

Read operations

findOne:

If the filter targets _id, check cache first.
Cache hit → return cached document.
Cache miss → query DB, populate cache, return.
skipCache?: boolean option available.

findMany / other query methods:

If the filter is a list of _id values, check cache for each.
Return cached hits, query DB for misses, populate cache with DB results.
Non-_id filters bypass cache entirely (cache is by-id only).

Write operations

insertOne:

After successful insert, put the document into cache.

insertMany:

After successful insert, put all documents into cache.

updateOne / updateMany / replaceOne:

After successful write, update the cache entry with the new document state.

deleteOne:

After successful delete, evict from cache.

deleteMany:

If deleting by ids, evict those ids from cache.
Filter-based deleteMany does not evict (no way to know affected ids without extra query). Future improvement possible.

Optimistic concurrency

On concurrency error (version mismatch), evict the stale entry from cache. This is critical — a concurrency error means someone else updated the record (e.g., from another node), so the cached version is stale.
If a caller provides expectedVersion and the cached document has _version, compare them in memory. If they don’t match, we know it’s stale without hitting the DB — cheap short-circuit.

`handle` method

Accepts id: string | string[] (overloaded: string returns single result, string[] returns array).
Read phase uses cache by default.
Write phase updates/evicts cache based on outcome.
skipCache?: boolean available in handle options.
For batch: loads all ids from cache, fetches misses from DB in one query, processes handler per document, batch writes to DB, updates cache after commit.

Transactions

Writes update cache only after commit. Not during the transaction — uncommitted data should not enter the cache.
Reads within a transaction check cache by default (most transactions are short-lived, cached data is probably valid). If the transaction modified a document and then reads it, cache miss falls through to DB which returns the correct uncommitted state.
Cache can be disabled per-session: startSession({ cache: 'disabled' }).
Per-session cache settings follow the cascade (session overrides collection).

Error handling

Cache provider errors are swallowed — treated as cache misses. The application never fails because of a cache failure.

Cache is an optimization, not a correctness requirement.
Errors trigger onError hook for debugging.
Future: errors will become metrics and trace events when Pongo’s observability layer lands.

Future considerations (out of scope for v1)

Cache warming: pluggable strategy for pre-populating cache on startup. Interface hook defined but no default implementation shipped.
Query-result caching: caching results of non-_id queries. Complex invalidation — deferred.
Cross-node cache sharing: Redis/Memcached providers. The interface supports it; no provider shipped in v1.
Filter-based deleteMany eviction: would require querying affected ids before delete.
Observability integration: metrics (hit ratio, latency saved, eviction counts), trace attributes, log entries for cache operations.

Implementation approach

Define PongoCacheProvider interface and CacheConfig types.
Implement in-memory provider using lru-cache.
Build Pongo cache wrapper (key prefixing, event hooks, error swallowing).
Wire cascading config resolution (client → db → collection → session → per-op).
Integrate into pongoCollection: intercept _id-based reads, update cache on writes, evict on deletes and concurrency errors.
Extend handle to accept string | string[] with batch support.
Add skipCache to operation options.
Tests: unit tests for cache provider, integration tests for cache + collection operations, e2e tests for concurrency scenarios.

Cheers!

Oskar

p.s. and yes Nano Banana also did pretty well generating the cover of this post!

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

The End of Coding? Wrong Question

Be careful what you wish for, because your wish may come true.

What LLMs revealed is how many people in our industry don’t like to code.

It’s intriguing that now they claim and showcase what they “built with Claude”, whereas usually that means they generated a PoC.

It’s funny, as people still focus on how they’re building, so it’s all about the code. And if that’s the message sent outside, together with the thought that LLMs are already better than “average coder Joe”, then the logical follow-up question is: why do we need those humans in the loop?

I think that most people look at the forest and see trees. The current way of working with LLMs is not scalable. It’s transition phase. I can’t imagine calling myself an engineer and doing ONLY stringly-typed development with chat or markdown.

What I can imagine is getting help from it, and using those tools as a help for research, for generating the OUTPUT, still keeping me responsible for the OUTCOME.

Why am I saying that we’re in the transition phase? As prompting in our natural language is not precise, it’s verbose, and adding a translation layer between our freeform prompts and programming languages is a waste of time (and tokens, which LLM vendors love).

I think that we’ll still be coding, but with some other layer, as LLMs are good with structured input, like programming languages. So we might need other programming languages than we have atm. Might we need different tools to evaluate LLMs’ output to make it deterministic? Might we need a different approach for engineering to make it scalable? Might we need more?

Still, I don’t see those discussions.

I mostly see: noise and celebrities who don’t code showing their beautiful PoCs. And doing a mic drop about the end of coding. Just like PoC represents the whole Software Development Life Cycle.

So, will we code or not?

If the answer is yes, let’s talk about what’s next, then let’s discuss how and when.

If the answer is not, then let’s talk about how our job will change, ask if it’s still engineering, etc.

Let’s try to make our discussions more precise, more focused on the essence, and avoid them from becoming just a bunch of anecdotal evidence.

Noone outside our industry cares how we code unless it changes the cost, quality, or delivery time.

Let’s discuss the impact that matters, rather than just the amount of code we produce (or not).

Now, guess who I am quoting:

Imagine you’re a software application developer. Your programming language of choice (or the language that’s been foisted on you) is Java or Typescript. You’ve been at this for quite a while and your job doesn’t seem to be getting any easier.

These past few years you’ve seen the growth of multiple incompatible architectures. Now you’re supposed to cope with all this and make your applications work in a distributed client-server environment. The growth of the Internet, the World-Wide Web, and “electronic commerce” have introduced new dimensions of complexity into the development process.

The tools you use to develop applications don’t seem to help you much. You’re still coping with the same old problems; the fashionable new object-oriented techniques seem to have added new problems without solving the old ones.

You say to yourself and your friends, “There has to be a better way”!

The Better Way is Here Now

Now there is a better way—it’s our new model. Imagine, if you will, this development world…

It’s still dead simple.

Your development cycle is much faster.

Your applications can be created across multiple platforms. Write your spec once, and you never need to port them—they will be recreated if you without your hand-rolled modification on multiple operating systems and hardware architectures.

Your applications are adaptable to changing environments.

Your end users can trust that your applications are secure, and you can use protection against viruses and tampering through security scans.

You don’t need to dream about these features. They’re here now.

And also this from another source:

When I started interviewing programmers in 2005, I would generally let them use any language or tool they wanted to solve the coding problems I gave them. 99% of the time, they chose Java.

Nowadays, they tend to choose LLMs.

Now, don’t get me wrong: there’s nothing wrong with LLN as an implementation tool.

Wait a minute, I want to modify that statement. I’m not claiming, in this particular article, that there’s anything wrong with LLM as an implementation tool. There are lots of things wrong with it but those will have to wait for a different article.

Instead what I’d like to claim is that LLM is not, generally, a hard enough programming tool that it can be used to discriminate between great programmers and mediocre programmers. It may be a fine tool to work in, but that’s not today’s topic. I would even go so far as to say that the fact that LLMs aere not hard enough is a feature, not a bug, but it does have this one problem.

Well, I cheated you, but only a bit. I changed “Java” to “LLM ” and cut some phrases.

The first one is from “The Java Language Environment” by Sun Microsystems, introducing Java in 1995.

The second one was from Joel Spolsky’s “The Perils of JavaSchools” article, written in 2001.

Let me be clear. I’m not trying to do grandpa talk on the old days and claim that it’s the same old thing.

What I’m trying to say is that we were continuously introducing new abstractions into our development cycle to scale it. By scale, I mean: getting more people to deliver more code. Even Java was invented for precisely this goal. Yes, the one that’s together with craftsmanship madness presented as “the enterprisy complex environment”. In the early days, it was just said that it’ll make us dumber.

The goal of abstraction is not to gatekeep but to allow us to reduce cognitive load. We invented new languages to help us, but then added more components like a distributed environment, multiregion, because of globalisation. For the same reasons, we use cloud-native tooling so we don’t have to deal with it.

Some of the architecture and security tools were commoditised by the cloud. We don’t need to think about much stuff we had to do before.

Will Claude-native do the same?

We don’t need to learn Assembler, C++, Lisp anymore; we have lost a lot of mechanical sympathy. We deal with higher abstractions, but we still engineer solution, we still code, is it a different coding? It is. Is it better or worse? Well, it is how it is. As Gerald Weinberg said:

Things are the way they are because they got that way

And now the question is whether we’re fine with the way we’re doing stuff, and where we’re getting to.

Personally, I don’t think that stringly-typed markdown or chat-based design will be the thing in the future.

Knowing how skilled we were always with:

breaking down tasks into smaller chunks,
writing precisely what we had in mind,
thinking before doing,
waging tradeoffs,

I’m optimistic about the Spec-Driven Design idea. It’s going to be great.

Not.

If we want to call ourselves engineers, we need to put more structure and determinism.

I agree that reviewing all code generated by the GenAI is not sustainable.

But I also don’t think that generating tons of code is, in general, sustainable.

With the current state of the art we have, sure, that’s some solution to just generate based on the tools we have.

But let’s start to think what’s next.

Simon Wardley brought an interesting point:

That said, it is fine for an entire culture to decide that producing outputs matters more than understanding mechanisms. You only have to compare the practical engineering of the Roman Empire and the loss of inquiry from science in the Hellenistic age to see this. When the Roman Empire collapsed, the practical knowledge embedded in those institutions (how to maintain aqueducts, how to produce certain grades of concrete) was lost remarkably quickly. Not because people decided to forget it, but because the knowledge was procedural, embedded in chains of practice rather than recorded as transferable understanding. When the chains of practice broke, that embedded knowledge went with them. We had to rediscover the art of inquiry (i.e. Science) to bring them back.

So if we don’t code, how do we hone our skills? How will newcomers from LLMSchools (using Joel’s terms) be able to decide whether something is wrong or right? I don’t think that you can be good on something without doing it.

The paradox with code is that it’s not an asset; it’s a liability. Some say that code doesn’t matter, only proper design. But how do you define “proper design”? Yes, code style doesn’t matter as long as it works as expected. But…

But code eventually matters, as that’s the source of truth for what’s on production. As Alberto Brandolini said:

It’s developers’ (mis)understanding, not domain experts’ knowledge, that gets released in production.

Now, it’s the developers’ and LLMs’ misunderstandings that are deployed to production, not the expert’s knowledge. Neither the markdown spec.

And coding is just one danger.

Outsourcing thinking is an even more dangerous path, as:

If LLMs are doing everything, then what are humans for? Aren’t we cutting the branch on which we’re sitting?
LLMs are statistical parrots. They repeat the most possible answer. Which means mediocre. This can still be fine enough for many cases, but for those we want to make a difference for? Definitely not.
Just like we’re losing our coding skills by not doing them, we’re losing design skills by not practising them.

Of course, whatever happens, LLMs will stay with us. How and where it’s hard to say. Unless you have a Magic 8 Ball of 100% correct predictions. I don’t.

That’s why I’d like our industry to finally start mature discussions on the real impact. I would like us to stop acting like children, bragging about generating code and then claiming that code doesn’t matter.

I’d like to think about how to reshape our SDLC process and make it sustainable.

I’d like us to think about what tools we need, and how to change what we have.

If we won’t finally start to do it, then things will be the way they are because they got that way.

And it might not be what we wished for.

Cheers

Oskar

p.s. to kinda prove that I’m more sceptic and pragmatic than hater, I recently started playing with building an Agent with Emmett to better understand those tools. If you’d like to read about the findings and honest thoughts I have while doing it, please comment!

Parse, Don't Guess

Last time, I shared with you how sneaky I was on transaction handling.. Today, the opposite: I’ll tell you how I fixed the issue when I tried to be too sneaky. I already told you that Sneaky Code Bites Back. The moral? Do as I tell, not how I do.

In some environments, we’re spoiled. We’re getting a lot from a Base Class Library or standard frameworks, so we stop thinking that those issues can exist. For instance, serialisation. Do you know how many data types JSON has? 6. Six. Sześć.

Exactly those:

string,
number,
boolean,
object,
array,
and (TADA!) null.

What about number precision and size? It is. That’s what I can tell you, but it’s not enough, e.g., to keep big int/long, etc. What about Dates? Also, there are none. I wrote about it longer here or how much fun that brings.

If you use statically typed languages and runtimes like C#, Java, etc., your serialiser can, in addition to parsing, perform mapping and, sometimes, validation. And it can also be tricky, as nicely Alexis King put in her “Parse, don’t validate”.

If you’re in a dynamic environment, like JavaScript, then you’re left with parsing and explicit mapping afterwards. What about TypeScript? Same case, types are only used during compilation, then erased and not visible at runtime. So, the place where we do parsing.

Because JSON was defined a long time ago, JavaScript moved on and now supports bigints (Big Integers) and Dates natively (what an achievement!), which created a gap I wanted to fill.

As you know from my previous articles (e.g. this one), big integers are quite important in distributed processing. You can represent the position in log with them. Since your log may be quite long, regular numbers aren’t enough. Or they’re long enough, until they overflow, then they’re not anymore.

I’m using those bigint types extensively in internals in Emmett and Pongo. And I store them in JSONs. I store them as alphanumeric strings, because strings don’t have a max length (or at least I don’t know such).

So, for instance, an event payload can look like:

{
  "type": "InvoiceIssued",
  "data": {
    "invoiceNumber": "123",
    "version": 1,
    "issuer": "John Doe",
    "issuedAt": "2026-02-23T14:07:20Z"
  },
  "metadata": {
    "streamPosition": "3",
    "globalPosition": "928391"
  }
}

As you can see in the metadata stream and global positions, the values are bigints (even if they’re smaller than the maximum value), and data can also use bigints if the user decides to (e.g., invoice number).

And encoding data is simple: you convert it to a string, call it a day. But how to get it back?

And here’s where my struggles started. How do you know that someone intentionally used bigint when they just wanted to store a number as a string?

There are several options. The first one is: encode value.

We could store it, for instance, as:

prefixed value: “_bigint:928391”. But then you need to find a prefix that will be unique enough not to cause conflicts,
nested object, e.g. { “_kind”: “bigint”, value: “928391” }.

Then, either based on the prefix or the object structure, we could automatically decode the value. Still, ther creates other issues, as the structure no longer matches the original value. If we’re just storing and retrieving, that shouldn’t be so bad, but… But remember that in Pongo I’m allowing the use of PostgreSQL and SQLite as document databases, supporting such queries:

const invoices = pongoDb.collection<Invoice>("invoices");

const invoiceNumber = 123n;
const invoice = await invoices.findOne({ invoiceNumber });

That gets translated into a fancy JSONB SQL query.

Of course, I could work around it by encoding the value, but… But I was lazy!

I decided to use a Get Out of Jail Free Card and just treat all strings with numbers as bigints. Sneaky. And it will get even sneakier.

In JavaScript, JSON.parse accepts a parameter that allows you to provide custom mapping logic. I decided to use it and check if the string is alphanumeric, and gulp, I’ve used a regular expression to parse it:

const bigIntReviver: JSONReviver = (_key, value) => {
  if (typeof value === 'string' && /^[+-]?\d+n?$/.test(value)) {
    return BigInt(value);
  }

  return value;
};

Yes, it’s either DNS or Regex. Or both.

I explained in another article that JavaScript runtime doesn’t like where you do CPU-heavy computations.

Small Regex isn’t CPU-heavy, but if you consider that ther will be done for each string in each document or event you try to deserialise, and multiply that by the number of concurrent requests? That can cause the JavaScript event loop to freeze.

What’s more, I plugged that automatically into node-postgres driver custom type handling, so each JSONB deserialization goes through it.

Again, not shit, Sherlock. I should have known it wasn’t the best choice, but I was busy trying to be sneaky at that moment.

Fortunately, a user, Dawid, benchmarked and noticed CPU freezes. It wasn’t catastrophic, but clearly needed a fix.

The Shift

And here I had several options. I could keep hacking on the same idea- maybe replace the Regex with a simpler string check, still globally. Or I could just ignore bigints during deserialisation entirely, let them stay strings, call it a day, move on. Or I could apply the encoding I mentioned earlier, prefixed values, and nested objects. All of those would fix the performance issue. And all of those would be the same kind of wrong choice I already made: trying to solve a schema problem without the schema.

Because that’s the actual mistake here, not the Regex. The pg driver has no idea what your schema looks like. It doesn’t know that “928391” is a bigint and “John Doe” is a name. It doesn’t know that “123” is an invoice number (bigint!) and “90210” is a zip code (string!). I asked it to guess, and it guessed wrong, because there is no right guess at that level.

Enough is enough. I had been planning to do ther properly for a while, and the performance issue gave me the push I needed.

Old rule says: “Make it work, make it right, make it pretty”. I had “make it work” covered for a long time. Now it was time for “make it right”.

And honestly? It wasn’t that hard. Maybe because “make it work” came first, I already understood the problem well enough to see the shape of the solution.

In Pongo, I dropped the automatic bigint parsing from the driver entirely. If you want bigint or date parsing, you say so at the client level:

const client = pongoClient({
  driver: databaseDriver,
  connectionString: postgresConnectionString,
  serialization: {
    options: {
      parseBigInts: true,
      parseDates: true,
    }
  },
});

By default, strings stay strings. You opt in. I didn’t want to break things for users who don’t care about bigint precision or don’t have performance-sensitive workloads. The serializer became an explicit parameter passed down to each query, each collection, each operation- instead of a global that silently changed everything.

That was the “make it right” part for Pongo. But disabling alone isn’t a solution, it’s a band-aid. Users who need bigints and dates still need a way to get them back after deserialisation. The question is: where does that conversion happen?

And that’s where upcasting comes in. Let me start with a simple example in Pongo, then build up.

Say you have a user document. In the database, dates are stored as ISO strings, and the version counter is a numeric string (because JSON). But in your application, you want proper Date objects and bigints:

type UserDocStored = {
  name: string;
  createdAt: string;
  lastLogin: string;
};

type UserDoc = {
  name: string;
  createdAt: Date;
  lastLogin: Date;
};

The upcast function does the conversion:

const upcast = (doc: UserDocStored): UserDoc => ({
  name: doc.name,
  createdAt: new Date(doc.createdAt),
  lastLogin: new Date(doc.lastLogin),
});

You wire it into the collection, and every read goes through it:

const collection = pongoDb.collection<UserDoc, UserDocStored>(
  'users',
  {
    schema: { versioning: { upcast } },
  },
);

What’s in the database:

{ "name": "Alice", "createdAt": "2024-01-15T10:30:00.000Z", ... }

What you get back:

{ name: 'Alice', createdAt: Date, ... }

That’s all. new Date(str) is cheap. Running a Regex against every string in the document is not. The CPU freeze Dawid spotted came from that check running millions of times at the driver level for every field on every concurrent request. With upcasting, the conversion runs only for the fields you declared, in a plain function, no Regex.

But ther is just type mapping - the simplest case. As I wrote about in my serialisation article, the explicit mapping pattern is useful for much more than just fixing types. It’s the same pattern you need for schema versioning. It defines the stored schema and the application schema separately together with function to transform one into the other.

Let’s say business requirements changed. You now need to group user data differently: a profile object for identity data, and a timestamps object for temporal data. The V1 documents are flat. The new V2 shape is nested:

type UserDocV1 = {
  name: string;
  createdAt: string;
  lastLogin: string;
};

type UserDocV2 = {
  profile: {
    name: string;
  };
  timestamps: {
    createdAt: Date;
    lastLogin: Date;
  };
};

Ther isn’t just a type change like string-to-Date anymore. The structure itself is different. Flat fields became nested objects, and field names moved into sub-objects. And you have thousands of V1 documents already stored. You can’t migrate them all at once (or don’t want to, because it’s risky, and some consumers might still expect V1). But your application now expects V2.

Compatibility FTW

Ther is where backward and forward compatibility come in.

Backward compatibility means: old data still works. V1 documents stored months ago need to be readable by the V2 code. The upcast handles ther. It reads the document in whatever shape it has and transforms it into V2.

Forward compatibility means: new data doesn’t break old consumers. If you have another service or an older deployment that still reads the V1 format, it needs to keep working. The downcast handles ther. When storing V2 documents, it writes the V1 fields alongside the V2 fields, so older readers can still find what they expect.

Together:

type StoredPayload = UserDocV1 & UserDocV2;

const upcast = (doc: StoredPayload): UserDocV2 => ({
  profile: doc.profile ?? { name: doc.name },
  timestamps: {
    createdAt: new Date(doc.timestamps?.createdAt ?? doc.createdAt),
    lastLogin: new Date(doc.timestamps?.lastLogin ?? doc.lastLogin),
  },
});

const downcast = (doc: UserDocV2): StoredPayload => ({
  name: doc.profile.name,
  createdAt: doc.timestamps.createdAt.toISOString(),
  lastLogin: doc.timestamps.lastLogin.toISOString(),
  profile: doc.profile,
  timestamps: doc.timestamps,
});

Look at the upcast: if the nested profile or timestamps fields exist (document was written by V2 code), it uses them. If they don’t exist (for the old V1 document), it falls back to the flat fields. One function handles both old and new documents: that’s backward compatibility.

And look at the downcast: it writes name, createdAt, lastLogin as flat string fields (V1 shape) alongside profile and timestamps (V2 shape). A service still reading V1 sees the flat fields and works fine. A service reading V2 sees the nested ones. That’s forward compatibility.

You wire both into the collection:

const collection = pongoDb.collection<UserDocV2, StoredPayload>(
  'users',
  {
    schema: { versioning: { upcast, downcast } },
  },
);

From here, your application code only deals with V2. The collection handles the translation in both directions:

const v2Doc: UserDocV2 = {
  profile: { name: 'Alice' },
  timestamps: {
    createdAt: new Date('2024-01-15T10:30:00.000Z'),
    lastLogin: new Date('2024-06-20T14:45:00.000Z'),
  },
};

await collection.insertOne(v2Doc);

What’s stored (downcasted, both shapes for compatibility):

{
  "name": "Alice",
  "createdAt": "2024-01-15T10:30:00.000Z",
  "lastLogin": "2024-06-20T14:45:00.000Z",
  "profile": {
    "name": "Alice"
  },
  "timestamps": {
    "createdAt": "2024-01-15T10:30:00.000Z",
    "lastLogin": "2024-06-20T14:45:00.000Z"
  }
}

Then you can read it back with:

const doc = await collection.findOne({ ... });

And get upcasted to V2 data in your application code:

{
  "profile": {
    "name": "Alice"
  },
  "timestamps": {
    "createdAt": "2024-01-15T10:30:00.000Z",
    "lastLogin": "2024-06-20T14:45:00.000Z"
  }
}

Same collection, V1 and V2 documents coexisting. insertMany, replaceOne, findOne: all go through the upcast/downcast. No batch migration needed. You roll out the new code, and old documents are handled transparently.

There’s another thing the downcast gives you: querying remains backwards-compatible. Because the downcast writes the flat V1 fields alongside the nested V2 ones, a query like collection.findOne({ name: ‘Alice’ }) still works even though V2 code doesn’t use name directly anymore. The V1 field is there in the stored document. That matters if you have queries or indexes built against the old shape. They don’t break.

Now, for events, ther matters even more. In event sourcing, stored events are immutable- the log is append-only, and you don’t modify what was already written. I wrote about versioning patterns in more detail. The core idea is: your business evolves, your code evolves, your event schemas evolve, but the events in the store stay as they were. You can’t go back and rewrite them (well, you can, but you really shouldn’t). Upcasting is how you bridge the gap.

For Emmett, the same pattern works at the event store level. You define the stored shape (what JSON gives you from the database) and the application shape (what your code works with):

type ShoppingCartOpenedFromDB = Event<
  'ShoppingCartOpened',
  { openedAt: string; loyaltyPoints: string }
>;

type ShoppingCartOpened = Event<
  'ShoppingCartOpened',
  { openedAt: Date; loyaltyPoints: bigint }
>;

And an upcast that handles each event type:

const upcast = (event: Event): ShoppingCartEventWithDatesAndBigInt => {
  switch (event.type) {
    case 'ShoppingCartOpened': {
      const e = event as ShoppingCartOpenedFromDB;
      return {
        ...e,
        data: {
          openedAt: new Date(e.data.openedAt),
          loyaltyPoints: BigInt(e.data.loyaltyPoints),
        },
      };
    }
    case 'ShoppingCartConfirmed': {
      const e = event as ShoppingCartConfirmedFromDB;
      return {
        ...e,
        data: {
          confirmedAt: new Date(e.data.confirmedAt),
          totalCents: BigInt(e.data.totalCents),
        },
      };
    }
    default:
      return event as ShoppingCartEventWithDatesAndBigInt;
  }
};

You pass it when reading a stream:

const { state } = await eventStore.aggregateStream<
  ShoppingCartState,
  ShoppingCartEventWithDatesAndBigInt
>(shoppingCartId, {
  evolve: evolveState,
  initialState,
  read: { schema: { versioning: { upcast } } },
});

Or in a command handler:

const handle = CommandHandler<ShoppingCart, ShoppingCartEvent>({
  evolve,
  initialState: () => ({ ... }),
  schema: { versioning: { upcast: upcastDatesAndBigInt } },
});

The difference with events is that you can’t update them in place. For documents, you have both directions: upcast on read, downcast on write. For events, upcasting is the main tool because the event store is append-only. Old events stay as they were written. But downcasting has its place too.

Consider ther: you have a projection or a subscriber that was built months ago against the old event schema. Maybe it’s a read model that listens to ShoppingCartOpened and expects clientId as a flat string. But your current code evolved. Now ShoppingCartOpened carries a client object with id and name:

// What old subscribers expect
type ShoppingCartOpenedV1 = Event<
  'ShoppingCartOpened',
  { clientId: string; openedAt: string }
>;

// What current code produces
type ShoppingCartOpenedV2 = Event<
  'ShoppingCartOpened',
  { client: { id: string; name: string }; openedAt: Date }
>;

Upcasting enables the current code to read older events. The ones stored with just clientId. Downcasting helps old subscribers consume new events. It transforms the new client object back into the flat clientId they expect. Same principle as with documents, but especially important here because event subscribers often live in separate services or deployments that you can’t update all at once.

And the same upcast function that started as simple type mapping string → Date, string → bigint handles ther structural change too. You just add another case to the switch:

case 'ShoppingCartOpened': {
  const e = event as ShoppingCartOpenedV1;
  return {
    ...e,
    data: {
      client: { id: e.data.clientId, name: 'Unknown' },
      openedAt: new Date(e.data.openedAt),
    },
  };
}

Old events get the client object synthesised from the flat clientId. New events already have it. The evolve function only deals with the V2 shape.

And here’s where things started to click for me. I added upcasting to fix the bigint problem: explicit type mapping instead of a Regex. But the same mechanism, without any changes, also handles structural versioning. The simple string → Date mapping from the first example is the same code path as the clientId → client migration above. It’s one function, one place, one pattern for all of it: type coercion, field restructuring, schema migration.

Right decisions stack

Right decisions stack. The Regex hack was blocking the slot where upcasting should have been all along. Once I removed it, the performance got fixed, and I got schema versioning on top. One fix created room for the next one, which created room for the next. That doesn’t happen when you keep patching around the same bad decision.

Looking back, maybe the Regex wasn’t the wrong first move. The rule is “make it work, make it right, make it pretty”, in that order. The Regex made it work. It had performance problems, but it still let me ship and learn where the real problem was. If I had tried to design the upcast/downcast system from scratch, without having lived with the Regex for a while, I might have over-engineered it or missed the connection to schema versioning entirely. The understanding came from living with the shortcut.

Dawid raised a performance issue with Pongo projections, but the same Regex was running in Emmett as well. I could have fixed it in one place and called it a day. Instead, I used it as a push to do the thing I’d been planning anyway and applied it to both Pongo and Emmett to keep things consistent. Because I already understood the problem well enough, “make it right” turned out easier than I expected.

You can recover from shortcuts. You should. But you also shouldn’t be afraid to take them in the first place, as long as you come back and do it properly.

Full changes:

Cheers!

Oskar

How I cheated on transactions. Or how to make tradeoffs based on my Cloudflare D1 support

We’re being told that software design is the art of making tradeoffs. But… Are we taught how to make them?

Not that it’s easy to teach tradeoffs, it’s a subtle art of explanation. You need to provide enough context and be precise so others don’t treat tradeoffs as a general best practice. Because they’re usually not such, they typically come from the muddy banks of the Wishkah River.

I think that a decent way is to tell the story. Not the fairy tale, but the specific tradeoff applied in practice. That’s what I’m going to do today: tell you how I cheated on database transactions.

Dumbo

Do you know Dumbo? Of course, flying elephant, such a sweetie. It’s also a codename of my Open Source project. I didn’t tell you about it so far, as it’s a shared dependency for Pongo and Emmett, responsible for:

connection pooling,
safe connection lifetime management,
handling transactions,
migrations,
SQL execution.

Not that small a scope for a Dumb tool, aye? Still, the intention is to make usage dumb, hiding the weirdness of a specific SQL database, so I solve it once and don’t need to be constantly distracted thinking about it, but focus on:

How to append and process events the best way in Emmett,
How to make the best JSON handling and translation into specific SQL dialects in Pongo.

Dumbo usage is quite simple:

import { dumbo } from '@event-driven-io/dumbo';
import { pgDumboDriver } from '@event-driven-io/dumbo/pg';

const pool = dumbo({ connectionString, driver: pgDatabaseDriver });

You need to set up a specific database driver (e.g. pg for PostgreSQL or sqlite3 for SQLite), as Dumbo now supports multiple relational databases.

Having that, you can do stuff like:

import { SQL } from '@event-driven-io/dumbo';

await pool.execute.batchCommand([
  SQL`CREATE TABLE test_users (id SERIAL PRIMARY KEY, name TEXT)`,
  SQL`INSERT INTO test_users (name) VALUES ('Alice'), ('Bob')`,
]);

And also do queries:

const count = pool.execute.query<{count: number}>(
  SQL`SELECT COUNT(*) as count
      FROM test_users
      WHERE ${SQL.in('id', userIds)}`,
);

It’ll handle query parameterisation, data escaping, etc.

It can also handle transactions:

const users = await pool.withTransaction(async (tx) => {
  await tx.execute.command(
    SQL`INSERT INTO test_users (name)
        VALUES (${firstUserName}), (${secondUserName})`,
  );

  return execute.query<User>(
    SQL`SELECT *
        FROM test_users
        WHERE ${SQL.in('id', userIds)}`,
   );
});

As simple as this looks, you should already have the question about the first tradeoff I made in your head.

Why on earth I think that’s a sane move to write my own multi-database driver?!

Well, I agree, that’s not the best move at the first glance, but let me explain to you why in MY context this actually makes more sense:

Yes, there are tools like Knex, Kysely, Drizzle, etc. in Node.js land that handle similar stuff. They’re nice, I really like them, I really do, but… But they are all big and are bringing a lot of their conventions, and when I’m building storage tools like Emmett and Pongo, I need to have more control. I don’t want those tools and their limitations drive my architecture decisions. I also don’t want to be surprised when the creator decides to drop working on it, or when I become a victim of a supply chain attack. Still, when I want a new kitchen table, I don’t start by going to the forest with a saw. I’m still using existing database drivers, what’s more I’m allowing people to choose the one they prefer. I just don’t want to be driven (and also my users be driven) by some other higher abstraction tool.
Even with that, this decision can seem like a bold “how hard can it be?!” statement. And it is, but… But I’ve built, or was co-authoring, such tools in the past. Some were proprietary, some were Open Sourced (see Weasel). And they served me well.
And last but not least, the last point. Well, I only had those two above. OK, I can add that I didn’t plan to make it a general-usage tool, just a small one for my needs.

So I did.

And here we’re at the moment when I had to cheat on transactions because of those decisions.

Cloudflare D1

All relational databases seem similar, but only until you start using them extensively, or until you need to write your own storage library. Then you learn stuff that not necessarily the stuff you’d like to spend time on. For instance:

What’s the difference between databases like PostgreSQL and single-threaded databases like SQLite or DuckDB, and how concurrent processing can be surprising
or that sqlite3 only calls the first query, but the next silently ignores,
etc.

Still, well, abstractions like Dumbo give the possibility to massage such cases behind the scenes.

And then Cloud Databases and SaaS Databases came into play, like Cloudlfare D1.

I was motivated by the generous sponsorship from Sam Hatoum to make supporting Emmett and Pongo a top priority. Thanks Sam, appreciate that! And I made it, but I had to cheat a bit.

Databases like Cloudflare D1 and Supabase expose databases as pay-as-you-go services. They optimise deployment, making it highly scalable, so you don’t need to care about it. It’s cost-effective for start-ups and scenarios when you’re not under a huge load. If you are, then you’ll need to pay more, but they also give you autoscaling at least.

They do it by exposing the database API through HTTP API, for instance PostgREST. This gives them easier management around throughput, security, etc., as each call is made as an HTTP request through the exposed API.

Yet, that kills some options like: ekhm, transactions. The challenge with transactions is that they don’t scale (that’s why MongoDB is WebScale). They don’t scale, as you’d need to open a transaction, do some freehand operations, and then commit or rollback. That means (typically) you need to keep a connection open during that time. If you’re building SaaS, it’s a no-go, because sneaky users would open it for a few hours, do crazy stuff, and kill your SaaS resources’ utilisation.

But, boy, aren’t transactions one of the selling points of relational databases? They do, so how to proceed?

Then you need to cheat, as I did.

Repeatable reads, batches, etc.

Cloudflare D1 doesn’t provide transaction data for the reasons I outlined, but it doesn’t leave us without other tools. Those two tools to let me do the smoke and mirrors trick are:

sessions,
SQL batches.

What are sessions? Per Cloudflare Docs:

A session encapsulates all the queries from one logical session for your application. For example, a session may correspond to all queries coming from a particular web browser session. All queries within a session read from a database instance which is as up-to-date as your query needs it to be. Sessions API ensures sequential consistency for all queries in a session.

Essentially, that means that we’re getting repeatable reads. So when we’re starting specific sessions, they will be handled sequentially.

And now, the second ingredient: Batches. Per Cloudflare docs

Sends multiple SQL statements inside a single call to the database. This can have a huge performance impact as it reduces latency from network round trips to D1. D1 operates in auto-commit. Our implementation guarantees that each statement in the list will execute and commit, sequentially, non-concurrently.

Batched statements are SQL transactions. If a statement in the sequence fails, then an error is returned for that specific statement, and it aborts or rolls back the entire sequence.

To send batch statements, provide D1Database::batch a list of prepared statements and get the results in the same order.

So Cloudflare didn’t allow us to do the full, freehand transaction, but they allowed us to send multiple statements that internally will be executed as a SQLite Transaction. When the request is handled, it’ll open a transaction, run the statements, and return the results.

Cool, let’s mix this soup together, as having that, I decided to:

Fail automatically if someone tries to create a transaction on Cloudflare D1 with an error:

D1 does not support SQL transactions (BEGIN/COMMIT/ROLLBACK/SAVEPOINT). Use { mode: “session_based” } to opt-in to session+batch semantics, or use ‘connection.execute.batchCommand() for atomic multi-statement execution.

Then they get clear information on the first try.

Allow users to explicitly open a session-based transaction by providing mode:

const users = await pool.withTransaction(
  async (tx) => {
    await tx.execute.command(
      SQL`INSERT INTO test_users (name)
      VALUES (${firstUserName}), (${secondUserName})`,
    );

    return tx.execute.query<User>(
      SQL`SELECT *
              FROM test_users
              WHERE ${SQL.in('id', userIds)}`,
    );
  },  
  { mode: 'session_based' }
);

When they do it, they will need to be aware of the limitations of the tool they have. So that this will internally create a D1 session, and only handle a single batch of operations properly. We’re mimicking the sequential processing by the session-based repeatable reads capability. Still, we need to remember that we won’t be able to roll back changes across multiple statements. Only a single command or batch command is an atomic operation.

We can’t, for instance, run a batch of updates, and fail the whole batch if one update didn’t change any record. The batch will only fail if the database throws an exception. An exception can be in SQLite only called by a table constraint or trigger.

By making this choice to require explicit mode and naming it explicitly, I didn’t manage to cover all cases, but at least made it safe, so people need to learn about this non-default behaviour and be more careful about it. When designing an API, it’s usually better to start with a more strict option and do a bit of “scarification” sometimes.

Still, when I implemented that in Emmett and Pongo, I intentionally used it to enable event appends, but document operations, etc.

If you’d like to try it, you can check Emmett’s or Pongo’s beta versions.

For Pongo, you can install it with:

npm install @event-driven-io/pongo@0.17.0-beta.21

And use it as:

import { d1PongoDriver } from '@event-driven-io/pongo/cloudflare';

const client = pongoClient({
  driver: d1PongoDriver,
  database,
  transactionOptions: { mode: 'session_based' },
});

Or in Emmett by installing:

npm install @event-driven-io/emmett-sqlite@0.43.0-beta.1

And use it as:

import { getSQLiteEventStore } from '@event-driven-io/emmett-sqlite';
import { d1EventStoreDriver } from '@event-driven-io/emmett-sqlite/cloudflare';

const eventStore = getSQLiteEventStore({
  driver: d1EventStoreDriver,
  database,
});

Still, even if you don’t care about Emmett, Pongo, and my Open Source project, I hope that this will give you decent inspiration for your own tradeoffs analysis.

I hope that you learned a bit about how design APIs work, how to check the guarantees of your tools and learn to walkaround them when you have to.

Cheers!

Oskar

On rebuilding read models, Dead-Letter Queues and Why Letting Go is Sometimes the Answer

In the last article, I explained how to rebuild Event-Driven Read Models in a safe and resilient way. I asked readers to let me know if they find any blind spots in my design.

Well, I found one myself.

This article is about that edge case, but more importantly, it’s about the rabbit hole I went down thinking about how to “fix” it. At the end of this hole, there is a nice learning about dealing with distributed systems. Sometimes the best engineering isn’t about preventing all failures. It’s about recognising your blind spots and making sure they don’t catch you off guard.

Everyone has a plan until they get punched in the mouth

Let me recap the situation described in the previous article. We’re using PostgreSQL to store events and read models. We want to rebuild an inline projection, one that’s applied in the same transaction as the appended event.

In the design from my previous article, during a rebuild, we:

Mark the projection as “rebuilding”.
Skip inline projections (the rebuild process will catch up anyway).
Process all historical events.
Mark the projection as “active” again.

The hybrid locking strategy with advisory locks and status checks ensures that inline projections know when to skip and that only one rebuild runs at a time.

Sounds solid. Here’s where it breaks:

The rebuild process checks for new events, sees none (because event 1000 is still in an uncommitted transaction), declares victory, and sets the projection to active. Meanwhile, the append transaction has already decided to skip the inline projection. When it finally commits, the event exists but was never projected. The rebuild process is done, and only new events will update the projection during the next update to the new event append.

We could handwave it with it won’t ever happen!. But, well, under load, this will happen eventually. You might not even notice it, but it will. The timing window is small, but with enough throughput, it becomes a certainty.

The Rabbit Hole of “Fixes”

My first instinct was to engineer my way out of this. Surely with enough clever coordination, we can close this gap?

Let me walk you through the rabbit hole.

Attempt 1: Wait for In-Flight Transactions

PostgreSQL provides pg_snapshot_xmin(pg_current_snapshot()) which tells us the oldest transaction that’s still running. The idea: after processing events, wait until all potentially-skipping transactions have completed before marking the projection as active.

Why it fails: We don’t know what we’re waiting for. While we wait for existing transactions to complete, new transactions start and make their own skip decisions. The target keeps moving. We can never “catch up” because new skips happen while we’re waiting for old ones to become visible.

Attempt 2: Use Transaction IDs as Boundaries

PostgreSQL assigns monotonic transaction IDs to each transaction when it starts. This seems useful—what if we record a “sealing” transaction ID when we’re ready to complete the rebuild, and use it to make decisions?

The logic would be:

When rebuild is ready to complete, record sealing_txid = current_transaction_id

Any event from a transaction with ID lower than the sealing point was “in flight” during rebuild, so it should be handled by async Events from transactions with higher IDs started after sealing, so they can safely use inline projections (the read model will be ready)

Sounds reasonable?

Here’s why it doesn’t work.

The core problem: transaction ID order ≠ commit order.

When a transaction starts, PostgreSQL assigns it the next available transaction ID. But transactions don’t commit in the order they started. A transaction that started earlier (lower ID) might take longer to complete and commit after a transaction that started later (higher ID).

Let’s trace through a concrete scenario:

Here’s what happened:

Tx 99 started first and got the lowest transaction ID. It inserted an event and decided to skip the inline projection (status was ‘rebuilding’). But then it got slow—maybe network latency, maybe the application did other work.
Tx 100 (the rebuild) started second, recorded sealing_txid = 100, and prepared to complete.
Tx 101 started third. It checked: “Is my transaction ID (101) >= sealing_txid (100)?” Yes, so it assumed the read model was ready and processed its inline projection. It committed successfully.
Tx 100 marked the projection as active and committed.
Tx 99 finally committed. But it had already decided to skip the projection back when status was ‘rebuilding’. That decision was made, the skip happened, and the event is now missing from the read model.

The fundamental issue: we can’t see uncommitted transactions. When Tx 100 set the sealing point, it had no way to know that Tx 99 was still out there, holding an event that would skip projection. Transaction 99 is invisible until it commits, but by then it’s too late.

You might think: “Just wait until all transactions before the sealing point have committed!” PostgreSQL even provides pg_snapshot_xmin(pg_current_snapshot()) which tells you the oldest active transaction. But this leads us back to Attempt 1—while we wait, new transactions start and make their own skip decisions. The target keeps moving.

I wrote about this exact problem in How Postgres sequences issues can impact your messaging guarantees. The same visibility challenges that affect outbox ordering apply here. Transaction IDs are useful for ordering within committed data, but they can’t help us coordinate with transactions that haven’t committed yet.

Attempt 3: Lock Appends During Transition

What if we use advisory locks more aggressively? The idea: when rebuild is ready to complete, acquire an exclusive lock that blocks the entire append path. While holding this lock, flip the status to ‘active’. No appends can be in progress during the flip, so no race condition, right?

Here’s the proposed flow:

Rebuild finishes processing historical events
Acquire exclusive advisory lock (all new appends must wait)
Set status = ‘active’
Release lock
Waiting appends resume and sees ‘active’, processes inline normally

It should work. In theory. We’re creating a synchronisation point where nothing can slip through. But there’s a subtle problem: the skip decision was already made inside the append transaction.

An append transaction doesn’t just check the lock and status at one instant. It does several things:

BEGIN transaction
INSERT the event
Try to acquire a shared advisory lock
Check projection status
Decide: process inline or skip?
Execute that decision
COMMIT

The decision in step 5 happens inside the transaction. If the transaction saw ‘rebuilding’ status, it was skipped. That decision is now part of the transaction’s pending work. The transaction might be waiting to commit or doing other work, but the skip decision has already been made.

Our exclusive lock in the completion flow blocks step 3 - new transactions can’t acquire the shared lock. But what about transactions that already passed step 5 and are just waiting to commit?

The timing here is tricky. Advisory locks in PostgreSQL can be either:

Transaction-scoped (pg_advisory_xact_lock): released automatically when transaction commits
Session-scoped (pg_advisory_lock): held until explicitly released or session ends

If we use transaction-scoped locks for inline projections (which makes sense—you want the lock tied to the transaction lifetime), then the append transaction might have already released its shared lock by the time we try to acquire an exclusive lock. The lock protected the status check, but the transaction is still running with its skip decision already made.

Even if we could perfectly synchronise the lock acquisition, there’s another problem: advisory locks are session-scoped, not durable.

If the connection dies while holding the exclusive lock:

The lock releases immediately (that’s how advisory locks work)
We might have partially updated the status
The system is now in an unknown state
Other transactions resume with potentially inconsistent data

We can’t build reliable coordination on something that vanishes when connections fail. This is exactly why the original article combined advisory locks with persistent status checks—but that combination doesn’t solve this particular race condition.

TLDR on Attempts

Every “fix” follows the same pattern:

Identify a coordination point.
Discover there’s a window before that point we can’t see.
Try to close that window.
Create a new window somewhere else.
Repeat.

We’re not solving the problem. We’re relocating it.

So is it a bug in the initial design? Depending how you look on that. I see it as an example of fundamental property of concurrent systems. PostgreSQL’s isolation guarantees mean uncommitted transactions are invisible to other transactions. That’s a feature, not a bug - but it means there’s always a window we can’t see into.

Stop Fighting, Start Tracking

After chasing my tail through various “solutions,” I stepped back and asked a different question.

Instead of: “How do we prevent events from being skipped?” (which requires blocking appends or seeing into uncommitted transactions—both unacceptable because of performance, guarantees etc.)

I started to think on: “How do we know when an event was skipped?” and “How do we ensure skipped events get processed eventually?”

Both of these are solvable.

If we can’t prevent skips from happening, let’s make them visible. When an inline projection skips, it could record that it skipped in the same transaction as the event append.

If the append transaction rolls back, the skip record also rolls back (there’s no event to worry about). If it commits, we have a durable record of exactly what was skipped.

In Emmett, I’m using a dedicated emt_system_messages table rather than reusing the regular emt_messages table or creating a simple “skipped events” table. This might seem like over-engineering—why not just create a simple table with projection ID and event position? Or why not just reuse the existing messages table?

Why a dedicated system messages table?

The regular emt_messages table is for business events—the actual domain events that drive your application. Mixing system-level concerns (like “this projection skipped this event during rebuild”) with business events pollutes the event log and makes it harder to reason about. A simple “skipped_events” table with just (projection_id, event_position) could be enough, but in the long term, it may be hard to maintain. As systems evolve, we’ll have more cases where we might want to skip events. Plus, having system events recorded with all metadata gives us full observability of the internals of our system. We could even make tables for projections and processors’ status, built as read models from system events! Still, let’s hold our horses and get back to our use case.

The system messages table could mirror the structure of regular messages with:

Global position sequencing (with all the transaction visibility handling from the outbox ordering article)
Transaction ID tracking for proper visibility checks
Archiving support via is_archived flag
Partitioning for performance

If we created a throwaway “skipped events” table, we’d need to solve all these problems again. We’d essentially be building a second event log with the same guarantees.

The skip could be stored as a system message where:

Stream ID = the projection identifier (name + version), so we can query all skips for a specific projection
Message data = reference to the original event (sequence ID from the event log)
Message metadata = processor ID, reason for skip, timestamp
Message type = indicates this is a “skip during rebuild” system event

Won’t this table grow too much? It may, but we can archive skip events when they’re no longer needed.

When the rebuild processor handles a skipped event, we could archive it. In Emmett, this means setting is_archived = true on the record. The table is partitioned by this flag. Archived records automatically move to a separate partition, which can even be on a different disk drive.

Why archive instead of delete?

Audit trail: You can see what was skipped and when it was processed
Debugging: If something goes wrong, you have history to investigate
Idempotency: If a processor crashes and restarts, it won’t reprocess already - archived skips

Later, we could define retention policies and clean up old archived records. We could do it more aggressively than for business events, since these are operational records, not business history. You might keep business events forever, but archived skip records only need to stick around long enough for debugging (days or weeks, not years).

The same processor that handles regular events for the projection could also process its skipped records. When it reads an event from the main event log and applies the projection, it also checks for and archives any corresponding skip record for that event position. This keeps everything in sync.

As mentioned, this also opens the door for other system events in the future - not just rebuild skips, but potentially failed projections, poison messages, or audit records. The infrastructure would already be in place, separated from the business event log.

The Final Flow

Let me walk through how this works in practice, bringing together all the pieces.

During rebuild

The projection status is ‘rebuilding’. The async rebuild processor works through historical events from the beginning. Meanwhile, normal operations continue. Events are appended, and inline projections check the status. When they see ‘rebuilding’, they skip the projection but record a skip message in emt_system_messages within the same transaction.

When rebuild catches up

The rebuild processor eventually reaches the current position where no new events are visible. At this point, it sets the status to ‘active’. From now on, new appends will process their inline projections normally.

But what about those skip records? The events they reference exist, but their projections were never applied.

Draining the skipped events

The rebuild processor (or a dedicated processor, or a manual trigger—your choice) now queries emt_system_messages for skip records belonging to this projection. Using the same transaction visibility rules from the outbox pattern (transaction_id < pg_snapshot_xmin), it only sees skip records from committed transactions.

For each skip record:

Find the referenced event in the event log (or use its data if stored in skipped event)
Apply the projection for that event
Archive the skip record (set is_archived = true)

The skip record and the event are committed in the same transaction. This gives us a simple invariant: if an event exists, either its projection was applied inline (status was ‘active’) or a skip record exists (status was ‘rebuilding’). There’s no third option where an event exists without a trace.

The drain process might find new skip records appearing. Transactions that were in flight during the status change, committed after we started draining. That’s fine. We keep querying until no more visible skip records exist. They will stop appearing as we alread stopped rebuilding processor, so no more inline projections should be skipped.

The “drain skipped events” phase can be automatically triggered by finishing projection rebuild. It could be handled as the 2nd phase of rebuild processor, or triggering a dedicated one. It could be also just handled by a human initiating the drain.

This flexibility lets user choose the approach based on the specific operational requirements.

The Dead Letter Queue Pattern

What we’ve built is essentially a Dead Letter Queue (DLQ)—a place where messages that couldn’t be processed normally are stored for later handling.

This pattern exists in every serious messaging system:

Apache Kafka: Dead Letter Topics for messages that fail consumer processing
RabbitMQ: Dead letter exchanges for rejected or expired messages
AWS SQS: Redrive policies that move messages to a DLQ after N failures
Azure Service Bus: Built-in dead-letter sub-queues for each queue

The pattern is universal because all messaging systems face the same fundamental problem: sometimes messages can’t be processed immediately, and you need a place to store them without blocking the rest of the system.

It’s a topic in its own right, as it’s not perfect and should be used cautiously. Many teams fail to apply them correctly. The DLQ becomes like a car alarm in a parking lot, technically signalling a problem, practically ignored by everyone. And that’s what I see in many systems: teams set up DLQs and do nothing about them.

It starts innocently. You configure the DLQ “just in case.” A few messages end up there during a deployment. Someone says, “We’ll look at it later.”

More messages accumulate. The DLQ becomes background noise—a number on a dashboard that nobody checks. Eventually, something critical lands there, and nobody notices until a customer complains.

That’s why in the discussed design, skip records aren’t meant to accumulate indefinitely. The rebuild processor drains them during completion.

Retention policies clean up archived records after a reasonable period. If skip records exist for too long, that’s a signal that something is wrong with the rebuild process - and users should know about it.

A DLQ is only helpful if it’s monitored, processed, and understood why messages end up there. Otherwise, it’s just a fancy way to lose data slowly rather than immediately.

The Broader Lesson

This article isn’t really about PostgreSQL advisory locks or projection rebuilding. It’s about how we approach problems in distributed systems.

When we find a race condition, the instinct is to fix it. Add a lock. Add a check. Add a coordination phase. But each “fix” often just moves the race condition somewhere else. We went through three attempts—waiting for transactions, using transaction ID boundaries, locking appends—and each one failed for a different reason. We weren’t solving the problem; we were relocating it.

At some point, I had to ask myself: am I making this system more reliable, or just more complicated?

The answer came when I changed the question. Instead of asking “how do I prevent events from being skipped?” I asked, “How do I know when an event was skipped, and how do I make sure it gets processed eventually?”

The first question has no good answer, not without blocking appends, which defeats the purpose. The second question is straightforward: record the skip in the same transaction as the event, and process it later.

A system isn’t trustworthy because it never fails. That’s impossible for anything sufficiently complex. A system is trustworthy because you know when it can fail, how it will fail, and how to recover. The skip tracking approach doesn’t prevent failures during the transition period. It makes them visible and recoverable. That’s a stronger guarantee than complex coordination machinery with hidden edge cases.

Sometimes the best engineering decision is to accept what you can’t control and focus on what you can. We can’t control PostgreSQL’s transaction visibility rules. We can’t see into uncommitted transactions. We can’t make the append decision and the rebuild completion atomic without stopping the world.

What we can control is recording skipped items, ensuring those skips are processed, and making the whole thing observable. That’s enough.

The blind spot I found wasn’t really about the specific race condition. It was a reminder that distributed systems have fundamental constraints we can’t engineer around, at least not without trade-offs worse than the original problem.

Transaction isolation and concurrent systems mean we can’t have zero-downtime rebuilds with perfect inline projection consistency and no coordination overhead, all at the same time.

Something has to give.

What we get instead is explicit tracking of what was skipped, guaranteed eventual processing via the system messages table, observability into the transition period, and crash recovery that doesn’t lose data. The read model might be briefly inconsistent during the transition, but we know exactly what’s missing, and we have a clear path to fix it.

That’s the kind of system I can reason about and trust.

If you’re dealing with similar challenges, I’m happy to help through consulting or mentoring. You can also join the discussion in our Emmett Discord—we have a nice community working through these exact problems.

Or check also other related resources:

Cheers!

Oskar

Rebuilding Event-Driven Read Models in a safe and resilient way

Let’s make a soup today: a blog soup. We’ll mix multiple ingredients like:

events (obviously),
read models,
inline and async projections,
rebuilding read models,
backfilling new ones with data from existing events,
scaling async processing horizontally,
distributed locking,
PostgreSQL and its Advisory Locks.

Sounds a lot? Well, the soup should be nutritious.

In an event-driven way, after handling business logic, we record new facts and call them events. They gather information about what has happened. That brings many benefits, such as business observability by keeping a log of them. Especially if we’re doing Event Sourcing, we can make the next decision based on them.

Typically, we’re using events in two ways:

reacting to them, triggering and integrating the steps of our business workflow,
projecting them, and getting the flattened interpretation of our system state inside read models.

Such processing can happen asynchronously, but doesn’t have to. If we’re using a transactional database (like PostgreSQL, SQLite, or even MongoDB), we can update our read models in the same atomic transaction that stores new events. Such a process is typically called inline projection. Event stores like Emmett and Marten allows that. Still, you can do the same if you’re using outbox pattern, then you can achieve the same without Event Sourcing.

Inline processing is tempting because we’re getting immediate consistency. Yet, there’s no free lunch here. We’re slowing down our event append as we need to process more, and our transactions will be open longer, which can cause deadlocks, etc. We also can’t take advantage of batching event processing.

Also, as Bastian Waidelich rightfully pointed out to me, using inline projections also increases the coupling and fragility of our business logic. If the inline projection fails (due to a bug in its projection logic, a database constraint or other random issue), then we won’t be able to append our event, which is counterintuitive, as why would read model block our business logic (e.g. confirming shopping cart).

My thumb rule is that for single stream, simple projections, I prefer inline projections, but for more complex or workflow processing, I’d go with async.

The big benefit of a durable event log is that we can correct past mistakes and gain more insights from existing data.

How does it look in practice? Let’s say we initially had a basic read model that showed a summary of the specific shopping cart. In TypeScript this could look as follows:

type ShoppingCartSummary = {
  _id?: string;
  productItemsCount: number;
  totalAmount: number;
};

Besides the data, we also need a method that represents how we apply events on top of the existing state to get the next, evolved state.

const evolve = (
  document: ShoppingCartSummary | null,
  event: ProductItemAdded | ProductItemRemoved,
): ShoppingCartSummary => {
  document = document ?? { totalAmount: 0, productItemsCount: 0 };

  switch (type) {
    case 'ProductItemAdded':
      return withAdjustedTotals({
        document,
        productItem: event.data.productItem,
        by: 'adding',
      });
    case 'ProductItemRemoved':
      return withAdjustedTotals({
        document,
        productItem: event.data.productItem,
        by: 'removing',
      });
  }
};

const withAdjustedTotals = (options: {
  document: ShoppingCartSummary;
  productItem: PricedProductItem;
  by: 'adding' | 'removing';
}) => {
  const { document, productItem, by } = options;
  const plusOrMinus = by === 'adding' ? 1 : -1;

  return {
    ...document,
    totalAmount:
      document.totalAmount +
      productItem.unitPrice * productItem.quantity * plusOrMinus,
    productItemsCount:
      document.productItemsCount + productItem.quantity * plusOrMinus,
  };
};

Sounds fine, but well, it may appear that either we, implementing it, or our business, through requirements, forgot about some requirements. Or we didn’t forget anything, but just requirements evolved as they tend to always do.

What if we need to handle now also:

Show the shopping cart status and show whether the shopping cart is open, confirmed or cancelled.
not only show totals, but also a list of product items with their details.

Our new read model would look like:

type ShoppingCartSummary = {
  _id?: string;
  productItemsCount: number;
  totalAmount: number;
  status: 'Opened' | 'Confirmed' | 'Cancelled';
  productItems: ProductItem[];
};

type ProductItem = {
  productId: string;
  name: string;
  quantity: number;
  unitPrice: number;
}

And here we should ask ourselves the following questions:

Is it really the same read model or another one that’ll be used in another place in our UI? Maybe the initial just shows basic data in the menu bar, and this will be used as the summary before confirmation?
If the read model is the same, then are you fine with downtime where you clean the old data, and reprocess events?
If it’s the new read model, do we need to backfill it with the old data?

There’s no best practice here; we need to do the drill and be prepared for multiple options.

Add new vs update existing.

In our case, my initial guess would be that this should be a new read model. We need to add significantly more data and new event handling for confirmation and cancellation.

You could say that:

Why add a new read model with similar data? Can’t we just do a subquery?

Of course, you can. I wrote about that more in How to create projections of events for nested object structures?. This can make sense if those read models will always evolve together.

If you need to run multiple views on the same read models, you’re increasing coupling. As such, when you’re rebuilding the read model, then potential downtime will impact both. Also, each time you adjust one view, ensure you haven’t broken the others.

You’re getting a smaller storage size, and potentially don’t need to remember multiple read models, but are doing it just one.

My experience shows that optimising for the end storage until we check that it’s too big isn’t a good driver. Nowadays, storage is cheap, so my default is to keep read models separated, and also not reuse the same evolve logic. A bit of code duplication won’t harm us, but we’ll see benefits as our models evolve. We’ll decrease the cognitive load.

Still, if those models are indeed the same, or we bet they’ll constantly evolve together, then it could be fine to reuse them.

In place update, blue/green rebuild and backfilling data

Ok, I was already using rebuilding word multiple times. But how do we actually do it?

If you’ve read articles on checkpointing or positions in event stores, you already know that each event in the event store/outbox can have its unique, monotonic position. We can subscribe to notifications about new events and process them one by one. That’s how consumers and processors work in Emmett. Once we process a specific event, we can store the checkpoint in the end storage. This enables resilient failover when our processor dies for some reason. When we restart, we’ll read the last processed checkpoint first and start listening for events from that point.

We could reuse this not only for failover but also for rebuilds. Instead of starting processing from the last known checkpoint, we could reset the checkpoint in our database to a specific point (e.g., the beginning of the log). Then we’ll get recorded events again.

And this is the moment we need to decide whether to do an in-place update or a blue/green rebuild.

In-place means that we’re the same storage target. We need to truncate it and then apply data from scratch, typically. The downside is that when we clear it, our read models won’t be up to date with the current state of our event store. We already have newer events recorded, but our read models are empty. We need to fill them in.

The same happens when we create a new read model based on the existing events. When we create a new read model, it won’t magically contain data from existing events.

Typically, we need to spin up a background worker (e.g., Emmett consumer with a projector plugged in) that pulls all events since the beginning and runs projection logic. Obviously, that means we need to run it asynchronously, and depending on the data size, we’ll need to wait until it catches up. During that time, querying for the data will return outdated information - eventual consistency in practice.

That’s why there’s another option. Instead of truncating existing data, we can keep it as it is, or even keep updating it in the old form. Having that, we can build a new read model in parallel. This read model could have a different name, or just a suffix, e.g. V2, V2.0.1, whatever we find helpful.

Then, once this new read model catches up (so it has processed all events, or is close enough to the latest event with a defined threshold), we can switch queries from the old to the new read model storage.

This is actually the preferred way, but it’s a bit more challenging when it comes to dynamically switching the query target. If you’re using Pongo, that’s not that hard, since you just switch the text-based collection name, which is just another table. But if you’re using an ORM, adding a new table dynamically and mapping it can be much more challenging.

Concurrency issues while (re)building read models

To make it harder, eventual consistency is not the only challenge we’ll face. We also need to deal with concurrency and parallel processing.

What should we do when a new event is appended while we’re rebuilding/backfilling the inline projection? If we try to update the end storage in the middle of the processing, we can end up with an inconsistent or erroneous state.

Another issue is what to do if we’re in a Kubernetes-like setup and don’t have full control over the number of instances of the same service? Or what happens if we accidentally spin up multiple rebuild workers?

Then we’re doomed, or at least the consistency of our read model data.

How do we solve it? Let me explain my plan for Emmett.

Distributed Locking and PostgreSQL advisory locks

I encourage you to check my other article: Distributed Locking: A Practical Guide. Yet, don’t worry, I won’t leave you with Read-The-Fucking-Manual type of answer.

Distributed locks are a fundamental tool for coordinating concurrency across systems. We can use a central place, typically scalable on its own, that’ll be used in multiple instances of our service, to ensure that exactly one can request a lock and run specific code.

In the mentioned article, I described multiple options and popular tools for handling that (e.g., Redis, Zookeeper, Kubernetes Replica Sets), as well as PostgreSQL and its Advisory Locks. Let me focus today on the last one and describe how I want to use it in Emmett’s PostgreSQL projection handling.

PostgreSQL gives us two options for coordination.

Row-level locks lock individual table rows. You do SELECT … FOR UPDATE, and anyone else trying to modify that row waits. The lock is tied to the specific row in a table.

BEGIN;
SELECT * FROM some_table WHERE id = @key FOR UPDATE;
-- make changes
COMMIT;

Such locks are straightforward, as we explicitly state what we want to lock. We could define the following table for projections:

CREATE TABLE IF NOT EXISTS emt_projections(
      version                       INT                    NOT NULL DEFAULT 1,  
      type                          VARCHAR(1)             NOT NULL,
      name                          TEXT                   NOT NULL,
      status                        TEXT                   NOT NULL,  
      PRIMARY KEY (name, version)
);

The background worker could lock the projection by a specific name and version and set the status to “rebuilding”. Then, during handling the inline projection, we could use the same lock and check whether the status is “active”; if not, skip processing. Using a lock-in inline projection would also prevent the rebuilding process from starting, as they wouldn’t acquire the lock.

And that’s a nuke option, as it would work but could create a performance problem. If every inline projection needs to grab a row lock on a coordination row, they will all be processed sequentially, one at a time. That’s a throughput killer when you’re appending thousands of events per second. Each of these append would require access to the lock for the specific projection type (e.g. our shopping cart summary), making not only updates but also event appends sequential. That’s not acceptable for most cases.

Advisory locks are different. They’re locks on arbitrary integers. PostgreSQL doesn’t care what the integer means—it just manages who holds the lock. No table rows involved. It also allows two modes: exclusive or shared.

-- Shared lock: many can hold simultaneously
SELECT pg_try_advisory_xact_lock_shared(12345);

-- Exclusive lock: only one holder, blocks shared locks
SELECT pg_advisory_lock(12345);

Those locks are either scoped to an open connection session or an opened transaction (with xact with name) and released automatically once they end.

Shared locks allow multiple sessions to access the lock with the same value. Exclusive lock blocks both other exclusive locks and new shared locks.

This allows a design where shared locks are used for readers, and exclusive locks for writers and maps directly to our problem:

Inline projections take shared locks as they run concurrently, and just need to check if there’s no async job updating these projections
Rebuilds take exclusive locks - they block inlines and other instances of async processing.

We just need to do one more thing: since advisory locks take integers, we need to map our projection name and version to them. We can do it by a consistent hash algorithm, either in the application code or in PostgreSQL.

PostgreSQL provides a built-in MD5 hash function. It’s not perfect, as it’s not a sophisticated hash, but it’s fast enough and predictable. In our case, we won’t have thousands of projections in our application, so the risk of a hash collision is negligible. If you’re still worried it’s too high, we could store id in our projections table and use it instead of hash-mapping. Still, if we used md5 function, it could look as follows:

// shared for inline projections
SELECT pg_try_advisory_xact_lock_shared(
        ('x' || substr(md5(?), 1, 16))::bit(64)::bigint
) AS acquired;

// exclusive for inline projections
SELECT pg_try_advisory_xact_lock(
        ('x' || substr(md5(?), 1, 16))::bit(64)::bigint
) AS acquired;

Where as query param, we’d pass the joined projection name and its version.

Thanks to that, multiple inline projections can access the lock if it’s not held exclusively by the async (re)building worker. Thanks to that, we’re not blocking event appends because of the lock on the inline projections.

An exclusive lock can be held only when there’s no single inline projection being applied at the moment.

What’s also cool is that advisory locks can be used with two strategies: fail fast when the lock is held, or wait for the lock to be released. The first option would be useful for inline projections, and the second for rebuilds.

Is that all? Well, not quite.

Why advisory locks alone aren’t enough

Advisory locks have a gap: they’re session-scoped. If the connection holding the lock dies, the lock releases automatically. For most cases, that’s fine and expected, but think through this scenario:

We have 1000 events in our event store.
Rebuild starts, acquires exclusive lock.
It truncates the projection and processes events 1-500.
Connection dies (network blip, out of memory kill, whatever).
Lock releases automatically.
Projection contains data only for events 1-500.
Inlines see no lock, start processing.
Inline applies event 1001, ending up with a potentially corrupted state (as there may be some events between 500 and 1001 that would impact the state).

Advisory locks can’t persist across connection failures. We need to add something that does.

The hybrid-locking approach

Emmett already maintains tables for tracking processors and projections. The relevant ones:

CREATE TABLE IF NOT EXISTS emt_projections(
    version         INT         NOT NULL DEFAULT 1,  
    type            VARCHAR(1)  NOT NULL,
    name            TEXT        NOT NULL,
    partition       TEXT        NOT NULL DEFAULT 'emt:default',
    kind            TEXT        NOT NULL,
    status          TEXT        NOT NULL,
    definition      JSONB       NOT NULL DEFAULT '{}'::jsonb,
    PRIMARY KEY (name, partition, version)
) PARTITION BY LIST (partition);

CREATE TABLE IF NOT EXISTS emt_processors(
    last_processed_transaction_id XID8    NOT NULL,
    version                       INT     NOT NULL DEFAULT 1,
    processor_id                  TEXT    NOT NULL,
    partition                     TEXT    NOT NULL DEFAULT 'emt:default',
    status                        TEXT    NOT NULL DEFAULT 'stopped',
    last_processed_checkpoint     TEXT    NOT NULL,    
    processor_instance_id         TEXT    DEFAULT 'emt:unknown',
    PRIMARY KEY (processor_id, partition, version)
) PARTITION BY LIST (partition);

The status column in the projections table can do what advisory locks can’t: persist status between connection crashes. Even if the connection dies, status = ‘rebuilding’ stays in the table. Inlines could check this and skip processing.

The processor’s table tracks checkpoint progress. When a rebuild runs, it updates last_processed_checkpoint as it goes. If it crashes and restarts, it can resume from where it left off rather than starting over.

Potentially, we could reuse the processor’s table, but having a dedicated projection table could also be useful for diagnostics (e.g., storing the projection definition) and for switching queries when a new projection version catches up. We could select the highest active projection version.

Having that, the query checking if the inline projection should be processed or skipped could look as follows:

WITH lock_check AS (
    SELECT pg_try_advisory_xact_lock_shared(
        ('x' || substr(md5($1), 1, 16))::bit(64)::bigint
    ) AS acquired
),
status_check AS (
    SELECT status = 'active' AS is_active
    FROM emt_projections
    WHERE partition = $2 AND name = $3 AND version = $4
)
SELECT
    COALESCE((SELECT acquired FROM lock_check), false) AS acquired,
    COALESCE((SELECT is_active FROM status_check), true) AS is_active;

Two checks, both must pass:

Can we get a shared advisory lock? Fails if a rebuild holds exclusive.
Is the projection status ‘active’? Fails if it’s ‘rebuilding’.

Then for async (re)building worker:

WITH lock_check AS (
    SELECT pg_try_advisory_lock(
        ('x' || substr(md5($1), 1, 16))::bit(64)::bigint
    ) AS acquired
),
ownership_check AS (
    INSERT INTO emt_processors (
        processor_id,
        partition,
        version,
        processor_instance_id,
        status,
        last_processed_checkpoint,
        last_processed_transaction_id
    )
    VALUES ($2, $3, $4, $5, 'running', '0', '0'::xid8)
    ON CONFLICT (processor_id, partition, version) DO UPDATE
    SET processor_instance_id = $5,
        status = 'running'
    WHERE emt_processors.processor_instance_id = $5 -- We already own it
       OR emt_processors.processor_instance_id = 'emt:unknown' -- Unclaimed
       OR emt_processors.status = 'stopped'  -- Previous instance finished or crashed
    RETURNING last_processed_checkpoint
)
SELECT
    COALESCE((SELECT acquired FROM lock_check), false) AS acquired,
    (SELECT last_processed_checkpoint FROM ownership_check) AS checkpoint;

Together Advisory locks prevent the race at the transition point where the connection was closed, and the rebuilding job is restarting. The status check handles crash recovery

The rebuild acquires the exclusive lock before updating the status. It waits for in-flight inlines (which hold shared locks) to finish.

If the checkpoint is null, the UPDATE matched no rows, then another instance owns the processor and is actively running. Back off.

If both succeed, you own the lock and the processor. The checkpoint tells you where to resume from (with a fallback to the beginning). Only then does it flip the status and start async processing.

If the rebuild crashes, the lock is released, but the status remains ‘rebuilding’. Inlines check status and skip.

The scenarios walkthrough

As a single good image speaks more than thousands of words, let me give you some diagrams to summarise how it works.

1. During inline projection while event appends:

Each inline projection grabs a shared lock before applying,
Multiple inlines can run concurrently (shared locks are compatible),
Projection updates happen normally.

2. When a rebuild starts:

Rebuild grabs an exclusive lock (waits for any in-flight inlines to finish),
Marks projection as “rebuilding”,
New inlines see the lock or status and skip,
Rebuild processes all historical events,
Marks the projection as “active” and releases the lock,
Inlines resume.

3. If rebuild crashes:

Lock releases automatically
Status stays “rebuilding”
Inlines keep skipping until another rebuild completes

4. Multiple async processors:

Each processor tries to acquire the exclusive lock,
First one wins, others wait or skip,
Guarantees that only one processor handles a projection at a time.

5. When you add a new projection:

No status row exists yet,
Inlines skip automatically,
First rebuild creates the row and backfills data.

TLDR

Just like making delicious soup, designing robust, fault-tolerant and performant distributed systems is not that easy. Building an event store is not that hard. But only if we exclude the async processing part.

I hope this walkthrough covers both the conceptual and practical aspects of handling projection rebuilds.

We used PostgreSQL and Advisory Locks, as PostgreSQL is cool and is a driving force in Emmett. But all the same principles apply to other tools and storage (with their specifics).

I explained why advisory locks and status columns complement each other:

Advisory locks handle the fast path (in-memory, no disk I/O for normal operations) and prevent races at transition points (rebuild can’t start until in-flight inlines finish)
Status column handles crash recovery (persists across connection failures) and new projection bootstrapping (inlines skip until first rebuild completes)

Neither alone is sufficient. Together, they provide the guarantees we need without external infrastructure. Just PostgreSQL doing what PostgreSQL does.

The cost on the hot path is microseconds: one in-memory lock check, one indexed read on a tiny cached table. For most systems, that’s an acceptable tradeoff.

I hope this article also shows you how to use distributed locking in practice.

Please tell me your thoughts and concerns, especially if you see any blind spots in this design! You can do that in our Emmett Discord, come on in, we have a nice community!

If you’re dealing with such issues, I’m happy to help you through consulting or mentoring. Contact me and we’ll find a way to unblock you!

Or check also other related resources:

Cheers!

Oskar

Architecture Weekly is 5 years old! 🎉

Welcome to the new week!

Who said that LinkedIn notifications are useless? I did, several times. Yet! Today, I opened my computer for the first time since last Tuesday and just saw a notification that Architecture Weekly is 5 years old now!

At this point, you probably know that I’m not the anniversary-type-of-guy (my wife loves that part of me…), but well, that’s a nice milestone!

Architecture Weekly started as a way to organise my readings. I’m a curious guy, and I like to read a lot about different topics. During my career, I worked on a lot of weird projects, which exposed me to many technologies and approaches. It was not always pleasant, but it taught me that technology is a tool, and there’s a big diversity of tools that can lead you to a proper solution.

So, I had always had hundreds of open browser tabs until my Firefox crashed from time to time (yes, I’m a Firefox-type-of-guy). I’m sure you know that too. I tried multiple ways to organise that: tools like Notion, e-mail threads, etc. None of them was perfect. At some point, my friends treated me like a librarian and asked me for interesting links on a topic or what interesting things I’d read recently. I decided to put my readings publicly into a Git repository and use Markdown, so I can easily share them. This repository still exists: https://github.com/oskardudycz/ArchitectureWeekly. At first, just links organised in groups.

I started sharing it with my friends, and they told me it was useful, so I started sharing it with their friends. It seemed so useful to others that I shared it on the other channels, socials, and groups I was part of. Feedback was still positive, and some people started pushing me to create the newsletter, even though they could subscribe to GitHub release e-mail notifications; they still landed in the same notification swamp, as usual, with other notifications. So they asked if I could set up “a proper newsletter”. I was reluctant to do it, but well, as you already know, I did. I used Substack because it helped me avoid the usual accidental complexity by writing my own blogging engine from scratch.

At some point, I decided to try making a paid version of the newsletter. I decided to build a Discord community for paid subscribers and my GitHub Sponsors. I started to also run webinars every month, also inviting guests.

Complete list of webinars we had so far:

And now, yes, all of them are available for free. You can watch them. It’s around 50 hours of free knowledge. I believe that it is of much better quality than some paid courses. Why did I make them public?

At some point, I realised that I don’t have much fun with organising those links; it was more of a burden, as curating such lists takes a lot of time. I’m also a perfectionist-type-of-guy, so I wanted to ensure that what I recommend is worth the read. It triggers the movement of brain cells. So if you read the article, you might disagree with it, but at least it’s sparking some thought. I also wanted to synthesise them and put my comments. It may seem like a simple thing to provide such articles. But try to keep it at the proper level week after week, and you’ll see it might not be as easy as it seems.

There were not enough paid subscribers to justify the time spent on it, even though I was named a Substack Bestseller. Yes, running a newsletter is not a great business plan. There are some exceptions to that, but even if you’re in the top few per cent, then it might not be enough to even justify the time spent on it.

I decided to try a different angle and made Architecture Weekly a fully paid newsletter. Starting with a deep dive into Database Connection Pooling. I tried to see if I could grind for a while, providing high-quality content that’s not available elsewhere. I think I managed to deliver a lot of articles like that, and the paid subscriber count grew, but it was still not enough to justify my work. I enjoyed writing such articles a lot, and I still do, but it has put more pressure on me. And preparing such content takes time and effort. The effort I could spend elsewhere, e.g., with my family, in better-paid work like my workshops and consulting, or in my OSS projects.

And now, here we are, 10 months later and well, I still delivered a lot of deep dives, but shifted closer to the stuff I do daily, so helping other humans build systems and getting benefit from the Event-Driven approach. As you see, recently I cross-posted articles to my blog and here. I also used the angle of my work on tools like Emmett to explain general concepts.

What’s next? I’ll probably continue doing it as is; I might also consider merging the blog and newsletter into one. I even own domain event-driven.news, so maybe some rebranding will happen. But it’s not yet decided.

I’ll definitely continue sharing what I learned, blogging, but the exact form? We’ll see. I’m open to suggestions! Comment under this post and tell me your thoughts.

What type of content would you like to see here? Or is there some that you’d be even open to paying for?

Can’t promise that I’ll deliver all of them, but it’ll give me a food for thought.

Nevertheless, it’s time for a small celebration!

Thanks for being here with me through this journey!

Cheers!

Oskar

Multi-tenancy and dynamic messaging workload distribution

There are several reasons why I’m blogging.

The first one is that I forget, and writing helps me to remember and organise my findings.

The other is that I like to share my journey with the intention of sparing you, my dear reader, some of my struggles.

Last but not least, it gives me the chance to learn from discussions inspired by them. It’s always a chance to meet new perspectives, correct what I did or just trigger some recollection.

After the article about consumer, processors and all that messaging jazz, there was a great discussion on Emmett Discord, there were a lot of interesting threads there, but the one that inspired what you read today came from the Ismael Celis question. Ismael was curious about:

Distributing the workload dynamically between processors

And as that’s something I haven’t planned to deliver initially, but it’s a great question, let me put down some notes on my plan around it.

Why would we want to distribute the workload dynamically? What does that even mean?

Let’s start from the classical approach. We can define projections that will build read models based on the upcoming events. In Emmett, it can look like that when using Pongo as the storage tool:

const cartsSummaryProjection = pongoSingleStreamProjection({
  collectionName: shoppingCartsSummaryCollectionName,
  getDocumentId: (event) => event.data.shoppingCartId,
  evolve,
  canHandle: ['ProductItemAdded', 'ShoppingCartConfirmed'],
  initialState: () => ({
    status: 'pending',
    productItemsCount: 0,
 }),
})

This means that when the projection handles only ProductItemAdded and ShoppingCartConfirmed event types. It’ll insert or update rows in the table based on the shopping cart id from the event data.

We can plug this projection into a projector (the specific type of processor responsible for running projections).

const cartsSummaryProjector = postgreSQLProjector({
  processorId: 'shoppingCartSummary',
  projection: cartsSummaryProjection,
 });

And plug it into the consumer:

const consumer = postgreSQLEventStoreConsumer({
  connectionString,
  processors: [
    cartsSummaryProjector
  ],
});

Now, consumers will poll the PostgreSQL event store and pipe filtered events by type into our projector.

That’s simple, we know the filtering criteria, as consumers will know which event types their processor(s) can handle.

Things get harder when we don’t know those criteria upfront. When can that happen? What if we were building an e-commerce SaaS product that lets shop owners buy a subscription to run their shops? Then we won’t know all of our tenants upfront. They will register as they go.

What options do we have for such a multi-tenant setup?

The simplest option would be to add a custom filter that allows filtering events on the consumer by the tenant.

Emmett doesn’t support such a feature yet, but it could (and will at some point). We could extend our event metadata to contain the tenant and spin up a consumer for a dedicated tenant.

We could even wrap it into a dedicated function.

function tenantedConsumer(
  connectionString: string,
  tenant: string,
  processors: MessageProcessor[]) {
  return postgreSQLEventStoreConsumer({
    connectionString,
    processors,
    filterBy: (event) => event.metadata.tenant === tenant,
    projection,
  });
}

Of course, this wouldn’t be fully dynamic, but we could make it dynamic at the infrastructure level, e.g., by passing the tenant ID from an environment variable and spinning up a new Container (e.g., a Kubernetes Pod).

const consumer = tenantedConsumer(
  process.env.PG_CONNECTION_STRING,
  process.env.TENANT_ID,
  [ cartsSummaryProjector ],
);

We could also start a new job from the API endpoint, running it in the existing deployment; there are plenty of options. You can go wild and think about other ways.

Still, logically, they would look more or less like that. Either you’re scaling horizontally and sharding physically, tenants

Or you’re scaling vertically and running multiple tenants inside your box. Then, instead of spinning up new containers, you’re spinning up new jobs (processes, threads, virtual threads, etc.).

That’s nothing special for Emmett; other tools also do this, e.g., Spring Boot Kafka concurrent listeners.

And hey, btw. there’s one more reason why I’m writing about my Emmett design: to let you benchmark your design against it. Even if you’re not planning to use it, then those are considerations for you either as:

internal tooling creator,
user of other OSS tooling, to check how they solved it.

Separating tenants through sharding give you:

better option to scale and align the needs to the specific tenant workload,
makes possible full separation in terms of networking, storage, etc.
can be more costly.

Separating tenants through partitioning the load inside one box:

is usually cheaper,
easier to manage,
doesn’t give you a full separation and can fall into Noisy Neighbour issue.

So there’s no golden rule to choose, it depends on your tooling, needs, etc.

You can, of course, have a mixed solution, so running most of the small tenants in the same box, and the one with bigger security needs gets a special setup. Especially that dynamic split doesn’t have to be only per tenant. You can, e.g., have per region, per product range, per chain, etc.

Also, such a dedicated setup can work for a set number of dynamic options. If you have 1000 tenants, would you spin up 1000 containers or jobs? You can, but just because you can doesn’t mean that you should. The more containers or threads you have, the more you pay for the coordination costs. At some point, it’s better to group processing into a manageable range of containers or threads.

Ok, but how do we know which messages to process where? We could use a consistent hash. I wrote about it in detail in Understanding Kafka’s Consumer Protocol: A Deep Dive into How Consumers Talk to Brokers.

Kafka, by default, partitions its data on the producer side. The topic represents a logical split (e.g. all messages from the E-Commerce module), and the partition represents physical layout. Consumer groups receive messages. Kafka guarantees that precisely one consumer from the consumer group will receive messages from the specific partition. If we have fewer consumers than partitions, then of course, we can get more partitions to handle.

The pseudo code for distributing load to consumers could look as follows:

const partitionId = message.headers.partitionId;
const hash = consistentHashFunction(partitionId);
const consumerId = hash % totalNumberOfConsumersWithinGroup;

In our case, we could use the tenant id as the partition id.

Kafka’s strategy is also simpler than we might need. The partitioning is done on the producer side. The producer sends a message to the topic that already has a certain number of partitions:

const hash =  consistentHashFunction(message.header.recordId);
const partition = hash % totalNumberOfPartitionsWithinTopic;

In our case, if we’d like to allow distribution by any property, we’d need to load the message, read the field we want to partition/shard on, and send it to the appropriate container or thread.

The mechanism can get pretty hefty. You’d need a more sophisticated mechanism to know where to spin up what, how to distribute the load between containers or threads, and to make it resilient. We’re getting into the area of distributed consensus algorithms such as Raft and Paxos.

Do I want to go into this area with Emmett? Definitely not for free! I don’t think that’d be even worth it, as we have mature solutions like Kafka, RabbitMQ, and other messaging systems that implement such algorithms and specialise in that. I’d prefer to make it easier to forward messages to them and let them do the work.

What is definitely on the plate is the second option, which allows partitioned producers. You could define it as such:

const cartsSummaryProjector = postgreSQLProjector({
  processorId: 'shoppingCartSummary',
  projection: cartsSummaryProjection,
  partitionBy: (event) => event.metadata.tenant,
 });

Then the consumer would poll messages from all tenants, forward them to projectors, and the projectors would internally spin up worker threads per tenant or group them by consistent hashing.

If we add to that distributed locking or detecting conflicting checkpointing to ensure that there’s only one worker instance handling messages for the processor and partition id, then this should be good enough for the majority of cases.

Keeping in mind the flexibility in which you can group (or not) projectors within consumers, you can define your own topologies.

What are your thoughts? How do you deal with such cases? Would you like me to expand more on some cases?

Or maybe you’d like to help me and sponsor my work in this area in Emmett? Then your project could also benefit faster from it!

Cheers!

Oskar

Checkpointing the message processing

Let’s start by asking you two questions.

What Super Frog has to do with messaging?
When was the last time you wrote if statements in SQL? If it’s been a long time, have you at least seen them? If not, (don’t) worry, you’ll see them today.

Will it be a post about weird SQL usage? Not necessarily.

We’ll talk today about checkpointing our processing.

I’ve started my relationship with computers with games. I still have my Amiga 500. In those days, computers didn’t always have a hard disk. You’ve got a bunch of diskettes with different chapters of the game. Not all of them were simple games; many were quite sophisticated, and it took some time to finish them.

Yet they were dealing with limited diskette space, so it was best if they didn’t have to use any of it. How can you then allow you to stop playing and return to the previous state? Or how to not force you to start from the beginning of the game when you fell from the platform, and well, you died? You died in the game, ofc, that at least you should be able to recover, right?

As mentioned earlier, the limited space on diskettes and the additional complexity that came with it led many game makers to adopt a simple solution: checkpoints with codes.

After you passed a level, you got a code you could type when you started the game, and instead of starting from the beginning, you could go directly to the place where you left off. That worked pretty well for the platform and race car games, since your game’s storyline was always the same, immutable. If you had to go to level 27, the starting point and your character would always look the same. Of course, for RPG and strategy games, that’s a different story.

Surprisingly, this parallel also matches the recovery from a business process failure.

Let’s say we’re using message-based communication to streamline and make it more resilient. We don’t want to make it vulnerable to scenarios where we store information in one system, our process dies, and we don’t manage to notify the other parts.

We’re using Outbox pattern to enable that technically. We’re storing messages in the relational table within the same transaction, updating the state after running business logic. Thanks to that, either both states are updated, and the message is scheduled, or none of it is. We’re getting (eventual) consistency thanks to that.

Now we’re on the receiving end, so where we were in the previous article with the explanation of Consumers, Processors and all that jazz.

Let’s say that we’re using PostgreSQL and our Outbox structure looks as explained in the other article:

CREATE TABLE outbox(
   -- the autoincremented position of the message to respect the order
   position        BIGINT                    GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
   -- used to detect gaps in numbering
   transaction_id  xid8                      NOT NULL,
   -- unique message id, which can be used for deduplication or idempotency
   message_id      TEXT                      NOT NULL,
   -- the message type, e.g. `TransactionRecorded`
   message_type    TEXT                      NOT NULL,
   -- serialised message data, e.g. to JSON
   data            JSONB                     NOT NULL,
   -- diagnostic information on when the message was scheduled
   scheduled       TIMESTAMP WITH TIME ZONE  NOT NULL    default (now())
);

As you can see, besides the message ID, type, and data, we’re also storing the (global) position number and transaction ID (to ensure we don’t skip in-flight transactions that have requested the global position number, read more here for reasoning).

Now we can be polling it with the query like:

SELECT
     position, message_id, message_type, data
FROM
     outbox
WHERE
     (
          (transaction_id = last_processed_transaction_id
               AND position > last_processed_position)
          OR
          (transaction_id > last_processed_transaction_id)
     )
     AND transaction_id < pg_snapshot_xmin(pg_current_snapshot())
ORDER BY
    transaction_id ASC,
    position ASC
LIMIT 100;

Now, thanks to that, we can have the global ordering guarantee on the receiving end. We’re trading a bit of performance for greater correctness. Not always acceptable, but for internal module communication or forwarding to the messaging system, that’s usually more than enough.

That’s also the place where we’re back to our checkpointing. How do we know the last processed position?

By default, that’s simple, we could either say:

start from the beginning and use -1 or some other hardcoded position,
start from the end and use the position of the last message in the table plus one.

Those cases can be fine when we need to handle all messages (e.g., adding a new process or reading a model), or when we don’t care about the past and need to process the newest notifications.

Still, the reality is just like in the old games, we’d like to start somewhere in the middle, precisely where we left off.

We need one more table for storing our checkpoints. It can look as follows:

CREATE TABLE processor_checkpoints
(
   -- subscription name
   processor_id                     TEXT      PRIMARY KEY,
   -- information about the position of the last processed message
   last_processed_position          INTEGER   NULL,
   -- used to detect gaps in numbering
   last_processed_transaction_id    xid8      NOT NULL
);

Simple stuff: the processor ID should be unique and reflect the processor’s logical name and the last processed position(s).

If that looks simple, then let me follow up with the next set of potentially simple questions:

How to store it?
When to store it?

If your answer is: “just do upsert statement”, then you’re kinda right, but that wouldn’t be simple, it’d be over-simplification. At least if you’d like to run it in production.

Let’s start with how. And, for that, let me bring you now the promised stored procedure with if statements, it’s a bit simplified version from Emmett:

CREATE OR REPLACE FUNCTION store_processor_checkpoint(
  p_processor_id      TEXT,
  p_position          BIGINT,
  p_expected_position BIGINT,
  p_transaction_id    xid8
) RETURNS INT AS $$
DECLARE
  current_position BIGINT;
BEGIN
  -- Handle the case when p_check_position is provided
  IF p_expected_position IS NOT NULL THEN
      -- Try to update if the position matches p_check_position
      UPDATE processor_checkpoints
      SET
        "last_processed_position" = p_position,
        "last_processed_transaction_id" = p_transaction_id
      WHERE "processor_id" = p_processor_id AND "last_processed_position" = p_check_position;

      IF FOUND THEN
          RETURN 1;  -- Successfully updated
      END IF;

      -- Retrieve the current position
      SELECT "last_processed_position" INTO current_position
      FROM processor_checkpoints
      WHERE "processor_id" = p_processor_id;

      -- Return appropriate codes based on current position
      IF current_position = p_position THEN
          RETURN 0;  -- Idempotent check: position already set
      ELSIF current_position > p_expected_position THEN
          RETURN 2;  -- Failure: current position is greater
      ELSE
          RETURN 3;  -- Default failure case for mismatched positions
      END IF;
  END IF;

  -- Handle the case when p_check_position is NULL: Insert if not exists
  BEGIN
      INSERT INTO processor_checkpoints ("processor_id", "last_processed_position", "last_processed_transaction_id")
      VALUES (p_processor_id, p_position, p_transaction_id);
      RETURN 1;  -- Successfully inserted
  EXCEPTION WHEN unique_violation THEN
      -- If insertion failed, it means the row already exists
      SELECT "last_processed_position" INTO current_position
      FROM processor_checkpoints
      WHERE "processor_id" = p_processor_id;

      IF current_position = p_position THEN
          RETURN 0;  -- Idempotent check: position already set
      ELSIF current_position > p_expected_position THEN
          RETURN 2;  -- Insertion failed, row already exists with a greater position
      ELSE
          RETURN 3;  -- Default failure case for mismatched positions
      END IF;
  END;
END;
$$ LANGUAGE plpgsql;

Oooh, even by copy and pasting, I’m already tired; there’s a fair reason why we’re not doing that too often nowadays.

Let me untangle that for you:

We’re trying to update the existing position or insert it if we’re storing it for the first time.
If all went fine, we’re returning 1 as a result to denote complete success.
If we saw that the checkpoint in the database had the same value, we’re returning 0.
If we saw that the checkpoint is different from what was expected and further away from it, then we’re returning 2.
Otherwise, we’re returning 3, which means that the checkpoint is different from the expected and older.

Essentially, by passing the expected position, we can detect whether we:

already handled the specific position,
have some competing instance of our processor handling our data.

That’s why we’re doing this fancy dance with IF statements and a stored procedure.

Detection assumes that we have a global ordering processing guarantee (thus, tricky bits with transaction ID).

It also shows why global ordering is useful.

By detecting that we’ve already handled a specific position, we can skip processing handling idempotency on the processor level.

By detecting that there’s another processor with the same id processing messages, we can make it more resilient and detect the noisy neighbour issue.

How would that look in the code?

async function handleBatch(messageBatch: RecordedMessage[], context: ProcessorContext): Promise<BatchHandlingResult> {
  const { checkpoint } = messageBatch[messageBatch.length - 1].metadata;

  return context.pool.withTransaction(async (transaction) => {
    for (const message of messageBatch) {
      await context.onMessage(message);      
    };

    // No error was thrown: proceed to store checkpoint of the last processed message
    const result = await storeProcessorCheckpoint(transaction.execute, {
      processorId: context.processorId,
      newCheckpoint: checkpoint,
      lastProcessedCheckpoint: context.lastProcessedCheckpoint,
    });

    if(result.success) {
      await transaction.commit();
    } else {
      // no need to do here, either we already handled it
      // or we have a mismatch of expected and existing checkpoints
      await transaction.rollback();
    }

    return result;
  });
}

type ProcessorContext = {
  processorId: string;
  lastProcessedCheckpoint: bigint | null;
  onMessage: (message: AnyMessage) => Promise<void>;
  pool: ConnectionPool;
}

type BatchHandlingResult =
  | {
      success: true;
      newCheckpoint: bigint | null;
 }
  | { success: false; reason: 'IGNORED' | 'FURTHER' | 'OLDER' };

async function storeProcessorCheckpoint(
  execute: SQLExecutor,
  options: {
    processorId: string;
    newCheckpoint: bigint | null
    lastProcessedCheckpoint: bigint | null;
    partition?: string;
 },
): Promise<BatchHandlingResult> {
  const { result } = await single(
      execute.command<{ result: 0 | 1 | 2 | 3}>(
        SQL`SELECT store_processor_checkpoint(
            ${options.processorId},
            ${options.newCheckpoint},
            ${options.lastProcessedCheckpoint},
            pg_current_xact_id()
        ) as result;`,
      ),
    );

    return result === 1
      ? { success: true, newCheckpoint: options.newCheckpoint! }
      : { success: false, reason: result === 0 ? 'IGNORED' : result === 2 ? 'FURTHER': 'OLDER' };
};

As you can see, thanks to:

global ordering,
checkpoint detection,
storing checkpoint where our side effects will be stored,
transactional capabilities of our end storage,

We can ensure that the entire batch is processed or not. We could even optimise it by storing the checkpoint first and not processing the business logic if there’s a mismatch, then committing only if the logic succeeds.

We’re getting by that generic idempotence check and detection of the noisy neighbour.

Of course, I still believe that idempotence check should happen on the business logic side. But why not both?

Being able to detect a noisy neighbour can help you automatically stop (or pause) one of the competing consumers and avoid inconsistency conflicts.

What are the tradeoffs of this approach?

This will work if we have a global ordering guarantee. Not many messaging solution gives us such. If we have a subscription-based outbox as explained, event store like Emmett, Marten or KurrentDB, Kafka, this will work, but not necessarily for solutions like RabbitMQ, SQS, Google Pub Sub, etc.
This works best if you have transaction capabilities. Batching updates generally improves performance, but sometimes can lead to long-lived transactions; beware of that. The subscription-based solution with a transaction ID also works best if your transactions are short. If they’re open for a long time, it can cause delays.
Still, even without transactions, it will work fine, as long as you’re fine with having retries more often. If the business logic fails and the checkpoint is not committed, it’ll reprocess the already-handled messages from the previously stored messages. Which means that they can be handled more than once, but you should not lose any messages. You may also not fully benefit from the idempotence check for skipping already handled messages.

As always, the choice is yours.

Still, I hope that this article will show you why:

having a global ordering guarantee can be useful,
why and how to checkpoint your processing, how they relate to level codes from old games like Super Frog,
What are the tradeoffs, and how to consider them,
…and that SQL IF statements are sometimes justified. But don’t go wild with them!

And hey, I also hope that’s not something that you’d like to maintain on your own. There are mature tools to deal with such stuff, like Emmett, which implements this for you.

What are your thoughts? Questions? Concerns?

Cheers!

Oskar

Consumers, projectors, reactors and all that messaging jazz in Emmett

Did you know that you can build an event store in one hour? I even did it a few times on the conference stage. Actually, it took me usually around 25 minutes; the rest was mistyping, lame jokes and a bit of explanation. See:

Yet, my final thought was: Kids don’t do it at home.

It’s a fun coding exercise, but using the outcome in production? Not as much fun running and maintaining it. Why though? How hard can it be?

Event Sourcing systems have two phases: appending events and processing them afterwards. The write side gets most of the attention in tutorials and talks - commands, deciders, event stores, optimistic concurrency, as you saw, I’m also one to blame.

Providing the guarantees on the write side is relatively simple, especially if you use a database like PostgreSQL as a storage. You need to provide features like:

appending an event at the end of the stream,
reading all events from the stream,
a guarantee of the ordering within the stream,
being able to read your writes,
strong-consistent, atomic writes and optimistic concurrency.

That can be solved with knowledge about transactions, database design, etc. So again, why so hard?

The processing side is where systems often struggle as they grow. This is where the Event Sourcing solution becomes an Event-Driven Messaging tool. And if you’ve read my previous articles, you know that this can be tricky at times.

How do you reliably process events to build read models? How do you trigger side effects without losing messages? How do you scale processing independently from writes? How do you make it performant and run multiple handlers in parallel?

I’ve been working on the message processing architecture in Emmett for a while now. I’ll try to explain how I designed the split between Consumers and Processors, the problems it solves, and the tradeoffs involved.

Why Split Consumers and Processors?

When processing messages, we already know that someone produced them. We’re on the receiving end. Facts are already known; now we need to do something about them.

When we process them, do we care about the source? Typically, we take the information it gathers and reason about it. For instance, when we received an event indicating that a room reservation was made, we may need to send an email with details to the consumer, update the reservations dashboard, and generate a pro forma invoice. We may have specific logic, depending whether it came from our internal reservation platform or Booking.com, but we know the source from the message payload.

That seems obvious, but it was an important realisation for me. When we’re building a read model in MongoDB, we don’t care if events come from PostgreSQL event store, EventStoreDB, RabbitMQ queue or Kafka topic.

It needs events and the projection logic. Of course, it needs to know the guarantees around: delivery, ordering, idempotency, etc., but besides that? The message’s source doesn’t matter to its logic.

Similarly, a component polling PostgreSQL for messages to publish them doesn’t care what happens to those events - whether they update read models or trigger webhooks is irrelevant to polling logic.

These concerns are orthogonal.

I realised that much of the complexity comes from coupling those two together. We wouldn’t like to change our processing logic because of an internal change in how they’re produced, or vice versa. I concluded that separating them means each can evolve independently. And I came with the initial idea for the split: Consumers and Message Processors.

Consumers are responsible for getting messages from a source and forwarding them to processors. Think of them as the “delivery mechanism.” They handle the “where do events come from” concern. A consumer might connect to:

A PostgreSQL event store, polling the events table,
EventStoreDB, using push-based catch-up subscriptions,
Kafka, consuming from topics,
Any other message source you might have.

Processors are responsible for doing something meaningful with those messages. They handle the “what do we do with messages” concern. A processor might:

Update a read model in PostgreSQL or MongoDB,
Call an external API when certain events occur,
Publish events to Kafka or fire webhooks,
Trigger workflow steps or saga operations.

This separation follows Unix philosophy: small, focused components connected by simple interfaces. Each piece does one thing well. You can plug any processor into any consumer. This gives you flexibility that matters in practice:

You can run the same projection logic against different event sources
Adding new processors doesn’t require changing consumer code
You can test processors in isolation with fake event streams
Consumers and processors can be scaled independently

Let me show you how this looks in practice.

How Consumers Work

I deliberately decided to keep Consumers as dumb as possible. A consumer’s entire job is:

Connect to a message source,
Poll messages in batches or subscribe to notifications (depending on the source specifics).
Forward them to all registered processors.
Go back to step 2.

That’s it. No business logic. No complex state management. No decision-making about what to do with messages. Consumers are essentially routers.

Why this simplicity? I’m a simple guy; I like clear boundaries for responsibility. They help me reason about both how to use the tool and how to handle it when things go wrong. When message delivery breaks, you want to know exactly where to look. With a simple consumer, the question is binary: did it deliver messages or didn’t it? There’s no complex interaction between delivery logic and processing logic to untangle at 3 AM.

For EventStoreDB, the consumer creates a single subscription and fans out messages to all registered processors. For PostgreSQL, it polls the message table in batches, handling ordering guarantees (we’ll get to why that later).

Here’s what a basic consumer setup looks like for PostgreSQL :

const consumer = postgreSQLEventStoreConsumer({
  connectionString,
  processors: [
    shoppingCartDetailsProjector,
    customerAnalyticsProjector,
    orderNotificationReactor
  ]
});

await consumer.start();

Set up looks accordingly for the other sources providing options specific for the source, e.g. for EventStoreDB store, you may want to provide the category stream name:

const consumer = eventStoreDBEventStoreConsumer({
  connectionString,
  from: { stream: '$ce-roomRservations', options: { resolveLinkTos: true } },
});

The consumer receives messages, batches them and forwards them to all processors. Each processor handles messages independently - they don’t know about each other, and they don’t need to.

Why Batching Belongs to Consumers

Who decides how many messages to fetch at once? That’s a tricky question. I think that batching can happen both at the consumer and processor levels. The consumer decides the batch size for the polling or receiving to tune the receiving throughput. The processor can either align with it, using those batches as a safe default or diverge to its specifics. Read more on Why you should batch message processing in my other article.

Different message sources have different optimal batch sizes. PostgreSQL might be efficient with 100-row fetches. EventStoreDB subscriptions don’t have built-in batching; they deliver events as they arrive. Kafka has its own batching semantics. These are all source-specific optimisations we should be able to apply without ending up with the lowest common denominator.

Processors, by default, can just receive batches and process them. Then they can decide whether to split batches into smaller chunks, group them into even larger chunks, or process them as single messages. For instance, PostgreSQL can handle random single updates pretty well, whereas Elastic prefers batching updates.

How Processors Work

If consumers are simple routers, processors are where the interesting work happens. They’re the smarter ones in this relationship. A processor is responsible for:

Processing logic: Actually doing something useful with events. Updating a read model, sending an email, calling an API.
Checkpointing: Tracking which messages have been processed. This is crucial - without it, you’d reprocess everything from the beginning every time you restart.
Error handling: Deciding what to do when processing fails. Retry? Skip? Stop everything?
Idempotency: Doing their best to ensure that reprocessing the same event doesn’t cause problems. Of course, still assuming that handlers should be idempotent, read more in my other article,
Backpressure: They need to be able to tell consumers that they cannot process more messages at the moment, and that the consumer needs to slow down delivery.

Those are general promises and common stuff for the message processing logic. Still, there are multiple reasons why you want to process incoming messages:

Read models (projections) transform events into queryable state. For instance, shopping cart events - ProductItemAdded, ProductItemRemoved, ShoppingCartConfirmed - need to become a document showing current items, quantities, and totals. Something your API can quickly return when a user opens their cart.
Reactions trigger side effects after a business fact has happened. When a shopping cart is confirmed, you may want to send a confirmation email, notify the shipment module, and register a new order. These things need to happen, but they’re not part of the core business logic.
Workflows coordinate multi-step processes across multiple streams. An order might involve payment processing, inventory reservation, and shipping coordination - each with its own state and events.
Integration means forwarding events to other systems. Other services in your systems might need to know about orders. External partners might need webhook notifications. You might publish to messaging systems for downstream consumers.

All of those processing needs a bit different ways to handle reliability, ordering, throughput, etc. Also, all tools we integrate with require a different approach: storing the read model in PostgreSQL will be quite different from forwarding a message to Kafka.

I wouldn’t like to handwave all of those specifics and end up with the lowest common denominator. That’s why I decided to group them into the following archetypes:

projectors,
reactors,
workflows,
allow custom message processors to allow people to tune it fully to their needs,
and in the future, stuff like forwarders, web hooks and others we find useful.

All of them should have a unified API that allows them to be plugged into different consumers, while also embracing differences in message processing and target API specifics.

That’s also why each message processing target (PostgreSQL, EventStoreDB, MongoDB, InMemory, Kafka, SQS, etc.) will have its own implementations.

I believe that this focused responsibility, different archetypes, and specific implementations for different tools will strike the right balance between reusability and avoiding the lowest common denominator. We’ll see if that’s not a famous last words.

Read also more in:

The example projector can look like that:

const projection = pongoSingleStreamProjection({
  collectionName: shoppingCartsSummaryCollectionName,
  evolve,
  canHandle: ['ProductItemAdded', 'ShoppingCartConfirmed'],
  initialState: () => ({
    status: 'pending',
    productItemsCount: 0,
 }),
});

const postgreSQLProjector = postgreSQLProjector({ projection });

const reactor = postgreSQLReactor({
    processorId: 'order-notifications',
    canHandle: ['ShoppingCartConfirmed'],
    eachMessage: (event) =>
      emailService.sendOrderConfirmation(event.data.customerId);    
  }

Native implementations of processors

Different storage requirements require different capabilities, and getting proper guarantees might involve deeper knowledge. For instance, Postgres sequences issues can impact your messaging guarantees. Those are cases where, when you’re starting, you might not anticipate. Test environments may not even catch it; you might realise you’re losing business data when you reach production. That’s why it’s, imho, better to have a tool that solves it rather than trying to maintain it on your own, making technical infrastructure something you need to keep working on instead of your business features. How does Emmett solve them? Let’s discuss them briefly. I’ll try to expand in the future posts about the details.

Resilience

What happens if the processor fails? By default, it stops processing. But only this one, the consumer keeps pushing events to the other processors that can continue. Consumer stops when all their processors are inactive.

Why? Consider this scenario: you have two processors, one updating MongoDB and another updating PostgreSQL. MongoDB becomes temporarily unavailable. Should that stop PostgreSQL updates?

Still, failure behaviour is configurable; your message handler can return:

void/ACK: message processed successfully, continue to the next one.
SKIP: Skip this message, useful for poison messages that consistently fail
STOP: Stop this processor entirely.

Why have skip separate from ACK? Consider a poison message - a message that causes your processor to fail every time. Without skip, you have two bad options: fail forever (blocking all processing) or ACK it (pretending you processed it). With Skip, you can move it to a dead-letter queue for investigation while continuing to process other messages.

For now, Emmett doesn’t support Dead Letter/Poison Message Queues out of the box, but they will be supported in the future. You could already append those events to some specific stream.

In upcoming releases, we’ll also have configurable retry policies based on error type and other factors. Just like we already have for command handlers (e.g. to retry 3 times with exponential backoff for Optimistic Concurrency error).

There’s no easy answer to when to stop and when to skip poison messages. Neither choice is universally correct. A financial system might need all-or-nothing semantics. A social media feed can tolerate inconsistency between views.

That’s also why you can freely group processors within consumers. Best if they share similar resiliency and desired throughput characteristics. If they’re very different, you can always spin up another consumer for the same source and process it differently.

Checkpointing processing

In Emmett, processors own their checkpoints. Each processor independently tracks the last message it processed. The consumer doesn’t maintain any checkpoint state.

When a consumer starts up, it asks all registered processors for their last processed position and starts polling from the earliest one.

It has several benefits:

Independent progress: Processors can move at different speeds. If your MongoDB projector is fast and your analytics processor can get slow at times, they each track their own progress. The slow one doesn’t hold back the fast one.
Isolated failures: If one processor’s checkpoint storage fails, only that processor is affected. Others continue working.
Easy replay: To rebuild a single projection, you just reset that processor’s checkpoint. No need to coordinate with other processors or manage a global position.
Flexibility: Processors can store checkpoints wherever makes sense - in the same database as their read model, in a separate checkpoint table, or anywhere else.
Capability to redistribute the load. As mentioned in the previous points, if you observe that one of the processors is slower or demands more resources, you can freely deploy it separately in a different consumer, and it’ll start where it left off.

The tradeoff? When a consumer restarts, it might poll events that most processors have already seen. If one processor is significantly behind, all processors receive those events again (they just skip them based on their checkpoints). This is why you should group processors by their typical processing pace - don’t put a real-time dashboard projector and a monthly analytics batch processor on the same consumer.

Backpressure: When Processors Can’t Keep Up

Backpressure occurs when processors can’t process messages fast enough for the consumer to deliver them. This is a real operational concern that needs explicit handling.

Emmett doesn’t support it at the moment, but here’s what I’m thinking about it.

There are several strategies, each with tradeoffs:

1. Ignore backpressure: Consumer keeps polling and pushing regardless of processor state.

Pro: Simple, maximum throughput when processors can keep up
Con: Memory grows unbounded, possible OOM, cascading failures

2. Stop on any slowdown: If any processor signals it’s overwhelmed, stop polling.

Pro: Safe, no resource exhaustion
Con: Slowest processor determines overall throughput

3. Force synchronised pace: All processors must process each batch before the next is fetched.

Pro: All processors stay in sync, predictable memory usage
Con: The Slowest processor becomes the bottleneck for all

4. Slow down ingress: Adaptively reduce polling rate based on processor feedback.

Pro: Balances throughput and stability
Con: More complex, needs tuning

5. Rolling buffer: Buffer messages up to a limit, retry delivery to slow processors.

Pro: Absorbs temporary slowdowns, maximises throughput
Con: Needs memory limits, complex failure handling

Different systems need different strategies. Real-time dashboards might use strategy 1 (drop messages rather than lag). Financial transactions might use strategy 3 (consistency over throughput). Event forwarding to Kafka might use strategy 5 (buffer temporary network issues).

I’m leaning toward making this configurable per consumer, with sensible defaults. The default would be a bounded buffer with adaptive polling slowdown.

Scaling: Current State and Future Plans

For now, the big benefit of having dumb consumers is that you can scale them horizontally. Of course, this works for offset-based solutions like event stores and streaming tools like Kafka. It may not always work for systems that remove the message once it’s handled. Still, current consumers are using only event stores as sources; Kafka will likely come next.

You can group processors into consumers by that, reducing the number of polling jobs (one consumer polls/subscribes to one source).

I already mentioned batching, which should also increase the throughput.

Running multiple instances of the same processor causes conflicts. Both process the same events, update the same read models, and corrupt the state. Emmett already has the basic capability to do distributed locking, but it’s not fully plugged yet. This will come in future releases.

For now, checkpointing can detect whether a newer checkpoint is already stored (which can suggest another processor is running) and stop processing.

The recommended approach is to run the consumer as a separate service from the API. Then you can scale it separately. You can also set replicas=1 for the specific consumer to ensure one instance.

Rebuilding Projections

Event Sourcing enables rebuilding read models from events. Bug in projection? Fix code, rebuild. New read model? Populate from history.

With processor-owned checkpoints, you can either rebuild read model from scratch by:

Stopping the processor.
Delete read model data.
Reset the checkpoint to the beginning.
Restart processing.

Or doing blue greeen by:

Creating a new version of your storage (with Pongo, it’s just adding a suffix or prefix to your collection name).
Start consumer since the beginning.
Check if read models are close enough, and stopthe old processor
Start processing.

In Emmett you have even some syntactic sugar on top of consumers and processors to make this easier:

import { rebuildPostgreSQLProjections } from '@event-driven-io/emmett-postgresql';

const rebuilder = rebuildPostgreSQLProjections({
  connectionString,
  projection: shoppingCartsSummaryProjectionV2
});

await rebuilder.start();

This will spin up a new consumer; other consumers and processors continue normally, with their checkpoints unaffected. You can specify the position from which you want to start, and also whether to truncate the end storage.

We’ll need more metrics like gap detection and distributed locking to make it more plug-and-play.

Wrapping Up

The consumer/processor architecture in Emmett is about making event processing concerns explicit and separable:

Consumers handle delivery - getting events from sources to processors. They’re simple by design. When delivery breaks, you know where to look.

Processors handle processing - doing useful things with events. They own their checkpoints, track their own progress, and handle their own failures.

This separation gives you:

Flexibility to mix and match consumers and processors,
Independent scaling of different processing workloads,
Isolated failure domains,
Easy projection rebuilds,
Testability at multiple levels.

The design makes tradeoffs explicit:

Partial progress over all-or-nothing (configurable soon),
Processor-owned checkpoints over global tracking,
Simplicity in consumers, complexity in processors,
Eventual consistency for async operations.

There’s more to build - distributed locking, partitioning, better backpressure handling. There’s still a lot to do, but I believe the foundation is there, and I know real applications are using it already.

I hope that this is a good food for thought, even if you’re not using Emmett. I’m curious about your thoughts and feedback. I’ll try to tackle those cases in more detail in dedicated articles.

If you have questions, feedback, or would like to help me speed up the planned stuff, come chat in the Emmett Discord. We have a small, but welcoming and awesome community.

Cheers!

Oskar

Requeuing Roulette in Event-Driven Architecture and Messaging

I’m always saying that there’s a thin line between good and bad practice, and this thin line is named “Context”.

That’s also true for the (anti-)pattern I’m calling “Requeuing Roulette”. Let’s discuss it today, continuing the “race condition series”:

What’s the Requeuing Roulette? As the name suggests, this technique involves putting a message back into the queue. It also (correctly) suggests that we’re hoping for the best. And sometimes we may be lucky to be true.

The basic primitive for a messaging system is a queue. The producer is putting messages into the queue, and the consumer is getting them on the other end. If everything goes well, the consumer receives them in the order the producer put them (thus, a queue, like a queue in a shop).

If the consumer is not available, the messaging system will try to deliver messages and handle retries for us.

We discussed it in detail in:

Ordering of processing works if we have a single consumer for a single queue. If we have more than one consumer, we lose the ordering guarantee. Why would we want to have more than one consumer? Obviously, to speed up processing. If messages in the queue are not causally correlated, then we can process them in parallel.

What does the smartass “causally correlated” even mean? For instance:

depositing money into a bank account is causally correlated to opening it, as we can’t deposit money if we don’t open it.
depositing money into a bank account is NOT causally correlated to other deposits, as we can deposit as much money as we have (of course, ignoring weird regulations),
money withdrawal is causally correlated to depositing money and other withdrawals, as we need to check the balance, and they may impact it.
Withdrawals and deposits are only causally correlated if they happen on the same bank account; other bank account operations can happen at any time.

You get the idea, aye?

So if we set up a queue to process money transfer events, then it could look as follows:

I assumed that we’re following the advice from the previous article. Besides the event type and payload, we’d also pass the record revision, which represents the logical order of events. It comes from the number incremented with each change. Assuming we’re publishing events after each successful business logic handling, it should be gapless.

You may notice that our queue actually has multiple timelines for each causally correlated message sequence. If we simplify our considerations and assume that all events from a certain account are causally correlated, then we could visualise them as:

It’s fine to process events that are not causally correlated, as by that we’re increasing throughput, not trading off correctness:

But it’s not fine to process messages from the same timeline in parallel, as we may get race conditions when the consumer processing the earlier message will be slower than the one processing the later message.

We actually have two orders in messaging systems:

Queue Order: The order messages are produced to the queue
Processing Order: The order messages are actually consumed and processed.

We already learned that we can detect the out-of-order processing by:

We can also apply techniques like the Phantom record, where we keep data as it comes and take the next steps only when defined conditions are met. And that’s also what I’d recommend in general, but…

But I promised you to talk today about the Requeuing Roulette (anti)pattern, aye? So let’s do it!

If we’re using tools like RabbitMQ, SQS and other classical messaging tooling (so not you Kafka! You’re a streaming or log solution!), then we can put the message back in the queue.

RabbitMQ message ordering documentation states:

Messages published in one channel, passing through one exchange and one queue and one outgoing channel will be received in the same order that they were sent. RabbitMQ offers stronger guarantees since release 2.7.0.

Messages can be returned to the queue using AMQP methods that feature a requeue parameter (basic.recover, basic.reject and basic.nack), or due to a channel closing while holding unacknowledged messages. Any of these scenarios caused messages to be requeued at the back of the queue for RabbitMQ releases earlier than 2.7.0. From RabbitMQ release 2.7.0, messages are always held in the queue in publication order, even in the presence of requeueing or channel closure.

With release 2.7.0 and later it is still possible for individual consumers to observe messages out of order if the queue has multiple subscribers. This is due to the actions of other subscribers who may requeue messages. From the perspective of the queue the messages are always held in the publication order.

The last paragraph seems promising, as it suggests the message will be put back before the next messages, since it was placed in the queue.

Unfortunately, it’s only a best effort. Another place in the documentation states:

When a message is requeued, it will be placed to its original position in its queue, if possible. If not (due to concurrent deliveries and acknowledgements from other consumers when multiple consumers share a queue), the message will be requeued to a position closer to queue head.

So in the worst case, it can even end up like that:

Then the fun will begin. Now we have two messages that we already see will be handled out of order, and we’ll need to requeue them, hoping they land after the message we need to process as the next one. What if we have multiple messages like that? What if our consumer randomly fails when processing our message with revision 12, and we need to requeue it again? How lucky will it be with Requeuing Roulette?

As you can see, the more correlated our messages are and the more we’d like to parallelise them, the more likely we are to face Requeuing Roulette. Typically, we put messages into the same queue so we can process them in order.

When Requeuing Roulette is helpful?

It can be useful if:

We want to get the best parallelism for message processing, and ordering best effort is good enough for us.
Our messages are most of the time not causally correlated or…
Events for the same records/processes are not quickly published one after another, so we could safely retry message before the next one will arrive,
Our consumers are stable, not failing too often.

As you see, those assumptions can be fragile and classical famous last words.

Of course, we can use one of the techniques like:

RabbitMQ routing key, correlation id,
AWS SQS message group id, visibility timeout,
Azure Service Bus sessions,
etc. see Ordering, Grouping and Consistency in Messaging systems for details.

Making this trade-off more in favour of the ordering guarantee or parallelism may reduce it to an acceptable level, but you need to be aware of the risk of an unexpected traffic spike or a different message distribution than you expected.

Dangers of Requeuing Roulette

Even when order doesn’t matter, requeueing has a hidden cost that becomes visible under load.

Suppose you reject the message with requeue set to true. In that case, it can be redelivered to your consumer almost instantly, resulting in a very high workload, since your consumer will reject it again.

Let’s say you have a message that fails because a downstream service is down. You requeue it. It immediately returns to the consumer (maybe even the same one), fails again, and is requeued. This can happen hundreds of times per second. It can also swamp the slow consumer, preventing it from even recovering.

In the worst case, your CPU can be spent processing and requeueing the same 10 messages over and over, while thousands of processable messages sit behind them in the queue (because RabbitMQ will try to put requeued messages before the next messages).

What about Kafka?

Well, in Kafka, this issue doesn’t exist as Messages with the same record key go to the same partition, maintaining order within that partition while allowing parallel processing across partitions.

So Kafka, for the win? Hold your horses!

Only one consumer from the consumer group can handle a specific partition. So, within a single partition, parallelisation isn’t possible. If we map the RabbitMQ queue to Kafka’s partition, then the conclusion can be that Kafka solved it by removing this feature.

Also, when we consume a message from the classical messaging system (like RabbitMQ), it will be removed from the queue. In a streaming solution like Kafka/Pulsar, etc., they will remain in the log until the retention policy kicks in and drops old messages from the partition.

Kafka maintains the offset of the last processed message in each topic partition. You don’t need to requeue messages; you can just rewind the offset to an older position when you want to reprocess messages.

TLDR

The “requeueing roulette” is a symptom of trying to solve a distributed systems problem with a technical solution.

The requeueing roulette is seductive because it promises something impossible: maintaining strict order in a distributed, concurrent system without sacrificing throughput. It’s trying to cheat the fundamental trade-offs of distributed systems.

Still, cheating can take us far enough, but there’s always a danger that we’ll be caught and handcuffed.

If you’re considering using Requeuing Roulette, then consider the other techniques I described in previous articles. I’d treat Requeuing Roulette as a temporary solution and a tradeoff.

In my opinion, if you decide to use it, then the question isn’t whether you’ll abandon it, but how much pain you’ll endure before you do.

The real skill isn’t in making requeueing work - it’s in understanding your actual ordering requirements and choosing the most straightforward solution that meets them. Often, that means accepting that perfect ordering is neither necessary nor worth its cost, especially in the long term.

Read also more in:

Cheers!

Oskar

Handling Events Coming in an Unknown Order

After the last article on Dealing with Race Conditions in Event-Driven Architecture with Read Models, I got such a question from Ben:

You described the scenario where you know what events you should receive, just not the order. But what if you don’t know that? For example, you get an ItemRemovedFromCart event, but the item doesn’t exist in your view of the current state of the cart. Is it an invalid event? Or is there an ItemAddedToCart event that hasn’t come through yet?

That’s a good question, and good questions usually require more depth to give a precise answer. That’s what we’re here for!

Let’s follow up and discuss how to determine whether we have complete information for our events!

What we learned so far

Communication in messaging systems works like a department store. It has multiple cash registers and separate queues for each of them. You can only guess which person will be handled in a specific queue: first in, first out. Between queues, you only know that the slowest will be the one you’re standing in.

Of course, you could put a single cash register and a single queue, and everything would be sequential. Such a setup can work for small groceries with few customers. Tho, for a supermarket, it’d end up with an extremely long waiting queue.

You can think about a single module as a single cash register in a department store or a small grocery store. Inside it, you can get strict ordering of processing, but not in the relationship with the outside world. For instance, you may know that on Monday morning, you’re getting the fresh fruits delivery, and at noon, you’re getting the dairy delivery. And it’s typically like that, but from time to time, because of the fruit delivery delay, you may get your fresh dairy first.

Is it an issue? Not a huge one, as you just want them to come asap so you have fresh stuff to sell. When would it be an issue? If you were running the Milk Shake Cafe and needed both to make your special strawberry shake recipe.

In many systems, ordering is not a key concern, especially when we partition the workload. The issue may arise when we need to correlate separate actions.

In the last article, we discussed the payment verification workflow. To make the final decision, we needed to correlate data from the external payment gateway with our own modules, which calculate fraud scores, check limits, and assess risk. Only after receiving data on available merchant limits and the fraud assessment score could we make the final decision. Those pieces of information could return to us at different times and in a different order.

To resolve it, we were just gathering and aggregating data as they went. Then, after each step, we checked whether we now have all the data. If we had, we were making the final decision; if not, we were storing it as is, assuming that at some point, data would arrive.

If we model that as the workflow, then it’d look like that:

function decide(
  current: PaymentVerification | null,
  event: PaymentVerificationEvent
):
  | PaymentVerification
  | { document: PaymentVerification; events: VerificationEvent[] } {
  current = current ?? {
    paymentId: event.paymentId,
    initialState,
  }

  switch (event.type) {
    // (...) other event handlers
    case 'MerchantLimitsChecked': {
      const updated = {
        ...current,
        merchantLimits: {
          withinLimits: event.withinLimits,
          dailyRemaining: event.dailyRemaining,
          checkedAt: event.checkedAt,
        },
        lastUpdated: event.checkedAt,
      };

      return tryCompleteVerification(updated, event);
    }
    case 'FraudScoreCalculated': {
      if (
        current.fraudAssessment &&
        event.calculatedAt <= current.fraudAssessment.assessedAt
      )
        return current;

      const updated = {
        ...current,
        fraudAssessment: {
          score: event.score,
          riskLevel: event.riskLevel,
          assessedAt: event.calculatedAt,
        },
        lastUpdated: event.calculatedAt,
      };

      return tryCompleteVerification(updated, event);
    }
  }
};

And:

function tryCompleteVerifications(
  current: PaymentVerification,
  event: PaymentVerificationEvent
):
  | PaymentVerification
  | { document: PaymentVerification; events: VerificationEvent[] } {
  // Ignore if we already made decision
  if (current.decision)
    return current;

  // Check if we now have BOTH critical pieces
  if (!current.fraudAssessment || !current.merchantLimits)
    // Don’t have both yet - stay in processing
    return {
      ...current,
      status: 'processing',
      dataQuality: 'processing',
    };

  const decision =
    current.fraudAssessment.riskLevel === 'high'
      ? {
          approval: 'declined',
          reason: 'High fraud risk',
          decidedAt: event.checkedAt,
        }
      : !current.merchantLimits.withinLimits
      ? {
          approval: 'declined',
          reason: 'High fraud risk',
          decidedAt: event.checkedAt,
        }
      : {
          approval: 'approved',
          reason: 'Verified',
          decidedAt: event.checkedAt,
        };

  return {
    document: {
      ...current,
      status: decision.approval,
      decision,
    },
    events: [
      {
        type: 'PaymentVerificationCompleted',
        data: decision,
      },
    ],
  };
};

This works fine, as we know precisely which steps need to happen, so we know what we’re waiting for. And that’s how we made the full loop to Ben’s question. What if we didn’t know which steps we’re waiting for?

How to know what we don’t know?

Let’s have a look at the case brought by Ben: the e-commerce flow. First, we complete the shopping cart by adding and removing items, then we confirm it. The example event flow could look as follows for the online food ordering:

ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
CartConfirmed       (cartId:1, confirmedAt: 2025-11-03 11:44:27)

We see here that someone added the first Pizza, then maybe accidentally added it again, corrected their mistake, and confirmed the order.

Then, if that was an online ordering system and we had it integrated with the kitchen ordering, then we could get those events in a different order, for instance:

ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
CartConfirmed       (cartId: 1, confirmedAt: 2025-11-03 11:44:27)
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)

We see that someone removed one Pizza from their shopping cart, which suggests that some information is missing. When we get a confirmation event, we still know that there’s more to come, as an order with a removed item doesn’t make sense. The same goes for the information that one pizza was added; when we correlate it with the removal event having the same cart identifier, we still see zero items in the shopping cart. Once we get the next event, we will finally know that we have more than one item in our shopping cart.

Can we then proceed? Maybe yes and maybe no. For this particular order, it’d be correct, but what if our real order:

ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
CartConfirmed       (cartId: 1, confirmedAt: 2025-11-03 11:44:27)

Also, since messaging systems retry to ensure delivery, how would we know that those “doubled” events for adding or removing are actually distinct events and not just retries?

For instance, in such a delivery case:

ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
CartConfirmed       (cartId: 1, confirmedAt: 2025-11-03 11:44:27)
ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)

Let’s discuss a few strategies to deal with that!

External vs Internal events

Doctor, it hurts when I bend my arm this way.

Then don’t bend it this way

One of the most common mistakes we learn too late is separating our events into internal and external (or private and public). Internal information can and should be more granular. We need it to be precise in capturing the business context and making our decision.

Yet, other parts of our system don’t need to know all of that. Is the kitchen interested in the details of all the changes procrastinating customer made to their shopping cart? No, they just want the final information on which meal they need to prepare.

So in our example, if we published to the outside world just:

CartConfirmed  {
    cartId: 1,
    items: [ { name: "Spaghetti Carbonara" } ],
    confirmedAt: "2025-11-03 11:44:27"
}

Such a type of event is also called a Summary Event. We should not mistake it with the latest state. It’s still an event because it tells what has happened business-wise. It gathers all the information needed for other modules and summarises the changes. And no more than that. It should still be as small as possible and expose only the information that other modules need. It’s a contract made between different teams. I wrote about it in detail Internal and external events, or how to design event-driven API.

We can define such an internal event API as:

type ItemAddedToCart = {
  type: 'sc:int:ItemAddedToCart';
  data: {
    cartId: string;
    productItem: ProductItem;
  };
};

type ItemRemovedFromCart = {
  type: 'sc:int:ItemRemovedFromCart';
  data: {
    cartId: string;
    productItem: ProductItem;
  };
};

type CartConfirmed = {
  type: 'sc:int:CartConfirmed';
  data: {
    cartId: string;
    confirmedAt: Date;
  };
};

type ShoppingCartEvent =
  | ItemAddedToCart
  | ItemRemovedFromCart
  | CartConfirmed;

type ProductItem = {
  productId: string;
  quantity: number;
}

and public as:

type CartOpened = {
  type: 'sc:ext:CartOpened';
  data: {
    cartId: string;
    openedAt: Date;
  };
};

type CartConfirmed = {
  type: 'sc:ext:CartConfirmed';
  data: {
    cartId: string;
    productItems: { productId: string; quantity: number }[];
    confirmedAt: Date;
  };
};

type ShoppingCartExternalEvent = CartOpened | CartConfirmed;

As you see, we can even have more than one summary event, and not even be one-to-one with an internal event. Maybe we also have an analytics module that analyses how long it takes the user to make a final decision after adding the first product. Then we may decide to expose such an event, hiding the details of the internal flow. We’re also defending ourselves and minimising the need for versioning when flow changes.

Ok, but how to map internal events into external?

We can enrich them using such a function:

import type { ShoppingCartExternalEvent } from './shoppingCart.external';
import type { ShoppingCart, ShoppingCartEvent } from './shoppingCart.internal';

const enrich = (
  event: ShoppingCartEvent,
  state: ShoppingCart | null,
): ShoppingCartExternalEvent | [] => {
  switch (event.type) {
    case 'sc:int:ItemAddedToCart':
      return state == null
        ? {
            type: 'sc:ext:CartOpened',
            data: {
              cartId: event.data.cartId,
              openedAt: new Date(),
            },
          }
        : [];
    case 'sc:int:CartConfirmed':
      return {
        type: 'sc:ext:CartConfirmed',
        data: {
          cartId: event.data.cartId,
          productItems: state?.productItems ?? [],
          confirmedAt: event.data.confirmedAt,
        },
      };
    default:
      return [];
  }
};

We can then subscribe to internal events in our module, load the state (best to build it from events if we’re using Event Sourcing), and publish enriched events externally.

If we’re using messaging, this means also separating queues/topics. If you’re using Kafka both for internal and external communication, then you should separate topics and have two different topics for outgoing communication, e.g.:

‘carts:events:int’
‘carts:events:out’.

Similarly, for RabbitMQ or similar tools, you should have separate queues for internal and external communications.

This is important, as you can now:

Publish messages that other modules need, decreasing the number of issues with ordering,
Have enrichment as an anti-corruption layer for your internal process changes,
You can have different scaling capabilities for internal and external events. Maybe for internal, you don’t even need a messaging system, maybe outbox or event store subscriptions will be enough? Maybe you could cut costs using AWS SQS for internal communication and AWS Kinesis for cross-module?
You can now define different security for those topics and retention policies.

Sweet, right?

It’s not me, it’s them

Maybe it’s sweet enough for you, but you may also say:

But Oskar, I’m not producing those events, it’s the other team and the external service. If I was responsible for that, I’d go this way, but I can’t change it how messages are published.

I could handwave it and say I pity you, but well, this actually can happen. Let’s see what else we could do about it.

The first idea could be: Let’s add timestamps!

Let’s see how it looks for our example:

11:40:10 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
11:40:10 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
11:42:13 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
11:43:18 - ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
11:44:23 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
11:44:27 - CartConfirmed       (cartId: 1)

And the out of order delivery:

11:40:10 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
11:40:10 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
11:44:23 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
11:44:27 - CartConfirmed       (cartId: 1, confirmedAt: 2025-11-03 11:44:27)
11:43:18 - ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
11:42:13 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)

Would that help? No, because how would we know, based on timestamps, that there will be two more events after confirmation? Timestamps only tell us when a certain operation happens. They could help us order items that were delivered, but they won’t help us know what we’re missing. Because how do we know that there’s a gap in our knowledge? Within a minute, one could do nothing and order or remove a few more items. Also, timestamps might work if the data is coming from the same node, but we don’t have any guarantees across nodes. Read more about clock drift.

What we actually need is the logical clock. One that increments after each operation. So something like:

1 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
2 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
3 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
4 - ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
5 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
6 - CartConfirmed       (cartId: 1)

If we had such, then our delivery would look as follows:

2 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
1 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
5 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
6 - CartConfirmed       (cartId: 1, confirmedAt: 2025-11-03 11:44:27)
4 - ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
3 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)

If this number were monotonic and gapless, we’d know that if the event has number 3, then we have completeness of information if we received three events; if not, then we’re missing something.

We still need to define the completion criteria and determine, from a business perspective, where we can make a decision or proceed to the next step. Here we know that we can start preparing a meal when the shopping cart is confirmed.

In this case, we got the events in the following order: 2, 1, 5, 6.

We know we’re missing events 3 and 4, so we need to wait for them. Only when we receive them can we proceed. Ok, but how to do it?

What if we kept a list of pending events in our data model? Let’s try that!

Our kitchen order could look as follows:

type KitchenOrder = {
  orderId: string;
  productItems: ProductItem[];
  status: 'Incomplete' | 'Ready' | 'InPreparation';
};

type KitchenOrderCommand =
  | {
      type: 'AddItem' | 'RemoveItem';
      productId: string;
      quantity: number;
    }
  | {
      type: 'Confirm';
      orderId: string;
    };

type ProductItem = {
  productId: string;
  quantity: number;
};

When storing it, we could store it with additional metadata:

type DocumentWithPendingCommands<State, Command> = State & {
  metadata: {
    lastProcessedRevision: number;
    pendingCommands: PendingCommand<Command>[];
  };
};

As you see, besides the regular data, we have two other properties: pending commands and last processed revision.

You can think about pending commands as your git repository on your local disk. It contains the list of all operations that you’ll eventually commit. The rest of the data is like the remote git repository. They will be updated when you push your changes there. Then, the last processed revision will be updated with the revision of the last applied command.

The code for that workflow could look as follows:

function handle(
  event: ShoppingCartEvent,
  document: DocumentWithPendingCommands<
    KitchenOrder,
    KitchenOrderCommand
  > | null,
): DocumentWithPendingCommands<KitchenOrder, KitchenOrderCommand> {
  const { metadata, ...data } = document ?? {
    metadata: {
      lastProcessedRevision: 0,
      pendingCommands: [],
    },
  };

  const state: KitchenOrder = {
    orderId: event.data.cartId,
    productItems: [],
    status: 'Incomplete',
    ...data,
  };

  const updated = {
    ...state,
    metadata: {
      lastProcessedRevision: metadata.lastProcessedRevision,
      pendingCommands: [
        ...metadata.pendingCommands,
        mapToPendingCommand(event),
      ],
    },
  };

  return handlePendingCommands(updated, decide);
}

It takes the current state of the kitchen order. If it doesn’t exist, then we need to set it up with default data. We’re also appending a pending command that’s built from an event.

function mapToPendingCommand(
  event: ShoppingCartEvent,
): PendingCommand<KitchenOrderCommand> {
  switch (event.type) {
    case 'sc:int:ItemAddedToCart':
      return {
        type: 'AddItem',
        productId: event.data.productItem.productId,
        quantity: event.data.productItem.quantity,
        metadata: {
          revision: event.metadata?.revision,
        },
      };
    case 'sc:int:ItemRemovedFromCart':
      return {
        type: 'RemoveItem',
        productId: event.data.productItem.productId,
        quantity: event.data.productItem.quantity,
        metadata: {
          revision: event.metadata?.revision,
        },
      };
    case 'sc:int:CartConfirmed':
      return {
        type: 'Confirm',
        orderId: event.data.cartId,
        metadata: {
          revision: event.metadata?.revision,
        },
      };
  }
}

Why not use a regular event here? We could. The benefit is that then we’d have all the data stored as it came. This could make troubleshooting and correction easier. Downside? We’re coupling the event with our internal business logic. Also, keeping the whole event payload increases the size of our data. The choice is yours.

As you see, we’re taking revision from the event metadata. We’ll get to that later, how to fill it on the producer side. Now, let’s focus on the workflow.

Let’s see what processing pending items looks like:

function handlePendingCommands<State, Command>(
  document: DocumentWithPendingCommands<State, Command>,
  decide: (command: Command, state: State) => State,
): DocumentWithPendingCommands<State, Command> {
  const { metadata, ...data } = document;

  const commandsToHandle = getCommandsReadyToHandle(
    metadata.pendingCommands,
    metadata.lastProcessedRevision,
  );

  // Nothing to do see here, please disperse
  if (commandsToHandle.length === 0) return document;

  let state = data as State;
  for (const command of commandsToHandle) {
    state = decide(command, state);
  }

  const lastCommand = commandsToHandle[commandsToHandle.length - 1];

  return {
    ...state,
    metadata: {
      lastProcessedRevision: lastCommand.metadata.revision,
      pendingCommands: metadata.pendingCommands.filter(
        (a) => a.metadata.revision > lastCommand.metadata.revision,
      ),
    },
  };
}

We need to get the commands ready to be handled. For instance, if we already had commands with the following revisions 2, 6, 1, 4, and now we got an action with number 3, then that means:

We can process actions 1, 2, 3, 4
Action 6 remains, as we’re missing 5.

The filtering can look as follows:

function getCommandsReadyToHandle<Command>(
  pending: PendingCommand<Command>[],
  lastProcessedRevision: number,
): PendingCommand<Command>[] {
  return (
    pending
      // filter out commands that have already been processed
      .filter((cmd) => cmd.metadata.revision > lastProcessedRevision)
      // sort by revision to ensure correct order
      .sort((a, b) => a.metadata.revision - b.metadata.revision)
      // only take commands that are consecutive in terms of revision
      .reduce<PendingCommand<Command>[]>((acc, command) => {
        const lastRevision = acc[acc.length - 1]?.metadata.revision;

        return !lastRevision || command.metadata.revision === lastRevision + 1
          ? [...acc, command]
          : acc;
      }, [])
  );
}

When we filter them out and have actions to process, we need to run the actual logic for each of them. For our case this could look like that:

function decide(
  command: KitchenOrderCommand,
  order: KitchenOrder,
): KitchenOrder {
  switch (command.type) {
    case 'AddItem':
    case 'RemoveItem': {
      if (order.status !== 'Incomplete') return order;

      const updatedItems = new Map(
        order.productItems.map((item) => [item.productId, item.quantity]),
      );
      const current = updatedItems.get(command.productId) ?? 0;

      const multiplier = command.type === 'AddItem' ? 1 : -1;
      const updated = current + multiplier * command.quantity;

      if (updated > 0) {
        updatedItems.set(command.productId, updated);
      } else {
        updatedItems.delete(command.productId);
      }

      return {
        ...order,
        productItems: Array.from(updatedItems.entries()).map(
          ([productId, quantity]) => ({
            productId,
            quantity,
          }),
        ),
      };
    }
    case 'Confirm':
      if (order.status !== 'Incomplete') return order;

      return {
        ...order,
        status: 'Ready',
      };
  }
}

type KitchenOrderCommand =
  | {
      type: 'AddItem' | 'RemoveItem';
      productId: string;
      quantity: number;
    }
  | {
      type: 'Confirm';
      orderId: string;
    };

After processing each command, we’re returning the document with the updated state, filtered-out processed commands, and the last processed revision set to the revision of the last processed command.

Not that big a hassle as it may seem, but…

What’s revision and how to get it?

A revision needs to be created on the producer side. The best is to use Optimistic Concurrency for that. If you don’t know what optimistic concurrency or locking is, then you should. Check my intros:

Essentially, if you’re:

using a typical implementation of optimistic concurrency where each record/document change increments its version/revision,
publishing an event after each business operation.

Then you can use this incremented state revision and pass it in your events metadata. Having that, you’d know precisely on which revision it was recorded and get gapless monotonic numeration. It’d also ensure that each message is produced in a certain order, as operations on the specific record will be sequential. Read also more on how revision can help in Dealing with Eventual Consistency and Idempotency in projections.

But Oskar, what if I have more than one record?

Well, then you either need to store multiple revisions. But this won’t help if you need to correlate data between them, as revision is monotonic and gapless for the specific record.

What about global positions? Well, they’re useful for knowing the order of things, but they won’t help here, as they’re monotonic but may have gaps.

TLDR

Proper modelling in Event-Driven Architecture can spare you a lot of complicated implementation tricks.

If you

define essential events for your process,
ensure that they have completeness of information,
shape contracts and communication between modules, respecting the internal and external split.

Then, when things get easier to handle, we can define conditions that tell us when to take action.

Still, sometimes we may:

have strict ordering needs,
be using a queue that doesn’t give us an ordering guarantee,
need to adjust to the other teams.

Then, using revision can be a decent option to solve things in an organised way.

If you’re dealing with such issues, I’m happy to help you through consulting or mentoring. Contact me and we’ll find a way to unblock you!

Read also more in:

Cheers!

Oskar

Dealing with Race Conditions in Event-Driven Architecture with Read Models

My events came out of order! What should I do?!

Are you familiar with the term “phantom record” and its benefits? No? Let me explain it to you today.

Everyone has a plan until they get punched in the mouth. Design, architecture, and modelling are important, but it’s the actual code that reaches production. That’s also the place where we see all those nasty issues that we haven’t foreseen, like: race conditions, eventual inconsistency, idemNotency, etc.

We realise that our BBC Architecture is just a Box-Box Cylinder on paper; in reality, things get messy. We go from a 2D layout to a 3D or even a 4D view.

We see that our business processes are not so linear and predictable as we draw them. Of course, we have somewhere in the back of our mind that subprocesses can go in parallel, but somehow our tests go down the happy path. Then bugs and incidents come in.

We learn the hard way that real processes have delays; they run in parallel. For instance, when you process a payment, fraud checking, risk assessment, and merchant validation happen simultaneously. That’s efficiency, not a flaw.

And the sad part is that it’s our role to provide this efficiency. Yelling at event-driven clouds won’t help.

Race Conditions in EDA

When we distribute our systems, we’re getting better isolation of failure, we can deploy changes at a different pace, but… But we also get communication going at various paces. We can no longer get a unified view of the business actions in the order of appearance. Now each service has its own linearity.

To integrate services predictably, we’re using messaging tools. They provide us with durability, retries when the recipient is unavailable, and ensure that information flows and data are delivered effectively. Still, they’re not magical creatures; they have their limits. We can’t cheat physics.

Between one service and the other, we’re putting in a queue. That’s the place where we can achieve an ordering guarantee. Typically, one queue, one consumer means that we can achieve ordering. Still, not all tools are giving us that.

Out of order issues may happen when the RabbitMQ queue has multiple consumers racing for messages. When we add consumers, we get better throughput, but we’re risking ordering issues.

Out of order issues may also happen when you’re using tools like SQS or Google PubSub, which only guarantee best-effort ordering.

Or when your outbox pattern deletes processed messages and loses sequence.

Or when network delays shuffle carefully ordered streams.

It’s safe to say that you won’t get any ordering guarantee between different queues. Since queues represent communication flows between modules, we should not assume strict ordering in cross-module communication.

Events vs Rumours?

Continuing with the payment verification process example: Fraud module publishes to one queue. Risk assessment for another. The payment gateway to a third. Messages arrive at your module in the order your system received them, not the order they were created. Fraud detection service flags a high-risk payment at 10:00:01. The payment initiation event from the gateway, created at 10:00:00, arrives at 10:00:02. The fraud score arrives before the payment exists in your system. You might think sorting by timestamp solves this. It doesn’t. Clock skew between services means timestamps lie. Each service node has its own time. It may be similar, but if you have a system with high throughput, the skew may be significant enough, causing more issues than actual help.

But, well, that’s the same case when you’re getting information from the outside world. You can get multiple friends sending you some news, you can get them at different paces, and read them in a different order. You may get obsolete news and then newer, but it can go the other way round. You’re only sure of which you received those messages; to get the truth, you need to correlate that information and do fact-checking based on some rules (e.g. which source is more reliable, newer, etc). Then you can deduce the actual information.

We’re saying that in the event-driven world, events are facts. But that’s just half the truth.

Those events that we store/publish are facts for us, or at least represent the current state of our knowledge.

The events from external systems are rumours at best. We need to interpret them to make them (our) facts.

Ok, but what can we do? Sit and cry?

Of course, we can do more. We’ll discuss today a simple technique with Read Models that can take you far enough.

Let’s say that you’re using Event Sourcing in your system. You take all the messages coming from other systems, and you store them in your event store.

Event Sourcing assumes you can rebuild state by replaying events in sequence. When FraudScoreCalculated arrives before PaymentInitiated, the payment doesn’t exist. You can’t apply a fraud score to nothing. You realise that when your carefully designed domain model throws exceptions. This should never happen, aye?

This isn’t specific to Event Sourcing. Even if you don’t use it but just update read models directly from events, you have the same problem. Your read model update handler for FraudScoreCalculated looks for a payment document to update. No document exists. The update fails.

Such issues are a recurring theme in my consulting, as they are a common concern for my customers. My advice is usually: Don’t try to fight them.

Store data as it arrives and “denoise” on your side. Interpret them and save your own “facts”. Acknowledge the need for a split between internal and external events. Use an Anti-Corruption Layer (ACL) pattern to protect from external chaos.

Read models as Anti-Corruption Layer

Read models can be used as such an ACL. A read model document can have a partial state, with optional fields that are filled as events arrive. You can process each event, updating the fields it can, ignoring those it can’t, and making decisions based on the available data.

Let’s explore how to apply this in practice using our payment orchestration scenario. Let’s say we’re gathering information about the payment verification process, displaying its state, and performing follow-up operations once the process is complete.

When we trigger payment, we need to correlate data from the external payment gateway with our own modules, which calculate fraud scores, check limits, and assess risk.

We could define events using TypeScript, as:

type PaymentVerificationEvent =
  | PaymentInitiated
  | FraudScoreCalculated
  | RiskAssessmentCompleted
  | MerchantLimitsChecked
  | PaymentCompleted
  | PaymentDeclined;

Having

type PaymentInitiated = {
  type: "PaymentInitiated";
  data: {
    paymentId: string;
    amount: number;
    currency: string;
    gatewayId: string;
    initiatedAt: Date;
  };
};

type FraudScoreCalculated = {
  type: "FraudScoreCalculated";
  data: {
    paymentId: string;
    score: number;
    riskLevel: "low" | "medium" | "high";
    calculatedAt: Date;
  };
};

type RiskAssessmentCompleted = {
  type: "RiskAssessmentCompleted";
  data: {
    paymentId: string;
    riskScore: number;
    factors: string[];
    assessedAt: Date;
  };
};

type MerchantLimitsChecked = {
  type: "MerchantLimitsChecked";
  data: {
    paymentId: string;
    withinLimits: boolean;
    dailyRemaining: number;
    checkedAt: Date;
  };
};

type PaymentCompleted = {
  type: "PaymentCompleted";
  data: { paymentId: string; completedAt: Date };
};

type PaymentDeclined = {
  type: "PaymentDeclined";
  data: { paymentId: string; reason: string; declinedAt: Date };
};

Let’s say the fraud system flagged the payment as high-risk before it even existed in your system. Approval happened before the risk assessment was completed. The example race condition can look as follows:

10:15:32.123 - FraudScoreCalculated (score: 85, high risk)
10:15:32.145 - PaymentInitiated (amount: $500)
10:15:32.167 - PaymentCompleted (approved by automated system)
10:15:32.201 - RiskAssessmentCompleted (risk: medium)
10:15:32.234 - MerchantLimitsChecked (within limits)

If we assume that things go as on the whiteboard, and handlers can’t process events for non-existent payments, then we’ll be the ones who get punched in the mouth by reality. They will fail to find documents to update.

The first step is to embrace that we have a problem. Actually, it’s not a problem, but rather a challenge - a scenario we just need to support.

We can start by defining a document that represents a payment’s verification state. The document should include optional fields since you can’t predict which event will arrive first. The projection function processes each event and updates this document, creating it if necessary.

It can look as follows:

type PaymentVerification = {
  paymentId: string;
  payment?: Payment;
  fraudAssessment?: FraudAssessment;
  riskEvaluation?: RiskEvaluation;
  merchantLimits?: MerchantLimits;
  decision?: Decision;
  status: "unknown" | "processing" | "approved" | "declined";
  completionPercentage: number;
  lastUpdated: Date;
  dataQuality: "partial" | "sufficient" | "complete";
};

type Payment = {
  amount: number;
  currency: string;
  gatewayId: string;
  initiatedAt: Date;
};

type FraudAssessment = {
  score: number;
  riskLevel: "low" | "medium" | "high";
  assessedAt: Date;
};

type RiskEvaluation = {
  score: number;
  factors: string[];
  assessedAt: Date;
};

type MerchantLimits = {
  withinLimits: boolean;
  dailyRemaining: number;
  checkedAt: Date;
};

type Decision = {
  approval: "approved" | " declined";
  reason: string;
  decidedAt: Date;
};

Besides the optional data that we’ll gradually fill as events arrive, we have some mandatory fields. The obvious one is paymentId, we need to be able to correlate upcoming data from events. If you have a look at them, all of them have such information. Thanks to that, we know which payments we are verifying. If we’re missing it, then we won’t be able to correlate upcoming data. Then we’re indeed doomed.

Of course, sometimes things can get harder. We do not always have a certain id, sometimes some other data, like external id, idempotence key, correlation id, whatever id. Still, we need to have some field, or multiple fields that allow us to point to that for this event we need to update that document.

How do we update our read model? We need a function that takes the current state (or null if it doesn’t exist) and the event we apply on top of it, returning the new state.

It can look as follows:

function evolve (
  current: PaymentVerification,
  event: PaymentVerificationEvent
): PaymentVerification | null {
  current = current ?? {
    paymentId: event.paymentId,
    initialState,
  }

  switch (event.type) {
    case "PaymentInitiated":
      return onPaymentInitiated(current, event);
    case "FraudScoreCalculated":
      return onFraudScoreCalculated(current, event);
    case "RiskAssessmentCompleted":
      return onRiskAssessmentCompleted(current, event);
    case "MerchantLimitsChecked":
      return onMerchantLimitsChecked(current, event);
    case "PaymentCompleted":
      return onPaymentCompleted(current, event);
    case "PaymentDeclined":
      return onPaymentDeclined(current, event);
};

const initialState: PaymentVerification = {
  paymentId: undefined!,
  status: 'unknown',
  completionPercentage: 0,
  lastUpdated: new Date(),
  dataQuality: 'partial'
};

// Using Emmett it could be defined as
export const paymentVerificationProjection = pongoSingleStreamProjection({
  // collection to where we store projection
  collectionName: 'paymentVerification',
  getDocumentId: (event) => event.data.paymentId,
  evolve,
  canHandle: [
    'PaymentInitiated',
    'FraudScoreCalculated',
    'RiskAssessmentCompleted',
    'MerchantLimitsChecked',
    'PaymentApproved',
    'PaymentDeclined',
  ],
  // use this state when there's no document in collection of certain id
  initialState: () => initialState,
});

And yes, even if the data is not in the state we expect, we can always create the Phantom Record that represents the best state we can in the conditions we have.

What if fraud scoring arrives first?

function onFraudScoreCalculated(
  current: PaymentVerification,
  { data: event }: FraudScoreCalculated): PaymentVerification {
  // Ignore event if we're already ahead of it
  if (current.fraudAssessment && event.calculatedAt <= current.fraudAssessment.assessedAt)
    return current;

  const updated = {
    ...current,
    fraudAssessment: {
      score: event.score,
      riskLevel: event.riskLevel,
      assessedAt: event.calculatedAt,
    },
    lastUpdated: event.calculatedAt,
  }

  if (event.riskLevel !== 'high')
    return {
      ...updated,
      status: 'declined',
      decision: {
        approval: 'declined',
        reason: `High fraud risk detected: score ${event.score}`,
        decidedAt: event.calculatedAt,
      },
    };

  return updated;
}

I told you that you should not use a timestamp, but I did it on my own. We cannot always be prim and proper. When doing ACL, sometimes we can achieve just good enough results. If the event doesn’t have any logical timestamp, then we need to live with what we have. Here, we’re assuming that FraudScoreCalculated always comes from the same source, and then we assume that timestamps have the chance to be consistent. With that, we can use them to see if we haven’t already applied this information, or the newer one. If it’s older, then we just don’t make any changes. Sometimes you’ve got to do what you’ve got to do.

Now, lets have a look on the payment initiation when fraud scoring might have already completed:

function onPaymentInitiated(
  current: PaymentVerification,
  { data: event }: PaymentInitiated): PaymentVerification {
  // Ignore if we already handled this event
  if (current.payment)
    return current;

  return {
    ...current,
    payment: {
      amount: event.amount,
      currency: event.currency,
      gatewayId: event.gatewayId,
      initiatedAt: event.initiatedAt,
    },
    lastUpdated: event.initiatedAt,
  };
};

If fraud scoring was completed first, current already has fraud data. The handler merges payment details into the current state.

We can also mark that in the field that I called dataQuality. It may not be the perfect name, but I used it to indicate that we can have a field that lets us decide whether data is as expected, or if we hit reality. Based on that, we can decide how to display it, or whether to display it at all. Thus, the name “phantom record”. It may look like phantom as it’s missing some data but it’s the best data we have.

function onPaymentCompleted(
  current: PaymentVerification,
  { data: event }: PaymentCompleted): PaymentVerification {
  // Ignore if we already made decision
  if (current.decision)
    return current;

  const decision =
    current.fraudAssessment?.riskLevel === 'high'
      ? {
        approval: 'declined',
        reason: `Approval attempted but overridden by fraud (score: ${current.fraudAssessment.score})`,
        decidedAt: event.approvedAt,
      } : {
        approval: 'approved',
        reason: `Approved by ${event.approvedBy}`,
        decidedAt: event.approvedAt,
      };

  return {
    ...current,
    status: decision.approval,
    decision,
    lastUpdated: event.approvedAt,
  };
}

Waiting for Dependencies

Some decisions need multiple pieces. The MerchantLimitsChecked handler waits for both fraud assessment and merchant limits. If we receive merchant limits first and don’t have a fraud assessment yet, we simply store the information; the same applies if it goes the other way. When we eventually get both, we can make the final update, e.g. about the final payment approval or decline.

function onMerchantLimitsChecked(
  current: PaymentVerification,
  { data: event }: MerchantLimitsChecked): PaymentVerification {
  // Ignore if we already handled this event
  if (current.merchantLimits)
    return current;

  const updated = {
    ...current,
    merchantLimits: {
      withinLimits: event.withinLimits,
      dailyRemaining: event.dailyRemaining,
      checkedAt: event.checkedAt,
    },
    lastUpdated: event.checkedAt,
  };

  // Check if we now have BOTH critical pieces
  if (!updated.fraudAssessment || !updated.merchantLimits)
    // Don't have both yet - stay in processing
    return {
      ...updated,
      status: "processing",
      dataQuality: "processing",
    };

  // Both present - can make final decision
  const decision =
    updated.fraudAssessment.riskLevel === "high"
      ? {
          approval: "declined",
          reason: "High fraud risk",
          decidedAt: event.checkedAt,
        }
      : !updated.merchantLimits.withinLimits
      ? {
          approval: "declined",
          reason: "High fraud risk",
          decidedAt: event.checkedAt,
        }
      : {
          approval: "approved",
          reason: "Verified",
          decidedAt: event.checkedAt,
        };

  return {
    ...updated,
    status: decision.approval,
    decision,
  };
}

And now we’re reaching the grey area, where this can become an anti-pattern. I even made this mistake when writing this article. I wrote initially, “we can make the final decision”, but then changed it to “we can make the final update”. Why?

Projections, ACL responsibility and Workflows

I’m sure you’ve used or seen Anti-Corruption Layers. They’re usually places where all the weird logic lands. We want to protect our core logic, pushing all hacks to one, hidden place. This strategy makes some sense; that’s why it’s called Anti-Corruption Layer. Yet…

Yet, we should not push it to the limits, as we’ll end up with an unmaintainable beast. With the last example, we’ve reached or even passed the limits of what projection should be responsible for. The projection should just interpret upcoming information and store the result. It should not make decisions. Business logic is responsible for making decisions.

What we actually ended up with in the last step is a form of process manager, or workflow, so a state machine that listens to events, gets its current state and makes further decisions. And what’s the best way to inform the outside world about making new decisions? Well, producing a new event.

We could define an event as:

type PaymentVerificationCompleted = {
  type: "PaymentVerificationCompleted";
  data: {
    approval: "approved" | " declined";
    reason: string;
    decidedAt: Date;
  };
}

As we’re making decisions, let’s be frank and rename evolve function to decide and produce a new event when we distilled it from the messy outside world?

It could look like that:

function decide(
  current: PaymentVerification | null,
  event: PaymentVerificationEvent
):
  | PaymentVerification
  | { document: PaymentVerification; events: VerificationEvent[] } {
  current = current ?? {
    paymentId: event.paymentId,
    initialState,
  }

  switch (event.type) {
    // (...) other event handlers
    case "MerchantLimitsChecked": {
      const updated = {
        ...current,
        merchantLimits: {
          withinLimits: event.withinLimits,
          dailyRemaining: event.dailyRemaining,
          checkedAt: event.checkedAt,
        },
        lastUpdated: event.checkedAt,
      };

      return tryCompleteVerification(updated, event);
    }
    case "FraudScoreCalculated": {
      if (
        current.fraudAssessment &&
        event.calculatedAt <= current.fraudAssessment.assessedAt
      )
        return current;

      const updated = {
        ...current,
        fraudAssessment: {
          score: event.score,
          riskLevel: event.riskLevel,
          assessedAt: event.calculatedAt,
        },
        lastUpdated: event.calculatedAt,
      };

      return tryCompleteVerification(updated, event);
    }
  }
};

We update the state and try to complete verification if we gathered both merchant limits and risk assessment. This could be encapsulated in a dedicated method, as the logic is the same now matter which event came first:

function tryCompleteVerifications(
  current: PaymentVerification,
  event: PaymentVerificationEvent
):
  | PaymentVerification
  | { document: PaymentVerification; events: VerificationEvent[] } {
  // Ignore if we already made decision
  if (current.decision)
    return current;

  // Check if we now have BOTH critical pieces
  if (!current.fraudAssessment || !current.merchantLimits)
    // Don't have both yet - stay in processing
    return {
      ...current,
      status: "processing",
      dataQuality: "processing",
    };

  const decision =
    current.fraudAssessment.riskLevel === "high"
      ? {
          approval: "declined",
          reason: "High fraud risk",
          decidedAt: event.checkedAt,
        }
      : !current.merchantLimits.withinLimits
      ? {
          approval: "declined",
          reason: "High fraud risk",
          decidedAt: event.checkedAt,
        }
      : {
          approval: "approved",
          reason: "Verified",
          decidedAt: event.checkedAt,
        };

  return {
    document: {
      ...current,
      status: decision.approval,
      decision,
    },
    events: [
      {
        type: "PaymentVerificationCompleted",
        data: decision,
      },
    ],
  };
};

We’re returning not only the new state, but also the new event. It can be published to the local module queue or just stored inside the event store.

Essentially, we’re making a chaotic outside world linear based on the order of our observations. We can’t change the outside world, but we can at least know why and where we’ve made our decisions.

This is actually the same pattern I showed during the webinar on modelling and implementing distributed processes:

We aggregate data until we have enough of it to make the next decision. Is it perfect? Nah, but sometimes best you can is good enough.

When This Approach Works (And When It Doesn’t)

This approach works when:

Your business logic can handle partial data: Payment approval can often proceed with fraud score and limits check, even if risk assessment is pending. E-commerce fulfilment can start when payment is confirmed, even if the recommendation engine results are still processing.

Eventual consistency is acceptable: The payment dashboard might show “processing” for a few hundred milliseconds while verification completes. Most business users can tolerate brief inconsistency in exchange for system responsiveness.

Decisions can be corrected or refined: A payment approved with partial data can be flagged for additional review when complete risk data arrives. An order can be expedited or delayed based on a complete customer analysis.

You have clear business rules for conflicts: When fraud assessment contradicts approval, fraud wins. When risk assessment arrives late, it updates data quality but doesn’t reverse committed decisions.

This approach struggles when:

Perfect consistency is required: Financial accounting, legal compliance, and safety-critical systems often can’t tolerate any inconsistency, even briefly.

Business logic requires complete data: Some decisions genuinely can’t be made with partial information. Credit limit increases might require a complete financial analysis before any approval.

Rollback costs are high: If reversing a decision is expensive or impossible, you need stronger ordering guarantees before committing.

Users can’t tolerate uncertainty: Some interfaces need to show definitive status immediately, not “processing” or “partial data available.”

Conclusion: Embracing the Chaos

The world is chaotic, and we can’t stop the chaos, but we can stop fighting the chaos. External events are rumours about what happened in other systems. Your read model can store the information you derive from those rumours. The evolve function processes whatever arrives, in whatever order, building state incrementally, making your local interpretation.

Undeniably, it’s a workaround of some sort. But it’s also an acknowledgement that distributed systems don’t guarantee order across boundaries. When you can’t control the topology, you adapt on your side. Store data as it arrives. Denoise in your projections. Create clean internal events for downstream systems.

Sometimes the proper solution is fixing your message topology. Route related events to the same partition. Use predictable identifiers for correlation. But when external systems constrain your options, when organisational boundaries limit your control, when legacy integrations force your hand - you’ve got to do what you’ve got to do.

Build your local models and live with partial state. Process events in any order. Make decisions with available data. That’s how you build reliable systems on unreliable foundations.

If you’re dealing with such issues, I’m happy to help you through consulting or mentoring. Contact me and we’ll find a way to unblock you!

Read also more in:

Cheers!

Oskar

On Messaging and Distributed Systems with Ian Cooper

Welcome to the new week!

New week, new video. Last week, you got my talk about event-driven modelling anti-patterns. A significant portion of it was to explain the difference between logical design and technical one.

Event-Driven Architecture helps us to design interactions in our system. We prefer async communication, informing others of what has happened by publishing events. Yet, if we model our interactions only with events, then we’d either end up with a passive-aggressive environment or a room filled with shouting people. That’s not great.

Messaging patterns help us to design our message flow optimally.

One of the best people in the messaging space is Ian Cooper. I have a lot of respect for his work and knowledge. He always does his homework.

Last week, I had the pleasure of recording an interview with him, thanks to an invitation from Avanscoperta.

We discussed Ian’s perspective on:

Why do we still need to learn messaging? Why isn’t it a commodity yet?
how we both started with messaging,
how Ian is learning and teaching others about distributed systems and messaging,
how to shape boundaries, and how data on the insight, and data on the outside can help in that,
testing strategies,
and more.

The trigger for discussion was Ian’s upcoming workshop, but I’m sure that if you’re looking for a lighthearted exchange of ideas on those topics, I think you’ll enjoy it.

You can also check it on:

Spotify,
Apple Podcasts.

Please share your thoughts with me on how you liked it!

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

New Recording on Event modelling anti-patterns from DDDEU

Welcome to the new week!

I’m always saying that with Event-Driven Architecture, the modelling effort pays back.

That goes both ways: the less effort we put in modelling, the more it’ll hurt later.

There’s a wide range of issues you may be facing, from overfocusing on the state instead of tracking behaviour, to asking others more often than allowing them to tell you what happened, and ending with race conditions and other unpleasant scenarios.

I packed as many of such cases into our talk, and the recording from this year’s DDD Europe just arrived:

During the session, I explained the specifics of event modelling (yes, no capital letter, and double l), starting with bad practices and knowing why and how to avoid them.

I told the story about the project that aimed to modernise legacy software into the event-driven world. In theory, artificial, but in practice, none of the examples were made up. Either I made those mistakes on my own, or I saw them in my projects or helped to fix them for my clients.

I tried to make it both entertaining and educational, bitter and sweet. It is not easy when you’re not a native speaker. There’s a thin line between being funny and being silly.

There’s also a thin line between bad and good practices. And its name is: context.

Also, as much some of those cases may seem wild, then I can assure you that all of those mistakes I either:

did by myself,
saw in my projects,
saw in my client’s project.

Check it out, why learn always from your mistakes? Learn from mine.

The talk also summarised my article series about anti-patterns in event modelling. Here’s the full list:

Check also more general considerations:

And hey, I’m also doing consulting and mentoring. If you’re struggling with your projects/organisation with such cases, I’m happy to help. Feel free to reach out to me through the email.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Sneaky Code Bites Back

Welcome to the new week!

Sneaky code bites back, I realised that again recently.

I realised I’d been writing terrible code when I couldn’t explain it to myself.

The code worked. It was adding SQLite support to Pongo. But when I tried to describe how I was implementing multi-database support, I heard myself saying:

“First it checks if the Promise is cached, then it creates a proxy that defers to the real implementation which doesn’t exist yet, but will be loaded dynamically when…”

Yes, I sounded insane.

What I Was Building

Pongo is my attempt to bring the familiar MongoDB API and use relational databases as document databases.

If you’ve used MongoDB, you know the API collection.find(), collection.insertOne(), collection.updateOne(). Pongo gives you that same API, but your data lives in PostgreSQL using JSONB columns.

Why would anyone want this? Teams often know MongoDB’s API but need PostgreSQL’s guarantees: real ACID transactions, mature replication, and existing infrastructure. Or they have PostgreSQL but want document storage without learning PostgreSQL’s JSON query syntax, which looks like this:

SELECT data FROM users

WHERE jsonb_path_exists(data, '$.address.history[*] ? (@.street == "Elm St")')

users.find({ "address.history": { $elemMatch: { street: "Elm St" } } })

Same result, familiar syntax. I’d built something similar before with Marten for .NET, so I knew the approach worked.

It seemed that people appreciated it. And when someone appreciates something, they usually want more…

Users started requesting support for other databases, e.g. SQLite for edge deployments, DuckDB for analytics, Spanner for distributed systems, etc. Made sense. The MongoDB API could work on any database that supports JSON.

The Challenge

Adding multi-database support meant dealing with drivers. Database drivers are how your application talks to databases. So code code handling connection protocols, type conversions, and query preparation. Each database needs its own driver.

The straightforward approach: make developers explicitly import the driver they need:

import { pongoClient } from '@event-driven-io/pongo';

import { databaseDriver } from '@event-driven-io/pongo/pg';

const client = pongoClient({
 driver: databaseDriver,

 connectionString: 'postgresql://localhost:5432/mydb'
});

This bothered me. The connection string already says ”postgresql“.

Why make developers say it twice?

I wanted automatic driver selection. Parse the connection string, load the right driver, only when needed.

The Sneaky Solution

I built a deferred loading pattern in Dumbo, my database abstraction layer. Dumbo sits under Pongo, handling connection pools, transactions, and SQL generation. Both Pongo and my event sourcing library, Emmett, use it.

Here’s the pattern:

export const createDeferredConnectionPool = <Connector, ConnectionType>(
  connector: Connector,
  importPool: () => Promise<ConnectionPool<Connection<Connector>>>,
): ConnectionPool<ConnectionType> => {
  let poolPromise: Promise<ConnectionPool> | null = null;

  const getPool = async () => {
    if (poolPromise) return poolPromise;
    return (poolPromise = importPool());
  };

  return createConnectionPool({
    connector,
    execute: createDeferredExecutor(connector, async () => {
      const connection = await getPool();
      return connection.execute;
    }),
    close: async () => {
      if (!poolPromise) return;
      const pool = await poolPromise;
      await pool.close();
    },
    // ... every method proxied through getPool()
  });
};

Let me explain what’s happening here, because it’s the core of why my design went wrong.

The connection pool appears to exist immediately, but it’s actually a proxy. Every method (execute, close, transaction, etc.) is wrapped to the first call getPool(). The first time getPool() runs, it imports the real driver and creates the real connection pool. Then—and this is crucial—it caches the Promise itself, not the result.

In JavaScript, a Promise is an object. Once created, it has an identity. You can await the same Promise multiple times, and the function will be evaluated once, the result will be cached and reused in subsequent awaits:

const work = doSomethingAsync();
const a = await work;
const b = await work; // Same Promise object, same eventual result

This is different from calling the async function twice:

const a = await doSomethingAsync(); // Creates Promise #1
const b = await doSomethingAsync(); // Creates Promise #2, does work again

By caching the Promise, multiple parts of code can request the connection pool simultaneously, but the import only happens once. Everyone waits for the same Promise to resolve. This pattern is called Promise memoisation.

In C# terms, it’s like storing a Task<T> and having multiple threads await it. In Python, it’s like functools.lru_cache for async functions. The concept exists across languages—cache the operation, not just the result.

This worked in Dumbo. The API stayed clean. Drivers are loaded on demand. You pay the differing cost only once.

Then I tried extending this pattern to Pongo itself.

The Cascade Effect

To support automatic driver selection in Pongo, I’d need to defer everything.

Currently, when you create a Pongo client, it knows its driver:

const client = pongoClient({ connectionString });
const db = client.db('myapp');
const users = db.collection('users');

Each level—client, database, collection—is a concrete object with real methods.

With automatic driver selection, none of these could be real until first use. The client wouldn’t know which driver to use until it parsed the connection string. However, parsing occurs asynchronously, as dynamic module loading in JavaScript can only happen with async imports. So the client also needs to become a proxy just like the Dumbo connection. The database becomes a proxy. The collection becomes a proxy.

Here’s what a deferred collection would look like:

class DeferredPongoCollection<T> implements PongoCollection<T> {
  private realCollectionPromise: Promise<PongoCollection<T>> | null = null;

  constructor(
    private name: string,
    private getDb: () => Promise<PongoDb>
  ) {}

  private async getRealCollection(): Promise<PongoCollection<T>> {
    if (!this.realCollectionPromise) {
      this.realCollectionPromise = this.getDb()
        .then(db => db.collection<T>(this.name));
    }
    return this.realCollectionPromise;
  }

  async find(filter?: Filter<T>, options?: FindOptions): Promise<T[]> {
    const collection = await this.getRealCollection();
    return collection.find(filter, options);
  }

  async findOne(filter: Filter<T>, options?: FindOptions): Promise<T | null> {
    const collection = await this.getRealCollection();
    return collection.findOne(filter, options);
  }

  async insertOne(doc: T): Promise<InsertOneResult> {
    const collection = await this.getRealCollection();
    return collection.insertOne(doc);
  }

  async insertMany(docs: T[]): Promise<InsertManyResult> {
    const collection = await this.getRealCollection();
    return collection.insertMany(docs);
  }

  async updateOne(filter: Filter<T>, update: Update<T>): Promise<UpdateResult> {
    const collection = await this.getRealCollection();
    return collection.updateOne(filter, update);
  }

  async updateMany(filter: Filter<T>, update: Update<T>): Promise<UpdateResult> {
    const collection = await this.getRealCollection();
    return collection.updateMany(filter, update);
  }

  // ... and 20+ more methods
}

Every single method becomes a proxy that waits for the real collection. But the collection needs a database, which needs a client, which needs a driver. Three levels of deferred proxies.

Now add TypeScript generics that need to flow through all these layers. Add error handling—what if the import fails? Add specialised MongoDB types like AggregationCursor, ChangeStream, BulkWrite. Each needs its own deferred proxy class.

The driver would register itself through a side effect:

// In @event-driven-io/pongo/pg
import { registerDriver } from '@event-driven-io/pongo';
registerDriver('postgresql', () => import('./postgresqlDriver'));

Then you could add just the import anywhere in the app:

import * from '@event-driven-io/pongo/pg';

And magic registration will happen, and as a user, you wouldn’t need to know about all the stuff I just wrote, right? RIGHT?!

Right, if you remember about this import, if you forgot, then the registration doesn’t happen. The error only surfaces at runtime when you try to connect.

The Failure

When I was into this design, I reminded myself on a similar thing I wanted to do in past.

Years ago, I nearly shipped distributed transactions using MSDTC (Microsoft’s distributed transaction coordinator). The code worked perfectly in testing. But when I imagined production failures, I realised I couldn’t debug them. The complexity was hidden, not gone.

This was the same mistake.

Consider what happens when something goes wrong. A user reports:

“I’m getting an error when inserting a document.”

Where’s the actual problem? Let’s trace through the layers:

User calls collection.insertOne(doc)
Deferred collection waits for real collection via getRealCollection()
Which waits for a deferred database via getDb()
Which waits for a deferred client to resolve
Which waits for the driver to be imported
Which needs the connection string to be parsed
Which needs the driver to be registered
Which depends on a side-effect import that may not have happened

The stack trace shows insertOne at the top. The actual error—maybe a typo in the connection string, maybe a missing driver import, maybe a network issue—is buried under seven layers of Promise resolution.

But there’s a worse problem: Promise memoisation means errors get cached forever.

If the first connection attempt fails, poolPromise holds a rejected Promise. Every subsequent request awaits the same rejected Promise. You can’t retry without restarting the process. This is a fundamental property of Promises—once settled (resolved or rejected), they never change state.

In production, this means that one failed connection attempt during startup breaks everything until a restart is performed. A temporary network glitch becomes a permanent failure.

The Nightmare

Let’s say you’re debugging this in production. You add logging:

async insertOne(doc: T): Promise<InsertOneResult> {
  console.log('insertOne called');
  const collection = await this.getRealCollection();
  console.log('Got real collection');
  return collection.insertOne(doc);
}

The logs show “insertOne called” but not “Got real collection”. Where’s the problem? Could be:

Driver not registered (missing import)
Connection string malformed
Network unreachable
Import failed (syntax error in driver)
Previous connection attempt failed (cached rejection)

You need to understand the entire deferred loading chain to debug a simple insert.

Now imagine explaining this to a contributor who wants to add a DuckDB driver.

“First, create your driver. Then register it via side effect. Understand Promise memoization because failures get cached. Know that everything is proxied through three levels of deferreds. When debugging, remember the stack trace lies about where errors originate…”

The Alternative

I chose explicit driver injection instead:

import { pongoClient } from '@event-driven-io/pongo';
import { databaseDriver } from '@event-driven-io/pongo/pg';

const client = pongoClient({
  driver: databaseDriver,
  connectionString: 'postgresql://localhost:5432/mydb'
});

One extra import. That’s the entire cost.

The benefits:

TypeScript catches missing drivers at compile time: “Property ‘driver’ is missing”
Stack traces point to real problems
Failed connections can retry
New developers understand it immediately
Each driver can expose database-specific features

The code is boring. No magic. No proxies. No deferred loading. It works or it doesn’t. When it fails, you know why and where.

The Pattern

This isn’t really about Pongo or database drivers. It’s about a pattern we all fall into.

We see repetition and think “I can abstract this.”

We see explicit configuration and think: “I can infer this”. We see upfront costs and think: “I can defer this”.

Sometimes we just don’t trust our colleagues, thinking that they’re too incompetent and we need to take precautionary steps.

Sometimes we’re right. Most often we’re not.

Too often, we’re optimising the wrong metric. I was optimizing for fewer imports, not simpler code. I was hiding essential configuration, not accidental complexity.

A minimal API with complex implementation isn’t simple—it’s a lie. The complexity doesn’t disappear. It moves from the visible surface to hidden internals, where it’s harder to understand, debug, and modify.

I’ve seen this pattern everywhere:

ORMs with query optimization: They generate “optimal” SQL you can’t debug. When queries are slow, you’re reverse-engineering what the ORM decided to do. The abstraction that was supposed to help becomes a barrier to fixing problems.

Dependency injection with auto-wiring: The container magically figures out what to inject where. Works great until it doesn’t. Then you’re debugging why the container chose the wrong implementation, and the magic configuration system has no good debugging tools.

Build tools with convention over configuration: They discover your project structure automatically. Perfect until your project doesn’t fit the convention. Then you’re fighting the tool’s assumptions, wishing for explicit configuration.

Test frameworks with magical setup: They auto-mock dependencies, auto-setup databases, auto-configure everything. The tests become harder to understand than the code they’re testing. When tests fail, you debug the framework, not your code.

The pattern is always the same. Good intentions. Hidden complexity. Eventual regret.

When Magic Becomes a Prison

Magic in code should be like magic in a good movie—the audience shouldn’t see how it works. But there’s a crucial difference: movie audiences never need to modify the magic trick.

Code has maintainers. They need to understand the magic, extend it, debug it. When the magic is at the surface—the API level—every maintainer becomes a magician’s apprentice.

The deferred loading pattern would have created a codebase where:

Reading code means understanding proxies and Promise memoization
Adding features means maintaining the deferred chain
Debugging means tracing through invisible layers
Onboarding means learning the entire magical system

It creates a priesthood of maintainers who understand the magic. Everyone else fears the code. They make small changes and hope nothing breaks. They don’t refactor because they might disturb the magic.

Let’s be honest about what I was trading:

The cost of explicit drivers:

One extra import line
Developers must know which database they’re using

The cost of automatic driver selection:

Three levels of proxy objects
Promise memoization complexity
Cached failures requiring restarts
Incomprehensible stack traces
Side-effect registration
Impossible debugging
Steep learning curve for contributors

Was saving one import worth all that complexity? Obviously not. But I didn’t see it until I started implementing it.

This is why I need to write code to understand ideas. My brain can’t model all the implications abstractly. I need to “sweat out” designs in real code. Sometimes that means building something just to realize it’s wrong.

The key is recognizing it before shipping.

The Lesson

Every time I’ve tried to be sneaky, it’s bitten me. The distributed transactions that worked until they didn’t. The code generators that saved typing but destroyed readability. The clever abstractions that became maintenance nightmares.

The problems sneaky code solves are almost always smaller than the problems it creates. But we don’t see that at design time. We see the elegance of the solution, not the pain of living with it.

I wanted to make Pongo simple. I was making its setup simplistic and obscure. Simple means easy to understand and maintain. Simplistic means oversimplified to the point of being inadequate for real-world use.

Now I’m removing the clever patterns. Each driver will be explicit. Each connection direct. Each error immediate and clear.

The new code is boring. Import a driver. Pass it to the client. It connects or fails with a clear error.

Boring code is debuggable. Explicit code is maintainable. Simple code—truly simple, not simplistic—is code that works and keeps working.

I got too clever. We all do sometimes. The key is catching it before it escapes into production, where it becomes someone else’s nightmare.

If you want to see “The Removal”, check this Pull Request and related follow-ups. I’m replacing the clever code with boring code. It’s less exciting than it sounds, and that’s the point.

And hey, there’s also another lesson: we all make mistakes, and fall into the rabbit hole. At least I do.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

PostgreSQL partitioning, logical replication and other Q&A about PostgreSQL Superpowers

Welcome to the new week!

Last Tuesday, I did a new version of the “PostgreSQL Superpowers” webinar, thanks to the invitation from the good folks at Particular. It’s available here:

Or check:

I’m happy with how it went, as looking at the number of questions, I managed to pique people’s curiosity about this topic!

Curious enough, that I didn’t manage to cover all the questions during the webinar, which is why I decided to answer them today as part of my humble newsletter.

I encourage you to watch the recording, but to recap what I showed:

PostgreSQL is not only a rock-solid Open Source database, but also how rich the plugin ecosystem is, and how that impacts its success
native PostgreSQL partitioning for efficient data storage management,
How to compose plugins like Timescale for time series data, PostGIS for spatial data (GPS tracking, etc.).
In general, how can plugins cut the time needed to deliver essential features
Automate reporting with continuous aggregations without having to poll the database
Stream database changes in real time by using logical replication as a push-notification system
Detect issues as they happen, for example, spotting unusual fleet activity, without constant polling

All of that is based on the simple, but real-world app of tracking business trips and managing reporting and alerting around them.

Ok, let’s move into the questions that I didn’t manage to cover during the webinar.

Questions & Answers

Partitioning

Michał: How many partitions is too much partitions? talking about weekly partitions for ~10 years?

If we follow the example from our webinar, the trips table looks like this:

CREATE TABLE trips (
    trip_time TIMESTAMPTZ NOT NULL,
    vehicle_id INT NOT NULL,
    driver_name VARCHAR(255) NOT NULL,
    start_location TEXT NOT NULL,
    end_location TEXT NOT NULL,
    distance_kilometers NUMERIC(10,2) NOT NULL,
    fuel_used_liters NUMERIC(10,2) NOT NULL,
    PRIMARY KEY (trip_time, vehicle_id)
) PARTITION BY RANGE (trip_time);

-- setup partitions for year

DO $$DECLARE
    month_start_date DATE := '2023-01-01';
    month_end_date DATE := '2023-12-01';
BEGIN
    WHILE month_start_date < month_end_date LOOP
        EXECUTE format('
            CREATE TABLE trips_%s PARTITION OF trips
            FOR VALUES FROM (%L) TO (%L);',
            TO_CHAR(month_start_date, 'YYYY_MM'),
            month_start_date,
            month_start_date + INTERVAL '1 month'
        );
        month_start_date := month_start_date + INTERVAL '1 month';
    END LOOP;
END$$;

Then, if we partition it by time, we’d get in 10 years:

month - 120 partitions (12 * 10),
week - 520 partitions (52 * 10),
day - 3650 partitions(365 * 10).

Obviously, 520 and 3650 sound like potentially a lot and should raise our eyebrows. PostgreSQL can handle up to a few thousand partitions fairly well, provided that typical queries allow the query planner to prune all but a small number of partitions. But here’s the catch - each partition requires its metadata to be loaded into the local memory of each session that touches it. See the nice research on how the number of partitions can impact performance: Partitioning in Postgres and the risk of high partition counts.

Nevertheless, we may notice that for our case, partitions fall into three groups:

current partition - where new trips are added,
active period partitions - e.g., the last 3 months, that are interesting for us from the alerting and reporting needs or maybe filling back missing trips,
old partitions - those that we don’t touch at all, or sporadically (e.g. older than 3 months).

This split is also essential from the perspective of another question:

Serge: Is there a notion of hot partition?

Yes, depending on our scenarios, some partitions may be the most used. That’s also why partition planning is critical, and it should be also made based on the business scenario. For our case, we should definitely care and keep focus on the current partition (the hot one, as it’s the most often used), and the active period.

What about the rest? They can be moved either to cheaper, slower storage. PostgreSQL allows this through setting up a so-called Tablespace.

We can do it by calling a SQL statement:

-- Create cheap storage
CREATE TABLESPACE cold_storage LOCATION '/mnt/slow_disk';

-- detach partion
ALTER TABLE trips DETACH PARTITION trips_2025_01;

-- set cheaper disk for this table
ALTER TABLE trips_2025_01 SET TABLESPACE cold_storage;

We could also use the pg_partman extension to automate it:

-- Tell partman to keep only 2 years attached
UPDATE partman.part_config
SET retention = '3 months',
    retention_keep_table = true  -- detach but don't drop
WHERE parent_table = 'public.trips';

-- Run maintenance (or let background worker do it)
SELECT partman.run_maintenance();

That also answers the next question:

Karol: How can we “move” older partitions to some less available/slower storage? Is available in postgres itself or some postgres cloud providers?

So it’s a native mechanism, but supported in the Cloud RDSes (also pg_partman is usually available there).

Worth noting is that detaching means removing a partition from the parent table while keeping it as a standalone table. The detached partition continues to exist as a standalone table, but no longer has any ties to the table from which it was detached. Read more in the partitioning docs.

We can attach it again, if we want, or just delete the content or even the whole partition table if we don’t need to access it.

That’s also important for the next question:

Matteo: what about a time range query? like time >= x AND time < y?. will Postgres touch only the relevant partitions?

Yes, PostgreSQL uses partition pruning to automatically skip irrelevant partitions when you query with conditions on the partition key.

Here’s how it works with our trips table when using the Explain tool:

EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM trips
WHERE trip_time >= '2025-06-15'
  AND trip_time < '2025-07-15';

We’ll get something close to:

Append  (cost=0.00..241.00 rows=4800 width=64) (actual time=0.015..0.467 rows=4752 loops=1)
  Subplans Removed: 10    -- ← 10 partitions skipped
  ->  Seq Scan on trips_2025_06  (cost=0.00..48.20 rows=1200 width=64) (actual time=0.014..0.128 rows=1188 loops=1)
        Filter: ((trip_time >= '2025-06-15'::timestamp) AND (trip_time < '2025-07-15'::timestamp))
  ->  Seq Scan on trips_2025_07  (cost=0.00..48.20 rows=1200 width=64) (actual time=0.012..0.115 rows=1176 loops=1)
        Filter: ((trip_time >= '2025-06-15'::timestamp) AND (trip_time < '2025-07-15'::timestamp))

PostgreSQL will only query (scan) partition tables trips_2025_06 and trips_2025_07 because our date range spans mid-June to mid-July. The other 10 monthly partitions were pruned away. Pruned means skipped.

Two types of pruning occur:

Planning time pruning - when query values are constants (like above)
Execution time pruning - with prepared statements or subqueries where values aren’t known until runtime.

This is why partitioning by time works well for time-series data - PostgreSQL only touches partitions that overlap with your date range. Still, the recommendation is to try avoid doing queries between partitions. We should make our partitions as autonomous as possible, as querying between partitions is costly (may even mean accessing different disks, etc.).

That’s also how we’re coming to the next two questions:

Barry: What happens when you run this on a table with data already in it? Will it automatically split the data into the relevant partition?

And:

Nikita: Please share your experience with rebalancing of partitions.

PostgreSQL doesn’t have automatic partition rebalancing like distributed databases. Once data lands in a partition, it stays there unless you manually move it. This becomes a problem when your partitioning strategy no longer fits your data patterns.

The most common issue? You chose the wrong partition size. Maybe you started with yearly partitions, but now queries are slow because each partition has too much data. You need to split them, e.g.:

-- Can't just alter boundaries - must detach and recreate
ALTER TABLE trips DETACH PARTITION trips_2025;

-- Create monthly partitions instead
CREATE TABLE trips_2025_01 PARTITION OF trips
FOR VALUES FROM ('2025-01-01') TO ('2025-02-01');
-- ... create all 12 months

-- Move data from the old yearly partition
INSERT INTO trips SELECT * FROM trips_2025;

-- drop old years partition
DROP TABLE trips_2025;

Another case is when you have data in the wrong partitions. Updating your partition key value won’t move your data to a different partition, so instead of updating the partition key value, you need to insert it into the new partition with the new key and drop it from the old one. You can do it as follows:

WITH moved AS (
  DELETE FROM trips_2024_12  
  -- condition to find record(s) you want to move
  WHERE trip_time = '2024-12-31' and vehicle_id = 123
  RETURNING
    vehicle_id,
    driver_name,
    start_location,
    end_location,
    distance_kilometers,
    fuel_used_liters
)
INSERT INTO trips (trip_time, vehicle_id, driver_name, start_location,
                   end_location, distance_kilometers, fuel_used_liters)
SELECT '2025-01-01', 123, driver_name, start_location,
       end_location, distance_kilometers, fuel_used_liters
FROM moved;

In the same way, if you have a table that’s not yet partitioned but has existing data, then PostgreSQL will refuse to partition it. You have two options. Either manually create new partitioned tables and migrate data first, or use pg_partman, which can partition existing tables:

SELECT partman.create_parent('public.trips', 'trip_time', 'native', 'monthly');
CALL partman.partition_data_proc('public.trips');

So, again, PostgreSQL extensions ecosystem FTW!

And final question on partitioning:

Dillan: Does ef core support partitions

Not at the moment, you’ll need to set up partitions on your own through custom migration, and AFAIK, the same story for other popular ORMs. Still, that’s not a big issue, as when you do the setup, you can query a partitioned table as any other table.

Logical Replication

I wrote in more detail about PostgreSQL Logical replication and Write-Ahead Log in:

Check also a great article from Gunnar Morling, an authority in this space:

The Wonders of Postgres Logical Decoding Messages

They should answer some of the questions. Let’s do a super quick TLDR of the key concepts.

A Write-Ahead Log is a structure on the disk to which all statements representing changes (INSERT, UPDATE, DELETE) are appended before applying them to tables upon transaction commit. Most of the databases are cleaning those entries, soon after they’re committed, PostgreSQL allows us to keep them as long as we tell it to.

This enables Logical replication, which streams database changes from a publisher to subscribers using the Write-Ahead Log. Unlike physical replication that copies everything byte-by-byte, logical replication sends actual SQL-like changes (thus logical, as they represent the logical change, not serialised bytes of data).

Key components:

Publication: Defines which tables to replicate. You set it up by calling:
```
CREATE PUBLICATION outbox_pub FOR TABLE outbo
```
Replication Slot: Tracks subscriber progress in WAL, ensures no data loss (pg_create_logical_replication_slot)
Subscriber: Connects to slot, receives changes, sends acknowledgements

So, answering Matteo’s question:

Matteo: How does logical replication relate to Change Data Capture?

Logical replication IS PostgreSQL’s native CDC. It captures changes from WAL and streams them out, the same as Debezium or other CDC tools.

The difference is scope. Debezium uses logical replication internally, providing a higher abstraction level and features such as connectors to Kafka. It handles batching, retries, and monitoring. With native logical replication, you build that plumbing yourself.

Muhammad: I assume WAL notifications are ordered?

Yes, strictly ordered by transaction commit order. WAL is append-only. PostgreSQL tracks subscriber progress through replication slots using LSN (Log Sequence Number - read more here).

Ordering is per replication slot.

Muhammad: What happens when there are multiple subscribers to WAL notifications?

Multiple subscribers of the same slot process at different speeds, so “ordered” messages might arrive out of order at different downstream systems. So it’s similar to RabbitMQ guarantees for multiple consumers of the same queue.

If you need global ordering across systems, you need to either have a single subscriber for the replication slot or you need additional coordination.

Muhammad: Do we get publisher confirms with this Postgres Publication

Not in the traditional message queue sense. The subscriber sends acknowledgements after processing batches of changes, not individual messages - that’s why we call SendStatusUpdate() in the code example.

PostgreSQL gives you position-based tracking. If your subscriber crashes after processing message 100 but before sending the status update, it’ll reprocess from maybe message 80. Your consumer must be idempotent.

Muhammad: Does WAL get auto flushed if there is no subscriber confirms?

No. WAL segments stick around until the subscriber confirms. Replication slots will prevent the removal of required resources even when there is no connection. But that also brings cost:

Nikita: Does WAL grow indefinitely when a subscriber goes down? Can we limit the growth?

Yes, indefinitely by default. To understand why, you need to know how WAL cleanup works.

PostgreSQL continuously writes changes to WAL files in the pg_wal directory. Each file is 16MB by default. Without replication slots, PostgreSQL deletes old WAL files after:

They’ve been checkpointed (data written to main tables)
They’ve been archived (if archiving is enabled)

Let’s compare that to how messaging systems deal with retention.

In Kafka, you set retention policies like “keep data for 7 days” or “keep 100GB”. Data gets deleted after this period, regardless of whether consumers read it. Late consumers simply miss old data.

PostgreSQL works differently. WAL retention is controlled by replication slots, not publications. A publication defines which tables to replicate - it has no retention policy. The slot tracks the consumer’s position and prevents WAL deletion until consumed.

With slots, PostgreSQL also checks every slot position before deletion. One slow slot blocks WAL cleanup for everyone.

If you want Kafka-like behaviour where data is available for future subscribers, you need WAL archiving:

-- Archive WAL to long-term storage
archive_mode = on
archive_command = 'cp %p /mnt/wal_archive/%f'

This copies WAL files before deletion. Future subscribers can replay from the archive, not the active WAL. But logical replication doesn’t automatically use archives - you’d need to restore them manually or use tools like pg_rewind.

You can also set the maximum size of data to keep for the slot:

ALTER SYSTEM SET max_slot_wal_keep_size = '10GB';

This works like a circuit breaker - slots that retain too much WAL get invalidated rather than filling your disk. But unlike Kafka, where late consumers just miss data, PostgreSQL slots become permanently broken.

The fundamental difference: Kafka decouples retention from consumption. PostgreSQL couples them tightly - data stays until consumed or the slot breaks. You can’t have “keep WAL for 7 days for future subscribers” without archiving.

So again, just like with partitioning, archiving strategy is the key!

Trivia

What are core key features not present in SQL Server?

Lots to mention! Jokes aside, MSSQL is a decent database, but from my perspective, Microsoft is lagging behind and underfunding efforts on it. Even on their conferences, they announce more news around PostgreSQL than MSSQL. The biggest difference is in the ecosystem and extensions being first-class citizens.

Still, here’s a quick list from the top of my head:

Extension system - The ability to add functionality without touching core code. That’s how we get pg_partman, PostGIS, TimescaleDB. SQL Server has nothing comparable.
Logical replication with custom output plugins - SQL Server has CDC, but for traditional replication, you can’t customise or plug into it as described for PostgreSQL.
Partitioning in MSSQL is just for the enterprise version, and IMHO, more complicated than in PostgreSQL.
Foreign Data Wrappers - Query external databases, CSV files, APIs as if they were local tables. SQL Server has linked servers but they’re not as flexible.
Custom types and operators - Define your own data types and how they behave. SQL Server is limited to built-in types.
JSONB with GIN indexing - While SQL Server has JSON support, PostgreSQL’s JSONB with indexing is more mature and performant.
Tablespaces - Place specific tables/partitions on different disks. SQL Server uses filegroups, but PostgreSQL’s approach is simpler for partitioning.
Arrays as first-class citizens - you can store and index arrays directly.
Rich ecosystem of various hosting services, besides cloud RDSes there’s a variety of bare metal VPCs, providers like Supabase, Neon. And cloud-native databases like AWS AuroraDB, Google Spanner, CockroachDB, Yugabyte, compatible with PostgreSQL syntax.
The PostgreSQL Docker image is lighter and much easier to run integration tests.

Peter: What misuse of PostgreSQL (feature or in general) has turned out to be useful?

I’d say that:

Using PostgreSQL as a document database - I wrote about it in PostgreSQL JSONB - Powerful Storage for Semi-Structured Data. What seemed to be a wild idea worked really well. For year, I was co-maintaining .NET tool Marten, which allows using PostgreSQL as a document db, now I’m building also Pongo - a Mongo-compliant alternative in Node.js. Also tools like FerretDB and even cloud providers used it, e.g. AWS in DocumentDB, Microsoft in CosmosDB.
Using PostgreSQL as a queue - that’s not something I’d recommend as the safe default, but if throughput of a few thousand messages per second is fine for you, then why not? That’s what Particular did in NServiceBus transport. There are popular tools like Supabase queues, pgBoss, beware of Just use PostgreSQL for all advice, as Just use X is dangerous advice, but can be a good choice.
Using PostgreSQL as an event store - that’s what I did in Marten, and do in Emmett, and also other tools are doing. Don’t write your own, now you have tools in all popular environments that’ll do it for you.
Full-Text Search Engine - Using PostgreSQL’s full-text search capabilities instead of Elasticsearch or Solr might not be always the best ide, but for many use cases, PostgreSQL’s tsvector and tsquery are more than sufficient for many cases and can eliminate operational complexity if you’re already using PostgreSQL and don’t have sophisticated needs.

Peter: What is a common thing you see as a consultant that many teams+orgs get wrong with postgres?

I think that the issues are similar to those in other environments:

Not using or misusing connection pooling,
not using explain and not tuning queries,
not using prepared statements for the commonly used queries,
not setting or oversetting indexes,
not using optimistic concurrency,
not thinking about the transaction isolation, and causing deadlocks.

From the issues that are special for PostgreSQL:

not knowing how PostgreSQL sequences works, and losing messages when using the outbox pattern or PostgreSQL as a message queue,
Not understanding VACUUM and autovacuum: PostgreSQL’s MVCC model means dead tuples accumulate and need cleanup. Many teams either disable autovacuum (terrible idea) or don’t tune it properly for their workload. This leads to table bloat, degraded query performance. Check more about it in:
- Internals of PostgreSQL VACUUM Processing
- Robert Haas: Understanding and Fixing Autovacuum

Woof, that was a lot! I hope that this was useful and I didn’t mess anything! How would you answer those questions? Or maybe you have more of them?

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Just use SQL they say... Or how accidental complexity piles on

Welcome to the new week!

“Just use SQL”, they say.

“No need for ORMs”, they add.

And they may be right, but…

Let’s say that you want to query users. In TypeScript, this could look like:

const sql = 'SELECT userName, email FROM users';

const result = await connection.execute.query(sql);

Easy peasy!

Now, let’s add a bit more, so filtering, let’s say that we’d like to find users with a specific e-mail.

const email = 'oskar@event-driven.io';
const sql = `
  SELECT userName, email
  FROM users
  WHERE email = '${ email }'
`;
const result = await connection.execute.query(sql);

Still a piece of cake, right? We’re using Template Strings, a feature available nowadays in most languages (oh well, Java, hey! I’m looking at you!).

This will work, but what if the email wasn’t the constant value, as I did, but was provided by the user? Well, then one could give an email with the value:

const email = `'; DROP TABLE users; SELECT 'drop da base!`;

…or something similar, depending on your SQL dialect, and cause (in) famous SQL Injection.

One solution would be to try to validate or sanitise the email value. Still, have you seen the e-mail regex? Not quite as easy as we’d like to: see here.

Luckily, databases give us parameterised queries so that we can provide params that will be sanitised and mapped. For instance, like that:

const sql = `
  SELECT userName, email
  FROM users
  WHERE email = ?
`;
const result = await connection.execute.query(sql, [email]);

Cool, but let’s say that we’d like to have email, be just one of the params, and our filtering params look as follows:

type UsersFilter = {
   userName?: string;
   email?: string
};

function filterUsers(filter?: UsersFilter): Promise<User[]> {
  const sql = `SELECT userName, email FROM users`;

  const params = [];

  if (filter && (filter.userName || filter.email)) {
     sql += " WHERE ";

     let hasParams = false;

     if(filter.userName) {
       sql += " userName = ?";
       params.push(filter.userName);
       hasParams = true;
     }

     if(filter.email) {
       if(hasParams) sql += " AND ";

       sql += " email = ?";
       params.push(filter.userName);
     }
  }

  return connection.execute.query(sql, params);
}

Boy, that escalated quickly…

And that’s just basic filtering of two parameters. What if there were 10? What if there were even more?

Of course, we could make it more generic, more sneaky, but I’m sure that you’re starting to get the idea where we’re getting into.

We’re getting into the Stringly Typed world where we’re doing complicated string manipulations. We not only need to generate SQL based on the provided criteria in a predictable manner, but also ensure that the parameters in the SQL query match the values provided in the parameters array when querying values.

What can we do about it? Let’s try to simplify the filtering query generation first.

type ParametrizedQuery {
  sql: string,
  params: unknown[]
}

const emptyQuery: ParametrizedQuery = { sql: '', params: [] };

We’re benefiting from the dynamic nature of the JS/TS here: each object is a record with field names and values. In other environments, we’d need to use either Reflection or code generation. Still, we find the field names,

Cool, now we can use it in our method:

function filterUsers(filter?: UsersFilter): Promise<User[]> {
  const sql = `SELECT userName, email FROM users`;

  const { sql: whereSql, params } = where(filter);

  return connection.execute.query(sql + whereSql, params);
}

That solved the issue for some time, but… What if we’d like to support not only equals but also: greater than, less than, LIKE, etc.?

Then we’d need to try harder!

type ColumnOperator = 'eq' | 'like' | 'gt' | 'lt' | 'gte' | 'lte';

type ColumnFilter = {
  columnName: string;
  value: unknown
  operator: ColumnOperator;
}

type WhereOperator = 'AND' | 'OR'

type WhereFilter = {
   columnFilters: ColumnFilter[];
   operator: WhereOperator;
}

We could define a column filter to contain the column name, value and operator. Then, the Where filter would be a collection of column filters and a binary operator.

We could map our filter object to the WhereFilter using:

type BuildWhereFilterOptions = {
   map?: (property: string, value: unknown) => ColumnFilter;
   operator?: WhereOperator
}

function buildWhereFilter(
  filter: Record<string, unknown>,
  options?: BuildWhereFilterOptions
): WhereFilter {
  const { map, operator } = options;

  const columnFilters = Object.entries(filter)
    .filter(([key, value]) => value !== undefined)
    .map(([prop, value]) => ((map ? map(prop, value): undefined) ?? {
      columnName: prop,
      value,
      operator: 'eq'
    }));

  return {
    columnFilters,
    operator: operator ?? 'AND'
  };
}

Besides the filter, we’re also allowing an optional pass mapping function to customise filtering and also the general binary operator.

We could use it in the where SQL generation:

function mapToSQL(columnFilter: ColumnFilter): string {
  const { columnName, operator } = columnFilter;

  switch (operator) {
    case 'eq':
      return `${columnName} = ?`;
    case 'like':
      return `${columnName} LIKE ?`;
    case 'gt':
      return `${columnName} > ?`;
    case 'lt':
      return `${columnName} < ?`;
    case 'gte':
      return `${columnName} >= ?`;
    case 'lte':
      return `${columnName} <= ?`;
    default:
      return `${columnName} = ?`;
  }
}

function where(filter: WhereFilter): ParametrizedQuery {
  const { columnFilters: filters, operator } = filter;

  return columnFilters.length > 0 ?
    {
      sql: ' WHERE ' + filters.map(mapToSQL).join(` ${operator} `),  
      params: filters.map(c => c.value)
    }
    : emptyQuery;  
}

And now we can customise our filtering, adding LIKE for user name with:

function filterUsers(usersFilter?: UsersFilter): Promise<User[]> {
  const sql = `SELECT userName, email FROM users`;

  const filter = buildWhereFilter(usersFilter, {
    map: (prop, value) => {
      if (prop === 'userName' && value) {
        return {
          columnName: prop,
          value: `%${value}%`,
          operator: 'like'
        };
      }
    }
  });

  const { sql: whereSql, params } = where(whereFilter);
  return connection.execute.query(sql + whereSql, params);
}

And that’s just a beginning! As we see, we have almost built a simple WHERE condition compiler! Or at least we’re not that far from it.

Where am I going with this? Why am I showing you such code?

Definitely NOT to tell you “just use ORM!”.

That’s not the point here. SQL is fine, ORMs are fine. When used in the proper context.

One of the reasons is that I’m working on such stuff in Pongo, adding multiple database support and some sneaky SQL handling. I don’t want to be alone with that!

The other, and most important, story behind it is to show you that it’s never “just”.

Just is enough for a simple or explicit context. Eventually, as our system grows, we’ll need to deal with complications. Popular tools might not be perfect; they may be too heavy, depending on design decisions, but as Gerald Weinberg said:

Things are the way they are because they got that way

You can live “just” with something. This can be a good starting point. Yet, we should always consider when deciding on DIY or “take off the shelf” solutions, where we need to land and what our main problem is to solve.

Because if it appears that we’ll need more advanced features, then we’ll follow the path similar to the one that people building popular libraries/tools have taken.

It’s always worth thinking whether “just” will be enough for us. Usually, “just” is enough only for some time. And it is better not to overlook this to change our approach, as then we can deal with accidental complexity related to wrongly weighted “just” as described in this article.

Read also more in:

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

My thoughts on Vertical Slices, CQRS, Semantic Diffusion and other fancy words

Welcome to the new week!

Vertical Slices in software architecture are pictured right now as the best thing since sliced bread. I won’t try to hide that, like it. I’ve written about CQRS and Vertical Slices over the years - how to slice the codebase effectively, shown examples, and explained why generic doesn’t mean simple, yet…

I still get questions about Vertical Slices Architecture (VSA). After a recent Discord discussion, I want to share some additional thoughts on how I see Vertical Slices Architecture, how it relates to CQRS, what different slicing strategies are, and (of course) the tradeoffs.

What Vertical Slices Architecture actually is

Funnily, many people using the Vertical Slices term don’t even know who coined it. Jimmy Bogard did it. In the original article contrasts traditional layered architecture with vertical organisation:

“In this style, my architecture is built around distinct requests, encapsulating and grouping all concerns from front-end to back. You take a normal “n-tier” or hexagonal/whatever architecture and remove the gates and barriers across those layers, and couple along the axis of change.”

He showed the famous image:

When you add or change a feature, you’re touching many different layers - the UI, the models, the validation, the data access. Instead of spreading these changes across multiple horizontal layers, you could group everything related to that feature. As Jimmy puts it:

“Minimize coupling between slices, and maximize coupling in a slice.”

So Instead of:

📁 controllers
  📄 ProductController.cs
  📄 OrderController.cs
📁 services  
  📄 ProductService.cs
  📄 OrderService.cs
📁 repositories
  📄 ProductRepository.cs
  📄 OrderRepository.cs

You organise more or less like:

📁 ecommerce
  📁 products
    📁 create-product
      📄 createProduct.cs
      📄 createProductHandler.cs
    📁 update-product
      📄 updateProduct.cs  
      📄 updateProductHandler.cs

Each vertical slice can decide for itself how to best fulfil the request. Simple CRUD? Use a transaction script. Complex domain logic? Bring in aggregates and domain services. The architecture doesn’t force a one-size-fits-all approach.

Why I think VSA is rebranded CQRS

For me, Vertical Slices Architecture is essentially CQRS with more prescriptive guidance on how to cut your architecture.

Let’s look at what CQRS actually says. Greg Young’s original definition is simple:

”Command and Query Responsibility Segregation uses the same definition of Commands and Queries that Meyer used and maintains the viewpoint that they should be pure. The fundamental difference is that in CQRS objects are split into two objects, one containing the Commands one containing the Queries.”

Ok, well, it’s simple but can be confusing. What are those “objects”? Data structures? Tables? Nay, they’re services that are handling specific business requests. We should segregate handling into commands and queries. Commands run business logic, and queries return data without causing side effects (so no-no to state change).

Essentially, CQRS is an architecture pattern explaining how to structure our application based on the behaviour. (Read more of my thoughts in CQRS facts and myths explained).

And hey, wait! Isn’t that already a vertical slice? Each command or query handler contains everything needed to process that specific request. The handler might talk directly to the database, or use domain objects, or call external services. The choice is encapsulated within that slice.

The difference is that VSA is more explicit about this organisational principle and extends it beyond just the command/query split.

It says: Don’t just separate commands and queries, organise your entire codebase this way. Put the API endpoints, the validation, the business logic, and even the data access for a specific feature all in one place.

In my experience, when you properly apply CQRS, you naturally drift towards Vertical Slices. The patterns reinforce each other. CQRS gives you the conceptual split (commands vs queries), and VSA gives you the organisational principle (group by feature, not by technical concern).

Two approaches to slicing

During our Discord discussion, I found myself drawing diagrams to explain two different approaches I’ve seen teams use successfully. Let me share them with concrete examples.

Approach 1: Pure vertical slices

In this approach, each slice is completely self-contained:

📁 room-reservations
  📁 reserving-room
    📄 reserveRoomEndpoint.ts    // ⬅️ API endpoint definition
    📄 reserveRoom.ts            // ⬅️ Business logic
  📁 confirming-reservation
    📄 confirmReservationEndpoint.ts
    📄 confirmReservation.ts
  📁 cancelling-reservation
    📄 cancelReservationEndpoint.ts
    📄 cancelReservation.ts

The endpoint file handles HTTP concerns and application logic:

app.post('/api/reservations/:id/confirm', async (req, res) => {
  const reservation =
    await reservationRepository.findById(req.params.id);

  const confirmedReservation =
    confirmReservation(
      reservation,
      mapToCommand(req)
    );

  await reservationRepository.save(confirmedReservation);

  res.status(200);
});

The business logic is pure:

function confirmReservation(
  reservation: Reservation,
  command: ConfirmReservation
) {
  const { confirmedAt, confirmedBy } = command;

  if (reservation.status !== 'pending') {
    throw new Error('Only pending reservations can be confirmed');
  }

  // some other business logic if needed

  return {
    ...reservation,
    status: 'confirmed',
    confirmedAt,
    confirmedBy,
    requiresPayment: daysBetween(new Date(), reservation.from) < 7
  };
}

Splitting the endpoint from business logic gives you:

Testing - You can test business logic without any dependencies. Just pass data in, assert on the result.
Reusability - The same business logic can be triggered from different sources (HTTP, message queue, CLI).
Focus - The endpoint deals with infrastructure (database, external services). The handler deals with business rules.
Evolution - When business logic grows complex, it’s already separated and ready to expand.

If you need to extract this into a microservice, you copy the folder. Everything related to confirming reservations is in one place.

Approach 2: Slices with a thin coordination layer

Sometimes you need some coordination - maybe all your reservation endpoints share authentication logic, or you want consistent URL patterns. Or maybe you’re a C# or Java developer, and taking away controllers is as hard (or harder) than taking your family away from you:

In this case, you might have:

📁 room-reservations
  📁 api
    📄 reservationsController.ts // ⬅️ Thin routing layer
  📁 reserving-room
    📄 reserveRoom.ts            // ⬅️ Business logic
  📁 confirming-reservation
    📄 confirmReservation.ts
  📁 cancelling-reservation
    📄 cancelReservation.ts

The routes file acts as a facade, handling HTTP concerns and orchestration:

export const reservationController = (app: Express) => {
  app.use('/api/reservations', authenticate);

  app.post('/api/reservations/:id/confirm', async (req, res) => {
    const reservation =
      await reservationRepository.findById(req.params.id);

    const confirmedReservation =
      confirmReservation(
        reservation,
        mapToCommand(req)
      );

    await reservationRepository.save(confirmedReservation);

    res.status(200);
  });

  // Other endpoints follow similar pattern
};

The business logic remains pure in each slice. The facade is purely about the entry point for the application logic and HTTP handling, while business logic stays in individual slices.

This facade serves several purposes:

It provides a consistent URL structure for all reservation operations
It handles cross-cutting concerns like authentication in one place
It maps HTTP concerns (status codes, headers) away from business logic
It makes it easier to see all available operations at a glance

But notice - the facade is thin. It’s not a “service layer” with business logic. It’s purely about HTTP-to-domain translation. The actual business logic remains in the vertical slices.

Both approaches have tradeoffs. The pure slices approach gives you maximum independence - each feature is truly isolated. But you might end up duplicating cross-cutting concerns like authentication or error handling. The facade approach gives you a central place to handle these concerns, but now you have coupling between the facade and the slices.

Neither approach solves all problems. With pure slices, you risk turning into a ”feature factory” where each feature is built in isolation without considering the overall system design. You might lose sight of the broader domain concepts because everything is scattered across independent slices. With the facade approach, you risk the facade growing into a bloated orchestration layer that knows too much about the domain.

The choice depends on your context. If you’re building a system where features are truly independent (maybe different teams own different features), pure slices work well. If you need consistency across related operations, a thin facade can help without sacrificing the benefits of vertical organisation.

Jimmy Bogard actually goes further than just backend organisation - he suggests that slices can and should include the UI. A vertical slice can go from the user interface all the way to the database. If the goal is to minimise coupling between features, why stop at the API layer? But the difference in tooling, team skill sets between backend and frontend often makes this challenging, which is why many teams slice vertically on the backend while keeping a more traditional organisation on the frontend.

CRUD and CQRS are orthogonal

People often ask if VSA or CQRS make sense for CRUD systems. They do. Your commands might be OnboardUser, UpdateProductDefinition, and CancelSubscription, but behind the scenes, they could be a typical Create, Update, and Delete implemented with Object-Relational Mapper. Naming is a key thing, as it already establishes a business process behind this action.

Neither CQRS nor VSA require complex domain logic. They’re structural patterns that work for simple scenarios, too.

I’ve heard dozens of times that CRUD is simpler, and CQRS and VSA bring additional overhead. And I just can’t objectively see that if we follow the basic principles. Why is naming things based on the business process an overhead? Why does reflecting this flow explicitly in the code add additional work?

I understand that requires thinking about what we’re doing, asking business and ourselves: “What is the feature that we want to deliver?”. Still, from my perspective, that’s the thing we’re brought to the game. Building a model to solve the specific business problem. That starts with the naming and the structure.

If we name things closer to the business, we’re giving yourself more options for future evolution.

At the beginning, things look simple, but when requirements come and the system evolves, quite often the system stops looking like a simple CRUD.

Vertical Slices can help in that, as you already established the split based on the features, start to think about them, but you can keep the implementation simple and evolve when you learn more about your domain.

Feature folders and nested modules

Speaking about organisation, we should not mistake folder layout with software architecture, but it’s also good if your code reflects the emphasis you make in your design.

As mentioned, Vertical Slices are focused on the specific business flow, yet… Beyond individual features, you need to organise at a higher level.

The approaches I’ve shown so far work well for individual features, but what happens when your system grows? How do you organise dozens or hundreds of features without ending up with a flat, unmanageable structure?

Kamil Grzybek wrote an excellent article about Feature Folders, where he contrasts technical folders (Controllers, Services, Repositories) with business-focused organisation. His key insight is about cohesion.

Elements that change together should live together. This principle directly reduces cognitive load.

When you’re working on a reservation feature, you don’t need to keep a mental map of where the controller lives, where the service is, or where the repository might be. Everything is in the ReservingRoom folder. As Kamil puts it, this creates high cohesion because “the elements inside a module belong together.”

In practice, this often means nesting your vertical slices within larger business modules:

📁 reservations
  📁 room-reservations
    📄 api.ts                    // ⬅️ Public API of room reservations
    📁 reserving-room
      📄 reserveRoom.ts
      📄 reserveRoomEndpoint.ts
    📁 confirming-reservation
      📄 confirmReservation.ts
      📄 confirmReservationEndpoint.ts
    📁 cancelling-reservation
      📄 cancelReservation.ts
      📄 cancelReservationEndpoint.ts
  📁 group-reservations
    📄 api.ts                    // ⬅️ Public API of group reservations
    📁 creating-group-reservation
      📄 createGroupReservation.ts
      📄 createGroupReservationEndpoint.ts
    📁 assigning-rooms-to-group
      📄 assignRoomsToGroup.ts
      📄 assignRoomsEndpoint.ts
📁 billing
  📄 api.ts                      // ⬅️ Public API of billing module
  📁 invoicing
    📄 api.ts                    // ⬅️ Public API of invoicing
    📁 generating-invoice
      📄 generateInvoice.ts
      📄 generateInvoiceEndpoint.ts
    📁 sending-invoice
      📄 sendInvoice.ts
      📄 sendInvoiceEndpoint.ts
  📁 payments
    📄 api.ts                    // ⬅️ Public API of payments
    📁 recording-payment
      📄 recordPayment.ts
      📄 recordPaymentEndpoint.ts
    📁 processing-refund
      📄 processRefund.ts
      📄 processRefundEndpoint.ts

This creates a hierarchy where:

Top-level folders represent major business areas (reservations, billing)
Second-level folders represent subdomains or aggregates (room-reservations, invoicing)
Leaf folders represent specific use cases (reserving-room, generating-invoice)

The api.ts files act as module boundaries, explicitly defining what’s public and what module is exposing and what features it has.

This gives you encapsulation and focuses on the responsibility. Other modules can only use what you explicitly expose. When modules need to communicate, they do so through their public APIs - though you need to be careful about coupling. Direct synchronous calls between modules can create distributed transaction problems.

This gives you encapsulation without requiring separate packages or complex dependency management. Other modules can only use what you explicitly expose. Thus, the facade pattern still has its point with Vertical Slices.

Such an approach scales naturally. When you’re working on invoicing, you’re in the Billing/Invoicing folder. Everything related to invoicing is there. You don’t need to jump between technical layers. The reduced cognitive load is dramatic - instead of thinking “where do I put this validation?” you think “I’m working on generating invoices, so it goes in GeneratingInvoice.”

This also aligns with team organisation. The billing team owns the Billing folder. The reservations team owns the Reservations folder. Conway’s Law works in your favour instead of against you.

Starting simple

The initial structure matters less than your ability to evolve it.

Start simple. If you’re building a proof of concept, don’t create elaborate folder hierarchies. But structure your code so that evolution is possible. That’s why I like CQRS even for simple systems - it gives you a place to put complexity when it inevitably arrives.

I’ve seen teams start with pure vertical slices and later add a coordination layer once they’ve identified patterns. I’ve seen the reverse, too - starting with controllers and gradually pushing logic into focused handlers. Both paths work because the team could evolve the architecture as they learned more about their domain.

The anti-pattern is the “big design up front” where you create elaborate abstractions for problems you don’t have yet. Start with the simplest thing that could possibly work, but keep the path open for evolution.

From my experience, slicing your code by features pays off much better and reduces the range of change than artificial cross-cutting splits by technical aspects.

Ok, so why is CQRS and Vertical Slices pictured as such complex beasts?

Semantic Diffusion: How good ideas get corrupted

Martin Fowler coined the term Semantic Diffusion back in 2006, describing what happens when:

“you have a word that is coined by a person or group, often with a pretty good definition, but then gets spread through the wider community in a way that weakens that definition.”

He compared it to the children’s game where a message gets passed from person to person - each retelling adds distortions until the original meaning is lost. Popular terms suffer from this the most because they create longer chains of reinterpretation.

How CQRS got diffused

CQRS is a perfect example of semantic diffusion in action. Greg Young’s original definition was crystal clear: separate your objects into commands and queries. That’s the entire pattern.

Somewhere along the way, CQRS accumulated baggage. Today, if you mention CQRS, people assume you mean:

Two databases - One for writes, one for reads. Greg never said this. You can do CQRS with a single table if you want.
Event Sourcing - Yes, they work well together, but CQRS doesn’t require events at all. You can update the state directly.
Eventual consistency - Again, not required. Your read model can be updated synchronously in the same transaction.
Message queues - Kafka, RabbitMQ, whatever. Nope. You can do CQRS in a single process with direct method calls.
Complex architecture - People think CQRS means microservices, distributed systems, and architectural astronautics. It doesn’t.

The original idea - separate commands and queries - is simple and useful. It’s arguably simpler than mixing reads and writes in the same objects. But semantic diffusion has turned it into a boogeyman.

This simplicity enables multiple options and extensions. And that posed the risk we’re facing—the risk of misinterpretation.

Vertical Slices turn for diffusion

I think that it’s actually good that Jimmy coined a new term instead of reusing CQRS. He (sub?)consciously tried to avoid Semantic Diffusion. But that didn’t help in getting this issue for VSA.

Now I’m watching the same thing happen to Vertical Slices Architecture. Jimmy Bogard’s original description was clear: organise code by features, not by technical layers. Couple along the axis of change.

But already the corruption has begun:

“You can’t share any code between slices” - Jimmy never said this. He said to minimise coupling between slices. Minimise isn’t zero.
“Every slice must have its own database table” - Where did this come from? Slices can share storage.
“You must copy-paste everything” - No. You can extract common patterns when they prove themselves useful.
“You need MediatR” (in .NET) - VSA is an architectural pattern, not a library choice.
“Slices must have exactly this folder structure” - Missing the point entirely. The structure should emerge from your needs.
“A vertical slice is an entire module” - Some people argue that a vertical slice should be an entire business module (like “Billing” or “Inventory”). But Jimmy’s original article talks about individual features - specific user requests. A single command or query. The confusion about granularity shows how far we’ve drifted from the source.

People teaching VSA often haven’t read Jimmy Bogard’s original article. Some don’t even know who coined the term.

Even worse, some communities are developing their own interpretations where vertical slices mean something completely different (e.g. modelling processes).

Each person who retells the pattern adds new constraints, removes important nuances, or changes the meaning to fit their agenda.

Why semantic diffusion happens

I think semantic diffusion happens because successful patterns attract attention. People want to be associated with them - “we’re doing CQRS” sounds more impressive than “we separated our reads and writes.” But most people learn about these patterns second or third-hand, and each retelling adds distortions.

The pattern also shifts from principles to prescriptions. Where Greg Young said “separate commands and queries,” people hear “you must have two databases.” Where Jimmy Bogard said “minimize coupling between slices,” people hear “zero shared code allowed.” Nuance gets lost in favour of absolute rules that are easier to remember and teach.

Framework authors and tool vendors contribute to this. They build “CQRS frameworks” or “VSA templates” that embed their own opinions and constraints. Soon, people can’t distinguish between the pattern and the tool’s implementation of it. The original simple idea gets buried under layers of accidental complexity.

Teams reject CQRS because they think it requires eventual consistency. Others implement complex event sourcing and messaging systems when all they needed was to put queries in different files from commands. Instead of asking “what problem are we solving?” teams ask “are we doing real CQRS?” as if there’s a certification board checking their implementation.

Closing thoughts

Both CQRS and Vertical Slices Architecture are great patterns. They’re even more useful together - CQRS provides the conceptual model (separate commands and queries) while VSA provides the organisational principle (group by feature).

But their usefulness depends on understanding what they actually are, not what common application has made them into.

CQRS is just about separating commands and queries. VSA is just about organising by feature instead of the technical layer.

I’ve noticed that it’s easy to contribute to semantic diffusion without realising it. You solve a hard problem with vertical slices, and suddenly, you’re seeing it as the answer for every system. Before you know it, you’re adding your own constraints and requirements that weren’t in the original.

So next time someone tells you that CQRS requires event sourcing, or that VSA forbids shared code, ask where they read that. Most of the time, they didn’t read it anywhere - they just heard it from someone who heard it from someone else.

So my final request is, if you want to learn about the pattern, try to get to the original sources. They’re usually shorter and clearer than you think.

For instance:

Or check my take on CQRS:

Cheers!

Oskar

p.s. This article will also be published on my blog. In the following weeks, I’m planning to centralise both this newsletter and my blog together. Stay tuned!

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

Compilers Aren't Just for Programming Languages

Welcome to the new week!

What does this streaming pipeline have to do with a programming language compiler?

const deliveryPipeline = pipe(
  Async.map(async order => {
    const restaurant = await fetchRestaurantDetails(order.restaurantId);
    return { ...order, restaurant, timestamp: Date.now() };
  }),
  filter(order => order.restaurant.isOpen),
  Async.map(async order => {
    const [driver, route] = await Promise.all([
      findNearestDriver(order.restaurant.location),
      calculateRoute(order.deliveryAddress, order.restaurant.location)
    ]);
    return { ...order, driver, route };
  }),
  map(order => ({
    ...order,
    estimatedTime: order.route.duration + order.restaurant.prepTime
  }))
);

At first glance? Nothing!

This looks like a typical functional pipeline that processes food delivery orders. But here’s what actually happens when you run this code: these operations don’t execute immediately. Instead, they create a description of what should happen - a data structure that represents your intent. Then, a compiler transforms this description into optimised code tailored to your specific JavaScript environment.

This separation between declaring what you want and how it executes is the same fundamental principle that powers programming language compilers. When you write int x = 5 + 3 in C, the compiler doesn’t just translate it word-for-word to assembly. It might optimise it to int x = 8, or eliminate it entirely if x is never used. The compiler understands your intent and finds the best way to achieve it.

This same principle can apply to streaming pipelines. The pipeline definition above would just be data - a series of step descriptions. The actual execution would depend on where and how you run it.

The JavaScript Beautiful Mess

JavaScript’s ecosystem perfectly illustrates why we need this approach. Node.js has had streams since the beginning - they’re fast, battle-tested, and deeply integrated with the runtime. They power everything from file I/O to HTTP. But Node.js streams don’t exist in browsers. Instead, browsers implement the Web Streams standard - a completely different API with different behaviour, different backpressure mechanisms, and different performance characteristics.

You might think:

“Great, let’s use Web Streams everywhere since Node.js added support for them.”

But here’s the reality: Web Streams in Node.js are a compatibility layer, not a native implementation. They’re slower than Node.js streams, sometimes significantly so. The polyfill for environments that don’t support Web Streams? It’s not fully compliant and has its own performance issues.

A naive implementation would force an impossible choice. Use Node.js streams, and your code won’t run in browsers. Use Web Streams, and you sacrifice performance in Node.js. Use async generators as a lowest common denominator, and you lose the benefits of native streaming entirely. Try to support everything with runtime checks, and you end up with:

async function processStream(source) {
  if (typeof window !== 'undefined' && source instanceof ReadableStream) {
    // Web Streams path
    const reader = source.getReader();
    // ... different processing logic
  } else if (source && typeof source.pipe === 'function') {
    // Node.js streams path  
    // ... completely different API
  } else if (Symbol.asyncIterator in source) {
    // Async generator fallback
    // ... yet another approach
  }
  // ... and this is just the beginning
}

This code quickly becomes unmaintainable. Each code path needs its own error handling, backpressure management, and performance optimisations. Worse, you can’t easily apply optimisations like operation fusion because the implementations are so different.

As I explored in my article about TypeScript’s performance, the key to JavaScript performance is keeping the event loop spinning and leveraging native APIs. But when those native APIs are incompatible across environments, we need a different approach.

This is where compilation comes in. Instead of writing environment-specific code, we write a description of what we want to happen. A compiler then transforms this description into the optimal implementation for each environment. Same API, different execution. Write once, optimise everywhere.

I’m building this approach into Emmett’s event processing system. Let’s call it codename FusionStreams.

As I outlined in my workflow engine design, modern event-driven systems need to handle complex workflows - not just simple event handlers, but multi-step processes with error handling, compensation, and state management. These workflows need to run efficiently whether they’re processing events on a server, updating projections in a browser, or handling commands at the edge.

How Compilers Actually Work

Understanding how FusionStreams could work requires building it from the ground up. First, operations need representation as data:

interface Step {
  type: string | symbol;
  fn?: Function;
  options?: any;
  isSync?: boolean;
}

When you write map(x => x * 2), it doesn’t execute anything. It returns a step description. This is crucial - by delaying execution, we can examine the entire pipeline before running it. We can see patterns and opportunities for optimisation that would be invisible if we executed operations immediately.

The compilation process transforms these step descriptions into executable functions. Traditional compilers work in phases:

lexical analysis,
parsing,
semantic analysis,
optimisation,
and code generation.

Our streaming compiler follows similar principles but is simpler.

During the analysis phase, the compiler examines the pipeline structure. It identifies consecutive synchronous operations, async boundaries, and error handling points. This is where the first major optimisation happens - sync operation fusion.

Consider this sequence:

map(x => x + 1)
filter(x => x > 10)  
map(x => x * 2)

A naive implementation would create three function calls for each item, allocating intermediate objects and jumping between functions. The compiler recognises these are all synchronous and fuses them into a single operation. Instead of three function calls, you get one tight loop. No intermediate allocations. The event loop keeps spinning.

This optimisation is only possible because we separated declaration from execution. If each operation were executed immediately, we couldn’t see the pattern. This is exactly how modern language compilers work - they analyse entire programs to find optimisation opportunities that aren’t visible when looking at individual statements.

Early programmers wrote assembly directly. They counted cycles, allocated registers by hand, and knew exactly what the CPU would execute. As programs grew from hundreds to millions of lines, this approach collapsed under its own weight.

High-level languages offered an escape, but early compilers were disappointingly literal. They translated your C code line-by-line into assembly, producing results that ran slower than hand-optimised code. Programmers faced an uncomfortable choice: write maintainable code that ran slowly, or write fast code that was impossible to maintain.

Modern compilers changed this equation. They don’t translate - they analyse and transform. When you write a for-loop that adds numbers, the compiler might recognise the pattern and replace it with a multiplication. It might notice your loop accesses memory sequentially and generate vector instructions. Or it might determine the result at compile time and replace the entire loop with a constant.

Each phase transforms the pipeline further from its declarative form into executable code. The analysis phase groups operations and identifies optimisation opportunities. The optimisation phase applies transformations like operation fusion. The code generation phase produces the final executable form tailored to the runtime environment.

This evolution from direct execution to analysed transformation appears everywhere in software:

Java took this idea further with Just-In-Time compilation. Your Java code compiles to bytecode, which the JVM then compiles again to machine code while your program runs, optimising based on actual usage patterns.
Database query planners are compilers too. They transform declarative SQL into execution plans, choosing between nested loops, hash joins, or merge joins based on table statistics and available indexes.
.NET’s LINQ doesn’t execute queries directly. When you write customers.Where(c => c.City == "London").Select(c => c.Name), it builds an expression tree. The LINQ provider then compiles this tree - to SQL for databases, to objects for in-memory collections, or to web service calls for remote data.
React’s new compiler analyses your components to understand data flow and dependencies. It then generates optimised code that eliminates unnecessary re-renders by compile-time analysis of which props actually affect rendering.

Each system recognised that separating “what you want” from “how to do it” enables optimisations impossible with direct execution. FusionStreams applies this same principle to streaming pipelines.

From Theory to Implementation

Let’s see how this works with a more complex operation. When processing multiple async operations, you might naturally reach for parallel execution:

Async.map(async order => {
  const [driver, route] = await Promise.all([
    findNearestDriver(order.restaurant.location),
    calculateRoute(order.deliveryAddress, order.restaurant.location)
  ]);
  return { ...order, driver, route };
})

This works, but what happens when you need to process thousands of orders? Without concurrency control, you might overwhelm your services. You may replace it later with a new operator mapParallel for processing multiple items in parallel, passing multiple functions returning promises. The compiled version could then manage a pool of concurrent operations, reuse resources, and adapt its strategy based on the environment.

In Node.js, it might use worker threads for CPU-intensive operations. In a browser, it might use Web Workers or carefully time-slice execution to avoid blocking the UI. The declaration would stay the same, but the execution adapts.

One powerful aspect of this approach would be extensibility. Because operations are just data structures, users could define their own operations and register compilers for them.

For instance, real-world streaming pipelines rarely follow a simple linear flow. They branch, merge, and coordinate complex operations. The fanout pattern shows how compilation enables these sophisticated patterns while maintaining performance. When processing a food delivery order, you need to notify the kitchen, find a driver, and process payment simultaneously:

fanout({
  kitchen: pipe(
    Async.map(order => notifyKitchen(order)),
    Sync.map(response => ({
      orderId: order.id,
      prepTime: response.estimatedMinutes
    }))
  ),

  driver: pipe(
    Async.map(order => findDriver(order.restaurant.location))
  ),

  payment: pipe(
    Async.map(order => chargePayment(order.payment))
  )
})

During compilation, this pattern transforms based on the environment’s capabilities. If worker threads are available, each branch might run in true parallel. Without them, the compiler might use Promise.all for concurrent execution. In memory-constrained environments, it might process branches sequentially. The same declaration compiles to different strategies.

Want rate limiting? Define what it means declaratively, then provide compilers that implement it efficiently for each environment.

const RATE_LIMIT = Symbol('rateLimit');

function rateLimit(requestsPerSecond: number): Step {
  return {
    type: RATE_LIMIT,
    options: { requestsPerSecond },
    isSync: true
  };
}

The rate-limiting compiler could use token buckets in memory for single-instance Node.js apps, Redis for distributed systems, or localStorage in browsers. The pipeline definition wouldn’t change - only the compilation target.

Naive implementation could work as:

builder.registerSyncCompiler(RATE_LIMIT, (step) => {
  const { requestsPerSecond } = step.options;
  let tokens = requestsPerSecond;
  let lastRefill = Date.now();

  return (item) => {
    const now = Date.now();
    const elapsed = (now - lastRefill) / 1000;
    tokens = Math.min(requestsPerSecond, tokens + elapsed * requestsPerSecond);
    lastRefill = now;

    if (tokens >= 1) {
      tokens -= 1;
      return item;
    }

    return {
      result: {
        ok: false,
        error: new Error(`Rate limit exceeded: ${requestsPerSecond} requests/second`)
      },
      context: item.context
    };
  };
});

This extensibility would transform the system from a fixed library into a compilation framework. Domain-specific operations could compile to efficient implementations. A database operation could compile to use connection pooling in Node.js but IndexedDB transactions in browsers. A caching operation could use Redis in production but memory in development.

TLDR

Building event-driven systems that work across various JavaScript environments presents a specific technical challenge. Each environment has different APIs and performance constraints. The traditional solution - writing environment-specific code or using compatibility layers - leads to either maintenance headaches or performance compromises.

Compilation offers a different trade-off. By treating pipeline definitions as data to be transformed rather than code to be executed, you can generate environment-specific implementations from a single source. The complexity moves from runtime to build time, where it can be managed systematically.

This approach works because it acknowledges that abstraction and performance don’t have to conflict - they conflict when the abstraction exists at runtime.

A compiler can understand that three consecutive map operations should become one function, that Node.js should use native streams while browsers use web streams, or we can fallback to async generators. These optimisations are mechanical transformations, not clever hacks.

The same pattern appears whenever systems process declarative definitions. SQL becomes query plans. GraphQL becomes resolver calls. Configuration becomes lookup tables. In each case, the transformation from declaration to execution provides an opportunity for optimisation that interpretation can’t match.

For event-driven architectures, where the same business logic might run in vastly different contexts with vastly different performance requirements, this separation becomes essential. The alternative is maintaining multiple implementations or accepting suboptimal performance.

Compilation provides a third path: write the logic once, optimise for each target when you need it.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

Requiem for a 10x Engineer Dream

Welcome to the new week!

People claim they get 10x productivity boosts with AI coding tools. After my recent experiments with Claude Code, I’m starting to think we’re not using these tools the same way. Or that they’re just lying. Or both.

Those hyped posts seem to me like teenagers talking about sex. A lot of talking, not much doing.

Yes, I’m LLM-sceptic. But honestly, when I did a few times quick check and asked:

“Ok, so what’s your development flow”

Then most of the answers I got could be summarised into:

“Well, I’m typing and copy/pasting into chat”.

“I’m doing Tab, Tab in Copilot.

Fine, that’s also what I was doing. Even as a sceptic (or hater, if you prefer), I like to try stuff on my own, before I make a judgement.

I see the biggest potential improvement in LLMs in the boring, repetitive tasks, ideally done in the background without much intervention from me. Maybe then I could get some boost working as a solopreneur. In theory, that’s what the new wave of CLI tooling promises.

I decided to give it a try and even paid for Claude Code Max.

I tried to find a task that would be a stereotypical perfect match for an AI assistant.

I have a workshop for The Light and the Dark Side of Event-Driven Architecture. I wanted to expand some hands-on exercises to use RabbitMQ and Kafka instead of an in-memory implementation. Sure, “adding Kafka” sounds heavy, but it’s not that.

I started with Java exercises. They use Spring Boot, which streamlines Kafka and RabbitMQ setup. Plus there are many resources in the Internet on how to do the setup. Essentially, the goal was to:

copy/paste existing code, that had tests, working implementation,
replace the in-memory implementation with Kafka or RabbitMQ,
Add Spring Boot config for Kafka or RabbitMQ,
Update the test setup to use TestContainers and add some awaits respecting the async nature of those tools.

So do the boilerplate that I didn’t want to spent much time.

I wrote a detailed spec and hoped Claude could handle this boring work while I did something else. After a week, even with hints from me, it was still spinning in circles, unable to finish.

Yet it was claiming that it got a huge success, getting 80% of tests passing. It wasn’t even able to complete the proper setup. Indeed, great success for the code that had a 100% passing rate. So much for autonomous AI assistance.

Tho I’m not the person who easily gives up. Last Friday, I tried again, this time staying fully engaged. I wanted to refactor Pongo’s SQL handling. I wanted to make it generate parameterised queries instead of the raw ones. Thanks to that, getting better performance by benefiting from the query plan reuse.

Again, that can sound complex, but it’s more complicated than complex. Mostly, it had to refactor how SQL was parsed, and update the execution to use not only raw SQL but also parameters.

With the Claude Code setup, I followed the nice guide from Harper Reed:

brainstorming,
spec generation,
implementation plan with prompts.

Everyone is saying that you need to give LLM a precise plan to make it work correctly. And that’s entirely true. I managed to get acceptable results, especially after doing the final polish on my own. So what’s the story then?

I also realised the fundamental flaw in this whole enterprise.

The more detailed you make your prompts, the better Claude Code performs. But to get it to work correctly, you need specifications so detailed that you’ve almost done the implementation.

You’re not necessarily saving time - you’re programming in Markdown instead of code.

It’s even worse than that, though. To be good at it, you need to become the world’s most annoying micromanager.

You know the type - watching over someone’s shoulder, correcting every keystroke, eventually grabbing the keyboard.

Just like my childhood friend Gabriel. He was the first one from the block to get a personal computer. Yes, I’m that old. You had to wait until he let you play Mortal Combat on his Amiga 600. But then you need to cherish your moment, as he was watching from behind, and quickly got annoyed and shouted:

“that’s not how you play!”

Then, grabbing the joystick to “show you how it should be done.”

So, to be effective, you need to behave just like Gabriel. You need to watch Claude plotting stuff in the console. As one moment it generates sensible code, then completely loses the plot the next.

When you hit unexpected problems (and you always do), Claude can’t assess whether it needs help. I told it to ask for clarification in these situations. Sometimes it does, usually it doesn’t, preferring to solve everything by throwing more code at it. Mountains of code. Complex solutions to simple problems.

Also, if you’d like to make it autonomous, working in the background, then you need to provide a longer plan. The longer it gets, the longer it needs to be worked on. Adding to that, it has to be precise; the likelihood and the consequences of missing something important will get more severe. Of course, we can fix the plan and regenerate it, but…

We’ve seen this movie before. It’s waterfall development with extra steps. Remember UML tools? I still have nightmares from IBM Rational RequisitePro. They promised that we were going to model the world perfectly, then generate all the code. Just describe every class, every relationship, every edge case in beautiful diagrams. The tools would handle the mundane coding part.

It failed because the specification was never the hard part - it was discovering what to build while building it. Now we’re making the same bet with prompts, hoping that if we just describe things clearly enough, the implementation will handle itself.

After a day of this, Claude had produced something that technically worked. But was it faster than doing it myself? Doubtful. Plus, I had to fix everything afterwards.

Someone might say, “but you’re experienced, it helps juniors more.” Except that less experienced developers have even less ability to judge if Claude’s solutions are correct or completely insane.

Speaking about junior devs. Many people claim that working with LLM is like working with a junior. I think that’s disrespectful and just plain wrong. Junior devs don’t have enough knowledge yet, but they learn, you can teach them, mentor them, and they will get better. They can also reason and react based on what they’re doing; they’re not just code outputters. LLMs won’t learn, as they don’t have memory; they just have context, which they happen to lose quickly and randomly.

And another aspect: joy. We’re hearing a lot about vibes.

But what vibes are they? Are they more like:

Or actually more like?

A lot of us came to programming to express our creativity. The puzzle-solving, the flow state, and the satisfaction of building something with our own hands.

Replace that with prompt engineering and micromanagement, and you’ve sucked all the fun out of the room.

I spent a day watching an LLM struggle to generate sensible code. It was exhausting. Not the good exhaustion of solving hard problems, but the bad kind - like teaching someone who forgets everything between lessons. If this is the future of programming, we’re going to see a new kind of burnout: LLM burnout. Death by a thousand prompts.

Those tools have a sneaky UI with intentional gamification. It’s kinda addicting watching those CLI and trying to force it to do what you asked. It’s like playing an MMORPG game. Still, it’s exhausting.

We already established Zoom fatigue, and I believe that we’re also about to establish LLM fatigue.

Of course, those tools will get better, any LLM fan will tell you that, as talking with GenAI fans is like talking to TV Series fans:

“Dude, check those series, they’re great! I mean, the first five seasons might not seem impressive if you’re not into such series, but watch it, the 6th season will be mindblowing!”

I could also improve my prompting skills, but will it change the scale of the issue? I’ll try, but for now, I doubt that it’ll change the outcome substantially.

There are practical uses of those tools. There’s value in using Claude for brainstorming. As a solopreneur, I don’t always have someone to challenge my ideas. Having something respond, even imperfectly, beats talking to yourself. But let’s be honest about what’s happening - we’ve invented a very expensive rubber duck that occasionally contributes ideas and can be useful to solving specific, narrowed-down coding challenges.

Maybe I’m missing something. Maybe those 10x gains are real for others. But after two serious attempts with real-world tasks, I’m convinced we’re chasing a mirage. We’re not getting dramatic productivity improvements. We’re getting a new flavour of an old fantasy - that if we just specify things precisely enough, computers will do the hard work for us.

So do you really want to be like Gabriel?

Do we want to believe again that micromanagement is good management? Or that waterfall this time will work?

Or maybe you want to be a Markdown Programmer?

What are your thoughts? Did you find the workflow or type of tasks that work for you?

Check also:

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

Why Open Source Isn't Always Fair. Dual licenses explained

Welcome to the new week!

Rug pull!

Open Source drama!

Yet another License change!

We’re being hit recently by headers on social media. And guess what, I’m also making a similar change in my OSS projects. And let me explain to you why.

Using my projects as an example, I’ll try to explain to you the current OSS landscape and licensing details. And why you should care!

What’s the story, Morning Glory?

I propose dual licensing Emmett and Pongo under AGPLv3 and SSPL. Users choose which license suits their needs. The intention is to establish clear, flexible licensing for users and due diligence to prevent exploitation, enabling a sustainable model for maintainers.

Problem

Emmett has no license by design. Without an explicit license, anyone wanting to use it should ask permission or clarification. This was intentional, but not for gatekeeping. I didn’t actually enforce that on anyone.

I kept it this way to avoid the trap of starting with a permissive license and later changing terms, which communities see as “rug pull”.

Pongo uses Apache 2.0, allowing anyone, including cloud providers, to take the code, wrap it in their services, and sell access without contributing back.

From the beginning, I’ve been transparent: to work on Emmett and Pongo properly, I need to make them sustainable. This isn’t theoretical. I maintained Marten for years, and despite its success, I couldn’t make it financially sustainable. Licensing won’t make it sustainable, so I don’t plan to provide a paid license for the core code, but it’s due diligence to set the right legal foundations.

Now, with contributors who’ve put significant work into these projects, I need to formalise a structure that’s fair to them while creating the sustainability I’ve always said I needed.

Background: The Pattern of Exploitation

The current Open Source model assumes symmetry between all users. But as David Whitney argues, in his NDC talk “Open-Source Exploitation”, when there’s a power imbalance between the largest organisations in the world and someone putting code on the internet, the organisation always benefits. When the OSI insists that cloud providers deserve equal treatment to individual developers, it forces projects into defensive positions.

The database world has experienced this firsthand:

MongoDB’s Stand (2018)

MongoDB thrived under AGPL until cloud vendors began offering MongoDB as a commercial service, keeping all revenue without contributing back. MongoDB created SSPL with the intent: if you offer our database as a service, open source your entire service infrastructure or pay us.

AWS built DocumentDB, a proprietary MongoDB-compatible database. MongoDB protected itself but lost AWS as a user. They remain SSPL-only, never adding back an OSI-approved option.

Elastic’s Confrontation (2021)

Elastic’s case reveals deeper issues. AWS offered Elasticsearch as a managed service and using the project name. Besides the similar sustainability issue as MongoDB had, that also created the confusion. The responsibility for fixes were conflated, and too often Elastic was blaimed for the issues that were specific to AWS Managed service.

The relationship deteriorated. Elastic switched from Apache 2.0 to dual licensing under SSPL and their proprietary Elastic License. AWS forked Elasticsearch to create OpenSearch, maintaining the Apache 2.0 license.

But here’s the twist: in 2024, Elastic added AGPLv3 as a third license option.

Redis’s Preemption (2024)

Similar twist happened to Redis. In March 2024, they switched from BSD to dual licensing under SSPL and RSALv2. Within weeks, the Linux Foundation announced Valkey, a Redis fork backed by AWS, Google, and others.

In 2025, Redis added AGPLv3 with Redis 8, integrating features from their previously commercial Redis.

The pattern becomes clear.

Why the Return to Open Source?

Elastic and Redis (but not MongoDB) added AGPLv3 after their license changes. Two reasons:

First, SSPL isn’t recognised as open source by the OSI because it discriminates against offering software as a service. The OSI maintains that open source must treat all users equally—even when those users are trillion-dollar companies exploiting smaller projects.

Second, strategy. AGPLv3 allows projects to incorporate improvements from forks legally. You can’t take BSD-licensed code from Valkey and add it to SSPL Redis. But between compatible copyleft licenses? That works. Yes, also if there’s more than one license for the same code.

The dual licensing trend represents adaptation to this reality. It’s not ideal, but it’s pragmatic.

The Licenses Explained

Permissive Licenses: The Current Problem

Apache 2.0, MIT, and BSD say: take this code, do whatever you want, just give attribution. This generosity becomes vulnerability when cloud providers monetise your work without reciprocating.

AGPLv3: Network Copyleft

AGPLv3 extends GPL’s copyleft to network use. If you modify AGPLv3 software and users interact with it over a network, you must provide the source code of your modifications.

“Users interacting over a network” means any service where users access the functionality—web applications, APIs, any network protocol. This applies only to modifications of the licensed software itself.

Many developers misunderstand this. They believe AGPLv3 requires open sourcing their entire application. It doesn’t. If you build a banking system using Emmett, your business logic remains proprietary. Only if you modify Emmett’s core event sourcing engine and expose that modified version as a service would you need to share those Emmett modifications.

SSPL: Explicit Service Protection

SSPL takes AGPLv3 and replaces Section 13. The new text:

if you make the functionality of the software available to third parties as a service, you must release the “Service Source Code”—all management software, user interfaces, APIs, automation, monitoring, backup systems, storage software, hosting software.

If AWS (or any other company) wants to offer “Emmett-as-a-Service,” they’d need to open source their AWS infrastructure used to run it. For normal usage—building applications—SSPL behaves exactly like AGPLv3.

The Fair-Code Movement

Fair-code describes software that is free to use, has source code available, can be modified and distributed, but includes commercial restrictions. As n8n explains, cloud providers capture value created by open source projects with little return to the original developers.

N8n created the Sustainable Use License, acknowledging that open source doesn’t mean fair. Fair-code isn’t a software license but a model where software is free to use and distribute, but commercially restricted by its authors.

We don’t want a scenario like that:

This is not fair to anyone and is highly dangerous to users. From both the service continuity and security perspectives.

The Proposal

Dual license Emmett and Pongo under AGPLv3 and SSPL. Users choose:

AGPLv3 if:

Your organization requires OSI-approved licenses,
You understand and accept network copyleft,
You want maximum compatibility with OSS tooling.

SSPL if:

You worry AGPLv3 might require open sourcing your application (it doesn’t, but SSPL makes this clearer),
You want explicit protection that using Emmett/Pongo doesn’t affect your application code,
You prefer the strongest terms.

This is explicitly pro-user. Those preferring OSI-approved licenses choose AGPLv3. Those wanting clearer terms choose SSPL. Those licences ensure that the core code of Emmett and Pongo remains open.

Implementation

License Files

There will be added License files explaining the licensing.

LICENSE-AGPL.txt (GNU AGPLv3)
LICENSE-SSPL.txt (MongoDB SSPL)
LICENSE.md explaining dual licensing and selection

Contributor License Agreement

To contribute code to Emmett or Pongo, people will need to sign a Contributor License Agreement. This ensures legal clarity and protects both contributors and maintainers. The key terms of the CLA are:

Rights Granted - contributors grant the project a license to use, modify, and distribute your contributions under both AGPLv3 and SSPL. This allows users to choose either license and ensures consistency across the codebase.
Copyright - Contributors retain full copyright over their contributions. The project does not take ownership of your work. Attribution will be maintained as required by both licenses.
Future License Options - Contributors allow the project to add additional licenses in the future, provided your contribution remains available under AGPLv3 and SSPL. This flexibility enables adaptation to changing legal or strategic needs while preserving existing license paths.
Contributor Protection - The CLA does not allow the project to relicense your work under more permissive or proprietary terms unless explicitly stated and agreed upon. The CLA applies only to the specific contributions you submit to this project.

Why the CLA Is Needed?

It ensures clear legal rights for all users of the code under both licenses.
It protects contributors from unintended license violations or uncertainty.
It supports a long-term strategy of sustainability and legal defensibility.

A link to the full CLA text will be provided in the project repository. Signing the CLA is required before pull requests can be merged. There will be CLA bot added to the GH pull request process.

Sustainability Model

Open Source Core

Emmett and Pongo core remains free under AGPLv3/SSPL. No gatekeeping for regular users.

Paid Products

To fund development and make Emmett and Pongo sustainable, I plan to provide additional paid products and services. For instance:

Advanced tooling for observability, troubleshooting and monitoring,
Web UI for project documentation, messages visualization,
Advanced multitenancy, security and other enterpise features,
Additional integrations with Cloud/AI/ML tooling.

The separation clarifies responsibilities. Open source means community-supported. Paid means I’m accountable for maintenance and support.

I also want to offer support services and paid features prioritisation. But that can come under the regular contract terms.

Personal Context

I maintained Marten for years. Despite adoption and success, I couldn’t make it sustainable. Good intentions don’t pay bills.

With Emmett and Pongo, I’ve been transparent from the start: these need to be sustainable for me to focus on them. Keeping Emmett unlicensed was deliberate, avoiding the permissive-to-restrictive transition that breaks trust.

Now, with meaningful community contributions, I’m formalising what I’ve always intended: a structure that’s fair to contributors while enabling sustainable development.

Risks and Trade-offs

Fork Risk

Cloud providers may fork rather than negotiate. Valkey shows this is real. But maintaining forks requires resources and community work. They will need at least expose it as an open source, so we could get the improvements back..

Adoption Barriers

Some organisations prohibit AGPL or SSPL. The dual approach maximises compatibility, but some potential users will be lost. Better sustainable development with narrower adoption than broad adoption of an abandoned project.

Complexity

Two licenses can create confusion. Clear documentation and decision trees help, but they add friction. That’s also why we’re doing it transparently as an RFC.

Trust

Even with transparency from the start, formalising licenses after contributions creates tension. The CLA process may deter some contributors.

FAQ

Can I choose either license for any use case?

Yes. You may choose to comply with either AGPLv3 or SSPL. You only need to comply with one license. The choice depends on your use case, compliance requirements, and comfort with the license terms.

Do I have to open source my entire application if I use Emmett or Pongo under AGPLv3?

No. AGPLv3 requires you to open source modifications to Emmett or Pongo themselves if users interact with those modified versions over a network. Your application logic, domain models, and other services remain yours. You are not required to open source your entire stack.

Can I write plug-ins or extensions to Emmett/Pongo and keep them proprietary?

Yes, if your plug-ins are separate modules that use public interfaces and do not modify the Emmett/Pongo codebase directly, you may keep them proprietary. However, if your extension modifies core internals or you redistribute a modified version, the license obligations may apply.

Why offer SSPL if AGPLv3 already provides protection?

SSPL makes the service-use restrictions more explicit and comprehensive. It removes ambiguity around what counts as a modification, and helps organisations that want stronger language around SaaS offerings without worrying about copyleft infecting unrelated application code.

What does SSPL require that AGPLv3 does not?

SSPL extends the copyleft obligation: if you offer Emmett or Pongo as a service to third parties, you must open source not only modifications to Emmett or Pongo, but all the infrastructure code used to run that service—monitoring, orchestration, authentication, deployment scripts, UI dashboards, etc. For internal use or building apps with Emmett/Pongo, SSPL behaves like AGPLv3.

What counts as a “modification” under AGPLv3 and SSPL?

A modification includes direct changes to Emmett or Pongo source code - e.g., altering the event store engine, changing how projections work, or rewriting part of the message pipeline. Plug-ins or separate modules that interact with Emmett/Pongo through stable interfaces are generally not considered modifications, but consult legal counsel if uncertain.

Is this open source?

Yes. AGPLv3 is an OSI-approved open source license. SSPL is source-available but not OSI-approved. The project remains open for use, modification, and distribution under AGPLv3.

Conclusion

This dual licensing addresses goals I’ve had from the beginning: creating sustainable development conditions while keeping projects accessible to everyone but also preventing my (and other contributors)’) work from exploitation.

Will it work? MongoDB protected itself but faces ongoing community friction. Elastic and Redis found a middle path but spawned major forks. Projects staying permissively licensed often struggle with sustainability or watch cloud providers profit from their work.

I’m choosing to formalise now what I’ve always planned, with community input. The alternative—continuing without proper licensing or choosing permissive licenses—guarantees either abandonment or exploitation.

This isn’t about getting rich or rug pull. It’s about creating conditions where these projects can survive in the long term, serving users and enabling me to get the full focus on them. This is the intended win-win for both sides.

I hope that this summary will also be helpful for you, even if you’re not planning to use Emmett or Pongo, and it will explain to you a bit more on the current OSS landscape.

You can comment either here or through the RFC published in the Emmett repository: https://github.com/event-driven-io/emmett/pull/260.

Feel also invited to join our Discord Channel if you prefer chat.

Thank you in advance!

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

Workflow Engine design proposal, tell me your thoughts

Welcome to the new week!

As you may know, I’m building Emmett - a library/framework aiming to make building business-focused apps smoother with an event-driven approach. It’s a community project, and with community projects, sharing is caring, so I’m sharing today with you an idea, hoping that you’ll also share something with me: your thoughts on it.

Summary

We’re adding a workflow processing engine to Emmett for coordinating multi-step/asynchronous business processes. It uses an event store behind the scenes to provide durable execution with full observability, treating each workflow instance as an event stream that acts as both inbox and outbox for handled messages.

Problem

Multi-step processes are everywhere in business applications, not just the typical order-fulfilment examples:

Hotel group checkout coordinating multiple room settlements,
Code review workflow with multiple approvers and checks,
Customer onboarding with verification steps,
Incident response coordination across teams,
Data ingestion with validation and enrichment stages.

Why Durable Execution Matters

These processes need to survive more than we often realise. A group checkout may be a relatively quick batch operation, but a document review workflow can span days with human approvals. An incident response could run for hours, coordinating different teams.

When things fail, and they will, we need to know exactly where we stopped and why. A network blip shouldn’t cause an in-progress checkout to be dropped in the middle. A service restart shouldn’t forget which steps it passed. A bug fix shouldn’t require manually tracking down stuck processes.

Of course, you could implement your own processing based on the existing abstractions in Emmett, like command handlers, consumers and async processors, but…

DIY solutions always have gaps. We handle the happy path, add some retry logic, then discover edge cases in production:

Messages lost during deployment
Processes stuck after partial failures
No way to resume after fixing bugs
Can’t answer “what happened to order X?”

Current Tools Work But Bring Complexity

Temporal, Restate, AWS Step Functions, and similar tools solve these problems. They guarantee processes are complete despite failures. They provide visibility into process state. But they come with cognitive overhead.

To write a simple approval workflow in Temporal, you learn about activities, workflows, workers, task queues, and their deterministic replay requirements. Your “send email for approval” becomes an activity with retry policies, timeouts, and heartbeats. Testing requires their test server, time-skipping APIs, and mock activities.

These tools require learning their entire programming model, even for simple workflows. You get powerful features, but pay the complexity cost upfront, whether you need those features or not.

What We Actually Need

We need durable execution that feels natural. Something that allows just writing regular code. Testing should be just calling functions. The patterns should match how we already think about our domain. Simple workflows should be simple to write, with complexity available when needed. Therefore, we need a model that reuses and repeats patterns, allowing for the learning ladder. This can be achieved by providing safe defaults and extension points for advanced scenarios. In other words, an accessible model enabling both newbies and advanced users.

Background: Event Sourcing as Natural Solution

What Makes Execution Durable

At its core, durable execution means three things:

Persist progress at each decision point
Resume from the last known state after failures
Know exactly what happened and why

Traditional approaches use state machines with database tables, workflow engines with proprietary storage, or complex checkpoint mechanisms. But there’s a more straightforward way.

Event Sourcing: Built-in Durability

Event sourcing stores every business decision as an immutable event. For workflows, this means every input received and output produced becomes an event in a stream. This isn’t just storage - it’s the recovery mechanism. We can expand this model and use the event store as the message store, storing all messages. This allows us to store both commands (intention to run business logic) and events (facts about what has happened, which can be used as triggers for next steps in the workflow).

Consider the following scenario:

Clerk initiating the group checkout for two guest stays, let’s say with ids 1 and 2.
We need to record that the group checkout was initiated and start regular checkout processes for each guest stay.
Then we need to gather information based on successful or failed checkouts and complete the group checkout.

With our workflow design, the message stream with each recorded message representing a step in the workflow can look as follows:

1: InitiateGroupCheckout command handled.

2: GroupCheckoutInitiated event recorded for guest stays 1 and 2.

3: CheckOut command scheduled, initiating checkout for guest stay 1.

4: CheckOut command scheduled, initiating checkout for guest stay 2.

Then, the background process can poll those messages and sequentially process them, storing the checkpoints after successful processing.

Now, we could get the next step:

5: GuestCheckedOut event received informing about successful checkout for guest stay 1.

6. Process crashes, and the second checkout was not processed.

That’s not an issue, as upon restart, we can

Read the last successful checkpoint (which happens to be 3).
Continue processing triggering Checkout for guest stay 2.
No special recovery code needed

The workflow doesn’t know it crashed. It just continues from where it left off.

Why Event Sourcing Makes Workflows Simple

Beyond recovery, event sourcing brings natural benefits to workflows:

Time Travel Debugging: When a workflow fails, replay event-by-event to see the exact decision sequence. No more “I wonder what state it was in.”

What-If Analysis: Fix a bug and replay to see if workflows would complete correctly. Test process changes against historical data.

Natural Audit Trail: Every decision is recorded with its trigger. Compliance and debugging come for free.

Error Analysis: We could also store errors as messages. See patterns across failed workflows. Understand systemic issues.

Emmett already provides these event sourcing primitives. We have event stores that handle both events and commands (message stores). We have the decide/evolve pattern for business logic. We have consumers for background processing. What’s missing is the orchestration pattern to coordinate across multiple streams with proper correlation. That’s what workflows add.

Solution Overview

Workflows in Emmett follow the same mental model as command handlers: they receive commands, make decisions based on state, and produce new messages. But unlike command handlers that only respond to commands, workflows can react to events from anywhere in your system.

The pattern has three functions:

decide: Business logic that takes input + state and returns outputs
evolve: Builds state based on events (both inputs and outputs)
initialState: Starting state for new workflow instances

Each workflow instance gets its own event stream. This stream serves as both an inbox (recording inputs) and an outbox (recording outputs). This self-contained design means everything about a workflow instance lives in one place.

This design draws huge inspiration from Yves Reynhout’s Workflow Pattern, adapting it to use Emmett’s abstractions while aiming to retain its power.

Check also:

How it Works - Technical Design

Processing Flow

When a message triggers a workflow, we use a double-hop pattern:

Input message arrives from source stream (event or command).
Router function (getWorkflowId) determines which workflow instance.
Store input in the workflow’s stream (as inbox).
Consumer delivers the stored message from the workflow stream
Rebuild state by replaying all events through evolve.
Call decide with the input and current state.
Store outputs back in the workflow stream (as outbox).
Output processor sends commands and publishes events.

This flow ensures durability at each step:

Code Example - Group Checkout

Let’s see how this works with a group checkout workflow that coordinates multiple individual checkouts:

export const GroupCheckoutWorkflow: Workflow
 GroupCheckoutInput,
 GroupCheckout,
 GroupCheckoutOutput
> = {
 decide,
 evolve,
 initialState,
};

The state tracks which guests have checked out:

export type GroupCheckout =
  | { status: 'NotExisting' }
  | {
      status: 'Pending';
      guestStayAccountIds: Map<string, GuestStayStatus>;
    }
  | { status: 'Finished' };

The decide function contains our business logic:

export const decide = (
  input: GroupCheckoutInput,
  state: GroupCheckout,
): WorkflowOutput<GroupCheckoutOutput> => {
  const { type } = input;

  switch (type) {
    case 'InitiateGroupCheckout': {
      return initiateGroupCheckout(input, state);
    }
    case 'GuestCheckedOut':
    case 'GuestCheckoutFailed': {
      return onCheckoutFinished(input, state);
    }
    case 'TimeoutGroupCheckout': {
      return timedOut(input, state);
    }
  }
};

and state evolution:

export const evolve = (
  state: GroupCheckout,
  {
    type,
    data: event,
  }: WorkflowEvent<GroupCheckoutInput | GroupCheckoutOutput>,
): GroupCheckout => {
  switch (type) {
    case 'GroupCheckoutInitiated': {
      if (state.status !== 'NotExisting') return state;

      return {
        status: 'Pending',
        guestStayAccountIds: event.guestStayAccountIds.reduce(
          (map, id) => map.set(id, GuestStayStatus.Pending),
          new Map<string, GuestStayStatus>(),
        ),
      };
    }
    case 'GuestCheckedOut':
    case 'GuestCheckoutFailed': {
      if (state.status !== 'Pending') return state;

      return {
        ...state,
        guestStayAccountIds: state.guestStayAccountIds.set(
          event.guestStayAccountId,
          type === 'GuestCheckedOut'
            ? GuestStayStatus.Completed
            : GuestStayStatus.Failed,
        ),
      };
    }
    case 'GroupCheckoutCompleted':
    case 'GroupCheckoutFailed':
    case 'GroupCheckoutTimedOut': {
      if (state.status !== 'Pending') return state;
      return {
        status: 'Finished',
      };
    }
  }
};

We register it with routing and type information:

export const groupCheckoutWorkflowProcessor = workflowProcessor({
  processorId: 'GroupCheckoutWorkflow',
  workflow: GroupCheckoutWorkflow,
  getWorkflowId: (input) => input.data.groupCheckoutId ?? null,
  inputs: {
    commands: ['InitiateGroupCheckout', 'TimeoutGroupCheckout'],
    events: ['GuestCheckedOut', 'GuestCheckoutFailed'],
  },
  outputs: {
    commands: ['CheckOut'],
    events: [
      'GroupCheckoutCompleted',
      'GroupCheckoutFailed',
      'GroupCheckoutTimedOut',
    ],
  },
});

Inputs and outputs have to be registered explicitly. We cannot detect them from types, as they’re erased on the JavaScript runtime. Still, they will have a proper intellisense and get compiler help, not allowing them to register more types than allowed.

We’re having an explicit registration for commands and events to make API explicit, and also not require additional wrapper functions in the decide method.

See the full usage sample:

Stream Structure

The workflow’s stream name comes from the getWorkflowId function. Here’s what the stream contains as the workflow executes:

Pos |  Kind   | Direction | Message
----|---------|-----------|------------------------------------------
1   | Command | Input     | InitiateGroupCheckout {groupId: '123'}
2   | Event   | Output    | GroupCheckoutInitiated
3   | Command | Output    | CheckOut {guestId: 'g1'}
4   | Command | Output    | CheckOut {guestId: 'g2'}
5   | Command | Output    | CheckOut {guestId: 'g3'}
6   | Event   | Input     | GuestCheckedOut {guestId: 'g1'}
7   | Event   | Input     | GuestCheckoutFailed {guestId: 'g2'}
8   | Event   | Input     | GuestCheckedOut {guestId: 'g3'}
9   | Event   | Output    | GroupCheckoutFailed

Notice how inputs and outputs interleave. The workflow receives inputs (positions 1, 6, 7, 8) and produces outputs (positions 2, 3, 4, 5, 9). Inputs can be either commands sent sequentially from the API, other message processing, or events recorded in regular streams.

Emmett’s event store, which is actually a message store that tracks both events and commands with their types.

This single stream contains the complete story, including what triggered each decision and what the workflow ultimately decided. No distributed tracing needed.

Why This Design

Actor Model: Self-Contained Instances

Each workflow instance acts like an actor with its own mailbox:

This isolation brings several benefits:

No shared state: Each workflow instance has its own stream
Horizontal scaling: Process many workflows in parallel
Clear boundaries: Easy to reason about each instance
Natural sharding: Distribute by workflow ID

Observable by Design

Everything about a workflow lives in one stream. Want to know why a checkout failed? Read the stream. Need to debug a stuck process? The entire history is there. This isn’t observability bolted on - it’s fundamental to how workflows work. When connected to OpenTelemetry, it should provide better visibility into business processes than regular solutions. Knowledge about abstractions like commands and events can provide durable context, not only for error debugging, but also for classical telemetry.

Durable by Default

Recovery isn’t a special case - it’s just reading the stream and rebuilding state. No checkpointing logic. The same code path handles normal processing and crash recovery. If we want to replay messages, we can just replay processing from a specific stream position.

Simple Testing

Workflows are pure functions. Testing them is straightforward:

// Unit test
const state = { status: 'Pending', guestStayAccountIds: new Map(...) };
const input = { type: 'GuestCheckedOut', data: { guestId: 'g1' } };
const outputs = decide(input, state);
// Assert expected outputs

You can test it without workflow runtime using regular tests, or built-in Emmett Behaviour-Driven Design testing capabilities, e.g.:

it('handles partial failures', () =>
  given([
    { type: 'GroupCheckoutInitiated', data: { guestIds: ['g1', 'g2'] } },
    { type: 'GuestCheckedOut', data: { guestId: 'g1' } },
  ])
    .when({
      type: 'GuestCheckoutFailed',
      data: { guestId: 'g2', reason: 'BalanceNotSettled' },
    })
    .then({
      type: 'GroupCheckoutFailed',
      data: { completedCheckouts: ['g1'], failedCheckouts: ['g2'] },
    }));

Why Emmett Over Established Tools

Familiar Patterns: If you know Emmett’s command handlers, you know workflows. Same decide/evolve pattern, just triggered by events too.

Progressive Complexity: Start with a simple workflow. Add timeout handling when needed. Add compensation if required. You don’t front-load complexity.

Lightweight Runtime: No separate workflow server. No special deployment model. Just your application with background processors.

Design Tradeoffs

Latency: The Double-Hop Cost

Storing inputs before processing adds latency. A message goes:

Source → Workflow stream (store)
Workflow stream → Processor (execute)

For background processes, this extra hop is negligible compared to the business operations. For latency-sensitive operations, use regular command handlers instead.

Future versions could make this configurable, allowing for direct processing for low-latency needs and a double-hop approach for full observability. We’d need community feedback on whether this flexibility matters.

Storage: Everything Twice

Messages exist in both source streams and workflow streams. A GuestCheckedOut event appears in:

The guest’s stream (source of truth)
The workflow’s stream (for processing)

This duplication is intentional. Different retention policies can help - keep workflow streams shorter since they’re derived data. The observability benefit outweighs the storage cost for most applications.

When Not to Use Workflows

Workflows aren’t for everything:

Simple request-response: Use command handlers.
Read models creation: Use projections.
Simple message handling: Use reactors, they’re simple enough for basic choreography,
Synchronous operations: Workflows can handle them, but for most cases, command handlers are simpler.

Workflows shine for multi-step, long-running processes that coordinate across streams.

This design is based on Yves Reynhout’s Workflow Pattern, which has proven itself in production systems. We’ve adapted it to fit naturally with Emmett’s abstractions.

Future Possibilities

Other Workflow Patterns

The current design provides a simpler alternative to traditional process managers. We could add Saga as a stateless router with compensation logic and simple event handlers for choreography.

These would build on the similar foundation.

Processing Mode Flexibility

Currently, we always store inputs first for full observability. Some workflows might prefer lower latency:

Full observability mode: Current double-hop design
Direct mode: Process immediately, only store outputs
Hybrid: Configure per message type

This needs careful design to maintain durability guarantees.

Enrichment Patterns

Workflows may sometimes need external data. How should they fetch it deterministically?

Option 1 - Pre-enrichment: Enrich at system edges before the workflow sees the message.

Option 2 - Enrichment Step: Dedicated enrichment workflows that add data.

Option 3 - Request/Response:

Workflow → FetchPrices query → Wait for Prices result.

The same can be used for external communication, and also for output enrichment/transformation.

Each has different tradeoffs for testing, performance, and complexity. We need user feedback.

Messaging System Support

Today, inputs come from event stores. Tomorrow they could also come from:

Kafka topics,
RabbitMQ queues,
SQS messages,
Supabase queues,
etc.

The workflow doesn’t care about the source. Streams remain in the event store for durability.

Error Handling Evolution

Future versions could:

Store errors as events for analysis
Configure retry policies per workflow
Dead letter handling for poison messages
Timeout escalation strategies

Paid Tools (Making Emmett Sustainable)

I want to make Emmett sustainable through optional paid tooling:

Visual Workflow Debugger: Step through workflow execution with time travel. See state at each decision point. Understand complex flows visually.

Business Process Analytics: How long do checkouts take? Where do they fail? Which steps are bottlenecks? Dashboard for business users.

State Diagram Generation: Generate visual documentation from running workflows. See all possible paths. Spot missing transitions.

Performance Analysis: Which workflows consume most resources? Where are the hot paths? Optimisation recommendations.

Documentation Generation: Turn workflow definitions into readable docs. Keep business and code in sync.

Question for the community: Would these tools provide enough value to justify payment? This would fund continued Emmett development while keeping the core open source.

Conclusion

This workflow engine brings durable execution to Emmett using patterns developers already know. It allows you to just write TypeScript code. Event sourcing provides durability and observability naturally, not as add-ons, but as fundamental properties.

We avoid the complexity of existing tools while keeping power available when needed. Simple workflows stay simple. Complex workflows become possible. Testing remains straightforward. We want to compose, avoiding vendor locking.

Together, we can build workflows that are both powerful and pleasant to use. We’re really open to your questions, suggestions and constructive criticism.

We’d also like to know if the additional paid add-ons mentioned are something you would consider paying for.

You can comment either here or through the RFC published in the Emmett repository: https://github.com/event-driven-io/emmett/pull/257.

Feel also invited to join our Discord Channel if you prefer chat.

Thank you in advance!

Cheers!

Oskar

p.s. that’s also an example of what I meant in last e-mail that I plan to share more explicitly on stuff I do.

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

The Order of Things: Why You Can't Have Both Speed and Ordering in Distributed Systems

Picture this scenario: An online retailer runs a flash sale with 500 units of a hot product. Within minutes, they’ve processed 742 orders. Their inventory shows -242 units. Customer service is about to have a very bad day.

This isn’t a story about bad code. The engineers did everything right: database transactions, inventory checks, proper error handling. They discovered that in distributed systems, you must choose between global ordering and performance.

When you have multiple servers processing requests simultaneously, “check inventory then charge customer” becomes a race condition. The milliseconds between “check” and “charge” are windows where other orders slip through.

You might think stronger locks could fix this. They could, while destroying your system’s performance and/or its availability. Let’s discuss today why.

PostgreSQL: When Correctness Is King

Let’s start with PostgreSQL, the database that prioritises correctness above all else. Consider this seemingly bulletproof code that could be generated by the ORM used in application:

BEGIN;

SELECT quantity FROM inventory WHERE product_id = 'console-1';

--- ⌛ Application checks if quantity > 0

INSERT INTO orders (user_id, product_id, amount) VALUES (123, 'console-1', 599.99);

UPDATE inventory SET quantity = quantity - 1 WHERE product_id = 'console-1';

COMMIT;

With one server, this works perfectly. With ten servers hitting the same database? You’ve got a problem.

The issue is that BEGIN doesn’t mean “stop the world while I work.” It means “group these operations together.” Between your SELECT and UPDATE, nine other transactions can read the same inventory count. All ten transactions see “5 units available.” All ten proceed with the sale.

The Locking Solution

PostgreSQL offers a solution: SELECT FOR UPDATE. This actually locks the row:

BEGIN;

SELECT quantity FROM inventory WHERE product_id = 'console-1' FOR UPDATE;

--- ⛔ Now other transactions must wait
INSERT INTO orders (user_id, product_id, amount) VALUES (123, 'console-1', 599.99);

UPDATE inventory SET quantity = quantity - 1 WHERE product_id = 'console-1';

COMMIT;

But what actually happens inside PostgreSQL when you use FOR UPDATE? Let’s look at the mechanics.

When Transaction A executes SELECT FOR UPDATE, PostgreSQL:

Acquires a RowExclusiveLock on the row with product_id = ‘console-1’
Records this lock in the lock manager
Other transactions attempting to read with FOR UPDATE or write must wait

You can see this in action:

-- ✅ Session 1
BEGIN;

SELECT * FROM inventory WHERE product_id = 'console-1' FOR UPDATE;

-- ⛔ Session 2 (in another connection)

SELECT pid, relation::regclass, mode, granted
FROM pg_locks
WHERE relation = 'inventory'::regclass;

-- Output shows something close to:
-- pid  | relation  | mode              | granted
-- 1234 | inventory | RowExclusiveLock  | t
-- 5678 | inventory | RowExclusiveLock  | f  -- waiting!

The waiting transaction appears in pg_stat_activity:

SELECT pid, wait_event_type, wait_event, query
FROM pg_stat_activity
WHERE wait_event IS NOT NULL;

-- Shows: wait_event_type = 'Lock', wait_event = 'tuple'

This creates a queue. If 100 concurrent users attempt to buy the same product, 99 wait while one is processed. The 100th user might wait several seconds—an eternity in web applications.

The performance impact is severe. During high traffic, you might process 50 orders per second instead of 1,000 per second. Every popular product becomes a serialisation point for your entire system.

For more details on PostgreSQL locking, see the official documentation on explicit locking.

The Sequence Problem

As I explored in my article about PostgreSQL sequences and their quirks, even the basic building blocks of ordering have surprises. PostgreSQL sequences—used for generating IDs—can have gaps even when all transactions succeed. Why? Because sequences grab their values before knowing if the transaction will commit.

Consider this race:

Transaction A starts at 10:00:00.000, gets sequence value 1
Transaction B starts at 10:00:00.001, gets sequence value 2
Transaction B commits at 10:00:00.100
Transaction A commits at 10:00:00.200

Your records appear “out of order” relative to their IDs. Record 2 exists before record 1 in the commit order. This becomes critical when you’re using sequences for ordering in patterns like the Outbox pattern.

The problem runs deeper. PostgreSQL allocates transaction IDs (XIDs) from a global counter. When combined with MVCC (Multi-Version Concurrency Control), this creates interesting effects:

-- Transaction XIDs determine visibility
SELECT xmin, xmax, * FROM inventory WHERE product_id = 'console-1';
-- xmin: transaction that created this row version
-- xmax: transaction that deleted/updated this row version

A row is visible in your transaction if:

Its xmin is committed and < your transaction’s XID
Its xmax is either null or > your transaction’s XID

This is why PostgreSQL can show you different data than another concurrent transaction—each sees a different “snapshot” of the database based on transaction ordering. Even PostgreSQL makes trade-offs between strict ordering and performance.

The Replication Lag

The problems compound with replication. Your primary database knows the correct order. But replicas apply changes asynchronously. A customer might:

Place an order (writes to the primary instance)
Check their order history (reads from replica)
See nothing—the replica hasn’t caught up

You can force reads from the primary, but now you’ve eliminated the performance benefit of having replicas.

MongoDB: The Speed Trap

MongoDB takes a different philosophy: speed first, consistency when you ask for it.

Here’s our inventory check in MongoDB:

const product = await db.products.findOne({ _id: 'console-1' });
if (product.quantity > 0) {
  await db.orders.insertOne({ userId: 123, productId: 'console-1', amount: 599.99 });
  await db.products.updateOne(
    { _id: 'console-1' },
    { $inc: { quantity: -1 } }
  );
}

This has the same race condition as our PostgreSQL example, but MongoDB makes it worse through its replica set architecture.

How MongoDB Replication Actually Works

MongoDB uses replica sets where one primary accepts writes and secondaries asynchronously replicate from the primary’s oplog (operations log). As I explained in the Write-Ahead Log article, the oplog is MongoDB’s version of a write-ahead log:

// Actual oplog entry structure
{
  "ts": Timestamp(1626900000, 1),     // Operation timestamp
  "t": NumberLong(10),                 // Term (for elections)
  "h": NumberLong(0),                  // Hash
  "v": 2,                              // Version
  "op": "u",                           // Operation type (update)
  "ns": "shop.products",               // Namespace
  "ui": UUID("..."),                   // Collection UUID
  "o": { "$v": 1, "$set": { "quantity": 4 } },  // Operation
  "o2": { "_id": "console-1" }         // Query
}

By default, MongoDB reads might hit different replica set members. The primary might have a quantity of 5, while the secondary still shows a quantity of 10. Your application checks inventory on the secondary (seeing 10), while another server updates the primary (setting it to 0).

You can see the replication lag yourself:

db.adminCommand({ replSetGetStatus: 1 }).members.forEach(member => {
  console.log(`${member.name}: ${member.optime.ts.getTime()}`);
});
// Primary:    Timestamp(1626900000, 1)
// Secondary1: Timestamp(1626899995, 1)  -- 5 seconds behind!

For deep details on MongoDB replication mechanics, see the MongoDB Architecture Guide.

MongoDB’s Atomic Operations

MongoDB offers findOneAndUpdate for atomic operations:

const result = await db.products.findOneAndUpdate(
  { _id: 'console-1', quantity: { $gt: 0 } },
  { $inc: { quantity: -1 } },
  { returnDocument: 'after' }
);

if (result.value) {
  // Inventory decreased successfully, proceed with order
  await db.orders.insertOne({ userId: 123, productId: 'console-1', amount: 599.99 });
}

This is atomic for the inventory update. But now you have a new problem: what if the order insert fails? You’ve decremented inventory but have no order record.

The Session Solution

MongoDB 4.0 added multi-document transactions:

const session = await client.startSession();

await session.withTransaction(async () => {
  const product = await db.products.findOne(
    { _id: 'console-1' },
    { session }
  );

  if (product.quantity > 0) {
    await db.products.updateOne(
      { _id: 'console-1' },
      { $inc: { quantity: -1 } },
      { session }
    );

    await db.orders.insertOne(
      { userId: 123, productId: 'console-1', amount: 599.99 },
      { session }
    );
  }
});

You’ve just made MongoDB behave like PostgreSQL, inheriting PostgreSQL’s performance characteristics. MongoDB’s own documentation warns that transactions impact performance—in benchmarks, throughput drops by 50-60% with transactions enabled.

Why? Because MongoDB must now coordinate:

Acquire locks across multiple documents
Ensure all replica set members can see a consistent snapshot
Maintain undo logs for potential rollback
Block concurrent operations on the same documents

The MongoDB transactions documentation explicitly states: “In most cases, multi-document transaction incurs a greater performance cost over single document writes.”

The Revision Pattern

In my article about handling projections in eventually consistent stores, I explored another challenge: eventual consistency in reads. When you write to MongoDB and immediately read, you might get stale data or even null results.

The solution involves tracking revisions and retrying until you see the expected state:

async function retryIfNotFound(find, options = { maxRetries: 5, delay: 100 }) {
  let retryCount = 0;

  do {
    try {
      const result = await find();
      if (result !== null) return result;

      if (retryCount === options.maxRetries) {
        throw new Error('Document not found after max retries');
      }

      const sleepTime = Math.pow(2, retryCount) * options.delay;
      await sleep(sleepTime);
      retryCount++;
    } catch (error) {
      throw error;
    }
  } while (true);
}

Usage with revision tracking:

const { productItems, revision } = await retryIfNotFound(() =>
  shoppingCarts.findOne(
    {
      shoppingCartId: event.data.shoppingCartId,
      revision: { $gte: expectedRevision },
    },
    { projection: { productItems: 1, revision: 1 } }
  )
);

This pattern, retry with exponential backoff until you see the version you expect—works. Normally, one call should be enough, but sometimes it can take multiple retries to get it.

Such an approach improves the issue but adds complexity and may increase latency. You’re fighting MongoDB’s natural behaviour instead of working with it. Each retry might take hundreds of milliseconds, turning a simple read into a multi-second operation under load.

Kafka: Distributed by Design

Kafka takes yet another approach: it embraces distribution but partitions the problem.

As I explained in The Write-Ahead Log article, Kafka is built entirely around the concept of logs. Topics provide logical grouping, while partitions are the physical append-only logs. Each partition guarantees ordering, but there’s no global order across partitions.

In Kafka, you might model our inventory system as events. Order service publishes event:

producer.send({
  topic: 'orders',
  key: productId, // Ensures all events for a product go to same partition
  value: {
    type: 'OrderPlaced',
    userId: 123,
    productId: 'console-1',
    amount: 599.99
  }
});

The Inventory service consumes and updates

consumer.on('message', async (message) => {
  if (message.value.type === 'OrderPlaced') {
    const inventory = await getInventory(message.value.productId);
    if (inventory > 0) {
      await decrementInventory(message.value.productId);
      await publishInventoryReserved(message.value);
    } else {
      await publishOrderRejected(message.value);
    }
  }
});

By using the product ID as the partition key, Kafka guarantees all events for a product are processed in order. But this only works within a single partition.

The Distributed Transaction Problem

What happens when a customer orders two different products in one checkout?

Order contains: Product A and Product B
Event 1: “Reserve inventory for A” → partition 1 (based on hash of “product-A”)
Event 2: “Reserve inventory for B” → partition 7 (based on hash of “product-B”)
Event 3: “Process payment for order” → partition 3 (based on hash of “order-123”)

These events may be processed in any order across partitions. To understand why, look at how Kafka consumer groups work:

// Consumer group with 10 instances, 10 partitions
// Each consumer gets 1 partition
Consumer 1: Reading partition 0 at offset 1000
Consumer 2: Reading partition 1 at offset 5000
Consumer 3: Reading partition 2 at offset 2000
// ... and so on

Each consumer processes its partition independently at its own pace. Partition 1 might be at offset 5000 while partition 7 is still at offset 100. Your payment might process before inventory is even checked.

Kafka provides ordering within partitions through its ISR (In-Sync Replica) mechanism:

# Check partition details
kafka-topics --describe --topic orders --bootstrap-server localhost:9092

# Output shows:
# Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
# Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1

The ISR ensures all in-sync replicas have the same messages in the same order. But this is per partition—there’s no coordination between partitions.

For more details, see Kafka’s design documentation.

Plus even if you have events on the same partition, then between publishing message and handling it there is always a latency. So data may not be immediately in the end database, and cause discrepancies. Kafka doesn’t support optimistic concurrency.

That’s why Kafka should not be treated as a database (nor as an event store). See more:

The Saga Pattern

When developers reach these ordering limits, they often turn to the Saga pattern. As I explained in Saga and Process Manager - distributed processes in practice, Saga doesn’t solve the ordering problem—it accepts that you can’t have distributed transactions and builds compensations around that fact.

The typical solution models the process as a series of compensating transactions:

Reserve Product A (can be undone by releasing the reservation)
Reserve Product B (can be undone by releasing the reservation)
Process Payment (can be undone by refund)
Confirm Reservations (convert to permanent inventory decrease)

But this creates new problems. You’re now managing distributed state machines across multiple services. Each service maintains its own state, for instance:

InventoryService: { “product-A”: “reserved”, “product-B”: “available” }
PaymentService: { “order-123”: “pending” }
OrderService: { “order-123”: “awaiting-payment” }

States can become inconsistent between services. What if payment succeeds but the inventory times out? What if a network partition happens during compensation?

Your simple inventory check has become a complex choreography of events, timeouts, and compensations. You haven’t eliminated the distributed transaction problem - you’ve just made it explicit in your application code. Which doesn’t have to be that bad, but introducing it in a simple cases can add latency. Each step requires network calls, persistence, and coordination. You’ve traded database locks for distributed complexity.

Kafka’s Performance Trade-offs

Even within single-partition ordering, Kafka makes trade-offs:

Without a built-in idempotence on producer, producing is faster, but can create duplicates on retry

const producer = new Kafka.Producer({
  'client.id': 'inventory-service',
  'metadata.broker.list': 'localhost:9092'
});

With an idempotent producer, we won’t have duplicates, but it’ll be slower

// 20% performance hit
const producer = new Kafka.Producer({
  'client.id': 'inventory-service',
  'metadata.broker.list': 'localhost:9092',
  'enable.idempotence': true,  // Prevents duplicates
  'max.in.flight.requests.per.connection': 5  // Must be ≤ 5
});

And it will be even slower with Kafka “transactions”:

const producer = new Kafka.Producer({
  'client.id': 'inventory-service',
  'metadata.broker.list': 'localhost:9092',
  'transactional.id': 'inventory-tx-1'  // Enables transactions
});

await producer.initTransactions();
await producer.beginTransaction();
await producer.produce({ topic: 'orders', messages: [...] });
await producer.commitTransaction();

Why such performance penalties? With transactions, Kafka must:

Coordinate with a transaction coordinator
Write markers to the log
Ensure all partitions involved commit atomically
Maintain transaction state across failures

And remember—this ordering only works within partitions. For global ordering, you’d need a single partition, which means single-threaded, sequential processing, one message after another. Your distributed system just became a very expensive single-threaded application. At that point, you might as well use PostgreSQL.

The truth we try to hide from ourselves

After exploring these three systems, a pattern emerges. Each makes different trade-offs, but they’re all making trade-offs:

PostgreSQL chose correctness. You pay with lock contention and reduced throughput.
MongoDB choose speed. You pay with application complexity and eventual consistency.
Kafka chose scalability. You pay with architectural complexity and partial ordering.

There’s no magic configuration or hidden feature that gives you everything. The trade-offs are fundamental.

Why Is Ordering So Expensive?

In a single-threaded application, ordering is free. Operations happen one after another. Period.

In a distributed system, establishing “after” requires coordination. Consider what happens at the network level:

Server A must send a message: “I did X” (minimum 1ms in the same datacenter)
Server B must acknowledge: “I heard you” (another 1ms)
Only then can Server B safely do Y “after” X

But it’s worse with multiple participants:

3 servers need 3 message exchanges (A→B, B→C, C→A)
10 servers need 45 message exchanges (n*(n-1)/2)
100 servers need 4,950 message exchanges

Each message exchange is at least:

~0.5ms in the same datacenter
~50ms across data centres in the same region
~150ms across continents

For comparison:

Reading from CPU cache: 0.0000001ms
Reading from RAM: 0.0001ms
Reading from SSD: 0.01ms

Network coordination is 5,000 to 1,500,000 times slower than local operations. This is physics, not poor implementation.

The Two-Phase Commit Trap

You might think: “Use two-phase commit! Coordinate across databases!”

Two-phase commit (2PC) does guarantee consistency across systems. Here’s the protocol:

Phase 1 - Voting:

Coordinator → Participant A: "PREPARE transaction-123"
Coordinator → Participant B: "PREPARE transaction-123"
Participant A → Coordinator: "READY" (locks resources)
Participant B → Coordinator: "READY" (locks resources)

Phase 2 - Completion:

Coordinator → Participant A: "COMMIT transaction-123"
Coordinator → Participant B: "COMMIT transaction-123"
Participant A → Coordinator: "DONE" (releases locks)
Participant B → Coordinator: "DONE" (releases locks)

The killer problem: those locks are held during network round-trip times. With just 2 participants:

2 round trips minimum
At 1ms per message: 4ms holding locks
At 50ms per message: 200ms holding locks

But the real problem is failure handling. If the coordinator crashes after Phase 1:

Participants have locked resources
They don’t know whether to commit or abort
They must keep the locks until the coordinator recovers
This could be forever

This is called the “blocking problem” of 2PC. Your entire system stops because one component fails. Modern systems avoid 2PC for this reason. Even with perfect networks, the performance cost is prohibitive.

Google Spanner uses atomic clocks and GPS synchronisation to reduce coordination needs, and it still has uncertainty windows of 7ms. Amazon DynamoDB uses vector clocks and still requires application-level conflict resolution. If companies with unlimited resources can’t eliminate these trade-offs, the trade-offs are fundamental to distributed systems.

Making Peace with Physics

So what do you do? You can’t beat physics, but you can work with it.

Pattern 1: Reduce Coordination Scope

Instead of coordinating everything, coordinate only what matters. In our inventory example, don’t coordinate this:

await checkUserAuthentication();
await validateShippingAddress();
await calculateTaxes();

But focus on the critical business check:

await atomicallyDecrementInventory();

Most operations don’t need global ordering. Identify the few that do and design around them.

Pattern 2: Business-Level Solutions

Sometimes the best solution isn’t technical. Amazon shows “Only 3 left in stock!” not just for marketing—it’s also setting expectations. If they occasionally oversell by one unit, customers understand.

Pattern 3: Choose Your Guarantees

Not all operations need the same guarantees:

Financial transactions: Use a database with proper locking (e.g. PostgreSQL). Speed matters less than correctness.

Product browsing: You can use a database with eventual consistency (MongoDB, ElasticSearch). Speed and feature set may matter more than perfect accuracy.

Analytics events: Use Kafka with at-least-once delivery. Order within user sessions is enough.

Pattern 4: Embrace Idempotency

Make operations safe to retry:

// 👎 Bad: Changes meaning if applied twice
UPDATE inventory SET quantity = quantity - 1;

// 👍 Good: Same result if applied multiple times
UPDATE inventory SET quantity = 5 WHERE version = 42;

With idempotent operations, you can retry without fear, compensate for reordering, and recover from partial failures.

Conclusion: Pick Your Poison

Every distributed system makes trade-offs between ordering and performance. There’s no configuration flag or architectural pattern that eliminates this trade-off—only different ways to manage it.

PostgreSQL gives you correctness but makes you pay in lock contention. MongoDB gives you speed but makes you handle consistency yourself. Kafka gives you scale but requires complex coordination patterns.

The key is understanding which trade-off you’re making and designing accordingly. Don’t fight your database’s natural behaviour—work with it.

Next time you see eventual consistent reads in MongoDB, you’ll understand why. When PostgreSQL seems slow under load, you’ll know what you’re paying for. When Kafka requires complex event choreography, you’ll see it as the price of distribution.

The speed of light isn’t just a suggestion. Network calls aren’t free. Coordination has a cost. Once you accept these constraints, you can stop looking for silver bullets and start making informed trade-offs.

The question isn’t “How do I get both?” The question is “What can I afford to give up?”

Cheers!

Oskar

Dealing with Eventual Consistency, and Causal Consistency using Predictable Identifiers

Users want to attach files and keep working. Systems need time to upload, store those files and link it to the specific process or entity.

You have two choices: block the interface during uploads, or accept temporary inconsistency.

I learned this while building a construction documentation system. Users attached blueprints to tasks, photos to safety reports, and contracts to milestones. Construction files can be pretty big, so we didn’t want users to have to wait for the upload and can’t do any tasks during that. We needed something better.

The previous article showed how predictable ids can help in having module autonomy. We discussed URNs (Unique Resource Names) and took the payments module as an example. We used them to generate upfront correlation between other modules and payments. Thanks to that, we inverted the dependency. The payment module could expose a generic API without being aware of the other modules’ existence. Payments just had to know about the correlation id generated upfront and forward it through its own operations.

Document management faces similar coupling challenges, as well as concerns regarding network reliability and user experience.

Let’s follow up and discuss today how predictable ids can also help in solving eventual consistency challenges.

Eventual Consistency vs Causal Consistency

Most of the time, we discuss strong consistency and eventual consistency.

Most systems prefer to have strong consistency. You perform the operation, wait till it’s finished and then proceed. We could handle uploads synchronously. User selects file → browser uploads to server → server stores file → server creates database record → server returns success. Simple, consistent, slow.

This creates problems:

Large files block the UI for minutes
Network interruptions force complete restarts
Servers become bottlenecks, streaming every byte through memory
Multi-gigabyte files can crash servers

Coupling user experience with implementation details can cause problems. Users don’t care about moving bytes across networks. They selected the file - to them, it’s attached.

Luckily, we learned that sometimes we have to live with an additional delay. Still, we should not stop that, as it’s a bit trickier than “a small delay”.

We usually assume that stuff will happen in a particular order, we just don’t know precisely when. And that’s actually something that’s called Causal Consistency. The difference between Eventual and Causal consistency is:

Eventual consistency - The system eventually reaches a consistent state. Order of operations doesn’t matter. Create a link, then upload the file, or upload the file, then create the link - the result is the same. The system tolerates temporary inconsistency.

Causal consistency - Operations must respect cause and effect. You can’t comment on a document before it exists. Effects follow causes.

Source: https://x.com/gregyoung/status/1101642600342265857

File uploads fit eventual consistency perfectly. When attaching a blueprint to a construction task, two things happen: storing the file and linking it to the task. The order is irrelevant. What matters is that both are completed eventually.

With eventual consistency for file uploads, we can show files as “attached” immediately. The actual upload happens in the background. Of course, we won’t be able to download them until the upload is finished, but the system continues working during the brief inconsistency.

The URN Solution: Embracing Eventual Consistency

In our system, we decided to separate cloud storage from the actual application behaviour. Our files module was a generic one, similar to the discussed payments.

The reasoning, as in construction projects, is that you may have multiple places where someone can upload documentation, scans, and photos. We wouldn’t like to couple everything together. We just let specific modules to take care of linking files to their processes.

What’s more, we didn’t even want to have a direct relationship on the backend between files and the business module. File URNs were generated on the UI before uploading. The UI knows the context in which the file will reside before transferring bytes.

When users select a file, we could generate a URN immediately:

urn:files:1:CONSTRUCTION:SAFETY_CHECK:DAILY_INSPECTION:site-photo.jpg

Structure breakdown:

files - namespace for documents
1 - version for format evolution
CONSTRUCTION - initiating module
SAFETY_CHECK - business process
DAILY_INSPECTION - subprocess
site-photo.jpg - filename

The URN structure also helps detect duplicates. The user won’t be able to upload the same file twice, as the key is unique.

Ok, but how will we know where to upload the file when we just know some magic URN?

Pre-Signed URLs: Direct Client-to-Storage Uploads

Modern cloud storage providers offer pre-signed URLs, which are time-limited URLs that allow direct uploads without requiring permanent credentials. Your server generates these URLs on demand, then hands them to the client. The client uploads directly to S3, Azure Blob Storage, or similar services.

But why are pre-signed URLs better than traditional browser uploads? With traditional uploads, your browser establishes a single HTTP connection to your server. If that connection drops - and on mobile networks, it will - the entire upload fails. The browser can’t resume; it must start from scratch. Your server, meanwhile, has been holding that partial upload in memory, consuming resources for nothing. With Cloud, that can mean huge costs, as you’re paying for each byte transferred outside your network.

Pre-signed URLs shift this burden to battle-tested cloud infrastructure. S3 presigned URLs allow anyone with valid security credentials to create a URL that grants temporary access to upload an object. The URL is limited by the permissions of the user who created it. They provide:

Automatic retry logic: The storage provider retries failed chunks without restarting.
Resumable uploads: Large files can be uploaded using the multipart upload API.
Global edge locations: Files are uploaded to the nearest datacenter, reducing latency.
Bandwidth optimisation: Cloud providers have better peering agreements than your servers.

Are they secure? That’s, of course, depends on your requirements.

Following the dumb pipes, smart endpoints principle, we’re treating the resigned URL as a dumb pipe. Our endpoint should be smart enough to ensure that we authenticate and authorise users to perform an operation.

For S3, generating a pre-signed URL requires only the target key and expiration. AWS documentation notes that: “When you create a presigned URL, you must provide your security credentials and specify a bucket name, an object key, an HTTP method (PUT for uploading objects), and an expiration date and time.”

The presigned URLs are valid only for the specified duration:

async generatePresignedUrl(key: string): Promise<string> {
  const command = new PutObjectCommand({
    Bucket: this.bucket,
    Key: key
  });

  // URL expires in 1 hour
  return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 });
}

Microsoft Graph API offers similar capabilities through upload sessions. According to Microsoft’s documentation: “Create an upload session to allow your app to upload files up to the maximum file size. An upload session allows your app to upload ranges of the file in sequential API requests. Upload sessions also allow the transfer to resume if a connection is dropped while the upload is in progress.” The implementation follows a similar pattern:

async createUploadSession(path: string): Promise<UploadSession> {
  const response = await this.graphClient
    .api(`/drive/root:/${path}:/createUploadSession`)
    .post({
      item: {
        "@microsoft.graph.conflictBehavior": "rename"
      }
    });

  return {
    uploadUrl: response.uploadUrl,
    expirationDateTime: response.expirationDateTime
  };
}

This pattern removes re-streaming through your servers. We also benefit from a built-in storage provider’s replication. A user in Singapore uploading a 100MB video sends data to the nearest AWS edge location, not to your servers in Europe. Your server handles business logic instead of moving bytes around.

But now you can’t track upload progress. So again, how do you know when the file arrives?

The “Magic Folder” Pattern

We took inspiration from Microsoft Graph API “Special Folders”. It provides a sneaky pattern for addressing files by path. Per docs:

“Special folders provide simple aliases to access well-known folders in OneDrive without the need to look up the folder by path (which would require localization), or reference the folder with an ID. If a special folder is renamed or moved to another location within the drive, this syntax will continue to find that folder.”

What are those special folders? Known from Windows folders like Documents, Recordings, Photos etc.

Thanks to understanding the URN structure, we could design specific modules with separate paths to drives or buckets, simply mapping them logically in the backend to specific buckets. This could be either full conventional-based, or stored in a static config or database.

No need to pre-create folder hierarchies.
Files automatically organise by business context.
Migration preserves logical structure.
etc.

When combined with our URN approach, this creates a powerful system. A URN like urn:files:1:CONSTRUCTION:SAFETY_CHECK:DAILY_INSPECTION:site-photo.jpg maps directly to a storage path: files/v1/construction/safety_check/daily_inspection/site-photo.jpg. The storage provider creates any missing folders automatically, just like Microsoft’s “magic folder” behavior.

In S3, this works because S3 doesn’t actually have folders - it uses key prefixes that look like paths. When you upload to files/v1/construction/safety_check/daily_inspection/site-photo.jpg S3 treats the forward slashes as part of the object key. The AWS console displays these as folders for convenience, but they’re just a naming convention. This flat structure with hierarchical naming gives us the benefits of organisation without the overhead of managing actual folders.

Instead of explicit registration:

Generate URN
Upload to the corresponding path
The system finds a file through URN pattern matching

Modules query for files matching their patterns:

class TaskModule {
  async getTaskWithAttachments(taskId: string): Promise<TaskWithFiles> {
    const task = await this.db.getTask(taskId);

    // Find all document links for this task
    const pattern = `urn:files:*:TASKS:*:${taskId}:*`;
    const links = await this.db.findDocumentLinks(pattern);

    // Return task with attachment metadata
    return {
      ...task,
      attachments: links.map(link => ({
        name: this.extractFileName(link.urn),
        status: link.status,
        uploadedAt: link.uploadedAt,
        downloadUrl: link.status === 'uploaded'
          ? this.generateDownloadUrl(link.urn)
          : null
      }))
    };
  }
}

The task module doesn’t need to know about the upload process. It queries for documents matching its pattern. Whether those documents are still uploading, have been completed, or have failed, the module displays the appropriate status to users.

The “magic” works through convention. Upload the file to the correct path, and it will become visible to the relevant modules.

Composition Through UI and Process-Based APIs

These patterns work well together. The UI generates URNs based on user context. Process-based APIs accept these URNs without knowing their origin. Storage services handle files based on URN patterns alone.

But what is task-based UI? Traditional CRUD interfaces are organised around data models: “Create Document”, “Edit Document”, and “Delete Document”. Task-based UIs organise around user goals: “Submit Safety Report”, “Approve Change Order”, “Complete Inspection”. Each interface shows only the actions relevant to that specific task.

This design provides the context needed for URN generation. When a safety inspector uploads a photo, the UI knows they’re performing a safety inspection. It knows which site, which inspection type, and which date.

The flow works like this. We need a function to generate a derived URN. It can be either as dumb as this, or more likely a bit smarter:

// Client-side URN generation
function generateFileUrn(
    module: string,
    process: string,
    subprocess: string,
    fileName: string
  ): string {

  const safeName = fileName.replace(/[^a-zA-Z0-9.-]/g, '_');
  return `urn:files:1:${module}:${process}:${subprocess}:${safeName}`;
}

Nevertheless, how creative we are in the mapping then we could use it as follows:

async function createRemediationTask(violationId: string, contractorId: string, file: File) {
  const urn = generateFileUrn('CONSTRUCTION', 'SAFETY_CHECK', 'VIOLATION_${violationId}', file);

  // Immediately show the file as attached
  showFileAsAttached(urn, file.name);

  // Request upload URL from backend
  const { uploadUrl } = await api.requestUploadUrl(urn);

  // Create remediation task with photo reference
  await api.createRemediationTask({
    violationId,
    assignedTo: contractorId,
    attachments: [urn],
    dueDate: calculateDueDate(severity)
  });

  // Upload in background
  uploadInBackground(file, uploadUrl);
}

The backend tracks these uploads using link metadata:

class DocumentService {
  async requestUploadUrl(urn: string, context: Context): Promise<{ uploadUrl: string }> {
    // Generate pre-signed URL using URN as the storage key
    const storageKey = this.urnToStorageKey(urn);

    // verify persmissions
    await hasPermissions(storageKey, context.user);

    const uploadUrl = await this.storage.generatePresignedUrl(storageKey);

    return { uploadUrl };
  }
}

Each component has one job. The UI handles user interaction and URN generation. The API manages business logic without dealing with file storage. The storage service handles bytes and pre-signed URLs. Clean separation of concerns through predictable identifiers.

Cleanup and Validation

Accepting eventual consistency means accepting temporary inconsistency. But “temporary” needs boundaries. Sometimes just accepting uploads as they go is not enough. We may need an audit around it, and simply using path traversal as we did is not sufficient for you. Or you’d like to see also pending uploads, or perform an evaluation of file content before showing it to users (e.g., an anti-virus scan).

We could also keep an explicit track of document links together with metadata, store them in the database when we’re initiating, and upload metadata.

interface DocumentLink {
  urn: string;
  uploadedAt?: Date;
  fileSize?: number;
  contentType?: string;
  status: 'pending' | 'uploaded' | 'failed';
  createdAt: Date;
  expiresAt?: Date;
}

We could either subscribe to the storage notifications (if they give us such) or just run a background service that validates uploads and cleans up failures:

class DocumentValidationService {
  async validatePendingUploads(): Promise<void> {
    // Find uploads pending for more than 15 minutes
    const staleLinks = await this.db.query(`
      SELECT * FROM document_links
      WHERE status = 'pending'
      AND created_at < NOW() - INTERVAL '15 minutes'
    `);

    for (const link of staleLinks) {
      const storageKey = this.urnToStorageKey(link.urn);
      const exists = await this.storage.checkExists(storageKey);

      if (exists) {
        // File uploaded successfully, update metadata
        const metadata = await this.storage.getMetadata(storageKey);
        await this.db.update('document_links', {
          urn: link.urn,
          status: 'uploaded',
          uploadedAt: metadata.lastModified,
          fileSize: metadata.size,
          contentType: metadata.contentType
        });
      } else if (link.expiresAt < new Date()) {
        // Upload expired, mark as failed
        await this.db.update('document_links', {
          urn: link.urn,
          status: 'failed'
        });
      }
    }
  }
}

The periodic job serves three critical functions:

1. Upload Status Reconciliation: Files might upload successfully but the client fails to notify the server. Network issues, browser crashes, or users closing tabs can interrupt the completion callback. The reconciliation job discovers these “orphaned successes” and updates their status.

2. Dead Link Detection: Users sometimes delete files directly from cloud storage. External tools, manual cleanup, or storage lifecycle policies can remove files the system expects to exist. The job detects these deletions and marks links accordingly, preventing 404 errors when users try to download.

3. Database Hygiene: Failed uploads and deleted file records accumulate over time. The job purges old records (typically after 30 days) to prevent unbounded table growth. This matters at scale - millions of uploads can generate significant metadata that serves no purpose after a reasonable retention period.

The timing of these jobs requires careful consideration. Too frequently, and you waste resources checking files that rarely change. Too infrequently, and users encounter stale data. Most systems find 5-15 minute intervals work well. Critical systems might run more frequently with smart batching to limit resource usage.

We could also trigger a follow-up operation in through such job, notifying the outside world about the upload failure or completion.

Provider Independence Through URNs

Last but not least: URNs hide storage details. The URN urn:files:1:CONSTRUCTION:SAFETY_CHECK:DAILY_INSPECTION:site-photo.jpg works whether files live in S3, Azure, or on-premises storage. For a testing environment, we could use a regular disk location.

Migration becomes straightforward. We just need to move files and provide the mapping function. Once the migration is made in the files module, we can just resolve them to a different place. The change will be opaque for other modules. The URN abstraction separates the system from storage details.

Conclusion

The final flow looks as follows:

Choosing eventual consistency for file uploads improves user experience. Users attach files and continue working while uploads happen in the background. Perfect consistency would force them to wait.

Pre-signed URLs and predictable URNs make this possible. Files are uploaded directly to storage while the system tracks progress through link metadata. URNs provide routing and correlation without coupling. The system stays functional during the brief inconsistency window.

As we saw with the payment module in the previous article, predictable identifiers change how modules communicate. For file uploads, they also change how we handle consistency. Instead of forcing synchronous behaviour on an asynchronous process, we work with the distributed nature of the system. The architecture scales with load rather than against it.

Working with system constraints beats fighting them. Eventual consistency is an example of this.

Cheers!

Oskar

Predictable Identifiers: Enabling True Module Autonomy in Distributed Systems

In distributed systems, we face a fundamental tension between module communication and module autonomy. Our modules need to exchange information and coordinate actions, yet we want each module to evolve independently without forcing changes across the entire system. This tension becomes evident in event-driven architectures, where modules communicate through events and messages.

Consider a typical e-commerce platform. The payment module needs to process requests from orders, handle reimbursements, manage subscriptions, and potentially deal with dozens of other payment scenarios. Each connection point between modules represents a potential coupling that can ripple through the system when requirements change.

The traditional approach involves modules directly referencing each other through event types, API endpoints, or shared data structures. This creates a web of dependencies where adding a new module or changing an existing one requires coordinated updates across multiple teams.

What if we could enable modules to communicate without knowing about each other’s existence? What if a payment module could process requests from any source without being programmed to handle specific scenarios?

We’ll explore today how predictable identifiers, specifically Uniform Resource Names (URNs), provide an elegant solution to this challenge.

By structuring identifiers in a predictable format, we enable infrastructure-level routing and correlation without module-specific logic. This approach transforms tightly coupled systems into collections of autonomous services that communicate through well-defined patterns rather than explicit dependencies.

Why Generic Modules Matter

Payment processing in modern applications rarely involves direct credit card handling. Instead, we integrate with payment service providers like Stripe, PayPal, Square, etc. Each provider offers different APIs, webhook patterns, and state machines. A payment module serves as an abstraction layer, wrapping these external services to present a unified interface to the rest of our system.

Consider what a typical payment module handles. It translates internal payment requests into provider-specific API calls. It manages authentication tokens and API keys for multiple providers. It handles webhook signatures and payload verification. It normalises responses across different providers, mapping various status codes and error messages into a consistent format. It implements retry logic and circuit breakers for provider outages.

This wrapper pattern makes sense from a separation of concerns perspective. Business modules shouldn’t need to understand Stripe’s payment intent flow or PayPal’s authorisation and capture model. They should simply request a payment and receive a result.

However, making this wrapper truly generic presents challenges. Different parts of our system need payments for different reasons. An order needs payment before fulfilment. A subscription needs recurring payments. A marketplace needs to split payments between vendors. A reimbursement needs to refund previous payments. Each scenario has unique requirements, yet they all need the same underlying capability: moving money.

The key insight is that payment processing, at its core, involves the same operations regardless of the business context. Whether we’re charging for an order or processing a reimbursement, we’re essentially creating a financial transaction with an amount, currency, and customer. The challenge lies in building a payment module that can serve all these scenarios without becoming coupled to any of them.

Modern payment providers like Stripe have recognised this pattern. Their Payment Intents API offers a standardised approach to handling various payment scenarios.

The Coupling Problem - Payment Module Example

Let’s examine how coupling emerges in a traditional event-driven architecture. Imagine our payment module listening for events from various parts of the system:

interface OrderCreated {
  type: 'OrderCreated';
  orderId: string;
  customerId: string;
  amount: number;
  currency: string;
}

interface ReimbursementRequested {
  type: 'ReimbursementRequested';
  reimbursementId: string;
  originalPaymentId: string;
  customerId: string;
  amount: number;
  reason: string;
}

interface SubscriptionRenewalDue {
  type: 'SubscriptionRenewalDue';
  subscriptionId: string;
  customerId: string;
  amount: number;
  planId: string;
  nextBillingDate: Date;
}

interface MarketplacePurchaseEvent {
  type: 'MarketplacePurchase';
  purchaseId: string;
  buyerId: string;
  splits: Array<{
    vendorId: string;
    amount: number;
    commission: number;
  }>;
}

The pseudocode for handling that could look as follows:

class PaymentModule {
  constructor(
    private eventBus: EventBus,
    private paymentProvider: PaymentProvider
  ) {
    // The payment module must know about every event type
    this.eventBus.subscribe('OrderCreated', this.handleOrderCreated.bind(this));
    this.eventBus.subscribe('ReimbursementRequested', this.handleReimbursementRequested.bind(this));
    this.eventBus.subscribe('SubscriptionRenewalDue', this.handleSubscriptionRenewal.bind(this));
    this.eventBus.subscribe('MarketplacePurchase', this.handleMarketplacePurchase.bind(this));
  }

  private async handleOrderCreated(event: OrderCreatedEvent) {
    const result = await this.paymentProvider.createPayment({
      amount: event.amount,
      currency: event.currency,
      customerId: event.customerId,
      metadata: { orderId: event.orderId }
    });

    // Must publish order-specific response event
    this.eventBus.publish({
      type: 'OrderPaymentCompleted',
      orderId: event.orderId,
      paymentId: result.id,
      status: result.status
    });
  }

  private async handleReimbursementRequested(event: ReimbursementRequestedEvent) {
    // Different logic for reimbursements
    const result = await this.paymentProvider.createPayment({
      paymentId: event.originalPaymentId,
      amount: event.amount,
      reason: event.reason,
      metadata: { reimbursementId: event.reimbursementId }
    });

    // Different response event type
    this.eventBus.publish({
      type: 'ReimbursementProcessed',
      reimbursementId: event.reimbursementId,
      refundId: result.id,
      status: result.status
    });
  }

  private async handleMarketplacePurchase(event: MarketplacePurchaseEvent) {
    // Yet another response event type
  }

  // More handlers for each business scenario...
}

This design creates multiple problems.

First, the payment module must know about every business module that needs payments. When a new feature requires payment processing, we must modify the payment module to handle its specific event type and business rules.

Second, each integration requires custom logic. Orders might need immediate capture, subscriptions need scheduled billing, and marketplaces need split payments. The payment module becomes a collection of business-specific handlers rather than a generic payment service.

Third, response handling creates reverse dependencies. The payment module must know how to notify each requesting module, publishing module-specific completion events. This doubles the coupling surface area.

The release train problem becomes apparent when we trace a simple change. Suppose the order module needs to add a discount field. This requires updating the OrderCreatedEvent structure, modifying the payment module’s handler to process discounts, changing the response event structure, and coordinating deployment of both modules. What should be a simple change to order processing ripples through the entire system.

Testing becomes increasingly complex as the payment module accumulates dependencies. Unit tests must mock events from every integrated module. Integration tests require coordinating multiple modules. Adding a new payment scenario means updating test suites across multiple codebases.

The payment module also loses its generic nature. Instead of being a thin wrapper over external payment providers, it becomes an orchestrator of business logic. It knows that orders need immediate payment, reimbursements relate to previous transactions, and subscriptions recur monthly. This business logic properly belongs in the respective business modules, not in the payment infrastructure.

Partial Solution: Command Pattern

One approach to reducing coupling involves exposing a command interface instead of listening for specific events. Rather than the payment module knowing about orders, reimbursements, and subscriptions, other modules send generic payment requests:

interface RequestPayment {
  amount: number;
  currency: string;
  customerId: string;
  description?: string;
  metadata?: Record<string, any>;
}

And handle it like:

class PaymentModule {
  // Generic payment processing - knows nothing about business context
  async requestPayment(command: RequestPayment): Promise<PaymentResult> {
    // Just wraps the payment provider
    const result = await this.paymentProvider.processPayment({
      amount: command.amount,
      currency: command.currency,
      customerId: command.customerId,
      description: command.description,
      metadata: command.metadata
    });

    return {
      paymentId: result.id,
      status: result.status,
      amount: result.amount,
      currency: result.currency
    };
  }
}

This inverts the dependency. Instead of the payment module depending on other modules’ event types, other modules depend on the payment module’s command interface. The payment module no longer needs to be aware of orders or reimbursements; it simply processes payment requests.

Now, business modules can use this generic interface:

class OrderModule {
  async processOrder(order: Order) {
    const paymentResult = await this..requestPayment({
      amount: order.total,
      currency: order.currency,
      customerId: order.customerId,
      description: `Order ${order.id}`,
      metadata: {
        orderId: order.id
      }
    });

    if (paymentResult.status === 'succeeded') {
      await this.markOrderAsPaid(order.id, paymentResult.paymentId);
    }
  }
}

However, this solution introduces new challenges. In distributed systems, payment processing often happens asynchronously. Stripe might take several seconds to process a payment. 3D Secure authentication requires user interaction. Webhooks arrive minutes or hours after the initial request. The requesting module needs to know when the payment completes, but how does the payment module notify the requester without creating a reverse dependency?

We could use callbacks or return addresses, but this reintroduces coupling in a different form. The payment module would need to understand how to communicate with each requesting module, whether through specific message queues, HTTP endpoints, or event types.

We’ve simply moved the coupling problem rather than solving it.

What we need is a way for modules to correlate requests and responses without explicit knowledge of each other. The requesting module should be able to say “process this payment” and later receive notification of completion without the payment module knowing anything about the requester. This requires a more sophisticated approach to identification and routing.

Complete Solution: URN-Based Predictable Identifiers

Uniform Resource Names (URNs) provide a standardised way to create globally unique, persistent identifiers. Unlike URLs, which specify location, URNs specify identity. By structuring these identifiers predictably, we can encode routing and correlation information directly into the identifier itself.

Consider this URN format for our payment system:

urn:payments:{version}:{source}:{tenantid}:{uniqueid}

Let’s break down each component. The payments namespace identifies this as a payment-related identifier. The source indicates which module or system initiated the payment request. The tenantid enables multi-tenant routing at the infrastructure level. The uniqueid ensures global uniqueness, often incorporating timestamps or UUIDs. The version allows for format evolution while maintaining backward compatibility.

Here are concrete examples:

urn:payments:1:ORD:TN1:20240115-0jdfj93
urn:payments:1:RMB:TN1:20240115-ksd8234
urn:payments:1:SUB:TN2:20240115-mnb9821
urn:payments:1:MKT:TN1:20240115-poi8734
urn:payments:1:EXT:TN1:CRM1-jdj292

The first identifier represents a payment initiated by the order module (ORD) for tenant TN1. The second comes from the reimbursement module (RMB). The third originates from subscriptions (SUB) for a different tenant. The fourth is from the marketplace module (MKT). The fifth shows an external system (EXT) integration where CRM1 might be an external CRM’s identifier.

URNs offer several advantages over ad-hoc identifier schemes. They follow RFC 8141, providing a well-documented standard. They’re hierarchical, allowing for logical organisation and pattern matching. They’re opaque to external systems while being meaningful to internal infrastructure. Most importantly, they’re self-describing, carrying their own routing information.

The implementation requires careful consideration of field naming and placement. We could call this field processId, emphasizing its role in process correlation, or correlationId, highlighting its use in linking related operations. Some teams prefer contextId to indicate that it carries contextual information beyond simple identification.

Let’s update our payment request to use URNs:

interface PaymentRequest {
  correlationId: string;  // The URN
  amount: number;
  currency: string;
  customerId: string;
  description?: string;
  metadata?: Record<string, any>;
}

And the implementation to:

class PaymentModule {
  async requestPayment(request: PaymentRequest): Promise<void> {
    // Process payment asynchronously
    await this.queue.publish('payment.requests', request);
  }

  private async processPaymentAsync(request: PaymentRequest) {
    try {
      const result = await this.paymentProvider.processPayment({
        amount: request.amount,
        currency: request.currency,
        customerId: request.customerId,
        metadata: {
          correlationId: request.correlationId,
          ...request.metadata
        }
      });

      // Publish response using URN for routing
      await this.publishResponse(request.correlationId, {
        paymentId: result.id,
        status: result.status
      });
    } catch (error) {
      await this.publishResponse(request.correlationId, {
        status: 'failed',
        error: error.message
      });
    }
  }

  private async publishResponse(correlationId: string, response: any) {
    // Infrastructure handles routing based on URN
    await this.queue.publish('payment.responses', {
      correlationId,
      ...response
    });
  }
}

Implementation: RabbitMQ Routing

Let’s implement this pattern using RabbitMQ to demonstrate infrastructure-level routing. RabbitMQ implement various strategies to route messages between queues, they do it through Exchanges.

Exchanges are routing layers that receive messages and route them to queues based on specific rules. They don’t store messages.

Queues are storage layers - they buffer messages until consumers process them. Messages physically reside here.

Think of it as a postal system: exchanges are sorting facilities, queues are mailboxes. Publishers are unaware of mailboxes (queues); they simply send messages to the sorting facility (exchange) with an address (routing key).

We’ll use topic exchanges. They enable routing based on routing keys with wildcards, perfect for our URN structure.

Source: https://www.cloudamqp.com/blog/part4-rabbitmq-for-beginners-exchanges-routing-keys-bindings.html

First, we’ll set up the infrastructure configuration:

class PaymentsMessagingConfiguration {
  async setupRouting(channel: Channel) {
    // Main exchange for payment responses
    await channel.assertExchange('payment-responses', 'topic', { durable: true });

    // Each module creates its own response queue
    await channel.assertQueue('orders-payment-responses', { durable: true });
    await channel.assertQueue('reimbursements-payment-responses', { durable: true });
    await channel.assertQueue('subscriptions-payment-responses', { durable: true });

    // Bind queues with routing patterns based on URN components
    // Orders module receives responses for payments it initiated
    await channel.bindQueue(
      'orders-payment-responses',
      'payment-responses',
      'payments.*.ORD.*.*'
    );

    // Reimbursements module receives its responses
    await channel.bindQueue(
      'reimbursements-payment-responses',
      'payment-responses',
      'payments.*.RMB.*.*'
    );

    // Tenant-specific routing for multi-tenant systems
    await channel.bindQueue(
      'tenant-tn1-payments',
      'payment-responses',
      'payments.*.*.TN1.*'
    );
  }
}

Now let’s implement the complete flow from order creation through payment completion:

class OrderModule {
  async payOrder(orderId: string) {
    const order = await this.load(orderId);

    // Generate URN for this payment
    const correlationId = `urn:payments:1:ORD:${order.tenantId}:${order.id}`;

    // Request payment with URN
    await this.paymentsClient.requestPayment({
      correlationId,
      amount: order.total,
      currency: order.currency,
      customerId: order.customerId,
      description: `Order ${order.id}`
    });
  }

  async handlePaymentResponse(
    event: PaymentCompleted | PaymentFailed
  ) {
    const { correlationId, status, paymentId } = event;

    // Extract order ID from URN
    const urnParts = correlationId.split(':');
    const orderId = urnParts[5];

    // Update order based on payment status
    if (status === 'succeeded') {
      await this.markOrderAsPaid(orderId, paymentId);
    } else if (status === 'failed') {
      await this.markOrderAsPaymentFailed(orderId);
    }
  }
}

The payment module processes requests and publishes responses without knowing the source:

class PaymentModule {
  async onPaymentCompletedProviderWebhook(result: PaymentGatewayPaymentCompleted) {    
    try {      
      // Extract routing key from URN
      // urn:payments:1:ORD:TN1:123 becomes payments.1.ORD.TN1.123
      // to match RabbitMQ naming convention
      const routingKey = request.correlationId
        .replace('urn:', '')
        .replace(/:/g, '.');

      // Publish response using URN-based routing
      await this.channel.publish('payment-responses', routingKey, {
        correlationId: request.correlationId,
        paymentId: result.id,
        status: result.status
      });

    } catch (error) {
      const routingKey = request.correlationId
        .replace('urn:', '')
        .replace(/:/g, '.');

      await this.channel.publish('payment-responses', routingKey, {
        correlationId: request.correlationId,
        status: 'failed',
        error: error.message
      });
    }
  }
}

Here’s the complete flow visualised:

Additional Benefits

The predictable identifier pattern yields benefits beyond basic routing and correlation.

Webhook correlation becomes straightforward when external systems call back. Mature payment providers allow us to pass the external id or send custom metadata, including our URN. When a webhook arrives, we extract the correlation ID and route it back to the originating module without any hardcoded logic.

Natural idempotency emerges from URNs. The correlation ID itself can serve as an idempotency key, preventing duplicate payment processing when modules retry failed requests. This is especially valuable for payment operations where accidental duplicates can be costly.

Infrastructure-level tenant routing becomes possible. Since the tenant ID is embedded in the URN structure, the routing infrastructure can direct messages to tenant-specific queues or processing clusters without the payment module being aware of multi-tenancy concerns.

Debugging and observability improve dramatically. Every operation carries its correlation ID through the entire flow, making it simple to trace a payment from initiation through completion. Log aggregation tools can group all events by correlation ID, providing a complete view of each transaction.

Most importantly, the payment module remains completely generic. It processes payments without knowing whether they originate from orders, subscriptions, reimbursements, or any future module. This is the essence of an open host service: a truly autonomous module that provides its capability without coupling to its consumers.

Security and Practical Considerations

While predictable identifiers offer numerous benefits, they also introduce security considerations. The structure itself reveals information about your system’s organisation, potentially exposing module names and tenant identifiers to external observers.

Protect against identifier traversal attacks logically can look as follows:

class SecurePaymentHandler {
  async handlePaymentResponse(
    message: Message,
    context: RequestContext
  ) {
    const { correlationId } = message;

    // Validate tenant access
    const urnTenant = this.extractTenant(correlationId);

    if (urnTenant !== context.tenantId) {
      throw new ForbiddenError('Access denied');
    }

    // Validation based on source
    const source = this.extractSource(correlationId);

    if (!context.allowedSources.includes(source)) {
      throw new ForbiddenError('Source not authorized');
    }

    // Additionally, you can store and verify correlation exists
    const exists = await this.correlationStore.exists(correlationId);
    if (!exists) {
      throw new NotFoundError('Invalid correlation');
    }

    // Process valid response
    return this.processResponse(response);
  }
}

Rate limiting can also leverage the URN structure:

class UrnBasedRateLimiter {
  async checkLimit(correlationId: string): Promise<boolean> {
    const source = this.extractSource(correlationId);
    const tenant = this.extractTenant(correlationId);

    // Different limits for different sources
    const limits = {
      ORD: 1000,  // Orders can make many payments
      RMB: 100,   // Reimbursements are less frequent
      EXT: 10     // External systems get lower limits
    };

    const limit = limits[source] || 50;
    const key = `ratelimit:${tenant}:${source}`;

    const count = await this.redis.incr(key);
    if (count === 1) {
      await this.redis.expire(key, 3600); // 1 hour window
    }

    return count <= limit;
  }
}

Consider encryption for sensitive components (or best avoid sending such):

class SecureUrnGenerator {
  generateUrn(source: string, tenantId: string, uniqueId: string): string {
    // Version 2 URNs use encryption
    const version = '2';

    // Encrypt sensitive parts
    const encryptedTenant = this.encrypt(tenantId);
    const encryptedId = this.encrypt(uniqueId);

    // Base64URL encode for safety
    const safeTenant = Buffer.from(encryptedTenant).toString('base64url');
    const safeId = Buffer.from(encryptedId).toString('base64url');

    return `urn:payments:${version}:${source}:${safeTenant}:${safeId}`;
  }

  parseSecureUrn(urn: string): ParsedUrn {
    const parts = urn.split(':');
    const version = parts[2];

    if (version !== '2') {
      throw new Error('Unsupported URN version');
    }

    return {
      source: parts[3],
      tenantId: this.decrypt(Buffer.from(parts[4], 'base64url')),
      uniqueId: this.decrypt(Buffer.from(parts[5], 'base64url'))
    };
  }
}

You can, of course, go further, support checksums, etc.

Conclusion

Predictable identifiers transform how modules communicate in distributed systems. By encoding routing and correlation information directly into identifiers, we enable true module autonomy. The payment module example demonstrates this transformation clearly - from a tightly coupled system where the payment module must know about every business scenario, to a generic service that processes payments without knowing their source.

The key insight is that identity can carry meaning without creating coupling. URNs provide a standardised way to structure this identity, enabling infrastructure-level routing while maintaining module independence. The payment module processes generic payment requests and publishes responses using the correlation ID for routing. It never needs to know whether a payment is for an order, reimbursement, or subscription.

This pattern extends beyond payments to any cross-module communication. Any module that provides generic services - notifications, document generation, audit logging - benefits from predictable identifiers. The pattern is particularly powerful in event-driven architectures where asynchronous communication and webhook handling are common.

Implementation requires thoughtful design choices around format, security, and infrastructure integration. However, the investment pays dividends in reduced coupling, improved debugging, and easier system evolution. Teams can work independently, modules can be added or removed without system-wide changes, and the promise of microservice architecture finally becomes reality.

As systems grow in complexity, the need for autonomous modules becomes critical. Predictable identifiers offer a proven pattern for achieving this autonomy without sacrificing coordination. They represent a fundamental shift in how we think about module communication - from explicit contracts to emergent behaviour, from tight coupling to flexible collaboration.

Are you already using URN-based strategies in your systems?

Read also more in:

Cheers!

Oskar

Start Alone, Then Together: Why Software Modelling Needs Solitary Brainstorming

Watch a jazz quartet improvise, and you might think they’re making it up as they go. They’re not. Every riff builds on practised scales and learned patterns. The constraints don’t stifle creativity—they make it work.

Image source: https://nolaschoolofmusic.com/blog/tips-for-learning-jazz-improvisation

Most software teams get this when writing code. You wouldn’t build a critical system without established patterns, architectural principles, or coding standards. But put those same developers in a modelling session, and they drop all structure. The CTO speaks first, and everyone else responds. The loudest developer dominates the whiteboard. The most persuasive architect steers every decision.

I’m calling that “the CTO effect”.

The problem isn’t, of course, malicious. Quieter voices simply don’t get heard when the first person to speak sets the direction.

The solution: start modelling sessions with 10-15 minutes of solitary brainstorming before anyone speaks.

This breaks the hierarchy cycle at its source. When everyone generates ideas simultaneously in silence, there’s less chance for the first voice to set the agenda. No senior person’s opening statement to shape everyone else’s thinking. The junior developer and the CTO start from the same blank point for those crucial first minutes.

Why Written Ideas Persist

Written ideas are harder to dismiss than spoken thoughts. When someone voices an idea in a meeting, it exists for thirty seconds before the next person talks over it. Write that same idea on a shared board, and it stays visible. It’s much harder to ignore something sitting there in front of everyone.

Once you’ve written something on a sticky note, you materialise it, and it exists. It’s mentally harder to handwave it. That requires justification. The same idea spoken aloud dies with a simple “hmm, interesting” before moving on.

It’s not a novel idea, of course, other companies are already doing it. AWS start meetings with a silent reading time. Everyone quietly reviews the same document before discussion begins. No talking, no interrupting, no immediate reactions. Ideas get absorbed without social dynamics taking over. By the time the discussion starts, everyone works from the same information baseline rather than whoever spoke first setting the agenda. Read more here.

When Hierarchy Shapes Thinking

This happens beyond just CTOs. Senior architects get deference. The developer who wrote the original system carries weight. The expensive consultant commands attention.

These people can be seen as those who know what they’re talking about. And quite often they do. But not always. Sometimes they just extrapolate their past experience, and that’s dangerous, as Mathias Verraes wrote in The “It’s Just Like…” Heuristic:

Your brain rewards you for classifying new information into existing buckets1. Looking for similarities costs less energy than understanding new differences. When you say “Oh I get it, it’s just like …”, then it probably is not “just like”. That’s very counterintuitive, because that “oh I get it” feeling is intuition, not reason2. You feel good about the insight, so it’s very tricky to then go “Oh I get it, so I must be wrong”.

So, they can be more often right than you, but then if they fail, that can be quite spectacular.

In other words, their presence narrows the solution space before alternatives get explored. You may be optimising for consensus, not quality. You build the system that the most senior person envisioned, not necessarily the system that best solves the problem.

Solitary brainstorming counters this. When everyone generates ideas simultaneously and silently, the junior developer’s insight about edge cases appears alongside the architect’s system design with equal visual weight.

Structure Helps Improvisation

Going back to our improvisation metaphor.

Jazz musicians work within scales, follow chord progressions, and respond to patterns the group establishes. The structure doesn’t limit creativity—it gives creativity a framework to work within.

Setting a format for your session doesn’t have to limit it; it can help you focus on the merit and reduce the likelihood of wandering off.

The brainstorming phase outcome provides the raw material, events, questions, and ideas from everyone’s perspective. The EventStorming, C4 and others notations help in not having discussions on how to write stuff. We can cut time for questions on “what does this colour mean?” or “what is this box about?”. The structure helps facilitate discussions.

The solitary brainstorming approach requires discipline, which explains why teams skip it. Starting with a discussion feels easier and more natural. But if you’re already blocking time for modelling sessions, you’ve committed to doing the work properly.

So, try it on your own, set a timer for 10-15 minutes. Everyone works silently, generating ideas related to the modelling question. Use a shared digital board (like Miro), where ideas appear in real-time, but refrain from commenting until the time expires. This is active brainstorming, not a time for preparation. Quantity beats quality at this stage.

After the timer goes off, you have a board full of ideas from everyone, not just whoever speaks first or loudest. The collaborative phase can focus on synthesis and evaluation—what groups do well—instead of generation.

Individual brainstorming creates raw material. Group collaboration refines it. Ask a group to do both simultaneously, and one suffers.

How EventStorming Works Here

EventStorming sessions show this approach in practice. Instead of having the domain expert explain how everything works while everyone takes notes, everyone spends the first phase independently writing domain events on orange sticky notes. Things that happen in the business that stakeholders care about.

The result can look messy. Some events are duplicates with different wording. Some are too abstract, others too specific. Some are probably wrong. That’s the point.

You now have raw material representing everyone’s understanding of the domain, not just whoever speaks first. The next phase, grouping similar events, identifying gaps, building a timeline, becomes collaborative work in reconciling different perspectives instead of rubber-stamping one person’s vision.

Keep every sticky note during initial grouping. Every event that someone thought worth writing about stays visible, even if it ends up in the “questions” or “parking lot” sections. This prevents the premature filtering that occurs when ideas only exist as spoken words.

What I’ve Observed

In my experience, modelling sessions that start this way become less about convincing others of predetermined solutions and more about exploring a shared problem space. When everyone contributes ideas upfront, people seem more willing to engage with the discussion.

This isn’t about protecting feelings or ensuring everyone gets heard. Software systems are complex enough that no single person’s mental model captures everything. The goal is to develop more effective systems.

I’ve found that individual work makes collaborative work more productive. Skip the solo brainstorming, and you haven’t saved time: you’ve just moved idea generation into group discussion, where it’s slower and less effective.

Software modelling is complex enough without predictable social dynamics making it more challenging.

What’s your take?

Read also more in:

Cheers!

Oskar

Secondary Indexes and the Specialized Storage Dilemma

In 2016, the Kafka team faced a problem that illustrates a fundamental challenge in distributed systems design. Users kept asking:

“How do I query the current state of my data without replaying the entire log?”

If you’re tracking user balances through a stream of transactions, you shouldn’t need to replay millions of events just to find out someone’s current balance.

The problem was that Kafka’s storage was optimised for sequential log consumption and was inefficient at point queries.

The Cost of Specialisation

Building your own storage engine means making specific trade-offs. Kafka’s log storage uses sequential writes to append-only segment files, enabling high throughput.

We discussed the details already in:

Consumers read these segments sequentially, often using zero-copy transfers directly from the file system cache to the network. But this design makes point queries expensive. Want to find a specific user’s latest state? You’ll need to scan through potentially gigabytes of log data. There’s no index because indexes would require random writes, which would hurt the sequential write performance that makes Kafka useful for streaming.

This is the paradox of specialised storage: the optimisations that make you successful also constrain your options. Your users chose your system because it excels at one thing, but success brings diverse use cases. They’ve already invested in your system, their data lives there, and they don’t want to set up another piece of infrastructure just to run different queries.

The Kafka State Store Solution

The Kafka team had three options:

Add indexes to the log storage (degrading its sequential write performance).
Tell users to export data to a database (increasing operational complexity).
Build a separate storage tier optimised for point queries.

They chose option three. Around this time, the Kafka team was developing Kafka Streams, a stream processing library. One of its features fit the state query need: embedded state stores.

So the team built Kafka Streams, a Java library that processes data from Kafka topics. They embedded a completely different database inside these processing applications. That database is RocksDB, an embedded key-value store that Facebook originally built for fast local storage.

Unlike Kafka’s append-only logs, RocksDB uses an LSM-tree structure that’s optimised for random reads and writes. You can look up any key instantly without scanning through unrelated data. By embedding RocksDB inside each stream processor, they created local “state stores” that could answer point queries.

This architecture means that your Kafka messages flow through a Java application that maintains a local database of the current state. If you’re tracking user balances, the application reads transaction events from Kafka and updates each user’s balance in its local RocksDB. Now you can instantly query any user’s current balance, without replaying the log.

However, you’ve created a new problem: the RocksDB instance is local to a single machine. If the machine fails, you lose the state. So Kafka added yet another component: a changelog topic.

Source: https://www.linkedin.com/pulse/kafka-streams-state-store-rob-golder/

Every state store write also produces an event for this changelog. Now you have durability, but look at what your “index” has become:

The original message in the input topic.
The state store update in RocksDB.
The changelog event for durability.

This write amplification continues to grow. The changelog topic itself gets replicated across brokers for fault tolerance. If you have multiple state stores for different query patterns, each needs its own changelog. A single business message might trigger a dozen physical writes across the system.

DynamoDB’s Similar Challenge

The DynamoDB team at AWS faced a similar problem. DynamoDB’s storage is optimised for partition key access: you can retrieve any item instantly if you know its partition key. The system achieves this through consistent hashing and partition-specific storage optimisation.

But users wanted to query by other attributes. “Find all orders placed yesterday” or “Get all items with status=‘pending’” required scanning entire tables, which was expensive and slow.

DynamoDB’s solution was Global Secondary Indexes, or GSIs. The name suggests they’re just indexes, but the implementation tells a different story. A GSI is actually a complete, separate DynamoDB table. It has its own partitions, capacity units, and replication. When you write to the main table, DynamoDB captures that change in DynamoDB Streams and asynchronously replicates it to each GSI.

Source: https://aws.amazon.com/blogs/database/how-to-use-dynamodb-global-secondary-indexes-to-improve-query-performance-and-reduce-costs/

It’s not an index in any traditional sense; it’s a derived table that happens to be automatically maintained.

The main table write.
DynamoDB Streams event.
Asynchronous writes to each Global Secondary Index table.

Your “index” needs its own capacity planning. It can throttle independently of your main table. Each GSI has its own replicas and its own failure modes. The write amplification is proportional to the number of query patterns you support.

Why Secondary Indexes Work Differently in Distributed Systems

In a traditional single-node database, a secondary index is an additional data structure that enables the efficient retrieval of records without requiring a full table scan. Think of a phone book: the main data is sorted by last name, but you might also want a separate index sorted by phone number. When you look up a phone number, you check the number index, find the person’s name, and then jump to their complete entry in the main book.

Databases implement this using B-trees, which are like hierarchical directories. To find a value, you start at the root and follow pointers down through intermediate nodes until you reach the leaf containing your data. This gives you logarithmic lookup time: finding one record among a million takes only about 20 comparisons, not a million.

The crucial part is that in a single-node database, updating the main table and its indexes happens atomically. When you change someone’s phone number, the database updates both the main record and the phone number index in one transaction. Either both succeed or both fail.

Distributed systems can’t work this way. You cannot atomically update multiple data structures across different nodes. The network can partition, nodes can fail, and coordinating updates would require distributed transactions that destroy performance.

Instead of fighting this constraint, both systems adopted eventual consistency. Secondary indexes become eventually consistent views of the primary data, maintained through asynchronous replication.

These aren’t indexes in the traditional sense. They can be entirely different storage systems with their own:

Partitioning schemes
Replication strategies
Consistency models
Failure recovery procedures
Capacity planning requirements

When someone says “add a secondary index” to a distributed system, they’re really saying “build and operate another storage system”.

How These “Indexes” Actually Work

Starting with Kafka State Stores: when you query a state store, you’re querying that local RocksDB instance I mentioned. The critical design decision is co-partitioning. Kafka ensures all events with the same key go to the same partition, and that partition is processed by one specific task. If you’re tracking user balances, all events for user “12345” go to the same place, and the state for user “12345” lives in the RocksDB instance on that same machine.

This co-partitioning is elegant but creates operational complexity. When your application wants to query a user’s balance, it needs to know which machine holds that user’s state. Kafka maintains a directory of which keys live where through its group membership protocol. The Interactive Queries API uses this directory to route queries to the correct instance. During failures or rebalancing, queries might fail or return stale data from standby replicas.

But what happens when that machine fails? That’s where the changelog topic comes in. Every update to the local RocksDB also produces an event to a special Kafka topic. If the machine dies, a new instance can rebuild the entire state by replaying this changelog. It’s durability through replication, but notice the cost: every state update is written twice, once to RocksDB and once to Kafka.

DynamoDB’s GSIs work through a similar async replication pattern but with different trade-offs. Remember, a GSI is a complete DynamoDB table. When you write to the main table, DynamoDB captures that change in DynamoDB Streams (essentially a change data capture log) and asynchronously propagates it to your GSIs.

Notice that word “asynchronously”. Your write to the main table succeeds immediately, but the GSI updates happen later, typically within one to typically within one to two seconds under normal load (however this can stretch to minutes or more if the GSI is under-provisioned or the table is experiencing heavy writes). This lag is fundamental: updating synchronously would require coordinating across multiple partitions, which would significantly impact DynamoDB’s performance.

GSIs can also be sparse, meaning only items with the indexed attribute appear in the index. If you index on “email,” items without an email field won’t exist in that GSI. This saves storage but means you need to understand which queries each GSI can actually serve.

Both systems chose different points in the design space based on their constraints. Kafka prioritised keeping related data together through co-partitioning, accepting the complexity of query routing. DynamoDB prioritises flexibility, allowing GSIs to have completely different partition keys than the main table, while accepting higher eventual consistency lag and capacity management overhead.

Understanding Write Amplification

Once you understand these architectures, you can calculate the true cost of query flexibility. In DynamoDB, if you have a main table with three GSIs, every single item write becomes four separate writes: one to the main table and one to each GSI. Each of these writes is then replicated (typically three times) for durability. So your single logical write has become twelve physical writes across the infrastructure.

Kafka Streams faces a similar multiplication. When a message arrives, you write it to the output topic (three replicas), update the local state store, and write to the changelog topic (another three replicas). If you maintain two different state stores for different query patterns, add another changelog topic. One business event can easily trigger nine or more physical writes.

Each write serves a purpose: the original for durability, the indexes for queries, the replicas for fault tolerance. The amplification factor directly correlates with the query flexibility you’re providing.

Living with Distributed Indexes

Running these systems in production reveals their operational reality. The most critical metric becomes replication lag: how far behind are your indexes? In DynamoDB, your GSIs typically lag one to two seconds behind the main table. Under heavy write load, this can stretch to minutes. Kafka state stores face similar challenges during rebalancing, when partitions move between machines, and changelog replay can take hours.

Because of this lag, you need to design applications differently. A user might update their profile and immediately query it through a GSI, retrieving their old data. Your code needs to handle this gracefully, perhaps by reading from the main table when freshness matters. Some teams implement “read your writes” patterns, tracking recent updates client-side.

Capacity planning becomes multidimensional. It’s not enough to provision for your write rate; you need to account for the amplified rate across all indexes. A DynamoDB table handling 1,000 writes per second with three GSIs needs 4,000 write capacity units total. Miss this calculation, and your indexes will throttle, lag will increase, and queries will return increasingly stale data.

Recovery scenarios add another layer of complexity. When a Kafka Streams instance crashes, its replacement must replay the entire changelog to rebuild state. For large state stores, this can take hours. DynamoDB faces similar challenges when adding a new GSI to an existing table: it must backfill the entire index, potentially processing billions of items.

When to Pay the Secondary Index Tax

Given these costs, when do secondary indexes make sense in distributed systems? It depends on your alternatives. If users need to find orders by customer ID and your only option is scanning terabytes of data, the write amplification suddenly looks reasonable. If queries are rare or can tolerate minutes of latency, maybe a periodic batch job is better.

You need to understand the full cost model from the start. Adding an index means multiple storage systems, eventual consistency in your application logic, multiplied infrastructure costs, complex recovery procedures, and new operational playbooks. Sometimes these costs are worth it. Sometimes you’re better off telling users to adjust their query patterns.

Successful teams treat each secondary index as a long-term commitment. They prototype with real workloads to understand the amplification factor. They build tooling to monitor lag and alert on degradation. They educate developers about eventual consistency. Most importantly, they push back on speculative indexes, adding them only when the use case is clear and the alternative is untenable.

This pattern appears everywhere in distributed storage:

Elasticsearch adds doc values for analytics queries
Cassandra implements materialised views
MongoDB maintains separate index structures per shard
Even PostgreSQL separates heap and index storage

They all face the same constraint: you can’t modify specialised storage to support different query patterns without ruining its original performance characteristics. So you build separate storage tiers, accept the write amplification, and manage the complexity.

Each system optimises for its specific use case. Kafka keeps data co-partitioned, so related state lives on the same machine. DynamoDB allows completely different partitioning schemes between main tables and GSIs, maximising query flexibility. Elasticsearch builds multiple specialised structures for different query types. Each choice reflects the system’s priorities, but the fundamental trade-off remains the same.

In distributed systems, there’s no such thing as “just” adding an index. There’s only building and operating another storage system, accepting the write amplification, and managing the complexity. Sometimes that trade-off is worth it. But now you understand why it’s a trade-off at all.

As users of those tools, we need to understand that there’s no magical glue. We can’t cheat physics. Before we choose to use secondary indexing capabilities, we should analyse if we really need them, as they are usually also second-class citizens in those tools. They can still be a good choice, as they can cut our application code, but sometimes it’s just better not to bend the tool to scenarios that it wasn’t designed for and solve the challenge differently.

I hope that this article will help you in better understanding what you’re signing for!

Cheers!

Oskar

Do we still need the QA role?

Welcome to the new week!

I’ve been seeing the same discussion popping up across tech forums and social media lately: is QA dead?

“The testing market is ending.”

Some argue.

“AI will handle whole testing soon.”

Others chime in.

This conversation has been going in circles for years, long before AI became everyone’s favourite scapegoat.

I think that the real question isn’t whether AI will replace testers. It’s whether we’ve ever understood what good testing looks like.

Why This Debate Even Exists

The reason we’re having this conversation at all comes down to how our industry has treated testing roles over the past decade. Companies marketed testing as the “easy entry point” into the tech industry.

“Anyone can click through an interface and report bugs. No coding required! Perfect for career changers.”

This created an army of “clicker” testers who went through the same manual scenarios without understanding the underlying systems or business logic.

When developers, especially backend developers, worked with these teams, they naturally concluded that testing was mindless work they could do better themselves.

However, here’s the interesting psychological aspect. When someone tells you your code has bugs, you can feel that they’re criticising your work. The natural human response? Get defensive.

“What does this person know? I wrote this code, I understand it better than anyone.”

It’s easier to dismiss the messenger than admit you might have missed something. We’re again too attached to our code, treating bugs in it as a personal thing.

Developers (especially backend ones) work in clean, logical systems with well-defined inputs and outputs. They write unit tests that verify their code works exactly as intended. From their perspective, what more could you need?

Here’s what I’ve observed: developers often test what they built, not what should have been built. They verify their implementation matches their understanding of requirements. They rarely question whether those requirements made sense or whether their interpretation was correct.

I’ve seen projects with 100% code coverage and comprehensive test suites still ship with obvious UX problems, business logic gaps, and integration issues that become apparent the moment real users interact with the system.

All tests passed, product still sucked.

The Automation Illusion

The push toward test automation made this worse. Automation is great for repetitive, well-understood scenarios, but there is a fundamental misunderstanding about what it can and cannot do.

Automated tests catch regressions and verify that known scenarios continue to function as expected. They’re ineffective at identifying new problems, challenging assumptions, or understanding user needs. They test exactly what you tell them to test, nothing more.

When developers write automated tests, they’re encoding their own assumptions about how the system should work. If those assumptions are wrong, the tests will happily pass while shipping broken experiences.

QA Isn’t Just Testing

Here’s one of the biggest misconceptions: QA equals testing. Tests are just one piece of quality assurance. Real QA thinking should start during design discussions, continue through development, and extend to release and beyond.

This is not only testing, but ensuring a proper flow, user experience and correlation between functionalities.

QA also ensures dedicated time and focus on exploring what can go wrong. As developers, we are often focused on the feature we are currently working on. Being in the zone, focused on delivering stuff, we may miss pieces that are essential for the overall quality.

As a manager, I always tried to involve QA engineers in planning sessions from day one. Their input on early phases, with their different perspectives, brought significant value. It helped to strengthen the design before any code was written, which meant we could fix potential issues in the design rather than discovering them weeks later.

This early involvement matters because it’s exponentially cheaper to fix problems in design than in code. When a test engineer identifies a user flow issue during design review, you adjust the flow and design accordingly. When they find it during testing, you’re looking at code changes, testing cycles, and delayed releases.

Quality should be baked into every development stage, not bolted on at the end. A good test engineer helps teams think about edge cases during requirements gathering, advocates for testability during architecture discussions, and ensures quality considerations are part of every technical decision.

Dev-Sec-QA-Ops pipe dream

Of course, I’m not trying to say that developers are not responsible for finding similar flows. We do, we all do. Still, it’s about the specialisation. I’m questioning whether we can all be great at everything.

This reminds me of the backend versus frontend debate. In theory, everyone should be full-stack. In practice, while most developers can work on both sides, very few are equally strong at both. The thinking patterns and skills are different enough that a healthy portion of specialisation usually produces better results.

The same principle applies to quality assurance. Should every developer care about quality? Obviously. Should everyone write tests for their code? Of course. But does that mean we don’t need someone who specialises in thinking about quality?

I don’t think so. Just as having a frontend specialist ensures better user experiences, even when backend developers can write JavaScript, having someone whose primary focus is on quality helps ensure better products, even when all developers prioritise testing.

The key is integration, not separation. The most effective teams I’ve worked with had test engineers as full team members, not external consultants. They participated in standups, reviews, architecture discussions, and retrospectives. They weren’t handed requirements to verify—they helped shape those requirements from a quality perspective.

We don’t expect developers also to be designers, product managers, and infrastructure specialists. We recognise these require different ways of thinking.

Testing, real testing, not mindless clicking, requires specialised knowledge. Understanding how users actually behave, designing test scenarios that uncover edge cases, and thinking about system-wide interactions. These are skills that take time to develop.

The problem is that our industry confuses “anyone can click a button” with “anyone can design effective test strategies.” They’re completely different things.

I’ll also highlight that we should not have a horizontal QA team. A horizontal responsibility split is a pathology, in my opinion. I’ve never seen it working in the long term. We should aim to have self-organising teams capable of delivering the complete feature. I believe that also includes QA engineers.

The Path Forward

The solution isn’t eliminating testing roles or making developers responsible for everything. It’s stopping the hiring of unqualified people into testing positions and treating testing as the specialised engineering discipline it actually is.

This means working with test engineers who can code, understand system architecture, automate routine tasks, and focus their intelligence on problems that actually require human intelligence. More importantly, it means involving these specialists from the very beginning.

When test engineers are embedded in teams from day one, participating in design discussions and helping shape requirements, the entire quality conversation changes. Instead of finding expensive problems after they’re built, you prevent them from being built at all.

Developers also need to check their egos and recognise that having someone specifically focused on finding problems in their work is valuable, not threatening. The best developers I know actively seek this feedback because they understand it makes their products better.

The market for unqualified clickers is ending, and that’s good. But the market for skilled test engineers who think systemically about quality? That’s not going anywhere, AI or no AI.

Our products are getting more complex, users are more demanding, and systems are more interconnected. Having someone whose job is thinking about all the ways things can go wrong - from the very beginning of development - isn’t a luxury. It’s basic common sense.

Still, I get that our industry is not shaped by common sense and rationality, but by narratives and trends. It may be that GenAI would be a justification for CEOs to drop the QA Engineer role. Still, if that happens, our product quality and user experience will definitely be hurt by that.

Cheers!

Oskar

Killer metrics, or why you should know upfront when to remove the new feature

Welcome to the new week!

Let’s get back to the story I told in Why to measure and make our system observable? How to reason on chaotic world.

I was working on another migration from on-premises to the cloud. The product and sales teams decided to bait the current customers by exposing full-text capabilities for on-premise system search through the cloud. Teams added a change that continuously synchronised data from on-premises to the cloud. Then, it was indexed in Elasticsearch and exposed through the API on the web and mobile.

Then boom bada bing, bada boom. Big announcement to all on-premise customers that they can now enable it for free and use it through our new Cloud services. Of course, most of the customers enabled it. For free is a decent price, right?

So, great success?

We didn’t have the proper business metrics, so we have to add them.

And guess what! Our full-text indexing of on-premises data appeared to account for around 90% (or more) of the spending. Then we compared it with the business usage of the Cloud search API feature, and you know where it is going.

Only one customer was using it. Rarely.

Before I close the curtain, for this story, I’ll add that when we asked the client if they were okay with removing this feature, they immediately waved their hands and said: “Yeah, kill it; we don’t need it”.

If we’d done the metrics upfront, we’d still need to remove our feature, but we’d have prepared for that and done it earlier, without wasting so much money.

It’s yet another example that we’re wasting more time and code on maintaining things t’re not needed from a product perspective, instead of addressing the technical issues.

We’re not wasting time coding too slowly; we’re wasting time doing stuff that doesn’t matter. How to do it better, then?

Understanding the business and product scenery is crucial. Otherwise, we’ll be doing useless work. Understanding business is a foundation for the proper tradeoff analysis.

So it’s not IT VS Business but IT AND Business, as IT IS a Business nowadays.

Once we realise and agree that working hand in hand with business is beneficial, we can start to collaborate on business metrics.

We can define measurable metrics before starting, not just vague goals. Thanks to them, we can avoid redundant work and redundant costs, knowing when our feature wasn’t successful, and we can kill it.

And that’s, of course, tricky, but we can learn that.

Killer metrics to the rescue

Killer, not because they’re so great, but because we define when we can kill the feature.

Let’s discuss it with the example when the business tells “build email digest feature”. Perhaps they also provide more specifics on how it should work, you could start implementing it. Maybe that would even work, but without understanding why, you may end up in the same scenario as I’ve had with on-premises indexing and full-text search in the cloud.

If you want to understand The Why, you could ask: “What specific user behaviour are we trying to change? What’s our current retention data? Do we know why users don’t return?”

Business could answer: “Users sign up but forget about us. We want them to come back weekly.”

A bit better, but still not perfect, as it still tells more about The What than The Why.

You could follow up with: “Thank you for expanding, could you also tell me more on why do we want them to return more often?”

Business could answer: “It’s nice when people are buying stuff on our E-Commerce platform, but it’s much more efficient if those who buy goods have already repeated that. They’re much more likely to buy stuff than the totally new users. If we offer them customised weekly recommendations through email, they can make additional purchases.

Now we’re talking! Still, we should not be satisfied with knowing The Why.

We should also know The When. The When to kill this feature. Yes, before even starting to work on it.

We could continue with: “Thank you, now I have a clear understanding on the need. I’d like to understand also more on the current use behavior. Do you know how often do users currently return? What engagement do our existing emails get? Do you know the research how email digests perform for similar products?”

Business comes back: “In average, 40% return weekly now. Our newsletter gets 5% click-through. Found studies showing weekly digests increase return rates by 20-30%.”

You collaborate: “Great, thanks for clarification. What would we do if users started to cancel their subscriptions when they began receiving more emails? What would be the satisfying factor of increased traffic? What would we do if there’s no change? I’ll need to perform exact calculations, but we’ll need to use an external email provider that ensures proper delivery through spam filters, etc. That stuff costs. Plus, we’ll need to postpone other features we have in the pipeline.”

Probably both you and the business will need to do their homework, math, etc. But the side effects of that are that you’re already starting to see the reality, so whether this feature is useful or not and whether there’s a financial reason to do it this way.

One of the side effects is that you can kill this idea immediately or change the way you implement it. Or you may come up together with an educated…

Killer metric: If weekly returns don’t improve by at least 5% within 2 months, or if the unsubscribe rate exceeds 8% within that timeframe, we kill it immediately.

And that’s great, as we’re investing in Removability over Maintainability. Thinking about Residuality, we discussed a week ago. You don’t need to maintain what you just removed!

Still, you can think now:

That’s great Oskar. I love it, but my business won’t cooperate.

My business people would tell me: “Just build the email digest feature.”

That’s, of course, a challenge, and it can be a showstopper. Still, we can try to do at least the homework on our side. Maybe we can’t gather all the data, but we can definitely gather some data.

We can:

Check our analytics: current user return patterns, email open rates, and signup-to-usage conversion
Check your SLA/SLO and customer commitments, they can tell you what you have to fulfil.
Mine production data: server logs showing user behaviour patterns, telemetry data on feature usage, database queries to find drop-off points
Look at 3-5 direct competitors: Do they have similar features? How prominent are they?

We could suggest a small experiment first: “Let’s try manual weekly emails to 100 users for 3 weeks to see engagement before building the full feature”.

Of course, we should be careful not to spend too much time so we’re not punished. Still, when we gather some, we can come back with real numbers.

We could tell: “Based on my quick research on current usage data, 40% of users return weekly. Our logs show most users only check the app once after signup. Check also this link, this research shows that we can get above 5%. I checked the pricing of our e-mail providers and this will be the additional cost of sending more e-mails. What would you say if we tried sending manual weekly emails to 100 users for 3 weeks to see engagement before building the full feature? Then we could gather more data. Or if you’re sure then maybe we could target 5% improvement, and if that doesn’t happen in 2 months, we discuss whether to keep this feature?”

Managers like numbers, and when we speak to them using business terms. Even if they don’t like it when we push back, they appreciate it when they can use our numbers to discuss with their managers. As managers of managers, they appreciate it when people provide them with precise information.

Of course, can’t tell that it works, but you can at least try, aye?

So when business tells you “Bring me solutions, not problems!” - we should respond “Bring me problems, not solutions!“. Make them define the actual problem before you build anything.

The positive outcome of such metrics and measuring usage is that when we discover we haven’t reached the expected outcomes, we may decide to drop this feature and not maintain it at all.

We should not treat removing features as our personal failure, but we should accept the evolution of our software.

We should accept that our current work has a finite lifespan and acknowledge that even if we remove something, it could bring us business value.

This is the essence of prioritising removability. Success isn’t measured by how long code lasts, but by the value it creates during its lifetime

If we understand our product and the environment we’re in, it’d be easier to break it down into removable parts.

Funnily, if we remove stuff, then we won’t need to maintain, and those that are left will also be easier to maintain as they will have naturally shaped boundaries.

Read also more in:

Cheers!

Oskar

Residuality Theory: A Rebellious Take on Building Systems That Actually Survive

I was sitting in a dark room at NDC Oslo when Barry O’Reilly started talking about lightbulbs. One hundred thousand of them, to be exact, all interconnected, each one either on or off. It was an odd way to start a talk about software architecture, but by the end, my mind was blown. What he presented wasn’t just another framework—it was a completely different way of thinking about how we build systems.

For years, I’ve advocated for proper risk analysis. I wrote about it in “The Risk of Ignoring Risks”—how we need to think upfront about finding risks for our solutions, probability assessments, and mitigation plans. I still believe in all of that. But Barry’s Residuality Theory added a dimension I hadn’t considered.

Taking risks is okay as long as we’re prepared for them. In the article mentioned, I explained the risk register, a tool that can be helpful for analysing risks. It is a simple table where we write down all risks associated with our solution.

We write the probability in the lines and the degree of risk in the columns. By multiplying these two data, we get a result that tells us how much we should focus on a given risk. If we find out that our risks are very probable and the consequences of their occurrence are severe, we need to take action and make a corrective design decision upfront.

If the risk is unlikely and has little consequence, we can treat it lightly. Usually, however, it is somewhere in the middle. We should write down what we will do when the risk occurs (e.g., when we have 100,000 requests per second).

Risk analysis assumes you can identify what might go wrong. You list payment processor outages, network failures, scaling issues. You rate them by probability and impact. You prepare.

But what about the things you never thought of? What about when TikTok makes your coffee shop go viral? What about when a health inspector shows up during your morning rush? What about when your competitor opens next door with half-price drinks?

Residuality Theory asks a different question.

Instead of:

“What risks should we prepare for?”

It asks:

“What happens to our system when ANY stress hits it?”

The difference is subtle but profound. Risk Matrices help us prepare for known unknowns, while Residuality Theory helps us build systems that survive unknowns.

Barry O’Reilly developed the residuality theory, a rebellious approach to software architecture that focuses on designing systems that withstand unexpected stresses and changes. This isn’t just philosophy. It is based on actual complexity science. Those 10,000 lightbulbs represent any complex system where components influence each other. The key insight is that systems move from stable (predictable but rigid) to a critical point, becoming chaotic (unpredictable cascading failures) as connections between components increase.

Many software starts chaotic—everything depends on everything else. The goal is to reduce coupling and the chain reaction of failures, until you hit a sweet spot where your system can adapt without collapsing.

Unlike traditional architectural approaches that design systems from a static, component-based perspective, Residuality Theory views software architecture as a collection of “residues”, the elements that survive after a system experiences stress.

The fundamental concept of residuality theory is random simulation. In a complex business system, it is impossible to accurately make point predictions about what will change, when it will change, how it will change, how often, and what things will change along with it.

We can’t predict the future of our software systems, and anyone who claims they can should make you skeptical.

In Residuality theory, instead of trying to guess what might go wrong, we deliberately imagine random scenarios that could break our system, even outlandish ones we don’t think will happen.

Think of it as stress-testing your architecture with unexpected challenges:

What if our database provider goes bankrupt?
What if our main content creator removes all their material?
What if a regulatory change forces us to restructure our entire user data model?
What if Godzilla attacks our data centre? Yes, even that case.

By testing our architecture against a wide range of random stressors, we build systems that survive not just the problems we can anticipate, but also the ones we can’t imagine yet. Residuality theory pushes us past these mental blocks by making unexpected scenarios a central part of the design process.

The result: software that withstands surprises better than traditionally designed systems.

Residuality Theory has four main concepts:

Stressors - Unexpected events that challenge your system (technical failures, market changes, regulatory shifts),
Residues: The elements of your design that survive after stressors hit,
Attractors: States that systems naturally tend toward when under stress,
Incidence Matrix: A tool to visualise relationships between stressors and components

The general process looks as follows:

Brainstorm a wide range of potential stressors (including extreme ones).
Map these stressors against your system’s components.
Identify which components are most vulnerable.
Design solutions that will survive the stressors.
Test your design against new, unforeseen stressors.
Rince & repeat!

Let’s try it by example!

Imagine a local coffee shop chain building a mobile app where customers can order and pay for coffee in advance and pick it up when ready.

The Goal is simple: reduce wait times and in-store crowding.

The traditional approach would be to design each component. We could come up with something like:

Order Intake - receiving orders,
Payment Processing - charging customers,
Kitchen Queue - showing baristas what to make
Pickup Flow - customer collection.

With Residuality Theory, you start by imagining stressors, all kinds of them, even ridiculous ones. The list could look as follows:

Payment provider outage (Stripe API down)
Shop internet fails (router/ISP issue)
Barista drops tablet (screen shattered)
Monday 7:45 AM rush (40 orders in 15 minutes)
Customer no-shows (orders made but not collected)
Competing shop offers 50% off (customer flight)
Health inspector arrives during rush (disruption)
TikTok makes our “secret menu” viral (unexpected demand)

Next, you map what breaks when each stressor hits. This can be eye-opening. When the shop’s internet fails, orders can’t come in, payments can’t be processed, the kitchen can’t see orders, and customers don’t get notifications. Everything fails together. The same is true for Monday rush - total system meltdown. Or when our local coffee chain goes viral thanks to some TikTok influencer.

The pattern becomes obvious. Your beautifully designed components are so tightly coupled that certain events cause complete collapse.

In Barry’s terms, your system has the Attractor - a state it naturally falls into - called “everything is on fire.” So yes, the negative one.

You can see that clearly when filling in the Incident Matrix:

States that systems naturally tend toward when under stress

Total Meltdown - Internet failure or viral rush affects all components (rows with 4s),
Payment-Order Death Spiral - Payment and orders always fail together (coupled in 5/8 stressors),
Kitchen Blindness - Orders paid, but the kitchen can’t see them,
Waste Accumulation - System works, but no-shows create hidden losses

Now, here’s where it gets interesting. Traditional thinking says add redundancy, get backup internet, beefier servers, and more staff. With Residuality Theory, you should still consider that. Yet the focus is different: redesign so components can fail independently.

The solutions, what Barry calls Residues, aren’t always technical. That internet outage? Sure, you could get expensive backup connections.

Or you could create “Internet Down Tuesday”, when systems fail. Walk-ins get 20% off, and the failure could be turned into a marketing event.

Is the morning rush overwhelming everything? Don’t try to scale all systems. Create an express menu from 7 to 9 a.m. with only five drinks. A simpler menu means faster service and less system load. The constraint becomes a feature.

Customers ordering but not picking up? Instead of eating the loss, create a 15-minute rule where unclaimed drinks go to waiting customers who need a pick-me-up. Waste becomes community goodwill.

After applying these changes, measure again. The tight coupling is broken. The updated Incident Matrix could look as follows:

We added some new components:

Technical:
- Dual payment providers (Stripe + Square),
- Receipt printer backup,
- Staff hotspot failover.
Business:
- “Internet Down Tuesday” - 20% off walk-ins when the system fails,
- Express menu 7-9 am (only 5 drinks available),
- 15-minute auto-donation policy for unclaimed drinks
- WiFi sharing agreement with the bookstore next door,
- “Regulars board” - an analogue backup for frequent customers

Thanks to that, we not only tech-arounded the issue, but we also changed the overall process to be resilient. Rush hour doesn’t crash everything because the express menu prevents overload. Internet outages don’t cascade because you’ve accepted failure as a valid state and made it profitable. Payment provider issues only affect payments, not the entire order flow.

This isn’t about preventing failures. It’s about building systems that degrade gracefully, sometimes benefiting from failure. The express menu you created for rush hour could become your differentiator against competitors. The community cup program builds customer loyalty.

If we repeat this analysis again and again, we’ll notice that we also have positive attractors. Some of the Residues that came up as a result of solving the stressors’ impact are also solving cases for the new ones!

Summary

Of course, all of that requires collaboration between IT and business. Still, nowadays, IT is Business. The old split is obsolete already. If we don’t work together but remain in our positions, throwing the ball over the fence, we’ll play the telephone game instead of the real design.

Design is not created in a vacuum.

Don’t be an architecture astronaut. Don’t work in a vacuum.

In my opinion, Residuality Theory doesn’t kill risk analysis. We still need to find them, but without limiting ourselves and claiming that we know what we don’t know. It forces us to do our homework. What components fail together? What states does my system naturally fall into? What survives when things go sideways?

This approach acknowledges something important: we can’t predict everything. The world is messy. Competitors appear, social media goes viral, inspectors show up, and equipment breaks. Instead of pretending we can plan for it all, we build systems that adapt and survive.

We should stop pretending we can control everything.

Risk analysis says:

“Prepare for what might happen.”

Residuality Theory says:

“Build systems that handle what you didn’t prepare for.”

Since that talk, I’ve started looking at systems differently. I started thinking more about the simulation aspect. It’s also kind of more aligned with how other architects work. The real architects work: before they build something, they perform stress analysis to understand if the building won’t fall.

I’m not an expert in Residuality Theory. I hope to see Barry within the next two days at Techorama and that he won’t bash my humble notes here!

You should check out Barry’s talk and read his book Residues: Time, Change, and Uncertainty in Software Architecture.

And most importantly, play with it, make dry runs, just like I did in this article. It’s a fun exercise that can bring you many insights into our work and your system design.

When you build for chaos, you stop fearing it!

Check also my other article in a similar spirit to what I described today:

Cheers!

Oskar

The Underestimated Power of Hot Spots and Notes in EventStorming

I’ve been facilitating the EventStorming sessions for years now, and something keeps catching my attention.

While teams naturally focus on mapping out those orange event sticky notes (the backbone of any EventStorming session), they often underestimate the power of two critical elements: Hot Spots and Notes.

For those less familiar with EventStorming, it’s a workshop technique developed by Alberto Brandolini that helps teams map complex business domains. You gather relevant people in a room, roll out a long paper on the wall, and start mapping out what happens in your system using different colored sticky notes. Orange ones represent domain events - the facts that have happened in your system and that domain experts care about. These events form the primary narrative of your process.

Who are “relevant people”? Obviously, domain experts, decision makers, and we, technical people, who’ll be trying to solve business problems with 0s and 1s.

If you’re looking for a quick intro, check this talk by Kenny Baas-Schwegler:

Still, no intros today, as today, I want to discuss the red and white sticky notes that don’t always receive the recognition they deserve.

Why Meetings Get Stuck

Most of us have been in technical meetings that start with good intentions but descend into circular discussions. You gather smart people in a room to map out a process, but an hour in, everyone’s debating some detail nobody fully understands. Half the room checks emails two hours in, while the other half argues about edge cases.

Or all people are engaged and moving along smoothly until we hit some complicated requirements. The discussion can quickly devolve into speculation. For example, nobody in the room knows the exact regulatory requirements, but instead of admitting this, people make conflicting assertions with increasing confidence. Forty minutes later, zero progress is made and the energy in the room is killed.

I hit this problem repeatedly in my early EventStorming sessions until I started taking Hot Spots and Notes more seriously.

Hot Spots to the rescue

Hot Spots are those bright red sticky notes that mark areas of uncertainty or disagreement.

Notes, typically white, capture assumptions, decisions, or context discovered during the session.

Source: https://github.com/ddd-crew/eventstorming-glossary-cheat-sheet?tab=readme-ov-file#core-concepts

What makes these tools valuable isn’t that they’re somehow more important than events (they’re not!), it’s that they give teams a structured way to handle the unknown without derailing progress.

When I started explicitly encouraging teams to use Hot Spots, sessions immediately became more productive. Instead of pretending to know everything, team members would place a Hot Spot on the timeline whenever they hit uncertainty:

“Not sure if this requires manual approval”

Hot Spot!

“Do we need to store this data for compliance?”

Hot Spot!

“System integration point unclear”

Hot Spot!

This small change had profound effects. Sessions that previously ground to a halt now maintained momentum. We could acknowledge gaps without getting stuck in them.

The Power of Documented Assumptions

Using the derailed workshop example mentioned earlier, it could be handled differently if we put more effort into noting issues and assumptions.

When the regulatory question was hit, someone could put a Hot Spot labelled:

“Compliance requirements for trade verification?”

Then, a note was added stating:

“Assuming 24-hour verification window is required for now.”

That ten-second action could save an hour of circular discussion. We would document our uncertainty and our working assumption, then continue mapping.

When the compliance team later clarified the actual requirement (e.g., 48 hours for certain trade types), we would have a clear record of how this impacted our model. The risk of proceeding with an assumption would still pay off by maintaining session momentum.

“If It’s Not on the Board, It Doesn’t Exist!”

I’ve watched too many important decisions evaporate as soon as meetings end. A team spends twenty minutes reaching an agreement, but since nobody wrote it down, they end up having the exact same conversation weeks later.

It’s worth having a rotation seat for the note-taker. Of course, everyone can (and should be putting them), but someone who focuses on ensuring that all agreements are on board is essential.

Then, you’re ensuring that good, long discussions that end in consensus or compromise won’t be lost. Whenever something important is said or agreed upon, capture it on a white Note and put an arrow to which it relates. These can be invaluable during implementation, preventing numerous repeated debates.

Thanks to that, it could save you some endless meetings. When you start to argue about something, someone can point to the wall and say:

”We already decided this three weeks ago.‘”

Working Around Missing Stakeholders

Let’s be honest, it’s hard to get everyone with crucial domain knowledge in the same room for a full day. Subject matter experts are usually the busiest people in the organization.

I faced this with multiple teams redesigning their processes. Managers who understood the requirements could only be available in a short timeframe. Rather than trying to cram everything into that half-hour, you could run your session without him, but place Hot Spots everywhere we had questions only he could answer.

Of course, that’s far from perfect. You’re risking that your assumptions will be invalidated and you demotivated. Still, when you’re not new to the domain, but have already spent some time, then those chances will get smaller. It’s still better to have progress than tell “Business Won’t Let Me!” or “We don’t have time for that!”.

Management should appreciate your efforts and your respect for their time. The result could be that you could have a focused discussion when they finally get their time. Instead of wasting management’s limited availability with a rambling overview, you could point to specific Hot Spots:

“We need to understand exactly when reconciliation happens,”

“Is manual approval required for settlements over €50,000?”

The Hot Spots should become a question collection mechanism, ensuring you make the most of limited stakeholder time.

Getting Stakeholders Interested

And here’s something unexpected I discovered: Hot Spots are great for bringing reluctant stakeholders into the process.

A team I was coaching couldn’t get their Product Managers to participate. They had mapped their system but were stuck on several critical decisions.

Instead of giving up, they brought the EventStorming artefacts to his office, with Hot Spots clearly marking where they needed his input. Seeing the complexity visualised - and the specific places where his expertise was needed - changed his perspective entirely. The visual representation communicated the challenges in a way no meeting agenda could have.

Not only did he help resolve our immediate questions, but they also joined the next session, bringing insights that significantly improved their model.

Making This Work in Practice

If you want to get the most from Hot Spots and Notes in your sessions:

Put the sticky notes where everyone can reach them. I distribute red and white stickies at multiple points along the modelling wall and explicitly tell participants (repeatedly!) they can add them anytime without asking permission.

Start with more rather than fewer. It’s easier to close a Hot Spot that turns out to be simple than to recover from a derailed session because someone pretended to know something they didn’t.

Review them periodically during the session. I typically pause after completing major process sections to look at accumulated Hot Spots. This helps determine which we can resolve immediately, which need investigation, and which might affect areas we haven’t modelled yet.

Assign owners to post-session Hot Spots. Make sure someone is responsible for finding answers to unresolved questions before they become implementation blockers.

Include Hot Spots and Notes in your documentation. These often contain the most crucial insights from the session - the areas of uncertainty or assumptions that could make or break your implementation.

Changing How Teams Handle Uncertainty

What I value most about Hot Spots and Notes isn’t just how they improve workshops - it’s how they change team culture around uncertainty.

In technical discussions, we often feel pressure to know everything immediately. Hot Spots create permission to say, “I don’t know yet” while still making progress. They transform uncertainties from conversation-killers into clearly defined next steps.

I’ve seen teams evolve from hiding what they don’t know to actively hunting for uncertainties as valuable information. Questions become assets rather than liabilities.

If you’re incorporating EventStorming into your toolkit, don’t underestimate these powerful elements. They might not get as much attention as domain events, but in my experience, they often separate a productive modelling session from a frustrating stalemate.

Check also my other resources around EventStorming:

And 15 tips on how to run meetings effectively.

Cheers!

Oskar

Hints on getting to speak at conferences

Occasionally, I’m asked how to get to conferences (especially the bigger ones).

My secret recipe is to: just try.

Many people are so afraid of being rejected that they’re paralysed. Impostor syndrome is kicking in. Then thoughts: ” Those speakers know stuff much better,” or maybe “You need to know The People to get into those conferences.” That may be true, but…

…but the magic is that there’s no magic. If you don’t try, if you won’t send abstracts, if you won’t be rejected numerous times, then you won’t learn and won’t get accepted.

To prove it to you, look at the screens below. I did them from Sessionize.com - the most popular application for sending conference abstracts. I have been nominated as the most active speaker for the last two years.

And hey, I should have also got the most rejected speaker badge.

So I’m not sure if I could give some advice on how to be accepted, but if I could, besides “just try”, then:

1. Work on the abstract and tell precisely what you want to show. If you think that doing that will show too much, think twice about whether it’s a really good topic for a talk.

2. Ensure that your talk is aligned with the conference leitmotif. For instance: sending C# related talk to Java conference won’t help. Same goes for sending talk related to specific tool or technology to architecture track or modelling conference. Align your topic with the audience. Or just don’t send it there.

3. Don’t use GenAI tools to write it. Make a personal touch.

4. When you write it, ask a few folks what they think and how they understand it.

5. Make it engaging, not boring and not too lengthy. Still, don’t make cheesy jokes, don’t be jumpy. You need to balance expressiveness with being on point.

6. If you don’t know how to do it, look at the abstracts of the talks that were accepted in previous years. Don’t look at celebrities, but at folks who are closer to you.

7. It’s best to back your abstract with links to your articles or recordings so people can check how you explain stuff and what you’ll show.

8. Don’t be afraid to be precise about your special sauce. Many submissions write something like “I’ll show you what solution worked out for us and how we solved it”. Even though you know what it was, and believe it was great, people selecting the talks don’t know it. Explain briefly the solution and why it’s worth showing, and use examples. If you’re afraid that you’ll show the audience too much of the essence in two sentences, then again, ask yourself if it’s really a topic for a talk.

9. Before getting to a big stage, it’s good to make your way through user groups and smaller conferences. It helps you get used to the stage, gather feedback, hone your speaking skills, and get your name recognised. Having such experience will help you to be accepted.

10. Don’t take rejection personally. As you saw, I’m constantly rejected, there are conferences that are always rejecting me. Some conferences have hundreds of submissions, you not always need to have a good talk, good abstract but also luck. So it’s not about you, it may be about abstract, but not about you. If you want to get there: analyse what you coud do better, try again, rinse, repeat.

**And most importantly: just try.

Rejections will happen, but acceptance also. Sooner or later.**

At least that’s how it was for me.

And hey, if you’re at the conference, be it as an attendee or a speaker, I’m happy to talk with you in person. I’ll be speaking at three conferences next month on three conferences:

See you there?

Cheers!
Oskar

Why We Should Learn Multiple Programming Languages

In a recent Discord chat, I heard the phrase that:

A good senior developer should be able to program in any language.

As someone who’s spent years working across multiple tech stacks, this got me thinking. Is language flexibility a core trait of senior engineers, or is deep specialisation more valuable?

During my almost 18-year career, I’ve used numerous languages, such as C#, Java, JavaScript/TypeScript, C++, Ruby, PHP, Python, Perl, and ActionScript. I’ve also played with even more. I’m not flexing, I’m not a polyglot programming wizard. That’s just how it goes if you’re long enough in this industry.

Back in 1999, Andrew Hunt and David Thomas advised developers in “The Pragmatic Programmer” book to “learn at least one new language every year.” I think that’s too far, as we should not fall into the trap of pushing ourselves too much, but… This wasn’t just about padding our resume but expanding our thinking. Different languages embody different approaches to solving problems, and exposure to these approaches enriches our toolbox regardless of your day-to-day language.

Source: https://commons.wikimedia.org/wiki/File:Pieter\_Bruegel\_the\_Elder\_-\_The\_Tower\_of\_Babel\_(Vienna)\_-\_Google\_Art\_Project.jpg

Without this exposure, we can end up “stewing in our own juices,” unable to see alternatives. I’ve witnessed this repeatedly, for instance, with backend developers who construct elaborate workarounds to avoid touching JavaScript, not realising they’re making their lives harder, not easier.

And hey, I’ve been there, I’ve been a JavaScript hater, even worse for TypeScript, and I left the project when I was forced to code in Perl. Still, I transitioned, and I’ve made my journey.

Looking back, each time I’ve learned a new language, I’ve brought back new insights to my primary stack. I just believe that having proficiency in more than just your favourite language opens doors to new ways of thinking.

Learning the Language vs. Speaking It Fluently

When developers switch languages, they bring their habits with them, for better or worse.

Look at TypeScript code written by longtime Java or C# developers. You’ll often see unnecessary class hierarchies, dependency injections, and repository patterns when simpler approaches would work better. They’ve learned the syntax but not the philosophy.

I’ve watched .NET and Java programmers struggle with Node.js because they’re trying to force it into patterns that don’t fit. Node.js is architecturally more similar to Go’s minimalism than enterprise C# or Java, but they don’t see that. They’re writing C# with a different syntax.

It’s like spoken languages - learning classroom English is different from speaking conversational English. Learning syntax doesn’t mean you’re writing idiomatic code that leverages the language’s strengths.

Transitioning takes time. You might learn Go’s syntax in a week, but understanding its ecosystem and idioms is a much longer journey. Most people reach a productivity plateau after 3-6 months, but true fluency takes longer.

When It Makes Sense To Add A New Language

I’ve seen cases where adding a new language to the stack was worth it. The reasons were rarely related to the specific technologies, but rather to the state of the industry.

There are legitimate cases where a new language makes sense:

When the specific business problem demands it. Sometimes, specific services benefit from being written in a particular language. For example, when working with Node.js and needing CPU-intensive operations (like TypeScript migrations), it makes sense to use a more performant language for that specific component. See more in TypeScript Migrates to Go: What’s Really Behind That 10x Performance Claim?

When it helps with hiring and retention, I’ve had several clients who migrated from C#/Java to Node.js because they found it easier to hire developers. The systems they were building didn’t actually require the ceremonial complexity that C#/Java provides; there was no business advantage beyond familiarity.

When it opens up new career opportunities. I personally benefited from refreshing my Java skills at one point. Most of my clients are Java, so that investment has paid off significantly in my consulting work. Some languages are also becoming obsolete. They’re still needed, but the market is shrinking (cheers, Cobol!).

But let’s be honest - not every technology experiment belongs in production. The key question is rarely

“Can we write this in X?”

but

“Who will maintain this X code long-term?”

I once had to deal with a pricing algorithm module written in F#. While I actually like F# a lot, the decision wasn’t justified in this context. One of my first decisions after taking over was painfully rewriting it in C#. The marginal benefits didn’t outweigh the cognitive load on the rest of the team. It was a classical example of CV-driven design. The company burned money on a rewrite. But it was still cheaper than hiring or teaching new people F# to maintain it.

In another case, a team introduced a Python microservice because they thought it would be faster to develop. It turned out to be slower, leading to yet another rewrite. The business wasn’t even aware they were paying for learning experiments and subsequent cleanup rather than actual features.

These failures usually happen because people don’t ask the hard questions:

What’s the actual business driver for this change?

Is it worth spending 3-6 months of reduced productivity while the team learns something new?

Who will maintain this code if the champion leaves?

Even with good intentions, mixing technologies creates maintenance challenges. Small integrations in different technologies become business-critical risks. When key people leave, what should be a 15-minute fix can turn into weeks of investigation.

That’s why such decisions usually need to be consulted and confirmed at a higher level than those of the inner teams.

Cross the Frontend-Backend Divide

There’s one language-learning area I consider especially important: the frontend-backend divide.

If you’re a backend developer, learning some frontend technologies means you won’t limit yourself to just half of the feature delivery. You can help the frontend team when needed and understand the whole system better.

Similarly, frontend developers benefit from understanding what happens on the server. I’ve noticed backend developers often go to extreme lengths to avoid writing JavaScript, even when it’s the simplest solution. This resistance ultimately makes their job harder, not easier.

You don’t need to be an expert on both sides, but having enough knowledge to collaborate effectively improves everyone’s lives.

Going back to our English classroom example. As we learn it to be able to communicate, learn in our projects and not be limited only to our native language projects and resources. We should learn both about the backend and frontend stack and the frontend, so as not to limit ourselves. We still can specialise in one of them, but you should be able to deliver a feature end-to-end, getting a little help from others, instead of waiting for them.

See also my interview with Tomasz Ducin on Frontend Architecture, Backend Architecture or just Architecture?

“But I’m an Architect - Do I Still Need to Learn New Languages?”

My answer: absolutely, perhaps even more so.

Architecture decisions have long-lasting consequences. If you’re not hands-on with the technologies you recommend, how can you truly understand their tradeoffs? You’ll end up making decisions based on blog posts and conference talks rather than real-world experience.

The best architects I know still code regularly, even if not full-time, on production systems. They prototype in different languages, build proofs of concepts, and occasionally dive into codebases to understand pain points. They recognise that staying technically sharp is part of the job.

Without practical knowledge of multiple languages and ecosystems, how can you evaluate when a new technology is appropriate versus when it’s just adding unnecessary complexity? How do you distinguish between legitimate technical concerns and simple resistance to change?

Architects who stop learning new technologies quickly lose touch with the realities of modern development, and their teams suffer for it.

Finding Your Balance

So where does this leave us? Should every senior developer be capable of programming in any language? I think that’s overstating it.

It’s more like: a senior developer should have the mental models to adapt to new languages relatively quickly, while recognizing that true mastery takes time and immersion.

The challenge isn’t the language syntax but the frameworks, libraries, and idioms around it. You might learn Go in a week, but that doesn’t mean you’re writing idiomatic Go code. You’re likely writing Java-style Go, which misses the point entirely.

For organizations, the calculus is a bit different:

For stable, long-term projects: Maintaining technological consistency usually trumps introducing new languages
For teams facing specific technical challenges: Targeted use of specialized languages may be worth the investment
For everyone: Having a process for evaluating when to introduce new technologies is crucial

The key business question often gets overlooked: What’s the actual driver for introducing a new language? Is it worth spending 3-6 months of reduced productivity while a team learns something new? Sometimes yes, but often no.

The question shouldn’t be

“Can we write this in X?”

but,

“Who will maintain this X code long-term?”

Investing in Growth, Mindfully

Learning new languages represents an investment, both personal and organisational. While syntax might come quickly, ecosystem familiarity takes longer. Even after spending years exploring a new language, many developers don’t feel ready to join a project in that language at a senior level.

Yet every senior developer I respect maintains curiosity about other approaches. They might not be experts in five languages, but they’re familiar enough with alternatives to know when their primary tool isn’t the best fit.

This isn’t about chasing trends or padding resumes. It’s about expanding your mental models so you can recognize patterns across languages and ecosystems. It’s about understanding that the “obvious solution” in your primary language might be fighting against the grain in another.

And perhaps most importantly, it’s about maintaining the humility to recognize when you’re forcing a problem into your favorite solution rather than finding the right approach for the challenge at hand.

Because ultimately, programming languages are tools, not identities. The best developers I know don’t call themselves “Java programmers” or “JavaScript developers”; they’re problem solvers who use the right tool for the job, whether it’s the one they’ve used for a decade or the one they started learning last month.

What are your thoughts?

Cheers!

Oskar

Business Won't Let Me and other lies we tell to ourselves

“Business won’t let us implement proper telemetry.”

“We can’t write tests because there’s no time.”

“The manager won’t approve refactoring.”

“We don’t have budget for proper CI/CD.”

“The deadline doesn’t allow for documenting the API.”

These excuses appear in development teams with the regularity of standups and coffee breaks. You’ve heard that numerous times, and I bet you also said it. Why? Because I did both.

Those phrases reveal a fundamental confusion about where business decisions end and technical decisions begin. They often mask our reluctance to take responsibility for our actions.

Whose Decision Is It Anyway?

Let’s cut to the chase: technical experts should make technical decisions. That’s us. Or at least that’s our role.

You don’t ask your manager for permission to create a new file, name a variable, or start a debugging session. You just do it. So why are we asking permission for implementing telemetry, writing tests, or cleaning up code?

Delivering a feature doesn’t end with written code; this is just a part of it.

Source: https://www.monkeyuser.com/2020/feature-complete/

Like in the comic above, we often treat all other aspects as annoyances. That’s also how we explain it to the business people.

Besides typing words on the keyboard, we should ensure that they make sense, form a feature of proper quality, that we can safely deploy them, and then later observe their behaviour. So testing, adding telemetry, etc., is part of the coding process; it shouldn’t be treated as some tedious additional stuff we need to do.

Mark Seeman’s article “The Programmer as Decision Maker” nails this point. As developers, we make hundreds of technical decisions daily. Every variable name, code structure, and design pattern choice is a technical decision we make without consulting management. And rightly so. That’s not just part of our job, it IS our job. We’re paid to apply our expertise to solve technical problems. We need to start acting like it.

Requirements, Not Permissions

The key distinction lies in understanding the difference between requirements and implementation details.

Business stakeholders should provide requirements and constraints, such as features, budgets, priorities and deadlines. They represent the “what” and “when” of our work, but they shouldn’t dictate the “how.” The implementation details and technical approach fall squarely within our domain as technical professionals.

When a product owner says:

“We need this feature by next month”

They probably don’t want to say: ignore quality (so “skip writing tests” or “don’t bother with proper error handling”). They’re giving you a constraint within which you, as the technical expert, must make appropriate judgments and trade-offs.

If you genuinely cannot deliver quality work within the given constraints, that’s when you go back to business, not to ask permission to “do it right.” Still, to present business consequences in business terms, they can understand and evaluate. For instance

“If we proceed as requested, we’ll likely see production outages within three months”

is something a business stakeholder can assess, or:

“We’ll be unable to quickly diagnose issues, resulting in extended downtime that will affect our SLAs”

speaks to business concerns in a way they can grasp. Especially if we back it up with the previous cases like that.

Notice this difference from:

“We need more time for writing unit tests.”

One focuses on the business impact that stakeholders can weigh against other priorities; the other asks them to evaluate a technical practice they’re not equipped to judge.

Then they can make a decision on changing the scope or delivery time.

When we frame our concerns in technical terms, we’re essentially asking non-technical people to make technical decisions, which is precisely what we then complain about.

The Role Confusion Problem

From my experience, when the roles and responsibilities between business and engineering are disturbed, the situation ends badly in the long run. The dysfunction typically follows a predictable pattern.

It begins innocently enough. A developer asks a business stakeholder whether they should add tests or monitoring to a feature. Not wanting to delay the timeline, the stakeholder says: “Let’s skip that for now.” The developer complies. The pattern repeats. Soon, the business stakeholder starts believing that making these technical decisions is part of their job. Meanwhile, developers, many of whom are oddly comfortable dodging accountability, keep pushing more technical choices upstairs.

Over time, this creates a perfect storm of dysfunction. Business leaders make increasingly complex technical decisions that they fundamentally don’t understand. Developers build systems they know are flawed but can conveniently blame on “business requirements.” The system gradually deteriorates under the weight of poor technical decisions. When it all eventually collapses, the finger-pointing begins in earnest. Nobody wins.

The truth is that this dynamic serves a perverse purpose for both sides. Business gets to feel in control of all aspects of the product, while developers get to avoid the hard work of making tough technical trade-offs under constraint. It’s easier to implement what you’re told, especially when you can blame the outcome on someone else’s decisions.

The Responsibility Dodge

The core issue here isn’t really about business overreach. It’s about developers actively looking for ways to avoid responsibility for the systems they build.

“Business won’t let me” is the perfect excuse in many ways.

It’s practically unverifiable in the moment. Who’s going to call an emergency meeting to confirm whether the business actually prohibits proper error handling? It conveniently shifts blame upward to people who likely won’t understand the technical implications anyway.

Moreover, it positions you as the helpless victim rather than the empowered expert you’re supposed to be. It becomes a ready-made get-out-of-jail-free card when the system inevitably falls apart.

This responsibility avoidance tactic is so common because it works on multiple levels. It prevents you from doing extra work now – implementing proper monitoring or writing thorough tests takes time, after all. It shields you from blame later when problems arise, because you “wanted to do it right”, aye? It requires no confrontation or difficult conversations about trade-offs and priorities. And perhaps most comfortably, it lets you maintain the self-image of a “good developer” who “knows better” but is constrained by clueless managers who just don’t understand quality.

Here’s the uncomfortable truth: if you build a technical solution you know is flawed, you own that failure. You’re supposed to be the expert. Your job isn’t just to code what you’re told—it’s to find the best technical approach within given constraints. Sometimes that means making difficult trade-offs. Sometimes it means having uncomfortable conversations about what’s truly necessary versus what can be deferred. But it always means taking responsibility for the technical implementation, not passing that responsibility upstairs.

Read also more in:

Finding the Real Problem

In my article “Bring me problems, not solutions!” I discussed how business stakeholders often present us with solutions instead of problems. A classic example is:

“We need a web app for recording working hours.”

when the actual problem might be:

“HR needs accurate time records to process payroll.”

This happens constantly with technical decisions, too, but in reverse—we interpret business requirements as technical directives.

When someone says:

“We don’t have time for telemetry!”

They’re not actually talking about telemetry. They probably don’t even know what telemetry is. What they’re saying is:

“We’re worried about the deadline.”

That’s the real problem – a concern about timely delivery. Your job as a technical expert is to recognise this translation error. Once you understand the actual concern—meeting a deadline—you can make informed trade-offs that address both the business need and technical integrity.

Perhaps you scale back the feature itself rather than the quality of its implementation. Maybe you implement it in phases, with the most critical functionality delivered first. Perhaps you can find ways to streamline other aspects of the work.

The point is, you’re making these decisions based on your technical expertise rather than asking business people to make them for you or, worse, assuming they’ve already made them through their requirements.

Making Trade-offs, Not Excuses

When faced with constraints, we should not throw up our hands and say: “Business won’t let me do it right.” We should analyse requirements and make appropriate trade-offs, using our technical expertise to guide those decisions. This is fundamentally different from simply cutting corners or delivering poor quality.

Consider what’s necessary for system stability and reliability. Some aspects of software development aren’t optional luxuries, they’re essential to a functioning system. Basic error handling, data validation, and security controls aren’t nice-to-haves. They’re fundamental. On the other hand, some features can be simplified while still providing value. Perhaps an advanced search can be trimmed back to basic functionality initially. Maybe a dashboard can display essential metrics first, with additional views later.

Some work can be deferred without decreasing quality; not all technical improvements need to happen immediately. And often, there’s a minimal viable version that would meet the immediate need while allowing for future enhancement. These are the kinds of nuanced trade-offs that technical experts should make.

This isn’t about cutting corners. It’s about making expert judgments about what’s technically crucial given the constraints. Sometimes you’ll need to scale back features, adjust timelines, or implement in phases—but you make these decisions based on your technical expertise, not by asking non-technical stakeholders to make them for you.

If you’re a senior developer or architect, take a hard look at what decisions you’re punting upstairs that should be yours.

Are you hiding behind “business won’t let me” when you should be saying “I’m responsible for making this work within our constraints”?

Technical leadership isn’t just about knowing the right technical choice, it’s about having the backbone to stand behind it. It’s about using business requirements and constraints as parameters within which you operate, not excuses for poor quality.

It’s about making expert technical decisions within those constraints, applying your knowledge and experience to find the best approach possible given the limitations.

It means owning those decisions and their consequences, both good and bad. When things go well, you share the credit. When they go poorly, you accept responsibility rather than pointing fingers at “business requirements.” It requires communicating regarding business impact, not technical details, so stakeholders can understand the actual consequences of various options without evaluating technical practices themselves.

And yes, it means having uncomfortable conversations about genuine business risks when necessary.

Sometimes, despite your best efforts, the constraints don’t allow for a viable solution. In those cases, you need the courage to clearly articulate the business risks in business terms, not as a request for permission to “do things right,” but as vital information for business decision-making.

As Grace Hopper famously said:

“It’s easier to ask forgiveness than it is to get permission.”

So, own your technical expertise.

Make decisions with confidence.

And for everyone’s sake, stop blaming “business” for your technical compromises.

Cheers!

Oskar

PostgreSQL JSONB - Powerful Storage for Semi-Structured Data

Object-oriented or relational? You’ve seen this battle in your projects. For many years, we tried to fit the business data, which is usually grouped by business use case, into a normalised table structure. Sometimes it fits better, sometimes worse. We learn to cheat it with Object-Relational Mappers. They fixed some issues but created others.

Then document databases appeared, trying to look at this perspective from a different angle. They tried to accept that real-world data is fuzzy and usually doesn’t follow the rigid structure, but it’s semi-structured. It’s the classic dilemma: rigid schemas that ensure data integrity but resist change, versus flexible document stores that adapt easily but sacrifice guarantees.

Still, those databases didn’t give us the consistency guarantees we were comfortable with, like transactions, isolation levels, etc.

PostgreSQL added the JSONB column type to answer the question: “Can we have both?”.

Then other databases like MySQL and SQLite followed.

Let’s discuss today how it works, and if it’s the actual answer.

Introduction

First, terminology: the B in JSON stands for Binary. PostgreSQL also has a JSON type, but they’re different implementations:

JSON stores data as text, requiring parsing on each operation.
JSONB uses a binary representation.

The binary format of JSONB enables efficient indexing and querying and eliminates the need for repeated parsing. The downside is that it takes up more storage space (as it stores not only text but also metadata) and loses the exact formatting and ordering of the original JSON input. That’s usually acceptable, but worth knowing.

JSONB Under the Hood: Not Just JSON Storage

You can think of a JSONB column as a table inside the column. Each path to property can be seen logically as a different column.

To visualise this conceptual model, imagine if our database could access any path directly without scanning the entire document:

+----+---------------+--------------------------+--------------------+
| id | customer.name | customer.contact.address | email              |
+----+---------------+--------------------------+--------------------+
| 1  | "John Smith"  | "Portland"               | "john@example.com" |
+----+---------------+--------------------------+--------------------+

Of course, that’s a bit of an oversimplification.

PostgreSQL doesn’t create these virtual columns, but the binary representation effectively allows it to access paths directly without scanning the whole document. This approach provides several benefits:

Path traversal is much faster than parsing text
Fields can be indexed individually
Query performance is more predictable

When Postgresql stores a JSONB document, it doesn’t simply dump JSON text into a field. It transforms the document into a binary representation.

Have a look on the following JSON:

{
  "customer": {
    "name": "John Smith",
    "contact": {
      "email": "john@example.com",
      "phone": "555-1234",
      "address": {
        "city": "Portland",
        "country": "USA"
      }
    }
  }
}

This hierarchical structure could be flattened into a set of paths for each nested structure:

Path: customer.name = "John Smith"
Path: customer.contact.email = "john@example.com"
Path: customer.contact.phone = "555-1234"
Path: customer.contact.address.city = "Portland"
Path: customer.contact.address.country = "USA"

The goal is: to performantly read a specific field without parsing the entire document. How does it achieve it? To understand it, let’s discuss…

PostgreSQL JSONB Implementation

Storage Internals

JSONB’s document is stored as a hierarchical, tree-like structure of key-value pairs, each containing metadata about its type and actual data. To enable versioning, it begins with a “magical” single-byte header that identifies the format’s version (currently version 1) and a header with metadata.

When new JSON is stored, it has to be parsed and converted from text to key-value pairs. This conversion happens through a process called “tokenisation“.

For example, using our customer data example:

{
  "id": "cust_web_123",
  "email": "web_user@example.com",
  "source": "website",
  "web_data": {
    "first_visit": "2023-03-12T15:22:47Z",
    "utm_source": "google_search"
  },
  "contact": {
    "phone": "+1234567890"
  }
}

Would be tokenised and stored with internal tree-like structures tracking:

Root: Object {
  Count: 5,
  Children: [
    Key: "id",
    Value: String "cust_web_123",

    Key: "email",
    Value: String "web_user@example.com",

    Key: "source",
    Value: String "website",

    Key: "web_data",
    Value: Object {
      Count: 2,
      Children: [
        Key: "first_visit",
        Value: String "2023-03-12T15:22:47Z",

        Key: "utm_source",
        Value: String "google_search"
      ]
    },

    Key: "contact",
    Value: Object {
      Count: 1,
      Children: [
        Key: "phone",
        Value: String "+1234567890"
      ]
    }
  ]
}

Of course, it’s a bit simplified form, but you can think that each array element is actually a different node, and the index in the array is just a nested path. Each key and value in a JSONB document is stored with its data type and metadata. For objects and arrays, PostgreSQL maintains counts and offsets to child elements.

PostgreSQL’s JSONB tokenisation is sneaky. It not only parses data but also preserves the actual data types of values. Each value in the binary format includes a type that identifies whether it’s a string, number, boolean, null, array, or object. Thanks to that PostgreSQL can:

ensure that data maintains its semantic meaning,
enable indexing,
enable type-specific operations (like numeric comparisons),
avoid type conversion when not needed.

What’s more, it’s not limited to JSON types. It distinguishes different numeric types (integers, floating-point) and handles string data with character encoding.

So, when you search for:

customer_data->'customer'->'contact'->'address'->>'city'

PostgreSQL can navigate directly to that specific token without scanning the entire document. It uses the described hierarchical structure, and gets the exact value with the type.

Internal Path Representation

PostgreSQL uses path extraction operators (-> and ->>) for querying the data inside JSON.

When PostgreSQL encounters such query, it:

Parses the path into individual segments.
Locates the root object or array in the binary structure.
Each path segment computes a hash of the key.
Uses the hash to look up the corresponding entry in the structure.
Navigates to the next level if needed.
Extracts and returns the value in the requested format.

This algorithm is heavily optimised for typical access patterns. For simple path expressions, PostgreSQL can retrieve values with near-constant time complexity, regardless of document size. However, the performance characteristics become more variable for more complex expressions, especially those involving arrays or filtering.

The algorithm also has specific optimisations for handling arrays differently from objects. Arrays are stored with implicit numeric keys, and PostgreSQL includes specialised operations for array traversal and element access. This becomes important when dealing with large arrays or performing operations like jsonb_array_elements().

Understanding this internal path representation explains why some JSONB queries perform better than others. Operations that align with PostgreSQL’s internal traversal algorithm (like direct path access to deeply nested values) can be remarkably fast, while operations that require scanning or restructuring (like complex filtering within arrays) might perform less optimally.

Read more in the documentation about JSON Functions and Operators.

JSONB in practice

To see how JSONB actually works, let’s discuss a scenario that manages customer data.

Here’s how a typical customer model might look in TypeScript:

interface Customer {
  id: string;
  email: string;
  source: string;
  name?: string;

  // Different structure depending on acquisition channel
  // At least one of these will be present per customer
  web_data?: {
    first_visit: string;
    utm_source: string;
    browser: string;
    pages_viewed: string[];
  };

  mobile_data?: {
    device: string;
    os_version: string;
    app_version: string;
    notification_token: string;
  };

  crm_data?: {
    account_manager: string;
    company: string;
    department: string;
    role: string;
    meeting_notes: string[];
  };

  contact: {
    phone?: string;
    address?: {
      city: string;
      country: string;
      postal_code?: string;
    };
  };

  purchases: Array<object>;
}

If you tried to model it with relational tables, you’d face a classic data modelling challenge. You’d end up with either:

A complex set of tables with joins: Customer_Web, Customer_Mobile, Customer_CRM with relationships between them.
A wide table with mostly NULL values: A single table with columns for every possible field across all sources.

With JSONB, you can implement this in PostgreSQL without forcing every customer profile into the same rigid structure:

CREATE TABLE customers (
    id TEXT PRIMARY KEY,
    email TEXT UNIQUE NOT NULL,

    customer_data JSONB NOT NULL
);

This allows storing profiles with wildly different shapes while keeping common fields queryable. The SQL implementation might look like:

INSERT INTO customers (id, email, customer_data) VALUES
-- Web customer
('cust_web_123', 'web_user@example.com',
'{
    "id": "cust_web_123",
    "email": "web_user@example.com",
    "source": "website",
    "name": "Alex Johnson",
    "web_data": {
        "first_visit": "2023-03-12T15:22:47Z",
        "utm_source": "google_search",
        "browser": "Chrome 98.0",
        "pages_viewed": ["/products", "/pricing", "/blog/postgresql-tips"]
    },
    "contact": {
        "phone": "+1234567890",
        "address": {
            "city": "Portland",
            "country": "USA",
            "postal_code": "97201"
        }
    },
    "purchases": [
        {
            "product_id": "hosting_basic",
            "amount": 9.99,
            "date": "2023-04-01",
            "subscription": true
        }
    ]
}'
),

-- Mobile customer with different structure
('cust_mob_456', 'mobile_user@example.com',
'{
    "id": "cust_mob_456",
    "email": "mobile_user@example.com",
    "source": "ios_app",
    "mobile_data": {
        "device": "iPhone 13",
        "os_version": "iOS 16.2",
        "app_version": "3.4.1",
        "notification_token": "eyJhbGdvcml0aG0iOiJITUFDLVNIQTI1NiIsIml"
    },
    "contact": {
        "address": {
            "city": "Berlin",
            "country": "Germany"
        }
    },
    "purchases": [
        {
            "item_id": "premium_subscription",
            "recurring": true,
            "interval": "monthly",
            "price": 14.99,
            "currency": "EUR",
            "payment_method": "apple_pay"
        },
        {
            "item_id": "in_app_coins",
            "package": "large",
            "quantity": 5000,
            "price": 49.99,
            "currency": "EUR",
            "payment_method": "apple_pay"
        }
    ]
}'
)

Despite the varying structures, you can still query across all customer types:

SELECT
    id,
    email,

    customer_data->>'source' AS source,

    COALESCE(
        customer_data->'web_data'->>'first_visit',
        customer_data->'mobile_data'->>'app_version',
        customer_data->'crm_data'->>'account_manager'
    ) AS source_specific_data,

    customer_data->'contact'->'address'->>'city' AS city,

    jsonb_array_length(customer_data->'purchases') AS purchase_count

FROM customers
WHERE (customer_data->'contact'->'address'->>'country') = 'USA';

Le’ts have a look first on_WHERE (customer_data->‘contact’->‘address’->>‘country’) = ‘USA’_ statement.

When getting such WHERE query, PostgreSQL does following steps:

Computes the path contact.address.country for each document
Extracts the string value at that path
Compares it to the literal ‘USA’

With a traditional JSON text column, this would require parsing the entire document. With JSONB, PostgreSQL can navigate directly to the relevant portion of the binary structure.

The data in the SELECT statement is processed similarly.

PostgreSQL also has syntactic sugar called JSON_TABLE see more in the documentation.

Indexing and Performance Optimisation

Thanks to the binary, hierarchical structure we just described, PostgreSQL can index JSONB data. It also offers several index types, each with distinct use cases. It’s worth knowing their pros and cons.

GIN Indexes

The GIN (Generalized Inverted Index) is PostgreSQL’s primary weapon for speeding up JSONB queries. A GIN index creates index entries for each key and value in your JSONB documents, making it particularly effective for containment checks and path existence.

Creating a GIN index is straightforward:

CREATE INDEX idx_customer_data ON customers USING GIN (customer_data);

When PostgreSQL builds this index, it:

Iterates through every JSONB document in the table
Extracts each key and value pair at every level of nesting
Creates index entries for each key and for each scalar value
Organises these entries in a tree structure optimised for lookups

The resulting index is usefull for certain operations like:

-- Find all customers with specific tags (contains operator)
SELECT id FROM customers WHERE customer_data @> '{"tags": ["premium"]}';

-- Find all customers with a specific field (existence operator)
SELECT id FROM customers WHERE customer_data ? 'promotion_code';

-- Find all customers with any of these fields (any existence operator)
SELECT id FROM customers WHERE customer_data ?| array['discount', 'coupon', 'promotion_code'];

These queries can use the GIN index directly without extracting paths, making them highly efficient even with large datasets. The execution plan would show something like:

Bitmap Heap Scan on customers  (cost=4.26..8.27 rows=1 width=32)
  Recheck Cond: (customer_data @> '{"tags": ["premium"]}'::jsonb)
  ->  Bitmap Index Scan on idx_customer_data  (cost=0.00..4.26 rows=1 width=0)
        Index Cond: (customer_data @> '{"tags": ["premium"]}'::jsonb)

However, GIN indexes have significant drawbacks:

They can be large, sometimes larger than the table itself.
They can slow down write operations.
They require more maintenance (vacuum).
They don’t help with range queries or sorting.

For each key or scalar value, the GIN index maintains a posting list of row IDs where that element appears. These posting lists are compressed and optimised for fast intersection operations, which is why containment checks are so efficient.

When PostgreSQL executes a containment query like:

customer_data @> '{"status": "active", "type": "business"}

It:

Looks up the posting list for the key “status” with value “active”
Looks up the posting list for the key “type” with value “business”
Computes the intersection of these posting lists
Returns the matching row IDs

This set-based operation is highly efficient compared to extracting and comparing values from each document.

B-Tree Indexes for Extracted Values

For simpler cases where you frequently filter on a specific JSON path, creating a B-Tree index on an extracted value is often more efficient:

-- Index for a specific extracted field
CREATE INDEX idx_customer_source ON customers ((customer_data->>'source'));

This creates a traditional B-Tree index on the extracted string value. Internally, PostgreSQL treats this as an expression index - it computes the expression (customer_data->>‘source’) for each row and indexes the result.

The execution plan for a query using this index would show:

Index Scan using idx_customer_source on customers  (cost=0.28..8.29 rows=1 width=32)
  Index Cond: ((customer_data->>'source'::text) = 'website'::text)

These indexes:

Are much smaller than GIN indexes,
Have less impact on write performance,
Support range queries and sorting,
Only help with queries that exactly match the indexed expression.

What’s more, you can also use it to set up unique constraints and get the same strong checks as for regular tables!

Understanding the internal mechanisms helps explain why you might want different index types for different query patterns.

Expression and Partial Indexes

For more specific query patterns, you can combine these approaches:

-- Location-based index for frequently filtered fields
CREATE INDEX idx_customer_country ON customers ((customer_data->'contact'->'address'->>'country'));

-- Specialized conditional index for specific customer types
CREATE INDEX idx_web_utm ON customers ((customer_data->'web_data'->>'utm_source'))
WHERE customer_data->>'source' = 'website';

The partial index only indexes rows where the condition customer_data->>‘source’ = ‘website’ is true. This makes the index much smaller while still accelerating queries for web customers:

-- Query that can use the partial index
SELECT id FROM customers
WHERE customer_data->>'source' = 'website'
AND customer_data->'web_data'->>'utm_source' = 'google';

Internally, PostgreSQL maintains separate metadata about which index portions apply to which queries. The query planner uses statistics about data distribution to decide whether to use an index or not, which explains why sometimes PostgreSQL might choose a sequential scan even when an index exists.

Suppose we have a table with 1 million customer records, and we frequently run queries to find customers from specific countries:

-- Without an index, this requires scanning all records
SELECT id FROM customers
WHERE customer_data->'contact'->'address'->>'country' = 'Germany';

The execution plan would show:

Seq Scan on customers  (cost=0.00..24053.00 rows=10000 width=32)
  Filter: ((customer_data->'contact'->'address'->>'country'::text) = 'Germany'::text)

After adding an appropriate index:

CREATE INDEX idx_country ON customers ((customer_data->'contact'->'address'->>'country'));

The execution plan becomes:

Index Scan using idx_country on customers  (cost=0.42..341.50 rows=10000 width=32)
  Index Cond: ((customer_data->'contact'->'address'->>'country'::text) = 'Germany'::text)

This can reduce query time from seconds to milliseconds. But the choice between GIN and B-Tree indexes isn’t always obvious. If we frequently need to find customers with specific combinations of attributes, a GIN index might be better:

-- Query that benefits from a GIN index
SELECT id FROM customers
WHERE customer_data @> '{"contact": {"address": {"country": "Germany"}}, "status": "active"}';

Performance is good enough for most typical cases, and if you don’t have a huge amount of data, the performance of JSONB with indexing is similar to traditional columns.

Still, if it’s worth knowing that JSONB:

Can perform worse for simple single-value lookups: Traditional columns with B-Tree indexes typically outperform JSONB path extraction,
can perform better for retrieving complete complex objects. Avoiding joins can lead to significant performance advantages,
Has variable performance for analytical queries: Depending on indexing strategy, can be either much faster or much slower than normalised data

While JSONB offers flexibility, it has performance characteristics you should understand, especially for large documents. PostgreSQL applies TOAST (The Oversized-Attribute Storage Technique) compression to JSONB documents exceeding 2KB, storing them outside the main table in a TOAST table. This introduces retrieval overhead, as accessing TOASTed data requires additional I/O operations and CPU time for decompression.

A common anti-pattern is storing large arrays or deeply nested structures in a single JSONB document. For example, storing an entire order history with hundreds of orders in a customer document might seem convenient, but it leads to document bloat, poor query performance, and contention issues. The same applies to storing the whole contents of text documents.

Other JSONB Features and Patterns

Unique Constraints

You don’t have to abandon all data validation when using JSONB. PostgreSQL allows you to enforce constraints on JSONB documents:

-- Ensuring required fields regardless of customer source
ALTER TABLE customers ADD CONSTRAINT valid_customer
CHECK (
    customer_data ? 'id' AND
    customer_data ? 'email' AND
    customer_data ? 'source' AND
    customer_data ? 'contact'
);

-- Source-specific validation
ALTER TABLE customers ADD CONSTRAINT valid_web_customer
CHECK (
    customer_data->>'source' != 'website' OR
    (customer_data ? 'web_data' AND customer_data->'web_data' ? 'first_visit')
);

These constraints leverage PostgreSQL’s ability to check path existence and values within the binary representation. Internally, PostgreSQL evaluates these expressions during insert and update operations, using the same path traversal algorithms that power queries.

JSON Path Queries

PostgreSQL 12+ introduces a powerful new way to work with JSONB through the SQL/JSON path language. This provides a more expressive syntax for complex data extraction:

SELECT jsonb_path_query(
    customer_data,
    '$.purchases[*] ? (@.amount > 10 && @.subscription == true)'
)
FROM customers;

This query finds all purchases that are subscriptions with an amount greater than 10. The path expression is evaluated against each document, and matching elements are returned.

Internally, PostgreSQL parses the path expression into an execution plan specific to JSON path traversal.

Aggregation for Heterogeneous Data

JSONB works well with PostgreSQL’s aggregation functions:

-- Find average purchase amount across all customers
SELECT AVG(
    (jsonb_array_elements(customer_data->'purchases')->>'amount')::numeric
)
FROM customers;

-- Group purchases by product type
SELECT
    COALESCE(p->>'product_id', p->>'item_id') AS product,
    COUNT(*)
FROM
    customers,
    jsonb_array_elements(customer_data->'purchases') AS p
GROUP BY
    product;

The jsonb_array_elements internally unwraps the binary representation of the array, extracting each element as a separate JSONB value. This enables the array to handle optimisations in binary format, making it efficient even for large arrays.

Hybrid Relational-Document Models

Of course, you can mix traditional columns with JSONB ones.

For those columns that you know will always exist, or are the same, you can set up regular tables, and those with weaker schema, you can use JSONB.

CREATE TABLE customers (
    id TEXT PRIMARY KEY,
    email TEXT UNIQUE NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT NOW(),
    source TEXT NOT NULL,
    -- Common fields as columns
    -- Variable parts as JSONB
    source_data JSONB NOT NULL,
    contact_info JSONB NOT NULL,
    purchases JSONB NOT NULL
);

This hybrid approach gives you the best of both worlds: schema enforcement where it matters and flexibility where needed. It’s often the most practical approach in real-world systems.

The hybrid aligns with PostgreSQL’s internal implementation. Traditional columns can use the highly optimised columnar storage and indexing mechanisms PostgreSQL has refined over decades, while JSONB columns leverage the specialised binary representation for flexible data.

Conclusion

JSONB represents a practical compromise between rigid relational schemas and completely schemaless document databases. It provides an excellent balance of flexibility and performance for most transactional data operations, and its filtering capabilities are sufficient for most business operations.

Of course, in scenarios requiring absolute top performance, like business analytics with complex filtering, data warehousing, or very high-throughput OLTP systems, traditional columns with specialised indexes often perform better. In practice, a hybrid approach works best: use traditional columns for fixed, frequently queried attributes and JSONB for variable parts of your data.

Big strength of JSONB is handling schema evolution. In traditional relational databases, schema changes often require adding columns with potential downtime, updating application code, and migrating existing data. These operations introduce risk and complexity, especially in production environments.

With JSONB, you can evolve your schema more gracefully. When business requirements change and you need to store additional data, you don’t need to modify the database schema. Your application can simply start including new fields in the JSON documents it writes, and older documents without those fields continue to work fine.

I’m biased, but I extremely like JSONB and have used it in the past in my system. It can speed up development for line-of-business applications. Syntax can be a bit tricky, but once you learn it, or provide some wrapper it works like a charm.

That’s alos why I created Pongo. It’s a JS/TS library that uses a Mongo-like API but uses Postgres underneath, leveraging JSONB data type to get strong consistency benefits. See: https://github.com/event-driven-io/pongo

Pongo treats PostgreSQL as a Document Database, benefiting from JSONB support. Unlike the plain text storage of the traditional JSON type, JSONB stores JSON data in a binary format. This simple change brings significant advantages in terms of performance and storage efficiency.

See also my video where I explained internals of both Pongo and JSONB:

Read also more in:

Cheers!

Oskar

Monolith-First - are you sure?

Welcome to the new week!

Our industry is highly related to fashion and trends. Trends set up by the tech influencers promoting some ideas, quite often that they haven’t even tried to apply in the real system. We’re the guinypigs.

Every few years, our industry rallies around a new architectural approach that promises to solve all our problems. A few years ago, we all wanted to be micro and distributed. Microservices for the win!

Lately, the pendulum swung, and we hear the “modular monolith first!”.

Or: “If you cannot deal with monolith, microservices won’t work for you!”.

I wrote an article some time ago about the hardship of breaking down monoliths into microservices and asked whether that makes sense in most cases. That could look like following the monolith-first gold advice, but actually it was not.

It sounds like our industry is filled with Siths, as we’re dealing with absolute blank statements. They’re nice hot takes to tell on social media, promote on Youtube or use in an online course, but the reality is much more complex.

On paper, it sounds perfect: start with a single codebase divided into clear modules, enjoy the simplicity of a monolith, and preserve the option to split into microservices later. But…

Simple vs Simplistic

Let’s acknowledge the appeal first. You get a single deployment pipeline, avoiding the distributed systems’ complexity tax. Your developers can navigate the entire codebase without jumping between repositories. Cross-cutting concerns have a single known location. Communication between modules happens through direct method calls rather than over a network.

Many systems don’t need distribution from day one. A well-modularized monolith should be easier to achieve than a constellation of small services.

The simplicity sounds real but can be deceptive.

Of course, officially, we want to be good citizens, invest in modular design, and avoid chattiness and complexity, but too often, instead of a simple solution, we’re getting a simplistic solution.

Essentially, we’re lying to ourselves, as the actual reason is that we do it because we want to take shortcuts.

We want to massage the boundaries with:

transactions spanning across multiple features or modules,
shared database to query other modules,
change multiple modules at once,
don’t care about data governance,
etc.

Our “modular monolith” is as modular as typical micro-services are “micro”.

I agree that our solutions should be simple, but they should not be simplistic.

The Boundary Problem

It’s kinda sad that we need to use the “modular” adjective in the “modular monolith”. It’s like claiming that a monolith, by definition, is not modular. Well, actually, it should be. If it’s not, then it’s not a monolith but just a big ball of mud.

Still, this issue with the name highlights the main challenge. In theory, a modular monolith has clear boundaries between its components. In practice, these boundaries erode fast.

Consider a typical e-commerce platform with “Orders,” “Products,” and “Customers” modules. Without physical separation, developers inevitably create shortcuts:

public void ProcessOrder(Order order)
{
    // Direct access to another module's repository
    var customer = this.customerRepository.GetById(order.CustomerId);

    // Logic that belongs in the Customer module
    if (customer.LoyaltyPoints > 1000)
    {
        order.ApplyDiscount(0.1m);
    }

    // ...
}

This happens not because developers are careless but because it’s the path of least resistance. When working on a feature spanning modules, it’s easier to add a direct dependency than to define interfaces and maintain module autonomy properly.

The boundary violations compound exponentially. Each small violation creates a precedent for more violations. The pattern I’ve seen repeatedly is that modules start with clean boundaries, but as feature development accelerates, these boundaries become increasingly porous. What makes this particularly insidious is that it happens in small increments. A tired developer working too long on a feature adds what seems like a harmless dependency. No alarms go off. Six months later, you’ve got a monolith that’s merely a traditional monolith with extra folders.

Of course, we can try to refactor, and instead of using the repository with direct access to the storage, we can have a “well defined public API” and use it as hexagonal architecture, e.g.

public void ProcessOrder(ICustomerService customerService, Order order)
{
    var discount =
        customerService.CalculateDiscountFor(
            order.CustomerId,
            order.TotalAmount
        );

    if (discount > 0)
    {
        order.ApplyDiscount(discount);
    }
}

But well, it is actually Potato vs Potahto.

The problem still exists; it’s a bit reduced, as the other team can have a bit more control over it and can set up some anti-corruption layer, but the boundaries are still mixed.

Release Train

The typical advice is to use mono-repo, keeping all modules in the same repo. That should streamline deployment and make it easier to do cross-cutting changes. That sounds great on paper, but… Choo-choo, here comes the release train!

If we have a monolithic deployment, then we need to deploy all modules at the same time. So if one of our friends from the other team merged something in their module, then if we want to deploy our stuff, then we need to deploy everything, including their changes.

I had numerous cases where the other team claimed that they tested everything, that it works, and that it won’t break anything, but then after deployment, it somehow didn’t work. Maybe migration failed, maybe some dependency clashed, maybe other random thing. Then it’s evening, you just wanted to deploy a small thing, and you’re trying to patch other teams’ changes.

Of course, you can and should do the following:

feature toggles,
avoid breaking changes,
version your changes, etc.

But well, are those monolith-first-influencers telling you about that? Read also in Don’t Oversell Ideas: Trunk-Based Development Edition.

Of course, the mature solution is to pack each module into the deployment package (NuGet, Maven, Npm, etc.). Then, each package is released independently. Having that, even if the latest code is updated in the repo, you don’t need to bump the dependency and release with the old version.

But then we’re falling into another issue, which is that we need to have a process for aligning dependencies and knowing what version of each module is actually deployed. And that’s how we’re coming to…

Tooling (im)maturity

One of the advantages showed for modular monolith is that it makes easier deployments and CI/CD process. You don’t need to use Kubernetes, Argo, and all that fancy stuff. But don’t you?

Unfortunately, managing mono-repos is not a common practice in most developed environments. You’re left with configuring pipelines manually and handle packages version and deployments.

Of course, there are tools like Nx, Bazel, TurboRepo that makes it easier, but frankly even though they can build various development technologies, they’re more optimised for frontend as that’s where they started. Also knowledge about them in environments like .NET, Java, so those leaning towards monoliths are at best low.

Of course there are approaches like Spring Modulith or .NET Aspire that trying to make that easier, but then, at least with Aspire it also ends up with simplistic instead of simple solution.

So, yes, you can have a mature and nicely defined CI/CD process, but it doesn’t come without hassle as it’s advocated; it needs to be well-planned and iterated, as the tooling is just not yet there.

Also, if you look at environments like C# or JavaScript, they weren’t defined for such a modular approach.You cannot restrict easily access and referencing other modules, as C# can only do that for assembly, JavaScript for ES module.

You can trick that with static analysis or architecture tests, but again, those are tricks and not stuff that you can get out of the box.

Don’t get me wrong; if you’re going with distributed systems like micro-services, you’ll also need to think about it. The CNCF-related stack can indeed be overcomplicated, but it can also be more mature, as it was built to support those specific scenarios. If you know what you’re doing, you can develop the simpler stack for the monolith, but it’ll still require work and thinking. If you don’t, then you can come up with a more fragile and less organised flow that will require constant tweaking.

It’s just not as simple as it’s pictured.

The Resource Contention Reality

In a microservices architecture, if the “Product Catalog” service has a memory leak, it doesn’t affect order processing. If the “Payment” service is CPU-bound, it doesn’t slow down inventory updates. This isolation is a fundamental advantage of distribution.

The problems with resource contention in modular monoliths are very real:

Memory leaks in one module bring down the entire system, requiring a full restart that impacts all business functions.

CPU-intensive calculations in one module make interactive user operations sluggish across the entire application.

A module with improper exception handling can crash the entire application.

One module upgrading a shared library can break other modules that aren’t ready for the change.

These issues aren’t theoretical—they’re daily operational realities for teams with modular monoliths in production.

The Database Coupling Nightmare

While we often focus on code-level coupling, the shared database in a modular monolith presents the most intractable problem. In a true microservices architecture, each service owns its data.

You can also have the same separation per database or schema in a modular monolith. Still, most of the teams are selecting separation module-per-schema. Why? Quite often, because you can spin up transaction across multiple schemas and query data from multiple schemas in the same connections. Cheating, remember?

A shared database creates invisible dependencies between modules. Consider what happens when the “Orders” module needs to change its schema to support a new feature. Since the “Analytics” module directly queries the Orders tables, you now have a cross-module dependency that isn’t visible in your code but will break spectacularly at runtime.

I’m sure that if you tried to distribute a portion of your monolith, taking out some module, you found multiple times such hidden dependencies.

Of course, you can set up the permission rules for schemas and ensure that other modules can’t access your data. But well, again that’s something that you need to plan and it doesn’t happen out of the box.

The Generic Unit of Work Trap

And here comes my favourite nightmare: the generic unit of work. Martin Fowler defines it as everything one does during a business transaction that can affect the database. When the unit of work is finished, it will provide everything that needs to be done to change the database as a result of the work.

In modular monolith, even if we cheat, we want to try to decouple our flow into smaller pieces. That can mean:

Handling different business operations as different commands,
publishing the events to trigger the next step of the workflow,
etc.

Most people using monolith are afraid of eventual consistency. They want to process all at once in the same process, most often also in the same transaction.

As they decoupled pieces logically, they wanted to use a generic mechanism to gather all the changes in the same request. As the mechanism is generic, we need to expect everything and prepare the implementation for everything.

It usually starts simply by opening transactions across the whole request, but that fails as we have more complex flows and start hitting deadlocks and performance issues (as multiple transactions try to modify the same data).

Then we optimise it, opening transactions only upon save. Before that, we’re caching our changes in a unit of work changeset, but then the challenge appears, as this data is not stored physically in the database; then, if we try to query it, it won’t be there. Why do we need to query? E.g. to access data stored in command processing in the next step of the workflow triggered by the event.

So we add a 2nd level cache and either load data by ID from the cache or from the DB (if it’s not in the cache). But then we’re hitting discrepancies and are not always able to detect what’s the actual source of truth. Concurrency issues are killing us. Ah, and at some point reading by id is not enough, we need to do more advanced filtering. Try to merge 2nd level cache with data from db…

I did all of that in past, and frankly, I hate every minute spent working on such a mechanism. It all looks simple when you’re starting, but it’s a long tail of accidental complexity.

Of course, there are solutions. You can set up a rule that:

you storing results, and even handling multiple commands for user facing, synchronous operations (Read more in How to handle multiple commands in the same transaction),
Each of the messages that are published as an outcome needs to be published through an Outbox Pattern.

That gives you a proper process isolation, delivery guarantee, simpler to reason flow, but can bring eventual consistency for multiple step workflows. That’s most of the time acceptable by business, can be tricked to be synchronous with long polling but it’s a tradeoff, and something to consider. Plus implementing outbox on your own doesn’t have to be so simple as it seems.

Monolith-first? Modular-first!

I’m not against monolith-first. Moreover, I agree with the general principles, but I wanted to debunk some myths around it.

The modular monolith isn’t a free lunch. Like all architectural approaches, it comes with tradeoffs. The problem isn’t that these tradeoffs exist but that they’re rarely acknowledged by proponents of the “modular monolith first” advice.

If you’re a technical leader making architectural decisions, don’t fall for the myth that a modular monolith gives you the best of both worlds without the drawbacks of either. Instead, carefully consider your team’s capabilities, business requirements, and growth projections.

Sometimes, starting with separate services is genuinely the right choice. Sometimes, a traditional monolith with clear internal structures is simpler and more honest than pretending you’re building something modular.

So it’s not precise to say:

“If you cannot deal with monolith, microservices won’t work for you!”.

The actual phrase should be: if you can’t deal with modularity, then neither monoliths nor microservices will work for you.

If you try to cheat and select modulith-first to cheat using a simplistic solution, it won’t help you; you will probably end up with a big ball of mud.

Monolith-first can give you a gradual process of reaching maturity, but you need to plan it and consider that the DevOps process related to it and tooling is not as simple and mature as glossed.

So, if you’re considering whether to go with microservices-first or monolith-first, you may be thinking the wrong way.

You should go Modular-First and then think about your deployment strategy.

Whatever you choose, you’ll need to do exercise and think about how to solve:

data isolation (modules, tenantc, etc.),
data consistency (transactional, eventual, causal, etc.)
modules boundaries,
process boundaries and how you model integrations,
DevOps process,
Code structure,
how to fit that with your team experience,
how to fit that with your budget and your product,
etc.

Saying to go monolith-first or micro-service first won’t give you proper answers and might not even be a strategy but just a blank statement that will hide the real complexity.

As always, be pragmatic and pick your poison!

What we don't know that we don't know

Welcome to the next week!

I’ve got good news and bad news for you today. The good news is that I no longer have the flu; the bad news is that it evolved into pneumonia… So, because of that, I have other news that could be good or bad, depending on how you look at it: it’ll be quick today. But I’ll try not to make it shallow!

I had a weird thought on Friday, right after leaving the hospital. I went there because I had complications from my flu. Scratch the details; let’s say they were looking pretty severe. My doctor sent me there to double-check her diagnosis.

I spent over 5 hours in the hospital, going through several medical treatments. In the end, it appeared that the initial diagnosis was right: pneumonia. I was released home with the note that prescribed antibiotics should be good enough.

And here we’re coming to my weird thought.

I was impressed by how I was treated. I was informed of what was happening at each stage, why, and what was next.

That gave me trust, that people knew what they were doing and that I was in good hands, but the weird thought was: but do I?

How do I know that I know? How do I know that I don’t know?

It reminded me of my other discussion with my Grandpa. We were watching some political debate when the TV signal went down. My Grandpa, in anger, said that it must “have been our prime minister acting not to let us see his opponent in the upcoming elections.”

I, of course, said, “Grandpa, I don’t like him either, but that’s just technical issue”, and I got the answer: “But how do you know, can you prove that’s not him?”.

And, well, I couldn’t say that I know that for 100% and I could prove that.

Common sense, aye? Actually, not, it can lead to multiple thinking issues like:

if something is not good, then it’s bad,
if something is not bad, then it’s good,
we can only judge decision by the outcome of it,
we can never judge decisions, because we never know,
etc.

Or the worst of the worst: “Challenge status quo!”.

Just like I didn’t know that our ex-prime minister turned off signal in my TV, I didn’t know that the decision made by doctors was actually right. Should I challenge the status quo and not take antibiotics? There is a place for such people: it’s called a cemetery.

So how do I know that I know, or how do I know that I don’t know? How to find the middle ground?

The first thing is to realise that we may be wrong and start to think about what to do about it. I wrote on it longer in The risk of ignoring risks.

The second is to differentiate decision from outcome.

Just like in Poker, we should calculate the odds and consider the risk of decision-making. In Poker, we can have the best situation, but the chance is never 100% that we’re right. If we see that we have, e.g., 90% of the chance to win looking at our hand and cards on the table, then making a specific decision can be right, but the outcome may still be a lost game. Those 10% can be enough if we have bad luck.

And in reality, we can have bad luck. But that doesn’t invalidate our decision, as in 9 of 10 cases, we’d be right. Should the random outcome invalidate our decision?

But okay, how do we know it’s 90%? It’s statistics based on the data we gathered throughout the game.

In our system design, that’s the homework we’re doing through preparation:

modeling,
design,
challenging our design with our stakeholders and peers,
proof of concepts,
validations,
simulations,
etc.

So that’s what I get in the hospital. Even though I ended up with the same diagnosis as I would have without it, it was much more backed with evidence and trustworthy.

And now the bonus point.

Remember that style matters. Someone can make bullshit sounds reasonable, and someone can’t make a great idea sounds tempting. It’s not enough to be right. You need to be persuasive.

Quite often, our goal as engineers or architects is to contribute to the general business plan focused on technical analysis. Too often, the final decision is made by people who have no idea if we’re right or wrong. They don’t know what they don’t know.

That’s a responsibility on us to prepare and to build trust in our professionalism. That’s why we need to learn to talk to other people involved in the process. Explain to them what’s happening, why and what’s next.

If we try to follow that, the chance that our ideas will be accepted grows.

And folks, please, do our homework, and make our idea be closer to hospital analysis than conspiracy theories. Or licked finger and the wind technique.

And folks, most importantly, stay healthy, that’s much more important than our work.

Cheers!

Oskar

We are no superhumans

Welcome to the next week!

If you want to make God laugh, tell him about your plans. Everyone has a plan until they get punched in the face. We can and should make plans but also learn to let go.

We are no superhumans, or at least I am not.

Sometimes, we have a bad week at work, sometimes our daughter falls and ends up with three stitches in her head, and sometimes we wake up feeling terrible with the flu. Surprisingly, such cases always happen together, aye?

All of that has happened to me in the last few days, especially the nasty flu, which stopped me from delivering the proper Architecture Weekly this week. And also stopped from going to conference, and maybe made me postpone the workshop…

Still, not to leave you with nothing, I have prepared a playlist with the 22 recordings of my conference talks. You’ll find there a lot of materials around Event-Driven Architecture, CQRS, and Event Sourcing.

Enjoy!

Cheers!

Oskar

TypeScript Migrates to Go: What's Really Behind That 10x Performance Claim?

Welcome to the new week!

Almost two weeks ago, Microsoft announced they’re porting the TypeScript compiler from JavaScript to Go, promising a staggering 10x performance improvement. This news spread like wildfire across tech communities, with everyone from TypeScript fans to language war enthusiasts chiming in.

The announcement leads with impressive performance numbers and ambitious goals, though some intriguing details remain unexplored. We’ll tackle those today. Why today and not a week ago? Well, in Architecture Weekly, we don’t want rushy clickbaits, aye? I intentionally wanted to wait for the noise to calm down.

Beneath the headline figures lies a story worth unpacking about design choices, performance trade-offs, and the evolution of developer tools.

Even if you’re not interested in compilers, there are suitable lessons learned for your system’s design, like:

Looking beyond headline performance claims
Matching your technology to your problem domain
Understand your runtime model.
Reconsidering foundations as projects evolve.

Let’s tackle that step by step, starting from the Microsoft article headline.

“A 10x Faster TypeScript” – Well, Not Exactly

First, let’s clarify something that might be confusing in Microsoft’s announcement title:

“A 10x Faster TypeScript.”

Is it really faster? Would your application written in TypeScript be 10x faster once MS releases the new code rewritten in Go?

What’s actually getting faster is the TypeScript compiler, not the TypeScript language itself or the JavaScript’s runtime performance. Your TypeScript code will compile faster, but it won’t suddenly execute 10x faster in the browser or Node.js.

It’s a bit like saying,

“We made your car 10x faster!”

and then clarifying that they only made the manufacturing process faster—the car itself still drives at the same speed. It’s still valuable, especially if you’ve been waiting months for your car, but not quite what the headline suggests.

Beyond the “10x Faster” Headline

Anders Hejlsberg’s announcement showcased impressive numbers:

Such remarkable stats deserve a deeper look, as this 10x speedup has multiple contributing factors. It’s not simply that “Go is faster than JavaScript”.

Let’s be honest: whenever you see a “10x faster” claim, you should view it with healthy scepticism. My first reaction was, “OK, what did they do suboptimal before?” Because 10x improvements don’t materialize from thin air—they typically indicate that something wasn’t made well enough in the first place. Be it implementation or some design choices.

There’s a widespread belief that “Node.js is slow,” which is more of a repeated stereotype than a truth. In some instances, it can be true, but it is not a general statement.

If someone says, “Node.js is slow” is slow, it’s almost like they’d say that C and C++ are slow. Why?

The Architecture of Node.js

Node.js is built on Google’s V8 JavaScript engine, which is the same high-performance engine that powers Chrome. The V8 engine itself is written in C++, and Node.js essentially provides a runtime environment around it. This architecture is key to understanding Node.js performance:

V8 Engine: Compiles JavaScript to machine code using Just-In-Time (JIT) compilation techniques
libuv: A C library that handles asynchronous I/O operations
Core Libraries: Many written in C/C++ for performance. That’s also why there are few significant performance differences between performance of database connectors used in Node.js, Go and Rust.
JavaScript APIs: Thin wrappers around these native implementations

When people talk about Node.js, they often don’t realize they’re talking about a system where critical operations are executed by highly optimized C/C++ code. Your JavaScript often just orchestrates calls to these native implementations.

Fun fact: Node.js has been among the fastest web server technologies available for many years. When it first appeared, it outperformed many traditional threaded web servers in benchmarks, especially for high-concurrency, low-computation workloads. This wasn’t an accident—it was by design.

Memory-Bound vs. CPU-Bound

Node.js was specifically architected for web servers and networked applications, which are predominantly memory-bound and I/O-bound rather than CPU-bound:

Memory-bound operations involve moving data around, transforming it, or storing/retrieving it. Think of:

Parsing JSON payloads
Transforming data structures
Routing HTTP requests
Formatting response data

I/O-bound operations involve waiting for external systems:

Database queries
Network requests
File system operations
External API calls

For typical web applications, most of the time is spent waiting for these I/O operations to complete. A typical request flow might be:

Receive HTTP request (memory-bound)
Parse request data (memory-bound)
Query database (I/O-bound, mostly waiting)
Process results (memory-bound)
Format response (memory-bound)
Send HTTP response (I/O-bound)

In this workflow, actual CPU-intensive computation is minimal. Most web applications spend 80-90% of their time waiting for I/O operations to complete.

Node.js was optimised for precisely for this scenario because:

Non-blocking I/O: While waiting for I/O operations, Node.js can handle other requests
C++ Foundation: Memory operations delegate to efficient C++ implementations
Event Loop Efficiency: Great at coordinating many concurrent operations with minimal overhead

Here’s what many people miss: I/O-intensive operations in Node.js work at nearly C-level speeds. When you make a network request or read a file in Node.js, you’re essentially calling C functions with a thin JavaScript wrapper.

This architecture made Node.js revolutionary for web servers. When properly used, a single Node.js process can efficiently handle thousands of concurrent connections, often outperforming thread-per-request models for typical web workloads.

It came essentially from the idea that multitasking, like in human tasks, is not always the optimal way of handling tasks. Synchronising multiple tasks and context switching adds overhead and does not always give better results. It all depends on whether a task can be split into smaller pieces that can be done concurrently.

Where Node.js Faces Challenges: CPU-Bound Operations

The real performance challenges for Node.js are CPU-intensive tasks—like (welcome again!) compiling TypeScript. These workloads have fundamentally different characteristics from the web server scenarios for which Node.js was optimized.

CPU-bound operations involve heavy computation with minimal waiting:

Complex algorithms and calculations
Parsing and analyzing large files
Image/video processing
Compiling code

In these scenarios, the bottleneck isn’t waiting for external systems—it’s raw computational power and how efficiently the runtime can execute algorithms.

The Single-Threaded Limitation

JavaScript was designed with a single-threaded event loop model. This model works best for handling concurrent I/O (where most time is spent waiting) but becomes problematic for CPU-intensive operations:

// Pseudocode of how the Node.js event loop works
while (thereAreEvents()) {
  const event = getNextEvent();

  processEvent(event);  // If this takes a long time,
                        // everything else waits
}

When a CPU-intensive task runs, it monopolizes this single thread. During that time, Node.js can’t process other events, handle new requests, or even respond to existing ones. It’s effectively blocked until the computation completes.

This is why running a complex algorithm in Node.js can make your entire web server unresponsive—the event loop is busy with computation and can’t handle incoming requests.

The Event Loop

Writing efficient CPU-intensive code in JavaScript requires understanding and respecting the event loop. The code must be structured to yield control, allowing other operations to proceed periodically:

//////////////////////////////////////////////
// Naive approach - blocks the event loop
/////////////////////////////////////////////

function processLargeData(data) {
  for (let i = 0; i < data.length; i++) {
    // Heavy computation that might take seconds
    processItem(data[i]);
  }
  return results
}


//////////////////////////////////
// Event-loop friendly approach
//////////////////////////////////

async function processLargeDataChunked(data) {
  const results = []
  const CHUNK_SIZE = 1000

  for (let i = 0; i < data.length; i += CHUNK_SIZE) {
    const chunk = data.slice(i, i + CHUNK_SIZE)
    // Process one chunk
    for (const item of chunk) {
      results.push(processItem(item))
    }
    // Yield to the event loop before processing the next chunk
    await new Promise(resolve => setTimeout(resolve, 0))
  }

  return results
}

This “chunking” approach works but introduces complexity and fundamentally changes how you structure your code. It’s a very different programming model from languages with native threading, where you can write straightforward CPU-intensive code without worrying about blocking other operations.

For a complex application like the TypeScript compiler, this dance with the event loop becomes increasingly difficult to manage as the codebase grows.

Compilers: The CPU-Intensive Beast

A compiler is practically the poster child for CPU-intensive workloads. It needs to:

Parse source code into tokens and abstract syntax trees
Perform complex type-checking and inference
Apply transformations and optimizations
Generate output code

These operations involve complex algorithms, large memory structures, and lots of computation—exactly the kind of work that challenges JavaScript’s execution model.

For TypeScript specifically, as the language grew more complex and powerful over the years, the compiler had to handle increasingly sophisticated type checking, inference, and code generation. This progression naturally pushed against the limits of what’s efficient in a JavaScript runtime.

Threading Models Matter: Event Loop vs. Native Concurrency

The performance gap between the JavaScript and Go implementations isn’t just about raw language speed—it’s fundamentally about threading models and how they match the problem domain.

As mentioned, Node.js operates on an event loop model:

   ┌───────────────────────────┐
┌─>│           timers          │
│  └─────────────┬─────────────┘
│  ┌─────────────┴─────────────┐
│  │     pending callbacks     │
│  └─────────────┬─────────────┘
│  ┌─────────────┴─────────────┐
│  │       idle, prepare       │
│  └─────────────┬─────────────┘      ┌───────────────┐
│  ┌─────────────┴─────────────┐      │   incoming:   │
│  │           poll            │<─────┤  connections, │
│  └─────────────┬─────────────┘      │   data, etc.  │
│  ┌─────────────┴─────────────┐      └───────────────┘
│  │           check           │
│  └─────────────┬─────────────┘
│  ┌─────────────┴─────────────┐
└──┤      close callbacks      │
   └───────────────────────────┘

This single-threaded approach means that for CPU-intensive work like compiling TypeScript, you need to write code that doesn’t monopolize the thread. In practice, this involves breaking work into smaller chunks that can yield control back to the event loop.

For a compiler, this creates significant design challenges:

Artificial Fragmentation: The natural flow of compiler phases (parse → analyze → transform → generate) needs to be broken into small steps that can yield.
Complex State Management: Since processing is fragmented across event loop iterations, the compiler state must be carefully managed and preserved between yields.
Locality Disruption: When the event loop processes unrelated tasks between compiler operations, CPU cache locality benefits are lost, hurting performance.
Dependency Challenges: Compilers have complex interdependencies between components. Breaking a naturally sequential process to accommodate an event loop often requires complex coordination logic.

Go: Native Concurrency with Goroutines

Go, by contrast, offers goroutines—lightweight threads managed by the Go runtime:

┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
│Goroutine│ │Goroutine│ │Goroutine│ │Goroutine│ ...  more
└────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘
     │           │           │           │
     └───────────┴───────────┴───────────┘
                       │
              ┌────────┴────────┐
              │   Go Scheduler  │
              └────────┬────────┘
                       │
                ┌──────┴──────┐
                │  OS Threads │
                └─────────────┘

This model allows compiler phases to be naturally parallelized with minimal coordination overhead:

Natural Parallelism: Different files can be parsed and type-checked concurrently.
Direct Thread Access: CPU-intensive operations can run directly on threads without yielding.
Efficient Coordination: Go’s channels and synchronization primitives are designed to coordinate concurrent work.
Memory Efficiency: Goroutines use minimal memory (a few KB each) compared to OS threads (MBs each).

In this model, file parsing, type checking, and code generation can all happen concurrently without explicit yield points. The code structure can more naturally follow the logical flow of compiler phases.

Read also more in:

Same Code, Different Execution Model

Anders Hejlsberg said they evaluated multiple languages, and Go was the easiest to port the code base into. The TypeScript team has apparently created a tool that generates Go code, resulting in a port that’s nearly line-for-line equivalent in many places. That might lead you to think the code is “doing the same thing,” but that’s a misconception.

The code may look the same but can behave very differently across languages due to their execution models.

In JavaScript:

All code runs on a single thread with an event loop,
Long-running operations need to be broken up or delegated to worker threads,
Concurrent execution requires careful handling to avoid blocking the event loop.

In Go:

Code naturally runs across multiple goroutines (lightweight threads),
Long-running operations can execute without blocking other work,
The language and runtime are designed for concurrent execution.

When you port code from JavaScript to Go without changing its structure, you implicitly change how it executes. Operations that would block the event loop in JavaScript can run concurrently in Go with minimal effort.

This can lead to a conclusion: when a direct port yields dramatic performance improvements, it might indicate that the original implementation wasn’t fully optimized for JavaScript’s execution model.

Writing truly performant JavaScript means embracing its asynchronous nature and event loop constraints—something that becomes increasingly challenging in complex codebases like a compiler.

But What About Worker Threads?

Some of you might be thinking:

“Doesn’t Node.js have Worker Threads now? Couldn’t they have used those instead of migrating to Go?”

It’s a fair question. Node.js introduced the worker_threads module in v10 as an experimental feature became stable in v12 (mid-2019). Worker threads provide true parallelism in Node.js, allowing CPU-intensive tasks to run on separate threads without blocking the main event loop.

How Worker Threads Work in Node.js

Unlike the single-threaded event loop that characterizes most Node.js applications, worker threads allow JavaScript to be executed in parallel:

// main.js
const { Worker } = require('worker_threads')

function runWorker(workerData) {
  return new Promise((resolve, reject) => {
    const worker = new Worker('./worker.js', { workerData })
    worker.on('message', resolve)
    worker.on('error', reject)
    worker.on('exit', code => {
      if (code !== 0) {
        reject(new Error(`Worker stopped with exit code ${code}`))
      }
    })
  })
}

// Run multiple CPU-intensive tasks in parallel
Promise.all([
  runWorker({ chunk: data.slice(0, middleIndex) }),
  runWorker({ chunk: data.slice(middleIndex) })
]).then(results => {
  // Combine results
})

// worker.js
const { parentPort, workerData } = require('worker_threads')

// Perform CPU-intensive work
const result = processData(workerData.chunk)

// Send the result back to the main thread
parentPort.postMessage(result)

Each worker runs in its own V8 instance with its own JavaScript heap, allowing true parallelism. Workers communicate with the main thread (and each other) by sending and receiving messages, which can include transferring ownership of certain data types to avoid copying.

Why Weren’t Worker Threads the Solution for TypeScript?

So why might the TypeScript team have chosen to Go over retrofitting the existing codebase with worker threads? That we don’t know, but there are several plausible reasons:

Legacy Codebase Challenges: The TypeScript compiler has been developing for over a decade. Retrofitting a large, mature codebase designed for single-threaded execution with a multi-threaded architecture is often more complex than starting fresh. Workers communicate primarily through message passing. Restructuring a compiler to operate this way requires fundamentally reimagining how components interact. The video showed that even single-threaded Go was faster, so the code would probably still require modification to better take care of event loop characteristics.
Data Sharing Complexity: Workers have limited ability to share memory. A compiler manipulates complex, interconnected data structures (abstract syntax trees, type systems, etc.) that don’t neatly divide into isolated chunks for parallel processing.
Performance Overhead: While worker threads provide parallelism, they come with overhead. Each worker has its own V8 instance with separate memory, and data passed between threads typically needs to be serialized and deserialized. They are not as lightweight as threads or goroutines.
Timeline Mismatch: When the TypeScript compiler was designed and implemented (around 2012), worker threads didn’t exist in Node.js. The architecture decisions made early on would have assumed a single-threaded model, making later parallelization more difficult.
Dead-End Assessment: The team may have concluded that even with worker threads, JavaScript would still impose fundamental limitations for their specific workload that would eventually become bottlenecks again.
Skill Set Alignment: The decision may have partly reflected organizational expertise and strategic alignment with other developer tools.

Go’s Approach vs. Worker Threads

Go’s approach to concurrency offers several advantages over Node.js worker threads for compiler workloads:

Lightweight Goroutines: Goroutines are extremely lightweight (starting at ~2KB of memory) compared to worker threads (which require a separate V8 instance), making fine-grained parallelism more practical.
Shared Memory Model: Go allows direct shared memory between goroutines with synchronization primitives, making working with complex, interconnected data structures easier.
Language-Level Concurrency: Concurrency is built into Go at the language level with goroutines and channels, making parallel code more natural to write and reason about.
Lower Overhead Communication: Communication between goroutines is significantly more efficient than the serialization/deserialization required for worker thread communication.
Mature Scheduler: Go’s runtime includes a mature, efficient scheduler for managing thousands of goroutines across available CPU cores.

Also, I think the migration to Go wasn’t just about “multithreading vs. single-threading” but about adopting a programming model where concurrency is a first-class concept, deeply integrated into the language and runtime.

The Evolution Problem

When TypeScript started in 2012, the team made reasonable technology choices based on the context at that time:

TypeScript was a Microsoft project extending JavaScript, so using JavaScript made sense
The initial scope and complexity were much smaller
Alternatives like Go and Rust were still in their early days
The performance demands were more modest

Over time, TypeScript evolved from a relatively simple superset of JavaScript into a sophisticated language with advanced type features, generics, conditional types, and more. The compiler grew accordingly but remained built on foundations designed for a simpler problem.

This is a classic example of how successful software often faces scaling challenges that weren’t anticipated in its early design. As the TypeScript compiler became more complex and was applied to larger codebases, its JavaScript foundations became increasingly restrictive.

Don’t you know that on your own?

How is legacy code created? Day by day.

Outstanding Questions and Future Considerations

While the performance improvements from the Go migration are impressive, several important questions haven’t been fully addressed in Microsoft’s announcement:

What About Browser Support?

TypeScript doesn’t just run on servers and development machines and’s used directly in browsers via various playground implementations and in-browser IDEs. How will Microsoft address this use case since Go doesn’t run natively in browsers?

There are a few potential approaches:

WebAssembly (WASM): Compiling the Go implementation to WebAssembly could allow it to run in browsers. While WASM performance has improved dramatically, there would still be overhead compared to native Go.
Dual Implementation: Microsoft might maintain a JavaScript version for browsers alongside the Go version for everything else. This would create challenges for feature parity and maintenance.
Browser-Specific Alternative: They might create a streamlined browser-specific implementation with reduced functionality optimized for common playground scenarios.
Cloud Compilation: Browser-based tools might send code to cloud endpoints running the Go compiler instead of performing compilation locally.

The announcement doesn’t clarify their approach, and it’s an important detail that will affect the TypeScript ecosystem.

Feature Parity vs. Performance Trade-offs

It’s worth noting that achieving performance improvements often involves trade-offs. One common approach is to reduce feature scope or complexity. While Microsoft claims they’re maintaining full feature parity, we should watch carefully to see if any subtle behaviours change or if certain edge cases are handled differently.

Historical examples from other language migrations show that 100% identical behaviour is challenging to achieve. Some questions to consider:

Will all existing TypeScript error messages remain exactly the same?
Will every edge case in type inference behave identically?
Will compilation options and flags have the same effects?
How will performance optimizations affect type system corner cases?

The TypeScript team has a strong track record of backward compatibility, but a ground-up rewrite inherently carries risks of subtle behavioural changes.

Extensibility and Plugin Ecosystem

TypeScript has a rich ecosystem of plugins and tools that extend the compiler. The migration to Go raises questions about the future of this ecosystem:

Will the plugin API remain compatible?
Will JavaScript/TypeScript-based plugins need to be rewritten in Go?
How will this affect the barrier to entry for creating TypeScript tooling?

These considerations will affect the broader TypeScript ecosystem beyond just compilation performance.

Why This Matters Beyond TypeScript

This case study has broader implications for technology choices:

Match your technology to your problem domain. CPU-bound tasks like compilers benefit from languages designed for computation and native threading. IO-bound tasks like web servers often work well with event-loop models.
Reconsider foundations as projects evolve. What worked for a small project might become a constraint at scale. Be willing to revisit fundamental architecture decisions. Nothing’s wrong with a bold move, like rewriting if it’s needed. But before you do it, read that first.
Look beyond headline performance claims. A “10x improvement” often has multiple contributing factors beyond just the change in the technology stack.
Understand your runtime model. Whether you use Node.js, Go, Rust, or any other environment, understanding how code executes is crucial for performance optimization.

Looking Forward

As a TypeScript user, building Emmett and Pongo is good news. If I can get the compiler working faster “for free”, that’s sweet. But on the other hand, I don’t see why we make such noise and use clickbait as a way of bringing that to the development community.

Focusing on 10x without giving enough context just created friction (I mercifully skipped C# developers’ cries of “why not C#?!“…).

That’s why I wanted to expand on this use case, as it offers valuable lessons about technology, language choices, performance optimization, and the evolution of successful projects.

The move from JavaScript to Go shouldn’t be taken as a confirmation that “Node.js is slow”. It’s better to look on that as a recognition that different problems call for different tools. JavaScript and Node.js continue to be good at what they were designed for: IO-intensive web applications with high concurrency needs.

Of course, it’d be better if Microsoft explained that in more detail, rather than making a clickbait claim, but this is the world we live in.

Check also previous releases on performance:

What do you think? Have you faced challenges similar to technology evolution in your projects? How did you tackle them? Did you ever get a 10x surprising improvement? How did you deal with that?

I’d love to hear your thoughts in the comments!

Cheers!

Oskar

Event-Driven by Oskar Dudycz — articles

Open Source, a relic, a charity or still the thing?

We all wanted to be Netflix, who do we want to be next?

Avoid Duke Nukem Forever Mode

Start Small, Grow Big: how to pick the first feature for Event Sourcing

Picking the right problem

Small doesn’t mean unimportant

Selection checklist

Bad picks

A decent pick

Don’t forget to audit the tooling

Showcasing the pilot outcome

Fractal Architecture, Cognitive Load, Vertical Slices and other terms that do(n't) fit your head

Vertical slices, their ownership and external dependencies

Slice, module, context

Slices boundaries in practice

Duck typing and dependencies

Everything outside the slice is external

Two entry points, one feature

Vertical Slices and Database

Vertical Slices and Frontend

TLDR

Fixing bugs in Event Sourcing is hard, for real?

The bug

What we have, with state

What that migration does

And then migration number two

The same bug, event-sourced

Knowing which events to look at

Fix forward, don’t rewrite

If we get the correction wrong too

The customer already fixed it

Make the correction a feature

“But now the history contains a bug”

Throw, Result, or neither?

An error as a business fact

Broken rules can throw

What Result would add

Returning an event doesn’t mean persisting it

Several decisions on one stream

So should we throw or not?

Never Throw in Asynchronous Handlers

Conclusion

On swimming, learning and my journey to being the last one

Addition by subtraction in Software Design

What’s the lesson then?

Announcing Strictland - new contract testing library for message compatibility

What’s Strictland?

Why such a name?

How to use it?

Should you use it?

You can fork a package, but can you own it?

To fork or not?

Dependency Inventory

Bus factor and rug pulls

LLM as a fork

What to do then?

How soon is now in PostgreSQL?

The bug

Why my tests didn’t catch it

What I’m taking away

TLDR

On mashing up modelling techniques for fun and profit

Don't overestimate domain expertise

Vibing, Harness and OODA loop

The example

Yoda Principle for better integrations

Anti-patterns in event modelling - Passive-Aggressive Events

TLDR

The one where Oskar explains Example Mapping

Interactive Rubber Ducking with GenAI

Interactive Rubber Ducking example

My idea

Q1: Cache library choice — lru-cache, keyv, or custom interface?

Q2: Where should the cache live in Pongo’s hierarchy?

Q3: Inheritance/override behavior for cascading cache config?

Q4: Cache interface shape — async, batch ops, scoping?

Q5: Cache key strategy

Q6: Cache invalidation and consistency with optimistic concurrency

Q7: Should skipCache be available on read methods too?

What `Result` would add

Q7: Should `skipCache` be available on read methods too?

Q8: Batch operations on `handle` — signature and handler shape

`PongoCacheProvider<T>`

`CacheConfig`

Where: `pongoCollection` factory function

`handle` method