![\"\"](\"/800452215e74cd33b63bafd5c96e24d1/2026-05-25-nasty.gif\")
How soon is now? In PostgreSQL, it’s not always as soon as you’d think. I learned that the hard way recently, so you don’t have to.
\nIt took me hours and wasn’t easy to reproduce, even though the fix is one line. I found it in a Cybertec post, as I quite often do when I’m staring at something odd in PostgreSQL. I’m supposed to know my way around the database, but I missed it, which is another reason I want to write this down.
\nI was working on distributed locking in Emmett. When you scale a service horizontally, you can easily end up with two instances of the same message processor running at once. That’s bad. Both instances would pull the same events, both would write to the same projection storage, and we’d get duplicated side effects, overwritten state and broken checkpoints. So we need to guarantee that exactly one instance of each processor is active at any time. Emmett does that using two things working hand in glove: PostgreSQL advisory locks and a row in the emt_processors table. The row keeps the durable side of ownership: which instance currently holds the processor (processor_instance_id), when it last checked in (last_updated), and what state it’s in (status). I described the full design in Rebuilding Event-Driven Read Models in a safe and resilient way, so I won’t bore you with the whole picture here.
For this story, the part that matters is what happens when an instance crashes. The crashed processor’s connection is gone, so its advisory lock has already been released. A new instance can grab the advisory lock without resistance. But the row in emt_processors still says status = 'running' and still points to the previous owner, because the crash didn’t give anyone a chance to clean it up.
From the outside, we can’t tell whether the previous owner has crashed or is just between heartbeats. So we wait. If the row’s last_updated is older than a configurable timeout, the new instance is allowed to claim ownership anyway. Anyone quiet for that long is treated as gone. To make this graceful, the lock acquisition runs inside a retry policy. A fresh instance starting just after a crash doesn’t fail straight away; it retries until the timeout window expires.
The takeover decision lives in the upsert against emt_processors. In the real function, that upsert sits inside a Common Table Expression (CTE) alongside a pg_try_advisory_xact_lock call.
For the record: the snippets below skip that wrapping (and trim a couple of unused parameters) to keep the focus on the upsert, where the bug lives. The full version is in the source.
\nCREATE OR REPLACE FUNCTION emt_try_acquire_processor_lock(\n p_processor_id TEXT,\n p_processor_instance_id TEXT,\n p_lock_timeout_seconds INT\n)\nRETURNS BOOLEAN\nLANGUAGE plpgsql\nAS $$\nBEGIN\n INSERT INTO emt_processors (processor_id, processor_instance_id, status, last_updated)\n VALUES (p_processor_id, p_processor_instance_id, 'running', now())\n ON CONFLICT (processor_id) DO UPDATE\n SET processor_instance_id = p_processor_instance_id,\n status = 'running',\n last_updated = now()\n WHERE \n -- same instance reconnecting\n emt_processors.processor_instance_id = p_processor_instance_id \n -- previous owner stopped cleanly\n OR emt_processors.status = 'stopped' \n -- previous owner timed out \n OR emt_processors.last_updated\n < now() - (p_lock_timeout_seconds || ' seconds')::interval;\n RETURN FOUND;\nEND;\n$$;The last branch is the takeover. It reads naturally: if the previous owner hasn’t checked in for longer than the timeout, the new instance can replace them. All tests were green. Stop me if you think you’ve heard this one before. The problem surfaced through user feedback (thanks, Martin!), and it took me a long time to reproduce; none of the existing tests covered the scenario that triggered it. Once I had a new end-to-end test that pinpointed the symptom, the rest was the usual, long, boring, debugging loop.
\nTo see why, open psql and run this:
BEGIN;\nSELECT now() AS tx_now, clock_timestamp() AS wall_clock;\nSELECT pg_sleep(2);\nSELECT now() AS tx_now, clock_timestamp() AS wall_clock;\nCOMMIT;You’ll get something like:
\n tx_now | wall_clock\n-------------------------------+-------------------------------\n 2026-05-25 10:00:00.123456+00 | 2026-05-25 10:00:00.124012+00\n\n tx_now | wall_clock\n-------------------------------+-------------------------------\n 2026-05-25 10:00:00.123456+00 | 2026-05-25 10:00:02.131845+00The first column is the same in both rows. The second one is two seconds apart. As it turns out, now() is a synonym for transaction_timestamp(): it returns the time the transaction began, and keeps returning that value for every statement inside the same transaction. A light that never goes out, in other words. clock_timestamp() reads the wall clock each time it’s called, so it advances as time does. Cybertec wrote a good walkthrough of the whole family of timestamp functions if you want the full picture.
What difference does it make? For a column like last_updated, the constancy of now() is usually what you want: every row touched in the same transaction shares a single timestamp, which keeps audit logs and write batches coherent. For asking “has enough time passed?” inside the same transaction, the same constancy works against us.
Now back to the retry. The consumer that calls tryAcquire looks roughly like this:
pool.withTransaction((tx) =>\n asyncRetry(\n () => tryAcquireProcessorLock(tx.execute, options),\n { retries: 10, minTimeout: 200, maxTimeout: 1000 },\n ),\n);pool.withTransaction opens a database transaction and passes its executor to the body. asyncRetry then repeatedly calls the stored procedure on that same executor, with a backoff between attempts. So even though the retries are spread out in real time, every call runs inside the same database transaction:
withTransaction (transaction starts at T)\n └── asyncRetry\n ├── call lock function → now() = T\n ├── call lock function → now() = T (200 ms later)\n ├── call lock function → now() = T (400 ms later)\n └── ...last_updated < now() - timeout evaluates the same way every iteration. The predicate is effectively constant for the lifetime of that transaction. From the database’s perspective, no time was passing between attempts, even though the retries were spread across real seconds. (of course, the valid question is whether retries should happen inside a transaction, but let’s say that this is out of scope of today’s article, deal?).
So what was the fix? Change the time source inside the function. PL/pgSQL lets you declare local variables, so I added one at the top, initialised from clock_timestamp(), and used it everywhere the function previously called now():
CREATE OR REPLACE FUNCTION emt_try_acquire_processor_lock(\n p_processor_id TEXT,\n p_processor_instance_id TEXT,\n p_lock_timeout_seconds INT\n)\nRETURNS BOOLEAN\nLANGUAGE plpgsql\nAS $$\nDECLARE\n v_current_time TIMESTAMPTZ := clock_timestamp();\nBEGIN\n INSERT INTO emt_processors (processor_id, processor_instance_id, status, last_updated)\n VALUES (p_processor_id, p_processor_instance_id, 'running', v_current_time)\n ON CONFLICT (processor_id) DO UPDATE\n SET processor_instance_id = p_processor_instance_id,\n status = 'running',\n last_updated = v_current_time\n WHERE emt_processors.processor_instance_id = p_processor_instance_id\n OR emt_processors.status = 'stopped'\n OR emt_processors.last_updated\n < v_current_time - (p_lock_timeout_seconds || ' seconds')::interval;\n RETURN FOUND;\nEND;\n$$;clock_timestamp() ignores transaction boundaries. Every call to the stored procedure reads the wall clock fresh, so each retry sees a slightly later value than the last. After enough retries inside the wrapping transaction, the takeover predicate flips, and the new instance wins.
The function uses the timestamp in two places: when setting last_updated on the new owner, and when comparing the previous owner’s last_updated to the timeout. I switched both to v_current_time, so the write and the check read from the same wall clock. Mixing clock_timestamp() on one side and now() on the other would leave a subtler version of the same bug.
A cleaner option for the future is to move the retry one layer up, so each attempt opens its own transaction. That would remove the trap entirely and let the function go back to plain now(). For now, the local variable does the job.
Now to the testing side. I had a careful suite of integration tests for the stored procedure. Two instances racing for the lock. The same instance reconnects after a crash. Takeover after a custom timeout. They all passed, and they could not have caught this bug. Here’s the shape of a typical one:
\nawait pool.withTransaction((connection) =>\n lock.tryAcquire({ execute: connection.execute }),\n);That’s the shape: set up some state, call tryAcquire once, check the result. A single call to the stored procedure works fine with now() in the WHERE clause. There is only one timestamp involved per call, and the predicate evaluates correctly against it. The bug only shows up when several calls share one transaction, which happens when the consumer’s withTransaction wraps a retry policy. The stored-procedure tests never put those two together.
The end-to-end consumer tests do go through the consumer’s withTransaction wrapper, but they covered the happy paths: clean start, clean stop, two consumers competing, and an instance reclaiming its own stale lock. None of them combined the three conditions that together expose the bug:
status = 'running' (a crash, not a graceful stop),Any one of those missing and the takeover predicate either succeeded on the first attempt (so the retry never fired), or the test finished before the retry’s failure to make progress was visible.
\nSo why did this slip through both layers? The stored-procedure tests never combined a retry policy with the stale-row state, so they never produced multiple calls inside one transaction. The end-to-end tests did exercise the retry path, but none of them happened to combine all three conditions above at once. Both layers had blind spots, and the bug lived exactly where they overlapped.
\nThe Pull Request that fixes the bug also adds a new family of tests that mount tryAcquire under the same transactional wrapper the real consumer uses, with the crash + new-instance + retry combination wired up on purpose. That’s the kind of test I should have had from the start.
Two things about now():
now() is the right tool when you want every row touched in the same transaction to share a timestamp. created_at, last_updated, audit columns. That stability is a feature.now() is the wrong tool when you want to ask “has time moved on?” from inside a transaction. Use clock_timestamp() when you genuinely mean the wall clock.And one harder thing about tests. Inner tests for stored procedures give me a tight feedback loop and pinpoint failures, but only inside the scaffolding I build for them. End-to-end tests run the real wiring, but I can’t enumerate every combination of timeouts, instance IDs and crash states without the suite collapsing under its own weight. The combination where this bug lived wasn’t reachable from either side by accident; it needed a deliberate setup.
\nI don’t have a clean rule for where to draw that line, and honestly, I don’t think there is one. My takeaway is to look at the seam: the spot where the inner test invokes the code differently from how the production caller invokes it. Here, it was a single-call test against a retry loop sharing one transaction. Wherever that gap sits, write a test there. Not at the unit level, not at the full end-to-end level, but in a setup that mirrors how the real caller actually drives the code and exercises the path you care about.
\nnow() returns the start of the current transaction, not the current moment. Inside a transaction it doesn’t change between statements. If you wrap a retry loop around a function that uses now() in a WHERE clause, and the retry loop runs inside one transaction, the predicate is frozen and the retries do nothing. Use clock_timestamp() when you mean “right now”. And pay extra attention to the seam between inner and end-to-end tests, because that’s where mismatches between how tests drive the code and how production drives it tend to hide.
The fix and the new tests live in Emmett Pull Request #339.
\nUff. That bug was nasty.
\n
Read also:
\nnow() vs now()::timestamp vs clock_timestamp(),Or my other articles about PostgreSQL:
\nCheers!
\nOskar
\np.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.
","excerpt":"How soon is now? In PostgreSQL, it’s not always as soon as you’d think. I learned that the hard way recently, so you don’t have to. It took…","fields":{"slug":"/how-soon-is-now-in-postgresql/","prefix":"2026-05-25","langKey":"en"},"frontmatter":{"title":"How soon is now in PostgreSQL?","author":"oskar dudycz","category":"PostgreSQL","disqusId":null,"useDefaultLangCanonical":null,"cover":{"childImageSharp":{"resize":{"src":"/static/38bff92c4f54acc9e02aab58e8a3d196/2a4de/2026-05-25-cover.png"}}}}},"authornote":{"id":"295c4649-a20b-5147-9ca5-5cb77fbcfe66","html":"Through my window, I see the result of good plans but poor execution. Opposite my flat, there is a partially completed construction place. Buildings were supposed to be eye-catching Mediterranean style apartments. Delivery date? Two years ago. Actual? More and more unknown.
\nSome time ago, I heard that using Event Sourcing makes creating Event-Driven Architecture easier. The arguments were correct, that if we’re already publishing events to trigger business workflows, then at some point, we may want to also store events to not lose information. Agreed. However, I also heard that keeping the state as events will simplify things. We’ll have a source of truth with a record of the system behaviour. This will allow, e.g. to confront the results of the operations with the recorded state. I’d agree with that, with one distinction. It’s easier as long as you already know Event Sourcing.
\nMany people in the DDD community claim that the essential is to properly break down the system into autonomous parts called bounded contexts. Once we have it, the rest is secondary and will sort itself out. For sure.
\nMany seasoned programmers speak similarly about new technologies. They claim that they can translate past experience into new technologies. That’s true that by analogy, they can catch the big picture quicker. But isn’t it a bold assumption to say that Win.Forms specialist will learn Angular quickly?
\nThe end result may differ a lot from the initial ideas. I saw the plan of those buildings next to me. Now I can see the effects of the execution. Or actually, the lack.
\nI believe that we should carefully acknowledge not only the point of view of our authorities but also their seating point. If we want to find out how to form a wall, do we ask an architect or a foreman? An architect may know the theory, but the practice is what we’re looking for. On the other hand, if you want to know where to put the wall, you prefer the architect to do measurements. At least if you don’t want to have the roof falling to your head.
\nAfter I had torn a ligament in my knee, I went to two qualified orthopedists. One said I should have surgery and do a reconstruction. The second stated that there is no need for that; rehabilitation should be enough. Guess which one had a specialization in surgery and which in rehabilitation?
\nPeople usually give us advice from the point where they’re currently standing. They are entitled to a biased view. An architect who rarely does programming will tend to downplay the value of implementation and tactical patterns. Midlevel developers will focus on technicalities instead of the global system impact. The team manager or consultant will emphasize the importance of soft skills (or esoteric techniques known only to them).
\nThe truth is that we need all of them. The excellent plan will fall on the bad execution. The best execution for the wrong case will be just a waste of time. We should carefully evaluate the advice considering what we need and what an expert can give us.
\nTherefore, when we’re reading an article, watching a talk, let’s also pay attention to the place where the person is standing. The perspective from there may be much different from where we are right now. That can be good, as it may push us in the right direction. But it may also be misleading, as we accidentally take biases of this person without understanding the tradeoffs. Personally, I prefer to follow not only people from pedestal but also those that are closer to my position. A bit further in the journey, but not too far. That helps me to calibrate my view as those people are more relative to my daily struggles.
\nPolish historical leader Józef Piłsudzki reportedly used to say: “Right is like an ass, everyone has its own”.
\nCheers!
\nOskar
"},"site":{"siteMetadata":{"facebook":{"appId":""}}}},"pageContext":{"slug":"/how-soon-is-now-in-postgresql/","lang":"en","langKey":"en","prev":{"id":"e95d157a-18c2-5bd5-938d-4561ffc9f983","fields":{"slug":"/on-mashing-up-modelling-techniques/","prefix":"2026-05-18","source":"posts","langKey":"en"},"frontmatter":{"title":"On mashing up modelling techniques for fun and profit","category":"Software Architecture"}},"source":"posts"}}, "staticQueryHashes": ["2742854296"]}