Recorded durability test
Baridi drives its breach alert with Temporal. To check that the alert is not lost when a process dies, we killed a worker in the middle of sending an alert and watched Temporal resume the work on a new worker. This page shows that captured run.
A worker dies mid-alert and the alert still fires once
Worker A ran with a 25 second delay inside the alert step, so it could be killed after the breach was known but before the alert was durably written. The breach count was already decided. Only the write had not happened yet.
| Time (UTC) | Event |
|---|---|
| 21:01:20.590 | Worker A started with a 25 second alert delay. |
| 21:01:24.595 | Workflow submitted. |
| 21:01:25.301 • | Worker A: alert START for excursion 10:04. The breach is known, the alert is not yet written.Breach |
| 21:01:25.614 • | Worker A KILLED with kill -9, mid-alert. No worker is running.Kill |
| 21:01:25.617 | alerts.log line count while down: 0. The alert was not written before the crash. |
| 21:01:30.623 | Worker B restarted, no delay. |
| 21:01:56.316 | Temporal redelivers the activity. The alert fires, about 31 seconds late. |
| 21:01:56.544 | Final alert count: 2, one per excursion. None lost, none doubled. |
final alert count: 2, none lost, none doubled, about 31 seconds late.
The alert fired late because Temporal retried after the worker died. It was not lost, and it did not double-fire. The two keys below are the final lines in the alert log, one for each excursion on the featured day.
The durability mechanism is Temporal. It checkpoints the workflow, so a crash resumes from the last step instead of dropping the alert. That is the same thing the home page calls an alert that survives a crash, shown here as a recorded kill-and-resume rather than words. On the live site the breach timeline is served by the read-only API and TimescaleDB. The worker in this test is a separate process and does not run on the web path.
Baridi Technologies