Stop Rebuilding Databases in Your Web Server

Why Rolling Your Own Write-Ahead Log Is a Terrible Idea

TL;DR

Building a custom write-ahead log inside your web service to track API requests sounds lightweight, but you’re actually rebuilding a distributed database. Real WALs handle durability, crash recovery, replication, and conflict resolution. Those are hard problems that took decades to solve. Use boring infrastructure like Postgres or Kafka with patterns like Outbox and Saga. Save your creativity for your product, not for reimplementing database internals.

---

🔥 Introduction

You’re building a REST API and you want to track every request. Who called it. What they did. When they did it. Maybe even which headers, which user, which tenant.

So far so good. That’s a pretty standard requirement: auditing, compliance, analytics, debugging.

Then someone on the team has a „simple“ idea:

„Instead of adding a database or message queue, let’s just write every request to a write-ahead log file on each application server. No extra dependencies.“

At first glance, it sounds clean and lightweight. Just append to a file, maybe sync it, and replicate between app instances. No Kafka, no Postgres, no „big infra.“

But here’s the catch: once you go down that road, you’re not avoiding complexity. You’re quietly building a distributed log, a database, a message queue, all inside your web service.

This article walks through why that’s a terrible idea by unpacking what real write-ahead logs, databases, and message queues actually do. Then we’ll look at better patterns like Outbox and Saga for solving this problem without reinventing a database in your codebase.

---

The Proposal: A Homegrown Write-Ahead Log in the App

Let’s spell out the idea more concretely.

You have multiple replicas of your REST API behind a load balancer. Each incoming request hits one of the app servers. That server writes a record to a local „write-ahead log“ file with who, what, and when. Another request might hit a different app server, which writes to its own local log file. Some asynchronous mechanism replicates logs between servers: replica1 talks to replica2, replica2 talks to replica3, and so on.

graph TB
    Client1[Client Request 1] --> LB[Load Balancer]
    Client2[Client Request 2] --> LB
    Client3[Client Request 3] --> LB

    LB --> App1[App Server 1]
    LB --> App2[App Server 2]
    LB --> App3[App Server 3]

    App1 --> WAL1[(Local WAL 1)]
    App2 --> WAL2[(Local WAL 2)]
    App3 --> WAL3[(Local WAL 3)]

    WAL1 -.async replication.-> WAL2
    WAL2 -.async replication.-> WAL3
    WAL3 -.async replication.-> WAL1

    style WAL1 fill:#f9f,stroke:#333,stroke-width:2px
    style WAL2 fill:#f9f,stroke:#333,stroke-width:2px
    style WAL3 fill:#f9f,stroke:#333,stroke-width:2px

The argument sounds reasonable at first. We reduce dependencies because we don’t need a DB or message queue. If Kafka or Postgres go down, our app fails. This way, we’re self-contained.

On paper, this looks like a lightweight append-only logging mechanism.

In reality, you’ve just proposed a distributed log with replication, with bootstrapping for new nodes, with recovery after crashes, with ordering and conflict handling. Which is exactly the problem space of databases and log systems like Kafka.[5]

To see why this is dangerous, we need to talk about what a proper WAL actually does.

What a Real WAL Does Inside a Database

A write-ahead log in a database is not just „a file where we append stuff.“

It’s a carefully engineered component designed to guarantee durability, consistency, and recoverability under brutal conditions: power loss, crashes, disk failures, concurrent writes.[1]

At a high level, when a database processes a transaction, it writes the intent (the changes) to the WAL. It flushes that WAL entry to disk using fsync before acknowledging success. It later applies those changes to the main data files in what’s called checkpointing.[2] If the database crashes, on restart it replays the WAL to bring data files to a consistent state.

sequenceDiagram
    participant Client
    participant DB as Database
    participant WAL as Write-Ahead Log
    participant DataFiles as Data Files

    Client->>DB: BEGIN TRANSACTION
    Client->>DB: INSERT/UPDATE/DELETE

    DB->>WAL: Write changes to WAL
    WAL->>WAL: fsync() to disk
    Note over WAL: DURABLE NOW

    DB->>Client: ACK Success

    Note over DB,DataFiles: Later (async)...
    DB->>DataFiles: Apply changes (checkpoint)
    DB->>WAL: Truncate old entries

    Note over DB: If crash occurs...
    DB->>WAL: Replay WAL on restart
    WAL->>DataFiles: Restore consistent state

To make this work correctly and efficiently, you need to handle atomicity, meaning either all changes of a transaction are persisted or none. You need ordering, so WAL entries reflect a clear, consistent order that matches transactional semantics. You need durability guarantees with careful control of fsync, batching, and buffering to avoid acknowledging writes that never actually hit disk.[3] You need crash recovery, the ability to replay from a known point without corruption or partial writes. And you need compaction and checkpoints, periodically folding WAL contents into data files and truncating the log.

None of that is „just write JSON lines to a file.“

If you implement your own WAL inside your app, you are on the hook for all of this. What happens if the process crashes halfway through a write? How do you avoid partial, corrupt records? How do you guarantee ordering across concurrent requests? How do you make sure an acknowledged write is truly durable?

Databases have spent years getting this right. PostgreSQL’s WAL implementation alone has evolved over 20+ years.[4] It’s literally their job.

Then You Add Multiple Application Servers: Welcome to Distributed Systems Hell

Everything above was just for a single node.

Your colleague’s idea is worse: each app server has its own WAL, and then you replicate between them asynchronously.[15]

Picture this. App server A writes request #1000 to its local WAL. App server B writes request #1001 to its local WAL. Replication happens asynchronously in the background. A temporary network partition hits A but not B. A restarts halfway through sending its log to B. Meanwhile, a new app server C is added that needs to be bootstrapped.

sequenceDiagram
    participant A as App Server A
    participant B as App Server B
    participant C as App Server C (new)

    Note over A: Request #1000 arrives
    A->>A: Write to local WAL

    Note over B: Request #1001 arrives
    B->>B: Write to local WAL

    Note over A,B: Async replication starts...
    A--xB: Network partition!
    Note over A: Can't reach B

    Note over A: Crash & restart
    A->>A: Partial state

    Note over C: New server joins
    C->>A: Bootstrap from A?
    C->>B: Bootstrap from B?
    Note over C: Which has truth?

    Note over A,B,C: Conflicts:<br/>- Which request came first?<br/>- Missing entries?<br/>- Duplicate IDs?

You now have to solve a mess of problems.

Global Ordering and Conflicts

If you want a global audit log of all API requests, you need some notion of ordering. Which came first, the request logged on A or the one logged on B? If both A and B write a record with the same ID (bug, replay, duplicate), how do you resolve conflicts?

Databases and systems like Kafka use leader election and log replication.[5] They use monotonic sequence numbers or offsets. They use consensus protocols like Raft or Paxos to agree on order.[6][7] These protocols took years of academic research and production hardening to get right.

With „WAL per app server plus replication,“ you’ve sidestepped all those battle-tested protocols and replaced them with what exactly?

If your answer is „we don’t care about exact order,“ you’re implicitly saying you don’t care about strong guarantees. That might be fine, but then you need to be honest about what guarantees your system does not provide.

Bootstrapping New Replicas

Your load fluctuates, so you scale horizontally. New app server D comes up. It needs all past data so it can have a full copy of the audit log. You must copy the logs from A, B, and C to D. While copying, new requests are still coming in and being logged.

sequenceDiagram
    participant D as New Server D
    participant A as Server A
    participant B as Server B
    participant C as Server C
    participant Client as Clients

    Note over D: Starts up, needs all history

    D->>A: Copy your WAL?
    D->>B: Or copy from B?
    D->>C: Or copy from C?
    Note over D: Which one has complete data?

    par Copying historical data
        A->>D: Sending entries 1-1000...
    and New writes arriving
        Client->>B: Request #1001
        B->>B: Write to WAL
        Client->>C: Request #1002
        C->>C: Write to WAL
    end

    Note over D: Crash mid-copy!
    D--xA: Connection lost

    Note over D: Restart, try again...
    D->>B: Resume from where?
    Note over D: Inconsistent state,<br/>missing entries,<br/>duplicates?

Questions you now own: From which node or nodes does D bootstrap? Do you stream logs while copying historical ones? How do you ensure D has a consistent snapshot? How long until D is „in sync“? What happens if D crashes mid-bootstrap?

Databases and log systems ship snapshots, checkpoints, and log segments with well-defined protocols to handle this.[14] You’ll end up reinventing some inferior version of that.

Scaling Down Safely

When you scale down and remove app server B, what if B has WAL entries that A and C haven’t yet replicated? How do you ensure no data is lost? Do you block termination until replication catches up? What if the platform (Kubernetes, autoscaler) kills the pod anyway?

sequenceDiagram
    participant K8s as Kubernetes
    participant A as Server A
    participant B as Server B (terminating)
    participant C as Server C

    Note over B: Has entries 500-520
    Note over A: Has entries 1-510
    Note over C: Has entries 1-505

    B->>A: Replicating 511-520...
    B-->>C: Replicating 506-520...

    Note over K8s: Scale down triggered

    K8s->>B: SIGTERM (30s to shutdown)

    Note over B: Still replicating...
    B->>C: Sending entry 515...

    K8s->>B: SIGKILL (timeout!)
    Note over B: KILLED

    Note over A,C: Entries 516-520<br/>LOST FOREVER

    Note over C: Inconsistent state:<br/>Missing entries<br/>Gaps in sequence

Again, databases and proper message queues have clear mechanisms for ensuring safe removal of nodes, promoting followers to leaders, and redistributing partitions.

Your app-level WAL has none of that out-of-the-box. You’ll build it all yourself or accept painful data loss and inconsistencies.

Durability, Backups, and Restore: You Also Own That Now

Let’s say you somehow get the replication story working. You’re still not done.

Backup

Regulations or business requirements will likely demand long-term retention of audit logs, the ability to export and archive them to cheaper storage like S3, and protection against accidental deletion or disk corruption.

With homegrown WALs, you need your own backup strategy. You need your own tools to verify backups. You need a plan for rotating old segments, compressing logs, and indexing them.[13]

Databases and log systems have native backup tools, point-in-time recovery options, and battle-tested storage formats and indexes.

Restore

When something catastrophic happens (disk corruption, misconfiguration that wipes data, bug in your replication logic), you must be able to restore from backup to a consistent point without silently dropping or duplicating entries.

Again, if you roll your own WAL, you own the entire restore story. You’ll discover all the edge cases the hard way, likely in production.

The „Dependency Reduction“ Fallacy

Your colleague’s core argument is: „If we add a database or message queue, our system depends on it. If it fails, our application fails. Let’s keep the app self-contained.“

On the surface, that sounds reasonable. Fewer moving parts, fewer failure modes.

But here’s what’s actually happening. You are not eliminating dependencies. You are internalizing them.

graph LR
    subgraph "Using Postgres/Kafka"
        App1[Your App] --> DB[(Postgres/Kafka)]
        DB --> Features1["✓ Battle-tested<br/>✓ Monitoring tools<br/>✓ Backup/restore<br/>✓ Community support<br/>✓ 20+ years of fixes"]
    end

    subgraph "Homegrown WAL"
        App2[Your App + Custom WAL] --> Mess["✗ Built in 2 weeks<br/>✗ Only you understand it<br/>✗ No tooling<br/>✗ Unknown edge cases<br/>✗ You own all bugs"]
    end

    style DB fill:#90EE90,stroke:#333,stroke-width:2px
    style Mess fill:#FFB6C6,stroke:#333,stroke-width:2px

Instead of depending on Postgres or Kafka, which is built by hundreds of engineers and used by thousands of companies, you create a dependency on your own ad-hoc distributed WAL system that only you understand and that you built in a few weeks.

That’s not reducing risk. That’s trading a mature, well-studied dependency for a fragile, custom one. Moving complexity from infrastructure into your application code. Losing out on monitoring, tooling, community knowledge, performance tuning, and hard-won operational experience.

It’s like saying: „We don’t want to depend on TLS libraries. We’ll just implement our own encryption so we control everything.“

Technically possible. Practically reckless.

What You Actually Need: An Audit/Event Log, Not a Homegrown WAL

If you step back from the implementation details, your requirement is simple: „I want a reliable, queryable record of every API request.“

This is an event and audit log problem, not a „micro-database in my web server“ problem.

The usual, sane building blocks are a database table (or a few tables) where you store audit records, or a message queue like Kafka where you publish events and then sink them elsewhere. Maybe a data lake or warehouse downstream for analytics.

These systems already solve durability, replication, scaling, backup and restore, and queryability.

The app’s job is to emit events reliably into those systems. That’s where known patterns come in.

Use the Outbox Pattern for Reliable Event Emission

If your concern is „What if we write to our main database but fail to send the event to Kafka (or vice versa)?“ then you’re in classic dual-write territory.[8]

The standard solution is the Outbox pattern.[9] When handling a request, write your normal business data (like „order created“) to the database. In the same transaction, write an „outbox“ event row into an audit_events or outbox table. A background process (or CDC tool like Debezium) reads from that table and publishes events to Kafka or wherever they need to go.[10] Once published, the outbox row can be marked as processed or removed.

sequenceDiagram
    participant Client
    participant App as Application
    participant DB as Database
    participant Outbox as Outbox Table
    participant Worker as Background Worker
    participant Kafka

    Client->>App: POST /orders

    App->>DB: BEGIN TRANSACTION

    App->>DB: INSERT INTO orders (...)
    App->>Outbox: INSERT INTO outbox<br/>(event: order.created)

    DB->>App: COMMIT SUCCESS
    Note over DB,Outbox: Both writes atomic!

    App->>Client: 201 Created

    Note over Worker: Polling outbox...

    Worker->>Outbox: SELECT * FROM outbox<br/>WHERE processed = false
    Outbox->>Worker: event: order.created

    Worker->>Kafka: Publish event
    Kafka->>Worker: ACK

    Worker->>Outbox: UPDATE processed = true

Benefits: if the transaction commits, both the main data and the audit record exist. That’s atomicity. No custom WAL needed. You piggyback on the database’s transactional semantics and WAL. Easy to scale because the outbox consumer can be scaled independently.

Your audit log is now durable, consistent with your main data, and stored in a battle-tested system.

Use the Saga Pattern for Multi-Step Workflows

If your REST calls trigger workflows across multiple services (payment service, inventory service, notification service) and you want to track that, then your problem might be bigger than „just log requests.“ You’re dealing with distributed transactions and long-running workflows.

The Saga pattern helps here.[11] Break a big transaction into a series of local transactions. Each step emits events to a durable log, either a database or message queue. If a step fails, you execute compensating actions like refund or restock.[12]

sequenceDiagram
    participant O as Orchestrator
    participant P as Payment Service
    participant I as Inventory Service
    participant N as Notification Service
    participant Log as Event Store

    O->>Log: Start Saga: order-123
    O->>P: Charge payment
    P->>Log: Payment charged

    O->>I: Reserve inventory
    I--xO: Out of stock!
    I->>Log: Inventory failed

    Note over O: Saga failed, compensate!

    O->>P: Refund payment
    P->>Log: Payment refunded

    O->>Log: Saga completed (failed)

    Note over Log: Full audit trail:<br/>- Payment charged<br/>- Inventory failed<br/>- Payment refunded

Again, the key point is this: the saga orchestration logic relies on a reliable event store. It does not rely on each app server maintaining its own custom WAL and replicating it around.

Operational Reality: Tooling, Observability, and Maintenance

Even if you somehow manage to get a homegrown WAL „working,“ you still have to live with it.

For observability: How do you visualize the state of your distributed logs? How do you debug out-of-sync replicas?

For schema evolution: How do you evolve the structure of your audit records? Are you versioning events? Backfilling old ones?

For query patterns: How do you efficiently answer „show me all requests from user X in the last 30 days“? Are you going to write a custom indexing layer on top of flat files?

All of this is exactly what databases, search engines, and log systems are designed for.

By avoiding them, you don’t dodge complexity. You just rebuild a worse version.

🎯 Conclusion: Use Boring, Battle-Tested Infrastructure

If you want to track every REST request, do this.

Accept that you need a durable, queryable store for audit data. Use a proper database like Postgres, or a log system and message queue like Kafka or Pulsar with a sink to storage.

Use patterns like Outbox to avoid dual writes. Use Sagas to orchestrate multi-step workflows.

graph TB
    subgraph "The Right Way"
        App1[REST API] --> Outbox1[(Outbox Table)]
        Outbox1 --> Worker1[Background Worker]
        Worker1 --> Kafka1[Kafka/Event Log]
        Kafka1 --> Storage1[(Long-term Storage)]

        Note1["✓ Use battle-tested infrastructure<br/>✓ Outbox pattern for atomicity<br/>✓ Sagas for workflows<br/>✓ Let experts handle WALs"]
    end

    subgraph "The Wrong Way"
        App2[REST API<br/>+ Custom WAL] --> Replicate[Manual Replication]
        Replicate --> Problems["✗ Reinvent consensus<br/>✗ Reinvent recovery<br/>✗ Reinvent backups<br/>✗ Discover bugs in prod"]
    end

    style Note1 fill:#90EE90,stroke:#333,stroke-width:2px
    style Problems fill:#FFB6C6,stroke:#333,stroke-width:2px

Let these systems handle WALs, replication, partitioning, and recovery. That’s what they are built for.

And when someone suggests „Let’s just write our own write-ahead log in the app and replicate between instances,“ you can calmly say:

„That’s not dependency reduction. That’s us trying to build a distributed database inside our REST API. And we’re not going to out-engineer Postgres, Kafka, or decades of database research during a sprint.“

Boring, well-understood infrastructure is a feature, not a bug. Save your creativity for where it actually matters: your domain logic and your product, not reimplementing a WAL.

---

References

[1] Write-Ahead Logging (WAL) – Wikipedia https://en.wikipedia.org/wiki/Write-ahead_logging

[2] PostgreSQL Documentation: Reliability and the Write-Ahead Log https://www.postgresql.org/docs/current/wal-intro.html

[3] „On Disk IO, Part 1: Flavors of IO“ – Alex Miller, Confluent https://www.confluent.io/blog/okay-store-data-apache-kafka/

[4] PostgreSQL WAL Evolution and History https://www.postgresql.org/docs/current/wal-internals.html

[5] „The Log: What every software engineer should know about real-time data’s unifying abstraction“ – Jay Kreps, LinkedIn https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

[6] „In Search of an Understandable Consensus Algorithm (Raft)“ – Diego Ongaro and John Ousterhout, Stanford https://raft.github.io/raft.pdf

[7] „The Part-Time Parliament (Paxos)“ – Leslie Lamport ACM Transactions on Computer Systems, 1998 https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf

[8] „Dual Writes – The Unknown Cause of Data Inconsistencies“ – Gunnar Morling https://www.confluent.io/blog/dual-write-problem/

[9] „Pattern: Transactional Outbox“ – Chris Richardson, Microservices.io https://microservices.io/patterns/data/transactional-outbox.html

[10] „Change Data Capture with Debezium“ – Red Hat https://debezium.io/documentation/reference/

[11] „Sagas“ – Hector Garcia-Molina and Kenneth Salem Princeton University, 1987 https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf

[12] „Pattern: Saga“ – Chris Richardson, Microservices.io https://microservices.io/patterns/data/saga.html

[13] „Kafka Log Retention and Cleanup Policies“ – Confluent Documentation https://docs.confluent.io/platform/current/kafka/design.html#log-compaction

[14] „Streaming Replication“ – PostgreSQL Documentation https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION

[15] „Fallacies of Distributed Computing“ – Peter Deutsch and James Gosling https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing

Additional Reading

Books:

• Designing Data-Intensive Applications by Martin Kleppmann (O’Reilly, 2017)
• Database Internals by Alex Petrov (O’Reilly, 2019)
• Building Microservices by Sam Newman (O’Reilly, 2021)

Engineering Blog Posts:

• LinkedIn: „The Log: What every software engineer should know about real-time data’s unifying abstraction“ – https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
• Shopify: „Deconstructing the Monolith“ – https://shopify.engineering/deconstructing-monolith-designing-software-maximizes-developer-productivity
• Netflix: „Application Data Caching Using SSDs“ – https://netflixtechblog.com/application-data-caching-using-ssds-5bf25df851ef
• Stripe: „Online Migrations at Scale“ – https://stripe.com/blog/online-migrations
• Uber: „Designing Schemaless, Uber Engineering’s Scalable Datastore“ – https://www.uber.com/blog/schemaless-part-one/
• Airbnb: „Avoiding Double Payments in a Distributed Payments System“ – https://medium.com/airbnb-engineering/avoiding-double-payments-in-a-distributed-payments-system-2981f6b070bb
• Confluent: „Kafka as a Commit Log“ – https://www.confluent.io/blog/okay-store-data-apache-kafka/
• AWS: „Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases“ – https://www.allthingsdistributed.com/files/p1041-verbitski.pdf
• DoorDash: „Building Scalable Real-Time Event Processing with Kafka and Flink“ – https://doordash.engineering/2022/08/02/building-scalable-real-time-event-processing-with-kafka-and-flink/
• Meta (Facebook): „Scaling Memcache at Facebook“ – https://research.facebook.com/publications/scaling-memcache-at-facebook/
• Square: „Implementing the Outbox Pattern“ – https://developer.squareup.com/blog/implementing-the-outbox-pattern/
• Datadog: „Kafka at Datadog“ – https://www.datadoghq.com/blog/kafka-at-datadog/