The Write Ahead Log or WAL is typical across many databases and streaming technologies such as Postgres or Kafka.
It is an append-only set of files that contains either new messages in the case of Kafka or recent changes to data, as in the case of relational databases like Postgres. Since this chapter is all about relational databases, we’ll talk about how this is important for Postgres.
Both Postgres and other services use the WAL. Internally, it’s read when a CHECKPOINT happens. The checkpoint is responsible for writing these changes to the data files. Externally, the WAL is the log in a typical publish/subscribe model. PG posts to the WAL and other processes can subscribe to it to get data changes.
External subscribers might be Postgres Follower servers or another service such as Kafka Connect, which can use the data for Change Data Capture (CDC) pipelines.
Typically, when a service subscribes to PG’s WAL, it will create a Replication Slot by running the query select pg_create_physical_replication_slot('slot_name'); It can then use this slot to get a feed of WAL changes.
The replication slot keeps track of which WAL files the subscriber has acknowledged (ACK). Once all slots have sent an ACK to a particular WAL file, and a CHECKPOINT has occurred, PG can delete it.
If a subscriber goes offline, Postgres will not delete WAL files, which could easily cause the hard drive to fill up completely. To fix, delete the replication slot or bring the subscriber back online. It’s good to monitor the subscriber’s health and available hard drive space to avoid server downtime.