GitHub

DB Diff

Know when your data drifts

demo

Supported engines

Two Postgres databases are enough to see it work. The checksum is built in SQL, so only the digests and the rows that differ leave the server.

PostgreSQL

Try it yourself:

$ db-diff --source="$SRC" --target="$DST" --table=orders

The MySQL dialect has to work around unsigned integers and charset decoding in SQL. The result is the same checksum, byte for byte, as the other two.

MySQL

Try it yourself:

$ db-diff --source="$MYSQL_SRC" --target="$MYSQL_DST" --table=orders

ClickHouse folds with groupBitXor and has to reverse the byte order of the digest. It still lands on the same value the golden test pins.

ClickHouse

Try it yourself:

$ db-diff --source="$CH_SRC" --target="$CH_DST" --table=orders

Use cases

Compare the columns that matter. Excluding the noisy ones drops the rows whose only difference lives there, and the checksum reads less data while it does it.

Column control

Try it yourself:

$ db-diff --source="$SRC" --target="$DST" --table=line_items --exclude=quantity

A cross engine pair is refused before it connects. Engines render values their own way, so guessing would report every row as changed.

Engine boundary

Try it yourself:

$ db-diff --source="$PG_SRC" --target="$MYSQL_DST" --table=orders

Get started

DB Diff is built for the case a dump and compare cannot handle: a table with millions of rows where you need to know which rows drifted, without transferring the table.

Three engines

PostgreSQL, MySQL and ClickHouse, with one checksum contract across all three

Checksums only

Server-side MD5 chunking, so only digests cross the wire when the tables match

Pinpoint drift

A differing segment is read row by row and reported as primary keys

Column control

Include or exclude columns to leave the noisy ones out of the comparison

CI ready

Exit 0 when the tables match, 1 when they differ, 2 when the run failed

Tunable segments

Trade query count against how much is read when something differs

Install in one line

$ go install matto.club/vetrina/db-diff@latest
$ brew tap clubmatto/vetrina https://github.com/clubmatto/vetrina && brew install clubmatto/vetrina/db-diff

Requires Go 1.24+ or Homebrew.

Read the docs

Requirements and limits

Engines: PostgreSQL 14 or newer, MySQL and ClickHouse. Both databases must run the same engine; cross engine comparison is DB Diff Pro.

Table: one integer primary key named id, with the same table name and column order on both sides.

Run: a differing segment is read whole, and the comparison is not a transaction. Pin the session timezone and charset on both connections.

License: MIT. The releases page has the current version and checksums.

Algorithm notes

DB Diff does the comparison inside the database. The wire carries checksums when the tables match, and rows only when they do not.

  1. 01

    Segment the table

    The table is cut into ranges of primary keys, 10,000 by default. Both sides can address the same range without moving a row.

  2. 02

    Checksum each segment

    Each side hashes every compared column on its own, folds the digests with XOR, and returns one 64-bit checksum. Equal checksums skip the segment.

  3. 03

    Read only what differs

    A differing segment is read row by row and compared column by column. The output is the primary keys that changed.

Row checksum 
md5( null_bitmap || md5(COALESCE(col_a, '')) || md5(COALESCE(col_b, '')) || ... ) -> first 8 bytes
segment = XOR(row checksums)

The comparison runs within one engine. Engines render values their own way: a boolean is true in PostgreSQL and 1 in MySQL, so a cross engine run is refused rather than reported as every row changed. DB Diff Pro adds the canonical form that makes it possible.

Read the full algorithm notes

DB Diff Pro

Need to compare across engines? DB Diff Pro adds cross engine support to DB Diff. A wall clock reading is kept distinct from an instant, and floats are compared bit for bit rather than by how the engine happens to print them.

Learn more