LegalDrift: Semantic Version Control for Law
Redlines tell you what words changed; LegalDrift tells you if the substantive meaning changed. By leveraging statistical NLP and embedding-based monitoring, this tool detects semantic drift across legal document revisions. It allows contract managers to verify that renegotiated agreements haven't silently introduced new obligations, treating legal documents with the same observability we apply to software state.
When a software engineer updates code, they use Git diffs to see exactly what changed. When a lawyer updates a contract, they use redlines.
The problem
Redlines have a fatal flaw: they operate at the character and word level. They tell you what words changed, but they cannot tell you if the substantive meaning changed.
A stylistic rewrite of a boilerplate clause will light up a redline document in terrifying red ink, even though the legal effect is identical. Conversely, changing the word "may" to "shall" is a tiny redline blip that completely alters the allocation of risk.
LegalDrift is a statistical monitoring tool for legal document revision, essentially semantic version control for law.
How it works
LegalDrift converts each document into a high-dimensional vector using a language model trained on legal text (Legal-BERT). But it doesn't just do a simple cosine similarity check; it treats the collection of vectors from a document (or chunks of a document) as a statistical distribution, then asks a formal question: are these two distributions the same, or have they diverged?
| Test | What it measures | Why it's included |
|---|---|---|
| Kolmogorov-Smirnov | Overall shape of the cumulative distributions | Works across all PCA-reduced dimensions |
| Mann-Whitney U | Location shift (median/mean drift) | Robust to outliers |
| Maximum Mean Discrepancy | Kernel-based distributional distance | Permutation-based p-values, catches subtle shifts |
| Energy Distance | Geometric separation between sample clouds | Sensitive to distribution shape, not just location |
A low combined p-value (typically < 0.05) indicates the two documents occupy measurably different regions of the embedding space. Here's a worked example of how to read the four-test output when a real drift signal is present:
Combined p-value fell below the 0.05 threshold.
No significant drift was detected here (p≈1.0 across all four tests), despite the substantive deontic changes, and that's worth explaining rather than hiding. Two things are going on. The bundled CLI (detect/chunks/compare) embeds each whole document as a single pooled vector, so its statistical tests always compare one sample against one sample: degenerate by construction, independent of input. Running at sentence granularity through the public API fixes that mechanically, but Legal-BERT's pooled sentence embeddings still didn't separate these two clauses by modality (may vs. shall, percentages, day counts); they cluster on lexical/topical similarity more than on deontic force. That's a real, useful finding about where this technique's sensitivity currently ends, not a contradiction of the method.
Localized drift: finding the needle in the haystack
Full-document comparison has a well-known weakness: a large, mostly unchanged 50-page contract can dilute a significant, malicious change inserted into a single clause.
To counter this, LegalDrift supports localized drift detection. It splits documents into semantically coherent chunks (paragraphs, sections, or sentences) and aligns them:
legaldrift chunks contract_v1.txt contract_v2.txt
Chunked Drift Detection Results
============================================================
🔴 [DRIFT] 3. PAYMENT
p=0.0034, severity=0.2417
🟢 [OK] 1. DEFINITIONS
p=0.8912, severity=0.0124
🟡 [NEW] 10. AI GOVERNANCE
added in current document
This tells the human reviewer exactly where to look. You can safely skim the definitions, but you need to heavily scrutinize the payment terms. (The output above is illustrative of the intended UX; see the note above the gif for what a real run against synthetic revisions actually produced.)
Concept extraction without heavy pipelines
Not every task requires the full statistical pipeline. LegalDrift includes a lightweight regex-based extractor that flags common legal concept classes without requiring a heavy NLP pipeline:
obligation("shall", "must")permission("may", "has the right")prohibition("shall not", "is prohibited")
This allows for rapid, offline metadata tagging before the heavier embedding tests run.
Audit trails and observability
Legal workflows require strict reproducibility. A core feature of LegalDrift is its DriftHistory module, which persists every detection run to an SQLite database, turning the tool from an ad-hoc CLI script into a persistent audit system that can prove a document was computationally verified for semantic stability on a specific date.
Limitations
A p-value of 0.01 means the embeddings are different. It does not mean the difference is legally material.
- A shifted comma in a damages cap can be legally catastrophic yet statistically invisible.
- A stylistic rewrite of a boilerplate clause can be legally trivial yet statistically detectable.
- Confirmed empirically, not just in theory: the bundled CLI compares whole-document embeddings (n=1 per side), which makes its statistical tests degenerate regardless of input. At sentence granularity the tests run correctly, but in testing, Legal-BERT's pooled embeddings did not reliably separate clauses that differ mainly in deontic modality (obligation vs. permission) rather than topic, exactly the kind of change that matters most legally and least lexically.
LegalDrift is a screening layer, not a verdict. It treats legal documents with the same observability we apply to software state, acting as an early warning system for the humans who make the actual call.
Check out the library on PyPI or contribute on GitHub: OsamaMoftah/LegalDrift.