1024 weights and a bias, on top of BAAI/bge-m3. It catches health information, credentials and personal data written as ordinary English — the kind a regex ruleset cannot see.
Patient ID: 40219, sk-ant-…,
AKIA…, a base64 image. Anything shaped like a marker.
Catches 10 of 72 sensitive examples. 13.9%.
“The woman from Tuesday’s clinic, 34, has a 7mm asymmetric lesion on her left shoulder and a biopsy booked for the 20th.”
No marker. No pattern matches. 62 of 72 look like this.
The gate is a score, not a verdict, so the trade-off is a dial. Every number below is recomputed live over the 127-example gold set. Scores are five-fold cross-validated — each one comes from a head that never saw that example.
| Text | Should be | Rules | Gate | Score |
|---|
Shaded rows are the ones that matter: the rules miss them and the gate catches them. A red edge marks a row the gate gets wrong at the current threshold. The last 20 rows are the fresh set — written after the head was trained, as a generalisation check.
send verdict is not an assurance that text is safe.
This is a second gate that may only ever add a hold, behind whatever
deterministic checks you already run. It is a net, not a proof — and it is not
production-validated: the evidence is 147 examples written by one person in one day.
| Approach | AUC | Notes |
|---|---|---|
| Qwen3.5-0.8B, prompted for a label | 0.4609 | At chance. Answered HEALTH for 106 of 127 and never once
SECRET or PII. Its 95.8% “catch rate” was a stopped clock. |
| Hashed word unigrams, no model | 0.7391 | Real signal, useless as a gate: 85.5% friction. Cannot tell talking about a password from containing one. |
| bge-m3 + logistic head | 0.9924 | All 72 sensitive examples caught, zero leaks. |
from sentence_transformers import SentenceTransformer
from huggingface_hub import hf_hub_download
import json, numpy as np
head = json.load(open(hf_hub_download("YauhenBichel/privacy-gate-llm", "head-v0.json")))
enc = SentenceTransformer("BAAI/bge-m3")
w, mu, sd = map(np.array, (head["weights"], head["mean"], head["stdev"]))
v = enc.encode(["her biopsy is booked for the 20th"], normalize_embeddings=True)[0]
score = float(((v - mu) / sd) @ w + head["bias"])
hold = score > head["threshold"] # True
normalize_embeddings=True is required, not cosmetic: the head was fitted
on unit vectors, and an un-normalised one scores confidently wrong rather than
failing. Verified equivalent to the Ollama path to cosine 0.99993.