1024 weights and a bias, on top of BAAI/bge-m3. It catches health information, credentials and personal data written as ordinary English — the kind a regex ruleset cannot see.
Patient ID: 40219, sk-ant-…,
AKIA…, a base64 image. Anything shaped like a marker.
Catches 10 of 111 sensitive examples. 9.0%.
“The woman from Tuesday’s clinic, 34, has a 7mm asymmetric lesion on her left shoulder and a biopsy booked for the 20th.”
No marker. No pattern matches. 101 of 111 look like this.
The gate is a score, not a verdict, so the trade-off is a dial. Every number below is recomputed live over the 206-example gold set. Scores are five-fold cross-validated — each one comes from a head that never saw that example.
| Text | Should be | Rules | Gate | Score |
|---|
Shaded rows are the ones that matter: the rules miss them and the gate catches them. A red edge marks a row the gate gets wrong at the current threshold. The last 41 rows are held out of training: 20 fresh sentences written after the head was trained, and 21 short probe prompts, the ones that showed v0 holding “ok” and “thanks!”.
send verdict is not an assurance that text is safe.
This is a second gate that may only ever add a hold, behind whatever
deterministic checks you already run. It is a net, not a proof — and it is not
production-validated: the evidence is 206 training examples and 41 held-out ones, all written by one person.
| Approach | AUC | Notes |
|---|---|---|
| Qwen3.5-0.8B, prompted for a label | 0.4609 | At chance. Answered HEALTH for 106 of 127 and never once
SECRET or PII. Its 95.8% “catch rate” was a stopped clock. |
| Hashed word unigrams, no model | 0.7818 | Real signal, useless as a gate: 83.2% friction. Cannot tell talking about a password from containing one. |
| bge-m3 + logistic head | 0.9927 | Head v1: 110 of 111 caught at 5.3% friction, and none of 15 held-out short harmless prompts held (v0 held 8). |
from sentence_transformers import SentenceTransformer
from huggingface_hub import hf_hub_download
import json, numpy as np
head = json.load(open(hf_hub_download("YauhenBichel/privacy-gate-llm", "head-v1.json")))
enc = SentenceTransformer("BAAI/bge-m3")
w, mu, sd = map(np.array, (head["weights"], head["mean"], head["stdev"]))
v = enc.encode(["her biopsy is booked for the 20th"], normalize_embeddings=True)[0]
score = float(((v - mu) / sd) @ w + head["bias"])
hold = score > head["threshold"] # True
normalize_embeddings=True is required, not cosmetic: the head was fitted
on unit vectors, and an un-normalised one scores confidently wrong rather than
failing. Verified equivalent to the Ollama path to cosine 0.99993.