|
I found this interesting repository BIRA: LLM Watermark Evasion via Bias Inversion https://github.com/ml-postech/Bias-Inversion-Rewriting-Attack
Due to EU regulations it isnt wise to market anything as a watermark remover rather a dual use tool that incidentally removes watermarks is the only real compliant way already.
My understanding now based on that is the model uses k=1 or k=2 really short last token windows. The providers have a secret cryptographic key and modify it with the last 2 tokens as a PRNG seed to label all output tokens as red or green at random, green are biased slightly when sampling tokens. This way validating doesnt need to replay through the model or require probabilities. Just knowing what tokens are red vs green.
To clean text thus requires every 5-8 words about replacing with a non green list token. Basically keeping the signal towards 50/50 indistinguishable as if a human wrote it while if you are 80/20 its a strong indicator the LLM produced it.
Incidentally turning up model temperature makes more strong probabilities so that basing green is less impactful.
In some ways these hidden biases would seem to be worse quality of output as well as maybe messing up high temperature or causing artificial limits on it. Running open weight models is an easy win though as you control the sampling PRNG. Mostly this is a problem from the frontier closed model labs though any cloud provider likely must add it. Running local has utility. Otherwise its still quite simple to replace every 5 to 8 words each replacement 50/50 chance of breaking the statistical signal
|