Rendered at 13:25:48 GMT+0000 (Coordinated Universal Time) with Netlify.
parliament32 19 hours ago [-]
Note that watermarking (yes, including text) is a requirement[1] of the EU AI Act, and goes into effect in August 2026, so I suspect we'll see a lot more work in this space in the near future.
[1] Specifically, "...synthetic audio, image, video or text content, shall ensure that the outputs of the AI system are marked in a machine-readable format and detectable as artificially generated or manipulated", see https://artificialintelligenceact.eu/article/50/
raincole 18 hours ago [-]
EU really like unenforceable regulations, doesn't it?
u1hcw9nx 18 hours ago [-]
It's regulation for providers, so it's easy to enforce.
pegasus 18 hours ago [-]
Yeah, they also outlawed murder. And stealing. And bribing officials. All universally unenforceable. Weird...
mikestorrent 18 hours ago [-]
Well, consider the case with murder: they're not demanding that people proactively implement a system to prevent it from happening, are they? You're just not allowed to do it, in the sense that the system will attempt to find you, prove your guilt, and punish you after the fact.
pegasus 18 hours ago [-]
I imagine it would be the same for making (use of) models which don't add these watermarks, no? The punishable crime is providing or using the service.
themafia 18 hours ago [-]
> they're not demanding that people proactively implement a system to prevent it from happening, are they?
What do you think a "background check" is?
bicx 18 hours ago [-]
Definitely not murder prevention
themafia 18 hours ago [-]
That's absolutely what they are. That and other crimes. That's why they're mandatory, by law, in certain industries. That's _precisely_ why we started using them: to prevent the easily preventable.
I suppose this logic stands in the way of a corporation getting what it wants and so it's automatically offensive to the HN "job seeking" crowd; however, even a basic reading of the history shows it's completely true.
squigz 18 hours ago [-]
There are various systems meant to (attempt to) prevent it from happening, yes, from firearms laws to police forces
But picking out murder and ignoring the other ones which are far more analogous to the regulations mentioned seems a bit disingenuous...
hansihe 18 hours ago [-]
What do you mean? There is nothing unenforceable about this.
nightski 18 hours ago [-]
How would you prove that something was generated by AI yet did not include a watermark?
pegasus 18 hours ago [-]
You generate it with that particular AI and look for the watermark :/
18 hours ago [-]
littlestymaar 18 hours ago [-]
You can trivially enforce that at the AI provider level, which covers 99% of the problem the law is designed to address.
Of course it doesn't cover the issue of foreign state psyop operations but the fact that enforcing laws against organized crime and adversary state actors is hard isn't specific to AI.
mikestorrent 18 hours ago [-]
Are you not aware of open-weights models and local generation? I think the vast majority of deepfake content is being genned in basements on RTX cards, not on public providers. People already have all this content, and have archives of it, and can run it airgapped. Cat is out of bag.
pegasus 18 hours ago [-]
I would be very surprised if that would be the case. Maybe you mean deepfake content generated by organized crime or state actors, but that surely is a tiny fraction of what's being generated on Grok or other platforms.
littlestymaar 18 hours ago [-]
I am well aware of them, and I'm well aware that they are very niche as I'm the only one of my surrounding to use one of those. And those very models are being developed by tech giants and VC backed companies, on which regulation have leverage.
The fact that a small black market exists doesn't mean regulating the mainstream market doesn't matters.
Also, most people like you fail to realizes that the EU only has mandate from the member states to regulate the economy. The EU has no business dealing with people using SDXL finetunes on RTX cards in their garage.
mikestorrent 11 hours ago [-]
> The EU has no business dealing with people using SDXL finetunes on RTX cards in their garage.
I agree in theory, but all it takes is one deepfake video to cause the kinds of trouble the regulations are designed to stop, right?
littlestymaar 1 hours ago [-]
No. Again, this regulation is about regulating businesses because that's what the EU is about.
The general use or creation of deepfake for porn, harassment, or election manipulation, is outside of what the EU can regulate as an institution, it is the responsibility of member states. (The same way the EU can impose rules on platform with respect to copyright violations, but cannot enact rules against piracy in general, these are always made by member states).
parliament32 16 hours ago [-]
You don't have to prove anything? You just have to mark the outputs of your slop generator appropriately. "Proving" one way or another is their problem when it comes to enforcement.
vee-kay 9 hours ago [-]
[dead]
jamiecode 18 hours ago [-]
The text watermarking is the more interesting problem here. Image watermarking is fairly tractable - you can embed a robust signal in spatial or frequency domains. Text watermarking works by biasing token selection at generation time, and detection is a statistical test over that distribution.
Which means short texts are basically useless. A 50-token reply has too little signal for the test to reach any confidence. The original SynthID text paper puts minimum viable detection at a few hundred tokens - so for most real-world cases (emails, short posts, one-liners) it just doesn't work.
The other thing: paraphrase attacks break it. Ask any other model to rewrite watermarked text and the watermark is gone, because you're now sampling from a different distribution. EU compliance built on top of this feels genuinely fragile for anything other than long-form content from controlled providers.
pegasus 18 hours ago [-]
Long-form content from controlled providers is by far the lion's share of what needs this regulation, at least at the moment. Perfect is the enemy of good enough. Or at least of better than the status-quo.
dpe82 18 hours ago [-]
The act doesn't explicitly require watermarking, does it?
ekjhgkejhgk 18 hours ago [-]
Link to the paper please?
doctorpangloss 18 hours ago [-]
haha "you" say this, when your comment was written by an LLM! it's watermarked!
gregorkas 19 hours ago [-]
I genuinely feel that in this AI world we need the inverse. That every analogue or digital photo taken by traditional means of photography will need to be signed by a certificate, so anyone can verify its authenticity.
Doesn't this require a paid certificate? that effectively blocks open source software/hardware from implementing it.
yjftsjthsd-h 19 hours ago [-]
And how do you fix the analog hole? Because if you can point your "verified" camera at a sufficiently high-resolution screen, we're worse off than when we started.
byfx 6 hours ago [-]
There are some techniques to detect recapture, e.g.: Moiré Pattern, Glare, JPEG Grid Artifacts, Channel Phase Shift, Screen Emission, Chromatic Aberration.
If those are combined, the effort and cost to fake a photo rises significantly.
cedws 19 hours ago [-]
Yes, I’m more worried about the false confidence such technology could create. Implement an authenticity mechanism and it will be treated as truth. Powerful people will have the means to spoof photographic evidence.
fny 18 hours ago [-]
You can have other sensors that tell you it's a screen, maybe require a Live Photo, maybe also upload to a third party service faster than generation is possible? In the end I think we'd end up somewhere like with cryptography: generating a real fake might be theoretically possible but it could be made prohibitively expensive to generate.
0x696C6961 19 hours ago [-]
Or just extract the certificate from the hardware you own.
staticassertion 19 hours ago [-]
That is presumably a very expensive endeavor. We already have hardware that attempts to mitigate this and while I think it's possible for the government it's certainly not trivial.
lern_too_spel 19 hours ago [-]
This is a "solved" problem. Vendors whose keys are extractable get their licenses revoked. The verifier checks the certificate against a CRL.
lern_too_spel 19 hours ago [-]
Depth sensor information.
gumby271 19 hours ago [-]
I'm sure Apple would love that too. More seriously, would that also mean all editing tools would need to re-sign a photo that was previously signed by the original sensor. How do we distinguish an edit that's misleading vs just changing levels? It's an interesting area for sure, but this inverse approach seems much trickier.
alwa 17 hours ago [-]
CAI’s Content Credential standard accommodates what you suggest, as far as re-signing/provenance, with a chain kind of approach. It supports embedding “ingredient thumbnails” in an image’s manifest, and/or the image’s manifest can embed or link back to source images that are in turn also signed [2].
It feels like the approach assumes a media environment where a professional wants to provably “show their work,” where authenticity adds value to a skeptical audience.
In that spirit, then, I understand CAI’s intention [0] to be to vest that judgment with the creator, and ultimately the viewer: if my purpose is to prove myself, I’d want to show enough links in the chain that the viewer checking my work can say “oh I see how A relates to B, to C,” and so on. If I don’t want to prove myself, well… then I won’t.
I don’t know Adobe’s implementation well enough to know how often they save a CC manifest, and their beta is vague in just referring to “editing history.” [1] I get the impression that they’re still dialing in the right level of detail to capture by default. Maybe even just “came from Firefly” and “Photoshop wuz here.”
But if I want to prove this Nikon Z9 recorded these pixels at this time and place, or “I am the BBC and yes I published this,” or “only the flying monkey was GenAI, the rest was real” I could conceivably put together a toolchain (independently of Adobe) to prove it in more detail.
You'd have to provide both images, and let the end user determine whether they think it's misleading.
hedora 19 hours ago [-]
Some cameras support this, but usually only for raw.
Note that your cell phone camera is using gen AI techniques to counteract sensor noise.
Was that famous person in the background really there, or a hallucination filling in static?
Who knows at this point? So, the signatures you proposed need to have some nuance around what they’re asserting.
graypegg 19 hours ago [-]
To be fair, I think just signing details about the way an image was assembled makes sense. Deciding on fake vs real doesn't have to be done at time of capture. We store things like the aperture size, sensitivity, camera name/model, etc in the EXIF data, including details about the image processing pipeline seems like a logical step. (With a signature verification scheme... and I guess also trying to embed that in the actual bitmap data)
There is no original image to recover, since we can't capture and describe every photon, so it's not a "fake vs real" image signature... that would be a UI choice the image viewer client would make based on the pipeline data in the image.
osculum 18 hours ago [-]
Years ago, I worked at Apple at the same time as Ian Goodfellow. This was before ChatGPT (I'd say around 2019).
I had the chance to chat with him, and what I remember most was his concern that GANs would eventually be able to generate images indistinguishable from reality, and that this would create a misinformation problem. He argued for exactly what you’re mentioning: chips that embed cryptographic proof that a photo was captured by a camera and haven't been modified.
andrewmcwatters 19 hours ago [-]
[dead]
omgmo 8 hours ago [-]
What about spoofing a SynthID false positive for a real image or video? Who can arbitrate what is true?
I think that AI service providers should have safeguards and encoded attribution. This solution helps when people lazily share things with friends or on social media I suppose, rather than stopping motivated bad actors.
The only way to actually implement this I think would be to ban all local models, and to have the service providers store perceptual hashes all generated images and video. It feels like the cat's out of the bag already though (for images at least).
Aldipower 18 hours ago [-]
As a synthesizer collector with serious GAS I find this particular name very offensive.
manbash 18 hours ago [-]
It's nice that they explain the "what" (...it is doing) but not the "why". Who is going to use it and for what reasons?
Also, if it's essentially a sort of metadata, can't the output generated image be replicated (e.g. screenshot) and thus stripped of any such data?
ainch 17 hours ago [-]
I've heard of journalists using it to try and figure out whether images sent by sources were generated. In their Nano Banana 2 release blogpost, Google mentioned that SynthID has been used ~20 million times, so there's clearly some interest in identifying AI-generated images.
throwaway13337 19 hours ago [-]
These sorts of tools will only be able to positively identify a subset of genAI content. But I suspect that people will use it to 'prove' something is not genAI.
In a sense, the identifier company can be an arbiter of the truth. Powerful.
Training people on a half-solution like this might do more harm than good.
greensoap 19 hours ago [-]
It will just be an arms race if we try to prove "not genAI." Detectors will improve, genAI will improve without marking (opensource and state actors will have unmarked genAI even if we mandate it).
Marking real from lense through digital life is more practical. But then what do we do with all the existing hardware that doesn't mark real and media that preexisited this problem.
throwaway13337 19 hours ago [-]
I agree. A mechanism to voluntarily attach a certificate metadata about the media record from the device seems like a better idea. That still can be spoofed, though.
In the end, society has always existed on human chains of trust. Community. As long as there are human societies, we need human reputation.
observationist 19 hours ago [-]
You could take a picture or video with your phone of a screen or projection of an altered media and thereby capture a watermarked "verified" image or video.
None of these schemes for validation of digital media will work. You need a web of trust, repeated trustworthy behavior by an actor demonstrating fidelity.
You need people and institutions you can trust, who have the capability of slogging through the ever more turbulent and murky sea of slop and using correlating evidence and scientific skepticism and all the cognitive tools available to get at reality. Such people and institutions exist. You can also successfully proxy validation of sources by identifying people or groups good at identifying primary sources.
When people and institutions defect, as many legacy media, platforms, talking heads, and others have, you need to ruthlessly cut them out of your information feed. When or if they correct their mistake, just follow tit for tat, and perhaps they can eventually earn back their place in the de-facto web of trust.
Google's stamp of approval means less than nothing to me; it's a countersignal, indicating I need to put even more effort than otherwise to confirm the truthfulness of any claims accompanied by their watermark.
sippeangelo 19 hours ago [-]
It is actively harmful to society. Slap SynthID on some of the photographic evidence from the unreleased Epstein files and instantly de-legitimize it. Launder a SynthID image through a watermark free model and it's legit again. The fact that it exists at all can't be interpreted in any other way than malice.
kingstnap 19 hours ago [-]
It's security through obscurity. I'm sure with the technical details or even just sufficient access to a predictive oracle you could break this.
But I suppose it ads friction so better than nothing.
Watermarking text without affecting it is an interesting seemingly weird idea. Does it work any better than (with knowledge of the model used to produce said text), just observing the perplexity is low because its "on policy" generated text.
u1hcw9nx 20 hours ago [-]
This technology could be used to copyrights as well.
>The watermark doesn’t change the image or video quality. It’s added the moment content is created, and designed to stand up to modifications like cropping, adding filters, changing frame rates, or lossy compression.
But does it survive if you use another generative image model to replicate the image?
elpocko 18 hours ago [-]
It doesn't. I don't have a link for you right now but there was a post on reddit recently showing that SynthID is removed from images by passing the image through a diffusion model for a single step at low denoise. The output image is identical to the input image (to the human eye).
lxgr 19 hours ago [-]
> This technology could be used to copyrights as well.
Extremely doubtful, due to the way that embedding and diffusion works. I would be utterly floored if they had achieved that.
ks2048 18 hours ago [-]
How about a database of verified non-AI images?
I'm thinking of historical images, where there aren't a huge number of existing images and no more will ever be created.
If I see something labeled "Street scene in Paris, 1905". I want to know if it is legit.
PaulHoule 19 hours ago [-]
...But it can be hard to tell the difference between content that’s been
AI-generated, and content created without AI.
Pro-Tip: Something like that Sherbet colored dog is always AI generated
pavel_lishin 19 hours ago [-]
You'd be surprised what dog owners do sometimes.
galleywest200 18 hours ago [-]
This is great, but there is no way for me to verify if groups or nation states can pay for a special contract where they do not have to have their outputs watermarked.
18 hours ago [-]
zelias 19 hours ago [-]
Seems like this really just validates whether a piece of AI content was generated by Google, not AI generated in general
What incentive do open models have to adopt this?
18 hours ago [-]
squigz 19 hours ago [-]
Looks like there's a lot more info here, at least about the text version.
This is from 2025. Did something new happen? What am I missing here?
gigel82 18 hours ago [-]
Reposting a comment I made on an earlier thread on this.
We need to be super careful with how legislation around this is passed and implemented. As it currently stands, I can totally see this as a backdoor to surveillance and government overreach.
If social media platforms are required by law to categorize content as AI generated, this means they need to check with the public "AI generation" providers. And since there is no agreed upon (public) standard for imperceptible watermarks hashing that means the content (image, video, audio) in its entirety needs to be uploaded to the various providers to check if it's AI generated.
Yes, it sounds crazy, but that's the plan; imagine every image you post on Facebook/X/Reddit/Whatsapp/whatever gets uploaded to Google / Microsoft / OpenAI / UnnamedGovernmentEntity / etc. to "check if it's AI". That's what the current law in Korea and the upcoming laws in California and EU (for August 2026) require :(
"Generate a pure white image." "Generate a pure black image." Channel diff, extract steganographic signature for analysis.
alibero 18 hours ago [-]
I've been looking into this. There seems to be some mostly-repeating 2D pattern in the LSB of the generated images. The magnitude of the noise seems to be larger in the pure black image vs pure white image. My main goal is to doctor a real image to flag as positive for SynthID, but I imagine if you smoothed out the LSB, you might be able to make images (especially very bright images) no longer flag as SynthID? Of course, it's possible there's also noise in here from the image-generation process...
Gemini really doesn't like generating pure-white images but you can ask it to generate a "photograph of a pure-white image with a black border" and then crop it. So far I've just been looking at pure images and gradients, it's possible that more complex images have SynthID embedded in a more complicated way (e.g. a specific pattern in an embedding space).
amingilani 19 hours ago [-]
I just tried this idea, and it looks like it isn't that simple.
> "Generate a pure white image."
It refused no matter how I phrased it ¯\_(ツ)_/¯
> "Generate a pure black image."
It did give me one. In a new chat, I asked Gemini to detect SynthID with "@synthid". It responded with:
> The image contains too little information to make a diagnosis regarding whether it was created with Google AI. It is primarily a solid black field, and such content typically lacks the necessary data for SynthID to provide a definitive result.
Further research: Does a gradient trigger SynthID? IDK, I have to get back to work.
[1] Specifically, "...synthetic audio, image, video or text content, shall ensure that the outputs of the AI system are marked in a machine-readable format and detectable as artificially generated or manipulated", see https://artificialintelligenceact.eu/article/50/
What do you think a "background check" is?
I suppose this logic stands in the way of a corporation getting what it wants and so it's automatically offensive to the HN "job seeking" crowd; however, even a basic reading of the history shows it's completely true.
But picking out murder and ignoring the other ones which are far more analogous to the regulations mentioned seems a bit disingenuous...
Of course it doesn't cover the issue of foreign state psyop operations but the fact that enforcing laws against organized crime and adversary state actors is hard isn't specific to AI.
The fact that a small black market exists doesn't mean regulating the mainstream market doesn't matters.
Also, most people like you fail to realizes that the EU only has mandate from the member states to regulate the economy. The EU has no business dealing with people using SDXL finetunes on RTX cards in their garage.
I agree in theory, but all it takes is one deepfake video to cause the kinds of trouble the regulations are designed to stop, right?
The general use or creation of deepfake for porn, harassment, or election manipulation, is outside of what the EU can regulate as an institution, it is the responsibility of member states. (The same way the EU can impose rules on platform with respect to copyright violations, but cannot enact rules against piracy in general, these are always made by member states).
Which means short texts are basically useless. A 50-token reply has too little signal for the test to reach any confidence. The original SynthID text paper puts minimum viable detection at a few hundred tokens - so for most real-world cases (emails, short posts, one-liners) it just doesn't work.
The other thing: paraphrase attacks break it. Ask any other model to rewrite watermarked text and the watermark is gone, because you're now sampling from a different distribution. EU compliance built on top of this feels genuinely fragile for anything other than long-form content from controlled providers.
https://www.hackerfactor.com/blog/index.php?%2Farchives%2F10...
It feels like the approach assumes a media environment where a professional wants to provably “show their work,” where authenticity adds value to a skeptical audience.
In that spirit, then, I understand CAI’s intention [0] to be to vest that judgment with the creator, and ultimately the viewer: if my purpose is to prove myself, I’d want to show enough links in the chain that the viewer checking my work can say “oh I see how A relates to B, to C,” and so on. If I don’t want to prove myself, well… then I won’t.
I don’t know Adobe’s implementation well enough to know how often they save a CC manifest, and their beta is vague in just referring to “editing history.” [1] I get the impression that they’re still dialing in the right level of detail to capture by default. Maybe even just “came from Firefly” and “Photoshop wuz here.”
But if I want to prove this Nikon Z9 recorded these pixels at this time and place, or “I am the BBC and yes I published this,” or “only the flying monkey was GenAI, the rest was real” I could conceivably put together a toolchain (independently of Adobe) to prove it in more detail.
[0] https://spec.c2pa.org/specifications/specifications/2.2/spec...
[1] https://opensource.contentauthenticity.org/docs/manifest/und...
[2] https://opensource.contentauthenticity.org/docs/c2patool/doc...
Note that your cell phone camera is using gen AI techniques to counteract sensor noise.
Was that famous person in the background really there, or a hallucination filling in static?
Who knows at this point? So, the signatures you proposed need to have some nuance around what they’re asserting.
There is no original image to recover, since we can't capture and describe every photon, so it's not a "fake vs real" image signature... that would be a UI choice the image viewer client would make based on the pipeline data in the image.
I had the chance to chat with him, and what I remember most was his concern that GANs would eventually be able to generate images indistinguishable from reality, and that this would create a misinformation problem. He argued for exactly what you’re mentioning: chips that embed cryptographic proof that a photo was captured by a camera and haven't been modified.
I think that AI service providers should have safeguards and encoded attribution. This solution helps when people lazily share things with friends or on social media I suppose, rather than stopping motivated bad actors.
The only way to actually implement this I think would be to ban all local models, and to have the service providers store perceptual hashes all generated images and video. It feels like the cat's out of the bag already though (for images at least).
Also, if it's essentially a sort of metadata, can't the output generated image be replicated (e.g. screenshot) and thus stripped of any such data?
In a sense, the identifier company can be an arbiter of the truth. Powerful.
Training people on a half-solution like this might do more harm than good.
Marking real from lense through digital life is more practical. But then what do we do with all the existing hardware that doesn't mark real and media that preexisited this problem.
In the end, society has always existed on human chains of trust. Community. As long as there are human societies, we need human reputation.
None of these schemes for validation of digital media will work. You need a web of trust, repeated trustworthy behavior by an actor demonstrating fidelity.
You need people and institutions you can trust, who have the capability of slogging through the ever more turbulent and murky sea of slop and using correlating evidence and scientific skepticism and all the cognitive tools available to get at reality. Such people and institutions exist. You can also successfully proxy validation of sources by identifying people or groups good at identifying primary sources.
When people and institutions defect, as many legacy media, platforms, talking heads, and others have, you need to ruthlessly cut them out of your information feed. When or if they correct their mistake, just follow tit for tat, and perhaps they can eventually earn back their place in the de-facto web of trust.
Google's stamp of approval means less than nothing to me; it's a countersignal, indicating I need to put even more effort than otherwise to confirm the truthfulness of any claims accompanied by their watermark.
But I suppose it ads friction so better than nothing.
Watermarking text without affecting it is an interesting seemingly weird idea. Does it work any better than (with knowledge of the model used to produce said text), just observing the perplexity is low because its "on policy" generated text.
>The watermark doesn’t change the image or video quality. It’s added the moment content is created, and designed to stand up to modifications like cropping, adding filters, changing frame rates, or lossy compression.
But does it survive if you use another generative image model to replicate the image?
That's been a thing for a while: https://en.wikipedia.org/wiki/Digital_watermarking
I'm thinking of historical images, where there aren't a huge number of existing images and no more will ever be created.
If I see something labeled "Street scene in Paris, 1905". I want to know if it is legit.
What incentive do open models have to adopt this?
https://ai.google.dev/responsible/docs/safeguards/synthid
We need to be super careful with how legislation around this is passed and implemented. As it currently stands, I can totally see this as a backdoor to surveillance and government overreach.
If social media platforms are required by law to categorize content as AI generated, this means they need to check with the public "AI generation" providers. And since there is no agreed upon (public) standard for imperceptible watermarks hashing that means the content (image, video, audio) in its entirety needs to be uploaded to the various providers to check if it's AI generated.
Yes, it sounds crazy, but that's the plan; imagine every image you post on Facebook/X/Reddit/Whatsapp/whatever gets uploaded to Google / Microsoft / OpenAI / UnnamedGovernmentEntity / etc. to "check if it's AI". That's what the current law in Korea and the upcoming laws in California and EU (for August 2026) require :(
Some previous discussion:
https://news.ycombinator.com/item?id=45071677
"Generate a pure white image." "Generate a pure black image." Channel diff, extract steganographic signature for analysis.
Gemini really doesn't like generating pure-white images but you can ask it to generate a "photograph of a pure-white image with a black border" and then crop it. So far I've just been looking at pure images and gradients, it's possible that more complex images have SynthID embedded in a more complicated way (e.g. a specific pattern in an embedding space).
> "Generate a pure white image."
It refused no matter how I phrased it ¯\_(ツ)_/¯
> "Generate a pure black image."
It did give me one. In a new chat, I asked Gemini to detect SynthID with "@synthid". It responded with:
> The image contains too little information to make a diagnosis regarding whether it was created with Google AI. It is primarily a solid black field, and such content typically lacks the necessary data for SynthID to provide a definitive result.
Further research: Does a gradient trigger SynthID? IDK, I have to get back to work.