Apparently people who specialize in AI/ML have a very hard time trying to replicate the desired results when training models with ‘poisoned’ data. Is that true?
I’ve only heard that running images through a VAE just once seems to break the Nightshade effect, but no one’s really published anything yet.
You can finetune models on known bad and incoherent images to help it to output better images if the trained embedding is used in the negative prompt. So there’s a chance that making a lot of purposefully bad data could actually make models better by helping the model recognize bad output and avoid it.
A Variational AutoEncoder is a kind of AI that can be used to compress data. In image generators, a VAE is used to compress the images. The actual image AI works on the smaller, compressed image (the latent representation), which means it takes a less powerful computer (and uses less energy). It’s that which makes it possible to run Stable Diffusion at home.
This attack targets the VAE. The image is altered so that the latent representation looks like a very different image, but still roughly the same to humans. The actual image AI works on a different image. Obviously, this only works if you have the right VAE. So, it only works against open source AI; basically only Stable Diffusion at this point. Companies that use a closed source VAE cannot be attacked.
I guess it makes sense if your ideology is that information must be owned and everything should make money for someone. I guess some people see cyberpunk dystopia as a desirable future. It doesn’t seem to be a very effective attack but it may have some long-term PR effect. Training an AI costs a fair amount of money. People who give that away for free probably still have some ulterior motive, such as being liked. If instead you get the full hate of a few anarcho-capitalists that threaten digital vandalism, you may be deterred. Well, my two cents.
Thank you for explaining. I work in NLP and are not familiar with all CV acronyms.
That sounds like it kind if defeats the purpose if it only targets open source models. But yeah, makes sense that you would need the actual autoencoder in order to learn how to alter your data such that the representation from the autoencoder is different enough.
So there’s a chance that making a lot of purposefully bad data could actually make models better by helping the model recognize bad output and avoid it.
The thing is data poisoning is a arms race that the Ai side will win with ease. You can either solve it with pre processing or filtering. All it does is make the images look worse. I can’t think of a way that you can poison data that doesn’t take more effort to unpoison than to poison.
Apparently people who specialize in AI/ML have a very hard time trying to replicate the desired results when training models with ‘poisoned’ data. Is that true?
I’ve only heard that running images through a VAE just once seems to break the Nightshade effect, but no one’s really published anything yet.
You can finetune models on known bad and incoherent images to help it to output better images if the trained embedding is used in the negative prompt. So there’s a chance that making a lot of purposefully bad data could actually make models better by helping the model recognize bad output and avoid it.
VAE?
A Variational AutoEncoder is a kind of AI that can be used to compress data. In image generators, a VAE is used to compress the images. The actual image AI works on the smaller, compressed image (the latent representation), which means it takes a less powerful computer (and uses less energy). It’s that which makes it possible to run Stable Diffusion at home.
This attack targets the VAE. The image is altered so that the latent representation looks like a very different image, but still roughly the same to humans. The actual image AI works on a different image. Obviously, this only works if you have the right VAE. So, it only works against open source AI; basically only Stable Diffusion at this point. Companies that use a closed source VAE cannot be attacked.
I guess it makes sense if your ideology is that information must be owned and everything should make money for someone. I guess some people see cyberpunk dystopia as a desirable future. It doesn’t seem to be a very effective attack but it may have some long-term PR effect. Training an AI costs a fair amount of money. People who give that away for free probably still have some ulterior motive, such as being liked. If instead you get the full hate of a few anarcho-capitalists that threaten digital vandalism, you may be deterred. Well, my two cents.
Thank you for explaining. I work in NLP and are not familiar with all CV acronyms. That sounds like it kind if defeats the purpose if it only targets open source models. But yeah, makes sense that you would need the actual autoencoder in order to learn how to alter your data such that the representation from the autoencoder is different enough.
This would be truly ironic
If users have verry much control and we can coordinate then you could gaslight the AI into a screwed up alternate reality
Until they come with some preprocessing step, or some better feature extractors etc. This is an arms race like there are many of
The thing is data poisoning is a arms race that the Ai side will win with ease. You can either solve it with pre processing or filtering. All it does is make the images look worse. I can’t think of a way that you can poison data that doesn’t take more effort to unpoison than to poison.