Cyber-security implications of highly accessible generative language models

[chiffa]

Looks like the problem with submission of the output of GPT 3.5 as an answer to questions on Stackoverflow is so bad they actually banned it: 

While this is harmless, given that the modern large language models are just autocomplete on a lot of steroids, it is possible to inject into their recommendations autocomplete suggestions that contain known vulnerabilities. All is needed for it are a set of repositories/sources from which they aspirate training data contain code that would contain likely prompts followed by the poisoned suggestion.

 

Another great, although slightly more involved application for them is spam and phishing message optimization. Given that they can be fine-tuned through reinforcement (aka a success/fail outcome), someone can set a spam filter and let the model run against it until it's able to bypass it. Or if they are running a large-scale phishing operation pull the click-through rate as the reinforcement signal, especially when attached to the recipient context information (age, account name, mail provider, ...). 

 

What other cyber-security implications the new generative models are out there? Or at least who is working on that topic?

 

[j91321]

If there is going to be some cybersecurity related implication I think it's going to be more in the scam related parts of the cybersecurity rather. I can see this allowing places like call-centre phone scams run more efficiently or to do automation on BEC scams without the need of operator input.

On 12/5/2022 at 8:38 PM, chiffa said:

Or if they are running a large-scale phishing operation pull the click-through rate as the reinforcement signal

I think if you do this your model might learn to make grammar mistakes 😄 I've heard this mentioned by law enforcement where they claimed that scam emails with grammar errors have a better chance at successfully scamming the victim because it acts as a selection filter, if you are not able to spot grammar errors, it's more probable you'll send money to somebody using gift coupons. I'm sure people running these scams will find a creative way of using it if there is an incentive for them to do so.

What I'd be more worried about is bad actors using this to run influence operations and echo chambers on social media. We can already see that troll farms are capable of creating genuine problems for societies. This could allow them to scale much better. An bunch of bot-idiots confidently yelling in the comments on Facebook, automated Telegram channel maintainers that just push radicalising content with occasional manual intervention from operators to create specific influences.  ChatGPT can already write content indistinguishable from posts on LinkedIn.

[munin]

Generated code is likely to have idiosyncracies that may not be noticed by the developer - because they're using it to generate large quantities of boilerplate so they may be too overwhelmed to notice, or they genuinely are not sure of what to do - which can lead to......basically everything OWASP cares about.

It'll also make phishing filters marginally less effective.

 

[Kris]

Really interesting question. It's a tool which opens up scale and improved efficiency to smaller groups or individuals. Coupled with AI focused on a different area, e.g., choosing people to target, or running disinformation campaigns, it's clearly of interest to anyone interested in influence. I think you've hit the nail on the head with the phishing optimisation for the most obvious benefit. Likely reduces the barrier for language too, as models in different languages become available an actor could target well written campaigns to multiple languages to a high standard.

Regarding who's working on it, it's an active area of research. Search 'text generation', 'natural language processing'... etc, into https://arxiv.org, https://scholar.google.com.

1. 'The Next Generation of Cyber-Enabled Information Warfare' - https://doi.org/10.23919/CyCon49761.2020.9131716  - open access PDF: https://ccdcoe.org/uploads/2020/05/CyCon_2020_13_Hartmann_Giles.pdf

2. 'SCIgen - An Automatic CS Paper Generator' - https://pdos.csail.mit.edu/archive/scigen/ 

There's a US Army think tank paper somewhere, where they horizon scan X-years into the future and image a world with various advancements and their impact. I can't find it, but I believe it mentions text generation. I'll see if I can find it.

[chiffa]

On 12/6/2022 at 9:38 PM, j91321 said:

run influence operations and echo chambers on social media

Well, figuring countermeasures to that is actually my team's job. For reasons completely unrelated to ChatGPT release, we are now looking for a junior NLP ML dev to add to our team, so if you know someone interested...

 

On 12/7/2022 at 7:01 PM, Kris said:

1. 'The Next Generation of Cyber-Enabled Information Warfare' - https://doi.org/10.23919/CyCon49761.2020.9131716  - open access PDF: https://ccdcoe.org/uploads/2020/05/CyCon_2020_13_Hartmann_Giles.pdf

Those guys are nice, but they are military before hackers, so some of the things they anticipate are more of "take something that existed before and sprinkle ML magic". In particular GANs they cover in this report do not pose that much of a problem with image deepfakes, in part large because FX and photoshop were already a thing in 1990s and everyone is pretty much vaccinated to them. In Ukrainian conflict we saw some deepfake usage from Russia, but it was ineffective (in part because they were about as competent about using them as about their logistics) and overwhelmingly underrepresented compared to existing image re-use and in-game footage use (3D rendering).

Smart chatbots however are a whole different cup of coffee - one deployed earlier this year on 4-chan managed to fool even those paranoids for about 48h hours despite 10k messages/hour posting (GPT-4chan). There is no way normies would survive a first contact with LLMs that are posting much more reasonably, but I have strong suspicion that there will be way more ways this will be happening than thinktanks are foreseeing.

 

On 12/7/2022 at 7:01 PM, Kris said:

There's a US Army think tank paper somewhere

Would be interested if you have even the thinktank's name - we are doing the same thing here in Switzerland, but given that thinktanks papers are neither research nor traditional media, finding them with a search engine is pretty much impossible if you don't know the title.

[Kris]

On 12/9/2022 at 7:09 PM, chiffa said:

Would be interested if you have even the thinktank's name - we are doing the same thing here in Switzerland, but given that thinktanks papers are neither research nor traditional media, finding them with a search engine is pretty much impossible if you don't know the title.

Found it! 'The New Dogs of War: The Future of Weaponized Artificial Intelligence' - https://apps.dtic.mil/sti/citations/AD1040008 - direct to PDF: https://apps.dtic.mil/sti/pdfs/AD1040008.pdf

Created by the (US) Army Institute at West Point and Arion University Threatcasting Lab.