Assessing the risks and opportunities posed by AI-enhanced influence operations on social media
(2024) In Place Branding and Public Diplomacy- Abstract
Large language models (LLMs) like GPT-4 have the potential to dramatically change the landscape of influence operations. They can generate persuasive, tailored content at scale, making campaigns using falsified content, such as disinformation and fake accounts, easier to produce. Advances in self-hosted open-source models have meant that adversaries can evade content moderation and security checks built into large commercial models such as those commercialised by Anthropic, Google, and OpenAI. New multi-lingual models make it easier than ever for foreign adversaries to pose as local actors. This article examines the heightened threats posed by synthetic media, as well as the potential that these tools hold for creating effective... (More)
Large language models (LLMs) like GPT-4 have the potential to dramatically change the landscape of influence operations. They can generate persuasive, tailored content at scale, making campaigns using falsified content, such as disinformation and fake accounts, easier to produce. Advances in self-hosted open-source models have meant that adversaries can evade content moderation and security checks built into large commercial models such as those commercialised by Anthropic, Google, and OpenAI. New multi-lingual models make it easier than ever for foreign adversaries to pose as local actors. This article examines the heightened threats posed by synthetic media, as well as the potential that these tools hold for creating effective countermeasures. It begins with assessing the challenges posed by a toxic combination of automated bots, human-controlled troll accounts, and more targeted social engineering operations. However, the second part of the article assesses the potential for these same tools to improve detection. Promising countermeasures include running internal generative models to bolster training data for internal classifiers, detecting statistical anomalies, identifying output from common prompts, and building specialised classifiers optimised for specific monitoring needs.
(Less)
- author
- Fredheim, Rolf and Pamment, James LU
- organization
- publishing date
- 2024
- type
- Contribution to journal
- publication status
- epub
- subject
- keywords
- AI, Digital diplomacy, Disinformation, Influence operations, LLM
- in
- Place Branding and Public Diplomacy
- publisher
- Palgrave Macmillan
- external identifiers
-
- scopus:85184437675
- ISSN
- 1751-8040
- DOI
- 10.1057/s41254-023-00322-5
- language
- English
- LU publication?
- yes
- id
- 24add63a-3da3-4c72-af0a-809f2dc255e2
- date added to LUP
- 2024-03-11 12:47:45
- date last changed
- 2024-03-11 12:49:02
@article{24add63a-3da3-4c72-af0a-809f2dc255e2,
abstract = {{<p>Large language models (LLMs) like GPT-4 have the potential to dramatically change the landscape of influence operations. They can generate persuasive, tailored content at scale, making campaigns using falsified content, such as disinformation and fake accounts, easier to produce. Advances in self-hosted open-source models have meant that adversaries can evade content moderation and security checks built into large commercial models such as those commercialised by Anthropic, Google, and OpenAI. New multi-lingual models make it easier than ever for foreign adversaries to pose as local actors. This article examines the heightened threats posed by synthetic media, as well as the potential that these tools hold for creating effective countermeasures. It begins with assessing the challenges posed by a toxic combination of automated bots, human-controlled troll accounts, and more targeted social engineering operations. However, the second part of the article assesses the potential for these same tools to improve detection. Promising countermeasures include running internal generative models to bolster training data for internal classifiers, detecting statistical anomalies, identifying output from common prompts, and building specialised classifiers optimised for specific monitoring needs.</p>}},
author = {{Fredheim, Rolf and Pamment, James}},
issn = {{1751-8040}},
keywords = {{AI; Digital diplomacy; Disinformation; Influence operations; LLM}},
language = {{eng}},
publisher = {{Palgrave Macmillan}},
series = {{Place Branding and Public Diplomacy}},
title = {{Assessing the risks and opportunities posed by AI-enhanced influence operations on social media}},
url = {{http://dx.doi.org/10.1057/s41254-023-00322-5}},
doi = {{10.1057/s41254-023-00322-5}},
year = {{2024}},
}