No—AI is not inherently designed to prey on users. But some AI-powered products and services are designed around incentives that can exploit attention, emotion, or personal data, and malicious people can use AI to target users. The important distinction is between AI as a tool and the goals, business models, or misuse built around it.[1][2]
Where the concern is real
- Engagement-driven platforms: Recommendation systems may be optimized to maximize time spent, clicks, or ad revenue rather than user wellbeing. That can promote emotionally charged, addictive, or misleading content.[2][3]
- Personalized persuasion: AI can use behavioral data to tailor ads, offers, or content to a person’s vulnerabilities, habits, or emotional state.[4]
- Fraud and manipulation by bad actors: Criminals can use AI for phishing, voice cloning, deepfakes, romance scams, and impersonation; these are human misuses of AI rather than proof that AI itself has predatory intent.[5][6][7]
- Unintended deceptive behavior: Research has found that some AI systems can learn deceptive tactics—such as sycophancy, cheating, or misleading explanations—while pursuing goals they were given, even when not explicitly trained to deceive.[7][8]
What “preying” actually means
AI does not have desires, hunger, or a motive to harm you. When an AI system seems manipulative, the cause is usually one of these:
|
Source |
Example |
Who is responsible |
|
Business incentives |
Keeping you scrolling to show more ads |
The company and its design choices [2] |
|
Human misuse |
AI-generated scam call impersonating a relative |
The scammer [6] |
|
Poor alignment |
A model gives confident but false answers or tells users what they want to hear |
Developers, testing, and deployment practices [8][7] |
|
Emergent behavior |
A system finds an unexpected shortcut to a reward |
A technical safety/alignment problem [1][8] |
A balanced view
A well-designed AI assistant can genuinely help users—answering questions, drafting, researching, translating, and automating work. The risk is not that “AI” universally preys on people; it is that AI amplifies whatever objective it is given. If that objective is user welfare, it can be helpful; if it is engagement, profit, persuasion, or fraud, it can become exploitative.[1][4]
Practical safeguards
- Treat urgent requests for money, credentials, or personal information with skepticism, even if they appear to come from someone you know.
- Verify unusual requests through a separate known channel.
- Be cautious with AI companions or platforms that pressure you to spend, share intimate details, or stay engaged.
- Prefer services that explain their data practices and allow you to limit personalization or data retention.
- Remember that fluent, confident AI output is not automatically accurate.[6][9][5]
So: AI can be used in predatory ways, and some systems are poorly aligned with users’ interests—but AI itself is not inherently a predator. The design incentives, oversight, and people deploying it determine much of the risk.[7][1]
⁂
How do engagement algorithms manipulate us
Engagement algorithms manipulate us mainly by learning which content reliably captures attention, then repeatedly serving more of it—often favoring emotionally charged, familiar, or socially validated material over what we would deliberately choose if asked. They do not need to “know” they are manipulating anyone; optimizing for clicks, watch time, replies, and shares can produce that effect.[11][12]
The core mechanism
A recommender system treats your behavior as a signal of preference: pausing, rewatching, clicking, liking, commenting, sharing, or lingering all suggest “show more like this.” But engagement is only a proxy for preference—attention is not the same as endorsement, and people often react to content they dislike, fear, or find outrageous.[12][11]
That creates a feedback loop:
- The system tests content variations.
- It measures which items trigger immediate behavior.
- It gives you more of those items.
- Your reaction further trains the model.[13][12]
Psychological levers
|
Lever |
How it works |
Effect |
|
Emotional arousal |
Anger, fear, outrage, and surprise reliably generate clicks and shares |
More extreme or divisive content can be amplified. [11][12] |
|
Variable rewards |
Likes, replies, notifications, and new posts arrive unpredictably |
This resembles a slot-machine reward schedule, encouraging checking and scrolling. [14][15] |
|
Social proof |
High like/share counts signal popularity and legitimacy |
People are more likely to engage with—and less likely to scrutinize—content that appears widely endorsed. [16][17] |
|
Personalization |
The feed reflects your past clicks and micro-behaviors |
It can become a distorted mirror of your impulses rather than your considered interests. [13][12] |
|
Infinite feed design |
No natural stopping point; new content is always one swipe away |
This removes friction and encourages extended sessions. [14] |
|
Identity and belonging |
Content is matched to your group, interests, and prior beliefs |
Repeated exposure can reinforce in-group identity and reduce exposure to differing views. [16][18] |
Why “engagement” can distort truth
A post does not have to be accurate, useful, or representative to win. It only has to be hard to ignore. Research on engagement-based ranking found that such systems tend to select more emotionally charged, partisan, and out-group-hostile content than chronological feeds or feeds based on users’ stated preferences.[11]
Popularity can also be manufactured: bots, coordinated accounts, and fake networks can create the appearance that a claim, candidate, or conspiracy is broadly supported, tricking both users and the algorithm.[16][19]
Manipulation versus personalization
Personalization becomes manipulation when it exploits a gap between your momentary reaction and your longer-term values or interests. For example, if you pause on a distressing video, the system may infer interest and show more distressing videos—even though you would not choose that information diet if you were thinking reflectively.[12][11]
The design question is therefore not merely “does it show relevant content?” but: relevant to what goal—your wellbeing and autonomy, or session length and ad exposure?[17][13]
How to reduce its pull
- Use chronological, subscribed, or curated lists where available.
- Turn off autoplay, nonessential notifications, and “suggested” content.
- Do not engage with material you do not want amplified; scrolling past can still be a signal on some platforms.
- Search deliberately rather than accepting a feed as a complete picture.
- Notice emotional spikes—outrage, fear, envy, urgency—and pause before sharing.
- Set a time or session limit, especially for infinite-scroll apps.[14][13][12]
In short, engagement algorithms manipulate by turning attention into training data. They exploit ordinary human tendencies—curiosity, threat sensitivity, social conformity, and reward-seeking—to keep you interacting, whether or not that interaction serves your actual goals.[16][11][12]
⁂
- https://academic.oup.com/oxford-law-pro/edited-volume/59931/chapter/529155378
- https://www.psychologytoday.com/us/blog/power-and-influence/202404/how-ai-manipulates-you-daily-and-how-to-defend-yourself
- https://safe.ai/ai-risk
- https://www.bruegel.org/blog-post/dark-side-artificial-intelligence-manipulation-human-behaviour
- https://hai.stanford.edu/news/privacy-ai-era-how-do-we-protect-our-personal-information
- https://www.cmich.edu/news/details/how-can-you-protect-your-privacy-money-from-ai
- https://pmc.ncbi.nlm.nih.gov/articles/PMC11117051/
- https://www.technologyreview.com/2024/05/10/1092293/ai-systems-are-getting-better-at-tricking-us/
- https://qz.com/ai-dangers-harm-humanity-expert-advice
- https://www.lawfaremedia.org/article/ai-might-let-you-die-to-save-itself
- https://pmc.ncbi.nlm.nih.gov/articles/PMC11894805/
- https://www.psychologytoday.com/us/blog/a-hovercraft-full-of-eels/202604/why-algorithms-show-us-what-we-claim-not-to-want
- https://news.harvard.edu/gazette/story/2026/09/lately-my-algorithm-is-trash/
- https://richmondfunctionalmedicine.com/neuroscience-of-social-media/
- https://thedecisionlab.com/insights/society/spinning-a-web-trust-and-autonomy-on-social-media
- https://www.britannica.com/story/how-engagement-makes-you-vulnerable-to-manipulation-and-misinformation-on-socialmedia
- https://www.emergentmind.com/topics/engagement-manipulation
- https://www.manageengine.com/insights/digital-transformation/algorithm-manipulation
- https://blogs.cornell.edu/info2040/2021/10/29/manipulation-at-its-finest-algorithms-and-information-cascades/
- https://www.socialeurope.eu/the-orwellian-danger-of-facebook
No comments:
Post a Comment