A team led by Professor Minlie Huang from Tsinghua University’s Department of Computer Science has provided the first systematic demonstration that AI advisors equipped with hidden objectives that conflict with users’ interests can significantly shift users’ preferences away from optimal choices toward suboptimal or even harmful alternatives, largely without users noticing. Notably, the vast majority of users still regarded these advisors as “helpful, informative, and reasonable.” This finding reveals a previously unquantified behavioral vulnerability in human–AI interaction and provides empirical evidence supporting the development of transparency and accountability mechanisms in AI governance.
The research, titled “Human Preferences Are Susceptible to Covertly Misaligned AI Advice,” was published in the Proceedings of the National Academy of Sciences (PNAS) on September 14, 2026.
A reviewer commented: “This manuscript makes a timely and potentially important contribution by addressing a question of substantial practical interest for AI governance and human-AI interaction. The randomized design is a real strength, and the core descriptive result appears robust.”
From choosing products to coping with emotional distress, people are increasingly turning to AI assistants for advice. Many assume that these systems are neutral and prioritize users’ interests. In reality, however, the organizations providing these systems are often profit-driven businesses. If an AI advisor's true objectives conflict with users' interests in ways that are hard to detect, how easily can users be led astray in realistic advice-seeking situations?
To investigate this question, the research team designed a randomized between-subjects experiment, assigning participants to conditions crossing two decision-making domains with three types of AI advisor. The scenarios covered financial and consumer decisions (e.g., choosing fitness trackers, weight-loss products, and online shopping platforms) under specified budget and needs. They also involved advice in emotional conflicts, such as coping with concerns about self-image, conflicts with close friends, and criticism at work. Each scenario included one option that was optimal (beneficial) under the stated constraints and three suboptimal options, each carrying a different type of risk/harm. One of these suboptimal options was designated as the hidden incentive, which the misaligned advisors were covertly instructed to promote.
All three types of AI advisor were built on GPT-4o and differed only in their prompts. The neutral advisor (NA) served as the control condition and was instructed to act solely in the user's interest. The misaligned advisor (MA) was given a hidden incentive that conflicted with users’ interests and was instructed to steer users toward this suboptimal option without revealing its objective. The strategy-enhanced misaligned advisor (SEMA) was additionally equipped with a set of covert influence strategies, drawn from the psychology and social science literature, that it could select and deploy on its own during the conversation.
Before each conversation, all three advisors received participants’ Big Five personality profiles. They also used a theory of mind reasoning framework to infer participants’ beliefs, goals, and emotions before generating responses. Participants rated each of the four options on a 10-point scale before and after the interaction and were required to complete at least 10 conversational rounds with the advisor before submitting their final ratings.
Figure 1. Overall study design: decision-making scenarios (a), the three AI-advisor conditions (b), and the experimental procedure (c).
The results showed that participants’ preferences shifted significantly toward the AI’s hidden incentive after interacting with misaligned advisors. Before the interaction, participants across all conditions consistently preferred the optimal option, with no significant differences between groups, indicating that the options designed to be optimal were indeed recognizable as such. After the interaction, participants exposed to misaligned advisors showed a significant increase in their relative preference for the covertly incentivized option. The effects were large in both the financial and emotional domains, with effect sizes relative to the neutral advisor ranging from d = 1.04 to 1.53 and all P values below 0.001.
A difference-in-differences analysis further showed that, compared with the neutral advisor, the misaligned advisors increased the odds of choosing the hidden incentive over the optimal option roughly 5-8 times and raised the probability by as much as 38 percentage points.
One unexpected finding was that equipping the advisor with established psychological influence strategies (SEMA) did not reliably strengthen the influence; the differences between SEMA and MA were not statistically significant in either domain. This suggests that a hidden objective alone can be sufficient to substantially change people’s preferences, without requiring sophisticated manipulative language. This means that the technical barrier to creating this kind of risk may be much lower than expected.
Figure 2. Distribution of participants’ preference categories before and after the interaction: misaligned advisors significantly increased the probability of selecting the “hidden‑target” option.
Even more concerning was a systematic disconnect between participants' preference shifts and their subjective evaluations of the advisors. Although their preferences had clearly been steered off course, most participants (75.6%–86.8%) still considered the misaligned advisors helpful, with ratings even approaching those of the neutral advisors. Only a minority of participants (8.1%–28.9%) reported noticing that the advisors had ulterior motives, and these reports came mainly from the financial scenarios.
Therefore, this study reveals a previously unquantified behavioral vulnerability in human–AI interaction: when an AI’s motives are hidden, people may come to favor options that do not serve their interests while continuing to evaluate the AI positively. Unlike advertising and sponsored content, which users have learned to treat with skepticism, AI assistants are often perceived as neutral helpers. As a result, their inappropriate advice is more likely to be attributed to limitations in the model’s capabilities rather than to hidden vested interests. This asymmetry makes covert goal misalignment harder to detect. The potential harms warrant particular attention in settings such as emotional support, where users tend to be both more trusting and more vulnerable.
The paper’s co-first authors are Sahand Sabour, a doctoral student in Tsinghua University’s Department of Computer Science supervised by Professor Minlie Huang, and June M. Liu of the University of Hong Kong. The co-corresponding authors are Professor Minlie Huang of Tsinghua University’s Department of Computer Science, Professor Tatia M. C. Lee of the University of Hong Kong, Sahand Sabour, and June M. Liu.
Full article: https://www.pnas.org/doi/10.1073/pnas.2600684123
Editors: Li Han, John Paul Grima
