Yihao Zhang张益豪
Third-Year Applied Mathematics PhD
School of Mathematical Sciences, Peking University
Currently @ Tencent Yuanbao Security Center
I am Yihao Zhang (张益豪), a Third-Year Applied Mathematics PhD at the School of Mathematical Sciences, Peking University. I am currently conducting research under the guidance of Professor Meng Sun and am a part of his research group.
I got my bachelor’s degree (Data Science, Math) from Peking University in 2024. I served as a visiting research assistant at Singapore Management University, from October 2023 to May 2024. My supervisor at Singapore Management University is Jun Sun.
My research interests include:
- LLM Agent Safety
- Formal Methods for LLM Agents
- Jailbreaking LLM
- Prompt Injection
- Mechanistic Interpretability for Large Language Models
- Representation Engineering in LLM
- AI Safety, Verifying Robustness/Fairness/Trustworthy Related Issues in AI
- Automated Interactive Theorem Proving, AI4ITP
I have published multiple papers at different international conferences with total google scholar 
If you wish to access my personal homepage before 2024, please click here.
💼 Experience
Tencent Yuanbao Security CenterCurrent
腾讯元宝安全中心
ByteDance Flow Security
字节跳动 Flow 安全
🔥 News
📝 Selected Papers
(*: Equal Contribution; ${}^\dagger$: Corresponding Author)
AgentWorm: Self-Propagating Attacks Across LLM Agent Ecosystems (NDSS 2027)
Yihao Zhang, Zeming Wei, Xiaokun Luan, Jiangrong Wu, Chengcan Wu, Zhixin Zhang, Haolin Wu, Huanran Chen, Jun Sun, Meng Sun
We present AgentWorm, a self-propagating attack that compromises persistent LLM agents and autonomously spreads across agent ecosystems. The work studies the full infection loop, its transferability, and practical defenses for deployed agent systems.
On the Duality Between Sharpness-Aware Minimization and Adversarial Training (ICML 2024)
Yihao Zhang*, Hangzhou He*, Jingyu Zhu*, Huanran Chen, Yifei Wang, Zeming Wei${}^\dagger$
Adversarial Training (AT), which adversarially perturb the input samples during training, has been acknowledged as one of the most effective defenses against adversarial attacks, yet suffers from a fundamental tradeoff that inevitably decreases clean accuracy. Instead of perturbing the samples, Sharpness-Aware Minimization (SAM) perturbs the model weights during training to find a more flat loss landscape and improve generalization. However, as SAM is designed for better clean accuracy, its effectiveness in enhancing adversarial robustness remains unexplored. In this work, considering the duality between SAM and AT, we investigate the adversarial robustness derived from SAM. Intriguingly, we find that using SAM alone can improve adversarial robustness. To understand this unexpected property of SAM, we first provide empirical and theoretical insights into how SAM can implicitly learn more robust features, and conduct comprehensive experiments to show that SAM can improve adversarial robustness notably without sacrificing any clean accuracy, shedding light on the potential of SAM to be a substitute for AT when accuracy comes at a higher priority. Code is available at this https URL.
Adversarial Representation Engineering: A General Model Editing Framework for Large Language Models (NeurIPS 2024)
Yihao Zhang, Zeming Wei, Jun Sun${}^\dagger$, Meng Sun${}^\dagger$
Since the development of Large Language Models (LLMs) has achieved remarkable success, understanding and controlling their internal complex mechanisms has become an urgent problem. Recent research has attempted to interpret their behaviors through the lens of inner representation. However, developing practical and efficient methods for applying these representations for general and flexible model editing remains challenging. In this work, we explore how to use representation engineering methods to guide the editing of LLMs by deploying a representation sensor as an oracle. We first identify the importance of a robust and reliable sensor during editing, then propose an Adversarial Representation Engineering (ARE) framework to provide a unified and interpretable approach for conceptual model editing without compromising baseline performance. Experiments on multiple model editing paradigms demonstrate the effectiveness of ARE in various settings. Code and data are available at this https URL.
Other Publications
(*: Equal Contribution; ${}^\dagger$: Corresponding Author)
Symbolic-Neural Soft-Logic Reasoning: Towards Robust and Verifiable Thinking Chains via Cooperative Evolution (Preprint, 2026)
On Mutation Testing of In-Context Learning Systems (Journal of Systems Architecture, 2026)
Automata-Based Steering of Large Language Models for Diverse Structured Generation (ICFEM 2025 · Best Paper)
Robust and Efficient Watermarking of Large Language Models Using Error Correction Codes (PETS 2025)
When Thinking LLMs Lie: Unveiling the Strategic Deception in Representations of Reasoning Models (Preprint, 2025)
Mitigating Fine-tuning Risks in LLMs via Safety-Aware Probing Optimization (Preprint, 2025)
Towards Formal Modeling and Verification of the Stellar Consensus Protocol in Z3 (QRS Companion 2025)
MedTiny Code Generation for Enhancing RegLang Smart Contract Reliability (SAC 2025)
Component Composition in MedTiny: Multi-Level Constructs and Operational Semantics (SEKE 2025)
Exploring the robustness of in-context learning with noisy labels (ICASSP 2025, ICLR 2024 R2-FM Workshop)
DocVerify: Constraint-Based Verification of LLM Outputs Against Document Knowledge (ICECCS 2026)
MedTiny: Enhanced Mediator Modeling Language for Scalable Parallel Algorithms (QRS 2023)
Sharpness-aware minimization alone can improve adversarial robustness (ICML 2023 AdvML-Frontiers Workshop)
Weighted automata extraction and explanation of recurrent neural networks for natural language tasks (JLAMP, Vol 136)
🎖 Honors and Awards
- President’s Scholarship, Peking University, 2025
- Huaixin Bachelor (怀新学士, Honours Degree), 2024
- Selected for the Elite Program (拔尖计划, Graduate) in the School of Mathematical Sciences, Peking University.
- University Scholarship, Peking University, 2023
- Second prize, Chinese Mathematics Competitions for Undergraduates (Beijing Division), 2023
📖 Educations
- 2023.10 - 2024.05, Research Assistant, School of Computing and Information Systems, Singapore Management University.
- 2020.09 - 2024.06, Undergraduate Student, School of Mathematical Sciences, Peking University.
- 2024.09 - 2028.06 (expected), PhD, School of Mathematical Sciences, Peking University.
💬 Talks
- 2024.09, Presented our work, On the Duality Between Sharpness-Aware Minimization and Adversarial Training, in Yantai University.
- 2024.04, Presented our work on Adversarial Representation Engineering in an internal talk in Huawei.
- 2023.06, Presented my paper “Using Z3 for Formal Modeling and Verification of FNN Global Robustness” on SEKE 2023.
💻 Projects
- 2026-2029, Safety Assurance of LLM-Driven Agents, Key Member, NSFC General Program.
- 2025-2027, LLM Feature Reconstruction & Manipulation, Key Member, Industry Collaboration.
- 2023-2025, Study on the interpretability of large language model architecture and algorithm, Program Director.
- 2022-2025, Trustworthy guarantee of deep learning system, Member.
👨🏫 Advisors
🔗 Links
- 🧑🎓 Co-authors: Zeming Wei, Xiaokun Luan, Xiyue Zhang, Huanran Chen, Jiangrong Wu