Yuetai Li
PhD Student
University of Washington
About Me
I am a second-year PhD student at Network Security Lab at University of Washington, advised by Prof. Radha Poovendran. I am fortunate to work closely with Dr. Xiang Yue at CMU LTI, Zhangchen Xu, Prof. Luyao Niu, and Prof. Bill Yuchen Lin at UW.
Prior to UW, I obtained my Bachelor degree in Communication Engineering at University of Glasgow (UofG) and the University of Electronic Science and Technology of China (UESTC). During my undergrad, I was advised by Prof. Lei Zhang at UofG. I am also fortunate to work with Prof. Jon Crowcroft at University of Cambridge.
Interned at Microsoft Research Redmond (Agent Env Simulation) in Summer 2025.
Start Internship at Meta MSL in Menlo Park (Agent Eval) from March 2026.
I am open to collaboration and discuss interesting ideas! Email yuetaili@uw.edu if you would like to share any opportunities, collaborations, or just chat～
Research Interests
My research focuses on dataset/benchmark curation for both Agent-Native and Human-Centric needs: What are the limits of agentic automation? What skills remain uniquely human? What tasks do humans genuinely want agents to do?
Aligning Agent Work with Human Will
Current benchmarks for occupational AI agents are scoped primarily by economic values, telling a replacement story. We introduce JobBench, which evaluates AI agents on the workflows that experts identify as high-priority for delegation, empowering humans based on their needs instead of replacing them with GDP value. JobBench covers 130 agentic tasks across 35 occupations. Each task is packaged as a workspace of heterogeneous reference files, requiring the agent to reason through the cluttered information streams of real professional work. Outputs are graded by a fact-anchored chain of rubrics, averaging 35.6 binary criteria per task. We evaluate 36 models; the strongest, Claude Opus 4.7 under Claude Code, reaches only 45.9%. We hope JobBench shifts the community's target labour-market effect from replacement to enhancement: building agents that do what humans actually want delegated, not only what is most economically valuable.
The largest-scale, broadest-coverage agent evaluation benchmark to date
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 sub fields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is 2.6%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded.
We hired 1000+ artists to ask: Can frontier model judge beauty?
We conduct a controlled human study to investigate whether aesthetic scores faithfully capture comparative preference. Our results show that score-derived rankings align poorly with direct comparative rankings, while direct ranking yields substantially higher agreement across annotators. We introduce VAB, which formulates aesthetic evaluation as comparative selection over candidate sets with matched subject matter. VAB comprises 400 tasks and 1,195 images across fine art, photography, and illustration, with labels derived from the consensus of 10 independent expert judges. Evaluating 20 frontier MLLMs and 6 dedicated visual quality reward models, the best model achieves only 26.5% accuracy vs. 68.9% by human experts. We further fine-tune a 35B model and show it approaches the performance of a 397B model, highlighting the transfer value of VAB's expert comparative supervision.
The convergence of LLM-powered research assistants and AI-based peer review systems creates a critical vulnerability: fully automated publication loops where AI-generated research is evaluated by AI reviewers without human oversight. We investigate this through BadScientist, a framework that evaluates whether fabrication-oriented paper generation agents can deceive multi-model LLM review systems.
Agents4Science 2025
🏆 Best Paper Award
We revealed that small models do not consistently benefit from long CoT or distillation from large teachers. Instead, they perform better on shorter, simpler reasoning chains that better align with their intrinsic learning capacity. We term this phenomenon as Small Model Learnability Gap.
We find that models RL on math tasks generalize well across non-reasoning domains such as alignment, while SFT models lose such capacity. Latent-space representation and token-space distribution shift analyses reveal that SFT induces substantial representation and output drift, while RL preserves general-domain structure. We finally show that sampling policy is key to generalization: off-policy RL on reasoning tasks compromise non-reasoning performance while on-policy SFT generalizes well.
We demonstrate that LLMs can simulate realistic environment feedback without access to actual testbed data or APIs. Inspired by this, we propose two frameworks: Simia-SFT, a pipeline that synthesizes SFT data by amplifying small seed sets into diverse trajectories in an environment-agnostic manner, and Simia-RL, a framework that enables RL training without real environment implementations through LLM-simulated feedback.
We investigated how long CoT impacts safety and revealed that long CoT does not necessarily enhance safety. We introduced SafeChain, a dataset designed to improve safety alignment while preserving reasoning capabilities.
ICLR BiAlign Workshop (Oral)
🏆 Best Honorable Mention
We observed that during RL training process of Deepseek-R1-1.5B model, 76.7% of AIME problems were solved correctly at some intermediate checkpoint, yet only 30% remained correct in the final model. This indicates that many problems answered correctly during training were ultimately incorrect in the final checkpoint. We term this phenomenon as Temporal Forgetting. Inspired by this, we proposed Temporal Sampling: This method utilizes training dynamics as a source of answer diversity by distributing inference samples across multiple distinct checkpoints from the training trajectory, rather than relying solely on the single final checkpoint.
Our study reveals that over 38% of model responses suffer from false negatives in answer verification for RL training of LLMs, severely impairing training efficiency. We propose TinyV, a lightweight LLM-based verifier that augments existing rule-based methods to provide more accurate reward estimates.
Four-stage pipeline for generating VisualSphinx of 660K visual logic data for RL training on multimodal reasoning models. In Step 1, we collect 4K seed puzzles with explanations and abstract them into structured rule descriptions using LLMs. In Step 2, we apply a rule-level genetic algorithm to cross over, mutate and diversify the seed rules, scaling them to 40K high-quality rules. In Step 3, each rule is paired with a rendering style and used to generate five correct and three incorrect images via LLM-generated Python scripts. The fifth correct image is designated as the answer option, while the three rule-breaking images serve as distractors. After deduplication, we obtain 110K image groups. In Step 4, we assemble puzzles from each group using three complementary strategies: default assembly, shuffled answer variants, and expanded distractor sets.
Trustworthy AI
A novel decoding algorithm that defends against various generative backdoor attacks, including advertisement injection, code injection, and malicious content generation.
We introduce SOSBench, a regulation-grounded, hazard-focused benchmark encompassing six high-risk scientific domains: chemistry, biology, medicine, pharmacology, physics, and psychology. The benchmark comprises 3,000 prompts derived from real-world regulations and laws, systematically expanded via an LLM-assisted evolutionary pipeline that introduces diverse, realistic misuse scenarios (e.g., detailed explosive synthesis instructions involving advanced chemical formulas).
Distributed Algorithms
Work includes Voting Validity,
Wireless Distributed Consensus,
Resource Allocation, and
Distributed Consensus Network.