👋 Welcome to Jian Hu’s Blog
Email: [email protected]
Homepage: https://hujian.website/
LinkedIn: https://www.linkedin.com/in/jian-hu-060979238
Google Scholar: https://scholar.google.com/citations?user=-xt5vGkAAAAJ
GitHub: https://github.com/hijkzzz
Zhihu: https://www.zhihu.com/people/chu-qi-6-41/posts
I’m a RLer + NLPer / 2 + MLSyser / 2
From FlashREINFORCE to Score Centering: What Importance Sampling Already Corrects
TL;DR: Viewed through Score Centering, FlashREINFORCE’s unclipped token-level IS core already satisfies SC’s zero-mean condition—and goes further by restoring target-policy action weights at each prefix, making an additional exact SC correction redundant.
Reward Hacking in Claude Code RL Training
TL;DR: Over two years of Claude model cards, reward hacking escalated from simple test-hardcoding to Mythos autonomously bypassing network restrictions and peeking at held-out test sets.
Stabilizing MoE RL Without Router Replay: The Online IcePop/Seq-level Mask TIS Solution
TL;DR: Solutions for stabilizing MoE RL without Router Replay: (1) Online IcePop = pure online policy updates + IcePop token-level masking of extreme IS weights; (2) Online Seq-level Mask TIS = seq-level (geometric-mean) log-ratio filtering + untruncated IS.
A Brief Analysis of KL Approximation Methods (k1, k2, k3) in RLHF/RLVR
TL;DR: We analyze k1/k2/k3 KL estimators and argue k2 best matches the reverse-KL gradient (k1 fails as a loss; k3 behaves like forward-KL and can be high-variance).