Feature/find ppo error

[lwaekfjlk]

Closes #

📑 Description

✅ Checks

• My pull request adheres to the code style of this project
• My code requires changes to the documentation
• I have updated the documentation as required
• All the tests have passed
• Branch name follows type/descript (e.g. feature/add-llm-agents)
• Ready for code review

ℹ Additional Information

[lwaekfjlk]

WE USE AN OVER-TRAINED SFT MODEL!!!

[lwaekfjlk]

with a pretrained value model + a small value coef at the second stage

gamma=1 and base model as the reference model, ppo epoch =2 make most settings work.