I was wondering if there are any other training curves for the skyrl-v0-8b model, such as the typical value range for the KL loss, or the evaluation curves on the eval dataset?
Here is my results：model.py:660 - step 1: {'train/loss': 0.0006256469641812146, 'train/pg_loss': 0.0, 'train/entropy_loss': 0.07461290061473846, 'train/pg_clipfrac': 0.0, 'train/ppo_kl': -0.20245233178138733, 'train/kl_loss': 0.6256468892097473, 'train/grad_norm': 0.03765935603564608, 'train/lr-pg_0': 1e-06, 'train/lr-pg_1': 1e-06, 'train/step': 1}
kl_loss seems a bit abnorma。
I was wondering if there are any other training curves for the skyrl-v0-8b model, such as the typical value range for the KL loss, or the evaluation curves on the eval dataset?
Here is my results：model.py:660 - step 1: {'train/loss': 0.0006256469641812146, 'train/pg_loss': 0.0, 'train/entropy_loss': 0.07461290061473846, 'train/pg_clipfrac': 0.0, 'train/ppo_kl': -0.20245233178138733, 'train/kl_loss': 0.6256468892097473, 'train/grad_norm': 0.03765935603564608, 'train/lr-pg_0': 1e-06, 'train/lr-pg_1': 1e-06, 'train/step': 1}
kl_loss seems a bit abnorma。