2026-11-20
- Fixed the poem viewer and sampler dropdowns in notebooks/rlhf_poem_demo.ipynb. Both limit the prompt to leave four held-out lines. The notebook resolves repository paths from its working directory and launches training with the active kernel's Python.
- Updated training/train_grpo.py for the installed TRL API and restricted single-process training to the first visible CUDA device. It streams reward and KL metrics to the notebook and saves the final trainer state with each model. Added optional step and checkpoint path overrides for smoke runs.
- Reused the training prompt and reward code for notebook inference. Fixed the generation attention mask and made the reward in training/reward.py decrease as the classifier's confidence in negative sentiment increases.
- Added notebook cell IDs for schema validation and switched the viewer, scorer, and sampler to direct HTML widget updates. Escaped poem and completion text and made the evaluation chart handle smaller PoetryDB samples and negative rewards.
- Verified the notebook in newGPU with one GPU training step per beta on a small PoetryDB sample. Both live plots updated, both checkpoints reloaded, the sampler produced three scored completions, and the 20-poem evaluation produced 60 scores.
- Set the demo to 100 training steps per beta run, for 200 total.
- Moved live reward and KL curves into standard PNG output in each running training cell. This removes the background widget update path while keeping the trainer log and the first-step and five-step graph updates.
- Verified both training cells in the newGPU Jupyter kernel with simulated trainer subprocesses that waited for the notebook to receive the PNG at steps 1 and 5.
