2026-11-20
- Created the v2 poetry RLHF demo. The notebook and command-line trainer share a GRPO configuration with KL beta 0.04 against the frozen base policy.
- Added live KL plotting and connected notebook evaluation to the saved live-training adapter. v2 uses its own checkpoint paths and reads the existing repository dataset.
- Verified in newGPU with a full 12-step notebook run, including both evaluation sections, and a five-step command-line run. Both logged nonzero KL after training updates.
- Updated the v2 notebook to train fresh adapters at beta 0.05 and 0.15 with matching settings. It overlays KL and reward curves and compares both adapters with the base model and original poet on shared held-out prompts.
- Ran the updated notebook end to end in newGPU. Verified 12 steps per beta, distinct saved adapter weights, and both interactive and six-poem comparisons.
- Adapted the v2 notebook to the 1B LoRA recipe from ../rlhf-happy. It uses four-line chat prompts, CardiffNLP signed sentiment, Dr. GRPO, matched training and evaluation decoding, and 30 live updates per beta. Evaluation shows full replies and reports four-line compliance. New notebook adapters are saved under v2/checkpoints/chat.
- Verified the updated notebook end to end in newGPU. On six held-out poems, the base model and beta 0.05 produced six four-line replies each, and beta 0.15 produced five. Mean sentiment was about 0.32 in all three conditions.
