# RLHF In-Class Demo Plan: Sentiment-Steered Poem Completion with GRPO

> [!IMPORTANT]
> **Goal:** Fine-tune Llama 3.2 (1B) with GRPO to complete poems positively, using a pretrained sentiment model as the reward signal — no reward model training required. Demo is instructor-led; students observe. Live training runs during class (~20 min). Demo delivered as a Jupyter notebook.

---

## 1. Overview

```mermaid
flowchart LR
    A["PoetryDB API\npoetrbydb.org"] --> B["Prompt builder\nlines 8–15 → neutral prompt"]
    B --> C["Base Llama 3.2 1B\n(reference policy)"]
    B --> D["GRPO Training Loop\nlive in class"]
    E["Sentiment model\n(DistilBERT SST-2)"] --> D
    D --> F["Tuned Policy\n(beta=0.04 & beta=0.4)"]
    C --> G["Notebook Demo\nSide-by-side comparison"]
    F --> G
```

**Core choices:**
| Component | Choice | Rationale |
|---|---|---|
| Dataset | PoetryDB REST API (`poetrydb.org`) | No auth required, JSON responses, filter by `linecount` |
| Base model | `meta-llama/Llama-3.2-1B-Instruct` | Small enough for in-class GPU, strong enough to generate coherent poetry |
| RL algorithm | GRPO via HuggingFace TRL `GRPOTrainer` | No separate reward model needed; stable, group-based advantage estimation |
| Reward signal | `distilbert-base-uncased-finetuned-sst-2-english` + line-count bonus | Fast inference, well-calibrated positive/negative scores |
| KL penalty β | Two runs: **β = 0.04** (aggressive) vs **β = 0.4** (conservative) | Shows the exploration-vs-fidelity tradeoff live |

---

## 2. Repository / File Structure

```
rlhf-s2/
├── data/
│   └── poem_loader.py          # PoetryDB API fetch & prompt construction
├── training/
│   ├── reward.py               # Sentiment + structure reward function
│   ├── train_grpo.py           # Script-based GRPO trainer (beta param CLI arg)
│   └── configs/
│       ├── grpo_beta_low.yaml
│       └── grpo_beta_high.yaml
├── notebooks/
│   └── rlhf_poem_demo.ipynb    # ← MAIN DEMO NOTEBOOK
├── checkpoints/                # saved model weights (gitignored)
└── requirements.txt
```

---

## 3. Data Pipeline

### 3a. Dataset Loading via PoetryDB API

PoetryDB (`https://poetrydb.org`) is a free, no-auth REST API returning JSON poem objects with `title`, `author`, `lines`, and `linecount`.

**Strategy:** Fetch all poems with `linecount` ≥ 12 (guarantees enough context lines). Use the `linecount` endpoint or bulk-fetch and filter locally.

```python
import requests

def fetch_poems(min_lines: int = 12) -> list[dict]:
    """Fetch poems from PoetryDB with at least min_lines lines."""
    # Bulk fetch a large random sample and filter
    url = "https://poetrydb.org/random/200"
    poems = requests.get(url).json()
    return [p for p in poems if int(p["linecount"]) >= min_lines]

# Or fetch all poems of a specific linecount range:
def fetch_poems_by_linecount(min_lc: int = 12, max_lc: int = 50) -> list[dict]:
    poems = []
    for lc in range(min_lc, max_lc + 1):
        resp = requests.get(f"https://poetrydb.org/linecount/{lc}")
        if resp.status_code == 200:
            poems.extend(resp.json())
    return poems
```

> [!NOTE]
> PoetryDB has ~3,000 poems in its database. Filtering to ≥ 12 lines typically yields ~800–1,000 usable poems — plenty for training and demo.

### 3b. Prompt Construction (`poem_loader.py`)

Each training example:
1. Split `poem["lines"]` (already a list of strings in PoetryDB JSON)
2. Sample 8–15 context lines randomly during training; fixed window during demo
3. Format as a **neutral** chat-style prompt — no sentiment instruction. The goal of RLHF is for the reward signal alone to steer behavior.

```python
import random

def build_prompt(poem: dict, n_context: int | None = None) -> list[dict]:
    lines = poem["lines"]
    if n_context is None:
        n_context = random.randint(8, min(15, len(lines) - 4))
    context = "\n".join(lines[:n_context])
    return [
        {
            "role": "system",
            "content": "You are a poet. Complete the following poem with exactly 4 lines."
            # ← NO sentiment instruction: steering comes from the reward signal only
        },
        {
            "role": "user",
            "content": f"{context}"
        }
    ]
```

> [!IMPORTANT]
> The system prompt must remain **sentiment-neutral**. Telling the model to "be positive" or "be uplifting" in the prompt defeats the entire point of RLHF — the whole demo would just be prompt engineering. The sentiment shift must be learned from the reward signal alone.

**Output format constraint:** The reward function penalizes completions that are not ≈ 4 lines (see Section 4).

---

## 4. Reward Function (`training/reward.py`)

The reward is a **weighted sum** of two components:

| Component | Weight | Signal |
|---|---|---|
| Sentiment score | 1.0 | DistilBERT SST-2 confidence in POSITIVE class |
| Line-count bonus | 0.2 | +1 if completion has exactly 3–5 newlines (≈ 4 lines), else 0 |

```python
from transformers import pipeline

_sentiment = pipeline(
    "sentiment-analysis",
    model="distilbert-base-uncased-finetuned-sst-2-english",
    device=0  # GPU
)

def _line_count_bonus(text: str) -> float:
    """Return 1.0 if text has 3–5 newlines (roughly 4 lines), else 0.0."""
    n = text.count("\n")
    return 1.0 if 3 <= n <= 5 else 0.0

def reward_fn(completions: list[list[dict]], **kwargs) -> list[float]:
    texts = [c[0]["content"] for c in completions]
    results = _sentiment(texts, truncation=True, max_length=128)
    rewards = []
    for text, r in zip(texts, results):
        score = r["score"]
        sentiment = score if r["label"] == "POSITIVE" else score - 1.0
        structure = _line_count_bonus(text)
        rewards.append(sentiment + 0.2 * structure)
    return rewards
```

> [!TIP]
> You can swap in `cardiffnlp/twitter-roberta-base-sentiment-latest` for a 3-class model (NEG / NEU / POS) if you want finer-grained scoring. It runs in ~50ms per batch on GPU.

> [!IMPORTANT]
> The structural bonus is pedagogically useful: it shows students that GRPO can simultaneously optimize *multiple* reward signals, and that reward engineering is an active design choice — not just "use a sentiment model".

---

## 5. GRPO Training (`training/train_grpo.py`)

### Key `GRPOConfig` Settings

```python
from trl import GRPOConfig, GRPOTrainer

config = GRPOConfig(
    output_dir=f"checkpoints/beta_{beta}",
    learning_rate=1e-5,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,       # effective batch = 8
    num_generations=8,                   # G: group size for advantage calc
    max_prompt_length=512,
    max_completion_length=128,
    num_train_epochs=1,                  # single pass for demo
    beta=beta,                           # KL penalty coefficient ← THE KEY PARAM
    logging_steps=5,
    save_steps=50,
    bf16=True,
    report_to="none",                    # or "tensorboard" for live curves
)
```

### Two β Settings

| Run | β | Effect |
|---|---|---|
| **Low β** | `0.04` | Model drifts farther from base; stronger sentiment optimization, less coherent poetry |
| **High β** | `0.4` | Stays close to base distribution; poetry coherent but sentiment shift modest |

> [!NOTE]
> GRPO advantage: for each prompt, generate G=8 completions, score them all, normalize within the group (mean-subtract, scale). No value network needed — just the policy and the reward function.

### Training Time Estimate (1× A100 / L4 GPU)
- Dataset: ~200 filtered poems → ~200 steps per epoch
- Each step: 2 prompts × 8 generations × 128 tokens ≈ **~10–15 min per run**
- Two runs total: **~25–30 min** — both run **live during class** while the notebook streams curves in real time

> [!NOTE]
> Run both β settings sequentially during class: start low-β first (~10 min), then high-β. The notebook polls the trainer's log file and updates plots live every N steps, so the audience watches reward climb in real time.

---

## 6. Jupyter Notebook Structure (`rlhf_poem_demo.ipynb`)

> [!NOTE]
> This is an **instructor-led live demo**. Students observe; no student GPU access needed. The notebook is the presentation surface — no slides.

### Section 0 — Setup & Imports
```python
# Install / imports, load tokenizer, base model, and sentiment pipeline
# Checkpoints are loaded lazily after training completes
```

### Section 1 — RLHF Concept Explainer *(markdown cells)*
- What is RLHF? PPO → GRPO comparison diagram
- Why skip reward model training? → pretrained sentiment model as proxy
- GRPO advantage: group relative scoring (formula cell with LaTeX)

$$A_i = \frac{r_i - \text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})} \quad \text{where } \mathbf{r} = [r_1, \ldots, r_G]$$

- Key point: **the system prompt is neutral** — RLHF is what does the work

### Section 2 — Dataset Explorer (Interactive)
- Widget: slider to pick a poem index (fetched from PoetryDB)
- Display: title, author, full text, highlighted prompt lines (lines 1–N in one color, remaining lines greyed out)
- Show distribution of poem lengths (histogram)

### Section 3 — Reward Function Demo
- Input any text → get sentiment score + line-count bonus live
- Show score on 5 hand-picked poetry lines (positive vs gloomy)
- Builds intuition for the reward signal before training begins

### Section 4 — 🔴 Live Training (runs during class)

```python
# Cell that launches GRPOTrainer for beta=0.04, streams logs
trainer_low = build_trainer(beta=0.04)

# Live plot: updates every logging_steps using a callback + IPython display clear
from IPython.display import clear_output

class LivePlotCallback(TrainerCallback):
    def on_log(self, args, state, control, logs=None, **kwargs):
        clear_output(wait=True)
        plot_training_curves(state.log_history)
```

- Plot updates every 5 steps: **mean reward vs step** and **KL divergence vs step**
- After low-β finishes, instructor narrates what happened, then runs high-β cell
- Side-by-side comparison chart appears automatically when both runs complete

### Section 5 — Poem Sampler (The Main Demo Cell)

Interactive widget with:
- **Dropdown:** pick a poem (by title, fetched from PoetryDB)
- **Slider:** choose number of context lines (8–15)
- **Button:** "Generate completions"

Outputs rendered side-by-side:

```
┌─────────────────────────────────────────────────────────────────┐
│  PROMPT (lines 1–N of "[Title]" by [Author]):                   │
│  [poem text]                                                     │
├──────────────────┬───────────────────┬──────────────────────────┤
│  Base Model      │  Tuned (β=0.04)   │  Tuned (β=0.4)          │
│  [completion]    │  [completion]      │  [completion]            │
│  Sentiment: 0.42 │  Sentiment: 0.89  │  Sentiment: 0.71         │
└──────────────────┴───────────────────┴──────────────────────────┘
```

### Section 6 — Quantitative Comparison
- Run 20 random poems through all 3 models
- Bar chart: mean sentiment score (Base vs β=0.04 vs β=0.4)
- Scatter: sentiment score vs "perplexity" (measured by reference model) → shows the coherence/sentiment tradeoff

---

## 7. Hardware & Environment

```text
Recommended: single GPU with ≥ 16 GB VRAM (A10G / L4 / T4)
Minimum:     T4 (Colab/Kaggle) with 4-bit quantization (use bitsandbytes + LoRA GRPO)
```

```text
# requirements.txt
torch>=2.3.0
transformers>=4.44.0
trl>=0.11.0            # GRPOTrainer stable in ≥ 0.9
peft>=0.12.0           # LoRA adapters
datasets>=2.20.0
accelerate>=0.33.0
bitsandbytes>=0.43.0   # optional, for 4-bit on small GPU
ipywidgets>=8.0
matplotlib>=3.9
pandas
```

> [!WARNING]
> Llama 3.2 1B requires accepting Meta's license on Hugging Face and a valid `HF_TOKEN`. Make sure students have this set before class, or use an open-weight alternative like `HuggingFaceTB/SmolLM2-1.7B-Instruct` which requires no gating.

---

## 8. In-Class Demo Flow (suggested ~55 min)

| Time | Activity |
|---|---|
| 0–5 min | Intro: What is RLHF, why GRPO? Walk through Section 1 |
| 5–10 min | Dataset explorer (Section 2) — fetch poems live from PoetryDB, pick a few, show neutral prompt format |
| 10–15 min | Reward function live demo (Section 3) — type examples, see sentiment scores + structure bonus |
| 15–28 min | 🔴 **Launch low-β training live** — curves update every 5 steps; instructor narrates what's happening |
| 28–30 min | Checkpoint saved; discuss what the reward curve shape tells us |
| 30–43 min | 🔴 **Launch high-β training live** — compare curves as it runs |
| 43–45 min | Show side-by-side training curve comparison (Section 4) |
| 45–55 min | **Live poem sampler** (Section 5) — pick poems, run inference, compare 3 models + Q&A |

> [!TIP]
> Pre-select 3–5 "showcase" poems before class: ones where the base model gives a neutral/melancholy ending and the tuned model gives a noticeably brighter one. Have their indices ready to type into the widget.

---

## 9. Implementation Phases

```mermaid
flowchart LR
    P1["Phase 1\nData & Reward\n(~1 hr)"] --> P2["Phase 2\nGRPO Training\n(~2 hrs incl. GPU time)"]
    P2 --> P3["Phase 3\nNotebook UI\n(~2 hrs)"]
    P3 --> P4["Phase 4\nDry Run & Polish\n(~1 hr)"]
```

### Phase 1 — Data & Reward (~1 hr)
- [ ] Load and filter poetry dataset
- [ ] Write `poem_loader.py` with prompt builder
- [ ] Write and test `reward.py` on 10 sample completions
- [ ] Verify sentiment model outputs sane scores

### Phase 2 — GRPO Training (~2 hrs incl. GPU time)
- [ ] Write `train_grpo.py` with CLI `--beta` argument
- [ ] Run low-β training → save checkpoint
- [ ] Run high-β training → save checkpoint
- [ ] Export training logs as CSV for notebook

### Phase 3 — Notebook (~2 hrs)
- [ ] Sections 0–3: setup + explainer + interactive demos
- [ ] Section 4: load CSVs, plot training curves
- [ ] Section 5: poem sampler widget (ipywidgets)
- [ ] Section 6: quantitative comparison over 20 poems

### Phase 4 — Dry Run (~1 hr)
- [ ] Run full notebook top-to-bottom on demo machine
- [ ] Check widget rendering in JupyterLab / VS Code / Colab
- [ ] Confirm both checkpoints load cleanly
- [ ] Prepare 3–5 "good" poems to highlight during live demo

---

## 10. Open Questions / Decisions

1. **Model gating:** Use gated `meta-llama/Llama-3.2-1B-Instruct` (need HF token) or ungated `HuggingFaceTB/SmolLM2-1.7B-Instruct`?
2. **Sentiment model:** Keep DistilBERT SST-2, or use `cardiffnlp/twitter-roberta-base-sentiment-latest` (3-class: NEG/NEU/POS)?
3. **Line-count bonus weight:** 0.2 feels reasonable — want it higher or lower relative to the sentiment signal?
