# In-Class RLHF Demo Plan: Sentiment-Steered Poetry Continuation

## 1. Executive Summary & Objective

This project delivers an interactive, classroom-ready **Reinforcement Learning from Human Feedback (RLHF)** demonstration. 
* **Base Model**: `meta-llama/Llama-3.2-1B-Instruct` (parameter-efficient via LoRA).
* **Dataset**: **PoetryDB** (extracting poems with at least 12–19 lines; slicing the first 8–15 lines as prompt, and the subsequent 4 lines as reference continuation).
* **Reward Model**: Pre-trained sentiment classifier (`distilbert-base-uncased-finetuned-sst-2-english`), rewarding positive sentiment in the generated 4 lines.
* **RL Algorithm**: TRL's Online Policy Optimization (`GRPOTrainer` / `RLOOTrainer`), which natively optimizes policies directly against custom reward models/functions without needing an unstable critic network.
* **Deliverable**: An interactive Jupyter Notebook configured for the `newGPU` environment with dual-mode execution (a quick 2-minute live training demonstration + pre-trained LoRA adapter loading for zero-wait in-class inference) and an interactive side-by-side poem comparison tool.

```mermaid
flowchart LR
    A["PoetryDB Dataset"] --> B["Poem Slicer<br/>(8-15 line prompt)"]
    B --> C["Base Policy<br/>Llama-3.2-1B"]
    C --> D["Generated 4-Line<br/>Continuation"]
    D --> E["Sentiment Reward Model<br/>(DistilBERT SST-2)"]
    E --> F["Positive Sentiment Score<br/>Reward: [0, 1]"]
    F --> G["RLHF Optimizer<br/>(GRPO / RLOO via TRL)"]
    G --> H["LoRA Adapter Updates"]
    H -.-> C
```

---

## 2. Technical Architecture & Environment

* **Conda Environment**: `newGPU` (`/home/canders1/miniforge3/envs/newGPU`)
  * PyTorch 2.8.0 + CUDA 13.0
  * Transformers 5.20.1, PEFT 0.22.0, TRL 1.18.1, Datasets 5.1.0
  * Hardware: 2x NVIDIA RTX A6000 (48GB VRAM each)
* **Storage Footprint**:
  * Base model (~2.5 GB bf16) cached locally.
  * LoRA adapter weights: < 50 MB.
  * Dataset cache: ~10 MB.

---

## 3. Detailed Component Design

### 3.1 Data Preparation (`data_prep.py` & Notebook Section 1)
* **Source**: Query PoetryDB API for classic poems (Shakespeare, Dickinson, Shelley, Byron, Keats, Milton, etc.) and save locally as `data/poetrydb_curated.json` to prevent network latency during class.
* **Filtering & Slicing**:
  * Filter poems with $\ge 16$ lines.
  * Dynamically sample prompt length $L \in [8, 15]$ lines.
  * Ground truth continuation = next 4 lines ($L+1$ to $L+4$).
* **Prompt Template**:
  ```text
  Continue the following poem with exactly the next four lines:
  
  [Line 1]
  ...
  [Line L]
  
  Next four lines:
  ```

### 3.2 Reward Modeling (Notebook Section 2)
* **Model**: `distilbert/distilbert-base-uncased-finetuned-sst-2-english`.
* **Reward Formulation**:
  * Given generated completion $y$ (the predicted 4 lines):
    $$r(y) = P(\text{Positive} \mid y) \in [0.0, 1.0]$$
  * Optional rule-based shaping: Small brevity/repetition penalties if the model rambles beyond 4 lines or produces gibberish.

### 3.3 RL Policy Fine-Tuning (Notebook Section 3)
* **Why LoRA (PEFT)?**
  * Freezes the base 1B model and trains low-rank adapters ($r=16, \alpha=32$).
  * Enables swapping between the **Base Model** and the **Tuned Model** in-memory simply by toggling `model.disable_adapter()` / `model.enable_adapter()`, eliminating the need to load two 1B models simultaneously.
* **Classroom Training Strategy**:
  1. **Pre-trained Checkpoint Provided**: Pre-train a checkpoint (e.g. 150–250 steps) beforehand so the lecture never depends on live GPU training.
  2. **Live Mini-Run (Interactive)**: An optional live cell that runs 5–10 steps in ~60 seconds to demonstrate policy gradients, reward trajectories, and loss logging on screen for the students.

### 3.4 Interactive Demo & Comparison Tool (Notebook Section 4)
* An interactive widget or function `compare_poem_completion(sample_idx=None)`:
  * **Input**: Random sample from test poems (or manual selection).
  * **Display Elements**:
    1. **Metadata**: Title, Author, line count.
    2. **Prompt**: The 8–15 line excerpt.
    3. **Ground Truth**: The original next 4 lines written by the poet.
    4. **Base Model Output**: Un-tuned Llama-3.2-1B completion + DistilBERT sentiment score.
    5. **RLHF-Tuned Output**: Tuned Llama-3.2-1B completion + DistilBERT sentiment score.
    6. **Delta ($\Delta r$)**: Clear visual highlighting of the sentiment shift (e.g., turning melancholic verse into joyful or optimistic continuation).

---

## 4. Implementation Stages & Roadmap

| Stage | Action Items | Output Artifacts |
|---|---|---|
| **Phase 1: Data Pipeline** | Fetch & curate PoetryDB poems, implement line slicer (8-15 prompt, 4 target), create train/eval split. | `data/poetrydb_curated.json`, `data_utils.py` |
| **Phase 2: Reward & Baseline Test** | Verify sentiment pipeline scoring on poem lines; test baseline Llama-3.2-1B prompt formatting. | `reward.py` validation script |
| **Phase 3: Pre-training RLHF Adapter** | Run offline RL training script using TRL (LoRA) to obtain the tuned adapter. | `checkpoints/llama3.2_1b_rlhf_sentiment/` |
| **Phase 4: Jupyter Notebook Creation** | Build the full self-contained in-class notebook with explanations, data inspection, training curves, and side-by-side interactive comparison widget. | `RLHF_Poem_Sentiment_Demo.ipynb` |
| **Phase 5: Verification & Rehearsal** | Execute the entire notebook end-to-end in `newGPU` environment, verifying cell outputs, memory usage, and UI rendering. | Verified demo notebook |

---

## 5. Pedagogical Discussion Points for the In-Class Demo

The demo is structured to highlight key conceptual takeaways for students:
1. **The Alignment Problem**: How the objective function (positive sentiment) competes with the natural style, rhyme scheme, and meter of classic poetry.
2. **Reward Hacking / Mode Collapse**: Pointing out cases where the model injects overtly joyful adjectives ("smiling sun", "happy flowers") at the expense of poetic subtlety.
3. **KL Penalty Role**: Showing why keeping the policy close to the reference model prevents the model from generating incoherent positive gibberish.
4. **Base vs. Tuned Comparison**: Demonstrating how RLHF steers language model priors without retraining from scratch.
