{"step_index":0,"source":"USER_EXPLICIT","type":"USER_INPUT","status":"DONE","created_at":"2026-11-20T17:11:59Z","content":"\n/plan Make a plan for an in-class demo of RLHF. I want to use the PoemDB dataset and tune a a Llama 3.2 (1B) model to take the first 8-15 lines of a poem and produce a 4 line completion that is positive in sentiment. Instead of training a reward model, we can use a pretrained sentiment analysis model. Use GRPO as the RL technique. In addition, I want a Jupyter notebook that shows live training, example data, and compares two beta settings. It should also allow us to sample a poem, look at the original lines, the base model completion, and the tuned model completion.\n\n\nThe current local time is: 2026-11-20T12:11:59-05:00.\n\n\nThe user changed setting `Model Selection` from None to Claude Sonnet 4.6 (Thinking). No need to comment on this change if the user doesn't ask about it. If reporting what model you are, please use a human readable name instead of the exact string.\n"} {"step_index":1,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:11:59Z","model":"claude-sonnet-4-6","input_tokens":13003,"cache_read_tokens":0,"output_tokens":248,"thinking":"The user wants a plan for an in-class demo of RLHF using GRPO. Let me create a comprehensive plan artifact for this.","tool_calls":[{"name":"run_command","args":{"CommandLine":"find /home/canders1/rlhf-s2 -type f | head -50 && echo \"---\" && ls /home/canders1/rlhf-s2","WaitMsBeforeAsync":5000,"toolSummary":"Explore workspace"}},{"name":"search_web","args":{"query":"PoemDB dataset HuggingFace RLHF GRPO Llama 3.2 1B sentiment"}}]} {"step_index":2,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:12:03Z","content":"Created At: 2026-11-20T12:12:03-05:00\nCompleted At: 2026-11-20T12:12:22-05:00\nCreated At: 2026-11-20T12:12:03-05:00\nCompleted At: 2026-11-20T12:12:22-05:00\nCommand exited with code 0.\n\nCommand output:\n---\r\n"} {"step_index":3,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:12:04Z","content":"Created At: 2026-11-20T12:12:04-05:00\nCompleted At: 2026-11-20T12:12:26-05:00\nCreated At: 2026-11-20T12:12:04-05:00\nCompleted At: 2026-11-20T12:12:26-05:00\nThe search for \"PoemDB dataset HuggingFace RLHF GRPO Llama 3.2 1B sentiment\" returned the following summary:\nIt looks like your query combines a few different concepts in the AI/Machine Learning ecosystem (possibly referring to poetry sentiment datasets like `mteb/poem_sentiment`[1] or structured poem datasets on Hugging Face[2], combined with modern alignment techniques like **GRPO**—Group Relative Policy Optimization[3]—and compact open-weight models like **Llama 3.2 1B**[4]). \n\nIf you are trying to build a pipeline, fine-tune, or align **Llama-3.2-1B** using **GRPO** (popularized by models like DeepSeekMath/R1 for reinforcement learning without a separate reward model or using rule-based/sentiment rewards), or if you are looking for specific code to evaluate sentiment on poem datasets, please clarify your goal! \n\nCommon tasks combining these keywords usually look like:\n1. **Loading a sentiment dataset** (e.g., `mteb/poem_sentiment`[1] or similar text/poetry corpora from Hugging Face).\n2. **Setting up GRPO** via Hugging Face TRL (`GRPOTrainer`) to align a small model like `meta-llama/Llama-3.2-1B`[4].\n3. **Defining a reward function** (e.g., maximizing positive sentiment or specific structural constraints in generated poetry).\n\nLet me know what specific script, dataset link, or implementation steps (e.g., TRL configuration, reward function design, or dataset formatting) you are looking for!\n\nSources:\n[1] [huggingface.co](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFna1UXScf1p16u1sZSv68p_GER06Q7buMih6_DE8y22fuDjjsahdyffMg6_uuHbfHlew89ZdHsyjJNt-Vz1k1zW0Ak0y00Jn9KQdKIDHBTTT3q5yflmdL8FIDQXyJjCz8vP5mrwLhoRH_CLMbKFcwc5sYTgijDQWeJ1NH7Hog=)\n[2] [huggingface.co](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHeMeHg3JL8z-3MzCOVUW_6sHqp-7A58QgtrMEmdZDFgPUFy25Q97f9wZ7CovH2g-4ah9tq5BUzVi4azBIHTu3VU8QrBNmcQMtDpBakWFLjUGAGiihwA_PEvEJ8LZuBUgcnTXhKazdkp0pH88E6z4jP0BpV27J91l0k0feg9T24itvHjRXl5VazzA==)\n[3] [youtube.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG6A1BI60CEr6KVcvvZ_xJr1NUbLyOAL7u0IT01QjF3lSeS6gJz8RtluH6dGhT7xMFDoPgRcLUsfDVdPCWl7Ic5c8oHvA2HDbkgs1_1YhfuDnoC3mJLTKWRymlijG-Tl07h)\n[4] [huggingface.co](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHf-tR8Gy1_8Ufp6DgfL3qiIUKdpLDYMObVF11aUPku74oZYL_yfrs-FFbszsNvUGUPCnbD-8cUSBee0eXIDAowAQeU3NsQijnl8WT4IHIyKRIH1-L9nfSr7OWqBjUSQRMe0DuV)\n\nNote: Source URLs are HTTP redirects; check where they resolve if you need the direct link."} {"step_index":4,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:12:26Z","model":"claude-sonnet-4-6","input_tokens":2165,"cache_read_tokens":12389,"output_tokens":112,"tool_calls":[{"name":"search_web","args":{"query":"PoemDB dataset huggingface poems collection"}},{"name":"search_web","args":{"query":"TRL GRPOTrainer example sentiment reward function 2024"}}]} {"step_index":5,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:12:27Z","content":"Created At: 2026-11-20T12:12:27-05:00\nCompleted At: 2026-11-20T12:12:32-05:00\nCreated At: 2026-11-20T12:12:27-05:00\nCompleted At: 2026-11-20T12:12:32-05:00\nThe search for \"PoemDB dataset huggingface poems collection\" returned the following summary:\nIf you are looking for poetry collections and text datasets on Hugging Face—similar to a comprehensive \"PoemDB\"—there are several popular, community-curated poetry datasets widely used for NLP tasks, style modeling, and fine-tuning generative models (like GPT-2 or LLaMA)[1][2][3]:\n\n### 1. Popular Poetry Datasets on Hugging Face\n* **`suayptalha/Poetry-Foundation-Poems`**\n * **Overview:** One of the most comprehensive collections available on Hugging Face, containing thousands of poems scraped from the Poetry Foundation[4][2]. \n * **Features:** Includes metadata such as poem titles, authors, text content, and thematic tags[4].\n * **Usage Example:**\n ```python\n from datasets import load_dataset\n dataset = load_dataset(\"suayptalha/Poetry-Foundation-Poems\")\n ```\n\n* **`merve/poetry`**\n * **Overview:** A lighter, widely-used curated poetry dataset frequently leveraged for lightweight fine-tuning tutorials and poem-generation projects[3][5].\n\n* **Collections & MTEB**\n * Hugging Face also organizes specialized groupings like **PoetryMTEB Collections** for embedding and semantic tasks involving verse[6].\n\n---\n\n### 2. Quickstart: How to Load a Poetry Dataset\nYou can easily load, inspect, or slice these collections using Python's `datasets` library[2]:\n\n```python\nfrom datasets import load_dataset\n\n# Load the Poetry Foundation collection\ndataset = load_dataset(\"suayptalha/Poetry-Foundation-Poems\")\n\n# Inspect a single poem\nprint(dataset[\"train\"][0])\n```\n\nIf you meant a specific local database schema or a different repository named \"PoemDB\", let me know and I can pinpoint the exact resource or provide code tailored to it!\n\nSources:\n[1] [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF86f6KUqYU_tEMixNLKskJt7Q4CA_AL65nvNzWH7zvmBJ_qXvrxHSqYo_lc6gR8gKd3EXQoVh3mNPfaNHgoHccqWyoR5AI6TnsYddO_1fffJL1ZXNh7bb9dArO1UnQbA==)\n[2] [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGvIOeB-jpp9K6Wvw5tq4KiyQQsqjE7aPhwRgIpCCYd-ijyAvFWzj5jC8Dp26wP9IDrfmlexdqJ4J5NcKDN0sK71E-3pyVsYCpnnBImy42iW85It82y40bSx43PhpYkv8k1Hrvpnmc5dMaFDJeHGE0dsZ5Sr-myAjS_Uawys7eMB2brQYnlOeOSZQe4cbb-ghsbhTrL3Om4SfR78lGkveAnfFeLTDFVAGTzxvoJu5kfkA==)\n[3] [huggingface.co](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHxTDmVkeFR6M4b_wLvE8GPd0i3UJebKhy2y-4ZUiobcjLeKe0ADywJQdxQHwhtocPCQOyuKyj6KUUwPYbQKSB_QePRoFUbIWL7kyhS0m9u9psz4uLcYaBxZtvzZjB5KVcoycMFcg==)\n[4] [kaggle.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEDWmc2QyhsZGKnu6pNQOOqt9iKg3Nu3KuxvEw-6OaPni_U9b2ILTWdY_ela8DUSWhXjyNxgkxasXXb_P_cXPd__05EHCZjfV2YtjXLZyp5mf08c8aUAawNcn6Y2mrFBTuvAVMiTq74K4znOXXQziqej94akZzs0qVkZmi4oQ==)\n[5] [huggingface.co](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGhGZVEy7eR5KOrrIwSDK4CALT1gO4xwfg01Z0vrXZ5QFKH2lpmjyQOaEkUKrvzWhU1LGYvDcj0_hL0eq7Xa3StzrNT72lS3tG3bSsxv45w2T0URGH2Mx0piCgIbXE2OZixtQ==)\n[6] [huggingface.co](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFStJuV__OCu_iQh5fF_UPF2cuN8JOg4MhoWjWVqRi__ahRRuRfqJ4ZjrzYw2Q9aY4NCUPsFGJ-1gBFewLIBrepnPH94PPX18Sk2-o32QzBxywDeDhIIC1Y3xyuA31KTCIAd2xY5nshO9ehPB8EZFo8Vpkg)\n\nNote: Source URLs are HTTP redirects; check where they resolve if you need the direct link."} {"step_index":6,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:12:28Z","content":"Created At: 2026-11-20T12:12:28-05:00\nCompleted At: 2026-11-20T12:12:37-05:00\nCreated At: 2026-11-20T12:12:28-05:00\nCompleted At: 2026-11-20T12:12:37-05:00\nThe search for \"TRL GRPOTrainer example sentiment reward function 2024\" returned the following summary:\nHere is an example of implementing a custom **sentiment reward function** for Hugging Face TRL’s `GRPOTrainer`[1][2]. \n\nIn this example, we use a lightweight sentiment analysis model (from Hugging Face `transformers` or a library like `textblob`/`vaderSentiment`) to score the completions generated by the model. The `GRPOTrainer` evaluates a group of completions for each prompt, and this reward function returns a list of scalar float values corresponding to how well each completion fits the desired sentiment[1][3].\n\n### Example: Sentiment-Based Reward Function with `GRPOTrainer`\n\n```python\nfrom datasets import Dataset\nfrom transformers import pipeline\nfrom trl import GRPOConfig, GRPOTrainer\n\n# 1. Initialize a sentiment analysis pipeline (e.g., to encourage positive responses)\n# For a more nuanced approach, you could use a fine-tuned roberta/distilbert sentiment model\nsentiment_pipeline = pipeline(\"sentiment-analysis\", model=\"distilbert/distilbert-base-uncased-finetuned-sst-2-english\")\n\n# 2. Define the custom sentiment reward function\ndef sentiment_reward_func(completions, **kwargs):\n \"\"\"\n Reward function that scores completions based on positive sentiment confidence.\n \n Args:\n completions (list[list[dict]]): List of generated completions in chat format.\n **kwargs: Captures additional dataset columns or trainer context.\n \n Returns:\n list[float]: A list of scalar reward scores for each completion.\n \"\"\"\n # Extract the raw text string from the chat message structure\n # TRL passes completions as a list of message lists: [[{\"role\": \"assistant\", \"content\": \"...\"}]]\n texts = [completion[0][\"content\"] for completion in completions]\n \n # Run the texts through the sentiment analyzer\n results = sentiment_pipeline(texts)\n \n rewards = []\n for res in results:\n score = res[\"score\"]\n # If the model predicts POSITIVE, give a positive reward proportional to confidence.\n # If NEGATIVE, invert or penalize the score.\n if res[\"label\"] == \"POSITIVE\":\n rewards.append(float(score))\n else:\n rewards.append(float(-score))\n \n return rewards\n\n# 3. Create a dummy dataset with prompts instructing the model to be uplifting/positive\ndataset = Dataset.from_list([\n {\"prompt\": [{\"role\": \"user\", \"content\": \"Write a short encouraging note to a friend who is feeling down.\"}]},\n {\"prompt\": [{\"role\": \"user\", \"content\": \"Compliment someone's hard work on a project.\"}]},\n])\n\n# 4. Configure GRPO training arguments\ntraining_args = GRPOConfig(\n output_dir=\"grpo_sentiment_model\",\n learning_rate=2e-5,\n per_device_train_batch_size=2,\n num_generations=4, # Number of candidate completions generated per prompt group\n max_prompt_length=256,\n max_completion_length=128,\n logging_steps=1,\n)\n\n# 5. Initialize the GRPOTrainer\ntrainer = GRPOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\", # Or any base/instruction model\n reward_funcs=[sentiment_reward_func],\n args=training_args,\n train_dataset=dataset,\n)\n\n# 6. Run reinforcement learning post-training\ntrainer.train()\n```\n\n### Key Considerations for TRL GRPO Reward Functions:\n1. **Signature Requirement:** The reward function must accept `completions` (and optionally `prompts` or dataset columns like `ground_truth`) and accept `**kwargs` for future compatibility[1].\n2. **Data Structure:** `completions` is a list of lists containing dictionaries (`[{\"role\": \"assistant\", \"content\": \"...\"}]`)[4]. You must parse out the string content via `completion[0][\"content\"]` before passing it to external reward models or processing it with RegEx[5][4].\n3. **Group Relative Advantage:** GRPO computes the advantage by taking the scores returned by your reward function, grouping them per prompt, and normalizing them (mean-centering and scaling) relative to the group[3]. As long as your function returns relative scalar scores (higher is better), the trainer handles the baseline subtraction automatically.\n\nSources:\n[1] [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGXCAzS1_x_2EqRxnjGqwE7PdJOCP1t1_Q1XrJi05fTPdj3Hj-U_RoHBpIDcFSaNdh8KLc7MP9ixUDBlpGJOf8ah9Q-V17BAdXNLe_A5norv7kyaY0aiXH3Ge4Se14A1R6qRBQQNA4Bkz0h2gOaejedNaYd2ZfztGZetjXOVsk=)\n[2] [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFpVstXnxvNuEJ1IZX5MkIOE-EP4tSunB8pS1O0L2rA64GA4naB4YM5PFFFlDWhvLgnYp6C_Mrts3x7g2vx0fRcMd6P1dq69cDtFlElamTF45laWAXpzjc_dstJcoorDeqvnPU7-x95b1h76eJRmM5-7qwu1xiTVT7VGturAnPEts8OHABTnz_633smo6W7SK7gNybbr-m1zTAGE0A=)\n[3] [satyarthi.online](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHy0f21I776vm1lBwzEl70eNY6RzAKcQeOOg0M_pw9O8PkJxwTTg0EaywyX3BQs3X1eKCdc2vEpu9AAjLPQp4dZNRUHcjpEXuVrgyBReVJUEbNzAmFXNP-VbcDa5vjUTfqm0qnRiTrH6t29LmZMMneGK5lH)\n[4] [axolotl.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFD-gIiU2GwyOqPuVJ29Le9vSt8S5kK2H03GBrP6qYaX89-BC-pABgjzR51dmbVhOfw8KlvAX2vuO5COdE2kwW6YOjm3Vw3bDWH0l6sHMhO7Ec7tN7gOhNCE9MU)\n[5] [google.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEZUlJfcb58vbrYk2IiPSNZWbJJZP1PkkU5y0KnKI_OzFE-JzG6CPcJY8JrcUAzAEL0xFuj5I8l4QtD0BBZUIGanAuFugdS5AbcfWglA54MTnkt36dgXI3z_Bx3h7nxgQU2XpLUO2eRCOREbFuWBXeEih2WuxOWJP20XLtctoobmy_yfZQA87diKNOpyCuTBRV18i4WYgzF81RH9suRZTQ4xkba7qNDlQS4)\n\nNote: Source URLs are HTTP redirects; check where they resolve if you need the direct link."} {"step_index":7,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:12:37Z","model":"claude-sonnet-4-6","input_tokens":4041,"cache_read_tokens":14553,"output_tokens":4055,"content":"Now I have enough context to build a thorough plan. Let me create the artifact.","tool_calls":[{"name":"write_file","args":{"ArtifactMetadata":{"RequestFeedback":true,"Summary":"Comprehensive plan for an in-class RLHF demo using GRPO to fine-tune Llama 3.2 (1B) on poem completion with a sentiment reward signal. Covers data pipeline, reward function, training configuration, Jupyter notebook structure, hardware requirements, demo flow, and implementation phases.","UserFacing":true},"Content":"# RLHF In-Class Demo Plan: Sentiment-Steered Poem Completion with GRPO\n\n> [!IMPORTANT]\n> **Goal:** Fine-tune Llama 3.2 (1B) with GRPO to complete poems positively, using a pretrained sentiment model as the reward signal — no reward model training required. Demo delivered as a live Jupyter notebook.\n\n---\n\n## 1. Overview\n\n```mermaid\nflowchart LR\n A[\"PoemDB\\n(Poetry Foundation)\"] --> B[\"Prompt builder\\nlines 8–15 → prompt\"]\n B --> C[\"Base Llama 3.2 1B\\n(reference policy)\"]\n B --> D[\"GRPO Training Loop\"]\n E[\"Sentiment model\\n(DistilBERT SST-2)\"] --> D\n D --> F[\"Tuned Policy\\n(beta=0.1 & beta=0.5)\"]\n C --> G[\"Notebook Demo\\nSide-by-side comparison\"]\n F --> G\n```\n\n**Core choices:**\n| Component | Choice | Rationale |\n|---|---|---|\n| Dataset | `merve/poetry` or `suayptalha/Poetry-Foundation-Poems` | Freely available, rich metadata |\n| Base model | `meta-llama/Llama-3.2-1B-Instruct` | Small enough for in-class GPU, strong enough to generate coherent poetry |\n| RL algorithm | GRPO via HuggingFace TRL `GRPOTrainer` | No separate reward model needed; stable, group-based advantage estimation |\n| Reward signal | `distilbert-base-uncased-finetuned-sst-2-english` | Fast inference, well-calibrated positive/negative scores |\n| KL penalty β | Two runs: **β = 0.04** (aggressive) vs **β = 0.4** (conservative) | Shows the exploration-vs-fidelity tradeoff live |\n\n---\n\n## 2. Repository / File Structure\n\n```\nrlhf-s2/\n├── data/\n│ └── poem_loader.py # Dataset loading & prompt construction\n├── training/\n│ ├── reward.py # Sentiment reward function\n│ ├── train_grpo.py # Script-based GRPO trainer (beta param CLI arg)\n│ └── configs/\n│ ├── grpo_beta_low.yaml\n│ └── grpo_beta_high.yaml\n├── notebooks/\n│ └── rlhf_poem_demo.ipynb # ← MAIN DEMO NOTEBOOK\n├── checkpoints/ # saved model weights (gitignored)\n└── requirements.txt\n```\n\n---\n\n## 3. Data Pipeline\n\n### 3a. Dataset Loading\n\nUse `merve/poetry` (HuggingFace) — ~573 poems with title, author, and content.\n\n```python\nfrom datasets import load_dataset\nds = load_dataset(\"merve/poetry\", split=\"train\")\n```\n\n> [!NOTE]\n> `suayptalha/Poetry-Foundation-Poems` is richer (~14k poems) and includes tags like \"joy\", \"sadness\" — useful for filtering to poems that have *room* for a positive turn. Either works; use `merve/poetry` for fast demo startup.\n\n### 3b. Prompt Construction (`poem_loader.py`)\n\nEach training example:\n1. Split poem text by newline\n2. Keep only poems with ≥ 12 lines (guarantees 8–15 lines of context)\n3. Sample a window of 8–15 lines (randomly during training, fixed during demo)\n4. Format as a chat-style prompt:\n\n```python\ndef build_prompt(poem_lines: list[str], n_context: int) -> list[dict]:\n context = \"\\n\".join(poem_lines[:n_context])\n return [\n {\n \"role\": \"system\",\n \"content\": (\n \"You are a poet. Complete the following poem with exactly 4 lines. \"\n \"Your completion should be uplifting and positive in sentiment.\"\n )\n },\n {\n \"role\": \"user\",\n \"content\": f\"Here is the beginning of a poem:\\n\\n{context}\\n\\nWrite the next 4 lines:\"\n }\n ]\n```\n\n**Output format constraint:** Reward function also penalizes completions that are not exactly 4 lines (optional length reward component).\n\n---\n\n## 4. Reward Function (`training/reward.py`)\n\n```python\nfrom transformers import pipeline\n\n_sentiment = pipeline(\n \"sentiment-analysis\",\n model=\"distilbert-base-uncased-finetuned-sst-2-english\",\n device=0 # GPU\n)\n\ndef sentiment_reward(completions: list[list[dict]], **kwargs) -> list[float]:\n texts = [c[0][\"content\"] for c in completions]\n results = _sentiment(texts, truncation=True, max_length=128)\n rewards = []\n for r in results:\n score = r[\"score\"]\n if r[\"label\"] == \"POSITIVE\":\n rewards.append(score) # [0, 1]\n else:\n rewards.append(score - 1.0) # [-1, 0]\n return rewards\n```\n\n**Optional bonus reward:** Add `+0.1` if the completion has 3–5 newlines (≈ 4 lines). Keeps outputs structurally poetic.\n\n> [!TIP]\n> You can swap in `cardiffnlp/twitter-roberta-base-sentiment-latest` for a 3-class model (NEG / NEU / POS) if you want finer-grained scoring. It runs in ~50ms per batch on GPU.\n\n---\n\n## 5. GRPO Training (`training/train_grpo.py`)\n\n### Key `GRPOConfig` Settings\n\n```python\nfrom trl import GRPOConfig, GRPOTrainer\n\nconfig = GRPOConfig(\n output_dir=f\"checkpoints/beta_{beta}\",\n learning_rate=1e-5,\n per_device_train_batch_size=2,\n gradient_accumulation_steps=4, # effective batch = 8\n num_generations=8, # G: group size for advantage calc\n max_prompt_length=512,\n max_completion_length=128,\n num_train_epochs=1, # single pass for demo\n beta=beta, # KL penalty coefficient ← THE KEY PARAM\n logging_steps=5,\n save_steps=50,\n bf16=True,\n report_to=\"none\", # or \"tensorboard\" for live curves\n)\n```\n\n### Two β Settings\n\n| Run | β | Effect |\n|---|---|---|\n| **Low β** | `0.04` | Model drifts farther from base; stronger sentiment optimization, less coherent poetry |\n| **High β** | `0.4` | Stays close to base distribution; poetry coherent but sentiment shift modest |\n\n> [!NOTE]\n> GRPO advantage: for each prompt, generate G=8 completions, score them all, normalize within the group (mean-subtract, scale). No value network needed — just the policy and the reward function.\n\n### Training Time Estimate (1× A100 / L4 GPU)\n- Dataset: ~200 filtered poems → ~200 steps per epoch\n- Each step: 2 prompts × 8 generations × 128 tokens ≈ **~8–12 min per run**\n- Two runs total: **~20–25 min** — run both before class, checkpoint saved\n\n---\n\n## 6. Jupyter Notebook Structure (`rlhf_poem_demo.ipynb`)\n\n### Section 0 — Setup & Imports\n```python\n# Install / imports, load tokenizer & models\n# Load both checkpoints (beta_low, beta_high) + base model\n```\n\n### Section 1 — RLHF Concept Explainer *(markdown cells)*\n- What is RLHF? PPO → GRPO comparison diagram (image or mermaid)\n- Why skip reward model training? → pretrained sentiment model as proxy\n- GRPO advantage: group relative scoring (formula cell with LaTeX)\n\n$$A_i = \\frac{r_i - \\text{mean}(\\mathbf{r})}{\\text{std}(\\mathbf{r})} \\quad \\text{where } \\mathbf{r} = [r_1, \\ldots, r_G]$$\n\n### Section 2 — Dataset Explorer (Interactive)\n- Widget: slider to pick a poem index\n- Display: title, author, full text, highlighted prompt lines\n- Show distribution of poem lengths (histogram)\n- Show class balance (sentiment of first stanzas vs last stanzas)\n\n### Section 3 — Reward Model Demo\n- Input any text → get sentiment score live\n- Show score on 5 poetry lines (positive vs negative)\n- Builds intuition for the reward signal\n\n### Section 4 — Live Training Curves\n- Load TensorBoard logs (or CSV from trainer state)\n- Plot: **mean reward vs step** for both β runs on same axes\n- Plot: **KL divergence vs step** for both runs\n- Plot: **reward std within group** (shows how spread collapses as model improves)\n\n```python\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfig, axes = plt.subplots(1, 3, figsize=(15, 4))\n# reward curves, KL curves, group reward std\n```\n\n### Section 5 — Poem Sampler (The Main Demo Cell)\n\nInteractive widget with:\n- **Dropdown:** pick a poem (by title)\n- **Slider:** choose number of context lines (8–15)\n- **Button:** \"Generate completions\"\n\nOutputs rendered side-by-side:\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ PROMPT (lines 1–N): │\n│ [poem text] │\n├──────────────────┬───────────────────┬──────────────────────────┤\n│ Base Model │ Tuned (β=0.04) │ Tuned (β=0.4) │\n│ [completion] │ [completion] │ [completion] │\n│ Sentiment: 0.42 │ Sentiment: 0.89 │ Sentiment: 0.71 │\n└──────────────────┴───────────────────┴──────────────────────────┘\n```\n\n### Section 6 — Quantitative Comparison\n- Run 20 random poems through all 3 models\n- Bar chart: mean sentiment score (Base vs β=0.04 vs β=0.4)\n- Scatter: sentiment score vs \"perplexity\" (measured by reference model) → shows the coherence/sentiment tradeoff\n\n---\n\n## 7. Hardware & Environment\n\n```text\nRecommended: single GPU with ≥ 16 GB VRAM (A10G / L4 / T4)\nMinimum: T4 (Colab/Kaggle) with 4-bit quantization (use bitsandbytes + LoRA GRPO)\n```\n\n```text\n# requirements.txt\ntorch>=2.3.0\ntransformers>=4.44.0\ntrl>=0.11.0 # GRPOTrainer stable in ≥ 0.9\npeft>=0.12.0 # LoRA adapters\ndatasets>=2.20.0\naccelerate>=0.33.0\nbitsandbytes>=0.43.0 # optional, for 4-bit on small GPU\nipywidgets>=8.0\nmatplotlib>=3.9\npandas\n```\n\n> [!WARNING]\n> Llama 3.2 1B requires accepting Meta's license on Hugging Face and a valid `HF_TOKEN`. Make sure students have this set before class, or use an open-weight alternative like `HuggingFaceTB/SmolLM2-1.7B-Instruct` which requires no gating.\n\n---\n\n## 8. In-Class Demo Flow (suggested ~45 min)\n\n| Time | Activity |\n|---|---|\n| 0–5 min | Intro: What is RLHF, why GRPO? Walk through Section 1 |\n| 5–10 min | Dataset explorer (Section 2) — pick a few poems, show sentiment of opening vs closing stanzas |\n| 10–15 min | Reward function live demo (Section 3) — type examples, see scores |\n| 15–20 min | Show pre-recorded training curves (Section 4) — discuss β tradeoff |\n| 20–35 min | **Live poem sampler** (Section 5) — pick poems, run inference, compare 3 models |\n| 35–45 min | Quantitative results (Section 6) + Q&A |\n\n---\n\n## 9. Implementation Phases\n\n```mermaid\nflowchart LR\n P1[\"Phase 1\\nData & Reward\\n(~1 hr)\"] --> P2[\"Phase 2\\nGRPO Training\\n(~2 hrs incl. GPU time)\"]\n P2 --> P3[\"Phase 3\\nNotebook UI\\n(~2 hrs)\"]\n P3 --> P4[\"Phase 4\\nDry Run & Polish\\n(~1 hr)\"]\n```\n\n### Phase 1 — Data & Reward (~1 hr)\n- [ ] Load and filter poetry dataset\n- [ ] Write `poem_loader.py` with prompt builder\n- [ ] Write and test `reward.py` on 10 sample completions\n- [ ] Verify sentiment model outputs sane scores\n\n### Phase 2 — GRPO Training (~2 hrs incl. GPU time)\n- [ ] Write `train_grpo.py` with CLI `--beta` argument\n- [ ] Run low-β training → save checkpoint\n- [ ] Run high-β training → save checkpoint\n- [ ] Export training logs as CSV for notebook\n\n### Phase 3 — Notebook (~2 hrs)\n- [ ] Sections 0–3: setup + explainer + interactive demos\n- [ ] Section 4: load CSVs, plot training curves\n- [ ] Section 5: poem sampler widget (ipywidgets)\n- [ ] Section 6: quantitative comparison over 20 poems\n\n### Phase 4 — Dry Run (~1 hr)\n- [ ] Run full notebook top-to-bottom on demo machine\n- [ ] Check widget rendering in JupyterLab / VS Code / Colab\n- [ ] Confirm both checkpoints load cleanly\n- [ ] Prepare 3–5 \"good\" poems to highlight during live demo\n\n---\n\n## 10. Open Questions / Decisions\n\n1. **GPU access:** Will students run inference themselves, or is this a live instructor demo?\n2. **Model gating:** Use gated `meta-llama/Llama-3.2-1B-Instruct` or ungated `SmolLM2-1.7B-Instruct`?\n3. **Training environment:** Pre-train checkpoints before class, or show training live (needs ~20 min in class)?\n4. **Line-count reward:** Add a structural bonus (penalize completions ≠ 4 lines) or keep reward purely sentiment-based?\n5. **β values:** Do `0.04` and `0.4` feel right for the tradeoff you want to show, or prefer a more dramatic gap (e.g., `0.01` vs `1.0`)?\n","Path":"/home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md"}}]} {"step_index":8,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:13:55Z","content":"Created At: 2026-11-20T12:13:55-05:00\nCompleted At: 2026-11-20T12:13:55-05:00\nCreated At: 2026-11-20T12:13:55-05:00\nCompleted At: 2026-11-20T12:13:55-05:00\nCreated file."} {"step_index":9,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:13:55Z","model":"claude-sonnet-4-6","input_tokens":4120,"cache_read_tokens":18593,"output_tokens":435,"content":"Here's the full plan → [rlhf-demo-plan.md](file:///home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md)\n\nHere's a quick orientation to the key decisions baked in:\n\n**Architecture**\n- Uses `trl.GRPOTrainer` with a plain Python reward function — no reward model training, just `distilbert-base-uncased-finetuned-sst-2-english` scoring completions from `[-1, 1]`\n- Two β runs (`0.04` vs `0.4`) demonstrate the KL-penalty tradeoff: coherent-but-cautious vs sentiment-maxing-but-drifty\n\n**Notebook flow**\n- Sections go: concept explainer → dataset explorer → reward model demo → training curves → **poem sampler widget** → quantitative bar chart\n- The sampler widget is the classroom highlight: pick any poem + context length → see base vs tuned (low β) vs tuned (high β) side-by-side with live sentiment scores\n\n**Training time:** ~20–25 min total on a single A100/L4 for both runs — plan to pre-train before class and load saved checkpoints\n\n---\n\n**Before I build the code, please weigh in on the 5 open questions at the bottom**, especially:\n\n1. **Model access** — Gated Llama 3.2 1B (need HF token) or ungated `SmolLM2-1.7B-Instruct`?\n2. **Training live in class** or instructor pre-trains + students just run inference?\n3. **β values** — `0.04 / 0.4` or a more dramatic spread?"} {"step_index":10,"source":"USER_EXPLICIT","type":"USER_INPUT","status":"DONE","created_at":"2026-11-20T17:18:19Z","content":"Comments on artifact URI: file:///home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md\n\nSelection:\n>GPU access: Will students run inference themselves, or is this a live instructor demo?\n\nComment: \"live demo; students won't run\"\n\nSelection:\n>Training environment: Pre-train checkpoints before class, or show training live (needs ~20 min in class)?\n\nComment: \"show live training\"\n\nSelection:\n>Line-count reward: Add a structural bonus (penalize completions ≠ 4 lines) or keep reward purely sentiment-based?\n\nComment: \"structural bonus makes sense\"\n\nSelection:\n>β values: Do 0.04 and 0.4 feel right for the tradeoff you want to show, or prefer a more dramatic gap (e.g., 0.01 vs 1.0)?\n\nComment: \"sure\"\n\nSelection:\n>Use merve/poetry (HuggingFace) — ~573 poems with title, author, and content.\n\nComment: \"I said use PoetryDB!\"\n\nSelection:\n>Your completion should be uplifting and positive in sentiment.\"\n\nComment: \"this defeats the whole purpose of RLHF. Use a neutral prompt.\"\n\n\n\n\nThe current local time is: 2026-11-20T12:18:19-05:00.\n"} {"step_index":11,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:18:19Z","model":"claude-sonnet-4-6","input_tokens":17345,"cache_read_tokens":6204,"output_tokens":393,"content":"Good feedback — two important corrections stand out:\n\n- **PoetryDB** is an API (`poetrydb.org`), not a HuggingFace dataset — the data pipeline needs a full rewrite around that\n- **Neutral prompt** is a crucial pedagogical point: telling the model to \"be positive\" in the system prompt bypasses RLHF entirely; the steering should come *only* from the reward signal","thinking":"The user has provided comments on the plan artifact. Let me address each comment:\n\n1. **Live demo; students won't run** - Remove student inference setup, focus on instructor-only demo\n2. **Show live training** - Plan needs to include live training in the notebook during class (~20 min)\n3. **Structural bonus makes sense** - Keep the line-count reward component\n4. **β values: 0.04 and 0.4 are fine** - Keep those\n5. **Use PoetryDB** - Not `merve/poetry`. PoetryDB is an API (poetrydb.org) with poems by author/title/lines. I'm switching the data source over to the PoetryDB API, and I'm also going to drop the system-prompt instruction to be positive. The sentiment should come from the RLHF/GRPO training itself, so the completion prompt stays neutral.\n\n","tool_calls":[{"name":"search_web","args":{"query":"PoetryDB API poetrydb.org endpoints documentation JSON"}}]} {"step_index":12,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:18:27Z","content":"Created At: 2026-11-20T12:18:27-05:00\nCompleted At: 2026-11-20T12:18:32-05:00\nCreated At: 2026-11-20T12:18:27-05:00\nCompleted At: 2026-11-20T12:18:32-05:00\nThe search for \"PoetryDB API poetrydb.org endpoints documentation JSON\" returned the following summary:\n**PoetryDB** (`poetrydb.org`) is a free, open-source REST API designed for developers and internet poets[1][2]. It requires **no authentication or API keys**[2][3] and returns data natively in **JSON format** (with optional `.json` or `.text` extensions)[2][4].\n\n---\n\n### 1. Base URL\n```text\nhttps://poetrydb.org\n```\n\n---\n\n### 2. Core API Structure & Endpoints\nThe PoetryDB endpoint architecture is flexible. You query resources by combining **search fields** and **values** separated by slashes (`/`), and you can request specific output fields or formats at the end of the URL.\n\n#### **General Pattern:**\n```text\nhttps://poetrydb.org///.\n```\n\n---\n\n### 3. Available Search Fields\nYou can query poems using single fields, or combine multiple fields using commas `,`.\n\n* `author` — Search by poet name (case-insensitive substring match by default)[1][4].\n* `title` — Search by poem title[5].\n* `lines` — Search by text/phrases appearing within the lines of the poem[2].\n* `linecount` — Search by the exact number of lines (e.g., `14`), or range (`-14` for 14 or fewer lines)[6][7].\n* `random` — Retrieve a randomized set of poems (takes an integer, e.g., `5` for 5 poems)[8][7].\n* `author,title` — Combine fields[1].\n\n---\n\n### 4. Common Endpoint Examples (JSON)\n\n#### **Get a list of all poets:**\n* **URL:** `https://poetrydb.org/author`[1]\n* **Description:** Returns an alphabetically sorted JSON array of all available author names[8].\n\n#### **Get poems by a specific author:**\n* **URL:** `https://poetrydb.org/author/Emily Dickinson`[1]\n* **Description:** Returns all poems written by the specified author in JSON.\n\n#### **Get poems by title:**\n* **URL:** `https://poetrydb.org/title/Ozymandias`[5]\n* **Description:** Returns poetry objects matching the exact or partial title.\n\n#### **Get poems by author and title combo:**\n* **URL:** `https://poetrydb.org/author,title/Shakespeare;Sonnet`[1]\n* **Description:** Finds matches using multiple criteria separated by semicolons `;`[9].\n\n#### **Get random poems:**\n* **URL:** `https://poetrydb.org/random/3`\n* **Description:** Returns 3 random poems from the database[8].\n\n#### **Get author biographies:**\n* **URL:** `https://poetrydb.org/author/Dickinson/info`[1][4]\n* **Description:** Returns structured biographical details about the author (including dates, nationality, themes, and notable poems)[10].\n\n---\n\n### 5. Advanced Modifiers & Parameters\n\n* **Exact Matching (`:abs`)** \n By default, search fields use case-insensitive substring matching. Appending `:abs` forces an exact match[11][4].\n * *Example:* `https://poetrydb.org/author/Emily Dickinson:abs`\n\n* **Selecting Specific Output Fields** \n You can append a trailing comma-delimited list of fields you want returned (e.g., `author,title,linecount` or just `lines`) to minimize payload sizes[4].\n * *Example:* `https://poetrydb.org/title/Ozymandias/lines`[6]\n\n* **Format Extension** \n While JSON is the default response type, you can explicitly enforce formatting by appending `.json` or `.text`[2][4].\n * *Example:* `https://poetrydb.org/title/Ozymandias/lines.json`[6]\n\n---\n\n### 6. Example JSON Response Structure\nA typical poetry query (e.g., `/title/Ozymandias/lines`) yields a JSON array structured like this[6]:\n\n```json\n[\n {\n \"title\": \"Ozymandias\",\n \"author\": \"Percy Bysshe Shelley\",\n \"lines\": [\n \"I met a traveller from an antique land\",\n \"Who said: \\\"Two vast and trunkless legs of stone\",\n \"Stand in the desert. Near them on the sand,\",\n \"...\"\n ],\n \"linecount\": \"14\"\n }\n]\n```\n\nSources:\n[1] [poetrydb.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG24c52U7ZS3jpojxAdN-Y0QdMF6fwGMERPxMLXMA8fbmGHvXAHCKNbX7PAnee1DSH6Z5mqxg6MpvrsCL1Tcr5ML241IoP83LE0wm8=)\n[2] [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHTozV11Vt0k6lC8e2erFt5OGugSvypfOxy2IBPYSyStb9qJHg7558eAOAhlz_Ot1nzP8llrxemBG3xu70i4bMUCow6hdB6GnY9KJLIfb9KbmPigi1y9OCGqW_nDPpbmg9t)\n[3] [readthedocs.io](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHlvrGOnlBTduMx9foDJj3S6MkPsPFwfYf1Lq50Kexvpaov3tH5hnzEgKluJCB8iYl4qxT1MSbpnmIpFqOC4yEhRUTOmmDZBXSwbTk3c_bMwJE3iSA=)\n[4] [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF1paZKnAGUF0XOef0V4JUba4QTLiJwYa1wCbozh1eWE7ROxcxZQSvmP7_LjYV6nNZCHYwZ3NwdkUFwHcmriv-p7-KkDzOzxlJvq9shuHm7Dl0VlNtdU9NxlHt8UdE=)\n[5] [publicapi.dev](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGIPb4pw0anlsSDMz-eIioaLO3InBxlwtyKgeBQc7-Ct-5Uib3oYm1O0CReIhety6jmKxYn8pyBP5wpXlqQoZu4hGaYSchZ-FRmo_vUNwOuMG6TQBqgca-LzA==)\n[6] [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGj5W3sAycILtM8QAwAul_Z4q2m7Yi9UwBsh-P2ZKABnrwLT4l82qhacP2Ape_wu0p7cYOkyVCx0EKASvZXFx5BWQZ0CpfWaUwgEZ0ihGuMGYy5k865YLxU4k9Gkz8=)\n[7] [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF1paZKnAGUF0XOef0V4JUba4QTLiJwYa1wCbozh1eWE7ROxcxZQSvmP7_LjYV6nNZCHYwZ3NwdkUFwHcmriv-p7-KkDzOzxlJvq9shuHm7Dl0VlNtdU9NxlHt8UdE=)\n[8] [hoshime.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF4FEOJ0cCOJu-OGOFCNi3qA_NaZ3yow6WuJFO-BPu2FDADR-y2BTEHIIxcvhtIHMiTAsv_4bpoiqmlvHUMdOuIQjYJa0CwEhXz9HhLXqB_Y5RSjiLJ34_ww9F3R19gJI6t)\n[9] [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFLBnie1xAVT6hQ6Ugd3FeE1imXNk3mG1Low30yA9RdHlWoYSvM2DBXuy6hJfxNwpIhhWCuuD5dF8SqXCnSE64AwAiNIIgb8Eo9w1rdV1IDihGY607qx0kanrB737vB_ip2EIqDLE6uHbKSHWpFVn8nX7Dx)\n[10] [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF1paZKnAGUF0XOef0V4JUba4QTLiJwYa1wCbozh1eWE7ROxcxZQSvmP7_LjYV6nNZCHYwZ3NwdkUFwHcmriv-p7-KkDzOzxlJvq9shuHm7Dl0VlNtdU9NxlHt8UdE=)\n[11] [rapidapi.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFtZw8OWW846HV4w0jGEYSf5CkYFzcSzeYScSKy9jhczIRofJ_DUTH6eM5hDA_EwnSyfZW_b9SOvV0hOPkONm6GC_P5GA0gU7hH9ibcvqEkzRTvBxfKfZCzRWuViyJnxTlb4hPK2i-wgyKz21MIhvMA3K8iLpQzjmbIozU9OXXNvhcmUrie-fyl)\n\nNote: Source URLs are HTTP redirects; check where they resolve if you need the direct link."} {"step_index":13,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:18:32Z","model":"claude-sonnet-4-6","input_tokens":3810,"cache_read_tokens":23166,"output_tokens":2794,"tool_calls":[{"name":"edit_file","args":{"ArtifactMetadata":{"RequestFeedback":false,"Summary":"Updated plan for in-class RLHF demo. Incorporates user feedback: PoetryDB REST API (not HuggingFace dataset), neutral system prompt (no sentiment instruction), structural line-count bonus in reward, live training in class, instructor-led demo only.","UserFacing":true},"Path":"/home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md","Replacement":"# RLHF In-Class Demo Plan: Sentiment-Steered Poem Completion with GRPO\n\n> [!IMPORTANT]\n> **Goal:** Fine-tune Llama 3.2 (1B) with GRPO to complete poems positively, using a pretrained sentiment model as the reward signal — no reward model training required. Demo is instructor-led; students observe. Live training runs during class (~20 min). Demo delivered as a Jupyter notebook.\n\n---\n\n## 1. Overview\n\n```mermaid\nflowchart LR\n A[\"PoetryDB API\\npoetrbydb.org\"] --> B[\"Prompt builder\\nlines 8–15 → neutral prompt\"]\n B --> C[\"Base Llama 3.2 1B\\n(reference policy)\"]\n B --> D[\"GRPO Training Loop\\nlive in class\"]\n E[\"Sentiment model\\n(DistilBERT SST-2)\"] --> D\n D --> F[\"Tuned Policy\\n(beta=0.04 & beta=0.4)\"]\n C --> G[\"Notebook Demo\\nSide-by-side comparison\"]\n F --> G\n```\n\n**Core choices:**\n| Component | Choice | Rationale |\n|---|---|---|\n| Dataset | PoetryDB REST API (`poetrydb.org`) | No auth required, JSON responses, filter by `linecount` |\n| Base model | `meta-llama/Llama-3.2-1B-Instruct` | Small enough for in-class GPU, strong enough to generate coherent poetry |\n| RL algorithm | GRPO via HuggingFace TRL `GRPOTrainer` | No separate reward model needed; stable, group-based advantage estimation |\n| Reward signal | `distilbert-base-uncased-finetuned-sst-2-english` + line-count bonus | Fast inference, well-calibrated positive/negative scores |\n| KL penalty β | Two runs: **β = 0.04** (aggressive) vs **β = 0.4** (conservative) | Shows the exploration-vs-fidelity tradeoff live |\n\n---\n\n## 2. Repository / File Structure\n\n```\nrlhf-s2/\n├── data/\n│ └── poem_loader.py # PoetryDB API fetch & prompt construction\n├── training/\n│ ├── reward.py # Sentiment + structure reward function\n│ ├── train_grpo.py # Script-based GRPO trainer (beta param CLI arg)\n│ └── configs/\n│ ├── grpo_beta_low.yaml\n│ └── grpo_beta_high.yaml\n├── notebooks/\n│ └── rlhf_poem_demo.ipynb # ← MAIN DEMO NOTEBOOK\n├── checkpoints/ # saved model weights (gitignored)\n└── requirements.txt\n```\n\n---\n\n## 3. Data Pipeline\n\n### 3a. Dataset Loading via PoetryDB API\n\nPoetryDB (`https://poetrydb.org`) is a free, no-auth REST API returning JSON poem objects with `title`, `author`, `lines`, and `linecount`.\n\n**Strategy:** Fetch all poems with `linecount` ≥ 12 (guarantees enough context lines). Use the `linecount` endpoint or bulk-fetch and filter locally.\n\n```python\nimport requests\n\ndef fetch_poems(min_lines: int = 12) -> list[dict]:\n \"\"\"Fetch poems from PoetryDB with at least min_lines lines.\"\"\"\n # Bulk fetch a large random sample and filter\n url = \"https://poetrydb.org/random/200\"\n poems = requests.get(url).json()\n return [p for p in poems if int(p[\"linecount\"]) >= min_lines]\n\n# Or fetch all poems of a specific linecount range:\ndef fetch_poems_by_linecount(min_lc: int = 12, max_lc: int = 50) -> list[dict]:\n poems = []\n for lc in range(min_lc, max_lc + 1):\n resp = requests.get(f\"https://poetrydb.org/linecount/{lc}\")\n if resp.status_code == 200:\n poems.extend(resp.json())\n return poems\n```\n\n> [!NOTE]\n> PoetryDB has ~3,000 poems in its database. Filtering to ≥ 12 lines typically yields ~800–1,000 usable poems — plenty for training and demo.\n\n### 3b. Prompt Construction (`poem_loader.py`)\n\nEach training example:\n1. Split `poem[\"lines\"]` (already a list of strings in PoetryDB JSON)\n2. Sample 8–15 context lines randomly during training; fixed window during demo\n3. Format as a **neutral** chat-style prompt — no sentiment instruction. The goal of RLHF is for the reward signal alone to steer behavior.\n\n```python\nimport random\n\ndef build_prompt(poem: dict, n_context: int | None = None) -> list[dict]:\n lines = poem[\"lines\"]\n if n_context is None:\n n_context = random.randint(8, min(15, len(lines) - 4))\n context = \"\\n\".join(lines[:n_context])\n return [\n {\n \"role\": \"system\",\n \"content\": \"You are a poet. Complete the following poem with exactly 4 lines.\"\n # ← NO sentiment instruction: steering comes from the reward signal only\n },\n {\n \"role\": \"user\",\n \"content\": f\"{context}\"\n }\n ]\n```\n\n> [!IMPORTANT]\n> The system prompt must remain **sentiment-neutral**. Telling the model to \"be positive\" or \"be uplifting\" in the prompt defeats the entire point of RLHF — the whole demo would just be prompt engineering. The sentiment shift must be learned from the reward signal alone.\n\n**Output format constraint:** The reward function penalizes completions that are not ≈ 4 lines (see Section 4).","Target":"# RLHF In-Class Demo Plan: Sentiment-Steered Poem Completion with GRPO\n\n> [!IMPORTANT]\n> **Goal:** Fine-tune Llama 3.2 (1B) with GRPO to complete poems positively, using a pretrained sentiment model as the reward signal — no reward model training required. Demo delivered as a live Jupyter notebook.\n\n---\n\n## 1. Overview\n\n```mermaid\nflowchart LR\n A[\"PoemDB\\n(Poetry Foundation)\"] --> B[\"Prompt builder\\nlines 8–15 → prompt\"]\n B --> C[\"Base Llama 3.2 1B\\n(reference policy)\"]\n B --> D[\"GRPO Training Loop\"]\n E[\"Sentiment model\\n(DistilBERT SST-2)\"] --> D\n D --> F[\"Tuned Policy\\n(beta=0.1 & beta=0.5)\"]\n C --> G[\"Notebook Demo\\nSide-by-side comparison\"]\n F --> G\n```\n\n**Core choices:**\n| Component | Choice | Rationale |\n|---|---|---|\n| Dataset | `merve/poetry` or `suayptalha/Poetry-Foundation-Poems` | Freely available, rich metadata |\n| Base model | `meta-llama/Llama-3.2-1B-Instruct` | Small enough for in-class GPU, strong enough to generate coherent poetry |\n| RL algorithm | GRPO via HuggingFace TRL `GRPOTrainer` | No separate reward model needed; stable, group-based advantage estimation |\n| Reward signal | `distilbert-base-uncased-finetuned-sst-2-english` | Fast inference, well-calibrated positive/negative scores |\n| KL penalty β | Two runs: **β = 0.04** (aggressive) vs **β = 0.4** (conservative) | Shows the exploration-vs-fidelity tradeoff live |\n\n---\n\n## 2. Repository / File Structure\n\n```\nrlhf-s2/\n├── data/\n│ └── poem_loader.py # Dataset loading & prompt construction\n├── training/\n│ ├── reward.py # Sentiment reward function\n│ ├── train_grpo.py # Script-based GRPO trainer (beta param CLI arg)\n│ └── configs/\n│ ├── grpo_beta_low.yaml\n│ └── grpo_beta_high.yaml\n├── notebooks/\n│ └── rlhf_poem_demo.ipynb # ← MAIN DEMO NOTEBOOK\n├── checkpoints/ # saved model weights (gitignored)\n└── requirements.txt\n```\n\n---\n\n## 3. Data Pipeline\n\n### 3a. Dataset Loading\n\nUse `merve/poetry` (HuggingFace) — ~573 poems with title, author, and content.\n\n```python\nfrom datasets import load_dataset\nds = load_dataset(\"merve/poetry\", split=\"train\")\n```\n\n> [!NOTE]\n> `suayptalha/Poetry-Foundation-Poems` is richer (~14k poems) and includes tags like \"joy\", \"sadness\" — useful for filtering to poems that have *room* for a positive turn. Either works; use `merve/poetry` for fast demo startup.\n\n### 3b. Prompt Construction (`poem_loader.py`)\n\nEach training example:\n1. Split poem text by newline\n2. Keep only poems with ≥ 12 lines (guarantees 8–15 lines of context)\n3. Sample a window of 8–15 lines (randomly during training, fixed during demo)\n4. Format as a chat-style prompt:\n\n```python\ndef build_prompt(poem_lines: list[str], n_context: int) -> list[dict]:\n context = \"\\n\".join(poem_lines[:n_context])\n return [\n {\n \"role\": \"system\",\n \"content\": (\n \"You are a poet. Complete the following poem with exactly 4 lines. \"\n \"Your completion should be uplifting and positive in sentiment.\"\n )\n },\n {\n \"role\": \"user\",\n \"content\": f\"Here is the beginning of a poem:\\n\\n{context}\\n\\nWrite the next 4 lines:\"\n }\n ]\n```\n\n**Output format constraint:** Reward function also penalizes completions that are not exactly 4 lines (optional length reward component)."}}]} {"step_index":14,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:19:18Z","content":"Created At: 2026-11-20T12:19:18-05:00\nCompleted At: 2026-11-20T12:19:18-05:00\nCreated At: 2026-11-20T12:19:18-05:00\nCompleted At: 2026-11-20T12:19:18-05:00\nMade 1 replacement in /home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md starting at line 1 (+55/-33 lines)."} {"step_index":15,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:19:18Z","model":"claude-sonnet-4-6","input_tokens":5961,"cache_read_tokens":23942,"output_tokens":997,"content":"Now update the reward section to add the structural bonus, and update training/demo sections for the live-training + instructor-led changes:","tool_calls":[{"name":"edit_file","args":{"Path":"/home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md","Replacement":"## 4. Reward Function (`training/reward.py`)\n\nThe reward is a **weighted sum** of two components:\n\n| Component | Weight | Signal |\n|---|---|---|\n| Sentiment score | 1.0 | DistilBERT SST-2 confidence in POSITIVE class |\n| Line-count bonus | 0.2 | +1 if completion has exactly 3–5 newlines (≈ 4 lines), else 0 |\n\n```python\nfrom transformers import pipeline\n\n_sentiment = pipeline(\n \"sentiment-analysis\",\n model=\"distilbert-base-uncased-finetuned-sst-2-english\",\n device=0 # GPU\n)\n\ndef _line_count_bonus(text: str) -> float:\n \"\"\"Return 1.0 if text has 3–5 newlines (roughly 4 lines), else 0.0.\"\"\"\n n = text.count(\"\\n\")\n return 1.0 if 3 <= n <= 5 else 0.0\n\ndef reward_fn(completions: list[list[dict]], **kwargs) -> list[float]:\n texts = [c[0][\"content\"] for c in completions]\n results = _sentiment(texts, truncation=True, max_length=128)\n rewards = []\n for text, r in zip(texts, results):\n score = r[\"score\"]\n sentiment = score if r[\"label\"] == \"POSITIVE\" else score - 1.0\n structure = _line_count_bonus(text)\n rewards.append(sentiment + 0.2 * structure)\n return rewards\n```\n\n> [!TIP]\n> You can swap in `cardiffnlp/twitter-roberta-base-sentiment-latest` for a 3-class model (NEG / NEU / POS) if you want finer-grained scoring. It runs in ~50ms per batch on GPU.\n\n> [!IMPORTANT]\n> The structural bonus is pedagogically useful: it shows students that GRPO can simultaneously optimize *multiple* reward signals, and that reward engineering is an active design choice — not just \"use a sentiment model\".","Target":"## 4. Reward Function (`training/reward.py`)\n\n```python\nfrom transformers import pipeline\n\n_sentiment = pipeline(\n \"sentiment-analysis\",\n model=\"distilbert-base-uncased-finetuned-sst-2-english\",\n device=0 # GPU\n)\n\ndef sentiment_reward(completions: list[list[dict]], **kwargs) -> list[float]:\n texts = [c[0][\"content\"] for c in completions]\n results = _sentiment(texts, truncation=True, max_length=128)\n rewards = []\n for r in results:\n score = r[\"score\"]\n if r[\"label\"] == \"POSITIVE\":\n rewards.append(score) # [0, 1]\n else:\n rewards.append(score - 1.0) # [-1, 0]\n return rewards\n```\n\n**Optional bonus reward:** Add `+0.1` if the completion has 3–5 newlines (≈ 4 lines). Keeps outputs structurally poetic.\n\n> [!TIP]\n> You can swap in `cardiffnlp/twitter-roberta-base-sentiment-latest` for a 3-class model (NEG / NEU / POS) if you want finer-grained scoring. It runs in ~50ms per batch on GPU."}}]} {"step_index":16,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:19:33Z","content":"Created At: 2026-11-20T12:19:33-05:00\nCompleted At: 2026-11-20T12:19:33-05:00\nCreated At: 2026-11-20T12:19:33-05:00\nCompleted At: 2026-11-20T12:19:33-05:00\nMade 1 replacement in /home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md starting at line 118 (+20/-8 lines)."} {"step_index":17,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:19:33Z","model":"claude-sonnet-4-6","input_tokens":1263,"cache_read_tokens":29770,"output_tokens":380,"tool_calls":[{"name":"edit_file","args":{"Path":"/home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md","Replacement":"### Training Time Estimate (1× A100 / L4 GPU)\n- Dataset: ~200 filtered poems → ~200 steps per epoch\n- Each step: 2 prompts × 8 generations × 128 tokens ≈ **~10–15 min per run**\n- Two runs total: **~25–30 min** — both run **live during class** while the notebook streams curves in real time\n\n> [!NOTE]\n> Run both β settings sequentially during class: start low-β first (~10 min), then high-β. The notebook polls the trainer's log file and updates plots live every N steps, so the audience watches reward climb in real time.","Target":"### Training Time Estimate (1× A100 / L4 GPU)\n- Dataset: ~200 filtered poems → ~200 steps per epoch\n- Each step: 2 prompts × 8 generations × 128 tokens ≈ **~8–12 min per run**\n- Two runs total: **~20–25 min** — run both before class, checkpoint saved"}}]} {"step_index":18,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:19:40Z","content":"Created At: 2026-11-20T12:19:40-05:00\nCompleted At: 2026-11-20T12:19:40-05:00\nCreated At: 2026-11-20T12:19:40-05:00\nCompleted At: 2026-11-20T12:19:40-05:00\nMade 1 replacement in /home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md starting at line 195 (+5/-2 lines)."} {"step_index":19,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:19:40Z","model":"claude-sonnet-4-6","input_tokens":646,"cache_read_tokens":30900,"output_tokens":1897,"tool_calls":[{"name":"edit_file","args":{"Path":"/home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md","Replacement":"## 6. Jupyter Notebook Structure (`rlhf_poem_demo.ipynb`)\n\n> [!NOTE]\n> This is an **instructor-led live demo**. Students observe; no student GPU access needed. The notebook is the presentation surface — no slides.\n\n### Section 0 — Setup & Imports\n```python\n# Install / imports, load tokenizer, base model, and sentiment pipeline\n# Checkpoints are loaded lazily after training completes\n```\n\n### Section 1 — RLHF Concept Explainer *(markdown cells)*\n- What is RLHF? PPO → GRPO comparison diagram\n- Why skip reward model training? → pretrained sentiment model as proxy\n- GRPO advantage: group relative scoring (formula cell with LaTeX)\n\n$$A_i = \\frac{r_i - \\text{mean}(\\mathbf{r})}{\\text{std}(\\mathbf{r})} \\quad \\text{where } \\mathbf{r} = [r_1, \\ldots, r_G]$$\n\n- Key point: **the system prompt is neutral** — RLHF is what does the work\n\n### Section 2 — Dataset Explorer (Interactive)\n- Widget: slider to pick a poem index (fetched from PoetryDB)\n- Display: title, author, full text, highlighted prompt lines (lines 1–N in one color, remaining lines greyed out)\n- Show distribution of poem lengths (histogram)\n\n### Section 3 — Reward Function Demo\n- Input any text → get sentiment score + line-count bonus live\n- Show score on 5 hand-picked poetry lines (positive vs gloomy)\n- Builds intuition for the reward signal before training begins\n\n### Section 4 — 🔴 Live Training (runs during class)\n\n```python\n# Cell that launches GRPOTrainer for beta=0.04, streams logs\ntrainer_low = build_trainer(beta=0.04)\n\n# Live plot: updates every logging_steps using a callback + IPython display clear\nfrom IPython.display import clear_output\n\nclass LivePlotCallback(TrainerCallback):\n def on_log(self, args, state, control, logs=None, **kwargs):\n clear_output(wait=True)\n plot_training_curves(state.log_history)\n```\n\n- Plot updates every 5 steps: **mean reward vs step** and **KL divergence vs step**\n- After low-β finishes, instructor narrates what happened, then runs high-β cell\n- Side-by-side comparison chart appears automatically when both runs complete\n\n### Section 5 — Poem Sampler (The Main Demo Cell)\n\nInteractive widget with:\n- **Dropdown:** pick a poem (by title, fetched from PoetryDB)\n- **Slider:** choose number of context lines (8–15)\n- **Button:** \"Generate completions\"\n\nOutputs rendered side-by-side:\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ PROMPT (lines 1–N of \"[Title]\" by [Author]): │\n│ [poem text] │\n├──────────────────┬───────────────────┬──────────────────────────┤\n│ Base Model │ Tuned (β=0.04) │ Tuned (β=0.4) │\n│ [completion] │ [completion] │ [completion] │\n│ Sentiment: 0.42 │ Sentiment: 0.89 │ Sentiment: 0.71 │\n└──────────────────┴───────────────────┴──────────────────────────┘\n```\n\n### Section 6 — Quantitative Comparison\n- Run 20 random poems through all 3 models\n- Bar chart: mean sentiment score (Base vs β=0.04 vs β=0.4)\n- Scatter: sentiment score vs \"perplexity\" (measured by reference model) → shows the coherence/sentiment tradeoff","Target":"## 6. Jupyter Notebook Structure (`rlhf_poem_demo.ipynb`)\n\n### Section 0 — Setup & Imports\n```python\n# Install / imports, load tokenizer & models\n# Load both checkpoints (beta_low, beta_high) + base model\n```\n\n### Section 1 — RLHF Concept Explainer *(markdown cells)*\n- What is RLHF? PPO → GRPO comparison diagram (image or mermaid)\n- Why skip reward model training? → pretrained sentiment model as proxy\n- GRPO advantage: group relative scoring (formula cell with LaTeX)\n\n$$A_i = \\frac{r_i - \\text{mean}(\\mathbf{r})}{\\text{std}(\\mathbf{r})} \\quad \\text{where } \\mathbf{r} = [r_1, \\ldots, r_G]$$\n\n### Section 2 — Dataset Explorer (Interactive)\n- Widget: slider to pick a poem index\n- Display: title, author, full text, highlighted prompt lines\n- Show distribution of poem lengths (histogram)\n- Show class balance (sentiment of first stanzas vs last stanzas)\n\n### Section 3 — Reward Model Demo\n- Input any text → get sentiment score live\n- Show score on 5 poetry lines (positive vs negative)\n- Builds intuition for the reward signal\n\n### Section 4 — Live Training Curves\n- Load TensorBoard logs (or CSV from trainer state)\n- Plot: **mean reward vs step** for both β runs on same axes\n- Plot: **KL divergence vs step** for both runs\n- Plot: **reward std within group** (shows how spread collapses as model improves)\n\n```python\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfig, axes = plt.subplots(1, 3, figsize=(15, 4))\n# reward curves, KL curves, group reward std\n```\n\n### Section 5 — Poem Sampler (The Main Demo Cell)\n\nInteractive widget with:\n- **Dropdown:** pick a poem (by title)\n- **Slider:** choose number of context lines (8–15)\n- **Button:** \"Generate completions\"\n\nOutputs rendered side-by-side:\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ PROMPT (lines 1–N): │\n│ [poem text] │\n├──────────────────┬───────────────────┬──────────────────────────┤\n│ Base Model │ Tuned (β=0.04) │ Tuned (β=0.4) │\n│ [completion] │ [completion] │ [completion] │\n│ Sentiment: 0.42 │ Sentiment: 0.89 │ Sentiment: 0.71 │\n└──────────────────┴───────────────────┴──────────────────────────┘\n```\n\n### Section 6 — Quantitative Comparison\n- Run 20 random poems through all 3 models\n- Bar chart: mean sentiment score (Base vs β=0.04 vs β=0.4)\n- Scatter: sentiment score vs \"perplexity\" (measured by reference model) → shows the coherence/sentiment tradeoff"}}]} {"step_index":20,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:20:11Z","content":"Created At: 2026-11-20T12:20:11-05:00\nCompleted At: 2026-11-20T12:20:11-05:00\nCreated At: 2026-11-20T12:20:11-05:00\nCompleted At: 2026-11-20T12:20:11-05:00\nMade 1 replacement in /home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md starting at line 205 (+36/-27 lines)."} {"step_index":21,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:20:11Z","model":"claude-sonnet-4-6","input_tokens":2163,"cache_read_tokens":31413,"output_tokens":670,"tool_calls":[{"name":"edit_file","args":{"Path":"/home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md","Replacement":"## 8. In-Class Demo Flow (suggested ~55 min)\n\n| Time | Activity |\n|---|---|\n| 0–5 min | Intro: What is RLHF, why GRPO? Walk through Section 1 |\n| 5–10 min | Dataset explorer (Section 2) — fetch poems live from PoetryDB, pick a few, show neutral prompt format |\n| 10–15 min | Reward function live demo (Section 3) — type examples, see sentiment scores + structure bonus |\n| 15–28 min | 🔴 **Launch low-β training live** — curves update every 5 steps; instructor narrates what's happening |\n| 28–30 min | Checkpoint saved; discuss what the reward curve shape tells us |\n| 30–43 min | 🔴 **Launch high-β training live** — compare curves as it runs |\n| 43–45 min | Show side-by-side training curve comparison (Section 4) |\n| 45–55 min | **Live poem sampler** (Section 5) — pick poems, run inference, compare 3 models + Q&A |\n\n> [!TIP]\n> Pre-select 3–5 \"showcase\" poems before class: ones where the base model gives a neutral/melancholy ending and the tuned model gives a noticeably brighter one. Have their indices ready to type into the widget.","Target":"## 8. In-Class Demo Flow (suggested ~45 min)\n\n| Time | Activity |\n|---|---|\n| 0–5 min | Intro: What is RLHF, why GRPO? Walk through Section 1 |\n| 5–10 min | Dataset explorer (Section 2) — pick a few poems, show sentiment of opening vs closing stanzas |\n| 10–15 min | Reward function live demo (Section 3) — type examples, see scores |\n| 15–20 min | Show pre-recorded training curves (Section 4) — discuss β tradeoff |\n| 20–35 min | **Live poem sampler** (Section 5) — pick poems, run inference, compare 3 models |\n| 35–45 min | Quantitative results (Section 6) + Q&A |"}}]} {"step_index":22,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:20:22Z","content":"Created At: 2026-11-20T12:20:22-05:00\nCompleted At: 2026-11-20T12:20:23-05:00\nCreated At: 2026-11-20T12:20:22-05:00\nCompleted At: 2026-11-20T12:20:23-05:00\nMade 1 replacement in /home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md starting at line 307 (+11/-6 lines)."} {"step_index":23,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:20:23Z","model":"claude-sonnet-4-6","input_tokens":936,"cache_read_tokens":33443,"output_tokens":495,"tool_calls":[{"name":"edit_file","args":{"Path":"/home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md","Replacement":"## 10. Open Questions / Decisions\n\n1. **Model gating:** Use gated `meta-llama/Llama-3.2-1B-Instruct` (need HF token) or ungated `HuggingFaceTB/SmolLM2-1.7B-Instruct`?\n2. **Sentiment model:** Keep DistilBERT SST-2, or use `cardiffnlp/twitter-roberta-base-sentiment-latest` (3-class: NEG/NEU/POS)?\n3. **Line-count bonus weight:** 0.2 feels reasonable — want it higher or lower relative to the sentiment signal?","Target":"## 10. Open Questions / Decisions\n\n1. **GPU access:** Will students run inference themselves, or is this a live instructor demo?\n2. **Model gating:** Use gated `meta-llama/Llama-3.2-1B-Instruct` or ungated `SmolLM2-1.7B-Instruct`?\n3. **Training environment:** Pre-train checkpoints before class, or show training live (needs ~20 min in class)?\n4. **Line-count reward:** Add a structural bonus (penalize completions ≠ 4 lines) or keep reward purely sentiment-based?\n5. **β values:** Do `0.04` and `0.4` feel right for the tradeoff you want to show, or prefer a more dramatic gap (e.g., `0.01` vs `1.0`)?"}}]} {"step_index":24,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:20:32Z","content":"Created At: 2026-11-20T12:20:32-05:00\nCompleted At: 2026-11-20T12:20:32-05:00\nCreated At: 2026-11-20T12:20:32-05:00\nCompleted At: 2026-11-20T12:20:32-05:00\nMade 1 replacement in /home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md starting at line 360 (+3/-5 lines)."} {"step_index":25,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:20:32Z","model":"claude-sonnet-4-6","input_tokens":761,"cache_read_tokens":34246,"output_tokens":400,"content":"Plan updated → [rlhf-demo-plan.md](file:///home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md)\n\nHere's a summary of every change made:\n\n| Feedback | Change |\n|---|---|\n| **Use PoetryDB** | Data pipeline now uses `https://poetrydb.org` REST API — no HuggingFace dataset. Fetches JSON by `linecount`, no auth required |\n| **Neutral prompt** | System prompt is `\"You are a poet. Complete the following poem with exactly 4 lines.\"` — zero sentiment instruction. GRPO + reward signal does all the steering |\n| **Structural bonus** | Reward is now `sentiment + 0.2 × line_count_bonus` — bonus is 1.0 if the completion has 3–5 newlines (≈ 4 lines) |\n| **Show live training** | Demo flow restructured to 55 min — both β runs fire live during class with a `TrainerCallback` that re-renders the plot every 5 steps |\n| **Instructor-led** | Removed all student GPU/inference setup; notebook is the presentation surface |\n| **β values** | Confirmed `0.04` / `0.4` |\n\nTwo remaining open questions at the bottom of the plan: **model gating** (Llama vs SmolLM2) and **sentiment model choice** (DistilBERT vs RoBERTa 3-class). Ready to build the code whenever you are — just say the word!"} {"step_index":26,"source":"USER_EXPLICIT","type":"USER_INPUT","status":"DONE","created_at":"2026-11-20T17:21:31Z","content":"Comments on artifact URI: file:///home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/rlhf-demo-plan.md\n\nSelection:\n>Model gating: Use gated meta-llama/Llama-3.2-1B-Instruct (need HF token) or ungated HuggingFaceTB/SmolLM2-1.7B-Instruct?\n\nComment: \"I have access to this model. Use newGPU env\"\n\nSelection:\n>Sentiment model: Keep DistilBERT SST-2, or use cardiffnlp/twitter-roberta-base-sentiment-latest (3-class: NEG/NEU/POS)?\n\nComment: \"keep 2 class\"\n\nSelection:\n>Line-count bonus weight: 0.2 feels reasonable — want it higher or lower relative to the sentiment signal?\n\nComment: \"sure sounds good.\"\n\n\n\n\nThe current local time is: 2026-11-20T12:21:31-05:00.\n"} {"step_index":27,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:21:31Z","model":"claude-sonnet-4-6","input_tokens":838,"cache_read_tokens":34874,"output_tokens":219,"content":"Decisions locked in. Let me check that environment before building.","thinking":"The user has confirmed the remaining decisions and mentioned \"Use newGPU env\" for the model. Let me check what that means - they likely have a `newGPU` conda environment or similar. Let me check the workspace for environment details, then start building the actual code.","tool_calls":[{"name":"run_command","args":{"CommandLine":"conda env list 2>/dev/null; echo \"---\"; ls ~/.conda/envs/ 2>/dev/null; echo \"---\"; which python; python --version","WaitMsBeforeAsync":8000,"toolSummary":"Check conda envs"}}]} {"step_index":28,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:21:36Z","content":"Created At: 2026-11-20T12:21:36-05:00\nCompleted At: 2026-11-20T12:21:51-05:00\nCreated At: 2026-11-20T12:21:36-05:00\nCompleted At: 2026-11-20T12:21:51-05:00\nCommand exited with code 0.\n\nCommand output:\n\r\n# conda environments:\r\n#\r\nbase * /home/canders1/miniforge3\r\nMapSAM /home/canders1/miniforge3/envs/MapSAM\r\nnewGPU /home/canders1/miniforge3/envs/newGPU\r\n /opt/anaconda3/envs/agnostics\r\n /opt/anaconda3/envs/booknlp2\r\n /opt/anaconda3/envs/env38\r\n /opt/anaconda3/envs/experiments\r\n /opt/anaconda3/envs/gpu-env\r\n /opt/anaconda3/envs/gpu_11\r\n /opt/anaconda3/envs/imageEnv\r\n /opt/anaconda3/envs/llava\r\n /opt/anaconda3/envs/p311\r\n /opt/anaconda3/envs/vllm\r\n /opt/anaconda3/envs/vpolyglot\r\n /opt/anaconda3/envs/yiru\r\n\r\n---\r\n---\r\n/home/canders1/miniforge3/bin/python\r\nPython 3.12.10\r\n"} {"step_index":29,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:21:51Z","model":"claude-sonnet-4-6","input_tokens":964,"cache_read_tokens":35404,"output_tokens":165,"tool_calls":[{"name":"run_command","args":{"CommandLine":"conda run -n newGPU python --version && conda run -n newGPU pip list 2>/dev/null | grep -iE \"torch|transformers|trl|peft|datasets|accelerate|bitsandbytes|ipywidgets|jupyter\"","WaitMsBeforeAsync":15000,"toolSummary":"Check newGPU packages"}}]} {"step_index":30,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:21:53Z","content":"Created At: 2026-11-20T12:21:53-05:00\nCompleted At: 2026-11-20T12:22:07-05:00\nCreated At: 2026-11-20T12:21:53-05:00\nCompleted At: 2026-11-20T12:22:07-05:00\nCommand exited with code 0.\n\nCommand output:\nPython 3.12.11\r\n\r\naccelerate 1.12.0\r\ndatasets 5.1.0\r\nfastrlock 0.8.3\r\nipywidgets 8.1.8\r\njupyter_client 8.6.3\r\njupyter_core 5.9.1\r\njupyter-events 0.12.0\r\njupyter-lsp 2.3.0\r\njupyter_server 2.17.0\r\njupyter_server_terminals 0.5.3\r\njupyterlab 4.4.10\r\njupyterlab_pygments 0.3.0\r\njupyterlab_server 2.28.0\r\njupyterlab_widgets 3.0.16\r\npeft 0.22.0\r\ntorch 2.8.0\r\ntorchaudio 2.8.0\r\ntorchvision 0.23.0\r\ntransformers 5.20.1\r\ntrl 1.18.1\r\n"} {"step_index":31,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:22:07Z","model":"claude-sonnet-4-6","input_tokens":985,"cache_read_tokens":35931,"output_tokens":207,"tool_calls":[{"name":"run_command","args":{"CommandLine":"conda run -n newGPU python -c \"import torch; print('CUDA:', torch.cuda.is_available()); print('GPUs:', torch.cuda.device_count()); [print(f' GPU {i}:', torch.cuda.get_device_name(i), f'({torch.cuda.get_device_properties(i).total_memory//1024**3}GB)') for i in range(torch.cuda.device_count())]\"","WaitMsBeforeAsync":15000,"toolSummary":"Check GPU availability"}}]} {"step_index":32,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:22:10Z","content":"Created At: 2026-11-20T12:22:10-05:00\nCompleted At: 2026-11-20T12:22:18-05:00\nCreated At: 2026-11-20T12:22:10-05:00\nCompleted At: 2026-11-20T12:22:18-05:00\nCommand exited with code 0.\n\nCommand output:\nCUDA: True\r\nGPUs: 2\r\n GPU 0: NVIDIA RTX A6000 (47GB)\r\n GPU 1: NVIDIA RTX A6000 (47GB)\r\n\r\n"} {"step_index":33,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:22:18Z","model":"claude-sonnet-4-6","input_tokens":729,"cache_read_tokens":36533,"output_tokens":163,"content":"2× A6000 (47GB each) — plenty of headroom, no quantization needed. Now I have everything. Let me build all the code files.","tool_calls":[{"name":"run_command","args":{"CommandLine":"mkdir -p /home/canders1/rlhf-s2/{data,training/configs,notebooks,checkpoints}","WaitMsBeforeAsync":5000,"toolSummary":"Create project dirs"}}]} {"step_index":34,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:22:22Z","content":"Created At: 2026-11-20T12:22:22-05:00\nCompleted At: 2026-11-20T12:22:30-05:00\nCreated At: 2026-11-20T12:22:22-05:00\nCompleted At: 2026-11-20T12:22:30-05:00\nCommand exited with code 0.\n\nNo output from the command."} {"step_index":35,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:22:30Z","model":"claude-sonnet-4-6","input_tokens":379,"cache_read_tokens":37123,"output_tokens":1377,"tool_calls":[{"name":"write_file","args":{"Content":"\"\"\"\npoem_loader.py\n--------------\nFetch poems from the PoetryDB REST API (https://poetrydb.org) and\nconvert them into GRPO-ready prompt dicts.\n\nPoetryDB returns JSON with fields:\n title : str\n author : str\n lines : list[str]\n linecount : str (note: string, not int)\n\"\"\"\n\nimport random\nimport requests\nfrom datasets import Dataset\n\nPOETRYDB_BASE = \"https://poetrydb.org\"\n\n\n# ---------------------------------------------------------------------------\n# Fetching\n# ---------------------------------------------------------------------------\n\ndef fetch_all_poems(min_lines: int = 12, max_poems: int = 2000) -> list[dict]:\n \"\"\"\n Fetch up to `max_poems` poems from PoetryDB with >= min_lines lines.\n Uses the /random endpoint in batches to avoid a full DB dump.\n Falls back to linecount range queries if random endpoint returns few results.\n \"\"\"\n poems = []\n seen_titles = set()\n\n # Strategy 1: random batches\n for _ in range(max_poems // 50 + 1):\n resp = requests.get(f\"{POETRYDB_BASE}/random/50\", timeout=15)\n if resp.status_code != 200:\n break\n batch = resp.json()\n if isinstance(batch, dict): # error response\n break\n for p in batch:\n title = p.get(\"title\", \"\")\n if title in seen_titles:\n continue\n if int(p.get(\"linecount\", 0)) >= min_lines:\n seen_titles.add(title)\n poems.append(p)\n if len(poems) >= max_poems:\n break\n\n # Strategy 2: linecount range queries (supplement if needed)\n if len(poems) < 100:\n for lc in range(min_lines, 60):\n resp = requests.get(f\"{POETRYDB_BASE}/linecount/{lc}\", timeout=15)\n if resp.status_code != 200:\n continue\n data = resp.json()\n if isinstance(data, dict):\n continue\n for p in data:\n title = p.get(\"title\", \"\")\n if title not in seen_titles:\n seen_titles.add(title)\n poems.append(p)\n if len(poems) >= max_poems:\n break\n\n print(f\"Fetched {len(poems)} poems with >= {min_lines} lines from PoetryDB.\")\n return poems[:max_poems]\n\n\n# ---------------------------------------------------------------------------\n# Prompt construction\n# ---------------------------------------------------------------------------\n\ndef build_prompt(poem: dict, n_context: int | None = None) -> list[dict]:\n \"\"\"\n Build a neutral chat-style prompt from a poem dict.\n\n n_context: number of lines to use as context (8–15).\n If None, sampled randomly within the valid range.\n\n NOTE: The system prompt is intentionally sentiment-neutral.\n GRPO + the reward signal do all the steering — not the prompt.\n \"\"\"\n lines: list[str] = poem[\"lines\"]\n max_ctx = min(15, len(lines) - 4) # leave at least 4 lines \"unseen\"\n if max_ctx < 8:\n raise ValueError(f\"Poem '{poem['title']}' too short for 8-line context window.\")\n\n if n_context is None:\n n_context = random.randint(8, max_ctx)\n n_context = max(8, min(n_context, max_ctx))\n\n context = \"\\n\".join(lines[:n_context])\n return [\n {\n \"role\": \"system\",\n \"content\": \"You are a poet. Complete the following poem with exactly 4 lines.\",\n },\n {\n \"role\": \"user\",\n \"content\": context,\n },\n ]\n\n\n# ---------------------------------------------------------------------------\n# Dataset builder\n# ---------------------------------------------------------------------------\n\ndef build_dataset(poems: list[dict]) -> Dataset:\n \"\"\"\n Convert a list of poem dicts into a HuggingFace Dataset with a\n 'prompt' column formatted for GRPOTrainer (list of chat message dicts).\n \"\"\"\n records = []\n for poem in poems:\n try:\n prompt = build_prompt(poem)\n records.append({\n \"prompt\": prompt,\n \"title\": poem.get(\"title\", \"\"),\n \"author\": poem.get(\"author\", \"\"),\n })\n except ValueError:\n continue\n\n print(f\"Built dataset with {len(records)} training examples.\")\n return Dataset.from_list(records)\n\n\n# ---------------------------------------------------------------------------\n# Convenience: load everything in one call\n# ---------------------------------------------------------------------------\n\ndef load_poetry_dataset(min_lines: int = 12, max_poems: int = 500) -> Dataset:\n poems = fetch_all_poems(min_lines=min_lines, max_poems=max_poems)\n return build_dataset(poems)\n\n\nif __name__ == \"__main__\":\n ds = load_poetry_dataset()\n print(ds)\n print(\"\\nSample prompt:\")\n for msg in ds[0][\"prompt\"]:\n print(f\"[{msg['role']}]: {msg['content'][:200]}\")\n","Path":"/home/canders1/rlhf-s2/data/poem_loader.py"}}]} {"step_index":36,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:22:50Z","content":"Created At: 2026-11-20T12:22:50-05:00\nCompleted At: 2026-11-20T12:22:50-05:00\nCreated At: 2026-11-20T12:22:50-05:00\nCompleted At: 2026-11-20T12:22:50-05:00\nCreated file."} {"step_index":37,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:22:50Z","model":"claude-sonnet-4-6","input_tokens":1518,"cache_read_tokens":37425,"output_tokens":1094,"tool_calls":[{"name":"write_file","args":{"Content":"\"\"\"\nreward.py\n---------\nReward function for GRPO poem-completion training.\n\nReward = sentiment_score + 0.2 * line_count_bonus\n\n sentiment_score : DistilBERT SST-2 confidence in POSITIVE class → [0, 1]\n If label == NEGATIVE: score - 1.0 → [-1, 0]\n line_count_bonus: 1.0 if completion has 3–5 newlines (≈ 4 lines), else 0.0\n\nThe sentiment model is intentionally loaded once at module level so it\nstays on GPU across all reward calls during training.\n\"\"\"\n\nimport torch\nfrom transformers import pipeline\n\n# ---------------------------------------------------------------------------\n# Load sentiment model once\n# ---------------------------------------------------------------------------\n\n_SENTIMENT_MODEL = \"distilbert-base-uncased-finetuned-sst-2-english\"\n_device = 0 if torch.cuda.is_available() else -1\n\n_sentiment_pipe = pipeline(\n \"sentiment-analysis\",\n model=_SENTIMENT_MODEL,\n device=_device,\n truncation=True,\n max_length=128,\n batch_size=16,\n)\n\nprint(f\"[reward.py] Sentiment model loaded on device={_device}\")\n\n\n# ---------------------------------------------------------------------------\n# Sub-rewards\n# ---------------------------------------------------------------------------\n\ndef _sentiment_score(text: str) -> float:\n \"\"\"Score ∈ [-1, 1]. Positive sentiment → positive score.\"\"\"\n result = _sentiment_pipe(text)[0]\n score = result[\"score\"]\n return float(score) if result[\"label\"] == \"POSITIVE\" else float(score - 1.0)\n\n\ndef _line_count_bonus(text: str) -> float:\n \"\"\"Return 1.0 if text has 3–5 newlines (≈ 4 lines), else 0.0.\"\"\"\n n = text.count(\"\\n\")\n return 1.0 if 3 <= n <= 5 else 0.0\n\n\n# ---------------------------------------------------------------------------\n# Combined reward function (GRPOTrainer interface)\n# ---------------------------------------------------------------------------\n\ndef reward_fn(completions: list[list[dict]], **kwargs) -> list[float]:\n \"\"\"\n GRPOTrainer reward function.\n\n Args:\n completions: list of completion message lists, each like\n [{\"role\": \"assistant\", \"content\": \"\"}]\n **kwargs: absorbs extra trainer kwargs (prompts, etc.)\n\n Returns:\n list[float]: one scalar reward per completion.\n \"\"\"\n texts = [c[0][\"content\"] for c in completions]\n\n # Batch sentiment inference\n results = _sentiment_pipe(texts)\n\n rewards = []\n for text, result in zip(texts, results):\n score = result[\"score\"]\n sentiment = float(score) if result[\"label\"] == \"POSITIVE\" else float(score - 1.0)\n structure = _line_count_bonus(text)\n total = sentiment + 0.2 * structure\n rewards.append(total)\n\n return rewards\n\n\n# ---------------------------------------------------------------------------\n# Convenience: score a single text (used in notebook demos)\n# ---------------------------------------------------------------------------\n\ndef score_text(text: str) -> dict:\n \"\"\"Return a breakdown dict for a single completion string.\"\"\"\n result = _sentiment_pipe(text)[0]\n score = result[\"score\"]\n sentiment = float(score) if result[\"label\"] == \"POSITIVE\" else float(score - 1.0)\n structure = _line_count_bonus(text)\n return {\n \"label\": result[\"label\"],\n \"raw_score\": float(score),\n \"sentiment\": round(sentiment, 4),\n \"line_count_bonus\": structure,\n \"total_reward\": round(sentiment + 0.2 * structure, 4),\n \"n_newlines\": text.count(\"\\n\"),\n }\n\n\nif __name__ == \"__main__\":\n sample = \"The sun rose bright over golden hills,\\nAnd all the world was filled with light.\\nThe birds sang sweet on morning trills,\\nAnd hope returned, so pure and right.\"\n print(score_text(sample))\n","Path":"/home/canders1/rlhf-s2/training/reward.py"}}]} {"step_index":38,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:23:07Z","content":"Created At: 2026-11-20T12:23:07-05:00\nCompleted At: 2026-11-20T12:23:07-05:00\nCreated At: 2026-11-20T12:23:07-05:00\nCompleted At: 2026-11-20T12:23:07-05:00\nCreated file."} {"step_index":39,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:23:07Z","model":"claude-sonnet-4-6","input_tokens":1222,"cache_read_tokens":38879,"output_tokens":1318,"tool_calls":[{"name":"write_file","args":{"Content":"\"\"\"\ntrain_grpo.py\n-------------\nTrain Llama-3.2-1B-Instruct with GRPO to produce positive-sentiment\n4-line poem completions, using DistilBERT SST-2 as the reward signal.\n\nUsage:\n conda run -n newGPU python training/train_grpo.py --beta 0.04\n conda run -n newGPU python training/train_grpo.py --beta 0.4\n\nBoth runs can be launched sequentially from the demo notebook.\nLogs are written to checkpoints/beta_/trainer_log.jsonl for\nlive plotting.\n\"\"\"\n\nimport argparse\nimport os\nimport sys\n\n# Allow imports from project root\nsys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))\n\nimport torch\nfrom transformers import AutoTokenizer\nfrom trl import GRPOConfig, GRPOTrainer\n\nfrom data.poem_loader import load_poetry_dataset\nfrom training.reward import reward_fn\n\n\n# ---------------------------------------------------------------------------\n# Args\n# ---------------------------------------------------------------------------\n\ndef parse_args():\n p = argparse.ArgumentParser()\n p.add_argument(\"--beta\", type=float, default=0.04,\n help=\"KL penalty coefficient (0.04 = aggressive, 0.4 = conservative)\")\n p.add_argument(\"--model\", type=str,\n default=\"meta-llama/Llama-3.2-1B-Instruct\")\n p.add_argument(\"--max_poems\", type=int, default=500,\n help=\"Max poems to fetch from PoetryDB\")\n p.add_argument(\"--num_generations\", type=int, default=8,\n help=\"Group size G for GRPO advantage computation\")\n p.add_argument(\"--epochs\", type=int, default=1)\n p.add_argument(\"--lr\", type=float, default=1e-5)\n p.add_argument(\"--batch_size\", type=int, default=2)\n p.add_argument(\"--grad_accum\", type=int, default=4)\n p.add_argument(\"--logging_steps\", type=int, default=5)\n p.add_argument(\"--save_steps\", type=int, default=50)\n return p.parse_args()\n\n\n# ---------------------------------------------------------------------------\n# Main\n# ---------------------------------------------------------------------------\n\ndef main():\n args = parse_args()\n\n beta_str = str(args.beta).replace(\".\", \"_\")\n output_dir = f\"checkpoints/beta_{beta_str}\"\n os.makedirs(output_dir, exist_ok=True)\n\n print(f\"\\n{'='*60}\")\n print(f\" GRPO Training | β = {args.beta}\")\n print(f\" Output dir : {output_dir}\")\n print(f\" Model : {args.model}\")\n print(f\" CUDA devices : {torch.cuda.device_count()}\")\n print(f\"{'='*60}\\n\")\n\n # -----------------------------------------------------------------------\n # Dataset\n # -----------------------------------------------------------------------\n dataset = load_poetry_dataset(min_lines=12, max_poems=args.max_poems)\n\n # -----------------------------------------------------------------------\n # GRPO config\n # -----------------------------------------------------------------------\n config = GRPOConfig(\n output_dir=output_dir,\n model_init_kwargs={\n \"torch_dtype\": torch.bfloat16,\n \"device_map\": \"auto\",\n },\n # Optimization\n learning_rate=args.lr,\n per_device_train_batch_size=args.batch_size,\n gradient_accumulation_steps=args.grad_accum,\n num_train_epochs=args.epochs,\n # GRPO-specific\n num_generations=args.num_generations, # G: group size\n beta=args.beta, # KL penalty\n max_prompt_length=512,\n max_completion_length=128,\n # Logging & saving\n logging_steps=args.logging_steps,\n save_steps=args.save_steps,\n save_total_limit=2,\n report_to=\"none\", # notebook does its own plotting\n # Precision\n bf16=True,\n )\n\n # -----------------------------------------------------------------------\n # Trainer\n # -----------------------------------------------------------------------\n trainer = GRPOTrainer(\n model=args.model,\n reward_funcs=[reward_fn],\n args=config,\n train_dataset=dataset,\n )\n\n # -----------------------------------------------------------------------\n # Train\n # -----------------------------------------------------------------------\n print(\"Starting training...\")\n trainer.train()\n\n # Save final checkpoint + training log\n trainer.save_model(output_dir)\n trainer.state.save_to_json(f\"{output_dir}/trainer_state.json\")\n print(f\"\\nDone. Model saved to: {output_dir}\")\n\n\nif __name__ == \"__main__\":\n main()\n","Path":"/home/canders1/rlhf-s2/training/train_grpo.py"}}]} {"step_index":40,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:23:24Z","content":"Created At: 2026-11-20T12:23:24-05:00\nCompleted At: 2026-11-20T12:23:24-05:00\nCreated At: 2026-11-20T12:23:24-05:00\nCompleted At: 2026-11-20T12:23:24-05:00\nCreated file."} {"step_index":41,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:23:24Z","model":"claude-sonnet-4-6","input_tokens":1446,"cache_read_tokens":40037,"output_tokens":10734,"content":"Now the main notebook — this is the biggest piece:","tool_calls":[{"name":"write_file","args":{"Content":"{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# RLHF In-Class Demo\\n\",\n \"## Sentiment-Steered Poem Completion with GRPO\\n\",\n \"\\n\",\n \"**Llama 3.2 1B** · **PoetryDB** · **DistilBERT SST-2 reward** · **Group Relative Policy Optimization**\\n\",\n \"\\n\",\n \"---\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Section 0 — Setup\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os, sys, json, time, random, threading, subprocess\\n\",\n \"import requests\\n\",\n \"import numpy as np\\n\",\n \"import pandas as pd\\n\",\n \"import matplotlib\\n\",\n \"import matplotlib.pyplot as plt\\n\",\n \"import matplotlib.gridspec as gridspec\\n\",\n \"import ipywidgets as widgets\\n\",\n \"from IPython.display import display, clear_output, HTML\\n\",\n \"\\n\",\n \"# Project root on path\\n\",\n \"PROJECT_ROOT = os.path.abspath('..')\\n\",\n \"if PROJECT_ROOT not in sys.path:\\n\",\n \" sys.path.insert(0, PROJECT_ROOT)\\n\",\n \"\\n\",\n \"print('Libraries loaded.')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline\\n\",\n \"\\n\",\n \"print(f'PyTorch: {torch.__version__}')\\n\",\n \"print(f'CUDA available: {torch.cuda.is_available()}')\\n\",\n \"for i in range(torch.cuda.device_count()):\\n\",\n \" p = torch.cuda.get_device_properties(i)\\n\",\n \" print(f' GPU {i}: {p.name} ({p.total_memory // 1024**3} GB)')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Model IDs ──────────────────────────────────────────────────────────────\\n\",\n \"BASE_MODEL_ID = 'meta-llama/Llama-3.2-1B-Instruct'\\n\",\n \"SENTIMENT_MODEL_ID = 'distilbert-base-uncased-finetuned-sst-2-english'\\n\",\n \"\\n\",\n \"CHECKPOINT_LOW = '../checkpoints/beta_0_04' # filled after training\\n\",\n \"CHECKPOINT_HIGH = '../checkpoints/beta_0_4'\\n\",\n \"\\n\",\n \"print('Config set.')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Load base model (used for reference inference & perplexity) ────────────\\n\",\n \"print('Loading tokenizer...')\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_ID)\\n\",\n \"tokenizer.pad_token = tokenizer.eos_token\\n\",\n \"\\n\",\n \"print('Loading base model...')\\n\",\n \"base_model = AutoModelForCausalLM.from_pretrained(\\n\",\n \" BASE_MODEL_ID,\\n\",\n \" torch_dtype=torch.bfloat16,\\n\",\n \" device_map='auto',\\n\",\n \")\\n\",\n \"base_model.eval()\\n\",\n \"print('Base model ready.')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Load sentiment pipeline ────────────────────────────────────────────────\\n\",\n \"sentiment_pipe = pipeline(\\n\",\n \" 'sentiment-analysis',\\n\",\n \" model=SENTIMENT_MODEL_ID,\\n\",\n \" device=0,\\n\",\n \" truncation=True,\\n\",\n \" max_length=128,\\n\",\n \")\\n\",\n \"print('Sentiment model ready.')\\n\",\n \"\\n\",\n \"def score_text(text: str) -> dict:\\n\",\n \" r = sentiment_pipe(text)[0]\\n\",\n \" s = r['score']\\n\",\n \" sentiment = s if r['label'] == 'POSITIVE' else s - 1.0\\n\",\n \" n_lines = text.count('\\\\n')\\n\",\n \" bonus = 1.0 if 3 <= n_lines <= 5 else 0.0\\n\",\n \" return {\\n\",\n \" 'label': r['label'],\\n\",\n \" 'sentiment': round(sentiment, 4),\\n\",\n \" 'line_bonus': bonus,\\n\",\n \" 'total_reward': round(sentiment + 0.2 * bonus, 4),\\n\",\n \" 'n_newlines': n_lines,\\n\",\n \" }\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"---\\n\",\n \"## Section 1 — RLHF & GRPO Concepts\\n\",\n \"\\n\",\n \"### What is RLHF?\\n\",\n \"**Reinforcement Learning from Human Feedback** aligns a language model's outputs toward a desired behavior by training it with a reward signal instead of supervised labels.\\n\",\n \"\\n\",\n \"### Standard RLHF pipeline (PPO)\\n\",\n \"```\\n\",\n \"[prompts] → policy → completions → reward model → scalar reward → PPO update\\n\",\n \" ↑___________________KL penalty (vs reference policy)__________|\\n\",\n \"```\\n\",\n \"Problems: requires a trained reward model, a value network, and a reference model — 4 models in memory.\\n\",\n \"\\n\",\n \"### GRPO (Group Relative Policy Optimization)\\n\",\n \"```\\n\",\n \"[prompt] → generate G completions → reward_fn(each) → normalize within group → policy gradient\\n\",\n \"```\\n\",\n \"**Key insight:** Instead of a learned value baseline, GRPO uses the **mean reward of the group** as the baseline:\\n\",\n \"\\n\",\n \"$$A_i = \\\\frac{r_i - \\\\mu_G}{\\\\sigma_G} \\\\quad \\\\text{where } G = \\\\{r_1, \\\\ldots, r_8\\\\} \\\\text{ for the same prompt}$$\\n\",\n \"\\n\",\n \"No value network. No reward model training. The reward function can be **any callable** — including a pretrained sentiment classifier.\\n\",\n \"\\n\",\n \"### Our reward function\\n\",\n \"$$R = \\\\underbrace{R_{\\\\text{sentiment}}}_{\\\\text{SST-2 confidence} \\\\in [-1,1]} + 0.2 \\\\times \\\\underbrace{R_{\\\\text{structure}}}_{\\\\mathbb{1}[\\\\text{3–5 newlines}]}$$\\n\",\n \"\\n\",\n \"### The KL penalty β\\n\",\n \"$$\\\\mathcal{L} = -\\\\mathbb{E}[A_i \\\\log \\\\pi_\\\\theta(y|x)] + \\\\beta \\\\cdot D_{KL}(\\\\pi_\\\\theta \\\\| \\\\pi_{\\\\text{ref}})$$\\n\",\n \"\\n\",\n \"| β | Effect |\\n\",\n \"|---|---|\\n\",\n \"| **0.04** (low) | Model drifts far from base; strong sentiment shift, may lose poetic coherence |\\n\",\n \"| **0.4** (high) | Stays close to base distribution; modest sentiment shift, more coherent poetry |\\n\",\n \"\\n\",\n \"> **Demo goal:** Show that the reward signal — not the prompt — drives the sentiment shift. The system prompt is completely neutral.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"---\\n\",\n \"## Section 2 — PoetryDB Dataset Explorer\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Fetch poems from PoetryDB ──────────────────────────────────────────────\\n\",\n \"print('Fetching poems from poetrydb.org...')\\n\",\n \"resp = requests.get('https://poetrydb.org/random/200', timeout=20)\\n\",\n \"all_poems_raw = resp.json()\\n\",\n \"POEMS = [p for p in all_poems_raw if int(p.get('linecount', 0)) >= 12]\\n\",\n \"print(f'Found {len(POEMS)} poems with ≥ 12 lines.')\\n\",\n \"\\n\",\n \"# Show length distribution\\n\",\n \"linecounts = [int(p['linecount']) for p in POEMS]\\n\",\n \"fig, ax = plt.subplots(figsize=(8, 3))\\n\",\n \"ax.hist(linecounts, bins=range(12, max(linecounts)+2), color='steelblue', edgecolor='white')\\n\",\n \"ax.set_xlabel('Line count'); ax.set_ylabel('# poems')\\n\",\n \"ax.set_title('Distribution of Poem Lengths (filtered ≥ 12 lines)')\\n\",\n \"plt.tight_layout(); plt.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Interactive poem viewer ────────────────────────────────────────────────\\n\",\n \"POEM_TITLES = [f\\\"{p['title']} — {p['author']}\\\" for p in POEMS]\\n\",\n \"\\n\",\n \"poem_dropdown = widgets.Dropdown(\\n\",\n \" options=list(enumerate(POEM_TITLES)),\\n\",\n \" description='Poem:',\\n\",\n \" layout=widgets.Layout(width='70%'),\\n\",\n \")\\n\",\n \"ctx_slider = widgets.IntSlider(\\n\",\n \" value=10, min=8, max=15, step=1,\\n\",\n \" description='Context lines:',\\n\",\n \" style={'description_width': 'initial'},\\n\",\n \")\\n\",\n \"out_poem = widgets.Output()\\n\",\n \"\\n\",\n \"def show_poem(change=None):\\n\",\n \" idx = poem_dropdown.value[0]\\n\",\n \" poem = POEMS[idx]\\n\",\n \" n_ctx = ctx_slider.value\\n\",\n \" lines = poem['lines']\\n\",\n \" with out_poem:\\n\",\n \" clear_output(wait=True)\\n\",\n \" display(HTML(f\\\"\\\"\\\"\\n\",\n \"
\\n\",\n \"

{poem['title']}

\\n\",\n \" — {poem['author']}\\n\",\n \"
\\n\",\n \"
\\n\",\n \" {''.join(f\\\"{l}
\\\" for l in lines[:n_ctx])}\\n\",\n \" {''.join(f\\\"{l}
\\\" for l in lines[n_ctx:])}\\n\",\n \"
\\n\",\n \"
\\n\",\n \" Blue = prompt context ({n_ctx} lines)  | \\n\",\n \" Grey = held-out ground truth\\n\",\n \"
\\n\",\n \"
\\\"\\\"\\\"))\\n\",\n \"\\n\",\n \"poem_dropdown.observe(show_poem, names='value')\\n\",\n \"ctx_slider.observe(show_poem, names='value')\\n\",\n \"show_poem()\\n\",\n \"display(widgets.VBox([poem_dropdown, ctx_slider, out_poem]))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"---\\n\",\n \"## Section 3 — Reward Function Demo\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Live reward scorer ─────────────────────────────────────────────────────\\n\",\n \"text_input = widgets.Textarea(\\n\",\n \" value='The sun rose bright over golden hills,\\\\nAnd all the world was filled with light.\\\\nThe birds sang sweet on morning trills,\\\\nAnd hope returned, so pure and right.',\\n\",\n \" description='Text:',\\n\",\n \" layout=widgets.Layout(width='70%', height='120px'),\\n\",\n \")\\n\",\n \"score_btn = widgets.Button(description='Score it', button_style='primary')\\n\",\n \"out_score = widgets.Output()\\n\",\n \"\\n\",\n \"def on_score(b):\\n\",\n \" with out_score:\\n\",\n \" clear_output(wait=True)\\n\",\n \" result = score_text(text_input.value)\\n\",\n \" bar_width = int(abs(result['sentiment']) * 300)\\n\",\n \" color = '#27ae60' if result['sentiment'] > 0 else '#e74c3c'\\n\",\n \" display(HTML(f\\\"\\\"\\\"\\n\",\n \"
\\n\",\n \" Label: {result['label']}
\\n\",\n \" Sentiment score: {result['sentiment']:+.4f}  \\n\",\n \"
\\n\",\n \" Line count bonus: {result['line_bonus']}  \\n\",\n \" (newlines detected: {result['n_newlines']})
\\n\",\n \" Total reward: {result['total_reward']:+.4f}\\n\",\n \"
\\\"\\\"\\\"))\\n\",\n \"\\n\",\n \"score_btn.on_click(on_score)\\n\",\n \"on_score(None)\\n\",\n \"display(widgets.VBox([text_input, score_btn, out_score]))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Batch score some hand-picked examples ─────────────────────────────────\\n\",\n \"examples = [\\n\",\n \" ('Uplifting', 'The light broke through the morning grey,\\\\nAnd filled the world with golden cheer.\\\\nThe flowers bloomed along the way,\\\\nAnd life felt sweet and bright and clear.'),\\n\",\n \" ('Neutral', 'The clock ticked slow upon the shelf,\\\\nThe room was still and full of shade.\\\\nHe sat alone beside himself,\\\\nAnd watched the light begin to fade.'),\\n\",\n \" ('Dark', 'The grave was cold, the wind was thin,\\\\nNo star relieved the blackened sky.\\\\nThe hollow hours closed him in,\\\\nAnd no one came to hear him cry.'),\\n\",\n \"]\\n\",\n \"print(f'{\\\"Example\\\":<12} {\\\"Sentiment\\\":>10} {\\\"Bonus\\\":>6} {\\\"Total Reward\\\":>13}')\\n\",\n \"print('-' * 50)\\n\",\n \"for label, text in examples:\\n\",\n \" r = score_text(text)\\n\",\n \" print(f'{label:<12} {r[\\\"sentiment\\\"]:>+10.4f} {r[\\\"line_bonus\\\"]:>6.1f} {r[\\\"total_reward\\\"]:>+13.4f}')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"---\\n\",\n \"## Section 4 — 🔴 Live GRPO Training\\n\",\n \"\\n\",\n \"We run **two sequential training runs** with different β values:\\n\",\n \"- **β = 0.04** — aggressive; model drifts far from base\\n\",\n \"- **β = 0.4** — conservative; model stays close to base\\n\",\n \"\\n\",\n \"Training curves update live every few steps.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Training utilities ────────────────────────────────────────────────────\\n\",\n \"import json, subprocess, time, threading\\n\",\n \"from pathlib import Path\\n\",\n \"\\n\",\n \"def load_trainer_log(checkpoint_dir: str) -> pd.DataFrame:\\n\",\n \" \\\"\\\"\\\"Load trainer_state.json log history into a DataFrame.\\\"\\\"\\\"\\n\",\n \" path = Path(checkpoint_dir) / 'trainer_state.json'\\n\",\n \" if not path.exists():\\n\",\n \" return pd.DataFrame()\\n\",\n \" with open(path) as f:\\n\",\n \" state = json.load(f)\\n\",\n \" logs = [e for e in state.get('log_history', []) if 'loss' in e or 'reward' in e]\\n\",\n \" return pd.DataFrame(logs)\\n\",\n \"\\n\",\n \"def plot_curves(df_low: pd.DataFrame, df_high: pd.DataFrame, ax_reward, ax_kl):\\n\",\n \" \\\"\\\"\\\"Plot reward and KL curves for both beta runs.\\\"\\\"\\\"\\n\",\n \" for ax in [ax_reward, ax_kl]:\\n\",\n \" ax.cla()\\n\",\n \"\\n\",\n \" reward_col = 'rewards/mean' if 'rewards/mean' in (df_low.columns if not df_low.empty else []) else None\\n\",\n \" kl_col = 'kl' if 'kl' in (df_low.columns if not df_low.empty else []) else None\\n\",\n \"\\n\",\n \" for df, label, color in [(df_low, 'β=0.04', '#e74c3c'), (df_high, 'β=0.4', '#2980b9')]:\\n\",\n \" if df.empty:\\n\",\n \" continue\\n\",\n \" x = df.get('step', df.index)\\n\",\n \" if reward_col and reward_col in df:\\n\",\n \" ax_reward.plot(x, df[reward_col], color=color, label=label, linewidth=2)\\n\",\n \" if kl_col and kl_col in df:\\n\",\n \" ax_kl.plot(x, df[kl_col], color=color, label=label, linewidth=2)\\n\",\n \"\\n\",\n \" ax_reward.set_title('Mean Reward vs Step'); ax_reward.set_ylabel('Reward')\\n\",\n \" ax_reward.legend(); ax_reward.grid(alpha=0.3)\\n\",\n \" ax_kl.set_title('KL Divergence vs Step'); ax_kl.set_ylabel('KL'); ax_kl.set_xlabel('Step')\\n\",\n \" ax_kl.legend(); ax_kl.grid(alpha=0.3)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Run β=0.04 training ───────────────────────────────────────────────────\\n\",\n \"# This cell launches training in a subprocess and streams live plots.\\n\",\n \"\\n\",\n \"PYTHON = '/home/canders1/miniforge3/envs/newGPU/bin/python'\\n\",\n \"TRAIN_SCRIPT = '../training/train_grpo.py'\\n\",\n \"CHECKPOINT_LOW = '../checkpoints/beta_0_04'\\n\",\n \"\\n\",\n \"fig, (ax_r, ax_k) = plt.subplots(1, 2, figsize=(13, 4))\\n\",\n \"plt.tight_layout(pad=2)\\n\",\n \"out_training_low = widgets.Output()\\n\",\n \"display(out_training_low)\\n\",\n \"\\n\",\n \"print('▶ Launching β=0.04 training...')\\n\",\n \"proc_low = subprocess.Popen(\\n\",\n \" [PYTHON, TRAIN_SCRIPT, '--beta', '0.04',\\n\",\n \" '--max_poems', '400', '--logging_steps', '5'],\\n\",\n \" cwd='..', stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,\\n\",\n \")\\n\",\n \"\\n\",\n \"def stream_low():\\n\",\n \" for line in proc_low.stdout:\\n\",\n \" with out_training_low:\\n\",\n \" print(line, end='')\\n\",\n \" # Replot every time a log line appears\\n\",\n \" df_low = load_trainer_log(CHECKPOINT_LOW)\\n\",\n \" df_high = load_trainer_log(CHECKPOINT_HIGH)\\n\",\n \" if not df_low.empty:\\n\",\n \" plot_curves(df_low, df_high, ax_r, ax_k)\\n\",\n \" fig.canvas.draw_idle()\\n\",\n \"\\n\",\n \"threading.Thread(target=stream_low, daemon=True).start()\\n\",\n \"print('Training running in background — plots update live. Run the next cell to watch...')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Wait for β=0.04 to finish, then run β=0.4 ─────────────────────────────\\n\",\n \"print('Waiting for β=0.04 run to finish...')\\n\",\n \"proc_low.wait()\\n\",\n \"print('β=0.04 done! Reloading tuned model...')\\n\",\n \"\\n\",\n \"tuned_model_low = AutoModelForCausalLM.from_pretrained(\\n\",\n \" CHECKPOINT_LOW, torch_dtype=torch.bfloat16, device_map='auto'\\n\",\n \")\\n\",\n \"tuned_model_low.eval()\\n\",\n \"print('β=0.04 model loaded.')\\n\",\n \"\\n\",\n \"print('\\\\n▶ Launching β=0.4 training...')\\n\",\n \"proc_high = subprocess.Popen(\\n\",\n \" [PYTHON, TRAIN_SCRIPT, '--beta', '0.4',\\n\",\n \" '--max_poems', '400', '--logging_steps', '5'],\\n\",\n \" cwd='..', stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,\\n\",\n \")\\n\",\n \"\\n\",\n \"def stream_high():\\n\",\n \" for line in proc_high.stdout:\\n\",\n \" with out_training_low:\\n\",\n \" print(line, end='')\\n\",\n \" df_low = load_trainer_log(CHECKPOINT_LOW)\\n\",\n \" df_high = load_trainer_log(CHECKPOINT_HIGH)\\n\",\n \" if not df_high.empty:\\n\",\n \" plot_curves(df_low, df_high, ax_r, ax_k)\\n\",\n \" fig.canvas.draw_idle()\\n\",\n \"\\n\",\n \"threading.Thread(target=stream_high, daemon=True).start()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Wait for β=0.4 and load it ────────────────────────────────────────────\\n\",\n \"proc_high.wait()\\n\",\n \"print('β=0.4 done! Loading model...')\\n\",\n \"\\n\",\n \"CHECKPOINT_HIGH = '../checkpoints/beta_0_4'\\n\",\n \"tuned_model_high = AutoModelForCausalLM.from_pretrained(\\n\",\n \" CHECKPOINT_HIGH, torch_dtype=torch.bfloat16, device_map='auto'\\n\",\n \")\\n\",\n \"tuned_model_high.eval()\\n\",\n \"print('β=0.4 model loaded. Both runs complete!')\\n\",\n \"\\n\",\n \"# Final side-by-side training curves\\n\",\n \"df_low = load_trainer_log(CHECKPOINT_LOW)\\n\",\n \"df_high = load_trainer_log(CHECKPOINT_HIGH)\\n\",\n \"fig2, (ax_r2, ax_k2) = plt.subplots(1, 2, figsize=(13, 4))\\n\",\n \"plot_curves(df_low, df_high, ax_r2, ax_k2)\\n\",\n \"plt.suptitle('Final Training Curves — β Comparison', fontsize=13, fontweight='bold')\\n\",\n \"plt.tight_layout()\\n\",\n \"plt.show()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"---\\n\",\n \"## Section 5 — Poem Sampler\\n\",\n \"Pick a poem, choose how many context lines to show, and compare completions from all three models.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Generation helper ──────────────────────────────────────────────────────\\n\",\n \"def generate_completion(model, poem: dict, n_context: int, max_new_tokens: int = 100) -> str:\\n\",\n \" lines = poem['lines']\\n\",\n \" context = '\\\\n'.join(lines[:n_context])\\n\",\n \" messages = [\\n\",\n \" {'role': 'system', 'content': 'You are a poet. Complete the following poem with exactly 4 lines.'},\\n\",\n \" {'role': 'user', 'content': context},\\n\",\n \" ]\\n\",\n \" input_ids = tokenizer.apply_chat_template(\\n\",\n \" messages, return_tensors='pt', add_generation_prompt=True\\n\",\n \" ).to(model.device)\\n\",\n \" with torch.no_grad():\\n\",\n \" output_ids = model.generate(\\n\",\n \" input_ids,\\n\",\n \" max_new_tokens=max_new_tokens,\\n\",\n \" do_sample=True,\\n\",\n \" temperature=0.8,\\n\",\n \" top_p=0.95,\\n\",\n \" pad_token_id=tokenizer.eos_token_id,\\n\",\n \" )\\n\",\n \" generated = output_ids[0][input_ids.shape[1]:]\\n\",\n \" return tokenizer.decode(generated, skip_special_tokens=True).strip()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Sampler widget ─────────────────────────────────────────────────────────\\n\",\n \"sampler_dropdown = widgets.Dropdown(\\n\",\n \" options=list(enumerate(POEM_TITLES)),\\n\",\n \" description='Poem:',\\n\",\n \" layout=widgets.Layout(width='70%'),\\n\",\n \")\\n\",\n \"sampler_slider = widgets.IntSlider(\\n\",\n \" value=10, min=8, max=15, step=1,\\n\",\n \" description='Context lines:',\\n\",\n \" style={'description_width': 'initial'},\\n\",\n \")\\n\",\n \"generate_btn = widgets.Button(\\n\",\n \" description='⚡ Generate completions',\\n\",\n \" button_style='success',\\n\",\n \" layout=widgets.Layout(width='220px'),\\n\",\n \")\\n\",\n \"out_sampler = widgets.Output()\\n\",\n \"\\n\",\n \"def card(title, text, reward_dict, color):\\n\",\n \" label = reward_dict['label']\\n\",\n \" total = reward_dict['total_reward']\\n\",\n \" rcolor = '#27ae60' if total > 0 else '#e74c3c'\\n\",\n \" lines_html = text.replace('\\\\n', '
')\\n\",\n \" return f\\\"\\\"\\\"\\n\",\n \"
\\n\",\n \"
{title}
\\n\",\n \"
{lines_html}
\\n\",\n \"
\\n\",\n \"
\\n\",\n \" {label} · reward = {total:+.3f}\\n\",\n \"
\\n\",\n \"
\\\"\\\"\\\"\\n\",\n \"\\n\",\n \"def on_generate(b):\\n\",\n \" idx = sampler_dropdown.value[0]\\n\",\n \" poem = POEMS[idx]\\n\",\n \" n_ctx = sampler_slider.value\\n\",\n \" context_text = '\\\\n'.join(poem['lines'][:n_ctx])\\n\",\n \"\\n\",\n \" with out_sampler:\\n\",\n \" clear_output(wait=True)\\n\",\n \" display(HTML(\\\"Generating... (may take 10–20s)\\\"))\\n\",\n \"\\n\",\n \" comp_base = generate_completion(base_model, poem, n_ctx)\\n\",\n \" comp_low = generate_completion(tuned_model_low, poem, n_ctx)\\n\",\n \" comp_high = generate_completion(tuned_model_high, poem, n_ctx)\\n\",\n \"\\n\",\n \" r_base = score_text(comp_base)\\n\",\n \" r_low = score_text(comp_low)\\n\",\n \" r_high = score_text(comp_high)\\n\",\n \"\\n\",\n \" with out_sampler:\\n\",\n \" clear_output(wait=True)\\n\",\n \" display(HTML(f\\\"\\\"\\\"\\n\",\n \"
\\n\",\n \"

{poem['title']} — {poem['author']}

\\n\",\n \"
\\n\",\n \" {context_text.replace(chr(10), '
')}\\n\",\n \"
\\n\",\n \"
↓ model completions ↓
\\n\",\n \"
\\n\",\n \"
\\n\",\n \" {card('Base Llama 3.2 1B', comp_base, r_base, '#7f8c8d')}\\n\",\n \" {card('Tuned β = 0.04', comp_low, r_low, '#e74c3c')}\\n\",\n \" {card('Tuned β = 0.4', comp_high, r_high, '#2980b9')}\\n\",\n \"
\\n\",\n \" \\\"\\\"\\\"))\\n\",\n \"\\n\",\n \"generate_btn.on_click(on_generate)\\n\",\n \"display(widgets.VBox([\\n\",\n \" sampler_dropdown,\\n\",\n \" sampler_slider,\\n\",\n \" generate_btn,\\n\",\n \" out_sampler,\\n\",\n \"]))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"---\\n\",\n \"## Section 6 — Quantitative Comparison\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Score 20 random poems with all three models ───────────────────────────\\n\",\n \"N_EVAL = 20\\n\",\n \"eval_poems = random.sample(POEMS, N_EVAL)\\n\",\n \"results = []\\n\",\n \"\\n\",\n \"print(f'Running inference on {N_EVAL} poems with 3 models...')\\n\",\n \"for i, poem in enumerate(eval_poems):\\n\",\n \" n_ctx = random.randint(8, min(15, int(poem['linecount']) - 4))\\n\",\n \" for model, label in [\\n\",\n \" (base_model, 'Base'),\\n\",\n \" (tuned_model_low, 'β=0.04'),\\n\",\n \" (tuned_model_high, 'β=0.4'),\\n\",\n \" ]:\\n\",\n \" comp = generate_completion(model, poem, n_ctx)\\n\",\n \" r = score_text(comp)\\n\",\n \" results.append({'poem': poem['title'], 'model': label,\\n\",\n \" 'sentiment': r['sentiment'], 'total_reward': r['total_reward']})\\n\",\n \" if (i + 1) % 5 == 0:\\n\",\n \" print(f' {i+1}/{N_EVAL} poems done...')\\n\",\n \"\\n\",\n \"df_eval = pd.DataFrame(results)\\n\",\n \"print('Done!')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# ── Summary bar chart ──────────────────────────────────────────────────────\\n\",\n \"means = df_eval.groupby('model')['total_reward'].mean().reindex(['Base', 'β=0.04', 'β=0.4'])\\n\",\n \"sems = df_eval.groupby('model')['total_reward'].sem().reindex(['Base', 'β=0.04', 'β=0.4'])\\n\",\n \"\\n\",\n \"fig, axes = plt.subplots(1, 2, figsize=(13, 4))\\n\",\n \"\\n\",\n \"# Bar chart: mean total reward\\n\",\n \"ax = axes[0]\\n\",\n \"colors = ['#7f8c8d', '#e74c3c', '#2980b9']\\n\",\n \"bars = ax.bar(means.index, means.values, yerr=sems.values, color=colors,\\n\",\n \" capsize=5, edgecolor='white', linewidth=1.2)\\n\",\n \"ax.set_title('Mean Total Reward by Model\\\\n(sentiment + 0.2 × structure bonus)', fontsize=11)\\n\",\n \"ax.set_ylabel('Mean Reward'); ax.set_ylim(-0.1, 1.2)\\n\",\n \"ax.axhline(0, color='black', linewidth=0.8, linestyle='--')\\n\",\n \"ax.grid(axis='y', alpha=0.3)\\n\",\n \"for bar, v in zip(bars, means.values):\\n\",\n \" ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 0.03,\\n\",\n \" f'{v:.3f}', ha='center', fontsize=10, fontweight='bold')\\n\",\n \"\\n\",\n \"# Strip plot: individual poem scores\\n\",\n \"ax2 = axes[1]\\n\",\n \"for j, (label, color) in enumerate([('Base', '#7f8c8d'), ('β=0.04', '#e74c3c'), ('β=0.4', '#2980b9')]):\\n\",\n \" vals = df_eval[df_eval['model'] == label]['total_reward'].values\\n\",\n \" jitter = np.random.uniform(-0.1, 0.1, size=len(vals))\\n\",\n \" ax2.scatter(np.full_like(vals, j) + jitter, vals, color=color, alpha=0.7, s=40)\\n\",\n \" ax2.plot([j - 0.3, j + 0.3], [vals.mean(), vals.mean()], color=color, linewidth=3)\\n\",\n \"\\n\",\n \"ax2.set_xticks([0, 1, 2]); ax2.set_xticklabels(['Base', 'β=0.04', 'β=0.4'])\\n\",\n \"ax2.set_title('Per-Poem Reward Distribution', fontsize=11)\\n\",\n \"ax2.set_ylabel('Total Reward')\\n\",\n \"ax2.axhline(0, color='black', linewidth=0.8, linestyle='--')\\n\",\n \"ax2.grid(axis='y', alpha=0.3)\\n\",\n \"\\n\",\n \"plt.suptitle('Quantitative Comparison — 20 Random Poems', fontsize=13, fontweight='bold')\\n\",\n \"plt.tight_layout()\\n\",\n \"plt.show()\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"newGPU\",\n \"language\": \"python\",\n \"name\": \"newgpu\"\n },\n \"language_info\": {\n \"name\": \"python\",\n \"version\": \"3.12.11\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n","Path":"/home/canders1/rlhf-s2/notebooks/rlhf_poem_demo.ipynb"}}]} {"step_index":42,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:25:27Z","content":"Created At: 2026-11-20T12:25:27-05:00\nCompleted At: 2026-11-20T12:25:27-05:00\nCreated At: 2026-11-20T12:25:27-05:00\nCompleted At: 2026-11-20T12:25:27-05:00\nCreated file."} {"step_index":43,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:25:27Z","model":"claude-sonnet-4-6","input_tokens":10862,"cache_read_tokens":41419,"output_tokens":187,"tool_calls":[{"name":"write_file","args":{"Content":"# Core ML\ntorch>=2.3.0\ntransformers>=4.44.0\ntrl>=0.9.0\npeft>=0.12.0\ndatasets>=2.20.0\naccelerate>=0.33.0\n\n# Data\nrequests>=2.31.0\n\n# Notebook / UI\nipywidgets>=8.0\njupyterlab>=4.0\nmatplotlib>=3.9\npandas\nnumpy\n","Path":"/home/canders1/rlhf-s2/requirements.txt"}}]} {"step_index":44,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:25:31Z","content":"Created At: 2026-11-20T12:25:31-05:00\nCompleted At: 2026-11-20T12:25:31-05:00\nCreated At: 2026-11-20T12:25:31-05:00\nCompleted At: 2026-11-20T12:25:31-05:00\nCreated file."} {"step_index":45,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:25:31Z","model":"claude-sonnet-4-6","input_tokens":315,"cache_read_tokens":52217,"output_tokens":355,"content":"Now do a quick sanity-check: verify imports work and PoetryDB is reachable:","tool_calls":[{"name":"run_command","args":{"CommandLine":"cd /home/canders1/rlhf-s2 && conda run -n newGPU python -c \"\nimport sys; sys.path.insert(0, '.')\nimport requests\n# Test PoetryDB\nresp = requests.get('https://poetrydb.org/random/5', timeout=15)\npoems = resp.json()\nprint(f'PoetryDB OK — got {len(poems)} poems')\np = poems[0]\nprint(f' Sample: \\\"{p[\\\"title\\\"]}\\\" by {p[\\\"author\\\"]} ({p[\\\"linecount\\\"]} lines)')\n\n# Test reward\nfrom training.reward import score_text\nr = score_text('The sun rose bright and filled the world with light.\\nAnd hope returned, so pure and right.\\nThe birds sang sweet on morning trills,\\nAnd life felt warm and clear.')\nprint(f'Reward fn OK — {r}')\n\"\n","WaitMsBeforeAsync":30000,"toolSummary":"Smoke test reward and PoetryDB"}}]} {"step_index":46,"source":"MODEL","type":"GENERIC","status":"RUNNING","created_at":"2026-11-20T17:25:39Z","content":"Created At: 2026-11-20T12:25:39-05:00\nTool is running as a background task with task id: 99450367-08ef-40e7-b8c5-eb7d1d3288e6/task-46\nTask Description: cd /home/canders1/rlhf-s2 && conda run -n newGPU python -c \"\nimport sys; sys.path.insert(0, '.')\nimport requests\n# Test PoetryDB\nresp = requests.get('https://poetrydb.org/random/5', timeout=15)\npoems = resp.json()\nprint(f'PoetryDB OK — got {len(poems)} poems')\np = poems[0]\nprint(f' Sample: \\\"{p[\\\"title\\\"]}\\\" by {p[\\\"author\\\"]} ({p[\\\"linecount\\\"]} lines)')\n\n# Test reward\nfrom training.reward import score_text\nr = score_text('The sun rose bright and filled the world with light.\\nAnd hope returned, so pure and right.\\nThe birds sang sweet on morning trills,\\nAnd life felt warm and clear.')\nprint(f'Reward fn OK — {r}')\n\"\n\nTask logs are available at: file:///home/canders1/.gemini/antigravity/brain/99450367-08ef-40e7-b8c5-eb7d1d3288e6/.system_generated/tasks/task-46.log"} {"step_index":47,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:25:49Z","model":"claude-sonnet-4-6","input_tokens":539,"cache_read_tokens":52468,"output_tokens":260,"tool_calls":[{"name":"run_command","args":{"CommandLine":"cd /home/canders1/rlhf-s2 && conda run -n newGPU python -c \"\nimport sys; sys.path.insert(0, '.')\nfrom data.poem_loader import build_prompt\npoem = {'title': 'Test', 'author': 'Test', 'linecount': '20',\n 'lines': [f'Line {i}' for i in range(20)]}\np = build_prompt(poem, n_context=10)\nprint('poem_loader OK')\nprint('System prompt:', p[0]['content'])\nprint('User content (first 60 chars):', p[1]['content'][:60])\n\"\n","WaitMsBeforeAsync":10000,"toolSummary":"Test poem loader"}}]} {"step_index":48,"source":"MODEL","type":"GENERIC","status":"DONE","created_at":"2026-11-20T17:25:53Z","content":"Created At: 2026-11-20T12:25:53-05:00\nCompleted At: 2026-11-20T12:25:56-05:00\nCreated At: 2026-11-20T12:25:53-05:00\nCompleted At: 2026-11-20T12:25:56-05:00\nCommand exited with code 0.\n\nCommand output:\npoem_loader OK\r\nSystem prompt: You are a poet. Complete the following poem with exactly 4 lines.\r\nUser content (first 60 chars): Line 0\r\nLine 1\r\nLine 2\r\nLine 3\r\nLine 4\r\nLine 5\r\nLine 6\r\nLine 7\r\nLine\r\n\r\n"} {"step_index":49,"source":"SYSTEM","type":"SYSTEM_MESSAGE","status":"DONE","created_at":"2026-11-20T17:25:56Z","content":"The following is a not actually sent by the user. It is provided by the system as important information to pay attention to.\n\n\n[Message] timestamp=2026-11-20T17:25:54Z sender=99450367-08ef-40e7-b8c5-eb7d1d3288e6/task-46 priority=MESSAGE_PRIORITY_HIGH content=Task id \"99450367-08ef-40e7-b8c5-eb7d1d3288e6/task-46\" finished with result:\nCommand exited with code 0.\n\nCommand output:\nPoetryDB OK — got 5 poems\r\n Sample: \"By the Rivers of Babylon We Sat Down and Wept\" by George Gordon, Lord Byron (18 lines)\r\n[reward.py] Sentiment model loaded on device=0\r\nReward fn OK — {'label': 'POSITIVE', 'raw_score': 0.999859094619751, 'sentiment': 0.9999, 'line_count_bonus': 1.0, 'total_reward': 1.1999, 'n_newlines': 3}\r\n\r\n\r\nLoading weights: 0%| | 0/104 [00:00"} {"step_index":50,"source":"MODEL","type":"PLANNER_RESPONSE","status":"DONE","created_at":"2026-11-20T17:25:56Z","model":"claude-sonnet-4-6","input_tokens":1036,"cache_read_tokens":52887,"output_tokens":464,"content":"Everything passes. Here's a summary of what was built:\n\n---\n\nAll decisions locked in and all files are built. Here's what landed in your workspace:\n\n| File | Purpose |\n|---|---|\n| [data/poem_loader.py](file:///home/canders1/rlhf-s2/data/poem_loader.py) | Fetches poems from `poetrydb.org` REST API, filters by line count, builds **neutral** chat prompts |\n| [training/reward.py](file:///home/canders1/rlhf-s2/training/reward.py) | Sentiment score (SST-2) + 0.2 × line-count bonus, with `score_text()` for the notebook demos |\n| [training/train_grpo.py](file:///home/canders1/rlhf-s2/training/train_grpo.py) | `GRPOTrainer` script — takes `--beta` as a CLI arg, writes `trainer_state.json` for live plotting |\n| [notebooks/rlhf_poem_demo.ipynb](file:///home/canders1/rlhf-s2/notebooks/rlhf_poem_demo.ipynb) | 6-section demo notebook |\n| [requirements.txt](file:///home/canders1/rlhf-s2/requirements.txt) | Pinned deps (all already in `newGPU`) |\n\n**Smoke tests passed:** PoetryDB live, reward function returns `1.1999` on an uplifting quatrain (sentiment 0.9999 + 0.2 bonus), poem_loader neutral prompt confirmed.\n\n**To kick off training in the notebook:** Section 4 has two cells — the first launches β=0.04 as a subprocess and plots live; the second automatically starts β=0.4 once the first finishes. Both use the `newGPU` Python interpreter directly."}