RLHF is user research with a feedback loop
Most PMs read about RLHF and file it under "ML team problem." That is a mistake. RLHF is the most product-shaped part of the modern AI stack. It is how user preference becomes model behavior. If you have ever written a survey, you already understand half of it.
The three artifacts you actually own
1. The rubric
Before any human labels anything, someone has to decide what "good" means. That is a product decision, not an ML decision. A good rubric is concrete, ordered, and disagreement-resistant. "Helpful" is not a rubric. "Answers the user's question in under 120 words, cites a source, and does not invent facts" is a rubric.
2. The preference dataset
Pairs of responses where a human picked the better one. The quality of your model ceiling is set here. The two failure modes: labelers who do not understand the rubric, and labelers who do understand it but disagree with each other. Both are PM problems.
3. The eval set
The fixed set of prompts you run before every release. Treat it like a regression suite. If your eval set is the same five prompts the founders ask, you do not have an eval set.
How to run the loop
- Ship a v0 with a written rubric, even if it is hand-tuned prompts and no fine-tuning.
- Instrument disagreements. Every thumbs-down, every regenerate, every silent abandonment is a label.
- Sample weekly. Pull 100 real conversations. Label them yourself. You will learn more in two hours than from a dashboard.
- Update the rubric before you update the model. The rubric is the source of truth; the model is the implementation.
- Re-run the eval set. If a release wins on the rubric and loses on the eval set, the eval set is right.
The mindset shift
Old PM craft: write the spec, ship the feature, measure the funnel. New PM craft: write the rubric, ship the loop, measure the preference delta.
The model is not the product. The feedback loop is the product. RLHF is just the formal name for taking that loop seriously.
— Aadhar