Review: Reinforcement Learning Basics
Reinforcement learning is a mathematical framework.
Demystify the reinforcement learning model, it’s a open-ended model using reward function to optimize agent to solve complex task in target environment.
Step by Step
For RLHF training method, here are three core steps:
- Pretraining a language model
- Gathering data(问答数据) and training a reward model
- Fine-tuning the LM with reinforcement learning
Step 1. Pretraining Language Models
Read this to learn how to train a LM:
OpenAI used a smaller version of GPT-3 for its first popular RLHF model - InstructGPT.
Nowadays, RLHF is new area, there’s no answer to which model is the best for starting point of RLHF and using expensive augmented data to fine-tune is not necessarily.
Step 2. Reward model training
In reward model, we integrate human preferences into the system.
