README

Setup

1. Create a Conda Environment Use the following command to create and activate a new environment for the DPO training:

   bashCopy

   conda create -n dpo_env python=3.10
   conda activate dpo_env
   

2. Install Dependencies After activating the environment, install all required dependencies by running:

   bashCopy

   pip install -r requirements.txt
   

3. Constructing DPO Data Provide the data for DPO trainng as follow: the jsonl file contains json object (each line).

   jsonCopy

   {
        {"prompt": "Prompt"},
        {"chosen": "The chosen response"},
        {"rejected": "The rejected response"},
   }
   

4. Training Once the environment is ready and the model paths are configured, run the DPo training by executing the following script:

   Copy

   DATA_PATH="/path/to/preference/data"
   SFT_MODEL="/path/to/sft/model"
   OUTPUT_DIR="/path/to/output"
   bash ./scripts/dpo_qwencoder.sh