docs/01 - Chat Tab.md
Used to have multi-turn conversations with the model.
The main action buttons are:
The hover menu (☰) that appears over the chat area contains:
Allows you to switch between the current and previous conversations with the current character, or between the current and previous instruct conversations (if in "instruct" mode). The available buttons are:
A search field is also available to filter conversations by name.
The sidebar (toggled via "Show controls") contains:
<|character|> is a placeholder that gets replaced with the bot name, and <|prompt|> is a placeholder that gets replaced with the full chat prompt.The most important input field. It defines how the chat prompt is formatted. There are 3 options: chat, chat-instruct, and instruct. It is worth going into more detail about this because it seems to not be obvious to a lot of people.
There are two kinds of models: base models, like Llama and GPT-J, and fine-tuned models, like Alpaca and Vicuna. Fine-tuned models are trained starting from base models, most often with the goal of getting the model to understand and respond to instructions just like ChatGPT does. Let's call such models instruction-following models.
Each instruction-following model was trained on a specific prompt format, and you have to use that exact prompt format if you want the model to follow your instructions as accurately as it can.
As an example, this is the Alpaca format:
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Hi there!
### Response:
Hello! It's nice to meet you. What can I help with?
### Instruction:
How are you?
### Response:
I'm doing well, thank you for asking! Is there something specific you would like to talk about or ask me? I'm here to help answer any questions you may have.
This format is characterized by a context string at the top, and alternating turns where each user input starts with ### Instruction: and each bot turn starts with ### Response:. There are also weirder formats, like the one used by the Llama-2-chat models released by Meta AI:
[INST] <<SYS>>
Answer the questions.
<</SYS>>
Hi there! [/INST] Hello! It's nice to meet you. What can I help with? </s><s>[INST] How are you? [/INST] I'm doing well, thank you for asking! Is there something specific you would like to talk about or ask me? I'm here to help answer any questions you may have.
In this format, there are special tokens at the end of each bot reply (</s>, the end of sequence token, and <s>, the beginning of sequence token); no new lines separating the turns; and the context string is written between <<SYS>> and <</SYS>>. Despite the intimidating look of this format, the logic is the same: there are user turns and bot turns, and each one appears in a specific place in the template.
It is important to emphasize that instruction-following models have to be used with the exact prompt format that they were trained on. Using those models with any other prompt format should be considered undefined behavior. The model will still generate replies, but they will be less accurate to your inputs.
Now that an instruction-following model is defined, we can move on to describing the 3 chat modes.
Used for talking to the character defined under "Character" tab using a simple chat prompt in this format:
Chiharu Yamada's Persona: Chiharu Yamada is a young, computer engineer-nerd with a knack for problem solving and a passion for technology.
You: Hi there!
Chiharu Yamada: Hello! It's nice to meet you. What can I help with?
You: How are you?
Chiharu Yamada: I'm doing well, thank you for asking! Is there something specific you would like to talk about or ask me? I'm here to help answer any questions you may have.
There are 3 adjustable parameters in the "Character" tab being used in this prompt:
Additionally, the Greeting string appears as the bot's opening message whenever the history is cleared.
The "Chat" option should typically be used only for base models or non-instruct fine tunes, and should not be used for instruction-following models.
Used for talking to an instruction-following model using the prompt format defined under "Parameters" > "Instruction template". Think of this option as an offline ChatGPT.
The prompt format is defined by the Instruction template parameter in "Parameters" > "Instruction template", which represents a Jinja2 template.
Note that when you load a model in the "Model" tab, the web UI will try to automatically detect its instruction template (if any) from the model metadata (e.g. tokenizer_config.json or GGUF metadata), and will update the values under "Parameters" > "Instruction template" accordingly. You should check the model card on Hugging Face to see if you are using the correct prompt format.
As said above, instruction-following models are meant to be used with their specific prompt templates. The chat-instruct mode allows you to use those templates to generate a chat reply, thus mixing Chat and Instruct modes (hence the name).
It works by creating a single instruction-following turn where a command is given followed by the regular chat prompt. Here is an example in Alpaca format:
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Continue the chat dialogue below. Write a single reply for the character "Chiharu Yamada".
Chiharu Yamada's Persona: Chiharu Yamada is a young, computer engineer-nerd with a knack for problem solving and a passion for technology.
You: Hi there!
Chiharu Yamada: Hello! It's nice to meet you. What can I help with?
You: How are you?
### Response:
Chiharu Yamada:
Here, the command is
Continue the chat dialogue below. Write a single reply for the character "Chiharu Yamada".
Below this command, the regular chat prompt is added, including its Context string and the chat history, and then the user turn ends. The bot turn starts with the "Character's name" string followed by :, thus prompting the instruction-following model to write a single reply for the character.
Note that you can get creative: instead of writing something trivial like "Write a single reply for the character", you could add more complex instructions like
This is an adventure game, and your task is to write a reply in name of "<|character|>" where 3 options are given for the user to then choose from.
And it works:
This defines the visual style of the chat UI. Each option is a CSS file defined under textgen/css/chat_style-name.css, where "name" is how this style is called in the dropdown menu. You can add new styles by simply copying chat_style-cai-chat.css to chat_style-myNewStyle.css and editing the contents of this new file. If you end up with a style that you like, you are highly encouraged to submit it to the repository.
The styles are only applied to chat and chat-instruct modes. Instruct mode has its separate style defined in textgen/css/html_instruct_style.css.
This menu is a built-in extension defined under textgen/extensions/gallery. It displays a gallery with your characters, and if you click on a character, it will be automatically selected in the Character tab.