docs/development/utilities.md
Important: despite the name prompt engineering UI, this tool allows testers to test any kind of parameters that are exposed by developers. Prompt is one kind of param. There can be other type of params that testers can tweak (e.g. top_k, temperature...).
In the development process, developers typically build the pipeline. However, for use
cases requiring expertise in prompt creation, non-technical members (testers, domain experts) can be more
effective. To facilitate this, kotaemon offers a user-friendly prompt engineering UI
that developers integrate into their pipelines. This enables non-technical members to
adjust prompts and parameters, run experiments, and export results for optimization.
As of Sept 2023, there are 2 kinds of prompt engineering UI:
For simple pipeline, the supported client project workflow looks as follow:
$ kotaemon promptui export <module.path.piplineclass> --output <path/to/config/file.yml>$ kotaemon promptui run <path/to/config/file.yml>The prompt engineering UI prominently involves from step 2 to step 7 (step 1 is normally done by the developers, while step 7 happens exclusively in Excel file).
Command:
$ kotaemon promptui export <module.path.piplineclass> --output <path/to/config/file.yml>
where:
<module.path.pipelineclass> is a dot-separated path to the pipeline. For example, if your pipeline can be accessed with from projectA.pipelines import AnsweringPipeline, then this value is projectA.pipelines.AnswerPipeline.<path/to/config/file.yml> is the target file path that the config will be exported to. If the config file already exists, and contains information of other pipelines, the config of current pipeline will additionally be added. If it contains information of the current pipeline (in the past), the old information will be replaced.By default, all params in a pipeline (including nested params) will be export to the configuration file. For params that you do not wish to expose to the UI, you can directly remove them from the config YAML file. You can also annotate those param with ignore_ui=True, and they will be ignored in the config generation process. Example:
class Pipeline(BaseComponent):
param1: str = Param(default="hello")
param2: str = Param(default="goodbye", ignore_ui=True)
Declared as above, and param1 will show up in the config YAML file, while param2 will not.
developers can further edit the config file in this step to get the most suitable UI (step 4) with their tasks. The exported config will have this overall schema:
<module.path.pipelineclass1>:
params: ... (Detail param information to initiate a pipeline. This corresponds to the pipeline init parameters.)
inputs: ... (Detail the input of the pipeline e.g. a text prompt. This corresponds to the params of `run(...)` method.)
outputs: ... (Detail the output of the pipeline e.g. prediction, accuracy... This is the output information we wish to see in the UI.)
logs: ... (Detail what information should show up in the log.)
The inputs section have the overall schema as follow:
inputs:
<input-variable-name-1>:
component: <supported-UI-component>
params: # this section is optional)
value: <default-value>
<input-variable-name-2>: ... # similar to above
params:
<param-variable-name-1>: ... # similar to those in the inputs
The list of supported prompt UI and their corresponding gradio UI components:
COMPONENTS_CLASS = {
"text": gr.components.Textbox,
"checkbox": gr.components.CheckboxGroup,
"dropdown": gr.components.Dropdown,
"file": gr.components.File,
"image": gr.components.Image,
"number": gr.components.Number,
"radio": gr.components.Radio,
"slider": gr.components.Slider,
}
The outputs are a list of variables that we wish to show in the UI. Since in Python, the function output doesn't have variable name, so output declaration is a little bit different than input and param declaration:
outputs:
- component: <supported-UI-component>
step: <name-of-pipeline-step>
item: <jsonpath way to retrieve the info>
- ... # similar to above
where:
The logs show a list of sheetname and how to retrieve the desired information.
logs:
<logname>:
inputs:
- name: <column name>
step: <the pipeline step that we would wish to see the input>
variable: <the variable in the step>
- ...
outputs:
- name: <column name>
step: <the pipeline step that we would wish to see the output>
item: <how to retrieve the output of that step>
Command:
$ kotaemon promptui run <path/to/config/file.yml>
This will generate an UI as follow:
where:
Upon clicking export, the users can download Excel file.
Chat pipeline workflow is different from simple pipeline workflow. In simple pipeline, each Run creates a set of output, input and params for users to compare. In chat pipeline, each Run is not a one-off run, but a long interactive session. Hence, the workflow is as follow: