Prompt Versioning

Prompt versioning refers to the process of curating and developing different prompt templates in a managed and versionable way, and enables you to better test, compare, and evaluate the performance of different prompts. This is done on Confident AI’s Prompt Studio.

Video Summary

Loading video...

How to create and version a prompt

0 views • 0 days ago

Confident AI

100K subscribers

Subscribe

0

Share

Two Types of Prompts

There are two types of prompts you can create:

• (Single) Text Prompt: Use this when you need a straightforward, one-off prompt for simple completions.
• Prompt Message List: Use this when you need to define multiple messages with specific roles (system, user, assistant) in a conversation. This format is ideal for few-shot prompting, where you can start with a system message that sets the context, followed by several example pairs of user questions and assistant responses, and end with your actual user message. This structured approach helps guide the LLM to provide more accurate and consistent responses.

[TODO: image of two types of prompts]

Create a Text Prompt

To create your first prompt, navigate to the Prompt Studio page in your project space. Click on the button that says Create Prompt, and you will be required to name your first prompt by providing it with an alias, as well as a toggle to create the prompt as a message list. DO NOT toggle this on, we’ll do this in the next section.

Commit a Prompt Version

Now that you’ve created a prompt, you can create the first version of your prompt using the Prompt Editor within the Prompt Studio. Once you’re done writing the prompt template, enter a commit message, and click Commit new version.

Committing a new version is similar to how you would commit changes on GitHub, and allows you to create a new version of your prompt template. In the future, you can also Update a current version instead of committing a new one, but this is only possible if you haven’t used the version before.

Include Dynamic Variables

You can include variables that can be interpolated dynamically in your LLM application later on. There are four ways currently for you to include variables:

• {variable}
• {{ variable }}
• {{variable}}
• ${variable}

For example, the {variable} interpolation type must follow these rules:

1. No spaces in the variable name
2. Exactly one space between the braces and the variable name

For example:

# ✅ Correct usage:
"Hi, my name is {name}."
"The temperature is {temperature} degrees."
"User input: {user_input}"

# ❌ Incorrect usage:
"Hi, my name is {variable name}."  # Spaces in variable name
"Hi, my name is {variable_name }." # Extra spaces around variable name

You can select which variable interpolation type you wish to use in the Prompt Editor.

Create a Prompt Message List

This is exactly the same as creating a prompt in the previous step, but this time toggle ON the “message list” option to create a prompt as a list of messages.

Commit a Prompt Version

A prompt version for a prompt message list, is a list of prompt messages, not individual prompts. Each prompt message has:

• "role" - either “system”, “user”, “assistant”, or “developer”
• "content" - same as a single prompt version

When you create a new version, you’re creating a list of prompts. To add a new message, click on the New Message button at the end of the list, and edit your prompt message accordingly. When you’re done, click Commit new version.

Committing a new version is similar to how you would commit changes on GitHub, and allows you to create a new version of your prompt template. In the future, you can also Update a current version instead of committing a new one, but this is only possible if you haven’t used the version before.

Include Dynamic Variables

Dynamic variables can be included in the prompt template of each prompt message the same way as how you would include a variable in for a single text prompt, which you can learn about in more detail above.

One thing to note is, whatever variable you include in one promtp message will be interpolated the same way as all other messages in the same prompt version.

Preview Prompts

You can also click on the Preview button to preview each prompt template in markdown mode.

Quick Disclaimer

Once a prompt version has been used in an evaluation, it becomes immutable - meaning you cannot update it anymore. This is a deliberate design choice in Confident AI to maintain data integrity and ensure evaluation results remain tied to the exact prompt version that generated them. The immutability of evaluated prompts helps in:

1. Maintaining a clear audit trail of which prompt version produced which results
2. Enabling reliable comparison between different prompt versions
3. Preventing accidental modifications that could invalidate evaluation data

If you need to make changes to an evaluated prompt version, you’ll need to commit a new version. This approach ensures you can iterate on your prompts while preserving the history and integrity of your evaluation results.