Annotate Datasets
Confident AI provides your team a centralized place to edit and manage evaluation datasets online. Datasets also offer revision history and on-demand daily backups.
Annotating datasets is a flexible; it has support for custom columns, storing free-form JSON in the additional_metadata field for each golden, and any comments your domain experts, technical or not, may have.
If you already have a dataset, you can either upload a CSV file of goldens of via API requests (through deepeval), which is covered in the next section on importing datasets.
Video Summary
How to Create and Annotate Datasets
Create a Dataset
To create your first dataset, navigate to the Datasets page in your project space. There, you’ll see a button that says Create Dataset, and you will be required to name your first dataset by providing it with an alias. This alias will be used to identify which dataset will be used for evaluation later on.
Create Golden(s)
Now that you’ve created a dataset, you can create a golden using the Goldens Editor within your dataset that will later be converted to an LLMTestCase during evaluation time (we’ll talk more about this later).
You can open the Goldens Editor by clicking on the Create Golden button in the Dataset Editor page, which will allow you to manually fill in your golden fields.
input is a required field for all goldens.If you’re not sure what to include in your goldens, simply enter the inputs you’re currently prompting your LLM application with when eyeballing outputs. You’ll be able to automate this process by creating a list of goldens out of inputs.
We HIGHLY RECOMMEND that you DO NOT pre-fill actual_output fields or run evaluations on pre-computed datasets. Your LLM application should always generate fresh outputs during evaluation to properly test the latest changes.
Edit Golden(s)
There are two ways to edit a golden:
- Click directly on any cell in the table to edit that field inline
- Click the pencil icon that appears when hovering over the leftmost side of a row to open the Goldens Editor
The Goldens Editor modal provides a larger view where you can edit all fields at once. Remember to click the Save button after making your changes.
If you need more time to complete editing a golden, you can mark it as unfinalized by setting the “Finalized” field to “No”. Unfinalized goldens are automatically excluded from evaluations, allowing you to safely revisit and complete them later.
Adding Custom Columns
You can add custom columns to your dataset to store additional information that doesn’t fit into the default golden fields. To add a custom column:
- Click the Add Column button in the Dataset Editor
- Enter a unique name for your column (it cannot match any default field names like “Input” or “Actual Output”)
You can also add a new golden in the Golden Editor and when uploading a dataset via CSV.
The additional_metadata field in a golden can often serve as an alternative
to custom columns for storing metadata. This field accepts any dictionary of
values, offering flexibility while custom columns provide more structure and
type safety.
Enable Backups & Revision History (recommended)
We recommend enabling dataset backups and revision history to protect your data and track changes. When enabled:
- Automatic backups are created daily at midnight UTC for data recovery
- Revision history tracks who made changes, what was changed, and when
- Changes will appear in the revision history after the first backup completes
To enable these features, go to the Revision History tab and click on the Enable Backups & Revision History button.