OpenTelemetry

OpenTelemetry is an open-source observability framework that allows teams to collect, analyze, and visualize telemetry data.

Confident AI provides a ConfidentSpanExporter that allows teams to export traces to the Observatory .

Quickstart


deepeval login --confident-api-key YOUR_API_KEY

Add ConfidentSpanExporter to your OpenTelemetry tracer:

example.py


import time
import json
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
 
from deepeval.tracing.otel.exporter import ConfidentSpanExporter
 
# Set up tracer provider
tracer_provider = trace.get_tracer_provider()
if not isinstance(tracer_provider, TracerProvider):
    trace.set_tracer_provider(TracerProvider())
 
# Add confident span exporter wrapped in batch span processor
tracer_provider.add_span_processor(BatchSpanProcessor(ConfidentSpanExporter()))
 
# Get tracer
tracer = trace.get_tracer("deepeval_tracer")
 
def llm(input: str) -> str:
    with tracer.start_as_current_span("llm_span") as span:
        span.set_attribute("confident.span.type", "llm")
        span.set_attribute("confident.llm.model", "gpt-3.5-turbo")
        span.set_attribute("confident.llm.attributes.input", [
            json.dumps({"role": "system", "content": "You are a helpful assistant."}),
            json.dumps({"role": "user", "content": input})
        ])
        time.sleep(0.5)
        span.set_attribute("confident.llm.attributes.output", "Hello world")
    
    return "Hello world"
 
def my_app(input: str):
    with tracer.start_as_current_span("my_app") as span:
        span.set_attribute("confident.span.input", input)
        res = llm(input)
        span.set_attribute("confident.span.output", res)
 
my_app("Hi")

Run the script:


python example.py

This will create a trace in the Observatory, with the span name my_app.

Attributes mapping

Confident AI uses confident.* namespace to map OpenTelemetry span attributes with llm tracing data model.

Trace attributes

Trace attributes are set in the span attributes.

confident.trace.name (of type str) used for updating trace name.
confident.trace.tags (of type list[str]) used for updating trace tags.
confident.trace.thread_id (of type str) used for updating trace thread id.
confident.trace.user_id (of type str) used for updating trace user id.


with tracer.start_as_current_span("custom_span") as span:
    # your code
    span.set_attribute("confident.trace.name", "test_trace")
    span.set_attribute("confident.trace.tags", ["tag1", "tag2"])
    span.set_attribute("confident.trace.thread_id", "123")
    span.set_attribute("confident.trace.user_id", "456")

Trace input and output

You can set trace input and output at runtime using the following attributes:

confident.trace.input (of type Any) used for updating trace input.
confident.trace.output (of type Any) used for updating trace output.


with tracer.start_as_current_span("custom_span") as span:
    # your code
    span.set_attribute("confident.trace.input", input)
    span.set_attribute("confident.trace.output", output)

💡

It is recommended to set trace attributes once in any span in a trace lifecycle. The value of the specific attribute will be updated to the latest value if set in multiple spans.

Spans

Span attributes start with confident.{span_type}.* namespace. Span type can be one of the following:

span: Default (custom) span type.
llm: LLM span.
agent: Agent span.
retriever: Retriever span.
tool: Tool span.

Given below is the sample codes for setting attributes for different span types.

Default Span


def meta_agent(input: str):
    with tracer.start_as_current_span("meta_agent") as span:
        span.set_attribute("confident.span.input", input)
        span.set_attribute("confident.span.output", input)
        span.set_attribute("confident.span.error", "Error")
        span.set_attribute("confident.span.feedback", json.dumps({
            "rating": 1,
            "expected_output": "Expected output",
            "explanation": "Explanation"
        }))
        span.set_attribute("confident.span.metric_collection", "My metrics")
        span.set_attribute("confident.trace.attributes", json.dumps({
            "name": "test_trace",
            "tags": ["tag1", "tag2"],
            "thread_id": "123",
            "user_id": "456"
        }))

LLM Span


def llm_agent(input: str):
 
    with tracer.start_as_current_span("llm_span") as span:
        span.set_attribute("confident.span.type", "llm")
        span.set_attribute("confident.llm.model", "gpt-3.5-turbo")
        span.set_attribute("confident.llm.cost_per_input_token", 0.01)
        span.set_attribute("confident.llm.cost_per_output_token", 0.02)
        span.set_attribute("confident.llm.attributes.input", [
            json.dumps({"role": "user", "content": input}),
            json.dumps({"role": "system", "content": "You are a helpful assistant."})
        ])
        span.set_attribute("confident.llm.attributes.output", "Hello world")
        span.set_attribute("confident.llm.attributes.output_token_count", 10)
        span.set_attribute("confident.llm.attributes.input_token_count", 10)

Agent Span


def agent_span(input: str):
 
    with tracer.start_as_current_span("agent_span") as span:
        span.set_attribute("confident.span.type", "agent")
        span.set_attribute("confident.agent.name", "agent_span")
        span.set_attribute("confident.agent.available_tools", ["llm_agent", "retriever_span", "tool_span"])
        span.set_attribute("confident.agent.agent_handoffs", ["llm_agent", "retriever_span", "tool_span"])
        span.set_attribute("confident.agent.attributes.input", json.dumps({"input": input}))
        span.set_attribute("confident.agent.attributes.output", json.dumps({"output": input}))

Retriever Span


def retriever_span(input: str):
    
    with tracer.start_as_current_span("retriever_span") as span:
        span.set_attribute("confident.span.type", "retriever")
        span.set_attribute("confident.retriever.embedder", "embedder")
        span.set_attribute("confident.retriever.attributes.embedding_input", input)
        span.set_attribute("confident.retriever.attributes.retrieval_context", ["asd", "asd"])
        span.set_attribute("confident.retriever.attributes.top_k", 10)
        span.set_attribute("confident.retriever.attributes.chunk_size", 10)

Tool Span


def tool_span(input: str):
 
    with tracer.start_as_current_span("tool_span") as span:
        span.set_attribute("confident.span.type", "tool")
        span.set_attribute("confident.tool.name", "tool name")
        span.set_attribute("confident.tool.description", "tool description")
        span.set_attribute("confident.tool.attributes.input_parameters", json.dumps({"input": input}))
        span.set_attribute("confident.tool.attributes.output", json.dumps({"output": input}))

LLM test case

LLM Test Case attributes can be used to unit test interactions within your LLM application. It can be set in any span type.

Given below is the example of running online evaluation for a span.


with tracer.start_as_current_span("confident_evaluation") as span:
    input = "What is the capital of France?"
    output = my_llm_app(input) # your LLM application
    
    span.set_attribute('confident.span.metric_collection', "<your_metric_collection>")
    span.set_attribute('confident.span.llm_test_case.input', input)
    span.set_attribute('confident.span.llm_test_case.actual_output', output)

💡

Make sure you have metric collection created on the platform for running online evaluation on this test case.

LLM test case attributes mapping:

Attribute	Type	Required	Description
`confident.span.llm_test_case.input`	`str`	Yes	Test case input
`confident.span.llm_test_case.actual_output`	`str`	Yes	Test case actual output
`confident.span.llm_test_case.expected_output`	`str`	No	Test case expected output
`confident.span.llm_test_case.context`	`list[str]`	No	Test case context
`confident.span.llm_test_case.retrieval_context`	`list[str]`	No	Test case retrieval context
`confident.span.llm_test_case.tools_called`	`str`	No	Should be stringified json of type Tools Called
`confident.span.llm_test_case.expected_tools`	`str`	No	Should be stringified json of type Expected Tools

Default span

Attribute	Type	Required	Description
`confident.span.input`	`Any`	No	Span input
`confident.span.output`	`Any`	No	Span output
`confident.span.error`	`str`	No	Span error
`confident.span.metric_collection`	`str`	No	Span metric collection

LLM span

To create a LLM span, set the confident.span.type to llm.

LLM span attributes: reference

Attribute	Type	Required	Description
`confident.llm.model`	`str`	No	LLM model
`confident.llm.cost_per_input_token`	`float`	No	Cost per input token
`confident.llm.cost_per_output_token`	`float`	No	Cost per output token

LLM attributes: reference

Attribute	Type	Required	Description
`confident.llm.attributes.input`	`str`, `list[str]`	Yes	LLM Span input (parsed to `list[dict]` if `list[str]` is provided)
`confident.llm.attributes.output`	`Any`	Yes	LLM Span output
`confident.llm.attributes.input_token_count`	`int`	No	LLM Span input token count
`confident.llm.attributes.input_token_count`	`int`	No	LLM Span input token count

Agent span

To create an agent span, set the confident.span.type to agent.

Agent span attributes: reference

Attribute	Type	Required	Description
`confident.agent.name`	`str`	Yes	Agent span name
`confident.agent.available_tools`	`list[str]`	No	Agent span available tools
`confident.agent.agent_handoffs`	`list[str]`	No	Agent span agent handoffs

Agent attributes: reference

Attribute	Type	Required	Description
`confident.agent.attributes.input`	`Any`	Yes	Agent span input
`confident.agent.attributes.output`	`Any`	Yes	Agent span output

Tool span

To create a tool span, set the confident.span.type to tool.

Tool span attributes: reference

Attribute	Type	Required	Description
`confident.tool.name`	`str`	Yes	Tool span name
`confident.tool.description`	`str`	No	Tool span description

Tool attributes: reference

Attribute	Type	Required	Description
`confident.tool.attributes.input_parameters`	`str`	No	Tool span input parameters (parsed to `dict` if `str` is provided)
`confident.tool.attributes.output`	`Any`	No	Tool span output

Retriever span

To create a retrieval span, set the confident.span.type to retriever.

Retriever span attributes: reference

Attribute	Type	Required	Description
`confident.retriever.embedder`	`str`	Yes	Retrieval span embedder model

Retriever attributes: reference

Attribute	Type	Required	Description
`confident.retriever.attributes.embedding_input`	`str`	Yes	Retrieval span embedding input
`confident.retriever.attributes.retrieval_context`	`list[str]`	Yes	Retrieval span retrieval context
`confident.retriever.attributes.top_k`	`int`	No	Retrieval span top k
`confident.retriever.attributes.chunk_size`	`int`	No	Retrieval span chunk size