Skip to main content
See HTTP URLs and authentication and the API reference for complete request and response formats. Annotation Queues organize Traces into review workflows so Annotations can be applied in bulk in AI Studio. To define annotations and apply them individually, through the API and SDK, or the CLI, see Annotations.

​
Create Annotation Queues

Annotation Queues help you organize and apply annotations effectively to relevant incoming Traces.
To create an Annotation Queue, head to AI Studio > Annotation Queue.Choose Create Annotation Queue.The following fields are configurable:
  • The Name of the queue
  • The Description of the Annotation Queue
  • The Project that owns the queue, chosen from Select a project. The queue shows the Annotations defined on that project.

​
Fill Annotation Queues

Once a queue exists, fill it with the Traces to review. Traces can be added automatically or manually.
Use Trace Automations to route Traces into a queue based on configured rules. Add an Add to Annotation Queue action to an automation and select the target queue. As matching Traces arrive, they are added to the queue without manual effort, which keeps a steady stream of relevant Traces ready for review.

​
Use Annotation Queues

Open an Annotation Queue to step through its Traces one at a time in the review screen. The screen is divided into three panels:
  • Left: details for the selected Trace.
    • Inputs: the variables mapped to inputs, when configured.
    • Metrics: latency, cost, and token usage.
    • Task: the model, provider, and other configuration parameters.
    The header shows the current position, the total number of items in the queue, and how many have already been reviewed.
  • Center: the full interaction for the selected Trace.
  • Right: the Annotations panel with the Annotations defined on the queue’s project, such as a rating with categorical buttons or an open comment field. Selecting a value saves immediately and marks the Trace as reviewed.
Navigate between items with K (previous) and J (next), or use the up and down buttons at the top left. When a data point is worth reusing, select Add to dataset to send the Trace to a Dataset for use in a future Experiment.
Adding a Trace to a Dataset copies annotations only when Copy annotations is selected. The annotation values stay on the originating Trace as well, where they remain queryable via the Toqan Router MCP.
Evaluator results shown in the review screen can also be corrected, not just annotated. See Correct an Evaluator Result on the Traces page for the full flow.

​
Delete or Clear Annotation Queues

Select the menu on a queue row and choose Delete to remove the queue together with its items. The same menu offers Edit for the queue’s name, description, and project. To empty a queue while keeping it and its configuration, clear it:
Clearing is irreversible: every item is removed from the queue and its item count resets to zero. The Annotations already applied to those Traces stay on the Traces. See Delete All Items and Delete an Annotation Queue for the request and response details.