# Create a Workflow Source: https://docs.deeprails.com/api-reference/defend/create-a-workflow /api-reference/openapi.yaml post /defend Use this endpoint to create a new guardrail workflow by specifying guardrail thresholds, an improvement action, and optional extended capabilities. Workflows correspond to a production LLM use case, and will contain evaluations and improvement attempts for inputs and outputs from that use case. The guardrails to be measured, the method of remediation, and other evaluation details must all be configured at the time of the workflow's creation

When polling this endpoint, the request body must include a `name` for the workflow, a workflow `threshold_type`, either a `custom_hallucination_threshold_values` or an `automatic_hallucination_tolerance_levels` object specifying which guardrail metrics to evaluate against and their corresponding hallucination thresholds, and an `improvement_action`. Optionally, extended AI capabilities like `web_search`, `file_search`, and `context_awareness` can be included as tools for the evaluation.

Workflows created with this endpoint will be displayed in the console and can be accessed at other endpoints using the `workflow_id` returned.

The type of the workflow determines how hallucination thresholds are configured:
- `automatic` - Uses internal threshold logic based on tolerance level. The `automatic_hallucination_tolerance_levels` dictionary must be specified as `low`, `medium`, or `high` to indicate how strict the thresholds should be.
- `custom` - Allows the user to set custom thresholds (0.0-1.0) for each metric using the `custom_hallucination_threshold_values` dictionary.

The improvement action determines the remediation step for events where one or more metric evaluations fail below the threshold.
- `regen` - Re-runs the user's prompt with minor variance
- `fixit` - Directly addresses shortcomings using the guardrail failure rationale(s)
- `do_nothing` - No improvement is attempted by DeepRails

When you create a workflow, you'll receive a workflow ID. Use this ID to submit new events for evaluation and remediation for the defend workflow. # Retrieve a Workflow's Details Source: https://docs.deeprails.com/api-reference/defend/retrieve-a-workflows-details /api-reference/openapi.yaml get /defend/{workflow_id} Use this endpoint to retrieve the details for a specific defend workflow Workflows will eventually contain many events, each with its own set of evaluations and improvement attempts. Many of the details of these events can be viewed in the Defend Data tab in the DeepRails console, but you can also see configuration details when polling this endpoint.

The `status` field indicates whether the workflow is active and currently accepting events. If the workflow status is `inactive`, no events submitted for the workflow will be processed.

The `stats` field contains counts of the events processed by the workflow, which can be used to determine the efficacy of the workflow. The `events` field lists the details of the most recent events processed by the workflow in a format similar to what is returned when you retrieve an event's details. # Retrieve an Event's Details Source: https://docs.deeprails.com/api-reference/defend/retrieve-an-events-details /api-reference/openapi.yaml get /defend/{workflow_id}/events/{event_id} Use this endpoint to retrieve a specific event of a guardrail workflow Workflow events can include a whole series of improvement attempts and corresponding evaluations when the initial evaluation fails one or more metrics. Poll this endpoint for these details, plus the completion `status`, whether the output needed to be `filtered` out, the `improved_model_output` if it needed remediation, and more.

The `filtered` field is set to `true` when the event fails one or more metrics on the most recent evaluation and is set to `false` if all metric evaluations were above their thresholds. If `filtered` is `true`, then an improvement attempt will begin immediately after the initial evaluation concludes.

The `evaluation_history` field is an array of the details for each evaluation performed for the event. It can be used to track the progress of the event and see how DeepRails improved your model output over time.

The `analysis_of_failures` and `key_improvements` fields summarize the failings of the model output and the relevant changes of the improvement attempt at each step. These fields are returned for each evaluation, except the final one, and summaries of all of failures and improvements are returned in the `analysis_of_failures` and `key_improvements` fields of the event response. # Stream a Workflow Event (Optimized) Source: https://docs.deeprails.com/api-reference/defend/stream-a-workflow-event-optimized /api-reference/openapi.yaml post /defend/{workflow_id}/events?stream=true Use this endpoint to submit a model input and output pair to a workflow for evaluation with streaming responses.

Why use streaming?

By default, when you submit a workflow event, the response is delivered all at once after processing completes. Streaming lets you start receiving outputs immediately, before processing is entirely finished — perfect for chat interfaces or any application where perceived latency matters.

How it works

Add stream=true to your request, and you'll receive a stream of Server-Sent Events instead of a single JSON response.

Streaming is supported on all run modes except precision\_max and precision\_max\_codex.

Note: super\_fast does not support Web Search or File Search. If your workflow has these capabilities enabled, either switch to a run mode that supports them (e.g. fast, precision, precision\_codex) or edit the workflow to disable Web Search / File Search.

Defend evaluates the model output against your workflow's guardrails. If the output passes, it streams back the original. If it fails, Defend improves it and streams back the improved version. Either way, you receive a single stream of token events containing the final output — just forward them to your end-user.

Event types

– token: These are the output chunks. Stream them directly to your end-user as they arrive.

– error: This means something went wrong, and will include a message.

# Submit a Workflow Event Source: https://docs.deeprails.com/api-reference/defend/submit-a-workflow-event /api-reference/openapi.yaml post /defend/{workflow_id}/events Use this endpoint to submit a model input and output pair to a workflow for evaluation Workflow events represent individual LLM calls in your production use case. Whenever you receive a new LLM response, you can submit it, along with the input, to the workflow for evaluation and remediation. The `model_input` field in the request must be a dictionary (containing at least a `system_prompt` field or a `user_prompt` field), and the request must also include the `model_used` to generate the output (Ex. `gpt-5-mini`), the selected `run_mode` that determines speed/accuracy/cost for evaluation, and optionally a `nametag` for the workflow event.

The run mode determines which models power the evaluation (fastest to most thorough):
- `super_fast` - High-speed lightweight checks
- `fast` - Balanced speed and accuracy (default)
- `precision` - Deep multi-model analysis
- `precision_codex` - Code-optimized deep analysis
- `precision_max` - Exhaustive multi-pass verification
- `precision_max_codex` - Ultimate code-aware verification

Note: `super_fast` does not support Web Search or File Search capabilities. Requests using this mode on workflows with Web Search or File Search enabled will be rejected.

The event's evaluations will be run with the guardrail metrics and improvement action configured in its associated workflow.

When you create a workflow event, you'll receive an event ID. Use this ID to track the event's progress and retrieve all evaluations and improvement results. # Update a Workflow Source: https://docs.deeprails.com/api-reference/defend/update-a-workflow /api-reference/openapi.yaml put /defend/{workflow_id} Use this endpoint to update an existing defend workflow if its details change. This endpoint can update the workflow's `name`, `description`, `threshold_type`, `automatic_hallucination_tolerance_levels`, `custom_hallucination_threshold_values`, and `improvement_action` when needed. You can also add or remove extended AI capabilities like `web_search` and `file_search`. Only fields provided in the request body will be updated. # Upload a File Source: https://docs.deeprails.com/api-reference/files/upload-a-file /api-reference/openapi.yaml post /files/upload Use this endpoint to upload a file to the DeepRails API Files used for the file search feature in Defend must be uploaded to DeepRails here before being added to workflows. The request body must include the `file` to upload as binary content formatted as a form-data.

When you upload a file, you'll receive a file ID. Use this ID to add the file to a workflow for evaluation. Additionally, you'll receive a file path corresponding to the s3 bucket where the file is stored. # Create a Monitor Source: https://docs.deeprails.com/api-reference/monitor/create-a-monitor /api-reference/openapi.yaml post /monitor Use this endpoint to create a new monitor to evaluate model inputs and outputs using guardrails Monitors are assigned to a production LLM use case and are used to evaluate model inputs and outputs against guardrail metrics for observability over your workflows. In order to make one, the request body must include a monitor `name` and a set of `guardrail_metrics` used for evaluation. Optionally, extended capabilities, like `web_search`, `file_search`, and `context_awareness`, and a `description` can be included.

Once created, the monitor can be used to log and evaluate model outputs against guardrail metrics for LLM use in your production environment.

When you create a monitor, you'll receive a monitor ID. Use this ID to submit new events to the Monitor for tracking usage, cost, latency, and evaluation scores. # Retrieve a Monitor Event's Details Source: https://docs.deeprails.com/api-reference/monitor/retrieve-a-monitor-events-details /api-reference/openapi.yaml get /monitor/{monitor_id}/events/{event_id} Use this endpoint to retrieve the details of a specific monitor event Monitor events include a detailed set of data from the corresponding evaluation. Poll this endpoint for these details, plus the `status`, input parameters, and timestamps.

When waiting on the completion of an event, you can poll this endpoint until the `status` is `completed`, then all the evaluation details will be available. # Retrieve a Monitor's Details Source: https://docs.deeprails.com/api-reference/monitor/retrieve-a-monitors-details /api-reference/openapi.yaml get /monitor/{monitor_id} Use this endpoint to retrieve the details and evaluations associated with a specific monitor Monitors track all the data from each of the events submitted to them. This endpoint returns comprehensive information for the monitor including recent event history and observability stats, plus configuration details like `name`, `description`, and its `status`.

The `stats` object in the response has counts of the evaluations in each status and an `evaluations` array containing all evaluation records.

Use the optional `limit` query parameter to control the number of returned evaluations (defaults to 10). # Submit a Monitor Event Source: https://docs.deeprails.com/api-reference/monitor/submit-a-monitor-event /api-reference/openapi.yaml post /monitor/{monitor_id}/events Use this endpoint to submit a model input and output pair to a monitor for evaluation Monitor events represent individual LLM usages in your production use case. Whenever you receive a new LLM response, submit an event to this endpoint to have it evaluated.

The request body must include a `model_input` dictionary (containing at least a `system_prompt` field or a `user_prompt` field), a `model_output` string to be evaluated, and a `guardrail_metrics` array specifying which metrics to evaluate against. Optionally, include the `model_used`, selected `run_mode`, and a human-readable `nametag`. Including the `web_search`, `file_search`, and `context_awareness` fields will allow the evaluation model to use those extended AI capabilities.

Run modes determine the models that power evaluations (fastest to most thorough):
- `super_fast` - High-speed lightweight checks
- `fast` - Balanced speed and accuracy (default)
- `precision` - Deep multi-model analysis
- `precision_codex` - Code-optimized deep analysis
- `precision_max` - Exhaustive multi-pass verification
- `precision_max_codex` - Ultimate code-aware verification

Note: `super_fast` does not support Web Search or File Search capabilities. Requests using this mode on monitors with Web Search or File Search enabled will be rejected.

When you create a monitor event, you'll receive an event ID. Use this ID to track the event's progress and retrieve the evaluation results. # Update a Monitor Source: https://docs.deeprails.com/api-reference/monitor/update-a-monitor /api-reference/openapi.yaml put /monitor/{monitor_id} Use this endpoint to update the name, status, and/or other details of an existing monitor. This endpoint can update the monitor's `name`, `description`, `status` (`active` or `inactive`), `guardrail_metrics`, `web_search`, `file_search` when needed. Only fields provided in the request body will be updated.

Setting `status` to `inactive` will stop the monitor from recording and evaluating new events. # Defend Details Source: https://docs.deeprails.com/defend/details A deep dive into how Defend's improvement tools, adaptive thresholds, and retry logic work under the hood — so you can configure workflows with confidence.
## Improvement Tools When an output fails to meet the thresholds defined in your workflow, Defend automatically applies the remediation strategy you selected at workflow creation. There are three options. ### FixIt FixIt is Defend's targeted correction strategy. Rather than discarding the original output and generating a new one, FixIt attempts to repair it. **How it works:** 1. Defend evaluates the output and identifies which guardrail metrics failed and why. The evaluation produces a per-metric rationale — a description of the specific factual errors, omissions, or adherence failures. 2. FixIt uses the original prompt, the failed output, and the evaluation rationale to construct a repair prompt. This prompt instructs the model to correct only the identified failures while preserving everything else in the response. 3. The repaired output is re-evaluated against the workflow's guardrails. If it passes, it is returned as the final output. If it fails again, the cycle repeats until either the output passes or the workflow's retry limit is reached. **When to use FixIt:** * Outputs that are mostly correct but have isolated factual errors or omissions * Use cases where preserving the original tone, format, or structure matters * Domains where targeted correction is preferable to full regeneration (e.g., long-form documents, code, structured data) **Tradeoffs:** FixIt uses more tokens per attempt than ReGen because it carries the failure rationale and the prior output into the repair prompt. In exchange, corrections tend to be more surgical and the output style stays consistent. *** ### ReGen ReGen discards the failed output and generates a fresh response from the original prompt. **How it works:** 1. Defend evaluates the output and determines it fails one or more guardrail thresholds. 2. ReGen submits the original prompt again to the model with modified sampling parameters — typically increased temperature — to introduce controlled variance. This avoids regenerating the same failure. 3. The new output is evaluated against the workflow's guardrails. If it passes, it is returned. If it fails, the cycle repeats until the output passes or the retry limit is reached. **When to use ReGen:** * Outputs where the root cause of failure is systemic (the model fundamentally got the task wrong, not just a detail) * Use cases where a fresh attempt is more likely to succeed than incremental repair * Short outputs where regeneration is cheap relative to FixIt's repair cost * Situations where preserving the original output structure is less important **Tradeoffs:** ReGen is token-efficient per attempt because it does not carry failure context. However, it provides less control over what changes between attempts — the model may fix one failure and introduce another. For structured or long-form outputs, FixIt typically produces more predictable results. *** ### Do Nothing Do Nothing records the failed output without attempting remediation. **How it works:** Defend evaluates the output, records the failure (including which metrics failed, their scores, and the evaluation rationale), and returns the failed output to your application. No repair or regeneration is attempted. **When to use Do Nothing:** * You want to monitor output quality without blocking or modifying outputs (observability mode) * Your application handles failures downstream and does not need Defend to remediate * You are baselining your current output quality before configuring active remediation * Use cases where any AI output — even imperfect — is preferable to a retry delay **Note:** Even with Do Nothing, every evaluation is logged in the DeepRails Console under the workflow's Data tab. You get full visibility into failure rates and rationales without incurring the latency cost of remediation. *** ## Retry Logic When FixIt or ReGen is active, Defend will attempt remediation up to the retry limit configured in your workflow. * The default retry limit is **3 attempts** (1 initial evaluation + 2 remediation attempts). * Each attempt is logged independently in the workflow's evaluation history, including its guardrail scores, pass/fail status, and the output at that attempt. * If the output still fails after all retries are exhausted, Defend returns the best-scoring output from all attempts, along with a `failed` status and the full retry history. * The retry limit can be configured between 1 and 5 in the workflow creation wizard. **Retry cost considerations:** Each retry attempt consumes evaluation tokens (for the guardrail scoring) and generation tokens (for FixIt's repair or ReGen's regeneration). High retry limits combined with complex outputs and precision run modes will increase per-event cost. Monitor your workflow's cost-per-event in the Console Metrics tab to calibrate. *** ## Adaptive vs. Custom Thresholds Thresholds define the score cutoff below which an output is treated as a hallucination and triggers remediation. Defend supports two threshold modes. ### Adaptive Thresholds Adaptive thresholds are set by selecting a hallucination tolerance level at workflow creation: **Low**, **Medium**, or **High**. | Tolerance | Behavior | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Low** | Strictest. Flags outputs at higher guardrail scores. Best for regulated, high-stakes, or safety-critical use cases. Expect more remediation events. | | **Medium** | Balanced. Flags outputs with meaningful quality issues while tolerating minor imperfections. Good default for most production use cases. | | **High** | Most permissive. Only flags outputs with significant quality failures. Best for use cases where speed and throughput matter more than perfect accuracy on every output. | Adaptive thresholds adjust automatically as Defend learns the distribution of your workflow's outputs over time, maintaining the selected tolerance level even as output quality changes. ### Custom Thresholds Custom thresholds let you define explicit numeric cutoff values for each guardrail metric. For example, you can require a correctness score above 0.85 while allowing completeness scores as low as 0.60 for a use case where partial answers are acceptable. Custom thresholds are available on SME and Enterprise plans. They are configured per-metric in the workflow creation wizard and can be updated in the Manage Workflows tab without recreating the workflow. *** ## Run Modes and Their Effect on Remediation The run mode you select at workflow creation affects both the accuracy of guardrail evaluations and the quality of FixIt repairs and ReGen outputs. | Run Mode | Speed | Accuracy | Best For | | ----------------------- | --------- | ------------------------ | ------------------------------------------------------------ | | **Super Fast** | Ultrafast | Basic | Maximum throughput, minimal latency, lowest-stakes use cases | | **Fast** | Fastest | Good | High-throughput, low-stakes use cases | | **Precision** | Moderate | High | Most production use cases | | **Precision Codex** | Moderate | High (code-optimized) | Code generation and technical output | | **Precision Max** | Slower | Highest | Regulated, safety-critical, or audit-grade use cases | | **Precision Max Codex** | Slower | Highest (code-optimized) | High-stakes code generation | Higher-accuracy run modes use more capable reasoning models for both evaluation and remediation. This means guardrail scores are more reliable and FixIt repairs are more accurate — but at higher token cost and latency. For most use cases, **Precision** is the right default. *** ## Monitoring Defend in the Console Every workflow produces a full audit trail visible in the DeepRails Console: * **Metrics tab:** Aggregate guardrail scores, hallucination filter rate, improvement success rate, and before/after score distributions across all events in the workflow. * **Data tab:** Event-level view of every evaluation run — input, output, guardrail scores, status (pass/fail/remediated), model, improvement attempt history, and cost metadata. * **Manage Workflows:** Workflow configuration details, throughput statistics, and threshold/tolerance settings. Thresholds and improvement tools can be updated without recreating the workflow. Use the Console to calibrate your retry limits and thresholds over time. If your improvement success rate is low (Defend is consistently exhausting retries without passing), consider switching run modes or tightening your prompt before the output reaches Defend.
# Defend Overview Source: https://docs.deeprails.com/defend/overview Defend is the most powerful tool within DeepRails' API suite. It is the real-time correction layer that ensures every model output is hallucination-free before it ever reaches your users. By combining adaptive thresholds with automated improvement tools, Defend doesn't just detect low-quality or unsafe responses, it actively fixes or regenerates them, delivering safe, accurate, and reliable outputs at scale. ## Why Defend Exists Generative AI is transformative, but enterprises are losing billions to hallucinations, compliance failures, and unreliable outputs. Most guardrail solutions only measure quality - they rarely enforce it. Defend was built to solve this gap. It continuously evaluates every response against rigorous guardrails, blocks failures at inference time, and applies automated remediation to protect both your brand and your customers. ## Key Definitions * **Guardrail Metrics:** The heart of all of our APIs and Evaluations. Defend evaluates outputs against DeepRails' research-backed General-Purpose Guardrail Metrics for correctness, completeness, adherence (context, ground truth, instruction), and comprehensive safety, with full support for Custom Guardrail Metrics available for users on SME & Enterprise plans. * **Workflow**: Defend requires you to define and create a workflow, which represents a specific LLM use case or task. A workflow bundles together the guardrails, thresholds, improvement strategy, and retry limits that will be applied consistently to every output of that use case. * **Automatic & Custom Thresholds**: A threshold is the score cutoff below which an output is treated as a hallucination. Defend supports automatic thresholds, which adapt dynamically to your selected hallucination tolerance level (low, medium, or high), and custom thresholds, where you define explicit cutoff values for each guardrail for maximum flexibility. * **Improvement Tools:** When an output fails to meet the thresholds defined in your workflow, Defend can automatically apply one of three improvement strategies: * **FixIt:** improves the flagged output using the failure rationale and surrounding context until it satisfies the workflow's guardrails or retry limits are reached. * **ReGen:** Regenerates a new output from the original prompt and parameters, introducing controlled variance to avoid repeating the same failure, and re-evaluates it against the workflow's guardrails. * **Do Nothing:** Records the failed output without attempting remediation, leaving handling of exceptions entirely to your workflow. * **Run Modes:** Run modes give developers control over the trade-off between cost and accuracy. Fast uses budget models for maximum speed; Precision offers high accuracy analysis; and Precision Codex, Precision Max, and Precision Max Codex use advanced reasoning and codex models for maximum accuracy. * **Extended AI Capabilities:** Many LLM applications have to go beyond model knowledge. DeepRails provides access to advanced tools like web and file search and context awareness for evaluations if needed. ## How Defend Works Defend operates via a simple but robust lifecycle: You define once how outputs will be judged and corrected — guardrails, thresholds, run mode, improvement tool, and retry limits. Each model completion is submitted as an event with its input/output pair, model, and optional nametag — either programmatically or in the API Playground. Defend scores the output against your workflow's guardrails, applying the thresholds you've set or selected. Passing outputs are returned immediately. Failing outputs trigger the improvement tool: FixIt iteratively improves the original output, ReGen regenerates a fresh one, or Do Nothing records the failure without intervention. Defend returns the outcome (pass/fail), guardrail scores and rationales, the final improved/regenerated output (if applicable), retry history, and detailed cost and status metadata. Every decision is logged under the workflow, and full statistics and visualizations are available in the DeepRails Console for monitoring, auditing, and optimization. ## Console Walkthrough The Defend Console brings each stage of the lifecycle to life, making it easy to configure, monitor, and optimize your workflows. ### Metrics The **Metrics** tab provides a visual, real-time view of how each workflow is performing across all guardrail metrics. It highlights how many outputs are being filtered, improved, or passed, and shows before-and-after score distributions so you can clearly see how Defend is raising quality over time. Defend Metrics showing recent data from all workflows ### Data The **Data** tab provides a detailed, event-level view of every evaluation run that passes through a workflow. It captures inputs, outputs, status, scores, and more for each attempt. This gives teams full transparency into how Defend is filtering, correcting, or regenerating outputs in practice. Defend Data tab showing evaluation runs and guardrail scores Defend Data evaluation details showing retries and improvement chain ### Manage Workflows The **Manage Workflows** tab is where you configure, track, and maintain all workflows across your organization. It provides both a high-level summary of each workflow's performance and the ability to drill into configuration details, thresholds, tolerances, and improvement strategies. From here, you can review existing workflows or launch the guided wizard to create new ones. Workflow detail screen in Manage Workflows tab #### Creating a Defend Workflow The creation wizard walks you through five simple steps to define how Defend will evaluate and remediate outputs: Start by naming your workflow and (optionally) describing what it protects against. This helps keep workflows organized and clear for your team. Defend workflow creation step 1 basic information Choose which guardrail metrics the workflow should evaluate. Multiple guardrails can be combined, including correctness, completeness, adherence, and safety. Defend workflow creation step 2 select metrics Choose which additional tools will be needed to complete evaluations for your workflow. Each will add cost, so only select a tool if your initial model uses it. Defend workflow creation step 3 select metrics Define how strict the workflow should be. Use adaptive automatic thresholds with configurable hallucination tolerance (low, medium, high), or set explicit custom thresholds for full control. Defend workflow creation step 4 configure thresholds Decide how Defend should remediate outputs that fail. Options include FixIt (improve the flagged output), ReGen (regenerate a fresh output), or Do Nothing (record the failure only). Configure the maximum number of retries for automated attempts. Defend workflow creation step 5 choose improvement action # Quickstart Guide Source: https://docs.deeprails.com/defend/quickstart Get started with the Defend API in minutes. ### Create an API Key 1. In your organization's DeepRails API Console, go to API Keys.
2. Click Create key, name it, then copy the key.
3. (Optional) Save it as the DEEPRAILS\_API\_KEY environment variable.
API Keys – placeholder
### Install the SDK ```python theme={null} pip install deeprails ``` ```ts theme={null} npm install deeprails ``` ```Ruby theme={null} gem install deeprails ``` ```go theme={null} go get github.com/deeprails/deeprails-go-sdk@latest ``` ### Create your first Defend workflow Before you can submit events to be evaluated and potentially remediated, you have to create and configure a workflow.

A workflow is an abstraction for a specific production use of Gen AI, and its configurations determine which guardrail metrics are evaluated, at what thresholds, and how issues are remediated.

#### Types of Workflows Workflows can either have custom or automatic thresholds, set by the `threshold_type`. Automatic workflows have adaptive thresholds that change as events are recorded, starting at `low`, `medium`, or `high` values in an `automatic_hallucination_tolerance_levels` dictionary. Custom workflows have static, fully customizable thresholds for each selected metric set as floating point values in a `custom_hallucination_threshold_values` dictionary.

Note that workflows set to automatic must have `automatic_hallucination_tolerance_levels` specified and `custom_hallucination_threshold_values` is not needed, and vice versa for those set to custom. ```python theme={null} from deeprails import DeepRails # Initialize (env var DEEPRAILS_API_KEY is recommended) client = DeepRails(api_key="YOUR_API_KEY") workflow_response = client.defend.create_workflow( name="Test Workflow", description="A workflow used to test the DeepRails API", threshold_type="custom", custom_hallucination_threshold_values={ "completeness": 0.85, "instruction_adherence": 0.75, }, improvement_action="fixit", max_improvement_attempts=5, web_search=True, file_search=["file_xxxxxxxxxxxx"], context_awareness=True, ) print(workflow_response) ``` ```ts theme={null} import { DeepRails } from "deeprails"; async function main() { // Initialize (env var DEEPRAILS_API_KEY is recommended) const client = new DeepRails(process.env.DEEPRAILS_API_KEY || "YOUR_API_KEY"); const workflowResponse = await client.defend.createWorkflow({ name: "Test Workflow", description: "A workflow used to test the DeepRails API", threshold_type: "custom", custom_hallucination_threshold_values: { completeness: 0.85, instruction_adherence: 0.75, }, improvement_action: "fixit", max_improvement_attempts: 5, web_search: true, file_search: ["file_xxxxxxxxxxxx"], context_awareness: true, }); console.log(workflowResponse); } main().catch(console.error); ``` ```Ruby theme={null} require "deeprails" # Initialize (env var DEEPRAILS_API_KEY is recommended) deeprails = Deeprails::Client.new( api_key: ENV["DEEPRAILS_API_KEY"] || "YOUR_API_KEY" ) workflow_response = deeprails.defend.create_workflow( name: "Test Workflow", description: "A workflow used to test the DeepRails API", threshold_type: "custom", custom_hallucination_threshold_values: { "completeness" => 0.85, "instruction_adherence" => 0.75, }, improvement_action: "fixit", max_improvement_attempts: 5, web_search: true, file_search: ["file_xxxxxxxxxxxx"], context_awareness: true, ) puts workflow_response ``` ```go theme={null} package main import ( "context" "fmt" "log" "github.com/deeprails/deeprails-go-sdk" "github.com/deeprails/deeprails-go-sdk/option" ) func main() { apiKey := "YOUR_API_KEY" // NewClient will use the api key set under DEEPRAILS_API_KEY if option.WithAPIKey is not specified client := deeprails.NewClient(option.WithAPIKey(apiKey)) workflow, err := client.Defend.NewWorkflow( context.TODO(), deeprails.DefendNewWorkflowParams{ Name: deeprails.F("Test Workflow"), Description: deeprails.F("A workflow used to test the DeepRails API"), ThresholdType: deeprails.F(deeprails.DefendNewWorkflowParamsThresholdTypeCustom), CustomHallucinationThresholdValues: deeprails.F(map[string]float64{ "completeness": 0.850000, "instruction_adherence": 0.750000, }), ImprovementAction: deeprails.F(deeprails.DefendNewWorkflowParamsImprovementActionFixit), MaxImprovementAttempts: deeprails.F(int64(5)), WebSearch: deeprails.F(true), FileSearch: deeprails.F([]string{"file_xxxxxxxxxxxx"}), ContextAwareness: deeprails.F(true), }, ) if err != nil { log.Fatal(err) } fmt.Printf("Created workflow with ID: %s\n", workflow.WorkflowID) } ``` #### Required Parameters | Field | Type | Description | | -------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | name | string | The name of the workflow. | | threshold\_type | string | The workflow type (either automatic or custom), which determines whether thresholds are specified by the user or set automatically. | | improvement\_action | string | The remediation strategy when outputs fail guardrail metrics. fixit rewrites the failing output to pass the metrics. regen prompts the LLM to regenerate the output from scratch. do\_nothing records the failure without attempting remediation. | #### Optional Parameters | Field | Type | Description | | -------------------------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | description | string | A description of the use case of the workflow or other additional information. | | custom\_hallucination\_threshold\_values | object | The mapping of guardrail metrics to floating point threshold values (0.0-1.0). Required when threshold\_type is custom. This determines which metrics Defend will evaluate and how strict each threshold is. | | automatic\_hallucination\_tolerance\_levels | object | The mapping of guardrail metrics to tolerance levels (low, medium, or high). Required when threshold\_type is automatic. This determines which metrics Defend will evaluate and how strict the adaptive thresholds start. | | max\_improvement\_attempts | integer | The maximum number of improvement attempts to be applied to one workflow event before it is considered failed. Defaults to 10. | | web\_search | boolean | Whether or not the extended AI capability, web search, is available to the evaluation and remediation models. | | file\_search | string\[] | A list of uploaded file IDs for the evaluation and remediation models to search using the extended AI capability, file search. Upload files first via /files/upload. | | context\_awareness | boolean | Whether or not the extended AI capability, context awareness, is available to the evaluation and remediation models. | ### Submit a Workflow Event Use the SDK to log a production event (input + output). This creates a **workflow event** and automatically triggers an associated **evaluation** using the guardrail metrics you pass.

If the evaluation fails for one or more metrics, the improvement action specified for the affiliated workflow will be used to remediate the output. Then, that improved output will be evaluated and potentially improved again, if needed.

The improvement process will repeat for that event until all guardrails pass or the maximum number of retries is reached. > Tip: You can also submit a workflow event via the DeepRails API Playground. ```python theme={null} from deeprails import DeepRails # Initialize (env var DEEPRAILS_API_KEY is recommended) client = DeepRails(api_key="YOUR_API_KEY") event_response = client.defend.submit_event( workflow_id="wkfl_xxxxxxxxxxxx", model_input={ "system_prompt": "You are a helpful tutor specializing in AP science classes.", "user_prompt": "Explain the difference between mitosis and meiosis in one sentence.", "ground_truth": "Mitosis produces two identical diploid daughter cells for growth and repair, while meiosis produces four genetically unique haploid gametes for sexual reproduction.", "context": [ {"role": "user", "content": "I have an AP Bio exam tomorrow, can you help me study?"}, {"role": "tutor", "content": "Sure, I'll help you study."} ] }, model_output="Mitosis produces two genetically identical diploid cells for growth and tissue repair, whereas meiosis generates four genetically varied haploid gametes for sexual reproduction.", model_used="gpt-4o-mini", run_mode="precision", nametag="test", ) print(event_response) ``` ```ts theme={null} import { DeepRails } from "deeprails"; async function main() { // Initialize (env var DEEPRAILS_API_KEY is recommended) const client = new DeepRails(process.env.DEEPRAILS_API_KEY || "YOUR_API_KEY"); const eventResponse = await client.defend.submitEvent( "wkfl_xxxxxxxxxxxx", { model_input: { system_prompt: "You are a helpful tutor specializing in AP science classes.", user_prompt: "Explain the difference between mitosis and meiosis in one sentence.", ground_truth: "Mitosis produces two identical diploid daughter cells for growth and repair, while meiosis produces four genetically unique haploid gametes for sexual reproduction.", context: [ { role: "user", content: "I have an AP Bio exam tomorrow, can you help me study?" }, { role: "tutor", content: "Sure, I'll help you study." }, ], }, model_output: "Mitosis produces two genetically identical diploid cells for growth and tissue repair, whereas meiosis generates four genetically varied haploid gametes for sexual reproduction.", model_used: "gpt-4o-mini", run_mode: "precision", nametag: "test", } ); console.log(eventResponse); } main().catch(console.error); ``` ```Ruby theme={null} require "deeprails" # Initialize (env var DEEPRAILS_API_KEY is recommended) deeprails = Deeprails::Client.new( api_key: ENV["DEEPRAILS_API_KEY"] || "YOUR_API_KEY" ) event_response = deeprails.defend.submit_event( "wkfl_xxxxxxxxxxxx", { model_input: { system_prompt: "You are a helpful tutor specializing in AP science classes.", user_prompt: "Explain the difference between mitosis and meiosis in one sentence.", ground_truth: "Mitosis produces two identical diploid daughter cells for growth and repair, while meiosis produces four genetically unique haploid gametes for sexual reproduction.", context: [ { role: "user", content: "I have an AP Bio exam tomorrow, can you help me study?" }, { role: "tutor", content: "Sure, I'll help you study." } ] }, model_output: "Mitosis produces two genetically identical diploid cells for growth and tissue repair, whereas meiosis generates four genetically varied haploid gametes for sexual reproduction.", model_used: "gpt-4o-mini", run_mode: "precision", nametag: "test" } ) puts event_response ``` ```go theme={null} package main import ( "context" "fmt" "log" "github.com/deeprails/deeprails-go-sdk" "github.com/deeprails/deeprails-go-sdk/option" ) func main() { apiKey := "YOUR_API_KEY" client := deeprails.NewClient(option.WithAPIKey(apiKey)) // Submit a workflow event event, err := client.Defend.SubmitEvent( context.TODO(), "wkfl_xxxxxxxxxxxx", deeprails.DefendSubmitEventParams{ ModelInput: deeprails.F(deeprails.DefendSubmitEventParamsModelInput{ SystemPrompt: deeprails.F("You are a helpful tutor specializing in AP science classes."), UserPrompt: deeprails.F("Explain the difference between mitosis and meiosis in one sentence."), GroundTruth: deeprails.F("Mitosis produces two identical diploid daughter cells for growth and repair, while meiosis produces four genetically unique haploid gametes for sexual reproduction."), Context: deeprails.F([]string{ "user: I have an AP Bio exam tomorrow, can you help me study?", "tutor: Sure, I'll help you study.", }), }), ModelOutput: deeprails.F("Mitosis produces two genetically identical diploid cells for growth and tissue repair, whereas meiosis generates four genetically varied haploid gametes for sexual reproduction."), ModelUsed: deeprails.F("gpt-4o-mini"), RunMode: deeprails.F(deeprails.DefendSubmitEventParamsRunModePrecision), Nametag: deeprails.F("test"), }, ) if err != nil { log.Fatal(err) } fmt.Printf("%+v\n", event) } ``` #### Required Parameters | Field | Type | Description | | -------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | workflow\_id | string | The ID of the Defend workflow associated with this event. (find it in Console → Defend → Manage Workflows) | | model\_input | object | Your prompt + optional context. Must include at least a user\_prompt. See model\_input fields below. | | model\_output | string | The LLM output to be evaluated and recorded with the event. | | model\_used | string | The model used to generate the output, like gpt-4o or o3. | | run\_mode | string | Run mode for the workflow event that determines which models are used to evaluate the event. Available run modes (fastest to most thorough): super\_fast, fast, precision, precision\_codex, precision\_max, and precision\_max\_codex. Defaults to fast. Note: super\_fast does not support Web Search or File Search — if your workflow has these enabled, use a different run mode or edit the workflow to disable them. | #### model\_input Fields
Field Type Required Description
user\_prompt string Yes The user prompt sent to the LLM.
system\_prompt string No The system prompt used to configure the LLM's behavior.
ground\_truth string No The expected correct answer. Required when the workflow evaluates the ground\_truth\_adherence metric.
context array No Structured context for the evaluation, such as conversation history or domain-specific facts. Each item should have role and content fields. Required when context\_awareness is enabled on the workflow.
#### Optional Parameters | Field | Type | Description | | -------------------- | ------ | --------------------------------- | | nametag | string | A user-defined tag for the event. | ### Retrieve Workflow and Event Details You can retrieve workflow details and fetch events from a specific workflow. Events are processed asynchronously, so you will need to poll using the event ID until the status is Completed. ```python theme={null} import time from deeprails import DeepRails # Initialize (env var DEEPRAILS_API_KEY is recommended) client = DeepRails(api_key="YOUR_API_KEY") WORKFLOW_ID = "wkfl_xxxxxxxxxxxx" EVENT_ID = "evt_xxxxxxxxxxxx" # Retrieve workflow details workflow_response = client.defend.retrieve_workflow( workflow_id=WORKFLOW_ID ) print(workflow_response) # Poll event until completion event_response = None while True: event_response = client.defend.retrieve_event( workflow_id=WORKFLOW_ID, event_id=EVENT_ID, ) if event_response.status == "Completed": break print(f"Event status: {event_response.status} — waiting...") time.sleep(2) # Check the improvement outcome tool_status = event_response.improvement_tool_status if tool_status == "improved": print(f"Improved model output: {event_response.improved_model_output}") elif tool_status == "improvement_failed": print("Improvement action failed to improve output") elif tool_status == "no_improvement_required": print("Improvement not needed!") ``` ```ts theme={null} import { DeepRails } from "deeprails"; async function main() { // Initialize (env var DEEPRAILS_API_KEY is recommended) const client = new DeepRails(process.env.DEEPRAILS_API_KEY || "YOUR_API_KEY"); const WORKFLOW_ID = "wkfl_xxxxxxxxxxxx"; const EVENT_ID = "evt_xxxxxxxxxxxx"; // Retrieve workflow details const workflowResponse = await client.defend.retrieveWorkflow(WORKFLOW_ID); console.log(workflowResponse); // Poll event until completion let eventResponse: any = null; while (true) { eventResponse = await client.defend.retrieveEvent(WORKFLOW_ID, EVENT_ID); if (eventResponse.status === "Completed") break; console.log(`Event status: ${eventResponse.status} — waiting...`); await new Promise((resolve) => setTimeout(resolve, 2000)); } // Check the improvement outcome const toolStatus = eventResponse.improvement_tool_status; if (toolStatus === "improved") { console.log(`Improved model output: ${eventResponse.improved_model_output}`); } else if (toolStatus === "improvement_failed") { console.log("Improvement action failed to improve output"); } else if (toolStatus === "no_improvement_required") { console.log("Improvement not needed!"); } } main().catch(console.error); ``` ```Ruby theme={null} require "deeprails" # Initialize (env var DEEPRAILS_API_KEY is recommended) deeprails = Deeprails::Client.new( api_key: ENV["DEEPRAILS_API_KEY"] || "YOUR_API_KEY" ) WORKFLOW_ID = "wkfl_xxxxxxxxxxxx" EVENT_ID = "evt_xxxxxxxxxxxx" # Retrieve workflow details workflow_response = deeprails.defend.retrieve_workflow(WORKFLOW_ID) puts workflow_response # Poll event until completion event_response = nil loop do event_response = deeprails.defend.retrieve_event(EVENT_ID, workflow_id: WORKFLOW_ID) break if event_response.status == "Completed" puts "Event status: #{event_response.status} — waiting..." sleep(2) end # Check the improvement outcome tool_status = event_response.improvement_tool_status if tool_status == "improved" puts "Improved model output: #{event_response.improved_model_output}" elsif tool_status == "improvement_failed" puts "Improvement action failed to improve output" elsif tool_status == "no_improvement_required" puts "Improvement not needed!" end ``` ```go theme={null} package main import ( "context" "fmt" "log" "time" "github.com/deeprails/deeprails-go-sdk" "github.com/deeprails/deeprails-go-sdk/option" ) func main() { apiKey := "YOUR_API_KEY" client := deeprails.NewClient(option.WithAPIKey(apiKey)) WORKFLOW_ID := "wkfl_xxxxxxxxxxxx" EVENT_ID := "evt_xxxxxxxxxxxx" // Retrieve workflow details workflow, err := client.Defend.GetWorkflow( context.TODO(), WORKFLOW_ID, deeprails.DefendGetWorkflowParams{}, ) if err != nil { log.Fatal(err) } fmt.Printf("%+v\n", workflow) // Poll event until completion var event *deeprails.WorkflowEventDetailResponse for { event, err = client.Defend.GetEvent( context.TODO(), WORKFLOW_ID, EVENT_ID, ) if err != nil { log.Fatal(err) } if string(event.Status) == "Completed" { break } fmt.Printf("Event status: %s — waiting...\n", event.Status) time.Sleep(2 * time.Second) } // Check the improvement outcome toolStatus := event.ImprovementToolStatus if toolStatus == "improved" { fmt.Printf("Improved model output: %s\n", event.ImprovedModelOutput) } else if toolStatus == "improvement_failed" { fmt.Println("Improvement action failed to improve output") } else if toolStatus == "no_improvement_required" { fmt.Println("Improvement not needed!") } } ``` ### Check Defend Outcomes via the API Console 1. Open DeepRails API Console → Defend → Data.
2. Filter by time range or search by workflow\_id or nametag to find events.
3. Open any event to see guardrail scores and remediation chains (FixIt/ReGen).
Defend data – placeholder ### Next Steps Explore the metrics behind evaluations—correctness, safety, completeness, and more. Learn how workflows, metrics, and remediation work together. # Hallucination Classification Source: https://docs.deeprails.com/engine/hallucination-classification How DeepRails classifies and responds to hallucinations in Defend. Hallucinations in DeepRails are defined by user set thresholds in Defend. Each guardrail metric returns a score. Any score below the workflow’s hallucination threshold for that metric is treated as a hallucination, meaning the output failed and needs remediation. ## How Defend Identifies Hallucinations * Users can customize each workflow by selecting guardrail metrics for its evaluations and either setting custom thresholds per metric or enabling DeepRails' automatic detection at either high, medium, or low tolerance. * Defend compares each metric’s score to its configured hallucination threshold. If **all metrics meet or exceed their thresholds**, the output passes. If **any metric falls below** its threshold, the output is labeled a hallucination. * Hallucinations detected in evaluations drive remediation for the failed metrics. Based on the workflow’s improvement action (FixIt, ReGen, or Do Nothing) and retry limit, Defend attempts to improve the output and re-evaluates until all metrics pass or the retry budget is exhausted. * DeepRails uses Multimodal Partitioned Evaluation to more intelligently analyze outputs, using two evaluation models in parallel to greatly reduce the chance of hallucinations being missed. ## What Happens to Hallucinations When Defend detects a hallucination, it sends the output to remediation using the workflow’s selected improvement tool (learn more about improvement tools in the Defend Overview). The improved output is re-evaluated as soon as the tool finishes, as if it was a new event, completely independent of the original input/output pair. This evaluation-remediation cycle continues until all selected metrics pass or the workflow’s max retries are reached. DeepRails Defend uses **Assumed Pass** logic to reduce cost and latency: after a retry, only the metrics that failed in the previous output are re-evaluated. Scores and rationales for metrics that already passed are carried over to subsequent improvement attempts. This keeps remediation fast and efficient even when outputs have persistent hallucinations. #### The Impact of Run Modes All of DeepRails' evaluations are affected by the selected run mode (all run modes and more details about them are listed here). Run modes like Precision Max and Precision Max Codex use advanced reasoning models, incurring more cost and latency, while Fast uses more balanced and cost effective models. The more expensive run modes are more consistent both at detecting hallucinations and providing higher quality improvements. Higher quality hallucination detection and correction are worth the cost and latency for many applications, so experiment with different run modes before finalizing workflow configurations. ## Threshold Types DeepRails offers two types of hallucination thresholds in Defend: automatic tolerances or custom thresholds. Automatic workflows are designed for easy use, while custom thresholds give experienced users more control. #### Automatic Tolerances * Choose qualitative tolerance levels (`low`, `medium`, or `high`) per guardrail using `automatic_hallucination_tolerance_levels`. * DeepRails translates each tolerance into numeric thresholds and adapts them based on workflow performance as more events are recorded. * Best for fast setup or evolving prompts.
Tolerance Behavior When to Use
low Lenient thresholds that allow for more creative or exploratory outputs while still filtering obvious failures. Early prototyping, creative writing, or low-risk internal tools.
medium Balanced thresholds tuned for general production use; catches most issues without over-blocking. Default choice for new workflows and broad user-facing assistants.
high Strict thresholds that only accept high-quality, confident outputs. Regulated domains, compliance-heavy workflows, or scenarios with near-zero tolerance for hallucinations.
#### Custom Thresholds * Provide numeric cutoffs (0.0–1.0) for each metric in console or via API. * Thresholds remain fixed until you change them, giving maximum control for regulated or benchmarked use cases. * Best when you are familiar with DeepRails and your use case, and you know the minimum acceptable scores for your chosen metrics. ## What a Hallucination Looks Like You can view all evaluations for your Monitors and Defend workflows on their respective data tabs, Monitor Data and Defend Data. Defend Data threshold example When testing DeepRails Defend in the Playground, you can view evaluation performance by metric near the bottom of the results page for any run. API Playground threshold example # Multimodal Partitioned Evaluation Source: https://docs.deeprails.com/engine/multimodal-partitioned-evaluation Multimodal Partitioned Evaluation (formerly known as HyperChainpoll) is DeepRails’ evaluation engine. MPE combines several evaluation techniques to deliver accurate, audit-ready scores across all metrics—without exposing your prompts or users to single-model bias. Multimodal Partitioned Evaluation (MPE) is the engine that powers all DeepRails evaluations across Monitor and Defend. MPE is not a separate product—it is the way our evals run under the hood for every guardrail metric. We intentionally describe MPE at a high level to protect our IP. Exact routing, partitioning, and aggregation heuristics are proprietary. ## What is MPE? Multimodel Partitioned Evaluation is an advanced member of a larger class of AI evaluations, LLM-as-a-Judge (LLMJ). This means that MPE relies on using several other models to evaluate the output from an LLM. MPE stands out from other forms of LLMJ by using partitioned judging, dual-model consensus, and confidence calibration. LLMs work better with more granular tasks, so MPE breaks down evaluations in several ways: partitioning the input into smaller parts, evaluating in parallel with two or more models, and assigning confidence values rather than having models internally aggregate scores. ## Why LLM-as-a-Judge Is Effective The LLM-as-a-Judge evaluation approach combines the nuanced understanding of human evaluators with the scalability and consistency of automated systems. By leveraging the reasoning capabilities of LLMs to evaluate outputs across a range of tasks and quality dimensions, this approach allows for more holistic assessments that align closely with human judgment. Moreover, LLMJ can process vast amounts of data quickly, meaning they can provide timely feedback crucial for iterative development. > “\[Human graded evals are] often expensive or not always practical” > > * Shyamal Hitesh Anadkat, Applied AI Engineer, OpenAI The adoption of the LLMJ approach by leading AI labs underscores its effectiveness. OpenAI employs its most advanced models to evaluate outputs of new models, guiding release decisions and performance benchmarks. Similarly, Anthropic integrates judge-style evaluations as a "pillar of safe scaling", actively supporting an external ecosystem to develop LLMJ tools and protocols. ## Why We Built MPE Single-judge evaluators often miss subtle errors, inherit model-specific biases, and struggle with complex prompts. MPE solves this by **breaking evaluations into smaller, checkable units**, judging each unit with **two different LLMs in parallel**, and **calibrating by confidence** before producing a final, interpretable score. * **Always two judges in parallel:** Every evaluation uses two distinct LLMs to reduce bias and increase reliability. * **Partitioned judging:** Big problems are split into focused checks aligned to each guardrail metric. * **Confidence-aware aggregation:** Judges estimate their own confidence and total scores are weighted accordingly. * **Reasoned evaluation:** Judging prompts promote structured, stepwise reasoning for better fidelity. * **Repetition:** Evaluations are run several times per model and averaged to minimize the impact of evaluation hallucinations. * **Plan-agnostic:** Available on all plans and across all APIs. Model selection follows your chosen Run Mode. ## The four pillars of MPE Large input/output pairs are segmented into smaller, verifiable units per guardrail metric (e.g., claims for correctness, context checks for adherence). This reduces distraction and makes scoring traceable. Two different LLMs (often cross-provider) judge each unit in parallel. The output model never judges itself. This mitigates single-model bias and improves stability. Each judge self-reports confidence for every sub-check. MPE aggregates with confidence-aware weighting to ensure the final score reflects the exactness of the evaluations. Prompts that encourage structured, chain-of-thought style reasoning are run multiple times. This improves faithfulness on complex, multi-step tasks and minimizes noise from hallucinations. ## How MPE runs (conceptual) The request is decomposed into focused checks per selected guardrail metrics (e.g., correctness claims, instruction adherence, safety). Two different LLMs evaluate each partition independently, producing a score, rationale, and self-reported confidence. MPE aggregates the per-partition results with confidence calibrated weighting to form metric-level scores. Scores, rationales, and metadata are returned and visualized in the Console—ready for monitoring, auditing, and optimization. ```mermaid theme={null} flowchart LR A["Input + Output"] B["Partition into sub-checks per metric"] C1["Judge A (LLM 1)"] C2["Judge B (LLM 2)"] D["Scores • Rationales • Confidence"] E["Confidence-aware consensus"] F["Final metric scores"] A --> B B --> C1 B --> C2 C1 --> D C2 --> D D --> E E --> F ``` # Run Modes Source: https://docs.deeprails.com/engine/run-modes DeepRails has six different run modes that let you balance cost, latency, and accuracy across all APIs. Choose the intelligence level of the models behind your evaluations to best fit your needs. Run Modes control how DeepRails executes evaluations across Monitor and Defend. Every run uses two different LLMs in parallel to reduce bias and improve accuracy. The selected Run Mode determines which models are selected — from compact cost-efficient models to advanced reasoning models — so you can optimize your workflow. ## Why Run Modes Matter Not every task requires the same evaluation depth. A simple summarization prompt can be tested cost effectively with smaller models, while multi-step reasoning (math generation, chained steps, or multi-task prompts) benefits from reasoning-capable models. DeepRails' Run Modes let you tune this balance. * **Always two models in parallel:** Every evaluation uses two distinct LLMs to generate scores, avoiding single-model bias. * **Reasoning vs. non-reasoning models:** For complex prompts, modes that include reasoning models yield better accuracy and interpretability. * **Available everywhere:** Run Modes function the same across the Monitor and Defend APIs on all plans. ## The Six Run Modes Super Fast}> High-speed lightweight checks. Rapid evaluations with minimal overhead, ideal for large-scale screening.
Accuracy  ●●○○○
Speed     ●●●●○
 Fast (default)}> Balanced speed and accuracy. The default mode — a strong balance of evaluation quality and throughput for most use cases.
Accuracy  ●●●●○
Speed     ●●●○○
 Precision}> Deep multi-model analysis. Recommended for complex prompts that benefit from reasoning.
Accuracy  ●●●●○
Speed     ●●○○○
 Precision Codex}> Code-optimized deep analysis. Recommended for code-based workflows that benefit from specialized code analysis.
Accuracy  ●●●●○
Speed     ●●○○○
 Precision Max}> Exhaustive multi-pass verification. Two reasoning models in parallel — best for mission-critical use cases where accuracy outweighs cost or latency. **Note: Does not support streaming.**
Accuracy  ●●●●●
Speed     ●○○○○
Streaming  ✕
 Precision Max Codex}> Ultimate code-aware verification. Two reasoning models with Codex-optimized deep analysis for the highest accuracy on code-based workflows. **Note: Does not support streaming.**
Accuracy  ●●●●●
Speed     ●○○○○
Streaming  ✕
 **Streaming Constraints**: `precision_max` and `precision_max_codex` do NOT support streaming due to their multi-pass verification architecture. If your application requires real-time streaming responses, use `super_fast`, `fast`, `precision`, or `precision_codex` instead. Choosing whether to use reasoning models is often part of the prompt engineering process. If your task involves multi-step logic, mathematics, or complex instructions, Precision or Precision Max are recommended. If your task requires complex code analysis, use Precision Codex or Precision Max Codex. ## Choosing the Right Run Mode ### At a Glance
Run Mode ID Accuracy Speed Streaming Web / File Search
Super Fast super\_fast ●●○○○ ●●●●○ Yes No
Fast (default) fast ●●●●○ ●●●○○ Yes Yes
Precision precision ●●●●○ ●●○○○ Yes Yes
Precision Codex precision\_codex ●●●●○ ●●○○○ Yes Yes
Precision Max precision\_max ●●●●● ●○○○○ No Yes
Precision Max Codex precision\_max\_codex ●●●●● ●○○○○ No Yes
### Detailed Comparison
Name Description When to Use Example Use Case
Super Fast High-speed lightweight checks; rapid evaluations with minimal overhead. Large-scale screening, early exploration, low-stakes triage. Screening 10,000 code-gen outputs to flag potential safety risks.
Fast (default) Balanced speed and accuracy; the default mode for most use cases. General-purpose workflows that need a good balance of quality and throughput. Monitoring daily regressions in a customer support chatbot.
Precision Deep multi-model analysis; strong reasoning coverage with balanced cost/latency. Complex prompts with logic/calculations or multi-step reasoning. Monitoring daily regressions in a legal research bot.
Precision Codex Code-optimized deep analysis; specialized for code-based workflows. Code reviews, software development tasks, and technical documentation requiring code analysis. Evaluating AI-generated code snippets for correctness and best practices.
Precision Max Exhaustive multi-pass verification; two reasoning models in parallel (very high cost and latency). Mission-critical evaluations, final QA sweeps, regulated or safety-sensitive domains. Compliance evaluation on a healthcare agent before production.
Precision Max Codex Ultimate code-aware verification (highest accuracy, lowest speed). Complex code creation or refactoring, debugging large systems, or similar software engineering tasks. Final security and correctness review of AI-generated code before merging to production.
**Super Fast** does not support **Web Search** or **File Search** capabilities. If your workflow or monitor has these enabled, requests using `super_fast` will be rejected. To resolve this, either switch to a run mode that supports these capabilities or edit the workflow/monitor to disable Web Search and File Search. This mode does support **Context Awareness** — you can still pass context in your `model_input`. # Completeness Source: https://docs.deeprails.com/guardrails/completeness LLM outputs can often fixate on subsets of complex prompts and can occasionally deviate from the user's intended topic. The Completeness metric evaluates whether a response completely answers a prompt without going off track. Completeness measures the degree to which an AI-generated response fully addresses every necessary aspect of the user's query, with sufficient detail. Off topic deviations and incoherence result in deductions. Completeness is returned as a continuous score ranging from 0 to 1: *** ## Understanding Completeness The Completeness metric evaluates each model response along five key dimensions: * **Coverage:** Does the response address all parts of the user's query? * **Detail and Depth:** Does it consistently provide sufficient elaboration? * **Factual Correctness:** Is the information given sufficiently accurate? * **Relevance:** Is the content strictly pertinent to the query? * **Logical Coherence:** Is the response clearly organized and well structured? A separate pass/fail assessment for each dimension is conducted before the evaluation model decides a final grade for Completeness.

Example Scenario

Completeness detects when a response may be technically correct but fails to satisfy the full intent of the user's query.
User query: "Describe how a bill becomes a law in the U.S., and give an example of a recent bill."
Model response: "A bill becomes a law after being approved by Congress and signed by the President."
Analysis: The content given in the response is factually correct and coherent, but it fails to mention an example and lacks procedural detail. It would score poorly on the Coverage and Detail dimensions, leading to a low overall Completeness score.
## Details of the Five Dimensions This aspect is the most important, since missing a sub-request is the most common error models make in terms of completeness. The evaluation model enumerates each request in the user prompt and verifies if each one was met by the model response. LLMs sometimes use "fluffy" sentences with little substance, and those responses cannot be considered completely satisfactory. Each sentence in the response is analyzed and the evaluation model fails the response if any of them could've included more detail. This dimension seems like it should be unnecessary at first glance since DeepRails provides a Correctness metric separate from Completeness. However, during development of the prompt, we found that evaluation models struggled to accurately assess completeness without a component confirming that the response's claims were true. Sometimes an AI response can be too complete. This dimension of the Completeness evaluation fails if any sentences in the response are not directly related to the user's query. A response's completeness is not all about its content. A finished response has a good logical flow and provides indentation, proper spacing, and bullet points as needed, and our evaluation model checks the response for all of this structure. ## Evaluation Process DeepRails performs a Multimodal Partitioned Evaluation of every model output to assess its completeness in each of the five dimensions as visualized below. This flow is completed on two separate models and their evaluations are averaged to arrive at a final score. See the MPE page for more information. ```mermaid theme={null} flowchart LR A["Input + Output"] B1["Coverage Evaluation"] B2["Detail and Depth Evaluation"] B3["Factual Correctness Evaluation"] B4["Relevance Evaluation"] B5["Logical Coherence Evaluation"] C1["Coverage Verdict"] C2["Detail and Depth Verdict"] C3["Factual Correctness Verdict"] C4["Relevance Verdict"] C5["Logical Coherence Verdict"] D["Score Summary Generation"] E["Score Aggregation"] A --> B1 A --> B2 A --> B3 A --> B4 A --> B5 B1 --> C1 B2 --> C2 B3 --> C3 B4 --> C4 B5 --> C5 C1 --> D C2 --> D C3 --> D C4 --> D C5 --> D D --> E ``` ## Improving Low Completeness Scores

Recommended Fixes

To improve low completeness scores:
Use prompt structure effectively: You should tailor your prompts to emphasize structure and completeness in response; add specific validation rules for each important part of your request to the end of your prompts.
Guide for elaboration: You should instruct the model to use examples and detail specifically in each of your prompts to reinforce its adherence to all aspects of your request.
*** Completeness evaluates the robustness of your AI outputs, but remember to combine it with Correctness, Instruction Adherence, or other metrics for a more holistic review of the response contents. # Comprehensive Safety Source: https://docs.deeprails.com/guardrails/comprehensive-safety One of the largest concerns with using LLMs in automation is that dangerous content could be output uncensored and exposed to thousands or millions of people before it's caught. The Comprehensive Safety metric evaluates whether a response is completely devoid of potentially dangerous statements of all categories. Comprehensive Safety evaluates whether an AI-generated response contains unsafe, harmful, or prohibited content across a set of critical safety categories. Comprehensive Safety is returned as a boolean. Detection of any safety violation in a response results in an automatic failure. *** ## Understanding Comprehensive Safety Our Comprehensive Safety metric evaluates each model response across the following risk domains: * **CBRN Misuse**: Chemical, Biological, Radiological, or Nuclear weapon-related content * **Personally Identifiable Information (PII)**: Exposure of sensitive identity details like full names, SSNs, or contact information * **Hate / Harassment**: Abusive, discriminatory, or hostile language targeting individuals or groups * **Violence / Threats**: Physical threats, weapon use or creation instructions, or incitement to harm * **Sexual Content**: Explicit or inappropriate sexual material, especially involving children * **Self-Harm / Harmful Instructions**: Content that encourages self-injury or dangerous behaviors ## Evaluation Process DeepRails performs a Multimodal Partitioned Evaluation of every model output to scan for safety violations. A few core pieces of logic ensure that the evaluation is as thorough and accurate as possible. We intentionally describe evaluation logic at a high level to protect our IP. Exact segmentation, verification, and aggregation logic is proprietary. The model output is decomposed into up to 5 segments or 'chunks', usually one or more closely related sentences or clauses. Each chunk is evaluated for safety across each of the six risk domains. Each chunk is assessed to determine if its content is related to any risk domains. For each domain identified, the evaluation model decides whether the content was discussed safely or not (a binary verdict). If any chunk is flagged for any category, then the entire response is failed. There is no partial credit assigned for Comprehensive Safety, due to the sensitive nature of the category. ## Addressing Unsafe Responses

Improving Safety Outcomes

To minimize unsafe completions:
Refine prompts and system instructions: Discourage content likely to result in triggering harmful categories.
Use category-level logging: Track which safety categories are most frequently violated and then guide your training or fine-tuning efforts to compensate.
Apply structured overrides: Use score-based logic to filter, flag, or escalate responses for human review in agentic workflows with chained LLMs.
# Context Adherence Source: https://docs.deeprails.com/guardrails/context-adherence One of the most powerful aspects of generative AI is models' abilities to adjust to a given context. As such, monitoring the model's adherence is the most critical metric for many use cases. Note, however, the evaluation will not work effectively if DeepRails does not detect enough context in the prompt. Only use Context Adherence when a notable set of context is passed along with the prompt. Context Adherence measures how closely an AI response aligns with the information in the context provided in or with the user prompt. **This metric is only appropriate for prompts that include a significant context window.** *** ## Understanding Context Adherence

Context Adherence vs. Other Metrics

Context Adherence is critical in tasks that require strict fidelity to provided documents or sources.
Context Adherence: Measures whether the response reflects the information available in the provided context (e.g., retrieved documents, input references).
Correctness: Measures whether the response is factually accurate based on external truth—regardless of the provided context.
Instruction Adherence: Measures whether the model followed *how* it was supposed to answer, based on explicit instructions (e.g., tone, format).
*** ## Evaluation Process DeepRails performs a Multimodal Partitioned Evaluation of every model output to assess whether each claim is grounded in the context of the input to the model. Since adherence metrics involve more analysis of the model input than metrics like Correctness, the evaluation flow is a bit more complex. ```mermaid theme={null} flowchart LR A["Context Identification"] B["Claim Identification"] C1["Insufficient Context"] C2["Claim/Context Matching"] D["Evaluation and Y/N Verdict Assignment"] E["Confidence Assignment"] F["Score Aggregation"] A --> B B --> C1 B --> C2 C2 --> D D --> E E --> F ``` Context Adherence evaluations will terminate prematurely if insufficient context is detected, and the output will be given a 100% score as a default. *** ## Addressing Low Context Adherence Scores

Improving Context Adherence

To reduce out-of-context generations and improve grounding quality:
Use clear and complete context: You should ensure that all facts needed for an ideal response are included in the context window.
Instruct against extrapolation: You should direct the model to avoid speculation or deviation in all prompts that use a context window.
Audit across prompt types: Compare how different tasks (e.g., summarization, QA) influence context drift, and use different models, prompts, and/or context windows for each.
*** Context Adherence is critical for grounded generation tasks like RAG, search, summarization, and citation. However, it should not be used in context-light applications. # Correctness Source: https://docs.deeprails.com/guardrails/correctness The scariest AI hallucinations are when outputs contain made up or false information. The Correctness metric measures how verifiably true each claim in the LLM output is. Correctness evaluates the factual accuracy of an AI model's response. It measures the approximate percentage of the response that is verifiably true, with gross errors being weighted more than slight missteps. Correctness is returned as a continuous metric ranging from 0 to 1: *** ## Understanding Correctness It's important to distinguish Correctness from other related metrics:

Correctness vs. Context Adherence

Correctness: Measures whether a model response contains factual information, regardless of whether that information is included in the provided context.
Context Adherence: Measures whether the model’s output is derived solely from the user-supplied context or documents.
Example: A model may state that “The Eiffel Tower is in Paris,” which is factually correct (high Correctness) even if that fact is not found in the provided context. However, if the model generates this information from its own prior knowledge rather than the given context, it may be rated low on Context Adherence because it did not rely on the source material it was instructed to use.

Correctness vs. Completeness

Correctness: Measures whether a model response's factual information is accurate, regardless of whether that information fully answers the user's prompt.
Completeness: Measures whether the model’s output includes all necessary statements to entirely fulfill the prompt.
Example: A model may state that "The Eiffel Tower is in Paris," which is factually correct (high Correctness) regardless if it's related to the user's prompt. However, if the user prompt requested more information about the Eiffel Tower, or if the Eiffel Tower is off topic for the user request, the response would be incomplete (low Completeness).
## Why We Evaluate Correctness The most concerning and imminently dangerous errors made by AI are caused when outputs give verifiably false information that could inform important decisions or opinions. A dedicated evaluation for the factual accuracy for all claims made in AI outputs is critical for essentially every production LLM use case. As such, we made sure our Correctness evaluations efficiently and accurately identify all false or misleading claims. ## Evaluation Process DeepRails performs a Multimodal Partitioned Evaluation of every model output to assess the truth in each of its claims. A few core pieces of logic ensure that the evaluation is as thorough and accurate as possible. We intentionally describe evaluation logic at a high level to protect our IP. Exact segmentation, verification, and aggregation logic is proprietary. The model output is decomposed into segments each containing a granular factual claims. The most important claims are intelligently selected for evaluation to ensure that minor details don't adversely impact evaluation of complex outputs. Each claim is reviewed for factual accuracy using model knowledge and outside references if necessary. A binary correctness judgment is assigned to avoid overassignment of partial credit, along with a confidence rating. All claim judgments are weighted by their confidence rating and consolidated into a final correctness score between 0 and 1. ## Addressing Low Correctness Scores

Improving Correctness

When responses score low on correctness, consider the following steps:
Pattern Analysis and Prompt Tuning: Identify patterns in incorrect outputs using DeepRails’ Monitor function or evaluation logs, then update prompts to guard against these patterns of unreliability.
Add Context and Structure: Modify model inputs to include more information on topics models may get wrong or even include a separate context field.
Model Selection: Experiment with different input models, as factuality varies widely across architectures and providers.
Verification Workflow: Use workflows with additional verification steps, either automated or human-in-the-loop, to review and override responses with consistently low correctness.
While Correctness provides a powerful factuality lens, it's important to combine it with other guardrails like Context Adherence and Completeness for robust production safety. # Ground Truth Adherence Source: https://docs.deeprails.com/guardrails/ground-truth-adherence Though it's more rarely used than other metrics, Ground Truth Adherence is very useful in some specific use cases. Situations where the model is provided a strict role to follow (the "ground truth") will need to use this metric to evaluate how well the model performed its role. Ground Truth Adherence measures how closely a model response aligns with a given, authoritative reference answer (or “ground truth"). **This evaluation is only possible for prompts that include one or more examples or reference answers in a `ground_truth` field in the model input.** If Ground Truth Adherence is selected as a metric, then a `ground_truth` field must be passed as part of the model input. The evaluation will fail if the `ground_truth` field is not included. *** ## Understanding Ground Truth Adherence

Ground Truth vs. Other Metrics

It's important to distinguish Ground Truth Adherence from the other Adherence metrics:
Ground Truth Adherence: Measures alignment with a known ideal answer or given behavior provided in a separate field. Think statements given as ground truth that may conflict with model knowledge like "Tokyo is the capital city of the land of Neolandia".
Instruction Adherence: Measures whether all subjective instructions were followed. Think rules about tone or sentence structure in the user prompt.
Context Adherence: Measures whether the model's response stayed within the information provided in a context window. Think an educational prompt tied to a specific Common Core learning standard.
## Evaluation Process DeepRails performs a Multimodal Partitioned Evaluation of every model output to assess whether each claim is aligned with the provided ground truth over everything else. This evaluation flow includes another step compared to the other adherence metrics to ensure that the ground truth rather than existing model knowledge is used to grade. ```mermaid theme={null} flowchart LR A["Ground Truth Identification"] B["Claim Identification"] C["Claim/Ground Truth Verification"] D["Confidence Assignment"] E["Final Consistency Check"] F["Score Aggregation"] A --> B B --> C C --> D D --> E E --> F ``` ## Addressing Low Ground Truth Adherence Scores

Improving Ground Truth Adherence

To reduce discrepancies between generated responses and reference answers:
Use few-shot prompting: Embed examples that reflect the reference style and structure to increase alignment.
Evaluate your references: Ensure given ground truths are internally consistent, accurate, and domain-relevant.
High Ground Truth Adherence reflects alignment, not necessarily quality. Pair it with Completeness to assess coverage, and Correctness to ensure factual accuracy for a richer assessment of model output. # Instruction Adherence Source: https://docs.deeprails.com/guardrails/instruction-adherence Prompts used in production workflows often are very complex and structured with dozens of validation rules. The Instruction Adherence metric assesses each of the input rules and evaluates whether the model response followed each consistently. Instruction Adherence measures how closely a model response follows the instructions defined in the user and/or system prompts. *** ## Understanding Instruction Adherence

How Instruction Adherence Differs from Other Metrics

While it sounds similar at first, Instruction Adherence is distinctly different from other metrics:
Instruction Adherence: Measures whether the response followed how it was supposed to answer—structure, tone, content constraints, formatting, etc.
Context Adherence: Measures whether the response reflects what was in the provided context (e.g., source documents).
Correctness: Measures whether the information in the response is factually accurate, regardless of whether it followed instructions or context.
## Evaluation Process DeepRails performs a Multimodal Partitioned Evaluation of every model output to assess the extent to which it follows all prompt instructions. A few core pieces of logic ensure that the evaluation is as thorough and accurate as possible. The model input is separated into explicit, atomic instructions. The most important claims are intelligently selected for evaluation to ensure the evaluation is completed in a timely manner. The model output is decomposed into segments each relating to an identified instruction. Each segment analyzed and determined to either follow or not follow its instruction. For each binary verdict, a confidence rating is given as well. All claim judgments are weighted by their confidence rating and consolidated into a final instruction adherence score between 0 and 1. ## Addressing Low Instruction Adherence Scores

Improving Instruction Adherence

When models fail to follow instructions, the resulting output may be irrelevant or unusable in later production steps. To improve instruction-following:
Refine prompts: Reword unclear or ambiguous instructions to be more direct, structured, and constraint-based.
Compare model variants: Some models are significantly more instruction-aligned than others. Use Adherence metrics to validate before selecting the model used in deployment.
## Best Practices Categorize instructions (format, tone, scope, etc.) to make it harder for model's to miss them. Write instructions that are structured, unambiguous, and directive (e.g., “Respond only in bullet points” or “Return valid JSON”). While the other metrics ensure the model gives all the right information, Instruction Adherence ensures that it gives it in the right way. This guardrail is essential for structured outputs, enterprise use cases, and task compliance. # Metrics Overview Source: https://docs.deeprails.com/guardrails/metrics-overview Explore the DeepRails Guardrail Metrics that were designed to holistically evaluate AI responses across all possible use cases. ## Understanding Guardrail Metrics DeepRails offers a unified, comprehensive suite of Guardrail Metrics built to diagnose, debug, and improve the behavior of large language models. Each guardrail targets a specific aspect of LLM output quality, like correctness or instruction adherence. Each metric uses refined evaluation logic to deliver a continuous score and feedback that highlights both strengths and needed improvements. The table below summarizes each DeepRails Guardrail metric, how it works, and where it is most useful in your AI workflow. Because the metrics evaluate independent dimensions of the output, enabling several together gives the fullest picture. #### DeepRails Metric Comparison
Name Description When to Use Example Use Case
Correctness Measures factual accuracy by evaluating whether each claim in the output is true and verifiable. When factual integrity is critical, especially in domains like healthcare, finance, or legal. Verifying whether a model-generated drug interaction list contains any false or fabricated claims.
Completeness Assesses whether the response addresses all necessary parts of the prompt with sufficient detail and relevance. When ensuring that all user instructions or question components are covered in the answer. Evaluating a customer support response to check if it fully answers a multi-part troubleshooting query.
Instruction Adherence Checks whether the AI followed the explicit instructions in the prompt and system directives. When prompt compliance to components such as tone, structure, or style guidance is important. Validating that a model-generated blog post adheres to formatting rules and brand tone instructions.
Context Adherence Determines whether each factual claim is directly supported by the provided context. When grounding responses in user-provided input or retrieved documents on top of the prompt text. Ensuring that a RAG-based assistant only uses company documentation to answer internal HR questions.
Ground Truth Adherence Measures how closely the output matches a provided correct answer (gold standard). When evaluating model outputs against a trusted reference, such as in benchmarking or grading tasks. Comparing QA outputs against annotated gold answers during LLM fine-tuning experiments.
Comprehensive Safety Detects and categorizes safety violations across areas like PII, CBRN, hate speech, self-harm, and more. When filtering or flagging unsafe, harmful, or policy-violating content in LLM outputs. Auditing model-generated transcripts for PII leakage and violent content.
#### How to Combine Metrics Every Monitor and Defend workflow can include any combination of guardrail metrics. Some metrics simply are not applicable to every use case. For example, Ground Truth Adherence is critical when models must follow sources of truth beyond their inherent knowledge but unnecessary elsewhere. DeepRails evaluates each guardrail metric in parallel. This means that latency is minimized for evaluations on many metrics, but the overall evaluation will still take as long as the longest individual metric evaluation. Note that cost still increases linearly with each metric added to the evaluation, so users should avoid adding metrics that are irrelevant to the workflow they are monitoring or improving. For chatbots where cost adds up fast and user retention matters most, you might enable only Instruction Adherence and Comprehensive Safety, while complex production use cases benefit from checking Correctness, Completeness, and more. ## Why DeepRails Uses Granular Scoring One of the things that sets DeepRails apart from other LLM evaluation services is the granularity of our evaluations. In any DeepRails evaluation, each selected metric receives a final score from 0% to 100%. The one exception is Comprehensive Safety, which is scored all or nothing because any safety violation causes a failure regardless of severity. This fine-grained analysis requires more complexity in the evaluation prompts. However, it yields much higher accuracy compared to competitors' evaluations, especially for middling outputs, which are the bulk of outputs for top end Gen AI use cases. That accuracy allows users to confidently set specific hallucination thresholds in our Defend service. The difference between a 65% and an 80% threshold is meaningfully different, unlike in other evaluation products lacking granularity, so users can adjust leniency in small increments when perfecting their Defend setups. #### DeepRails vs. Amazon Bedrock To validate the efficacy of our evaluations, DeepRails Correctness and Completeness evaluations were compared against Amazon Bedrock evaluations in the same two categories for sixty input/output pairs, repeated three times for each service. The results showed that DeepRails aligned much more closely with human-assigned Completeness and Correctness grades for scores between 40% and 80%. This is the range where most customers draw remediation thresholds in Defend. The comparison is shown in two charts below: one covering the 40-60% score range and another covering the 60-80% range. The differences between 50% and 70% accuracy that DeepRails surfaces can determine whether an AI system remains in production. DeepRails v. Bedrock 40-60% DeepRails v. Bedrock 60-80% These charts also highlight how Bedrock uses extremely rigid evaluations, only giving a 0, 25, 50, 75, or 100. That 5-point scale can be adequate for an initial prompt check but is unusable for viewing gradual trends in long-term prompt monitoring. An inaccurate Bedrock evaluation could move a score of 25 to a 0 or a 50, creating a huge swing. The margin of error for DeepRails is much smaller, about 7-12 percentage points in either direction. Bedrock is also more inclined to give perfect scores, which can mislead users who want to take evaluations from good to great. # DeepRails Overview Source: https://docs.deeprails.com/index The only AI reliability platform that detects hallucinations and automatically corrects them before they reach your users. DeepRails is a production AI reliability platform built for teams that cannot afford wrong answers. We provide two core services that work together to ensure your AI applications remain accurate, safe, and trustworthy throughout their lifecycle — across healthcare, legal, financial services, education, and any domain where output quality is mission-critical. Continuously monitor your AI applications for hallucinations, quality regressions, and performance drift in production. Catch hallucinations in real time and automatically correct them before they reach your users. Additionally, the intuitive **DeepRails Console** provides a central dashboard to visualize and explore evaluation data, manage defend workflows and monitors, and configure guardrails efficiently. ## The Challenge - Evaluating Model Performance > **"Lack of evaluations has been a key challenge for deploying to production"**\ > [*- OpenAI, DevDay Conference*](https://youtu.be/XGJNo8TpuVA?feature=shared\&t=1089) AI systems can generate significantly varied outputs for identical inputs, complicating benchmarks and making consistent evaluation difficult. Current evaluation methods struggle to identify subtle inaccuracies, hallucinations, or early indicators of performance drift, exposing organizations to critical risks. Additionally, as models evolve, previously reliable methods quickly become obsolete. This requires the need for evaluation tools that keep pace with continuous changes in AI behavior to consistently provide trustworthy insights and guardrails against critical failures. > **".. don't consider prompts the crown jewels. Evals are the crown jewels"**\ > [*- Jared Friedman, Y Combinator Lightcone Podcast*](https://www.youtube.com/watch?v=DL82mGde6wo\&t=859s) The best performing prompts are guided by continuous rounds of high quality evaluations. ## What Makes DeepRails Unique Most AI safety tools stop at detection: they flag problems, block outputs, or log failures for you to handle later. DeepRails goes further. The Defend API corrects hallucinated responses automatically, so your users always get accurate answers. * **Monitor API**: Real-time detection and observability across your production AI outputs * **Defend API**: Real-time detection and automatic correction, verified before delivery In independent benchmarks against AWS Bedrock Guardrails, DeepRails is **45% more accurate on correctness** and **53% more accurate on completeness**. Your AI applications self-heal in production, reducing support tickets and user frustration. ## How DeepRails Works DeepRails delivers highly performant, research-driven metrics, continuous monitoring capabilities, and real-time guardrails designed specifically for critical AI applications. Our Guardrails guide both our proprietary Multimodal Partitioned Evaluation (MPE) engine and our one-of-a-kind remediation service for AI hallucinations. Each guardrail was selected based on years of generative AI experience and rigorous research. The most important metrics were perfected and delivered first, and more are being designed by the DeepRails team continuously. As part of development, each Guardrail has a Multimodal Partitioned Evaluation prompt individually created and tested. MPE prompts outperform other evaluators by breaking inputs down into granular chunks before evaluating and then aggregating scores. Learn how DeepRails scores AI outputs using multi-model evaluation, confidence-weighted scoring, and adaptive run modes. Connect with our team to explore DeepRails' capabilities for your organization. # Monitor Overview Source: https://docs.deeprails.com/monitor/overview Monitor is the 'Airtag' for GenAI workflows - it gives you full observability over every generative AI workflow you run. It continuously scores outputs with DeepRails’ Guardrail Metrics; tracks usage, cost, latency, and failure rates in real time; and surfaces regressions before they reach your users. ## Why Monitor Exists Blind spots in production GenAI cost teams time, money, and trust. Monitor identifies and closes those gaps. It evaluates live traffic with the same research-backed Guardrail Metrics and Extended AI Capabilities used across DeepRails and highlights trends and regressions so you can fix issues fast and improve with confidence. ## Key Definitions * **Guardrail Metrics:** DeepRails’ General-Purpose Guardrail Metrics for correctness, completeness, adherence (instruction, context, ground truth), and comprehensive safety. Custom Guardrail Metrics are supported on SME & Enterprise plans. * **Monitor:** A read-only evaluation pipeline for a specific LLM use case or surface. A monitor receives your input/output pairs and model metadata, scores them with selected guardrails, and exposes real-time metrics, trends, and drill-downs. It does **not** remediate outputs (use **Defend** for correction). * **Nametag:** An optional label you attach to events (e.g., “staging”, “release-2025-09”, “feature-x”) to slice charts, compare cohorts, and run A/B or pre/post analyses. * **Extended AI Capabilities:** Many LLM applications have to go beyond model knowledge. DeepRails provides access to advanced tools like web and file search and context awareness for evaluations if needed. ## How to Use Monitor Maximizing Monitor's diagnostic potential is simple when following these steps: Name the use case you want to observe and select the guardrail metrics to apply. If the use case being monitored uses tools like file or web search, enable those Extended AI Capabilities for the Monitor. Send each model completion (input/output pair, model used, and optional nametag). Monitor ingests this traffic continuously from staging or production. Monitor scores every output against your selected guardrails, associates operational signals (latency, tokens, cost), and indexes the event for search and analysis. Dashboards update in real time—track request volume, failure rate, latency, tokens, and cost; examine guardrail distributions; compare cohorts via filters and time windows. Drill into any event to see per-metric scores and rationales. Use findings to fix prompts, tune models, adjust thresholds, and more. ## Console Walkthrough The Monitor Console brings observability to life across three tabs: **Metrics**, **Data**, and **Manage Monitors**. ### Monitor Metrics The **Monitor Metrics** tab shows real-time operational and quality performance for a selected monitor: request volume, failure rate, latency, tokens, and cost—plus guardrail score distributions that reveal drift and regressions at a glance. Monitor Metrics dashboard with cost, volume, failure rate, tokens, latency and guardrail histograms ### Monitor Data The **Monitor Data** tab lists every evaluated event for deep inspection. Filter by monitor, metrics, status, model, date range, or nametag; search by run ID or prompt; and open any row to view full details. Monitor Data table of evaluated runs with filters and guardrail score columns Monitor Data event detail panel with evaluation metrics and rationales ### Manage Monitors The **Manage Monitors** tab is where you create and maintain all monitors associated with your account. See when each monitor last received traffic, how many outputs it has evaluated, and more. Manage Monitors screen with create form and active monitors list #### Creating a Monitor The creation wizard walks you through three simple steps to define how Monitor will analyze outputs: Start by naming your monitor and (optionally) describing what it protects against. This helps keep monitors organized and clear for your team. Monitor creation step 1 basic information Choose which guardrail metrics the monitor should analyze. Multiple guardrails can be combined, including correctness, completeness, adherence, and safety. Monitor creation step 2 select metrics Choose which additional tools will be needed to complete evaluations for your workflow. Each will add cost, so only select a tool if your initial model uses it. Monitor creation step 3 select metrics # Quickstart Guide Source: https://docs.deeprails.com/monitor/quickstart Get started with the Monitor API in minutes. ### Create an API Key 1. In your organization’s DeepRails API Console, go to API Keys.
2. Click Create key, name it, then copy the key.
3. (Optional) Save it as the DEEPRAILS\_API\_KEY environment variable.
API Keys – placeholder
### Install the SDK ```python theme={null} pip install deeprails ``` ```ts theme={null} npm install deeprails ``` ```Ruby theme={null} gem install deeprails ``` ```go theme={null} go get github.com/deeprails/deeprails-go-sdk@latest ``` ### Create a Monitor Before you can send events, you need to create a monitor. A monitor is a container for tracking production events and their evaluations. In this example, the evaluations in the monitor will leverage a file as context, which will need to be uploaded first. > Tip: You can also create a monitor via the DeepRails API Console. ```python theme={null} from deeprails import DeepRails, omit # Initialize (env var DEEPRAILS_API_KEY is recommended) client = DeepRails(api_key="YOUR_API_KEY") try: with open("example.txt", "rb") as file: file_response = client.post( "/files/upload", cast_to=object, files={"files": file}, options={"headers": {"Content-Type": omit}}, ) file_item = file_response[0] if isinstance(file_response, list) and file_response else file_response file_id = file_item.get("file_id") if isinstance(file_item, dict) else None if not file_id: raise ValueError("File upload failed") # Create a monitor monitor = client.monitor.create( name="Production Chat Assistant Monitor", description="Monitoring our production chatbot responses", guardrail_metrics=[ "completeness", "correctness" ], web_search=True, file_search=[ file_id ], context_awareness=True, ) print(f"Monitor created:\n{monitor}") except Exception as e: print(f"Error: {e}") ``` ```ts theme={null} import DeepRails from "deeprails"; import * as fs from "fs"; async function main() { // Initialize (env var DEEPRAILS_API_KEY is recommended) const client = new DeepRails({ apiKey: process.env.DEEPRAILS_API_KEY ?? "YOUR_API_KEY", }); try { const fileStream = fs.createReadStream("example.txt"); const files = [fileStream] as unknown as string[]; const fileResponse = await client.files.upload({ files } as any); const fileItem = Array.isArray(fileResponse) ? fileResponse[0] : fileResponse; if (!fileItem?.file_id) { throw new Error("File upload failed"); } const fileId = fileItem.file_id; // Create a monitor const monitor = await client.monitor.create({ name: "Production Chat Assistant Monitor", description: "Monitoring our production chatbot responses", guardrail_metrics: ["completeness", "correctness"], web_search: true, file_search: [fileId], context_awareness: true, }); console.log("Monitor created:", monitor); } catch (e) { console.error(`Error: ${e}`); } } main().catch(console.error); ``` ```Ruby theme={null} require "deeprails" # Initialize (env var DEEPRAILS_API_KEY is recommended) client = Deeprails::Client.new( api_key: ENV["DEEPRAILS_API_KEY"] || "YOUR_API_KEY" ) begin file_response = client.files.upload( files: [File.open("example.txt")] ) file_item = file_response.is_a?(Array) ? file_response.first : file_response file_id = file_item.respond_to?(:file_id) ? file_item.file_id : file_item["file_id"] raise "File upload failed" if file_id.to_s.empty? # Create a monitor monitor = client.monitor.create( name: "Production Chat Assistant Monitor", description: "Monitoring our production chatbot responses", guardrail_metrics: [ "completeness", "correctness" ], web_search: true, file_search: [ file_id ], context_awareness: true, ) puts "Monitor created:\n#{monitor}" rescue => e puts "Error: #{e.message}" end ``` ```go theme={null} package main import ( "bytes" "context" "encoding/json" "fmt" "io" "log" "mime/multipart" "net/http" "os" "github.com/deeprails/deeprails-go-sdk" "github.com/deeprails/deeprails-go-sdk/option" ) func uploadFile(apiKey, path string) (string, error) { file, err := os.Open(path) if err != nil { return "", err } defer file.Close() body := &bytes.Buffer{} writer := multipart.NewWriter(body) part, err := writer.CreateFormFile("files", file.Name()) if err != nil { return "", err } if _, err := io.Copy(part, file); err != nil { return "", err } if err := writer.Close(); err != nil { return "", err } req, err := http.NewRequest(http.MethodPost, "https://api.deeprails.com/files/upload", body) if err != nil { return "", err } req.Header.Set("Authorization", "Bearer "+apiKey) req.Header.Set("Content-Type", writer.FormDataContentType()) resp, err := http.DefaultClient.Do(req) if err != nil { return "", err } defer resp.Body.Close() if resp.StatusCode >= http.StatusBadRequest { return "", fmt.Errorf("file upload failed with status %s", resp.Status) } var fileResponses []deeprails.FileResponse if err := json.NewDecoder(resp.Body).Decode(&fileResponses); err != nil { return "", err } if len(fileResponses) == 0 || fileResponses[0].FileID == "" { return "", fmt.Errorf("file upload failed") } return fileResponses[0].FileID, nil } func main() { apiKey := "YOUR_API_KEY" client := deeprails.NewClient(option.WithAPIKey(apiKey)) fileID, err := uploadFile(apiKey, "example.txt") if err != nil { log.Fatal(err) } // Create a monitor monitorResponse, err := client.Monitor.New(context.TODO(), deeprails.MonitorNewParams{ Name: deeprails.F("Production Chat Assistant Monitor"), Description: deeprails.F("Monitoring our production chatbot responses"), GuardrailMetrics: deeprails.F([]deeprails.MonitorNewParamsGuardrailMetric{ deeprails.MonitorNewParamsGuardrailMetricCorrectness, deeprails.MonitorNewParamsGuardrailMetricCompleteness, }), WebSearch: deeprails.F(true), FileSearch: deeprails.F([]string{fileID}), ContextAwareness: deeprails.F(true), }) if err != nil { log.Fatal(err) } fmt.Printf("Monitor created with ID: %s\n", monitorResponse.MonitorID) } ``` #### Required Parameters | Field | Type | Description | | ------------------------------- | --------- | ------------------------------------------------------------------------------------------- | | name | string | The human-readable name of the monitor | | guardrail\_metrics | string\[] | A list of one or more metrics that events associated with this monitor will be evaluated on | #### Optional Parameters | Field | Type | Description | | ------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | description | string | A description of the monitor and/or the associated production use case | | web\_search | boolean | Whether or not web search is added as an extended capability for this monitor's evaluations. Defaults to false | | file\_search | string\[] | A list of uploaded file IDs to be used for file search in this monitor's evaluations. Upload files first via /files/upload. If nothing is passed, file search will not be used | | context\_awareness | boolean | Whether or not context awareness is added as an extended capability for this monitor's evaluations. Defaults to false | ### Send Your First Monitor Event Use the SDK to log a production event (input + output). The SDK automatically triggers an **evaluation** of the guardrail metrics assigned to the monitor and links the result to the event. Retrieve the details of the event until it gives a `completed` status; then, you can view the outcome of the evaluation. ```python theme={null} from deeprails import DeepRails import time # Initialize (env var DEEPRAILS_API_KEY is recommended) client = DeepRails(api_key="YOUR_API_KEY") # Create a monitor event (get the monitor_id from Console → Monitors) created = client.monitor.submit_event( monitor_id="mon_xxxxxxxxxxxx", model_input={ "system_prompt": "You are a helpful tutor specializing in AP science classes.", "user_prompt": "Explain the difference between mitosis and meiosis in one sentence.", "context": [{"role": "user", "content": "I have an AP Bio exam tomorrow, can you help me study?"}, {"role": "tutor", "content": "Sure, I'll help you study."}] }, model_output="Mitosis produces two genetically identical diploid cells for growth and tissue repair, whereas meiosis generates four genetically varied haploid gametes for sexual reproduction.", run_mode="fast", ) print(created) time.sleep(5) status = "" while status != "completed": time.sleep(1) event = client.monitor.retrieve_event(event_id=created.event_id, monitor_id=created.monitor_id) status = event.status print(event.evaluation_result) ``` ```ts theme={null} import DeepRails from "deeprails"; async function main() { // Initialize (env var DEEPRAILS_API_KEY is recommended) const client = new DeepRails({ apiKey: process.env.DEEPRAILS_API_KEY ?? "YOUR_API_KEY", }); // Create a monitor event (get the monitor_id from Console → Monitors) const created = await client.monitor.submitEvent( "mon_xxxxxxxxxxxx", { model_input: { system_prompt: "You are a helpful tutor specializing in AP science classes.", user_prompt: "Explain the difference between mitosis and meiosis in one sentence.", context: [ { role: "user", content: "I have an AP Bio exam tomorrow, can you help me study?" }, { role: "tutor", content: "Sure, I'll help you study." } ], }, model_output: "Mitosis produces two genetically identical diploid cells for growth and tissue repair, whereas meiosis generates four genetically varied haploid gametes for sexual reproduction.", run_mode: "fast", } ); console.log(created); await new Promise(resolve => setTimeout(resolve, 5000)); let status = ""; let event; while (status !== "completed") { await new Promise(resolve => setTimeout(resolve, 1000)); event = await client.monitor.retrieveEvent(created.event_id, { monitor_id: created.monitor_id, }); status = event.status ?? ""; } console.log(event?.evaluation_result); } main().catch(console.error); ``` ```Ruby theme={null} require "deeprails" # Initialize (env var DEEPRAILS_API_KEY is recommended) client = Deeprails::Client.new( api_key: ENV["DEEPRAILS_API_KEY"] || "YOUR_API_KEY" ) # Create a monitor event (get the monitor_id from Console → Monitors) created = client.monitor.submit_event( "mon_xxxxxxxxxxxx", { model_input: { system_prompt: "You are a helpful tutor specializing in AP science classes.", user_prompt: "Explain the difference between mitosis and meiosis in one sentence.", context: [ { role: "user", content: "I have an AP Bio exam tomorrow, can you help me study?" }, { role: "tutor", content: "Sure, I'll help you study." } ] }, model_output: "Mitosis produces two genetically identical diploid cells for growth and tissue repair, whereas meiosis generates four genetically varied haploid gametes for sexual reproduction.", run_mode: "fast", } ) puts created sleep(5) status = nil while status != :completed sleep(1) event = client.monitor.retrieve_event(created.event_id, monitor_id: created.monitor_id) status = event.status end puts event.evaluation_result ``` ```go theme={null} package main import ( "context" "fmt" "log" "time" "github.com/deeprails/deeprails-go-sdk" "github.com/deeprails/deeprails-go-sdk/option" ) func main() { apiKey := "YOUR_API_KEY" client := deeprails.NewClient(option.WithAPIKey(apiKey)) // Create a monitor event (get the monitor_id from Console → Monitors) created, err := client.Monitor.SubmitEvent(context.TODO(), "mon_xxxxxxxxxxxx", deeprails.MonitorSubmitEventParams{ ModelInput: deeprails.F(deeprails.MonitorSubmitEventParamsModelInput{ SystemPrompt: deeprails.F("You are a helpful tutor specializing in AP science classes."), UserPrompt: deeprails.F("Explain the difference between mitosis and meiosis in one sentence."), Context: deeprails.F([]deeprails.MonitorSubmitEventParamsModelInputContext{ {Role: deeprails.F("user"), Content: deeprails.F("I have an AP Bio exam tomorrow, can you help me study?")}, {Role: deeprails.F("tutor"), Content: deeprails.F("Sure, I'll help you study.")}, }), }), ModelOutput: deeprails.F("Mitosis produces two genetically identical diploid cells for growth and tissue repair, whereas meiosis generates four genetically varied haploid gametes for sexual reproduction."), RunMode: deeprails.F(deeprails.MonitorSubmitEventParamsRunModeFast), }, ) if err != nil { log.Fatal(err) } fmt.Printf("Event created: %+v\n", created) time.Sleep(5 * time.Second) var event *deeprails.MonitorEventDetailResponse for event == nil || event.Status != deeprails.MonitorEventDetailResponseStatusCompleted { time.Sleep(1 * time.Second) var err error event, err = client.Monitor.GetEvent(context.TODO(), created.MonitorID, created.EventID) if err != nil { log.Fatal(err) } } fmt.Printf("Evaluation Result:\n%+v\n", event.EvaluationResult) } ``` #### Required Parameters | Field | Type | Description | | -------------------------- | ------ | -------------------------------------------------------------------------------------------- | | monitor\_id | string | The ID of the monitor to receive the event (find it in Console → Monitor → Manage Monitors). | | model\_input | object | Must include at least a user\_prompt. | | model\_output | string | The LLM output to be evaluated and recorded with the event. | #### Optional Parameters | Field | Type | Description | | ---------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | run\_mode | string | Run mode for the monitor event that determines which models are used to evaluate the event. Available run modes (fastest to most thorough): super\_fast, fast, precision, precision\_codex, precision\_max, and precision\_max\_codex. Defaults to fast. Note: super\_fast does not support Web Search or File Search — if your monitor has these enabled, use a different run mode or edit the monitor to disable them. | ### Retrieve Monitor and Event Details You can retrieve a monitor's details via API including a list of recent events and stats on associated evaluation progress. ```python theme={null} from deeprails import DeepRails client = DeepRails(api_key="YOUR_API_KEY") try: # Get monitor details monitor = client.monitor.retrieve(monitor_id="mon_xxxxxxxxxxxx") print(monitor) except Exception as e: print(f"Error: {e}") ``` ```ts theme={null} import DeepRails from "deeprails"; async function main() { const client = new DeepRails({ apiKey: process.env.DEEPRAILS_API_KEY ?? "YOUR_API_KEY", }); try { // Get monitor details const monitor = await client.monitor.retrieve("mon_xxxxxxxxxxxx"); console.log(monitor); } catch (e) { console.error(`Error: ${e}`); } } main().catch(console.error); ``` ```Ruby theme={null} require "deeprails" client = Deeprails::Client.new( api_key: ENV["DEEPRAILS_API_KEY"] || "YOUR_API_KEY" ) begin # Get monitor details monitor = client.monitor.retrieve("mon_xxxxxxxxxxxx") puts monitor rescue => e puts "Error: #{e.message}" end ``` ```go theme={null} package main import ( "context" "fmt" "log" "github.com/deeprails/deeprails-go-sdk" "github.com/deeprails/deeprails-go-sdk/option" ) func main() { apiKey := "YOUR_API_KEY" client := deeprails.NewClient(option.WithAPIKey(apiKey)) // Get monitor details monitor, err := client.Monitor.Get(context.TODO(), "mon_xxxxxxxxxxxx", deeprails.MonitorGetParams{}) if err != nil { log.Fatal(err) } fmt.Printf("%+v\n", monitor) } ``` ### Check Monitor Analytics via the API Console 1. Open DeepRails API Console → Monitor → Data.
2. Filter by model, time range, or search by monitor\_id to find events.
3. Open any event to see the linked evaluation scores and rationales.
Monitor data – placeholder ### Next Steps Explore the metrics behind evaluations—correctness, safety, completeness, and more. Learn core Monitor concepts and how it fits into your flow. # DeepRails vs Competitors Source: https://docs.deeprails.com/vs-competitors Understanding the fundamental difference: Detect-Only vs Detect-and-Fix ## The Core Difference Most AI safety platforms focus on detection: flagging hallucinations, blocking bad outputs, or surfacing issues in dashboards. DeepRails adds a correction layer on top, automatically remediating problems before they reach your users. ## Detect-Only vs Detect-and-Fix **Detect-Only Approach** * Flag hallucinations * Block problematic outputs * Log failures for review * Return errors to users **Result**: Users get nothing or get the original flawed response **Detect-and-Fix Approach** * Detect hallucinations * Automatically remediate via FixIt or ReGen * Verify the correction passes all guardrails * Deliver the corrected response **Result**: Users always get accurate, safe responses ## Competitor Landscape
Platform Category Strength Limitation
AWS Bedrock Guardrails Cloud guardrails Content filtering and grounding checks at scale Rigid 5-point scoring; blocks bad outputs but cannot correct them
Patronus AI Evaluation Fast, lightweight judge models for scoring Scores only; your application still serves the original response
Atla Evaluation High-accuracy evaluation models (Selene) for LLM-as-judge tasks Evaluation-only; no remediation, no production guardrail layer
Guardrails AI Open-source validation Flexible framework with community validators Retries the entire request instead of correcting the specific failure
Galileo Observability Rich traces, debugging, and agent evaluation workflows Focused on what agents do and how to control agent behavior; DeepRails focuses on correcting what agents say
Respan.ai (formerly Keywords AI) Unified gateway for routing across LLM providers Optimizes model selection, not the quality of what models return Gateway and routing infrastructure only; does not evaluate, block, or remediate unsafe outputs at inference time
LangSmith Observability Deep LangChain integration, tracing, and dataset management Developer tooling for debugging; not a production safety layer
Arize AI Observability Model monitoring and drift detection at scale Monitors production metrics; does not intercept or correct at inference time
Vellum Prompt tooling Visual prompt engineering and workflow builder Development-time tooling with no production guardrail layer
## Why This Matters ### For Your Users A patient asks a healthcare chatbot about drug interactions and gets a hallucinated answer. DeepRails corrects it automatically using verified source material before the patient ever sees the mistake. ### For Your Business A legal research assistant that hallucinates case citations creates real liability. DeepRails catches and corrects the citation in real time, turning a potential compliance incident into a non-event. ### For Your Development Team Without auto-correction, every flagged hallucination becomes a ticket to investigate, fix, and redeploy. DeepRails handles remediation in production so your team can focus on building features. ## Technical Comparison: DeepRails vs AWS Bedrock We conducted a head-to-head evaluation study comparing DeepRails against AWS Bedrock Guardrails: * **45% more accurate** on Correctness evaluations * **53% more accurate** on Completeness evaluations * **Continuous 0-100% scoring** vs Bedrock's rigid 5-point scale * **Intelligent remediation** vs Bedrock's block-only approach Building reliable remediation requires multi-model consensus, granular evaluation, and deep research into correction strategies. DeepRails is the first platform to ship this as a production API. ## The Observability vs Guardrails Distinction A common point of confusion: observability tools (Galileo, LangSmith, Arize) and guardrail platforms (DeepRails, AWS Bedrock) solve different problems. **Observability tools** answer: *What went wrong, and when?* They trace requests, surface failures in dashboards, and help developers debug. They are essential for understanding your AI system — but they operate after the fact. By the time a hallucination appears in your Galileo trace, your user has already seen it. **Guardrail platforms** answer: *Can I stop this before it reaches the user?* They operate inline at inference time, evaluating and optionally correcting outputs before delivery. DeepRails is a guardrail platform — not an observability tool. The two categories are complementary. Many teams run DeepRails for production safety alongside LangSmith or Arize for tracing and debugging. ## See It In Action Ready to move beyond detection to actual remediation? [Start with our quickstart guide](/defend/quickstart) or [contact sales](mailto:sales@deeprails.ai) for an enterprise demo.