AI Inference Port Runbook

This runbook is the operating contract for DartCodeAI AI traffic. Product code, Suite modules, and licensed customer integrations must call the DartCodeAI ai-gateway; they must not import model-provider SDKs, call vendor APIs directly, or depend on provider-shaped response JSON.

The goal is to make the model backend replaceable while keeping entitlement, metering, auditing, token accounting, error handling, and response shape stable.

1. Architecture Boundary

Runtime AI requests follow this path:

Suite modules or licensed product integrations
  -> backend/ai-gateway
  -> dartcodeai_ai_runtime provider registry and router
  -> registered upstream provider
  -> AiRuntimeResponse-shaped gateway response

The boundary is intentionally narrow:

backend/ai-gateway/ is the only product-facing runtime entry point for model calls.
backend/ai_runtime/ owns the Dart DTOs, provider registry, router, runtime switch policy, error taxonomy, and parity tests.
aimodel/ owns self-hosted model implementation details and inference services.
Product feature modules consume the canonical output map from ai-gateway; they do not parse OpenAI choices, Gemini candidates, or any other upstream-specific shape.

This separation is what lets DartCodeAI switch from hosted providers to a self-hosted model without changing Suite feature modules or licensed product integrations.

2. Canonical Response Envelope

Every successful runtime response must normalize into the AiRuntimeResponse shape from backend/ai_runtime/lib/src/ai_runtime_contract.dart.

{
  "requestId": "request-123",
  "providerId": "internal_vllm",
  "output": {
    "text": "Generated result text"
  },
  "usage": {
    "promptTokens": 11,
    "completionTokens": 7,
    "totalTokens": 18
  },
  "finishReason": "stop",
  "confidence": 0.88,
  "metadata": {
    "routeKey": "chat.completions"
  },
  "completedAt": "2026-06-30T00:00:00.000Z"
}

Required fields:

requestId: the runtime request identifier.
providerId: the normalized provider that produced the response.
output: the provider-neutral payload consumed by product code.
metadata: routing, model, fallback, and diagnostic metadata.
completedAt: UTC completion timestamp.

Optional fields:

usage.promptTokens, usage.completionTokens, and usage.totalTokens: standardized token counts. The gateway maps upstream token fields into these names before recording usage.
finishReason: normalized upstream completion reason.
confidence: numeric confidence from 0 to 1 when the provider supplies one.

Provider adapters may receive provider-specific JSON, but that shape must be normalized before the response leaves ai-gateway.

3. Canonical Error Taxonomy

Runtime failures must normalize into the shared error taxonomy from AiRuntimeErrorCode and the gateway’s canonicalError response body.

Code HTTP status Retryable Meaning

Code	HTTP status	Retryable	Meaning
`rate_limited`	`429`	Yes	Provider or account-level rate limit.
`context_length_exceeded`	`413`	No	Request is too large for the selected model context window.
`auth_error`	`401`	No	Provider credential or gateway authorization failed.
`content_filtered`	`422`	No	Provider safety or policy filter blocked the request.
`timeout`	`504`	Yes	Upstream provider timed out.
`provider_unavailable`	`503`	Yes	Provider connection failed or upstream returned a service failure.
`capability_unsupported`	`422`	No	Selected provider does not support the requested runtime capability.
`invalid_provider_response`	`502`	No	Upstream response could not be normalized into the canonical envelope.

rate_limited

429

Yes

Provider or account-level rate limit.

context_length_exceeded

413

Request is too large for the selected model context window.

auth_error

401

Provider credential or gateway authorization failed.

content_filtered

422

Provider safety or policy filter blocked the request.

timeout

504

Yes

Upstream provider timed out.

provider_unavailable

503

Yes

Provider connection failed or upstream returned a service failure.

capability_unsupported

422

Selected provider does not support the requested runtime capability.

invalid_provider_response

502

Upstream response could not be normalized into the canonical envelope.

Gateway errors should expose the same response shape for every provider:

{
  "error": "provider_unavailable",
  "message": "Provider request failed.",
  "retryable": true,
  "failoverEligible": true,
  "canonicalError": {
    "code": "provider_unavailable",
    "message": "Provider request failed.",
    "httpStatusCode": 503,
    "retryable": true,
    "failoverEligible": true,
    "providerId": "internal_vllm"
  }
}

4. Provider Registry And Switching

Provider selection is config-driven. The gateway reads AI_GATEWAY_PROVIDER_REGISTRY when present and converts it into an AiProviderRegistry. The registry declares the stable provider, optional candidate provider, provider adapters, upstream URLs, model names, supported capabilities, and credential references.

{
  "stableProviderId": "openai",
  "candidateProviderId": "internal_vllm",
  "providers": [
    {
      "id": "openai",
      "adapter": "openai-compatible",
      "upstreamBaseUrl": "https://provider.example/v1",
      "model": "gpt-4.1-mini",
      "capabilities": ["chatCompletions", "embeddings"],
      "credentialRef": "OPENAI_API_KEY"
    },
    {
      "id": "internal_vllm",
      "adapter": "openai-compatible",
      "upstreamBaseUrl": "https://internal-model.example/v1",
      "model": "dartcodeai-codellama",
      "capabilities": ["chatCompletions"],
      "credentialRef": "AI_GATEWAY_PROPRIETARY_API_KEY"
    }
  ]
}

Rules:

Keep credential values out of committed config. Use credentialRef; gateway startup resolves credentials from environment variables or Secret Manager.
Declare each provider capability explicitly. The gateway rejects requests when the selected provider does not support the requested capability.
Put experimental hosted-provider use behind the existing allowlist policy. OpenAI is allowed only as an approved experiment, fallback, or configured stable provider.
Runtime switching must preserve the canonical response and error shape. Switching stableProviderId from openai to internal_vllm must not require product code changes.

5. Adding A New Provider

Use this process whenever a new hosted or self-hosted backend is added.

Define the provider id and capabilities.

Use normalized lower-case provider ids. Add only the capabilities the provider can satisfy: documentation, qa, releaseManagement, embeddings, or chatCompletions.
Add or select the adapter boundary.

Provider-specific SDKs and HTTP shapes belong inside the approved adapter or inference boundary. Product modules and Suite features must continue to call ai-gateway.
Add a registry entry.

Add the provider to AI_GATEWAY_PROVIDER_REGISTRY with id, adapter, upstreamBaseUrl, model, capabilities, and credentialRef. Do not commit secret values.
Prove success parity.

Run the runtime and gateway parity tests. The new provider must emit the same canonical response shape as the existing providers.
Prove error parity.

Run provider error cases through the gateway. Rate limits, auth errors, timeouts, unavailable provider failures, content filters, and invalid provider responses must normalize into the shared canonicalError shape.
Prove product boundary compliance.

Run the import guard. If it fails, move the provider-specific dependency or direct runtime reference into the approved adapter boundary instead of allowing product code to bypass ai-gateway.

6. Required Validation

Run the narrow checks for the area you changed before moving a work item to Peer Review.

# Runtime DTO, registry, switch policy, and parity tests.
cd dartcodeai/backend/ai_runtime
dart test

# Gateway request handling and provider parity tests.
cd ../ai-gateway/api
dart test test/provider_parity_test.dart test/server_handler_test.dart

# Architecture boundary guard from the repository root.
cd ../../..
python tools/lint/import_guard.py --root .
PYTHONPATH=tools/lint python -m unittest discover -s tools/lint -p 'test_*.py'

# Published docs build when this runbook changes.
cd dartcodeai/docs
npm run build

GitLab CI also runs the docs child pipeline for changes under dartcodeai/docs/**. Backend or lint changes should also pass the validate_architecture_import_guards job.

7. Import Guard Failures

The import guard exists to prevent accidental architecture drift. A failure means code or documentation introduced a direct provider dependency, direct provider import, or legacy local-model endpoint reference outside the approved adapter boundary.

Resolution steps:

Read the path and rule printed by tools/lint/import_guard.py.
If the violation is in product, Suite, licensing, or gateway orchestration code, remove the direct provider call and route through ai-gateway.
If the direct dependency is truly required, move it into the approved inference adapter boundary and add focused tests there.
Re-run the guard and its unit tests.
Do not bypass the guard by adding one-off exclusions unless the architecture boundary itself has deliberately changed.

8. Operational Checks

Before treating a provider switch or new model as ready:

Gateway startup validates the provider registry and credential references.
A request with Authorization: Bearer <api-key> and x-dartcodeai-product-key succeeds through ai-gateway.
The response contains the canonical envelope, not upstream provider JSON.
Usage records contain standardized token counts.
The selected provider is visible in providerId, x-dartcodeai-provider, or gateway metadata.
Error responses preserve the shared canonicalError shape.
Hosted-provider experiments remain allowlisted.
Product integrations do not gain direct model-provider credentials.

9. Ownership

Keep future changes in the correct owner:

Area Owns

Area	Owns
`backend/ai-gateway/`	Runtime AI ingress, entitlement checks, usage metering, request routing, provider policy, canonical response normalization.
`backend/ai_runtime/`	Runtime DTOs, registry abstractions, router, switch policy, canonical error taxonomy, provider parity tests.
`aimodel/`	Self-hosted model implementation, inference service behavior, model runtime operational details.
Product and Suite modules	Feature behavior that consumes `ai-gateway` output. These modules must not own provider-specific credentials or response parsing.

backend/ai-gateway/

Runtime AI ingress, entitlement checks, usage metering, request routing, provider policy, canonical response normalization.

backend/ai_runtime/

Runtime DTOs, registry abstractions, router, switch policy, canonical error taxonomy, provider parity tests.

aimodel/

Self-hosted model implementation, inference service behavior, model runtime operational details.

Product and Suite modules

Feature behavior that consumes ai-gateway output. These modules must not own provider-specific credentials or response parsing.

When this boundary is preserved, DartCodeAI can add, replace, or roll back model providers without rewriting product features or customer integrations.