Sort providers and control fallback

Price#

Price ranking compares the applicable input-tier rate plus the base output rate. It is not a prediction of the total request bill. Review input, output and cache prices for the selected provider and your expected workload.

Latency and throughput#

Latency uses observed completed-request elapsed time, not time to the first token. Throughput uses observed output tokens per elapsed second. Recent observations cover a five-minute window. These measurements guide routing; they do not guarantee a response time or capacity.

Providers without the required price or performance observation rank after measured providers in either direction. Stable identifiers break ties.

Order before sorting#

An explicit provider order takes precedence over sorting. Within the remaining eligible choices, the selected sort determines ranking. Ignored providers, providers outside only, unsupported operations and providers excluded by key permissions remain ineligible.

Fallback keeps the same model#

Provider fallback changes eligible infrastructure for the exact requested model. It does not switch to another model tag. Any upstream vendor choice made inside the ShareAI provider is separate from the public creator identity and execution-provider policy.

Handle interrupted requests#

Routing does not automatically retry after successful dispatch, an uncertain response or partial output. This avoids duplicate work and charges. Preserve partial output and let the application decide whether to start a new request. Avoid unlimited retry loops.

Examples#

{
  "provider": {
    "sort": {
      "by": "latency",
      "direction": "asc"
    },
    "allow_fallbacks": true
  }
}

To prefer highest throughput, use {"by":"throughput","direction":"desc"}. Reverse either direction only when that order matches your intended test or workload.