Configure AI Models Instead of Hard-Coding Them

Configure AI Models Instead of Hard-Coding Them

Revised
Published
~ 5 min read

#ai#architecture#economics#software engineering

On 30 July, OpenAI reduced the API price of GPT-5.6 Luna by 80% and Terra by 20%. It also introduced Fast mode for Sol, which runs at up to 2.5 times the speed for twice the standard price.

These changes affect the cost and response-time assumptions used when choosing a model. Keep model names in configuration rather than repeating them throughout the application.

Model Configuration

Application code should request a type of work rather than a specific model. For example:

export const modelPolicy = {
    planning: process.env.AI_PLANNING_MODEL ?? "gpt-5.6-sol",
    implementation: process.env.AI_IMPLEMENTATION_MODEL ?? "gpt-5.6-luna",
    review: process.env.AI_REVIEW_MODEL ?? "gpt-5.6-terra",
} as const;

export type ModelRole = keyof typeof modelPolicy;

export function getModel(role: ModelRole) {
    return modelPolicy[role];
}

The rest of the application calls getModel("planning"), getModel("implementation") or getModel("review"). A model change is made in the policy or environment configuration.

The policy can also contain timeouts, budgets and regional restrictions. If the application supports several providers, store the provider and model together and validate the pair. Keep provider-specific features visible where the application uses them. Tool calling, structured output, caching and streaming differ between APIs.

Abstract illustration of a stable application core separated from changing model and price choices by a clear routing boundary

Application code requests a type of work. Configuration selects the model.

Cost per Successful Result

OpenAI’s published prices put Luna at $0.20 per million input tokens and $1.20 per million output tokens. Terra costs $2 and $12 respectively. OpenAI also reports the performance figure for Fast mode. These are vendor figures and need testing against the work performed by the application.

Token prices are one part of the cost. Use the cost of a successful result:

model and tool charges + retry charges + manual-review cost + error cost
------------------------------------------------------------------------
                           successful results

Consider a customer service process that extracts an order number, classifies the issue and drafts a reply. A cheaper model may reduce the cost of the first attempt. It may still cost more overall if it sends additional cases for manual review.

Record the task, model version, token and tool cost, response time, retries, test result and any manual intervention. Use these figures when comparing models.

Model Choice by Task

OpenAI suggests using Sol to plan a change and Luna to implement it and run the tests. The same approach can be applied to other model providers.

A planning error can affect every later step. Implementation can use a cheaper or faster model when the task is narrow and has deterministic tests. Review can use a different model or provider where an independent result is useful.

Select each route according to the cost of failure and the checks available for that task.

Escalation from a cheaper model also needs care. A failed attempt may already have edited a file, called an API or changed external state. Isolate attempts and make side effects idempotent or deduplicate them. Use rollback where neither is possible.

Abstract illustration of different model paths handling planning, implementation and verification before converging on one tested outcome

Planning, implementation and review can use different models.

Testing a Route

Create a small replayable set of work from the application. Include normal inputs, known failures and cases where an incorrect result would be expensive. Score the result that matters, such as a valid extraction, an accepted patch or a correct sequence of tool calls.

Run the same set against the current and proposed routes. Compare pass rate, response time and cost per successful result. Change the route when the proposed model meets the required standard at a lower cost, or when a more expensive model improves the result enough to justify its price.

This test set can expose behavioural regressions after a model alias, prompt or tool changes. Run each case more than once where model output varies.

OpenAI’s engineering account describes work on routing, scheduling, caching, kernels and context management. If a task makes 30 model requests, one extra second per request adds 30 seconds to the task. Measure the model and the code around it together.

Task Limits

Lower model prices can increase total usage. An agent may make repeated model and tool calls while planning, correcting and reviewing its work.

A task-limit configuration might look like this:

const taskLimits = {
    maxAttempts: 3,
    maxToolCalls: 20,
    maxTokens: 100_000,
    maxCostUsd: 0.5,
};

The orchestration code must update the usage totals after each call and check them before starting another one. Attempt and tool-call limits can be hard limits. Token and cost limits are soft circuit breakers unless the code reserves the maximum cost before each request and sets a suitable output-token limit. Record why the task stopped.

A Sensible Starting Point

A small application with light usage may need one model and no routing logic. It should still keep the model name in one configuration file.

Search the repository for the current model ID. Move repeated values into configuration, record the cost and result of each task, and add another route when the tests show a useful difference.

When model prices change, update the configuration and run the tests again.

all posts →