Can the Decisions API Tell When to Retry a Deployment? A Test with 10 Failure Logs

A deployment fails and the last line says timed out. Would clicking Retry fix it? We reproduced two failures with identical first-attempt logs on a local machine. Only one succeeded on retry.

To see what advice we could offer on a failed deployment screen, we sent the first-attempt records to OpenAI’s Decisions API. It chose “Not enough information” for both cases with identical logs. In another case, the log showed that the same endpoint returned HTTP 200 immediately after the timeout. The API chose “Retry.”

First-attempt log shared by cases A and B

GET /artifact: TimeoutError: timed out
error: deployment step failed (TimeoutError: timed out)

Case A on retryThe server returned HTTP 200 and the step passed.

Case B on retryThe request timed out again and the step failed.

Figure 1. The first-attempt logs matched exactly, but retrying with the same settings produced different outcomes.

Below is the first-attempt log for the timeout followed by a successful probe. Compared with the log above, the second line records that the same endpoint returned HTTP 200.

GET /artifact: TimeoutError: timed out
follow-up probe of the same artifact endpoint: HTTP 200
error: deployment step failed (TimeoutError: timed out)

Reproduce failures and choose advice from the first attempt

We ran a small program locally to simulate the build, startup, and readiness stages of a deployment. We scripted what a local mock HTTP server would return on each request, so a retry with the same settings would succeed in some cases and fail in others. The program recorded the HTTP responses and Python exceptions from these simulated runs. We did not use production deployment logs or live services.

We made the first attempt fail in 10 cases and retried each one once with the same settings. Each HTTP request had a 200 ms timeout. The retry began 50 ms after the first-attempt log was complete. In recovery scenarios, we made one follow-up request to the same URL after the initial failure.

Input to the decisionFirst-attempt log, failed stage, and the follow-up response when available.

Select adviceChoose one of three options from the records provided.

Evaluate against the retryUse the result of one retry with identical settings to evaluate the decision.

Figure 2. We recorded the retry first for evaluation and sent only the first-attempt records to the API.

We set the criteria in advance and included them in the API’s option descriptions. Choose “Retry” if the record points to a temporary failure, such as when the endpoint has recovered. Choose “Check configuration” if it shows a specific configuration problem. Choose “Not enough information” if the record cannot distinguish between those possibilities.

We sent the failed stage and the first-attempt log to the Decisions API. We used choice with gpt-6-luna to select one of three options. The retry log and outcome were held back for evaluation. We made one API call per case, for a total of 10. How the Decisions API selects from fixed options

The estimated API cost for all 10 calls was about US$0.00026. The usage records showed 2,603 uncached input tokens. The base Decisions API rate for gpt-6-luna was US$0.10 per million uncached input tokens as of November 5, 2026. That gives 2,603 ÷ 1,000,000 × 0.10 = 0.0002603 US dollars. Output is free. The experiment ran locally, so there were no PaaS deployment charges. OpenAI Decisions API pricing

The two cases the final line could not distinguish

For comparison, we used a rule that reads the final line. It recommends “Retry” for a timeout, a dropped connection, HTTP 503, or HTTP 429, and “Check configuration” for HTTP 404 or a configuration error. Here is how often each approach matched the advice we defined in advance.

Expected advice	Decisions API	Last-line rule
Retry	4/4	4/4
Check configuration	4/4	4/4
Not enough information	2/2	0/2
Total	10/10	8/10

Table 1. Number of recommendations that matched the expected advice for each category.

In the four “Retry” cases, a dependency first returned HTTP 503 or HTTP 429, dropped the connection, or timed out. A follow-up request returned HTTP 200 in all four cases, and each retry passed with the same settings. The four “Check configuration” cases involved a missing dependency file, a missing required environment variable, the wrong readiness URL, or an invalid settings file. All four retries failed.

We designed the remaining two cases to be indistinguishable from the first-attempt records alone. The last-line rule recommended retrying both, and one retry failed. The API chose “Not enough information” for both. For these cases we evaluated whether the evidence available at the time supported recommending a retry. The later outcome was a separate observation.

This comparison gives the API the full first-attempt log and failed stage and the rule only the final line. Comparing them with the same information, and measuring accuracy on real deployments, would require further tests.

Logs from all 10 cases

The logs below came from the local simulation. Paths and setting names were created for the experiment. The logs contain no real service hostnames, credentials, or user data.

Each case shows the failed stage, the advice selected by the Decisions API, and both attempts’ logs. Where we sent a follow-up probe, its result appears in the first-attempt log.

1. The dependency temporarily returned HTTP 503

Failed stage: Build. Decisions API advice: Retry.

First-attempt log

GET /artifact: HTTP 503
follow-up probe of the same artifact endpoint: HTTP 200
error: deployment step failed (HTTP 503)

Retry log (succeeded)

GET /artifact: HTTP 200
deployment step passed

2. The dependency file was missing

Failed stage: Build. Decisions API advice: Check configuration.

First-attempt log

GET /artifact/not-in-lock: HTTP 404
the locked dependency artifact path returned not found
error: deployment step failed (HTTP 404)

Retry log (failed)

GET /artifact/not-in-lock: HTTP 404
the locked dependency artifact path returned not found
error: deployment step failed (HTTP 404)

3. The retry succeeded after a timeout (case A)

Failed stage: Build. Decisions API advice: Not enough information.

First-attempt log

GET /artifact: TimeoutError: timed out
error: deployment step failed (TimeoutError: timed out)

Retry log (succeeded)

GET /artifact: HTTP 200
deployment step passed

4. A required environment variable was missing

Failed stage: Startup. Decisions API advice: Check configuration.

First-attempt log

required runtime setting is missing: 'PORT'
error: startup validation failed

Retry log (failed)

required runtime setting is missing: 'PORT'
error: startup validation failed

5. The connection to the dependency dropped

Failed stage: Build. Decisions API advice: Retry.

First-attempt log

GET /artifact: RemoteDisconnected
follow-up probe of the same artifact endpoint: HTTP 200
error: deployment step failed (RemoteDisconnected)

Retry log (succeeded)

GET /artifact: HTTP 200
deployment step passed

6. The readiness check used the wrong path

Failed stage: Readiness check. Decisions API advice: Check configuration.

First-attempt log

GET /ready: HTTP 404
configured readiness path is /ready. App route /health: HTTP 200
error: deployment step failed (HTTP 404)

Retry log (failed)

GET /ready: HTTP 404
configured readiness path is /ready. App route /health: HTTP 200
error: deployment step failed (HTTP 404)

7. The dependency temporarily returned HTTP 429

Failed stage: Build. Decisions API advice: Retry.

First-attempt log

GET /artifact: HTTP 429
follow-up probe of the same artifact endpoint: HTTP 200
error: deployment step failed (HTTP 429)

Retry log (succeeded)

GET /artifact: HTTP 200
deployment step passed

8. The settings file could not be parsed

Failed stage: Build. Decisions API advice: Check configuration.

First-attempt log

could not parse build-settings.json: Expecting value: line 1 column 16 (char 15)
error: build configuration is invalid

Retry log (failed)

could not parse build-settings.json: Expecting value: line 1 column 16 (char 15)
error: build configuration is invalid

9. The retry also timed out (case B)

Failed stage: Build. Decisions API advice: Not enough information.

First-attempt log

GET /artifact: TimeoutError: timed out
error: deployment step failed (TimeoutError: timed out)

Retry log (failed)

GET /artifact: TimeoutError: timed out
error: deployment step failed (TimeoutError: timed out)

10. A follow-up request confirmed recovery after a timeout

Failed stage: Build. Decisions API advice: Retry.

First-attempt log

GET /artifact: TimeoutError: timed out
follow-up probe of the same artifact endpoint: HTTP 200
error: deployment step failed (TimeoutError: timed out)

Retry log (succeeded)

GET /artifact: HTTP 200
deployment step passed

Check the failed stage and endpoint before retrying

If a build could not fetch a dependency file, check whether the URL for that exact file responds now. If the readiness check failed, compare the path the app serves with the configured check path. If all you can see is a timeout, look for logs that identify which stage and endpoint failed.

On a deployment failure screen, we could show the advice alongside the facts that support it. For example, “The dependency file could not be fetched, but the endpoint responded to the follow-up request” gives users a reason to retry. In this experiment the API returned a choice. The facts displayed next to it would come from the logs.

KamuiDash is a PaaS for deploying web apps from GitHub. KamuiDash