Job debugging

Diagnose failed Livepeer AI and video jobs: error patterns by HTTP status, cold model timeouts, validation errors, and the network-direct troubleshooting flow.

Errors by HTTP status code

Cold model timeouts

A model is “cold” when no orchestrator on the network has it loaded in GPU memory. The first request triggers a model load, which takes 30 seconds to 5 minutes depending on model size and GPU.

A cold model returns a 503 or a long pending response. This is not an error in your code.

Mitigation: use warm models for latency-sensitive applications. The following models are kept warm across the network:

Pipeline	Warm model
text-to-image	SG161222/RealVisXL_V4.0_Lightning
image-to-image	timbrooks/instruct-pix2pix
audio-to-text	openai/whisper-large-v3
image-to-text	Salesforce/blip-image-captioning-large
LLM	meta-llama/Meta-Llama-3.1-8B-Instruct

Implement retry with exponential backoff for any request that may target a cold model:

async function callWithRetry(fn: () => Promise<any>, maxRetries = 5): Promise<any> {
  let delay = 5000; // 5 seconds initial
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (err: any) {
      if (err.statusCode === 503 && attempt < maxRetries) {
        await new Promise(r => setTimeout(r, delay));
        delay = Math.min(delay * 1.5, 60000); // cap at 60 seconds
      } else {
        throw err;
      }
    }
  }
}

Diagnosing a non-responsive job

When a request hangs with no response, check in this order:

1. Network connectivity to the gateway:

   curl -I https://dream-gateway.livepeer.cloud/text-to-image
   # Expect: HTTP 405 (method not allowed on GET is correct)
   

2. Request construction using curl to isolate from SDK behaviour:

   curl -v -X POST https://dream-gateway.livepeer.cloud/text-to-image \
     -H "Content-Type: application/json" \
     -d '{"prompt": "test", "model_id": "SG161222/RealVisXL_V4.0_Lightning"}'
   

3. Model availability: use a known warm model to confirm the integration works, then switch to your target model.

4. Gateway availability: if using a self-hosted or third-party gateway, confirm the gateway process is running and the HTTP port is reachable. For self-hosted gateways, check livepeer_cli -status and the TicketBroker deposit balance.

5. Orchestrator availability: check tools.livepeer.cloud/ai/network-capabilities to confirm orchestrators are advertising the pipeline and model you need.

422 validation errors

A 422 response includes a body that identifies the failing field:

{
  "detail": [
    {
      "loc": ["body", "model_id"],
      "msg": "field required",
      "type": "value_error.missing"
    }
  ]
}

Common causes:

• model_id is missing (required on all pipelines)
• model_id format is wrong: must be a Hugging Face model ID string, e.g. SG161222/RealVisXL_V4.0_Lightning
• Image input sent as JSON instead of multipart/form-data (image-to-image, upscale, segment-anything-2)
• Dimension values are not integers (use 1024, not "1024")

Video transcoding and streaming failures

Video jobs fail for different reasons than AI jobs:

Symptom	Likely cause	Fix
RTMP push rejected	Stream key wrong or stream deleted	Verify stream key matches the stream object
HLS manifest empty	Transcoding has not completed first segment	Wait 5-10 seconds after RTMP connection
Playback fails with 403	Access control policy blocking playback	Check JWT or webhook playback policy configuration
Recording not available	Stream did not go idle cleanly	Check stream.idle webhook; recording generates on idle
Webhook not firing	Webhook URL unreachable or signature mismatch	Verify webhook URL is publicly reachable and Livepeer-Signature verification passes

Getting help

If the above steps do not resolve the issue:

• Discord: #lounge and #ai-research channels in the Livepeer Discord. Include your request body (redact API keys), the response status, and the response body.
• Forum: forum.livepeer.org for in-depth debugging questions.
• GitHub: File an issue against livepeer/ai-runner for AI failures or livepeer/go-livepeer for video and node issues.

See help for the full channel reference.