We are moving from a world where ‘Software is a Document’ (Code) to a world where ‘Software is a Process’ (Inference). In this new world, the compiler isn’t our last line of defense—the runtime is.
TL;DR: The Infrastructure Shift
- The Model is Not the System: In production, AI is no longer a prompt/response task; it is a distributed runtime problem involving state, memory, and orchestration.
- AI systems reintroduce durable execution concerns: Traditional cloud-native “stateless” patterns fail AI agents. Reliability requires Durable Execution—where a system doesn’t “forget” its train of thought if a container restarts.
- Code as a Process: We are moving from a world where software is a document (static code) to a world where software is a process (live inference). Success depends on runtime supervision, not just model benchmarks.
Introduction
Today, most of the current AI discussions focus on models:
- model capability
- benchmarks
- prompting
- fine-tuning
- context windows
But once AI systems move toward agentic, conversational, or long-running AI workflows and beyond isolated demos and into production environments, a different set of problems begins to dominate.
The difficult part is often no longer the model itself. It becomes the runtime. Traditional web apps (REST/CRUD) treat every request as a clean slate. AI requires “Session Persistence” on steroids.
The shift from model-centric to system-centric thinking
An LLM on its own is relatively simple: Input –> inference –> output
But production AI systems rarely operate in isolation. Real systems involve:
- orchestration
- tool execution
- memory
- retrieval
- retries
- supervision
- permissions
- temporal workflows
- streaming events
- human interaction
- distributed state
- observability
- failure handling
At that point, the architecture starts to resemble a distributed runtime system more than a traditional request-response application. It’s not just about saving a chat history in a database; it’s about Execution State.
In a stateless architecture, a container restart midway through a 5-step agentic workflow isn’t just a hiccup; it’s a lobotomy. The system loses its ‘train of thought’ because that state wasn’t durable.”
Runtime complexity emerges quickly
Even relatively small AI-enabled systems introduce runtime concerns such as:
- long-running execution
- partial failure
- coordination between components
- asynchronous workflows
- state persistence
- conversational continuity
- cost-aware execution
- retry semantics
- execution tracing
- non-deterministic behaviour
These are not prompt engineering problems. They are runtime architecture problems. In a runtime environment, you can’t “prompt” your way out of a 504 Gateway Timeout or a rate-limit error. If your system logic lives inside a prompt, you lose the ability to use standard Circuit Breakers or Backoff policies effectively. A “Runtime Problem” approach treats the LLM as a fallible library call, not the “brain” of the application.
Why traditional web architecture patterns are insufficient
Many existing application architectures assume:
- short-lived requests
- deterministic execution
- stateless services
- predictable control flow
AI-native systems challenge these assumptions. Agentic and conversational systems often require:
- persistent context
- dynamic planning
- adaptive execution paths
- background processing
- memory over time
- orchestration across multiple capabilities
This starts to push architectures toward:
- workflow runtimes
- actor systems
- event-driven orchestration
- distributed state models
- graph execution systems
The big gap I see here is that the industry is settled on cloud-native stateless architecture and not thinking about AI native stateful architecture. Many will introduce AI into their products with much support from the business. The side effects will start when production usage increases. The risk is that systems will become unstable, which will lead to the mistaken belief that the AI is at fault.
The return of orchestration
One of the most interesting developments in AI systems is the renewed importance of orchestration. For years, many architectures moved toward increasingly stateless service models. But intelligent systems frequently require:
- coordination
- supervision
- execution tracking
- runtime decision-making
- stateful interaction over time
This creates architectural pressure toward systems capable of managing:
- execution lifecycles
- state transitions
- temporal reasoning
- distributed coordination
In many ways, this resembles problems already explored in:
- actor systems
- workflow engines
- distributed systems
- event sourcing
- stream processing
The actor model provides supervision trees which act as runtime supervisors. Imagine an agent (worker) and a deterministic script (supervisor). The supervisor doesn’t just watch; it has the power to kill, restart, or roll back the Agent’s state. This moves AI safety (excluding AGI) from “ethics” to “systems engineering”.
Debugging Nightmare
In traditional software, if a bug happens at runtime, you can often replicate it with a unit test. In AI, because of non-determinism, the “bug” might only happen 1% of the time under specific concurrency loads. If we view AI as a runtime problem, we need tools that can “record” the exact state, tokens, and environmental variables to reconstruct a failure.
Observability becomes critical
AI systems also introduce new operational challenges. Traditional observability already struggles with distributed systems. AI-native systems add:
- probabilistic behaviour
- reasoning chains
- dynamic execution graphs
- runtime tool selection
- evolving context state
Understanding why a system behaved a certain way becomes significantly harder. This makes observability a first-class architectural concern rather than an operational afterthought.
Runtime architecture may become the real differentiator
Model capability will continue to improve and commoditise. But the systems capable of operating intelligently, safely, and reliably in production will increasingly depend on runtime architecture quality:
- orchestration
- supervision
- memory
- execution semantics
- observability
- coordination
- resilience
The future of AI systems may depend less on who has the best model, and more on who builds the best runtime.
Example
Closing thoughts
The industry is still early in understanding what production AI architecture really requires. Many current systems are still model-centric. But over time, I suspect the focus will increasingly shift toward runtime design:
- how intelligent systems coordinate?
- how they maintain state?
- how they reason over time?
- how they recover from failure?
- how they remain observable?
- how humans interact with them safely?
The interesting architectural problems are only just beginning.
Leave a Reply