Serverless functions and fine-grained microservices solve many deployment problems. They simplify scaling, reduce operational burden for individual teams, and map well to organisational boundaries. But they introduce latency in ways that monolithic architectures do not. A single user-facing request that fans out across a dozen Lambda functions, each making its own downstream calls, accumulates delays at every hop. Cold starts, serialisation overhead, connection setup, and queue wait times compound. The result is often acceptable median latency but painful tail latency, the p99 and p99.9 that determine whether the system feels fast or sluggish for the users who matter most.

This article examines where latency originates in serverless and microservice environments, then works through practical strategies for reducing it: cold start mitigation, caching patterns, queue-based load levelling, connection management, and latency budgeting. These concerns build on foundations covered in queueing theory and latency budgets, multi-layer caching, and the observability requirements needed to diagnose latency problems across heterogeneous infrastructure.

Where latency comes from in serverless

Understanding where time goes is the prerequisite for reducing it. In serverless and microservice architectures, latency accumulates from several distinct sources.

Cold starts are the most discussed source, though not always the most significant. When a serverless platform needs to create a new execution environment for a function, it must allocate a container or microVM, load the runtime, initialise the application code, and establish any connections the function needs. For a Java function with a large dependency tree, this can add 1 to 5 seconds. For a lightweight Node.js or Python function, cold starts are typically 100 to 500 milliseconds. The impact depends on how frequently cold starts occur relative to warm invocations.

Service-to-service calls add network round-trip time at every hop. In a microservice architecture where a single request touches eight services sequentially, even 10 milliseconds per hop adds 80 milliseconds. Fan-out patterns, where a service calls multiple downstream services in parallel, bound latency by the slowest responder rather than the sum, but the slowest responder is often slower than you expect due to tail latency effects.

Serialisation and deserialisation consume CPU time at every boundary. A service that receives a 50 KB JSON payload, processes it, and serialises a response pays this cost twice per hop. Binary formats like Protocol Buffers reduce this cost but add schema management complexity.

Queue wait time applies whenever asynchronous processing is involved. A message sitting in an SQS queue or Kafka topic partition adds latency equal to its wait time. Under low load this is negligible. Under high load, as queueing theory predicts, wait times grow non-linearly as utilisation increases.

DNS resolution and connection setup affect the first request to any endpoint. TLS handshakes in particular add one to two round trips. In environments where functions are short-lived and connections are not reused, this cost is paid repeatedly.

Cold start mitigation

Cold starts cannot be eliminated entirely in a true serverless model, but they can be reduced to the point where they rarely affect user-facing latency.

Provisioned concurrency (AWS Lambda) and equivalent features on other platforms keep a specified number of execution environments warm and ready to handle requests. This converts cold starts from a runtime problem into a capacity planning problem: you pay for the provisioned environments whether or not they are serving traffic, but you guarantee that the first N concurrent requests never experience a cold start. The trade-off is cost. Provisioned concurrency at scale can be expensive, so it should be applied selectively to latency-sensitive functions rather than blanket-applied to everything.

Snap Start (AWS Lambda for Java) and similar snapshot mechanisms capture the initialised execution environment after the init phase. Subsequent cold starts restore from the snapshot rather than re-running initialisation, dramatically reducing startup time for JVM-based functions.

Container reuse optimisation means structuring code so that expensive initialisation happens once per container lifecycle. Database connection pools, SDK clients, and configuration should be initialised outside the handler function and reused across invocations.

Smaller deployment packages reduce the time the platform spends loading your code. Pruning unused dependencies, using tree-shaking for JavaScript bundles, and avoiding monolithic frameworks when a lightweight alternative exists all contribute to faster cold starts. For compiled languages, static linking produces a single binary with no runtime dependency resolution.

Runtime selection matters. Go and Rust produce small, statically linked binaries with minimal startup time. Python and Node.js start quickly but can be slow if they import large libraries at module level. Java is the slowest to cold-start in traditional configurations, though Snap Start and GraalVM native images have narrowed the gap significantly.

Caching patterns for microservices

Caching is the most effective latency reduction tool available, but in microservice architectures the question is where to cache and how to manage consistency.

Function-level caching stores results in the execution environment's memory. This works for data that is read frequently, changes rarely, and is tolerant of staleness. A Lambda function that resolves feature flags on every invocation can cache the flag values for 60 seconds and avoid a remote call on most invocations. The limitation is that each execution environment has its own cache, so there is no sharing across instances. Cache hit rates depend on how traffic distributes across environments.

Shared cache layers (Redis, Memcached) provide a cache all instances can read from. In serverless environments, the network round trip to a regional cache (1 to 3 milliseconds) replaces a database query taking 10 to 50 milliseconds. The cache hierarchy from edge to origin applies: function-local cache, shared regional cache, and origin database form a tiered system where each layer absorbs a fraction of reads.

Edge caching for APIs pushes responses closer to users. CDN-based caching works well for read-heavy, personalisation-light endpoints. Cache key design is critical: over-specific keys destroy hit rates, while over-general keys serve stale data.

Write-through and write-behind patterns manage cache consistency when data changes. Write-through updates the cache synchronously with the database, ensuring consistency at the cost of write latency. Write-behind updates the cache immediately and persists asynchronously, offering lower latency but risking data loss if the async write fails.

Queue-based load levelling

Queues decouple producers from consumers, allowing each to operate at its own pace. This is fundamental to managing latency under variable load.

The core pattern is simple: instead of a service calling a downstream dependency synchronously and waiting for a response, it places a message on a queue and either returns immediately (for fire-and-forget operations) or polls for a result later. The downstream consumer processes messages at a sustainable rate regardless of how fast they arrive. This prevents the downstream service from being overwhelmed during traffic spikes, which would cause latency to climb non-linearly as utilisation approaches capacity.

Back-pressure is the mechanism by which a system signals upstream that it is approaching capacity. Queue depth serves as a natural signal. When it exceeds a threshold, the system can scale consumers, throttle producers, or shed low-priority work. Without back-pressure, queues grow without bound during sustained overload. This connects to the broader discussion of failure modes and retry behaviour: a system that accepts work faster than it can process it will eventually fail.

Priority queues allow latency-sensitive work to be processed ahead of background tasks. A payment message should not wait behind analytics events. Separate queues with dedicated consumers provide stronger isolation than a single queue with priority levels.

Dead letter queues capture messages that fail processing repeatedly. Without them, poison messages (malformed, referencing deleted resources, triggering bugs) block the queue and prevent subsequent messages from being processed. Dead letter queues isolate these failures and allow the main processing path to continue.

Connection management

Serverless functions have an awkward relationship with persistent connections. Traditional connection pooling assumes long-lived application processes that maintain and reuse a pool of database or service connections. Serverless functions are short-lived (sometimes processing a single request before being frozen or recycled), which makes naive connection pooling problematic.

Connection exhaustion is the most common symptom. A traffic spike creating hundreds of new Lambda instances, each opening its own database connection, can overwhelm a database that supports a few hundred concurrent connections.

Connection proxies (AWS RDS Proxy, PgBouncer, ProxySQL) sit between functions and the database, multiplexing many short-lived callers onto a shared connection pool. This should be considered mandatory for any serverless workload that connects to a relational database.

Keep-alive for HTTP connections within a function's lifetime avoids repeated TLS handshakes. Most HTTP client libraries support keep-alive by default, but in some serverless runtimes the default configuration is overly aggressive about closing idle connections. Ensuring that HTTP clients reuse connections across invocations within the same execution environment eliminates a significant source of per-request overhead.

Measuring and budgeting latency

Reducing latency without measuring it is guesswork. Effective latency management requires both measurement infrastructure and a budgeting framework.

Latency budgets allocate a total time budget to a user-facing request and distribute it across services in the call chain. If the budget is 500 milliseconds across five services, each gets roughly 100 milliseconds, though in practice the distribution is unequal. The important discipline is that every team knows their allocation and is accountable for staying within it.

Percentile measurement matters more than averages. A service with a 20-millisecond average but a 2-second p99 is not a fast service. It is a service that is fast most of the time and catastrophically slow for 1% of requests. Tail latency is where user experience degrades and where TTFB and origin latency problems surface most clearly. Measuring p50, p90, p99, and p99.9 separately gives a complete picture.

Distributed tracing is the primary tool for understanding where time goes within a request. Each span in a trace captures the duration of a specific operation: a function invocation, a database query, a cache lookup, a queue wait. Aggregating span durations across many traces reveals which services and operations dominate the latency distribution. This requires the end-to-end observability discussed elsewhere, particularly in multi-cloud environments where traces must propagate across provider boundaries.

Continuous regression detection automates the process of noticing when latency increases. A deployment that adds 50 milliseconds to p99 latency may not trigger an alert if the absolute value is still within SLO bounds, but over many deployments these incremental regressions accumulate. The performance regression checklist provides a structured approach to catching and investigating these changes before they compound into a user-visible problem.

Serverless and microservice architectures distribute latency across many components, making it harder to see and easier to accumulate. The strategies here are not individually novel. Their value lies in applying them systematically, with clear ownership, measurable budgets, and the observability to verify that they work.

Related notes

Back to topic • Back to Notes index