Multi-Region Deployments for Serverless Workloads on AWS

DevOps
1

Multi-Region Deployments for Serverless Workloads on AWS

An in-depth guide to designing, implementing, and operating a safe and scalable multi-region deployment system for serverless infrastructure.

Background and Motivation

As a platform matures, deploying infrastructure to a single AWS region is often no longer sufficient. Common requirements that drive multi-region adoption include:

  • Global availability across regions
  • Faster and repeatable regional rollouts
  • Identical infrastructure behavior in every environment
  • Safe updates for already-provisioned tenants

Manual or sequential region-by-region deployments quickly become slow, error-prone, and hard to scale. A single deployment mechanism that can reliably deploy the same infrastructure across multiple AWS regions—without introducing operational risk—is the natural solution.

This guide explains how to implement multi-region deployment, the design decisions involved, and how deployments execute end to end.

Core Design Principles

Before implementation, align on a few non-negotiable principles.

1. Consistency Across Regions

All regions must receive infrastructure built from the same source, using the same deployment logic, with no manual intervention.

2. Regional Isolation

Each region should be self-sufficient:

  • Region-specific S3 buckets
  • Region-specific Lambda artifacts
  • No cross-region runtime or deployment dependencies

3. Safety by Default

Especially in production:

  • Fail fast on errors (set -e)
  • Avoid partial deployments
  • Preserve existing tenant configuration

4. Scalable Execution

The solution must handle:

  • Dozens of services
  • Multiple regions
  • Hundreds of tenants
    …without hitting AWS API throttling or system limits.

High-Level Deployment Architecture

At a high level, the deployment system performs the following steps:

  1. Validate deployment inputs
  2. Discover and filter deployable services
  3. Package services per region
  4. Upload artifacts to regional S3 buckets
  5. Update infrastructure definitions in AWS Service Catalog
  6. Optionally roll out changes to existing tenants

Each step is automated and executed in a controlled, predictable manner.

Challenges to Anticipate

Serverless Framework Hardcodes the Region into CloudFormation Templates

This is the most significant packaging challenge when targeting multiple regions.

When you run sls package --region <region>, the Serverless Framework bakes the target region directly into the generated CloudFormation template. Resource ARNs, S3 references, and environment-specific values all resolve to that specific region at package time—not at deploy time.

This means a template packaged for us-east-1 cannot simply be reused for eu-west-1. The region is not a parameter passed into the stack; it is a hardcoded literal embedded throughout the template.

What does not work:

Packaging once and reusing the same CloudFormation template for all regions. Every region-specific reference (ARNs, S3 paths, environment variables resolved at build time) will still point to the original packaging region.

What to do instead:

For every service and for every target region, run a separate packaging pass:

sls package --stage "$STAGE" --region "$region"

Each pass produces a region-correct CloudFormation template and Lambda ZIPs. These are then uploaded to the region-specific S3 bucket for that target region.

This is why the deployment loop iterates over both services and regions, rather than packaging once and distributing:

for each batch of services:
  for each target region:
    clean artifacts
    sls package --region <region>      ← region-specific template generated
    upload template + ZIPs → regional S3 bucket

The cost is packaging time multiplied by the number of regions. The benefit is that every region receives a template with accurate, region-native values—no substitution, no post-processing hacks.

Invocation Interface

The deployment script exposes a clean CLI interface:

./deploy.sh --stage <stage> [--regions <region1,region2,...>] [--update-product]
  • --stage (required) — deployment environment (dev, qa, prod)
  • --regions (optional) — comma-separated list of AWS regions (defaults to us-east-1)
  • --update-product (optional) — when present, triggers tenant-level rollout after infrastructure definitions are updated

This makes it straightforward to run region-specific or full multi-region deployments from CI/CD pipelines.

Deployment Flow

Below is the full deployment flow with the real execution order and built-in safeguards.

Start
  │
  ├─► Parse CLI Arguments (--stage, --regions, --update-product)
  │
  ├─► Validate Required Parameters
  │     └── Exit early if --stage is missing
  │
  ├─► Set Primary Region + Deployment Artifact Bucket
  │
  ├─► PHASE 1: Service Processing
  │     ├─► Discover all services, apply exclusion list
  │     ├─► Process in batches of 8 (per region, in parallel)
  │     │     ├─► Clean previous artifacts
  │     │     ├─► Package via Serverless Framework
  │     │     └─► Upload CloudFormation templates + Lambda ZIPs to regional S3
  │     └─► Process Lambda Authorizer (sequentially, per region)
  │
  ├─► PHASE 2: Init Package Deployment
  │     ├─► Package init stacks via serverless-compose
  │     └─► Sync nested CloudFormation stack files to regional S3 buckets
  │
  ├─► PHASE 3: Service Catalog Update
  │     ├─► Compare new template with current active version
  │     ├─► Create new version only if changes are detected
  │     └─► Store new artifact ID in SSM Parameter Store
  │
  └─► PHASE 4: Update Provisioned Products (opt-in via --update-product)
        ├─► Query all provisioned products
        ├─► Update in parallel batches (up to 8)
        └─► Preserve existing parameters via UsePreviousValue=true

The deployment is intentionally structured into clear phases to ensure that infrastructure definitions are always updated before any tenant-level changes occur.

Flow Walkthrough

Start → Parse Command-Line Arguments

The deployment begins by parsing command-line arguments, including:

  • --stage (required)
  • Optional region list
  • Optional rollout flags (e.g. updating existing tenants)

This establishes the deployment context up front.

Validate Required Parameters

The script immediately validates mandatory parameters, most importantly the deployment stage (dev, qa, prod).
If validation fails, the deployment exits early, preventing accidental or incomplete execution.

Set Primary Region and Deployment Bucket

A primary region is resolved along with the deployment artifact bucket associated with the target stage.
This bucket becomes the source of truth for CloudFormation templates and nested stack artifacts.

Phase 1: Service Processing

This phase handles application-level services and is designed to balance speed with safety.

Service Discovery and Filtering

  • All services are discovered from a shared services directory
  • Certain services are always excluded (legacy or internal-only)
  • Some services are conditionally excluded based on environment (e.g. test-only services in production)

This keeps deployments intentional and avoids unnecessary infrastructure changes.

Batched Service Execution

Services are processed in batches of 8.

Why batching matters:

  • Prevents AWS API throttling
  • Controls system resource usage
  • Provides predictable deployment progress

Each batch completes across all regions before the next batch begins.

Per-Region Packaging and Upload

For every service in the batch and for every target region:

  1. Previous build artifacts are removed (.serverless/ cleaned)
  2. The service is packaged using the Serverless Framework (sls package --stage --region)
  3. CloudFormation templates and Lambda ZIPs are generated
  4. The correct S3 key path is extracted directly from the CloudFormation template using jq, ensuring ZIPs are placed in exactly the location the template expects
  5. All artifacts are uploaded to region-specific S3 buckets

This guarantees clean, deterministic, and region-isolated deployments.

Lambda Authorizer (Sequential Processing)

The Lambda authorizer is processed sequentially rather than in parallel.
This avoids conflicts during packaging and ensures stability for shared authorization components.

Phase 2: Package Deployment

This phase handles foundational infrastructure that must exist before application-level services can be provisioned or updated.

  • serverless-compose is used to package one or more initialization stacks in a single command
  • Nested CloudFormation stack files are synced in parallel to all regional deployment buckets

These templates define the baseline infrastructure layer—shared resources, networking, or IAM foundations—that other stacks depend on.

Phase 3: Service Catalog Update

AWS Service Catalog acts as the control plane for infrastructure definitions.

Version Comparison

For each Service Catalog product:

  • The newly generated template is downloaded and diff’d against the currently active version in S3
  • A new provisioning artifact is created only if changes are detected

This avoids unnecessary version churn and keeps version history meaningful.

Artifact Version Management

When a new version is created:

  • Version numbers are auto-incremented (e.g. V12 → V13) by parsing the existing artifact name
  • The new template is uploaded to the deployment bucket
  • The latest artifact ID is stored in AWS SSM Parameter Store

This allows future deployments and tenant provisioning to reliably reference the correct version without hardcoding IDs.

Phase 4: Update Provisioned Products (For Optional Deployment)

Tenant-level updates are explicit and opt-in, triggered only when --update-product is passed.

When the update flag is enabled:

  • All provisioned products for each Service Catalog product are queried
  • Updates are executed in parallel batches (up to 8, capped to avoid API throttling)
  • The latest record ID and associated CloudFormation stack are resolved per product
  • Existing CloudFormation parameters are preserved using UsePreviousValue=true

This ensures:

  • No configuration drift
  • No accidental behavior changes
  • Predictable and safe rollouts

Failures in individual tenant updates are logged and do not block other tenants from updating.

Deployment Completion

The deployment completes once:

  • Infrastructure definitions are fully updated in Service Catalog
  • Optional tenant updates finish successfully (or are intentionally skipped)

This clear separation between definition updates and tenant rollouts is a key safety boundary in the design.

Production Deployment Strategy

In production, deployments should follow a phased approach:

Phase 1 – Catalog Update Only

Update Service Catalog definitions without impacting existing tenants.

Phase 2 – Validation

Provision test tenants and validate functionality across regions.

Phase 3 – Tenant Rollout

Explicitly trigger updates for existing tenants once confidence is established using --update-product.

Operational Optimizations

Several low-level optimizations are critical at scale:

Optimization Detail
AWS CLI retry mode adaptive with max 3 attempts
Interactive paging Disabled via AWS_PAGER=""
File descriptor limit Raised to 4096 via ulimit -n
Concurrency cap Max 8 parallel jobs to avoid throttling
Artifact cleanup .serverless/ removed before each package run

These details significantly improve deployment stability and predictability at scale.

Key Outcomes

  • Consistent, repeatable multi-region deployments from a single script invocation
  • Faster regional rollouts through controlled parallel execution
  • Reduced operational risk through phased, opt-in tenant updates
  • Safer production updates with full parameter preservation

Key Takeaways

  • Multi-region deployment is an architecture problem, not just automation
  • Region isolation simplifies reliability and troubleshooting
  • AWS Service Catalog provides strong guardrails for infrastructure evolution
  • Separating infrastructure definition updates from tenant rollouts is essential for safe production operations
  • Small details—retry modes, file descriptor limits, artifact cleanup—matter significantly at scale
1 Like