Migrating from ECS Fargate and Aurora PostgreSQL to KamuiDash

Migrating a system that is already in production is a different job from deploying a new one.
You have to avoid losing data, avoid taking the service down while people are using it, and keep a way back if something goes wrong. Everything else follows from those three constraints.

The starting point

The system has a Next.js frontend, a Flask backend, and a PostgreSQL database. On AWS it ran like this.

ECS Fargate — Frontend and backend as separate services, exposed through an ALB.
Aurora PostgreSQL 14 — In a private subnet. Not reachable from the internet.
Route 53 — The apex domain served the frontend, an api subdomain served the backend. Both were ALIAS records pointing at the ALB.
ACM — A certificate including a wildcard, attached to the ALB, validated via DNS.

Throughout this article the real domain names are replaced with example.com and api.example.com.

Before and after

Before, a single ALB received both the apex and the api hostname and routed to ECS Fargate. After, each app has its own custom domain and the KamuiDash edge routes by hostname. The database moved to Postgres inside the same project.

Split the work into three phases

Doing the whole migration in one go leaves you with nothing to isolate when it fails. We split it so that exactly one step is visible to users.

Phase 1  Move the database        no impact (it is only a copy)
Phase 2  Move DNS hosting         no impact (records stay the same)
Phase 3  Switch the domain        the only step users can notice

Phases 1 and 2 finish without touching the running service. Once everything is prepared, Phase 3 flips it. Rolling back means undoing Phase 3 and nothing else.

Order of work and the point of impact

Phases 1 through 3-a never touch the running service, so they can take as long as they need. Only Phase 3-b is visible to users, and if it goes wrong you recover by pointing the record back. Data freshness carries the condition described in the next section.

What "no maintenance window" actually requires

There is a premise here that needs stating plainly. pg_dump captures a point in time. Anything written to the old database after that point does not reach the new one.

The service stays up, so the migration is non-disruptive. That alone does not make it lossless. To get both, no writes may occur against the old environment between taking the dump and completing the cutover.

There are three ways to satisfy that.

A. Fit inside a window with no writes — Keep the whole span from dump to cutover inside a period you have measured to be write-free. Nothing has to be stopped, but this does not work for a system that is written to continuously.
B. Stop writes and do a final sync — Put the app in read-only mode or stop it just before the cutover, then dump and load again in that state. You accept a few minutes without writes, and you get a guaranteed match.
C. Keep syncing with logical replication — Stream changes until the cutover. The write pause is close to zero, but this is by far the most work to set up and verify.

We chose A, having first confirmed that the system in question is an internal tool used only during business hours. As described later, we pulled four weeks of traffic data, identified a window with no writes, and fit the dump and the cutover inside it.

If you take route A, verify afterwards that no writes reached the old environment. Load balancer request counts and write-path entries in the application log will tell you. If something did get through, you either move those rows by hand or start over with route B.

For a service written to around the clock, plan on B or C instead. Everything else in this article applies to all three routes. The only thing that changes is when you take the dump.

Dumping a database you cannot reach

Aurora sat in a private subnet with PubliclyAccessible set to false. Its security group allowed port 5432 only from specific security groups. You cannot run pg_dump against it from your laptop.

Session Manager remote host port forwarding solves this. If a bastion with the SSM agent exists inside the VPC, you do not need a PostgreSQL client on that bastion at all. You borrow the tunnel and write the dump to your own machine.

aws ssm start-session \
  --target i-0123456789abcdef0 \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["mycluster.cluster-ro-xxxxxxxx.ap-northeast-1.rds.amazonaws.com"],
                 "portNumber":["5432"],
                 "localPortNumber":["15432"]}'

Note the reader endpoint (cluster-ro-). A dump only reads, but making the path itself incapable of writing removes one way to make a mistake.

You may be able to attach an existing application security group to the bastion. If the database already allows that group, you can complete the migration without modifying any production security group.

Three traps in the dump step

The tunnel worked, and then three separate things went wrong. All of them are common in migrations.

1. The master password had rotated

With an RDS-managed master password, the value lives in Secrets Manager and rotates automatically. In our case the interval was seven days.

The application, meanwhile, read a copy stored in Parameter Store. That copy does not update itself. A password that worked a few days earlier now fails with password authentication failed.

SECRET_ARN=$(aws rds describe-db-clusters \
  --db-cluster-identifier my-cluster \
  --query "DBClusters[0].MasterUserSecret.SecretArn" --output text)

export PGPASSWORD=$(aws secretsmanager get-secret-value \
  --secret-id "$SECRET_ARN" --query SecretString --output text \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)["password"])')

There is a reason the ARN goes through a variable. RDS-managed secret names contain rds!cluster-, and in zsh the ! triggers history expansion even inside double quotes, producing event not found. Use single quotes, or put it in a variable first.

2. pg_dump and pg_restore were different versions

A custom-format archive carries a format version tied to the pg_dump that produced it. Reading a newer archive with an older pg_restore fails like this.

pg_restore: error: unsupported version (1.16) in file header

On a machine with several PostgreSQL clients installed, pg_dump and pg_restore can easily resolve to different installations. Check --version on both before you start.

3. localhost resolved to IPv6

The Session Manager plugin binds only to 127.0.0.1. On macOS, -h localhost tries ::1 first and fails, which adds noise to the output.

connection to server at "localhost" (::1), port 15432 failed: Connection refused

libpq falls back to IPv4 so it eventually connects, but the real error is buried on the following lines. Writing -h 127.0.0.1 keeps the output readable when something else breaks.

What to check after the restore

A KamuiDash database has a user for external connections and a user for the application. Restoring over the external connection makes the external user the owner of every table. The application then hits ownership errors when it runs migrations.

REASSIGN OWNED BY app_external TO app_user;

Tables and sequences move together, so nothing is left behind. The requirement is that the executing user is a member of the target role.

The other thing worth checking is the current value of every sequence. If it lags behind the data, the first INSERT fails on a duplicate primary key.

SELECT t.tablename,
       s.last_value,
       (xpath('/row/max/text()',
              query_to_xml(format('SELECT max(id) AS max FROM public.%I', t.tablename),
                           false, true, '')))[1]::text::bigint AS max_id
FROM pg_tables t
JOIN pg_sequences s ON s.sequencename = t.tablename || '_id_seq'
WHERE t.schemaname = 'public'
ORDER BY 1;

A full dump includes setval, so last_value should already exceed max_id. You can confirm this without writing a single row.

Privileges can be inspected the same way, with has_table_privilege and has_sequence_privilege. There is no need to insert test data to find out.

Move DNS without moving traffic

Route 53 ALIAS records can only point at AWS resources. When you move to another DNS provider, there is no direct equivalent. The apex domain is the harder half, because DNS does not allow a CNAME there.

A provider with CNAME flattening lets you put a CNAME at the apex anyway. We used Cloudflare for this.

The important part is that moving the zone and moving traffic are separate things. If the record values stay the same, only the answering nameserver changes. The destination does not.

example.com            CNAME  dualstack.my-alb-000000.ap-northeast-1.elb.amazonaws.com
api.example.com        CNAME  dualstack.my-alb-000000.ap-northeast-1.elb.amazonaws.com
_xxxxxxxx.example.com  CNAME  _yyyyyyyy.acm-validations.aws

After the nameserver change, traffic still reaches the ALB. Confirm resolution in that state before going further.

Automatic scans cannot reproduce an ALIAS

Some providers scan your existing records during onboarding. An ALIAS looks like an A record from the outside, so the scan produces A records containing the load balancer's current IP addresses.

Those addresses are not stable. During this migration one of the three changed within a day. Hard-coded IPs break quietly later. Use a CNAME to the load balancer's DNS name instead.

Turn the proxy off

On DNS providers with a built-in CDN, proxying is often enabled by default on new records. Keep it off during the migration. With it on, TLS terminates somewhere else, and depending on your origin configuration you get certificate errors or redirect loops.

Keep it off after switching to a KamuiDash custom domain as well. Stacking one proxy in front of another is not a supported arrangement.

You can verify before you switch

Before changing the nameservers at the registrar, query the new nameservers directly.

dig @new-nameserver example.com +short
dig @new-nameserver api.example.com +short

If they return the same addresses as today, switching changes nothing about the result. That check turns the nameserver change into a purely administrative step.

If DNSSEC is enabled on the zone, disable it at the registrar before changing nameservers. Switching with DNSSEC active can make the domain unreachable.

Why the backend needs a custom domain too

KamuiDash issues a URL of the form appname.kamui-platform.com the moment you create an app. It is covered by a wildcard certificate, so HTTPS works immediately.

That invites the idea of giving only the frontend a custom domain and leaving the backend on its issued URL. We considered exactly that, and it does not work here.

The reason is the session cookie. This system authenticates with cookies, and the frontend sends credentials: 'include' on every API call.

example.com  ->  api.example.com
  same registrable domain       first-party cookie   OK

example.com  ->  myapp.kamui-platform.com
  different registrable domain  third-party cookie   blocked

Whether a cookie is sent is decided by the SameSite attribute and the browser's third-party cookie policy. With no SameSite attribute set, Chrome treats the cookie as Lax and will not send it on a cross-site fetch. Safari blocks third-party cookies by default through ITP.

Allowing CORS does not fix it

This is the part that is easy to confuse. CORS and cookie transmission are different layers.

1. CORS            the server permits an origin to read the response
                   Access-Control-Allow-Origin / Allow-Credentials

2. Cookie sending  the browser decides whether to attach the cookie
                   SameSite attribute / third-party cookie policy

Access-Control-Allow-Credentials: true only clears the first gate. It does not compel the browser to send anything. Fail the second gate and the request arrives without a cookie, so the CORS headers look correct while authentication silently fails.

curl and other HTTP clients implement neither SameSite nor ITP, so they will not reproduce this. As long as CORS passes, everything looks fine. Test in a real browser, and include Safari.

So the frontend and the backend need to live under the same registrable domain. If the frontend is example.com, the backend has to be api.example.com.

TXT validation is what makes it zero-downtime

Registering a custom domain on KamuiDash gives you a CNAME and a TXT record to create. The TXT proves domain ownership and drives certificate issuance.

CNAME  api.example.com                  custom.kamui-platform.com
TXT    _acme-challenge.api.example.com  the value shown in the dashboard

This is the key design point of the whole migration. Because validation happens over TXT, the certificate can be issued independently of where the CNAME points.

1. Add only the TXT       CNAME stays as it is, no impact
2. Wait for validation    the service keeps running on the old stack
3. Switch the CNAME       everything is ready, so it responds immediately

With HTTP validation — serving a file at a given path — you would have to switch the CNAME first and accept downtime until validation completes. TXT validation is the reason no maintenance window was required.

There is also no rush. Adding the TXT one day and switching the CNAME the next is a perfectly reasonable pace.

Watch for settings baked in at build time

Next.js inlines environment variables prefixed with NEXT_PUBLIC_ into the JavaScript bundle at build time. Changing them at runtime has no effect.

If your API endpoint is passed that way, switching the domain requires a rebuild. We used this order.

1. Point api.example.com at the KamuiDash backend
2. Set the frontend API URL to https://api.example.com and redeploy
3. Point example.com at the KamuiDash frontend

Doing 2 before 1 aims the frontend at an endpoint that has not moved yet. Skipping 2 and going straight to 3 leaves the frontend calling the platform-issued URL, which lands you back in the third-party cookie problem.

You can confirm what the deployed bundle actually calls by fetching it. Read the artifact being served rather than the settings screen.

curl -s https://example.com/ \
  | grep -oE '/_next/static/[^"]+\.js' | sort -u \
  | while read -r p; do curl -s "https://example.com$p"; done \
  | grep -oE 'https://[a-zA-Z0-9._-]+\.example\.com' | sort -u

Choose the cutover window from data

"Nobody uses it on weekends" is an assumption worth verifying before you rely on it. We pulled four weeks of ALB request counts.

Sundays were consistently quiet across all four weeks. One Saturday, however, carried weekday-level traffic, and the hourly breakdown made it obvious that people were working. Do not stop at day-of-week averages; look at the hourly shape.

If you have access logs, they are the strongest evidence available. Get to a point where you can say "this window had zero requests for four weeks" instead of "I think it is quiet."

During the migration, one URL can mean two different things

This is what cost us the most time.

DNS does not switch instantly. Nameserver TTLs default to 172800 seconds — two days — and the moment of change differs per resolver. Throughout that window, the same URL can resolve to either the old or the new environment.

Worse, records propagate at different times from each other. We ended up in this state.

example.com      → old environment (not propagated yet)
api.example.com  → new environment (propagated first)

The old frontend was built to call api.example.com. So in that combination:

the page is served by the old environment
the API is answered by the new environment
both share a registrable domain, so cookies work normally

and everything looks perfectly fine in the browser. You can log in, Safari included, and writes land in the new database. It looks exactly like a successful migration.

The problem is that none of that is evidence. The new frontend has not been exercised at all. It only becomes visible once propagation finishes, and that is when the defects show up.

In our case the new frontend still had the old API endpoint baked in. Because the old frontend kept serving a correct bundle, that fact stayed hidden for two full days. It surfaced when propagation completed and logins stopped working.

Verify the origin, not the URL

"I opened the URL and it worked" is not verification. You have to establish which environment answered.

# Does the address belong to the old stack or the new one?
dig +short example.com

# Identify the origin from the response headers
curl -sI https://example.com/ | grep -iE "server|via|x-powered-by"

# Address the new environment by name, via the platform-issued URL
curl -s https://myapp.example-platform.com/ | head

The most reliable approach is to reach the new environment without DNS at all. Use the platform-issued default URL if there is one, or port-forward straight to the container. That removes DNS from the path entirely.

Also confirm in the registry that a build actually happened. The environment variable shown in a settings screen is the configured value, not the value inside the artifact being served. In our case no image had been produced since the variable was changed.

aws ecr describe-images --repository-name myapp \
  --query "sort_by(imageDetails,&imagePushedAt)[-1].[imagePushedAt,to_string(imageTags)]" \
  --output text

If no image is newer than the moment you changed the variable, the change is not in the artifact. Whether a redeploy button restarts or rebuilds varies by platform. Judging by whether a new image appeared is the reliable test.

Verifying after the switch

Once the records are changed, confirm from the outside that things moved as intended. All of these are read-only.

Query several resolvers — Your local resolver caches nameservers. Asking a few public resolvers shows how far propagation has reached.
Read the bundle — Confirm which API the frontend actually calls, from the files being served.
Watch the old stack drain — If load balancer request counts fall to near zero, traffic has moved.
Check the CORS headers — Confirm the new origin is allowed. Even an authenticated endpoint tells you this from the headers on its 401.

What to keep for rollback

Do not delete the old stack as soon as the migration completes. With a short TTL, pointing the record back recovers within minutes. How long you can keep that option is how safe the migration is.

The record most likely to be discarded by mistake is the ACM DNS validation record.

_xxxxxxxx.example.com  CNAME  _yyyyyyyy.acm-validations.aws

ACM managed renewal reads this record again before the certificate expires. Delete it because it looks unused after the migration, and renewal fails and the old certificate expires. If the old stack is your rollback target, this record is part of it.

The rule is simple. Clean it up when you delete the old load balancer, and not before.

Summary

Three things carried this migration.

Reduce the risky step to one — The database copy and the DNS delegation happened first, leaving the CNAME switch as the only user-visible change.
Measure the write-free window — A dump is a snapshot, so later writes do not follow it. Decide up front whether you fit inside a quiet window or stop writes just before the switch.
Use TXT validation — Preparing the certificate separately from switching traffic is what removes the maintenance window.
Stay able to roll back — Keep the old stack, and keep the records its certificate renewal depends on.

Most of a migration is investigation. Where the data lives, what points at what, and which settings freeze at which moment. Once you know that, the execution itself is short.

For the thinking behind KamuiDash and the basics of deploying from GitHub, see Why Use KamuiDash.