HelpWithWebGet Help Now
← Back to Blog
Debugging5 min read

next build Deleted My Production Site. Twice.

A Next.js standalone deployment in Docker has a trap: next build clears its output before compiling, and Docker recreates the bind-mount points as root. A failed build then can't be retried. Two outages, the root cause, and the deploy script that fixes it.

ByDino Bartolome
Code editor with colorful syntax highlighting on dark background
Photo by Florian Olivo on Unsplash

Here is a failure mode worth knowing about if you run Next.js in standalone mode inside Docker with bind mounts, because it is silent, it is repeatable, and the first time it bites you it will be while you are doing something routine.

next build deletes your previous build before it makes the new one. If the build then fails, you do not have a new site and you no longer have the old one either.

That is survivable on its own. What makes it a trap is a second detail that has nothing to do with Next.js, and the two together took a live site down twice in one afternoon.

The setup

A fairly ordinary Next.js standalone deployment. The compiled output lives on the host and is bind-mounted into the container:

volumes:
  - ./.next/standalone:/app/.next/standalone
  - ./.next/static:/app/.next/standalone/.next/static
  - ./public:/app/.next/standalone/public

Note that the second and third mounts are nested inside the first. That is the whole problem, and it looks completely reasonable until it isn't.

What actually happens

  1. You run npm run build.
  2. Next deletes .next/standalone to make way for the new output.
  3. The build hits an error — any error. Bad code, a lint failure, a lock held by another build.
  4. It exits. There is now no .next/standalone/server.js.
  5. The container restarts, finds no server, and crash-loops.

So far this is just "a failed build broke my deploy", which is bad but obvious. Here is the part that turns one bad afternoon into two:

When Docker starts a container and a bind-mount source directory doesn't exist, the daemon creates it — as root.

Because .next/static and public are mounted inside .next/standalone, and because the build just deleted .next/standalone, the daemon helpfully recreates that whole path on the next container start. Owned by root. Your user cannot delete it.

Now run the build again to fix things:

Error: EACCES: permission denied, rmdir '/app/.next/standalone/.next/static'

next build tries to clear the output directory, hits a root-owned subdirectory it cannot remove, and dies — after having already deleted everything it could delete. You are now further from a working site than when you started, and every subsequent attempt fails the same way.

That is the trap. A failed build creates the exact condition that makes the next build fail.

Fix number one, which is not enough

The obvious response is to stop the container writing root-owned files, so run it as your own user:

user: "1001:1001"
environment:
  - HOME=/tmp

HOME matters more than it looks. Inside the image, uid 1001 has no home directory, so anything in your startup command that writes to ~ fails. In our case a git config --global call at startup failed, and because it was chained with &&, the container never reached the line that actually starts the server. We swapped one failure mode for another for about ten minutes.

This change is worth making — it stops the container creating root-owned files during normal operation. But it does not fix the trap, because the directories that break the build are created by the Docker daemon, which runs as root no matter what user: says.

We learned that by making the change, feeling pleased, and watching the next build fail identically.

Fix number two, which is the actual fix

Stop trying to make the build safe to run against a live container, and make the deploy safe to fail instead. Three properties, in order of importance:

1. Never delete the only working copy. Snapshot the current build before touching it, and restore it automatically if the build fails. A hardlink copy costs almost nothing in time or disk, and it survives the rm -rf that next build performs, because the inodes stay alive through the backup:

cp -al .next/standalone .next/standalone.bak

2. Fail before you touch anything. Typecheck while the site is still serving. If the code is broken, you find out at a point where nothing has been deleted and the container hasn't been stopped.

3. Only one deploy at a time. Our third near-miss was two processes deploying at once: one had stopped the container, the other held Next's internal build lock. Neither brought the site back. A flock fixes that in two lines, and waiting out any build started elsewhere fixes the rest.

Here is the script, trimmed to the parts that matter:

set -euo pipefail
cd "$(dirname "$0")"

# one deploy at a time
exec 9>.deploy.lock
flock -n 9 || { echo "another deploy is running"; exit 1; }

# wait out a build started elsewhere
while pgrep -f "$PWD/node_modules/.bin/next build" >/dev/null; do sleep 5; done

# fail before touching the container
npx tsc --noEmit -p tsconfig.json

docker compose stop
docker run --rm -u root -v "$PWD/.next:/x" node:20-slim chown -R 1001:1001 /x

# snapshot: hardlinks, near-instant, survives the build's rm -rf
rm -rf .next/standalone.bak
cp -al .next/standalone .next/standalone.bak

docker run --rm -u root -v "$PWD/.next:/x" node:20-slim rm -rf /x/standalone

if ! npm run build || [ ! -f .next/standalone/server.js ]; then
  echo "build failed — restoring previous build"
  rm -rf .next/standalone
  mv .next/standalone.bak .next/standalone
  docker compose up -d
  exit 1
fi

rm -rf .next/standalone.bak
docker compose up -d

Note the last check. npm run build can print ✓ Compiled successfully and still fail afterwards during the copy step — we watched it happen. Exit code alone is not enough; verify the artefact you actually need exists.

Two mistakes worth admitting

Suppressing the output of a safety step. The second outage happened because the cleanup command was written as:

docker run --rm -u root ... rm -rf /x/standalone >/dev/null 2>&1

It failed. Silently. The build then ran against a directory that hadn't been cleared, deleted the old output, and died. Redirecting the noise from a step whose entire job is to prevent a failure is how you delete the step without noticing.

Knowing about a trap is not the same as avoiding it. We had already documented this exact failure — in writing, in a comment, at the top of the deploy script — and then ran a bare npm run build anyway because it was a small change and we were in a hurry. Documentation does not protect you. A script that refuses to do the dangerous thing protects you.

Test the failure path, not the happy path

The obvious way to test a deploy script is to run it and see the site come up. That tells you almost nothing, because the happy path was never the problem.

Test it by making the build fail on purpose. We forced it by copying the script with the build command replaced by false, then watched: container stopped, snapshot taken, build "failed", previous build restored, site back at HTTP 200 in six seconds. That is the test that matters.

Deliberately broken source turned out to be a poor way to force it, incidentally — Next tolerated a throw at module scope and a generateStaticParams that threw, and built successfully both times. Substituting the build command is more reliable.

The general lesson

A deployment whose failure mode is "the site is down" is not a deployment process, it is a gamble that runs a little worse each time you take it.

The property you want is that a failed deploy leaves you exactly where you started: same site serving, same version, nothing lost, and an error message telling you what to fix. That is achievable in about forty lines of bash, and it is worth writing before the outage rather than after it.

We wrote ours after.

Need Help With Your Website?

I fix these problems every day. Send me a message and I'll take a look.

Get Help Now
CallTextMessage