Cloudflare Pages already keeps your preview deploys out of Google

I wanted to review every change to this site on a real URL before it went public. Not localhost. The actual rendered page, deployed, so I could catch the layout problems a dev server hides.

Cloudflare Pages gives you that: push a branch, get a preview deployment at its own URL. But those preview URLs are public, and my first thought was the obvious one. I don’t want half-finished drafts turning up in Google. So I reached for an Astro conditional to stamp noindex on preview builds.

Then I checked whether I actually needed it. I didn’t. Cloudflare already does it, one layer below my application, and the reason my clever conditional was pointless is the interesting part of this post.

The setup: two systems, one job each

• Cloudflare Pages deploys. Production from main, a preview for the other branches it is configured to build. It deploys every branch that builds, good change or bad, because a preview is for looking at. It runs no checks beyond the build command.
• GitHub Actions is the quality gate. It runs format, type-check, and build on every pull request, and deploys nothing. It is what blocks a bad merge, once its job is a required status check in a branch protection rule or ruleset.

Connect the repo, set the build command (pnpm build) and output directory (dist), and choose the production branch (main here, but that is a project setting, not a rule). By default Cloudflare deploys your non-production branches as previews, and a pull request opened from the repository gets a preview URL posted on it. You can scope this to all branches, none, or a chosen set. Each preview gets an immutable hash URL and a per-branch alias.

That is the whole deploy configuration. Now the part I got wrong.

The noindex I didn’t need to write

Here is the code I was about to add to my base layout:

---
const isPreview =
  process.env.CF_PAGES === '1' && process.env.CF_PAGES_BRANCH !== 'main';
---

<head>
  {isPreview && <meta name="robots" content="noindex" />}
</head>

It is not wrong. Cloudflare injects CF_PAGES=1, CF_PAGES_BRANCH, CF_PAGES_URL, and CF_PAGES_COMMIT_SHA into every Pages build, so at build time a static Astro site can tell it is a preview and emit the tag. The mechanism is real.

It is just unnecessary, because Cloudflare already handles this a layer down. From their docs:

By default, every preview deployment generated by Cloudflare Pages includes the X-Robots-Tag: noindex HTTP response header.

You can watch it happen. Curl a preview URL and the production URL and compare headers:

$ curl -sI https://<hash>.my-project.pages.dev | grep -i robots
x-robots-tag: noindex

$ curl -sI https://my-project.pages.dev | grep -i robots
# (nothing)

Preview deployments carry noindex. Production does not. I had not written a line of code, and the exact behavior I wanted was already there.

Why the header is the right place and my meta tag was the wrong one

There are two ways to tell a crawler to skip a page:

<meta name="robots" content="noindex" />

X-Robots-Tag: noindex

They mean the same thing to Google, but they live at different layers. The meta tag exists only inside HTML that you generated. The HTTP header applies to the response itself, every response, whatever your app rendered and whether or not it is even HTML.

That difference is the whole point. Cloudflare knows a deployment is a preview because it created the preview. It does not need my application to re-derive that fact from an environment variable and act on it. The system that owns the fact applies the correct HTTP semantics, and my app never has to care.

That is the lesson worth keeping: do not solve an infrastructure problem inside your application. My isPreview check was pulling a fact that belongs to the deploy platform up into my page templates. It would only have covered HTML rendered through that layout, not images, PDFs, or any page that skips it. Cloudflare already answered the question, once, at the edge, for every response.

A static _headers file can also set X-Robots-Tag by hostname, using placeholders: https://:project.pages.dev/* for the production pages.dev address and https://:version.:project.pages.dev/* for previews, if you ever need indexing rules the defaults do not cover. For keeping previews out of search, you do not need it.

The layers, in one picture

                    git push / PR
                          │
             ┌────────────┴────────────┐
             ▼                         ▼
       GitHub Actions            Cloudflare Pages
       (quality gate)               (deploy)
             │                    ┌────┴────┐
   format / check / build         ▼         ▼
             │                  main      branch
             ▼                    │         │
        pass → merge              ▼         ▼
                              production  preview
                                  │         │
                              indexable   X-Robots-Tag:
                                            noindex

Cloudflare deploys and decides indexing. GitHub Actions decides whether a commit is allowed to become production. Neither job belongs in my page templates.

The one thing Cloudflare doesn’t do for you

Cloudflare noindexes previews, but your production *.pages.dev stays indexable. If you also serve the site on a custom domain, my-project.pages.dev and mydomain.com are now the same content at two addresses, which is a duplicate-content problem. The fix is not a preview conditional. It is a canonical URL on every page pointing at your real domain (this site sets one in its base layout), which search engines treat as a strong hint, or, to settle it, a Bulk Redirect from the pages.dev hostname to your domain. The _redirects file cannot match on hostname, so it cannot do this one.

And noindex is not access control. Anyone with a preview URL can still open it. If a draft must stay private, put the previews behind Cloudflare Access.

The quality gate

The deploy side does none of the checking, so a separate workflow runs the same commands I run locally, and a red check blocks the merge once the job is marked as a required status check.

name: CI
on:
  push:
    branches: [main]
  pull_request:

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - run: pnpm check-format # prettier --check .
      - run: pnpm check # astro check
      - run: pnpm build

pnpm/action-setup reads the pnpm version from the packageManager field in package.json; without that field, give it a version. --frozen-lockfile fails the install if the lockfile is out of date, so CI builds the exact tree I committed. The three checks are the same scripts in package.json, so pnpm check-format && pnpm check && pnpm build locally means green in CI. cancel-in-progress kills a superseded run, so a burst of commits only spends minutes on the last one.

What I actually learned

I came in ready to write application code to keep previews out of Google, and the useful outcome was not writing it. Cloudflare deploys previews and marks them noindex. GitHub Actions gates the merge. The site sets a canonical URL. Three layers, each owning one job, and none of them my page templates.

The best infrastructure code is often the code you were about to write and then didn’t have to.