Node.js and Next.js apps — the app-hook forwarder

What you get

The app-hook forwarder observes requests that never run JavaScript — AI assistants and crawlers fetching your pages — from inside your own application, where you can see exactly what your app serves.

It is an observer, never a gatekeeper:

It records request metadata only — method, path (including the query string), the raw client-address header, and the user-agent. It runs before your response exists, so it never sees or touches response bodies, and it never reads request bodies or cookies.
It never blocks, redirects, or mutates a request or response. Your middleware calls it and then returns whatever your app was going to return anyway.
Everything it captures is buffered in memory and shipped to your ingest endpoint in batches, entirely off the request path.

Query strings are shipped untouched and sanitised centrally on our side, with per-parameter accounting — keeping the raw evidence intact is what lets bot verification stay honest, and central sanitisation means a fix to redaction never requires you to redeploy.

This page describes the app-hook component. Choose another component only when your stack needs an unusual setup.

Prerequisites

A Node.js runtime. The forwarder is written against only the Node.js standard library and the global fetch that Node 22 ships built in — no native modules, no runtime dependencies. It must run on the Node.js runtime, not an Edge runtime — the durable queue writes files.
Writable local storage for the queue directory, ideally persistent across restarts. On an ephemeral filesystem the forwarder still works and your site is still safe, but events buffered during an outage would not survive a container replacement.
Your onboarding credentials:
- your ingest endpoint (shown here as https://ingest.example-analytics.com — use the one from your onboarding),
- your site’s public key (pk_…),
- your forwarder secret key (sk_…). Keep it in your environment or secret manager only — it is never written to disk or logged by the forwarder, and it should never appear in code or version control.
npm (or your usual package manager). The forwarder is an npm package with zero runtime dependencies, and it ships compiled JavaScript — there is no build step on your side and nothing is fetched at runtime.

How you receive the software

The forwarder’s distribution channel is the public npm registry, under the name @aurascope-analytics/forwarder-app-hook — so you install it the way you install anything else:

npm install @aurascope-analytics/forwarder-app-hook

That gives you the two things a handover cannot: a lockfile entry your build and your dependency-audit tooling already understand, and npm update as the upgrade path when we ship a new version. Published versions are immutable — the bytes behind a version you have locked can never change under you.

We will tell you the exact version to install as part of onboarding, and the package name above is exact — check both against what we give you, so you cannot end up on a similarly-named package that is not ours.

One honest note on what that does and does not prove. npm guarantees a version’s integrity, and this package is published only from our automated build, with no long-lived credential involved; it does not carry a cryptographic provenance attestation linking the tarball to that build, because our source repository is private and npm does not generate provenance for private repositories.

Install steps

1. Install the forwarder

From your project root:

npm install @aurascope-analytics/forwarder-app-hook

That is the whole step. The package ships compiled JavaScript with type declarations beside it, so there is nothing to build and nothing to vendor — your app imports it by name like any other dependency.

Our automated checks run this exact sequence on every change to the forwarder: they pack the package, install the tarball into a throwaway project, and import it by that bare specifier on plain Node with no flags — so the two lines above are executed on our side before you run them, not merely described.

2. Configure the environment

Set these variables in your app server’s runtime environment:

# Required
AURASCOPE_INGEST_URL=https://ingest.example-analytics.com   # your ingest endpoint from onboarding
AURASCOPE_SITE_KEY=pk_your_site_key
AURASCOPE_FORWARDER_SK=sk_xxx...redacted                    # your forwarder secret; env/secret-manager only

Optional tuning (safe defaults shown):

Variable	Default	What it does
`AURASCOPE_QUEUE_DIR`	`.aurascope/app-hook-queue` (under the process working directory)	Where durable batches are written while waiting to ship. Point it at persistent storage.
`AURASCOPE_WORST_CASE_RATE_PER_SEC`	`10`	Your site’s worst-case request rate, used to size the buffer.
`AURASCOPE_TARGET_OUTAGE_SEC`	`86400`	How long an ingest outage the buffer should ride out (one day by default).
`AURASCOPE_BUFFER_MAX_EVENTS`	rate × outage window (`864000` with the defaults)	Explicit buffer-capacity override, if you prefer to set it directly.
`AURASCOPE_IP_EVIDENCE_HEADER`	`x-forwarded-for`	Which header carries the client-address chain in your infrastructure. The complete raw header value is shipped; which hop is the real visitor is resolved on our side from your configured proxy depth, because picking a hop at the edge is both spoofable and unrecoverable.
`AURASCOPE_PROBE_MARKER`	unset	Optional declaration that every event this process ships is synthetic traffic you generate yourself (a load-test target, a staging copy), so it can be separated from real traffic. It travels as batch metadata (a request header on each shipped batch) — your events, including the user-agent, are never modified. Never set it on a production instance — it would mark all your real traffic as synthetic.

Buffer capacity is worst-case rate × outage window. Size both from your own peak traffic and how much disk you are willing to devote — the default (864,000 events) is a starting point, not a measurement of your site.

3. Wire the middleware

For Next.js, in your middleware.ts at the project root:

import type { NextRequest } from "next/server";
import { NextResponse } from "next/server";
import { createAuraScopeMiddleware } from "@aurascope-analytics/forwarder-app-hook";

export const config = { runtime: "nodejs" };

const auraScope = createAuraScopeMiddleware();

export function middleware(request: NextRequest) {
  auraScope(request);
  return NextResponse.next();
}

The import is a plain package specifier, so it works from wherever your middleware.ts sits — no relative paths to keep in step with your layout. The package ships its own type declarations, so your editor and your build see the same types we do.

Two things to notice, because they are the safety design:

auraScope(request) returns nothing and is synchronous — it copies a handful of request fields into memory and returns. No file access, no network access, no await happens on your request path. Shipping runs on a background loop (every 5 seconds, or sooner once 50 events are staged).
You return the response (NextResponse.next() here). The forwarder has no way to change what your app serves.

If you already have middleware, add the auraScope(request) call inside it, and put it before any branch that returns early — the forwarder only observes requests that reach the call, so calling it first means every request is seen. It composes with your existing logic and changes nothing about your responses.

The created middleware exposes .forwarder when startup succeeds. If your app has a graceful-shutdown hook, call await auraScope.forwarder?.stop() there — it performs one final flush so a clean shutdown loses nothing. Without the hook you are still safe, just slightly less complete: batches already written to the queue directory survive and ship on the next start, but events observed since the last flush live only in memory until a flush writes them to disk, so an abrupt kill can lose up to the last few seconds of observations (flushes run every 5 seconds).

Other Node frameworks import the same specifier: createAuraScopeMiddleware() accepts any request object with method, url, and a headers.get(name) accessor, so an adapter for your framework’s request type is a few lines. What the milestone end-to-end run proved — against a live ingest service — is the delivery path that every wiring shares: observation into the durable queue, crash recovery from the queue directory, and idempotent shipping with no double-counting. The framework wiring on top is a thin synchronous wrapper around that proven core.

4. Restart your app

Configuration is read once at startup. On boot the forwarder also scans the queue directory and re-ships, in order, any batches a previous process left behind.

Verify your install

There is no self-serve dashboard for this request measurement yet, so verification uses the forwarder’s own local signals — and one of them is genuinely end-to-end: the accepted count comes from the ingest service’s response, and our ingest never acknowledges an event it has not durably stored. accepted climbing is proof your events are safely on our side.

Confirm the forwarder started. auraScope.forwarder is undefined when a required setting is missing or invalid (the factory then returns a harmless no-op rather than crashing your app — see failure modes). Log it once at startup:
```
console.log("aurascope forwarder active:", auraScope.forwarder !== undefined);
```
Send a few requests to any page of your site (a plain curl is fine — the forwarder sees every request, browser or not).
Read the metrics snapshot after ~10 seconds, from anywhere you can log:
```
console.log(auraScope.forwarder?.metrics());
```
Healthy install:
- observed — climbing with your traffic,
- accepted — catching up to observed (this is the ingest acknowledgment),
- staged and pendingLines — returning to 0 (the queue drains),
- breakerState — "closed",
- dropped — 0.
Check the queue directory drains. Batches wait in <queue dir>/pending/*.ndjson only while unshipped; an empty pending/ directory after a flush means everything was delivered and acknowledged.

If accepted stays at zero, check the forwarder’s error log lines: a wrong secret produces AuraScope app-hook authorization failed; retaining the durable queue and retrying slowly — note retaining: even a misconfigured secret loses nothing, and delivery resumes as soon as the secret is fixed.

Failure modes — and what they do to your site

The short answer is nothing, in every case. That is not a promise of good behaviour, it is how the code is shaped: the only work on your request path is a synchronous in-memory copy, every error path is swallowed, and all disk and network activity happens on a background loop. The added request latency is under a millisecond at the 99th percentile.

What goes wrong	What happens to your site	What happens to your data
Our ingest service is down or unreachable	Nothing. Requests are captured in memory and to disk as usual.	Batches are retained on disk. A circuit breaker pauses shipping and retries with capped exponential backoff (or the interval our service asks for), then resumes and ships everything. Nothing is dropped merely because the breaker is open.
Our service is overloaded (rate-limits or asks you to back off)	Nothing.	Same pause–retain–resume, honouring the server’s requested retry interval.
Wrong or revoked secret key	Nothing. One error line is logged.	The queue is retained and retried slowly (every 5 minutes) until the key is fixed.
Your process crashes or restarts	Your app restarts as it normally would; the forwarder adds no startup dependency.	Persisted batches are recovered from the queue directory in order and re-shipped. Each batch carries the same idempotency key on every attempt, so a batch delivered just before a crash is recognised and never double-counted — proven end-to-end: a byte-identical replay was answered with 0 accepted, 4 duplicates. Events observed in the few seconds since the last flush were still in memory and are the one thing a hard crash can lose (the wiring step explains the window).
A malformed event reaches ingest	Nothing.	Ingest answers success with a per-line rejected count and records the fault on our side for accounting — it never answers with a server error, so a bad line can never put your forwarder into an error loop. Proven end-to-end.
Queue directory unwritable at startup, or configuration missing/invalid	Nothing — the factory returns a no-op middleware and your app runs exactly as before.	No telemetry until fixed (verification step 1 catches this).
Disk write fails mid-run	Nothing — request-path capture does no disk I/O.	The flush logs an error; staged and pending events remain retained and the next flush retries.
Outage outlives your buffer capacity	Nothing.	Fail-open wins over completeness: the oldest data is dropped first and every dropped event is counted in the local `dropped` metric — loss is bounded, contiguous, and never silent.

One deliberate consequence of “acknowledgment means durably stored”: if our service returns an unreadable acknowledgment, the forwarder treats it as no acknowledgment and retries the batch. Duplicates are always resolved on our side by the idempotency key — the design never risks losing your data to avoid re-sending it.

Troubleshooting

`auraScope.forwarder` is `undefined` at startup

A required setting is missing or invalid, or the queue directory is not writable. The factory deliberately returns a harmless no-op instead of crashing your app, so this is silent unless you log it. Check the three required variables and the permissions on AURASCOPE_QUEUE_DIR.

`observed` climbs but `accepted` stays at zero

Events are being captured but not acknowledged. Look for one of the forwarder’s own error lines: AuraScope app-hook authorization failed; retaining the durable queue and retrying slowly means the sk_ secret is wrong or revoked — fix it and the retained queue ships. No error line at all points at the ingest endpoint value.

The queue directory grows and never drains

Shipping is paused. That is the circuit breaker doing its job during an ingest outage or a rate-limit — nothing is lost while there is capacity, and it resumes on its own. If it persists, check outbound HTTPS from the app server.

Everything works, then events disappear after a deploy

Your queue directory is on an ephemeral filesystem, so a container replacement takes unshipped batches with it. Point AURASCOPE_QUEUE_DIR at persistent storage.

Fewer requests are recorded than your CDN reports

Requests answered from a CDN cache never reach your app, so an in-app forwarder cannot see them. That is a structural limit, not a fault. See the edge-worker reference for the forwarder that observes at the edge.

Uninstall

(Optional, for a clean flush) call await auraScope.forwarder?.stop() in your shutdown hook, or simply stop the app.
Remove the auraScope(request) call and the createAuraScopeMiddleware() wiring from your middleware (restore your previous middleware.ts if the forwarder was its only content), then npm uninstall @aurascope-analytics/forwarder-app-hook. The package has no runtime dependencies, so nothing else leaves your lockfile with it.
Remove the AURASCOPE_* environment variables.
Delete the queue directory (default .aurascope/app-hook-queue under the process working directory). It only ever contains pending event batches — plain files, no hooks anywhere else in your system.
Tell us during offboarding so we can revoke the sk_ forwarder secret.

Uninstalling is as inert as installing: nothing else in your app referenced the forwarder, so removal is deleting the call site, the config, and a directory.