Web-server access logs — the origin-log forwarder

The origin-log forwarder is the universal option: if your web server (nginx, Apache, or anything else that writes standard combined-format access logs) already writes an access log, this works — no framework hook, no CDN, no JavaScript, no change to your web server configuration.

What you get

A single small binary (aurascope-origin-log) that runs as its own process on your server, follows your access-log file, and ships each completed log line to us byte-for-byte, unmodified. It never rewrites, reformats, or redacts your lines at the source, and it never writes to your log file — it only reads it.
Server-side visibility of the traffic that browser JavaScript can never see: AI assistants and crawlers fetching your pages. These request events feed verification — is this visitor really the bot it claims to be? — on our side, at read time.
Zero presence in your request path. The forwarder is not a proxy, not a module, and not middleware. It reads a file. If it crashes, stalls, or loses connectivity, your site serves traffic exactly as before; the only thing that can ever be affected is our measurement.
Any secret-looking values in shipped URLs (tokens, keys in query strings) are redacted on our side before durable storage — nothing is silently discarded on yours.

This page describes the origin-log component for sites where reading the access log is where traffic is best observed.

One caveat to know up front: this forwarder sees what reaches your origin. Requests answered entirely by a CDN cache never appear in your origin access log, so it cannot see them. If a caching CDN fronts your site, ask us about pairing with the edge-worker forwarder.

Prerequisites

Combined-format access logs. Your server must write the standard “combined” log format — nginx’s built-in combined format, or Apache’s combined (NCSA extended) LogFormat. A line looks like:
```
203.0.113.9 - - [16/Jul/2026:10:15:42 +0000] "GET /pricing HTTP/1.1" 200 5316 "https://www.example.com/" "Mozilla/5.0 (compatible; ExampleBot/1.0)"
```
The format must be exactly combined — nothing appended after the closing user-agent quote (no extra timing fields, no custom suffix). Our parser for the combined-v1 format is strict end-to-end on purpose: a line that does not match is never silently dropped — it is stored and counted as a parse fault on our side so we can see it and add support — but it will not become a measurement. If your log format carries extra fields, tell us before installing.
A place to run one long-lived process with:
- read access to the access-log file,
- write access to two small state directories (a durable queue and an offset file),
- outbound HTTPS to your ingest endpoint.
Your credentials from onboarding:
- your ingest endpoint URL (shown below as https://ingest.example-analytics.com — a placeholder; use the endpoint you were given at onboarding),
- your public site key (pk_…),
- your per-forwarder secret key (sk_…). Treat the sk_ value like a password: environment only, never committed to a repository or written into logs.
A Go toolchain (Go 1.26+) to build the binary from the source package you receive at onboarding. It need not be on the origin host — see below.

How you receive the software

Today: a source archive, aurascope-origin-log.tar.gz, handed to you at onboarding. You unpack it and compile the binary yourself — step 1 below is that, in full.

Shortly: prebuilt binaries. We are standing up a published download channel for this forwarder — versioned, checksummed and signed static binaries with a one-line install script that picks the right build for your platform, so you will not need a Go toolchain at all. The archive handover above is the interim until that is live. Ask us where it stands when you install; if it has landed, we will point you at the script instead of an archive.

Install steps

1. Unpack the source and build the binary

The archive unpacks into a single directory, aurascope-origin-log/, whose root is the Go module itself — there is no enclosing project to descend into:

tar -xzf aurascope-origin-log.tar.gz
cd aurascope-origin-log
go build ./cmd/aurascope-origin-log

This produces a self-contained aurascope-origin-log binary with no runtime dependencies. Place it wherever you keep operational binaries, e.g. /usr/local/bin/aurascope-origin-log.

You can build on any machine with a Go toolchain and copy the resulting binary to the origin host — it has no runtime dependencies, so the host needs no toolchain of its own.

2. Configure the environment

The binary is configured entirely by environment variables — it takes no command-line flags.

Variable	Required	Value
`AURASCOPE_INGEST_URL`	yes	Absolute HTTPS base URL of your ingest endpoint from onboarding, e.g. `https://ingest.example-analytics.com`
`AURASCOPE_FORWARDER_SK`	yes	Your per-forwarder secret, starts with `sk_`
`AURASCOPE_SITE_KEY`	yes	Your public site key, starts with `pk_`
`AURASCOPE_LOG_PATH`	yes	The access-log file to follow, e.g. `/var/log/nginx/access.log`
`AURASCOPE_FORMAT_ID`	yes	The log-format contract; for combined logs this is `combined-v1`
`AURASCOPE_QUEUE_DIR`	yes	A durable directory the forwarder owns for its send queue, e.g. `/var/lib/aurascope/queue`
`AURASCOPE_OFFSET_DIR`	yes	A durable directory for the read-position state, e.g. `/var/lib/aurascope/offset`
`AURASCOPE_BUFFER_MAX_LINES`	no	Local buffer capacity in lines. Defaults to 864,000 — sized as 10 lines/second for a full day-long outage
`AURASCOPE_PROBE_MARKER`	no	For staging or synthetic-load installs only — see “A note on test-traffic markers” below. Leave unset in production

Misconfiguration is loud, never silent: a missing or malformed value prints an AuraScope origin-log configuration error: … message and the process exits with status 2 before touching anything.

3. Run it

AURASCOPE_INGEST_URL="https://ingest.example-analytics.com" \
AURASCOPE_FORWARDER_SK="sk_xxxxxxxxxxxxxxxx_redacted" \
AURASCOPE_SITE_KEY="pk_your_site_key" \
AURASCOPE_LOG_PATH="/var/log/nginx/access.log" \
AURASCOPE_FORMAT_ID="combined-v1" \
AURASCOPE_QUEUE_DIR="/var/lib/aurascope/queue" \
AURASCOPE_OFFSET_DIR="/var/lib/aurascope/offset" \
./aurascope-origin-log

Run it under whatever process supervisor you already use — it only needs the environment above and restarts safely at any time (see “Failure modes”). A minimal systemd example:

[Unit]
Description=AuraScope Analytics origin-log forwarder
After=network-online.target

[Service]
Environment=AURASCOPE_INGEST_URL=https://ingest.example-analytics.com
Environment=AURASCOPE_SITE_KEY=pk_your_site_key
Environment=AURASCOPE_LOG_PATH=/var/log/nginx/access.log
Environment=AURASCOPE_FORMAT_ID=combined-v1
Environment=AURASCOPE_QUEUE_DIR=/var/lib/aurascope/queue
Environment=AURASCOPE_OFFSET_DIR=/var/lib/aurascope/offset
# Keep the secret out of unit files readable by other users, e.g.:
EnvironmentFile=/etc/aurascope/origin-log.env
ExecStart=/usr/local/bin/aurascope-origin-log
Restart=always
User=aurascope

[Install]
WantedBy=multi-user.target

(Put AURASCOPE_FORWARDER_SK=sk_… in the root-only EnvironmentFile, not the unit. The service user needs read access to the log file and write access to the two state directories.)

Notes for your firewall and your expectations:

The forwarder makes exactly one kind of outbound call: POST to <your ingest URL>/v1/logs/origin-log over HTTPS, authenticated with your sk_ secret as a bearer token.
It polls the log file once per second and ships complete lines only (a partially written line is held until its newline arrives, even across restarts).
On first start it reads your log file from the beginning, so the file’s current contents ship immediately. If you do not want existing history shipped, rotate the log just before first start so the forwarder begins on a fresh file.
Lines are shipped in durable batches of at most 500, each with an idempotency key that is stored with the batch and reused verbatim on every retry — so a retried batch can never be double-counted on our side.

A note on test-traffic markers

Like the other AuraScope forwarders, this one accepts an AURASCOPE_PROBE_MARKER environment variable to declare self-generated test traffic (a load-test target, a staging copy) so it is never counted as real. Set it only on an install whose traffic is entirely synthetic, and leave it unset in production — it marks every line the install ships.

The marker travels as batch metadata (a request header sent alongside each batch), never inside your log lines: this forwarder’s contract is that your lines ship byte-for-byte unmodified, and that holds with or without a marker. Earlier versions had no safe way to carry the marker and refused to start when it was set; that refusal is gone — the same variable now simply works, with your lines still pristine.

Verify your install

There is no self-serve install-check dashboard for this request measurement yet, so verification uses the forwarder’s own observable signals plus a confirmation from us:

Startup is silent when healthy. The binary logs only problems. If it prints configuration error or startup error and exits, fix the reported variable and start again.
Watch the queue drain. Captured lines appear as batch files under $AURASCOPE_QUEUE_DIR/pending/ (pairs of *.ndjson and *.meta.json). Within seconds of a successful ship they are deleted. A pending/ directory that empties after activity is an end-to-end success signal: the batch was accepted and durably stored before the forwarder was allowed to delete it — our ingest never acknowledges data it cannot keep.
Send yourself a request. Fetch a distinctive URL on your own site (e.g. curl https://www.your-site.example/?install-check=1), confirm the line appears in your access log, then confirm pending/ fills and drains.
Check the failure channel is quiet. In the process’s log output, lines beginning AuraScope origin-log … report problems (capture failure, authorization failure, buffer exhaustion). Healthy operation prints nothing.
Ask us to confirm. During onboarding we confirm your request events are landing and parsing as combined-v1. In particular, if every line were failing to parse (wrong log format), your side would still look healthy — the batches are accepted and the queue drains — so this confirmation is the check that catches a format mismatch.

Failure modes — and what they do to your site

Short answer: nothing. The forwarder is a separate process that opens your log file read-only. It is not in the path of any visitor request, it never modifies your web server configuration, and every internal error is logged and swallowed rather than raised. The worst possible failure is a gap in our measurement, never an effect on your site. Specifics:

Our service is unreachable (outage, DNS, network). The forwarder pauses sending and retains everything in its durable on-disk queue, retrying with exponential backoff (starting at 1 second, capped at 5 minutes) and honoring any Retry-After we send. When service returns, the queue drains oldest-first. Nothing is lost within buffer capacity, and the default capacity is sized for a full day-long outage.
The buffer fills anyway. Only then are the oldest complete batches evicted, keeping the newest contiguous window, and every eviction logs the exact number of dropped lines locally. Loss is always counted, never silent. Raise AURASCOPE_BUFFER_MAX_LINES if you want more headroom.
Wrong or revoked credentials. Rejected batches are retained, not deleted, and retried slowly (every 5 minutes) so a credential rotation can never destroy data — fix the sk_ value and the retained queue ships.
The forwarder crashes or the host reboots. Queued batches and the read position are persisted with write-through file sync and atomic renames; a restarted process recovers both and resumes exactly where it stopped. Retries reuse the original idempotency key, so recovery cannot double-count.
Log rotation. Plain appends resume at the saved position; a renamed-and-recreated or truncated file (both common rotation schemes) is detected by file identity and size, and the forwarder restarts from the top of the new file — the new file’s first lines are not lost.
The log file is temporarily missing (mid-rotation). Not an error: the forwarder waits and polls again.
A line doesn’t parse on our side. It is stored and counted as a fault, never silently discarded — and never becomes a wrong measurement.

Troubleshooting

The process exits immediately with a configuration error

The message names the variable. Every required value is checked before anything is opened, so this always happens at start and never mid-run.

Nothing appears under `pending/` even though the site is being visited

The forwarder is not reading the file you think it is, or cannot read it at all. Confirm AURASCOPE_LOG_PATH is the file your web server actually appends to, and that the service user can read it.

`pending/` fills but never empties

Batches are being retained rather than lost. The line

AuraScope origin-log authorization failed; retaining durable batches and retrying slowly

means the sk_ secret is wrong or revoked; no line at all points at outbound HTTPS or the ingest endpoint value. Fix it and the retained queue ships in order.

Everything looks healthy on your side, but we see no usable requests

The most likely cause is a log format that is not exactly combined — every line then lands as a parse fault rather than a measurement, while your queue still drains normally. This is why the confirmation step exists; send us a sample line.

The whole log history shipped on first start

Expected: the forwarder starts at the beginning of the file. Rotate the log just before first start if you do not want existing history shipped.

Requests your CDN reports are missing

Requests answered from a CDN cache never reach your origin, so they never appear in your access log. That is a structural limit, not a fault. See the edge-worker reference for the forwarder that observes at the edge.

Uninstall

Stop the process (SIGTERM is handled gracefully — the forwarder attempts one final flush of its queue, then exits).
Delete the binary and, if you want a clean removal, the two state directories (AURASCOPE_QUEUE_DIR, AURASCOPE_OFFSET_DIR). If you might reinstall later, keep the offset directory so already-shipped lines are not shipped a second time.
Tell us so we can revoke the forwarder’s sk_ secret.

Nothing else changes: the forwarder never touched your web server configuration, your log files, or your site — there is nothing to roll back.