Working Through Log Data at Datacenter Scale
Here's a design problem I've been chewing on: five AI datacenters around the globe, big multi-tenant GPU clusters, and every host constantly writing logs - training output, kernel messages, heartbeats, security events, whatever a process feels like saying. How would you handle all of it?
The problem is deliberately open-ended, and the fun is imposing structure on it: the napkin math, the architecture, the alert path, and the grubby mechanics of tailing a log file without losing your place. At the end I'll show you the sketch I drew before thinking any of this through, because the gap between that sketch and the final design turned out to be the most useful part.
Do the math before you draw boxes
Assume 20,000 hosts per datacenter across five datacenters. GPU nodes are chatty - training stdout, CUDA noise, kernel messages - call it 10 KB/s per host on average. That works out to roughly 1 GB/s aggregate, about 86 TB of raw logs per day, with 5-10x spikes when a big job starts crash-looping. Compress it around 10:1 and you're storing ~9 TB/day. Hold 90 days and you're at petabyte scale.
Two things fall out of that math, and they decide everything else:
- You cannot full-text-index everything. Search-indexing 86 TB/day costs more than the electricity for the GPUs.
- You should not ship raw logs across regions. Bandwidth, blast radius, and data sovereignty all point in the same direction.
"Generic system logs" are not one stream
What most "Kafka plus Elasticsearch" answers miss is that these logs have very different consumers and very different service levels. Classify at the edge:
| Class | Volume | Latency need | Retention |
|---|---|---|---|
| Security / audit events | tiny | seconds | 1+ year, tamper-evident |
| Kernel / hardware (Xid, ECC, NVLink) | small | seconds-minutes | months |
| Heartbeats | medium | n/a | days - convert to metrics |
| Training output (customer stdout) | ~90% of volume | best-effort | tenant-defined |
Keep everything regional, share only what has to be global
Each datacenter runs a self-sufficient pipeline. The only things that cross regions are alerts, security events, and aggregates.
On every host an agent tails journald, kernel logs, and container stdout. It stamps each line with host, cluster, tenant, and job metadata. It buffers to local disk for crash safety, and it rate-limits per source so one crash-looping process cannot flood the pipeline. Per datacenter: a load-balanced ingest tier feeds Kafka, stream processors parse a light envelope and route by class, and everything lands in two storage tiers - a hot store (1-7 days) and a cold store (90+ days, columnar files on object storage).
Why a label index plus brute-force scan instead of indexing everything: at this volume, indexing just the labels (tenant, host, job, severity) and scanning compressed chunks for the actual grep is 10-50x cheaper, and a few seconds of search latency is fine for debugging. Save real full-text indexing for the small security stream.
How fast can an alert reach a customer?
Scale the fleet down to 1,000 servers and ask: how fast can a security or error event reach the customer? At that size the pipeline is nowhere near the bottleneck. All of the latency comes from decisions, not from hardware:
- Agent flush interval. Batching defaults favor throughput. Give the alert class its own 100-250ms flush.
- Detection semantics. A stateless rule (severity=critical, known signature) fires in milliseconds. A windowed rule - "5 failed logins in 60s" - cannot fire before the window closes, no matter how good the infrastructure is.
- Alert-manager grouping. This one gets missed a lot. Default group-wait settings add 30 seconds. Set it to ~0 for the critical class.
- Delivery channel. Webhook or dashboard socket is sub-second. Email is 5 seconds to forever.
Sum the tuned path and the technical floor is about 1-2 seconds end to end. If you need it faster than that, run the high-confidence rules on the host agent and fire straight to the alert gateway, skipping Kafka. That gets you 200-500ms. The same event still goes through the durable path for forensics.
Tailing a file without losing your place
Log files are just files that keep growing. How do you pull data and always resume where you left off? This is called checkpointed tailing with durable offsets. Vector, Filebeat, and Fluent Bit all do it:
loop: seek to saved byte offset # from the checkpoint registry read what's new consume only up to the last \n # partial lines stay unread ship the batch wait for the ACK then atomically persist new offset # write temp file + rename
Checkpoint after the ack, never after the read. Crash between ship and checkpoint and you re-send one batch: duplicates, never gaps. That's at-least-once delivery, and (host, file, offset) makes a natural dedupe key if you need better.
The part that bites everyone is file identity. You can't key checkpoints on filenames, because rotation renames files out from under you. Key on inode plus a fingerprint of the file's first kilobyte, and rotation handling falls out naturally. A renamed file keeps its inode, so you drain it to the end through the open descriptor. A truncated file shows a size smaller than your saved offset, so you reset to zero. A brand-new file at the old path starts fresh.
If the ingest tier goes down, the agent stops advancing offsets. The log file itself is your backpressure buffer. Monitor agent lag - file size minus checkpointed offset - and you get your early warning for free.
Rust for the agent, Go for everything else
The seconds in the latency budget come from batching and delivery, not language speed, so "fastest language" is the wrong axis. What matters is predictable tail latency with no GC pauses on the hot path, a small footprint on hosts shared with paying training jobs, and development speed for the 90% of the system that is just network services. That lands on Rust for the host agent and Go for the control plane, which is where the industry landed too. Vector is Rust; Prometheus, Loki, and Alertmanager are Go.
Why not C or C++, since raw performance ties? Because the agent runs as root on every host in a multi-tenant fleet, and its core job is parsing attacker-influenced bytes - log lines are whatever customer workloads write to stdout. A heap overflow in a C parser under that profile is tenant-to-root privilege escalation replicated across the fleet. Fluent Bit's CVE-2024-4323 is the case study. Rust keeps C's performance and deletes the entire memory-corruption bug class at compile time.
Does it survive 15,000 servers? 100,000?
15,000 hosts is ~150 MB/s aggregate, ~13 TB/day raw. A modest Kafka cluster per region yawns at that. What changes is that a few things you could be sloppy about become hard requirements: correlated alert storms become the top problem (one dead switch makes 2,000 hosts scream, and without rollup you page a customer 2,000 times), ingest reconnects need jitter so a restart doesn't self-DDoS, and the agent fleet needs canaried staged rollouts, because a bad release of a root-privileged agent is now a 15,000-host incident.
The design genuinely breaks around 50-100k hosts per region, and the fix is cells: independent pipeline stacks of 10-20k hosts with a thin federation layer. At that scale throughput stops being the constraint and blast radius becomes it.
The sketch I drew first
Before working through any of this, I sketched my instinctive answer: a cron shell script (pull_logs.sh) polling a /LOGS endpoint on every server, "check dates, leave others" for resume, per-customer SQLite files for storage, and a Node API with auth in front.
Here's the honest scorecard. The problems I identified were the right ones: resumable ingestion, per-tenant isolation, authenticated customer APIs, even a hint of log classification. The mechanisms were all v0 picks that break on schedule: pull instead of push (alert latency becomes your cron interval, plus a log endpoint exposed on every host), timestamps instead of offsets (clock skew and same-millisecond lines silently lose data), and SQLite as a log store (single-writer locks fall over fast). And there was no alerting path at all - the design stored logs, but nothing watched them.
I built it (and it bit back)
Two days after writing this, I built the home-scale version: Howl - three of my machines (a MacBook and two Ubuntu servers) shipping their logs to a Cloudflare Worker, with hot storage in D1, week-old days rolled nightly to R2 as gzip files, phone alerts on errors and on silence, and agents that update themselves within fifteen minutes of any push. All of the rules above held up. Here is what went wrong.
Scar one: the agent watched itself. Within the first hour on a systemd box, the pipeline was ingesting twice a second, forever. The unit's stdout went to journald, Ubuntu forwards journald into /var/log/syslog, and the agent watches syslog - so it shipped a line, logged "shipped 1 line," read its own log line, shipped that, and looped. The oldest trap in log collection, and I walked right into it:
Scar two: "works in my terminal" proves nothing about services. The agent ran perfectly in every shell test, then crash-looped under launchd because launchd's PATH has no idea where Homebrew keeps node. The Linux twin followed a day later: systemd's bare PATH versus an nvm-managed node. Then sudo git clone failed because sudo authenticates SSH as root, not you; then the self-updater could build new versions but not restart into them until a one-line sudoers rule existed. All four are the same problem: the service manager's environment is nothing like your shell. The first install on a new platform now gets treated like new code.
Scar three: migrations need two doors. Giving the receiver a custom domain silently disabled the old workers.dev endpoint - the one every agent pointed at - and briefly deafened the fleet. Days later, macOS's resolver cached a stale "no such host" for the new domain while real DNS was fine the whole time. Both incidents ended the same way: agents backed off, checkpoints froze, files buffered, everything drained on reconnect with zero loss - and the old endpoint, deliberately kept alive, was the fallback door. Migration rule: keep the old path open until the new one has soaked. The alert that fired during the outage was the system doing its job.
Scar four: the fingerprint earned its keep in testing. The hardened agent holds a descriptor per file, drains rotated and deleted files dry before switching (the diagram in section 05 is now literally what ships), fingerprints each file's first kilobyte to catch inode reuse and in-place rewrites, and force-consumes a quarter-megabyte no-newline blob rather than wedging. A nine-assertion harness drives a real agent binary through rotation-with-trapped-tail, deletion, truncation, and rewrite-across-restart: 166 lines, zero gaps. My own test failed twice by asserting fingerprint behavior on files too small to carry one - the sub-1KB blind spot is documented precisely because I fell into it.
Here is what the fleet actually looks like now - note that data and code flow in opposite directions, and nothing ever connects to a machine: