On Hooks and Heartbeats

You can tell when someone has crossed over from using AI to building with it. They stop asking "what should I put in the prompt?" and start asking "what happens when I'm not in the room?"

Those are different questions with different answers. The first is about a conversation. The second is about a system.

Most of the available advice is about the conversation. Prompt patterns, context engineering, model selection, how to phrase the instruction so the thing actually does what you meant. That advice is real and I've written plenty of it. But it has a ceiling, and you hit the ceiling the moment you need the same behavior twice.

Everything you write in a prompt is a request. A very good request, maybe. Carefully worded, repeated three times, in bold, with a threat. Still a request. The model can honor it, misread it, or lose it entirely when the context window compacts and your careful instruction gets summarized into "the user prefers careful work." You can't tell which happened by looking at the output, because output that followed the rule and output that ignored the rule look identical.

If something must be true every time, it cannot live in a prompt. It has to live in code that runs whether the model cooperates or not.

That's what a harness is: deterministic scaffolding around a nondeterministic core. It has two load-bearing primitives, and the difference between them cost me about a hundred dollars and a week of rework to learn properly.

The word is becoming standard, which is a good sign — Anthropic now publishes guidance under exactly that heading, on building harnesses for agents that run longer than one context window. I'm not coining a term here, I'm working in one, from the other direction: not what the labs recommend in principle, but what a working setup actually needs after it has failed at you a few times.

The Distinction

I learned it in the form of a question I asked after burning money on a mistake I had already fixed once that same morning.

The setup: I rent GPUs by the hour for batch video renders. A rented GPU that nobody turns off bills around the clock, so there's a watchdog process that retires an idle machine. To keep the watchdog from killing a machine mid-job, a running job holds a lock file. Simple.

The sweep finished at 12:05. I looked at 14:57. The machine had been sitting there, idle, billing, for nearly three hours — roughly nine dollars of nothing — because my sweep script had taken the lock and exited without releasing it. The watchdog saw a lock file, concluded a job was running, and dutifully protected an empty machine.

The irony was that I'd built the anti-stranding logic that morning. The guard was fine. The lock lied to it.

So I asked: do we need a hook or a heartbeat or something?

The answer:

A heartbeat, not a hook. A hook would fire at a moment; the failure was a persisting state nobody revoked.

Hooks are for moments. Something happened — a tool is about to run, a session started, context is about to compact. Code fires, synchronously, inside the session, at that boundary. Hooks are the involuntary nervous system: reflexes that fire whether the conscious part is paying attention or not.

Heartbeats are for states that persist. Nothing happened, and that's exactly the problem. A lock still held, a machine still running, a claim still registered, an inbox still unread. Heartbeats are circulation: something that keeps moving between the moments of attention, and that treats continued validity as something you have to keep proving.

A lock file is a moment-shaped solution to a state-shaped problem. It records that a job started. It cannot record that the job is still going. The fix was to make the lock expire — long jobs re-touch it while they run, and a forgotten one goes stale on its own:

def lock_held(ttl_min):
    """Is /workspace/.busy present AND FRESH?

    THE LOCK IS A HEARTBEAT, NOT A FLAG (cost ~3h of idle GPU).
    A sweep script touched .busy, finished, and exited without removing it. The
    watchdog dutifully "held" a completely idle pod for three hours. A lock that
    outlives the job holding it is indistinguishable from a lock nobody cleaned
    up - so it must EXPIRE. Never trust mere existence.
    """
    try:
        age = time.time() - os.path.getmtime(LOCK)
    except OSError:
        return False, None
    return age < ttl_min * 60, age

Never trust mere existence. A flag that says "something is true" is only as good as the discipline of whoever clears it. A heartbeat says "something was still true as of thirty seconds ago," and it degrades honestly when nobody is left to say otherwise.

Everything below is from systems I actually run: a multi-agent coordination bus wired into five hook events, a scheduled briefing that has been running for months, and a GPU watchdog that has now killed several of my own machines on purpose.

Part One: Hooks

The mechanical picture

A hook is a shell command the harness runs when a specific event occurs. It gets JSON on stdin describing what's happening. It writes JSON to stdout, or doesn't. It exits with a status code. That's the entire contract.

You wire it up in settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Edit|Write|NotebookEdit",
        "hooks": [
          {
            "type": "command",
            "command": "\"C:/Python/python.exe\" \"C:/Users/me/.claude/hook.py\" pre-tool-use",
            "timeout": 10
          }
        ]
      }
    ]
  }
}

The matcher is a regex against the tool name. The timeout is your protection against a hook that hangs — and you want it, because a synchronous hook on PreToolUse sits directly in the path of the tool call.

Input arrives on stdin with a common envelope — session_id, cwd, transcript_path, permission_mode, hook_event_name — plus event-specific fields like tool_name and tool_input.

Output has two channels. First, the exit code:

Exit code	What happens
`0`	Success. stdout is parsed as JSON if it looks like JSON.
`2`	Blocking error. stdout ignored, stderr fed back to the model, action blocked.
anything else	Non-blocking error. Execution continues, first stderr line shows in the transcript.

Exit 2 is the blunt instrument: a script that returns 2 and prints "run the formatter first" to stderr stops the tool call and tells the model why. For many use cases that's the whole implementation.

Second, structured JSON on stdout, which is where the real power is:

{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "permissionDecision": "deny",
    "permissionDecisionReason": "Migrations are frozen during the release window",
    "additionalContext": "Text injected into the model's context"
  },
  "systemMessage": "A warning shown to the human",
  "suppressOutput": true
}

The event surface

There are close to thirty hook events. You'll use six. Knowing the rest exist is still worth ten minutes, because the one you need for a weird problem is usually already there.

The load-bearing ones:

Event	Fires when	Can it block?
`SessionStart`	A session begins or resumes	No — context injection only
`UserPromptSubmit`	The human submits a prompt, before processing	Yes
`PreToolUse`	Before a tool executes	Yes — allow/deny/ask
`PostToolUse`	After a tool succeeds	Yes — can also rewrite tool output
`PreCompact`	Before the context window is compacted	Yes
`SessionEnd`	The session terminates	No

Also worth knowing: PostToolUseFailure (a tool errored — a separate path from success, which matters more than you'd think), Stop and SubagentStop (blocking here continues the conversation with your feedback, which is how you build "you're not done until the tests pass"), PermissionRequest, SubagentStart, TaskCompleted, FileChanged, PostCompact, and InstructionsLoaded.

This, by any other name

I'm going to say "hooks" for the rest of this piece, and I mean the Claude Code implementation because that's what I run. But the primitive is not proprietary, and if you work in another stack you already have it under a different name:

	Claude Code	Google ADK	OpenAI Agents SDK
Observe	hooks on lifecycle events	plugin callbacks returning `None`	`RunHooks` / `AgentHooks`
Govern	`permissionDecision`, exit 2	callback returning a value (short-circuits)	guardrails with tripwires
Scope	`matcher` regex on tool name	`before_tool_callback`, `before_model_callback`, …	input, output, and tool guardrails

Google's Agent Development Kit draws the observe/govern line in exactly the same place, and states it more crisply than I did: to observe, implement a hook with no return value; to intervene, return a value, which short-circuits the runner and skips the action entirely. That's the same boundary I arrived at by breaking things, which is either reassuring or annoying depending on how much time you spent getting there.

OpenAI splits the two roles across separate concepts — lifecycle hooks observe, guardrails govern — and formalizes governing further into tripwires that raise typed exceptions.

The translation matters because the rules that follow are not Claude Code rules. They're consequences of running deterministic code inside a nondeterministic loop, and they'll bite you the same way in any of the three.

What hooks are actually for

Every hook I've written does one of three things, and they have very different risk profiles. People reach for the dangerous one first.

Observe. The hook watches and records — a log line, a state file, a metric, a ping. It changes nothing about the session. Start here. An observe-only hook cannot break your work, which means you can leave it running for months and forget it exists.

Inject. The hook adds context. SessionStart returning additionalContext puts facts into a session before the first prompt: what other agents are doing, what the last deploy did, which files are hot. Strictly more reliable than telling the model to go look something up, because the model may not bother, and if the fact is already in context there's nothing to bother with.

Govern. The hook decides — denying a tool call, blocking a stop, rewriting an input. Powerful, and the one that will hurt you. A governing hook is load-bearing: if it's wrong, work stops.

A worked example: the coordination bus

I run multiple agent sessions at once, sometimes on the same machine, often in the same repositories. Two agents editing the same file with no awareness of each other produces exactly the mess you'd expect.

So there's a bus: an append-only log of events — an agent said hello, claimed a path, recorded a decision, handed off work. The interesting part isn't the log. It's that no agent has to remember to use it, because five hook events drive the whole thing:

SessionStart   READ  : fold the bus, inject a bounded digest
PreToolUse     READ  : warn if another agent holds a claim on the path
PostToolUse    WRITE : renew my own claims; throttled heartbeat
PreCompact     WRITE : checkpoint note so the bus outlives the context window
SessionEnd     WRITE : bye

The entry point is 171 lines of Python. Here are the rules I extracted from building it, each of which is really a bug I shipped first.

Rule 1: Fail open, always

if __name__ == "__main__":
    try:
        main()
    except Exception:
        pass          # fail open, always
    sys.exit(0)

Every meaningful block inside is individually wrapped too, so a failure in one section can't take out the others.

This looks like sloppy engineering if you squint. It's a deliberate statement about what tier of infrastructure this is. The bus is advisory. Its value is coordination, and coordination is worth less than the work being coordinated. A hook that crashes on a malformed payload and takes the session down with it has inverted that priority.

The critical part is the last line. A bare except is not enough — Python still exits non-zero on some paths, and a non-zero exit from PreToolUse is a blocking error. The explicit sys.exit(0) is what enforces fail-open.

And because this is the most important property of the whole system, it isn't documented in a comment. It's asserted in a test that feeds the hook garbage and requires exit 0. Prose about the fail-open rule rots into confident wrongness; a test that feeds it garbage fails the moment somebody breaks the guarantee.

Rule 2: Hooks are frozen at session start

This is the most surprising thing about hooks and it made me publicly wrong in front of the person paying me.

Skills hot-reload. Drop a new skill on disk and the running session picks it up. I watched that happen, generalized, and told Bill the hooks were live too. They weren't. I was seeing bus events from other sessions and reading them as proof my own hooks had armed.

Claude Code reads the hooks block once, when a session starts. That config is frozen for the session's lifetime. Skills hot-reload; hooks don't. Edit settings.json and every already-running session keeps the old configuration until it restarts.

The retraction matters less than how the question got settled, which is the transferable part. I designed a probe: plant a claim as a fake agent, then write to that exact path and see whether the warning appears. PreToolUse was chosen deliberately — unlike the heartbeat it has no throttle, so its silence is unambiguous. And I ran a control: pipe the same payload to the hook script by hand, and confirm it produces the warning when called directly.

Without the control, "no warning appeared" has two explanations — the hook didn't fire, or the hook is broken — and you cannot tell them apart. That's the general shape of testing anything that watches. More on this later, because it's the section people skip.

Rule 3: Hooks do not inherit your shell

Hooks inherit the harness's environment, not your interactive shell's. Every export in your shell profile, every variable you set before launching — invisible.

The failure mode isn't a crash. The bus location was configurable by environment variable. Set it in the shell, and the CLI wrote to the configured bus while the hooks — which never saw the variable — beat into the default one. Two buses. Both working. Neither complete. Claims registered by one agent lapsed silently because the renewals were landing somewhere else.

I had this live on my own session: hooks writing me under one identity to one bus, my deliberate notes going to another bus under a different identity. Two identities, one session. Nothing errored. The system just quietly stopped coordinating.

The fix is to derive configuration from something the hook actually receives. The payload includes cwd; a pointer file keyed on that resolves the right bus regardless of environment:

cwd = inp.get("cwd") or os.getcwd()
comms.resolve_home(cwd)

A hook's inputs are stdin and the filesystem. Treat anything else as unavailable.

Rule 4: Identity comes from the payload

Agents on the bus need stable identities. The obvious approach is deriving a handle from configuration — a role name, a machine name. The problem is that two sessions started the same way on the same machine derive the same handle, and then their events interleave in one partition and attribution is gone.

That happened. Two sessions on my desk both wrote the same partition with the same derived session id, and were literally indistinguishable in the status output — we only worked out which was which by file modification times. That window's events are permanently unattributable. You can't fix it after the fact; the information was never recorded.

The fix was to stop deriving identity and start receiving it. Every hook payload carries session_id, which is authoritative and unique:

sid = inp.get("session_id")
if sid:
    os.environ["AGENT_COMMS_SESSION"] = str(sid)

Human-readable handles are still useful. But they're labels, not identity. Identity comes from the harness, which is the only component in a position to know.

Rule 5: Warn, don't block

The PreToolUse hook detects that another agent holds a claim on the file you're about to edit. The tempting move is permissionDecision: "deny".

It's wrong. Claims expire on a TTL. Agents crash. A crashed agent leaves a claim that's live by the clock and dead in reality. If a stale claim can block an edit, one crashed process wedges every other agent out of a file until a timer runs out — and the human has no idea why their session refuses to work.

So the hook allows and attaches a message:

_emit({"hookSpecificOutput": {"hookEventName": "PreToolUse",
                              "permissionDecision": "allow"},
       "systemMessage": "\n".join(lines)})

It names the file, the holder, the reason, and the time remaining, then says: proceeding is allowed, consider coordinating first, check whether the claim is stale.

Deny only when you can prove the operation is wrong. When you merely suspect it, say so loudly and let work continue. There are real cases for a hard deny — secrets about to be committed, a migration during a freeze, production credentials in a connection string. Notice those are all provable from the input. A claim check isn't. That's the line.

That line has a name in OpenAI's governance guidance, which layers checks deliberately: deterministic pre-flight checks run first (pattern matching, PII detection — things code can decide), and model-assisted judgment runs only where deterministic checks can't reach, with a confidence threshold attached. Deterministic first, model judgment only where necessary. If your check needs the model's opinion to decide whether to block, you're building a classifier, and classifiers have false-positive rates that your users will experience as the tool refusing to work.

There's a second axis I under-weighted for a long time: where the check runs relative to the work. OpenAI's guardrails make this explicit with a parallel-versus-blocking choice. Run the check in parallel and you get lower latency, but as their docs put it, if the guardrail fails "the agent may have already consumed tokens and executed tools before being cancelled." Run it blocking and the agent never executes at all — no tokens, no side effects — at the cost of sitting in the critical path.

PreToolUse is inherently the blocking kind, which is why the timeout matters and why the matcher should be narrow. If you find yourself wanting an expensive check — a model call, a network round trip, a full test run — that is a signal you want it after the fact or out of band, not in front of every tool call. Which is the other half of this piece.

Rule 6: Activity is the renewal signal

Claims have a TTL because agents die and their claims must not be immortal. Mine is fifteen minutes.

Then an agent held a file across a long task and the claim lapsed mid-edit. Other agents saw the path as free while it was very much not free.

The bug wasn't the TTL. Renewal was manual — something an agent had to remember, which is exactly the class of thing that doesn't happen. So renewal moved into PostToolUse:

fp = (inp.get("tool_input") or {}).get("file_path")
if fp:
    comms.renew_claims(fp, cwd=cwd)

Editing a file under your own claim is proof you still want it. The evidence was already flowing through the system; it just wasn't wired to the thing it was evidence for.

Generalize this. Whenever you have a lease, a lock, a session, or a cache that expires, look for a signal already passing through your hooks that proves continued interest. Deriving renewal from real activity beats asking anyone — human or model — to remember. (This is the same lesson as the nine-dollar lock file, arriving from the opposite direction.)

Rule 7: Beat only into silence

Before the hooks existed, I had a polling loop: a shell while true that pulled the bus every sixty seconds and emitted a liveness beat every ninth iteration.

It was a disaster in a specific, instructive way. Ninety-two of one hundred seven events on the log were beats. Eighty-six percent of the signal was the system saying "still here" into a log nobody could now read for content. It also had a one-hour ceiling, died with the session it ran in, and still let claims lapse — because it proved presence without proving interest in any particular file.

Moving to a PostToolUse hook made the noise problem worse before it made it better, because a hook fires every turn. So the throttle went on the write side:

def beat_if_stale(interval, cwd=None):
    """The anti-balloon rule at the WRITE side. Heartbeats were 86% of the old
    log; a hook firing per turn would make that worse. Any event we already
    wrote is itself proof of life, so only beat into actual silence."""

Any event you already wrote is itself proof of life. A separate liveness signal is only needed during genuine silence. That one reframe took the log from 86% noise to nearly none, and it applies to every keepalive you will ever write.

Rule 8: PreCompact is a durability boundary

PreCompact fires just before the context window gets summarized. It is the last moment at which the session's full working state exists.

I learned what that's worth by losing it. Mid-project, a compaction ate the working method — and the model came back confidently doing the wrong thing, applying transitions in the wrong places, reintroducing an effect we'd explicitly rejected. Not a crash. A smooth, plausible, wrong continuation. The instruction that followed is the reason half my tooling exists:

Send an agent back through our session to learn our process and techniques and hard code it to disk please so we don't get lobotomized again on auto compact.

Everything not written down by then is gone. Not deleted — summarized, which is worse, because a summary reads like knowledge while having quietly dropped the specific detail you needed. The exact error. The decision you'd already made and are about to re-litigate. The lyrics you pasted three times before anyone persisted them.

So PreCompact writes a checkpoint. It's small. It means the record of what was happening survives into the next context window through a channel that isn't the context window.

If you build one governing hook this year, make it this one. Have PreCompact dump working state to disk — current task, open questions, decisions made, what you already tried. Have SessionStart read it back. That pair converts compaction from an amnesia event into a checkpoint.

This has a name, and I didn't invent it. Anthropic calls it structured note-taking: notes persisted to memory outside the context window, pulled back in later. Their harness guidance for long-running agents builds the same loop out of a progress file, a feature list, and git history — a first session that initializes those artifacts, and every later session that reads them before touching anything. My PreCompact hook is the same idea with the writing automated instead of requested.

One concrete detail from their work that I'd have taken years to discover: write state files as JSON rather than Markdown. The observation is that the model is measurably less likely to inappropriately rewrite or overwrite a JSON file. Prose invites editing. Structure invites appending. If your checkpoint file keeps mysteriously losing entries, that's probably why.

And the reason all of this matters more than it sounds: compaction isn't lossless summarization, it's lossy summarization that reads as complete. Anthropic's term for the underlying pressure is the model's attention budget, which depletes as tokens accumulate — and context rot, where recall degrades as the window fills. You are not protecting against a crash. You're protecting against a confident, fluent continuation built on a summary that quietly dropped the one detail you needed.

Rule 9: Design for the empty case

The first version announced arrival at SessionStart. Reasonable — say hello.

Then I looked at the log:

11:12:45  beat   repo=agent-protocol
11:14:12  hello  repo=PC     11:14:13  bye
11:15:56  hello  repo=PC     11:15:56  bye
11:19:56  hello  repo=PC     11:19:56  bye

Six of seven events in ten minutes were arrival-and-departure records of sessions that did nothing. Five junk partitions in half an hour.

The announcement moved to first real activity. SessionEnd got a matching guard:

if comms.session_has_events():
    comms.append_event({"type": "bye"}, cwd=cwd)

Only say goodbye if you ever said anything. Hooks fire on every session, including all the ones you forgot about. Anything a hook writes unconditionally, it writes thousands of times.

Rule 10: Assume it runs twice

Google's callback guidance includes a line that reads like boilerplate until it costs you something: design callbacks to be idempotent where possible, especially for external side effects.

Here's why it isn't boilerplate. A hook fires on every matching event, and the events are not as discrete as you'd like. A tool gets retried. A session resumes and replays. You reinstall and — if your installer matches its own entries by path instead of by shape — every hook is now registered twice, so every event fires twice. That last one happened to me, and duplicated hooks double every event silently. Nothing errors. The log just quietly becomes wrong.

If your hook appends an event, appending twice should be harmless or detectable. If it increments a counter, you have a counter that lies. If it sends a notification, someone gets two. The bus survives this because appending a duplicate beat is inert and claim renewal is a timestamp write — both naturally idempotent. That was luck the first time and a design rule afterward.

The test is easy and worth running: pipe the same payload to your hook twice and confirm the state afterward is the same as running it once.

Rule 11: The installer is part of the hook

Three bugs lived in installation rather than logic, and they're the ones nobody warns you about.

Match your own entries by shape, not by path. My installer recognized prior installs by looking for a literal path string. Change the path, reinstall, and it doesn't recognize its own previous entry — so it appends instead of replacing. Duplicated hooks double every event. Now it matches structurally.

Scope the matcher. PostToolUse with matcher * spawns a Python interpreter on every single tool call, including every read. Mine is Edit|Write deliberately. A hook you pay for on every read is a hook you'll eventually turn off.

Write an absolute interpreter path. On Windows, python3 resolves on PATH to a Microsoft Store stub that prints an advertisement and exits 49. It looks installed. It runs. It does nothing you wanted. My installer probes candidates by running them and writes the absolute path that answered — with forward slashes, because the value lands in JSON and goes through a shell.

And then the line the installer prints when it finishes, which is Rule 2 wearing work clothes: hooks load at session start, so restart to arm them.

Part Two: Heartbeats

Hooks only run when a session runs. Close the terminal and your beautifully instrumented harness is inert. Everything it knows is frozen at the last event.

Heartbeats are the other half. I'll show two, because they solve different problems and the second one is where the money was.

Before that, an honest calibration. It would be convenient for this essay if nobody had thought about the out-of-session half, and that isn't true. Claude Code ships scheduled tasks that re-run a prompt on a cron expression — though they're scoped to a session, which limits them for exactly the purposes here. Google's Agent Executor does durable execution properly, with an event log and snapshotting so a long-running agent can resume after an outage. The infrastructure exists and some of it is very good.

What I've found genuinely thin is the layer above the plumbing: not how do I run something on a schedule, but what should a scheduled process conclude from the absence of activity? Nearly all the published lifecycle guidance — every vendor, all three stacks — is about the moment something happens. Almost none of it is about the moment something stops happening and no one revokes it. That gap is where my nine dollars went, and it's what the rest of this section is about.

Worth saying too: heartbeats are not free of hazards of their own. There's active research into background agent execution silently polluting memory — a scheduled run is a real turn, and a real turn can write. A heartbeat that thinks is a heartbeat that can be wrong on your behalf while you sleep. Mine are deliberately dumb wherever possible, and the one that isn't has a budget cap.

The briefing heartbeat

This one runs on a timer and does five things:

Run cheap shell checks — calendar, mail, repositories, weather.
Concatenate the results and hash them.
Compare to the previous run's hash. If nothing changed, exit.
If something changed, invoke a model once to synthesize a briefing.
Write briefing and state to disk. Exit.

Two hundred and seventeen lines of bash, and the whole design is step three.

PREVIOUS_HASH=""
if [ -f "$STAGING_FILE" ]; then
    PREVIOUS_HASH=$(shasum "$STAGING_FILE" | awk '{print $1}')
fi

echo "$STAGING" > "$STAGING_FILE"
CURRENT_HASH=$(shasum "$STAGING_FILE" | awk '{print $1}')

if [ "$PREVIOUS_HASH" = "$CURRENT_HASH" ] && [ -f "$BRIEFING_FILE" ]; then
    log "No changes detected, skipping synthesis"
    exit 0
fi

This is the difference between a heartbeat you keep and one you turn off after the first bill. The checks are free — shell commands and API reads. Only synthesis costs money. Gate synthesis on a diff and the system's cost becomes proportional to how much actually happened rather than to how often you look.

There's a second governor on the call itself:

BUDGET=$(jq -r '.budget_per_beat_usd // 0.50' "$CONFIG_FILE")
claude -p --model sonnet --max-turns 3 --max-budget-usd "$BUDGET" "$PROMPT"

A per-beat budget cap, a turn cap, a small model. An unattended process that can spend money needs a ceiling enforced by the invocation, not by intention.

Degrade to raw, never to nothing

BRIEFING_CONTENT=$(claude -p ... "$PROMPT" 2>/dev/null) || {
    log "Claude synthesis failed, writing raw output"
    BRIEFING_CONTENT="# Pulse Check — $NOW
(Synthesis failed, raw output below)

$STAGING"
}

If the model call fails, the raw check output gets written instead. Uglier, still a briefing, every fact present.

Unattended systems fail while you aren't looking, and a failure you find out about is much better than a silence you mistake for "nothing happened." Never let the fancy part's failure mode be an empty file.

Make the safety rule structural, not textual

The briefing reads mail. It must never send mail. An unattended loop with a model in it and send access is a bad night waiting to happen.

The weak enforcement is instructions: tell the model it's read-only, put it in the prompt, put it in the docs. Requests, all of them.

The strong enforcement is filesystem layout. Read credentials live in one directory. Send credentials live in a send/ subdirectory. The function that enumerates accounts only looks at the first directory. It does not have a permission it chooses not to use — it has no path to the credential at all.

The read-only rule is a property of the layout, not of anybody's good behavior. When you need a capability boundary in an agentic system, build it out of structure. Separate directories, separate credentials, separate processes, separate machines. Anything you enforce with a filesystem or a permission bit will hold. Anything you enforce with a sentence in a prompt is a preference.

A monitor that cries wolf gets ignored

The failure mode of automated briefings isn't being wrong. It's surfacing the same six things every cycle until you stop reading.

Two mechanisms fight it: the prompt gets a list of resolved and dismissed items and is told not to resurface them, plus an "active work" registry of known states — uncommitted files, unpushed commits, a deploy failing on purpose — with instructions not to flag those as problems. And the standing rule: skip anything that's all quiet.

A monitoring system's real metric isn't coverage. It's whether a human still reads the output in week three.

The watchdog that lives on the machine it kills

The second heartbeat is the one that came out of the GPU work, and it has the better war stories.

Rented GPUs bill by the hour. A forgotten one bills 24/7 — at roughly forty-four cents an hour, ten dollars a day for nothing. So: a watchdog that retires an idle machine.

The first design instinct is to run the watchdog locally. That instinct is wrong, and the code says why:

# a local watchdog dies with the laptop/session/agent, and a forgotten pod bills
# 24/7 ($0.44/hr = $10/day). This one lives on the machine it kills.

A watchdog that shares a fate with the thing it watches is not a watchdog. If your laptop sleeping can strand the machine, your laptop sleeping will strand the machine. So the process runs on the rented box, polls every thirty seconds, and terminates its own host.

Four rules came out of that watchdog, each paid for.

Unreachable is not idle. The watchdog checks whether the render queue is busy by polling the local API. When that call fails it returns "unknown," and unknown is treated as busy. A booting machine is unreachable and very much alive. Kill on "no answer" and you'll kill every machine during startup.

Refuse to start without the credential that lets you stop. The watchdog checks for its API key at launch and exits if it's missing, with the reason spelled out: it could never stop the pod. A watchdog that starts successfully and can't perform its one action is worse than no watchdog, because you'll believe you're covered.

Retiring is a decision, not a reflex. When the machine goes idle, the watchdog picks between stopping (resumable, keeps the disk) and terminating (destroys it) based on whether GPU stock is thin and whether outputs have been synced. If work is unsynced and stock is thin, it refuses to do either and logs loudly on every poll until a human syncs.

That third rule exists because of the most expensive lesson in the set. A long batch ran on a machine writing to local disk with no copy-back. The provider stopped the pod, the GPUs were reclaimed, and it could not restart — not enough free GPUs on the host machine. Somewhere between two and four hundred rendered frames were on that disk. All of them were lost, along with about twenty-two dollars of compute that produced nothing.

The comment at the top of the sync script now reads:

# WHY THIS EXISTS: A long batch was run ON a pod writing to pod-local disk, with
# no copy-back. The pod stopped, the GPUs were reclaimed, the pod could not restart
# and EVERY frame was lost.

A completion marker is not completion. A download script printed DL_ALL_DONE after a total failure, because set -e doesn't catch a no-op. Monitor artifacts on disk. Never trust a script's own success message — including your own.

Part Three: Which One Do You Need?

You now have both shapes. The decision is usually fast:

Reach for a hook when the trigger is an event and the response is immediate. Something is about to happen and you want to observe, enrich, or veto it. Hooks are cheap, synchronous, and bounded — and they cannot help you at all once the session ends.

Reach for a heartbeat when the failure is a state that persists and nobody revokes it. Nothing fires. Nothing errors. A resource stays held, a claim stays registered, a machine stays billing. If your honest description of the failure contains the words "still" or "never got cleaned up," you need a heartbeat.

The tell: ask what the absence of activity should mean. Hooks have nothing to say about absence — they only speak when something happens. Heartbeats exist precisely to make silence meaningful. A lock that expires, a claim that lapses, a machine that retires: each is a way of saying "silence now counts as a negative answer."

Most real systems need both, wired together.

Part Four: The Loop Between Them

Hooks and heartbeats are more useful together, and the seam is the filesystem.

A hook can't wait around — it runs in the path of a tool call with a timeout. A heartbeat can't observe — it isn't in the session. But a hook can write, and a heartbeat can read.

in-session:   hook observes  →  writes state to disk
out-of-session:              →  heartbeat reads state, acts, writes back
next session: SessionStart hook reads  →  injects into context

That's continuity without a running process. The coordination bus is exactly this loop: hooks append events, a fold reduces them to current state, SessionStart injects the digest. Nothing is resident in memory. Nothing has to be.

Which reframes what you're building. Not an agent with memory — a filesystem with a schedule, that sessions attach to and detach from.

Testing The Thing That Watches

Here's the part everyone skips, including me, right up until it cost real money.

A hook that silently does nothing looks exactly like a hook with nothing to report. A check that always returns clean looks exactly like a system that is clean. You cannot distinguish a working detector from a broken one by looking at a passing result.

A cheap example from the week I wrote this. I wanted to know whether a /writing section existed on a site of mine, so I checked the URL. It returned 200. Section exists, obviously. Then I checked /obviously-not-a-real-page-xyz. Also 200 — the host serves the homepage for every unmatched path, so every URL on that domain "exists." My detector could not return a negative. Trusting the first result would have meant building on a page that wasn't there.

That's a free lesson from a URL check. The expensive version is the same mistake inside a watchdog, where a guard that can't fail protects an idle machine for three hours.

Concretely:

Mutation-test every guard. Break the thing it's supposed to catch and confirm it catches it. When I rewrote the lock check, I verified it by editing the function to always return True and confirming the test suite went red, then restoring it. A test that passes against deliberately broken code is not a test.

Keep a control. When a probe produces silence, you need an independent way to prove the mechanism works when invoked. Pipe a payload to the hook by hand. Otherwise "nothing happened" has two explanations and you can't tell them apart.

Run the hook by hand. It's a command that reads JSON on stdin:

echo '{"session_id":"test","cwd":"/tmp","tool_input":{"file_path":"/tmp/x"}}' \
  | python hook.py pre-tool-use; echo "exit=$?"

Check the exit code explicitly. Piping into head gives you head's exit status, which is a great way to convince yourself a broken hook is fine.

I should be honest about how well that rule works. It has been written down in my own instructions file for months. While preparing this piece I ran a check to confirm a machine was dead, piped it to tail, and read the reported exit status of 0 as success. The real status was 255. I then commissioned a crawl of my own sessions looking for hook opportunities, and it came back with that exact pattern ranked second — 89 instances where the discarded exit code sat inside an && chain, including one where a PowerShell parse error was swallowed by | tail -3 and the very next sentence was "Tests pass."

That is the entire argument for hooks in one paragraph. I wrote the rule. I understood the rule. I was writing an article about the rule. And I broke it inside the hour, because a rule you have to remember is a rule you will eventually not remember. It is becoming a hook — a warning, not a block, because ls | head is fine roughly fourteen hundred times in that same corpus and a guard with false positives gets switched off.

Feed it garbage. Empty stdin, malformed JSON, missing fields, a payload from a different event. A hook that only works on well-formed input will meet malformed input eventually, on a Friday.

Test the honest failure. Make the hook throw on purpose and confirm the session survives. If your fail-open guarantee has never been tested, you don't have one — you have an intention.

Assert the replacement, not just the removal. When the stale-lock bug was fixed, the old tripwire — which asserted the lock file existed — had itself become the bug. It wasn't deleted. It was rewritten to assert the new guarantee: that no code path treats a bare existing lockfile as authoritative. Deleting a test because it now fails throws away the reason it was written.

Where To Start

Don't build the bus. Build one hook that observes, and let it run for a week.

A minimal hook. Log every file edit:

#!/bin/bash
# PostToolUse, matcher: Edit|Write
python3 -c "import sys,json; d=json.load(sys.stdin); \
  print(d.get('tool_input',{}).get('file_path',''))" \
  >> ~/.claude/edits.log 2>/dev/null
exit 0

It cannot break anything, it exits 0 unconditionally, and after a week you have a log of every file you touched across every session. That log will tell you something true about your own work that you did not know.

A minimal heartbeat. Cheap check, diff, gate:

#!/bin/bash
NEW=$(git -C ~/repos/thing status --porcelain)
OLD=$(cat /tmp/thing.state 2>/dev/null)
echo "$NEW" > /tmp/thing.state
[ "$NEW" = "$OLD" ] && exit 0
# something changed — now do the expensive thing

Everything else is elaboration.

Then climb. Add SessionStart injection so sessions begin informed. Add PreCompact state-dumping so context survives compaction. Add expiry to anything you currently treat as a flag. Reach for a hard deny last, and only when a false positive is cheaper than a false negative.

And restart your session after editing settings.json. You will forget this at least once.

The Argument

There's a durability ladder I use for anything I want to still be true in six months. A test is a lesson that cannot rot — it fails when the lesson is forgotten. A hook is a lesson that cannot be skipped — the harness enforces it. A lint rule is a lesson the tooling repeats for you. A document is prose that decays silently into confident wrongness. A comment is a document nobody indexed.

Most people trying to make an AI system behave reliably are working at the bottom of that ladder. Longer instruction files, more emphatic wording, another paragraph explaining what not to do. It's the accessible move, and it degrades the moment context is tight.

Hooks are a rung up because they don't depend on cooperation. Heartbeats are a rung up because they don't depend on presence. Neither makes the model smarter. Both make the system dependable, which is the thing you actually wanted.

The model is not the system. The harness is. Build the harness.

Further reading. If you want the lab-side treatment of the same ground: Anthropic on effective harnesses for long-running agents and context engineering; Google's ADK callback patterns and plugins; OpenAI's guardrails and practical guide to building agents. Read them for the principles. Come back here for what happens when you run them on a Tuesday.

First in a series on agentic harness engineering — the scaffolding around the model rather than the model itself. Next: memory that survives compaction, and why the context window is a budget rather than a container.