The context waterline

TL;DR There’s no one-size-fits-all policy for deciding what context is the direct responsibility of a person, and what can be left to the AI. I think of this distinction as the “context waterline”. It varies by project, but deserves deliberate consideration so that the things that need human attention don’t get lost in the noise.

---

Getting my ticketing system and knowledge graph side project quickly off the ground was satisfying. It has been harder to figure out what direction the rest of the context management experiment should go. I usually have a fairly practical approach: squeeze the most juice out of the tools available, don’t reinvent the wheel, don’t wish we had Tool X instead of Tool Y. Good rules of thumb, but since one of my goals is learning-by-building, I do in fact need to reinvent the wheel.

My first goal with ACME could be described as building an “agent orchestration via a ticketing system”. That was pretty straightforward, since breaking down and scheduling work by projects, milestones and tickets is tried and tested. It also followed on naturally from already having an established routine revolving around Linear.

I struggled to define the second goal. Not for lack of ideas, but having too many of them. I don’t want to build a software factory or figure out the perfect spec-driven development system.

One way or another, the next step after managing tickets is managing files. Perhaps naively, I hoped I’d settle on an effective workflow with a simple heuristic like, “the agent owns everything in the repo, the human directs everything via tickets”. Or “just use a folder system, edit via Obsidian”. I never found a policy I was happy with.

There’s no easy dividing line

The problem is, I keep hitting exceptions. A good example is the capabilities map in Mock Machines. I want a page where I can see what product features have been delivered or planned. Fundamentally that needs human judgement. It takes a good product mindset to decide what gets prioritised, what gets cut.

At the same time, the map has to stay aligned to the code. I built tooling specifically so that the agent can annotate the code, and then the map updates automatically to reflect what has actually been built and tested.

It’s an anti-pattern to try and force a “neat” solution, so I’m parking any further work on the bidirectional sync between the knowledge graph in ACME, and the files and folders on disk. For one complication or another, I decided that wasn’t a good use of time.

I did settle upon a metaphor to keep my head clear: the “context waterline”. It’s my way of recognising that the boundaries of responsibility between human and agent are fluid - but not entirely unpredictable. Things above the waterline (the blue wave) are a human’s responsibility, things below the waterline are an agent’s. It doesn’t perfectly align to a system boundary such as acme vs. git, but every project has its context waterline.

I always enjoy stretching a metaphor to breaking point, so let’s keeping going! Imagine that the code represents deep, impenetrable water. An ocean of logic whose depths you don’t necessarily understand. The development team sails along on top, nice and dry. The coding agents work below the surface, equipped with thru-water comms to get their goals and to ask question of the surface crew. The crew also has sonar: an ability to constantly scan the code instead of rely on their out-of-date maps. All combined, we have clear boundaries with tools and processes to communicate across them.

Coming up for metaphorical air, let’s make that more practical. By thru-water comms, I means the agents need an API to get context i.e. query the context management system. The sonar is project tooling so that critical docs are automatically and deterministically kept up to date. For instance, I have a capabilities map where a manually structured and reviewed configuration file drives an automated code scan, to give me a reliable overview of features implemented.

And sometimes, someone on the crew needs to just dive in to the code and do the job themselves.

Having stretched the metaphor to breaking point, here’s how it breaks down in terms of some high level requirements:

Above the waterline

• Product decisions - purpose, audience, architecture, capabilities, work schedule
• Must be fast and easy to manually edit, visually structured for easy review
• Full history; avoids ideas being lost or lessons learned being forgotten
• Thru-Water Comms: tools for agents to access additional context on demand.

Below the waterline

• Engineering outputs - code, tests, CI/CD jobs
• Must be in repo: version controlled and lives within the sandbox
• Focused on current and future state; avoids biasing the agent towards past decisions and research
• Sonar: regular scans of code base to populate dashboards or reports for human review

Technical implementation

The context waterline isn’t intended to drive a technical specification so much as give me a frame of reference for when I reviewing the overall state of the project and code base. Nonetheless a few things do fall out as requirements e.g.

• When I’m working on “loose” ideas, I use paper or local files. For better or worse, I’m sensitive to the delay that you get from networking roundtrips, and tools I generally like such as Notion I find annoying. In practice, this means I use Obsidian as my daily driver.
• For important and relatively static diagrams, Mermaid kinda sucks. I need ACME to handle SVG and Typst.
• Set up a separate folder to the project workspace to hold any docs that are useful but might confuse an agent if included in the repo. Agents cannot reliably ignore information while executing a task, so I now keep research docs, design mockups, completed plans in a different location.
• Version control on planning docs etc does not need to be full git version control, file system history is good enough.
• Build tooling to scan code regularly. That is useful for me as the developer to review what capabilities have been delivered and whether they have test coverage, demonstration content, or both. It can also provide information back to the agent e.g. to identify planned features not implemented or not testable.

I like the waterline metaphor because a waterline is more of a reference than a set boundary. A boat can ride high or low in the water, or even be taken out entirely. Similarly, I have bigger projects with some kind of “sonar”, and smaller throwaway projects where it would be wasted. Yet in all cases, I can ask myself “what’s the context waterline on this project?” as a reminder to ensure that I do have docs and tooling appropriate to its scale.

Next steps

The extended metaphor above is probably a fair reflection of how I went in a few circles trying to figure out what would be my ideal system for managing the context of a complex project! Sometimes, the vision is just to ill defined to make real. I did come away with some practical next steps:

• Pause on bi-directional file sync in favour of just adopting a file+folder+Obsidian structure for PRDs, research and brainstorming docs, the capabilities maps, etc. That will move an entire category of documentation out of the repo but still let me experiment and refine.
• Make sure the code annotation and scanning process is well documented. I’ve been using it manually to date, but it has potential to drive a self-improvement process that gives autonomous agents an automated, deterministic process.