Building a backend for context management

When I shifted focus from building out the base platform to reviewing how development had been going, the thing that felt most important to me was improving my context management during coding. I wondered if there was “some kind of sweet spot between feeding an agent harness, the ticketing system (Linear in my case), and having the old “second brain” idea of knowledge graphs and expert systems.”

It seemed like a great side project to build something myself, not because I can build a better Linear but because I could have something that works exactly as I’d like for solo projects. And hopefully, learn some things along the way. I need to drill down into what I really want to build, but my instinct remains to use a knowledge graph as the central concept.

I’ve even got one lying around, from a time when I was diving in to recursive CTEs on Postgres. It may not be the best solution but “just use Postgres” rarely goes wrong for a prototype so I’ll start there and focus on the job to be done.

Problem Statement

Current State

An agentic coding project has several primary sources of context. Each serves a purpose, I don’t see any obvious candidates for deletion. Yet the range and volume of context also means that it is difficult to be sure what context is being used (or should be) to perform any given task.

Common sources of context

Here’s my quick rundown of the most common context in my projects:

• Project specific
  • The prompt given to the agent
    • Plus any context eg a PRD
  • The code repository
    • Source code
    • Guidance
      • AGENTS.md
    • Skills
      • Installed
      • Custom
    • Temporary project files
      • research outputs
      • design mockups
      • code reviews
  • Tickets system eg Linear
• Global
  • Global config eg ~/.claude/CLAUDE.md
  • Skills
    • Installed
    • Custom

Recurring points of failure

• Ticketing via Linear has proven unreliable and complicated. Sometimes MCP needs re-authentication. I’ve found the systems to maintain integrations poorly designed, inconsistent, subject to frequent change. Agents varied between using the issue description, the comment system, and sub-tickets to manage tasks. Overall, the experience has reduced the efficiency gains expected from off-the-shelf-software (and the cost could be used to pay for the VPN services for turning my Mac Studio into a platform server).
• Document management via Linear is difficult. It isn’t easy to search docs, browse content. Agents need to update entire documents at a time, unable to make targeted edits to a file on disk. I’ve had cases where the agent decided not to update issue descriptions (the body of the issue) because of this, and therefore used comments. This results in stale data continuing in the issue description, and being used in future decisions.
• Tracking research is impossible without confusing coding agents. AI agents “always think of an elephant”, in other words they are not capable of ignoring information in their context window. A human-centric approach where it is easy to separate current state from future state from rejected alternatives is not possible. I’ve had multiple cases where, having considered multiple options and clearly stating the way forward, an agent nonetheless greps through the files describing the rejected alternatives and uses rejected concepts in its work.
• Maintaining rules and guidance across multiple projects. While building with a coding agent, I will often encounter repeated cases of bad judgement. This can be done to training inherent to the model that interacts badly with the specific project environment, a bad interpretation of a guideline, or a badly authored guideline interpreted fairly. I’ve yet to find an approach that works for me that balances “global defaults” with project-specific guidance. For instance, a single set of global contexts does not distinguish between throwaway code and a production-grade project.
• Designing, visualising, enforcing the architecture. Coding agents will very quickly write spaghetti code, full of duplicate functions, pointless wrappers and convenience functions, complicated cross-dependencies. Architecture linting rules are one technique for tackling this problem, as they be tested and therefore enforced. Even knowing this, I have found it very easy to let mistakes slip through e.g. when adopting a CLI package to simply ACME’s code, I manually verified that the CLI help syntax worked, but did not catch that the agent was throwing away the CLI package’s parsed command, and re-parsing the entire command line before dispatching the command.
• Spec-driven development is too waterfall-like. While I have no doubt that in practice spec-driven development approaches are flexible in practice (Kiro, GitHub Spec Kit, Tessl), I bump up hard against the way they are described. I like throwaway work that feeds into the project architecture and plan, heavily interactive with the coding agent, with minimal planning up front. Every spec-driven argument always seems to be that the spec must come first, becoming a re-invention of the waterfall model.
• Swapping between agents / harnesses. Even at the most basic level of AGENTS.md vs CLAUDE.md, it feels to me like the competing agents and harnesses want to “own” my context. Judicious use of symlinks might be a simple solution here! Nonetheless it feels like a friction to me to be forced to think about the specific agent, model or harness in use while deciding how to structure and maintain context.
• Folder systems are difficult to maintain. Like a lot of people, I’ve become a big fan of Obsidian and generally using folders of text documents with agents. For instance, using the Claude Agent SDK to do some competitive intel research. This works well for a piece of research, which might involve many documents and websites. I don’t think this scales well to a wide variety of knowledge. It takes a lot of discipline to maintain a consistent folder structure, documents are themselves simply not well suited to indexing, links aren’t easy to label or tag.

Desired State

Business Goals

As an “inhouse personal project” there are no business goals as such, but an equivalent commercial project might aim to measure the following:

• Reduce time spent tidying up bad code. The biggest observable cost, significantly so.
• Increased pace of capabilities delivered. The opportunity cost of “fake” progress is that fewer deliverables get shipped. One technique for covering this up is to ship features and enhancements, but you should be trying to track “real capabilities” rather than individual features.
• Improved ability to predict delivery scope and timescales. High variance between estimates and actuals means that delivery is not living up to promises made. This tends to mean a lots of time us spent resolving internal frictions and keeping frustrated customers on board. That time would be better spent building, planning, selling.

User Goals

• Consistent workflow for defining architecture and project goals
  • Fast-and-easy-to-use UI for day-to-day work and progress tracking
  • Stop using Linear for personal projects
• Improved agent autonomy
  • Agents are consistently able to perform a range of tasks without significant error or re-work
  • CLI tool for agents
• Keep track of the advanced technology landscape
  • AI, Robotics, Quantum Computing, Energy, Space
  • Significant undertakings (Stanford, Mock Machines, ACME) all help build the knowledge base
• Successful, long running “programmes” or knowledge bases for personal projects
  • Mock Machines, ACME
  • Serious hobbies: music, photography, video

What’s next

I once built the start of a knowledge system using Retool and a Supabase backend. It was clunky, but had a graph-shaped core and some nice dynamic schema features. As a first pass, I’m going to take that idea and run two streams in parallel.

For the agents, a CLI tool that can use the backend to add and update Linear-style tickets. For me, a web app, potentially even read-only to start with. A basic ticket tracking system will be a good MVP.