Layering AI within your product

TL;DR It’s all to easy to describe an application as “AI Native” without really saying what that means. Here are six areas, splint into user-facing and developer-facing, where you can look to see if AI is genuinely baked in to a product.

User-facing vs Developer-facing

LLMs are a general purpose technology. That means that it can be used as part of a product, and it can be used to build a product. If only to give some structure to the list, I find it useful to think in terms of user-facing vs developer-facing ways of implementing AI.

User-facing

It was interesting to note when proof-reading this article that two out of three users I was thinking of are agents, not humans! That’s definitely one of the trends: are you building your product so that autonomous agents can discover and use them?

• Providing information that agents can consume as context
• Interfaces that agents can use
• Threading LLMs and workflows into the user experience

Developer-facing

• Training and evals to improve quality over time
• Treating agentic coding as part of the product
• Supporting the deployment and distribution of the product

The initial rush of platform and feature building on Mock Machines was a great experience in leveraging a coding agent (I used Claude) to move quickly. Everything from the core modelling engine to the LLM workflows and APIs to an in-app Typst template editor came together quickly.

Writing close to half a million lines of code (https://jonwalls.dev/blog/benefits-and-weaknesses-of-agentic-coding/) in a couple of months, however, demands some time is taken for reflection. One of the lessons from my time at Rill Data is that it’s all to easy to pay lip service to the idea of being “agentic” or “AI native” while really not changing your approach. I want Mock Machines to be a good showcase of using AI when building a product.

These are my notes on where AI was either most useful building the initial product, or where on review I’ve decided to focus more effort as it evolves.

User-facing

Providing information that agents can consume as context

I tend to believe that feeding the AI good information is the single most consistently impactful thing you can do. In my case of having a tool that generate simulation models with a bespoke file format, the LLM does a better job when it has access to a library of already generated models. That’s partly why I put time in to creating dozens of working scenarios — and it has a compounding effect as you watch the agent reading through past scenarios when creating a new one.

It is challenging though, especially for customer-facing workflows where it is difficult to give an agent the same rich access to information as you can when using an agent that has access to the product development sandbox. Some examples of what I’ve focused efforts on context management:

Enrich the information sent to the LLM. For instance, since my product involves a code editor, I instrumented the internal compiler so that context is always available about where the user’s cursor is, any selected text, the type of object (is it a state transition or a dimension field). This is more effective than sending an entire code file. In the UI, enabling the user to see the context being sent. Especially for technical users, making it easy to understand what items of context is valuable for getting the most out of an agent. Breaking up the workflow to focus on one problem at a time. I’ve found the LLMs struggle to successfully complete an entire scenario generation unless developing locally in a sandbox. So the “build your first scenario” workflow breaks down the process step-by-step, enabling targeted context to be sent with each new step in the flow.

The next steps I’m planning around context are to review the library of sample scenarios, to overhaul the documentation, and to evolve the way I evaluate the workflows. The sample scenarios and documentation need to demonstrate the full range of options available. As the modelling engine becomes more capable, I need the files that an agent is most likely to find to be up-to-date. I also need to spend time evaluating what context gives the best results.

Interfaces that agents can use

It was very important to me that agents were considered first class citizens when building Mock Machines. It should be possible for an agent to perform all the work on behalf of a user, from discovery to execution.

There’s three such interfaces I’ve built out:

• MCP
• API
• CLI

I don’t see any of these as optional. That’s not just about AI, a product with technical users wants all three options anyway. The next thing I’m going to look at here is to spend some time focusing on the “agent journey”. I don’t want to just wave my hands and say, “they can discover the docs and mcp endpoint for themselves”. I want to be confident that an autonomous agent can complete an end-to-end journey that does something useful.

Given the debate I see online about which of these is your best choice, I think we some time away from a common understanding on what “agent-first” looks like. Here’s my personal experience: CLI is generally more reliable than MCP, in that I have a pretty reliable connection to GitHub via gh but not Linear via their MCP server. I have not examined this closely, but my impression is that the agents are great at composing CLI commands but either they or their harness are not so great at knowing what MCP tools are available.

Direct access to APIs is generally less immediately useful than tooling via MCP or CLI, one reason being that the sheer size of an OpenAPI spec can be too much context. However since APIs are easy enough to wrap in an SDK, I am curious whether there is a world in which some applications are better served by agents writing their own tools based on an API spec or using an SDK. When I wrote a Python SDK for Rill Data, the most useful thing I added beyond the raw API spec was the query builder. Factory methods are much easier for coding than directly constructing a complex JSON object.

Many agentic flows and tool calls do not want direct access to an API so much as some degree of baked-in orchestration to achieve a particular task. My instinct is that the is an agentic equivalent of progressive disclosure in app design, where a raw API is available but most agents and tasks are solved by higher level tool calls. It’s an interesting thought that maybe agents need both the “pre-baked” MCP server or CLI tool, and also an API or SDK by which they can construct their own tools to suit their needs.

Threading LLMs and workflows into the user experience

The chatbot user experience is a good default, but thinking “AI = chatbot” is not thinking deeply enough. People have now encountered enough “old product with a chatbot slapped on top” examples that we’re all developing some discernment for what constitutes good design.

I’m enjoying experimenting with this. Rather than stating any conclusions, here are ideas I’ve implemented that have felt useful to me as a user:

• Rich, interactive context e.g. if there is a validation error, the user can click on the message to be taken to that line in the code, and a “Fix” button will automatically propose a fix.
• For the “Call To Action” on the website, give the user a visual, information rich feedback of every step of the scenario generation process. Not just a spinning logo.
• Adding voice and speech synthesis to the chat interface.
• Having separate “Edit” and “Ask” tabs. It might be a personal thing, but I have often been frustrated while working when an agent has decided to make edits while I was just reviewing some code.

For my taste, I see a lot of products exposing their AI features to the user in a way that demands attention, which can be distracting. If the workflows are threaded in to the fabric of the product, then the goal isn’t demanding attention but rather exposing the right functionality at the right point in the user’s workflow.

As one example of a capability I think my product needs, it should go beyond validating a simulation scenario, and offer suggestions for improvement. For instance, if simulating a population of some kind, like customers of an online store, there are tools for spawning and end entities that a user might not think to look for. I get the impression many users are tuning out the “Ask AI” buttons that are appearing everywhere, I’m curious what design patterns will prove to be the best way to expose advice and suggestions.

This is particularly important for what I’m building because I know that building a simulation to generate data is not something that most people are going to take the time to do. It’s too time consuming to get to an initial result. Generative AI solves that problem, as long as it is easy to apply at every point of friction.

Developer-facing

Training and evals to improve quality over time

During my initial build, I regularly did quick comparison exercises comparing the outcomes of using different models, context, workflows. Simple things like adding a model switch into the UI let’s you do fast, manual testing. These are all practical choices when there isn’t any baselines or infrastructure to work with yet.

Now that I do have a working product, there’s a clear need to evolve. In fact, I expect that the biggest code change will be an overhaul of the backend that includes capturing the execution traces. There’s a number of areas where I expect to spend time:

• Trace capture
• Test datasets
• Context management
• Workflow design
• Model selection

A particular concern I have is finding the best level of granularity to work at. Modelling scenarios in Mock Machines can get quite complex, and my “all options included” generation workflow has a high failure rate. The agent is trying to construct a large simulation model but struggles to pass all the validation rules. Simply put, it has not been so easy translating what works in a local IDE with access to the entire code base, into the workflow execution on the cloud platform.

Building a reliable flow wasn’t difficult at all: the website’s “try me!” call-to-action works every time. But there’s a tradeoff: a constrained workflow that works reliability, versus a more open ended approach that can take advantage of the full capabilities of the modelling engine. Good evals should give me confidence that I’m getting the right balance of flexibility vs. reliability.

Treating agentic coding as part of the product

As I started ramping up on this project, I wrote up some notes on how I’d set up myself up for agentic coding (https://jonwalls.dev/coding). My focus at the time as things like configuring the local project rules and skills versus the global configuration for the agent, having a consistent approach to using Linear and GitHub. My framing at the time could be described as “this is how I do agentic coding”.

The biggest shift of perspective that I’ve made almost no changes to my global configuration for Claude or Codex, instead preferring to focus entirely within the local context of the project. Some things that have been working well for me:

Code quality tests and architecture linting. The worst code cleanups I’ve experienced, in terms of time lost and grey hairs earned, have been the spaghetti code from agents solving problems without boundaries. I’ve never personally used automated tooling to enforce import rules between different parts of a code base, but have become huge advocates of them – coding agents simply do not have (yet) the instincts of a developer to keep code well organised.

Adding agent skills as I go to automate tasks. Running the test servers, writing new content, updating the modelling syntax. I see a lot of other people doing this. Obvious perhaps, but a shift from originally thinking “I’ll build a library of skills I can re-use” to “aiming for re-use is a distraction from building this product better.”

Keeping research docs and mockup designs outside of the main repo. This is also not what I aimed initially. Any project I started from about Oct 2025 to June 2026 had a folder dedicated to organising all my planning docs for a project. The concept made sense to me (still does), but the agents would just fairly regularly make a decision based on a rejected idea in an old doc it had found.

That last point touches on the biggest opportunity I see for significant improvement: figuring out how I like to do context management for coding. There is a lot of work going on with agent harnesses and so on, but I think I’ll treat this as an entire side project in its own right. I wonder if there is some kind of sweet spot between feeding an agent harness, the ticketing system (Linear in my case), and having the old “second brain” idea of knowledge graphs and expert systems.

Supporting the deployment and distribution of the product

Strictly speaking, this is about fairly traditional, deterministic DevOps or CI/CD workflows. The general theme of my notes has been about agent-specific developments. I wanted to include it though because LLMs have made it so much faster to get environments stood up. A test deployment to GCP Cloud Run, for instance, including getting the agent to do a full shell script for reference and repeatability, is vastly superior to doing manual setup via the console.

It has also been useful performing cross-language tasks such as creating a Python module using the CAPI interface and publishing to PyPI, or handling the Rust crate for the document editor. Task switching between language ecosystems is mentally taxing, for me at least, so being able to delegate these distribution challenges is wonderful.

If working with a full team, arguably the value here would diminish or disappear – at scale I’d certainly want to have a DevOps or SRE team with good security instincts than worry a developer has missed a security risk with handling credentials and tokens.

For my purposes, however, I now typically have a “tools” directory that stores scripts and code for deployments, benchmarking, code review, project reports.