Five weeks ralph-looping a document extractiona and review tool

Contents

• §0 — TLDR
• §1 — HOW IT STARTED · JUN → JUL 7
• §2 — RESEARCH THE THING · JUL 9–14
• §3 — THE INITIAL PIPELINE · JUL 7–10
• §4 — THE FRONT DOOR AND THE LOCK · JUL 15–25
• §5 — THE LIFE OF /ralph-arch · JUL 15–29
• §6 — THE THREE-LAYER DOCUMENT MODEL
• §7 — THE REFACTOR LANE · JUL 18
• §8 — /ralph-retro. MEASURE ITSELF, MEASURE ME · JUL 26–27
• §9 — /ralph-design · JUL 25–27
• §10 — SCRIPTS
• §11 — ADDING PI AND QWEN3.8 · JUL 31 → AUG 5
• §12 — FINAL PRODUCT VERSION
• §13 — NEXT STEPS
• §14 — WHAT WENT GREAT, AND WHAT I WOULD DO DIFFERENTLY
• §15 — CLOSING

§0 TLDR

36 days · 1,699 commits · 152 issues · 211 locked architecture decisions · 22 skills

§1 HOW IT STARTED · JUN → JUL 7

Me and a friend that I’ve been working with were investigating a venture together, and he is in the real estate market. I have some interest here, and I’m a tech person, so we thought the goal was to investigate this.

Our main thesis was that leases are the main driver of value for real estate companies through profit on a lease, through cost of capital, such as the value of real estate. It’s also very suitable data, so we thought that maybe we can do some AI magic here. We started talking and looking around and got some starting hypotheses, and then we built some prototypes. The first one was just a UI prototype or a very simple prototype to show what we are doing here and what we are thinking, which aligned us.

And then, based on this, we got some ideas of what we were going to build. Now the next step was to do customer discovery interviews, but unfortunately it was vacation month in Sweden. During this time, however, I thought that it would be great to build something, and probably it would be lowest risk to build something around leases. I wanted to take ralph-loops a step further, do something bigger with it, and kind of prepare for hopefully what was to come. The goal was to build something agnostic, so not really lock into too much, just set the general good architecture that we could extend upon later.

I did also build an initial brief prototype to test some techniques on extraction and find out that, in order to extract things, Opus, PyMuPDF, and the re-rank with Haiku got really good extraction and grounding. I could rule out some purely deterministic alternatives.

Repo started 2026-07-07 with Initial commit: ai-infra-template (pristine, pre-customization) — the template extracted from the Klara build in the last post.

§2 RESEARCH THE THING · JUL 9–14

One big lesson from the other project was that there are going to be a lot of questions or a lot of interaction around architecture. If you want to remove assumptions from the AI, you need to deliver the assumptions yourself. I knew that my gap here is that I have never built something of this scale, especially with the constraints that I had. I wanted it to eventually be able to have:

• Document store. The original PDFs, kept as they came in.
• Mining. Getting the text out of the leases, scanned or not, and having the AI pull out the fields (rent, dates, parties) with a pointer to where on the page each one came from.
• Review. A person checks whatever the AI wasn’t sure about.
• A database. Leases, buildings, and everything that’s been checked.
• Tools. Calculations built for what each customer needs, like rent indexation.
• Chat. Asking questions about your own portfolio.
• Inference. Calling the AI models, without locking into one of them.
• Where it runs. US, EU, or our own servers, depending on where the customer’s data is allowed to live.
• Auth. Logins, and who gets to see what.
• Collaboration. Owners and property managers working on the same buildings, across companies.
• Invoicing. Plugging into whatever the customer already runs: Vitec, Fortnox, Peppol e-invoices.
• Training. Maybe, later, our own models fine-tuned on corrected data.

All of this, developed step by step, without painting me into a corner. We ended on a Docker in Docker setup, one .yaml to rule them all, and each module neatly containerized so each ralph-loop-vm could develop independently.

I also wanted to be able to deploy this in my current Docker setup, if possible, so that the agents could work. I wanted it all nicely compartmentalized and modularized, but I also wanted agents to be able to work independently and autonomously on this. With that kind of thought in mind, and also knowing how to connect invoicing and many other questions, Fable was just temporarily released and I pointed it at these topics for a few days straight with a coherent research plan. You can read about the research that I did for that in this post here.

And the result of this for this project has actually been really great. The research pattern was also useful throughout. I saved it as a how-to-research.md, with a plan template, and instructions to grill me. Whenever I needed referenceable deep research on a topic I wanted to explore in the codebase, or an issue I had, I used it to great effect. For example:

• Jul 23, decision escalation → the /decision-escalation skill, same day
• Jul 25–27, UI-driven feature pipeline → /ralph-design, same day
• Jul 27, how to speed up the loop → flow instrumentation + /ralph-loop-analyze, same day
• Jul 28–30, first-mate orchestration → ralph/decision-ui, same day

§3 THE INITIAL PIPELINE · JUL 7–10

Now, I did a previous project in the Klara project, and I knew there were things that I liked and I disliked. Some things I didn’t like were the drift in the double user story list, for example. It was also a much smaller project where I wouldn’t have the same need for the role of a product manager that would kind of drive a backlog of things, etc. It contained nearly all of the complexity that this project would entail, so I started with the smallest slice of that version of things that I knew I liked:

• ralph-loop.sh (the driver)
• ralph-loop-pick-gh-user-story.sh (pick + claim)
• /ralph-prio (stock the backlog)
• /ralph-create-issues (grill → issues)
• /ralph-tdd-plan + /ralph-tdd (plan, then build, in an autonomous loop)
• ralph-loop-done.sh (the done gate — it closes #N only after a pushed commit referencing #N and a green ./scripts/run-tests.sh, then releases the claim and SIGKILLs its own session)

Also in from day one: ralph-loop-plan-done.sh, ralph-fetch-issue.sh, ralph-find-claimed-issue.sh, ralph-issue-drift.sh, and the two PROMPT-*.md templates.

flowchart TD
    classDef human   fill:#fde2b3,stroke:#d98c00,color:#5a3d00
    classDef system  fill:#cfe3ff,stroke:#2f6fd0,color:#0b2c5c
    classDef trigger fill:#eeeeee,stroke:#888888,color:#333333
    classDef done    fill:#d9f2d9,stroke:#3a9c3a,color:#14501a
    classDef header  fill:#eaeaea,stroke:#bbbbbb,color:#111111

    subgraph AUTH[" "]
        direction TB
        Ahdr["My machine — human in the loop"]:::header
        Ahdr --> A1(["Ask for the next stories to build"]):::human
        A1 --> A2[["/ralph-prio<br>research the specs, rank the stories<br>→ user-story-list.md"]]:::human
        A2 --> A4[["/ralph-create-issues<br>thorough research and grill session to align"]]:::human
        A4 --> A5["GitHub issue #N created<br>label: agent-ready"]:::trigger
    end

    subgraph VM[" "]
        direction TB
        Vhdr["Loop VM — no human present"]:::header
        Vhdr --> L1[/"./ralph-loop.sh"/]:::system
        L1 --> PICK[/"ralph-loop-pick-gh-user-story.sh<br>find current claim or claim new issue #N"/]:::system
        PICK --> D{"current-plan.md exists?"}:::trigger
        D -- "NO" --> NOPLAN["inject #N into PROMPT-plan.md<br>run claude"]:::system
        NOPLAN --> PS[["/ralph-tdd-plan<br>→ current-plan.md"]]:::system
        PS --> PD[/"ralph-loop-plan-done.sh<br>delete subagent reports and SIGKILL"/]:::system
        PD --> PICK
        D -- "YES" --> HASPLAN["inject #N into PROMPT-tdd.md<br>run claude"]:::system
        HASPLAN --> TS[["/ralph-tdd<br>red-green-refactor + e2e"]]:::system
        TS --> TD[/"ralph-loop-done.sh<br>tests green? close #N · unclaim · SIGKILL"/]:::system
        TS --> TO["TIMEOUT — SIGKILL"]:::trigger
        TO --> PICK
        TD --> PICK
    end

    A5 ~~~ Vhdr

One user story, start to finish. Amber = human in the loop, blue = the loop on its own. Rounded = me asking, double-bordered = a skill invocation, slanted = a bash script. Each iteration runs a fresh claude session under a hard timeout and gets SIGKILLed — the claim label and current-plan.md are the only memory in between.

The main principles were:

1. Fresh context per iteration. Plan and build are deliberately split across two iterations so the build never starts in a context already filled by the planning fan-out. Durability lives in git history, the plan file, and the GitHub claim label — not in a session.
2. The issue is the spec. A fresh-context iteration builds exactly what the issue pins down and invents the rest, which is why /ralph-create-issues records answers verbatim.
3. Green-gated advance. An issue can’t close without both a pushed commit referencing #N and a green ./scripts/run-tests.sh.
4. Loop skills delegate and check, never implement — plan fans out and reconciles, tdd implements, subagents do the heavy lifting

Bootstrap commit: 2026-07-10 Bootstrap P0.0 (#1): architecture spec + backend skeleton + loop scripts.

§4 THE FRONT DOOR AND THE LOCK · JUL 15–25

Alright, that’s kind of where the work was at start. I also knew that, with the research, I had front-loaded a lot of the architecture, so I need to incorporate this somehow. I also knew that I need to keep track of all of the architecture decisions because it wasn’t complete. It needed to be found more easily than by mining this huge research folder ad hoc so it could be referenced. I knew I needed something there, and it’s also the role of a product engineer: what am I going to build right now, and how do I build it?

For this, I made three main skills:

1. /ralph-product-planning, which took the role of a product manager. It created milestones with startup de-risking skill through a conversation.
2. /ralph-find-user-stories, which, through an interview and repo research, created the five next user stories.
3. /ralph-arch, which was repurposed to just mine these five stories, cross referenced with the previous architecture planning, and figure out decisions to make and to lock them down.

/ralph-arch was created to mine the research and to make sure that all of the decisions that needed to be locked were locked. It could take easy, obvious decisions itself and ask me about things that had both pros and cons. /ralph-arch would lock these decisions in a decision map in a folder structure. The architecture didn’t really look like I planned out the software map with trees and file trees and everything like that. It was more about how things were referenced to each other. Supposedly, this worked well for agents. For me, it’s hard to read this one, I will admit, but this is a bunch of decisions that have to do with architecture. They reference each other, supersede each other, and retire each other if needed. Only appendable, never editable.

Every user story row also got a flag: - ***Arch:*** ⬜ pending, written by /ralph-prio when it files the story. /ralph-arch is what flips it — to ✅ <D-refs> · verified <date>, where the D-refs span every module the story touches. The flag is the gate: /ralph-create-issues refuses to create an issue for any story whose row still reads ⬜ pending, and /ralph-arch itself refuses a track that still has ***Design:*** ⬜ pending, so an arched story necessarily has its design resolved too.

/ralph-arch-missing-stop was added and an instruction to the implementation skills to call it if an architecture gap was ever found. It diagnoses, posts a comment, removes the agent-ready label, and sigkills itself and the loop moves on. It never guesses.

flowchart TD
    classDef human   fill:#fde2b3,stroke:#d98c00,color:#5a3d00
    classDef system  fill:#cfe3ff,stroke:#2f6fd0,color:#0b2c5c
    classDef trigger fill:#eeeeee,stroke:#888888,color:#333333
    classDef done    fill:#d9f2d9,stroke:#3a9c3a,color:#14501a
    classDef header  fill:#eaeaea,stroke:#bbbbbb,color:#111111

    subgraph AUTH[" "]
        direction TB
        Ahdr["My machine — human in the loop"]:::header
        Ahdr --> A0[["/ralph-product-planning<br>milestones, riskiest assumption first<br>→ product-plan.md"]]:::human
        A0 --> A1[["/ralph-find-user-stories<br>interview + repo research<br>→ the next ~5 stories"]]:::human
        A1 --> A2[["/ralph-prio<br>dedup, rank, file the story<br>Arch: ⬜ pending"]]:::human
        A2 --> A3[["/ralph-arch<br>mine the research, grill the open choices<br>lock decisions → Arch: ✅"]]:::human
        A3 --> A4[["/ralph-create-issues<br>refuses any story still Arch: ⬜ pending"]]:::human
        A4 --> A5["GitHub issue #N created<br>label: agent-ready"]:::trigger
    end

    subgraph VM[" "]
        direction TB
        Vhdr["Loop VM — no human present"]:::header
        Vhdr --> L1[/"./ralph-loop.sh"/]:::system
        L1 --> PICK[/"ralph-loop-pick-gh-user-story.sh<br>find current claim or claim new issue #N"/]:::system
        PICK --> D{"current-plan.md exists?"}:::trigger
        D -- "NO" --> NOPLAN["inject #N into PROMPT-plan.md<br>run claude"]:::system
        NOPLAN --> PS[["/ralph-tdd-plan<br>→ current-plan.md"]]:::system
        PS --> PD[/"ralph-loop-plan-done.sh<br>delete subagent reports and SIGKILL"/]:::system
        PD --> PICK
        D -- "YES" --> HASPLAN["inject #N into PROMPT-tdd.md<br>run claude"]:::system
        HASPLAN --> TS[["/ralph-tdd<br>red-green-refactor + e2e"]]:::system
        TS --> TD[/"ralph-loop-done.sh<br>tests green? close #N · unclaim · SIGKILL"/]:::system
        TS --> TO["TIMEOUT — SIGKILL"]:::trigger
        TO --> PICK
        TD --> PICK
        PS -. "architecture gap" .-> MS
        TS -. "architecture gap" .-> MS
        MS[["/ralph-arch-missing-stop<br>comment the gap · remove agent-ready · SIGKILL<br>→ back to me in /ralph-arch"]]:::trigger
        MS --> PICK
    end

    A5 ~~~ Vhdr

Same pipeline, with the lock’s other half: if /ralph-tdd-plan or /ralph-tdd hits an architecture question no locked decision covers, it parks the issue and hands it back to /ralph-arch instead of guessing. Amber = human in the loop, blue = the loop on its own, dashed = the escape hatch.

§5 — THE LIFE OF /ralph-arch · JUL 15–29

• Jul 15 — born. One architecture file became a folder that grows per module and /ralph-arch appeared between /ralph-prio and /ralph-create-issues: mine the research, grill every open choice, append dated decisions, then run a fresh-context adversarial pass per story and tick the story’s flag. The invariant was written down on day one: the headless loop never decides architecture.
• Jul 15, later the same day — went cross-module. Where a story is filed got separated from which modules’ architecture it locks; the story’s ✅ line, whose decision refs span several module files, became the completeness gate.
• Jul 16 — the store restructured so specs stop bloating. Each module file became a folder: an overview plus one immutable file per track.
• Jul 16 — it started committing and pushing its own work instead of deferring git to a phantom orchestrator that didn’t exist.
• Jul 17 — it learned to answer its own questions before asking me. After two grill questions the research already answered, every choice now has to come back in one of three tiers — settled by research, settled by the backlog, genuinely open — with a citation, and only the genuinely open ones reach me. New hard rule: grill only what the evidence leaves open.
• Jul 17 — open forks come with options, pros/cons and downstream impact before the recommendation, not as a bare one-line question I have to interrogate.
• Jul 25 — concurrency day. Two /ralph-arch runs drew the same next-free decision number and the first push silently overwrote the other’s decisions. The fix: a git-based claim lock at the top of each track file, and decision IDs namespaced per track so parallel runs can’t collide.
• Jul 26 — it started logging what it costs me. A close-out step logs every ask to the retro ledger — which axis it was on, what happened to it, and whether the agent could have answered it itself.
• Jul 27 — milestone gate. It now arches the current milestone only — after 20+ corner-case issues (~77 founder-minutes of interview on one track pair) got authored and built while the milestone gate itself went un-run.
• Jul 27 — it refuses un-designed tracks and mines the design-node files as evidence, the same way it mines research.
• Jul 28 — decisions land at close-out, not mid-sitting. Now the sitting drafts into a scratch file, the pass edits drafts, and IDs are minted once, on my explicit “lock it”. Immutability starts at the write.
• Jul 28 — one stop removed. It picks the lowest eligible track itself and announces the touched modules instead of asking me to confirm them — the adversarial pass already catches a missed module, so the confirmation round-trip cost a stop and added no coverage.
• Jul 28 — the adversarial pass asks “can these both be true?” It now also receives the already-locked decisions and hunts for pairs that contradict — after one sitting locked a contradiction twenty lines apart in the same file.
• Jul 29 — “a cite is not a reading.” Subagent claims were being trusted from a file:line pointer without anyone opening the file — the ledger’s highest override rate, 7 of 9 asks overridden. Now the orchestrator must open the cite before it becomes an ask.
• Jul 29 — overrides declared on both ends. A decision that supersedes an older one writes a trailer on itself and a warning band on top of the thing it overrode, enforced by a checker in the green gate.

As you can see /ralph-arch was under constant work. This was mostly based to the somewhat boring nature of this work. Sitting in the terminal, reading through it’s decisions and suggestions, was not rewarding work, so it was very motivating to streamline it.

§6 THE THREE-LAYER DOCUMENT MODEL

Besides the actual code, there was three layers of information we kept pointing to in our skills and issues, when needed:

• workflow/research/** — exploration agents mine for reference, and where the high-level architecture was decided
• AI-Info/architecture-spec/ — the locked architecture decision points
• AI-Info/software-architecture/ — the as-built map, so agents quickly can navigate the codebase

The research was done when needed, starting with the initial architecture research, but many research runs were done pre-milestone creation. /ralph-arch read and extended architecture-spec, which was referenced when creating issues in /ralph-create-issues and also by /ralph-tdd-plan /ralph-tdd had instructions before closing any issue: extend the software-architecture with the new files. It is basically a huge file tree, per module, where it prints out the current lay of the land of the software. Architecture-spec had decisions, software-architecture had the ongoing results of these decisions. The goal was to give each issue more surgical access to relevant files, without relying on grep, glob or find, as it started to bloat context when looking for information.

§7 THE REFACTOR LANE · JUL 18

Another thing that happened in the previous project that I wanted to add as a kind of safety mechanism, I guess, was to add an automatic refactor script. It’s called ralph-refactor-scan.sh. There was a refactor scan that was run after every issue was done, and when the loop was done, this script checked if there was any file containing above 450 non-blank lines.

If so, it opens an agent-ready issue that explains the problem and what to do. When the ralph-loop picks an issue with the refactor label, it puts in a different prompt and forks off to a refactor thing. The only thing it does is to make sure that unit tests are solid and end-to-end pass. It does an end-to-end pass before refactoring, and then it does an end-to-end pass after and verifies the result. If both of these checks pass, then it’s a go.

It tries to make sure that the user journey is the same and the unit tests are the same, especially for refactors. It has only fired five times, which I account to the improved architecture work of this loop.

§8 — /ralph-retro. MEASURE ITSELF, MEASURE ME · JUL 26–27

In previous projects, and also from real-life sprints and stuff, I know that retros are good to evaluate things. I was thinking, “Okay, how can I add this continuous improvement to my workflow?” When I started evaluating this, it was clear that having numbers to base decisions on was needed. Pretty quickly, I made a retro skill and during the first run it suggested I needed numbers, so logging was added:

• the subagent runs of all the things I did
• questions and answers from /ralph-arch
• how much time was spent on founder questions or on interviews from escalation stuff
• what type of runs were aborted in the ralph-loop and how long they were running

This was actually super useful, as one can imagine when one thinks about it. I started seeing outliers in certain things: things that took a lot of time, things that took a lot of effort. For example, certain runs in certain decisions were killed very often, and when they were killed, they were often killed multiple times before they succeeded.

Of the 26 issues in the measured window, 20 took at least one killed iteration and 10 took two or more — and all 26 still passed their done-gate in the end. 31.6% of all iterations ended by the clock rather than by choice, and 47.3% of all iteration-seconds sat inside an iteration that ended in a kill.

When I implement these things in the /ralph-retro, I created hypotheses of how this would turn out to track if it was effective or not, from a lesson in a previous project. I created this KPI, which I believe will be impacted, and I try not to do more than one thing per KPI in each retro to see if my hypothesis would fall out. With this, I wrote some scripts with minimal additions. I wrote some scripts for helper stuff. For example, in step 7 in /ralph-arch, it said: call ralph-retro.sh --instructions, and then in there, it would print all instructions at that time. Even if it was a long-running arch session, at the end, when it did the second-to-last step before committing, it would get fresh instructions on how to do the next step. I also wrote a ralph_flow.py script that would generate numbers since the last run, so I didn’t have to manually crunch the numbers every time. I could just run one script that would get the relevant numbers from since the last /ralph-retro and compare it.

The things that I did here, for example, were to create specific subagents for different tasks, because I wanted to have more model observation and better logging. I created agents for each task in the /ralph-tdd loop. I also added this ralph-research-worker agent that I could reuse both in the TDD stuff and for /ralph-arch.

flowchart TD
    classDef human   fill:#fde2b3,stroke:#d98c00,color:#5a3d00
    classDef system  fill:#cfe3ff,stroke:#2f6fd0,color:#0b2c5c
    classDef trigger fill:#eeeeee,stroke:#888888,color:#333333
    classDef done    fill:#d9f2d9,stroke:#3a9c3a,color:#14501a

    A["Authoring skills log every ask at close-out"]:::system --> L1["Ask ledger — 32 sittings · 134 asks · 91 reversals"]:::done
    B["The loop logs every event, best-effort — logging can never fail the loop"]:::system --> L2["Flow logs — 3,223 events"]:::done
    L2 --> F["ralph_flow.py counts — one comparable snapshot row per run"]:::system
    L1 --> R[["/ralph-retro judges"]]:::human
    F --> AN[["/ralph-loop-analyze judges"]]:::human
    R --> C["Improvement catalogue — append-only, frozen on write, one change per KPI"]:::done
    AN --> C
    C --> V{"Next run — did the named metric move?"}:::trigger
    V -- "3 worked · 1 no effect · 1 superseded · 5 open" --> C

The measurement loop: the skills and the loop write ledgers, scripts count, two skills judge — and every change carries a falsifiable prediction scored on the next run.

Of the 134 asks: 90 accepted as recommended (67%), 26 overridden (19%), 10 needed a second round-trip, 7 accepted with modification. The ledger also carries a confession field — could the agent have answered this itself? — and on 47 of the 134 it names the exact file that already contained the answer. A third of my interruptions were avoidable, by the agent’s own account. The load was also front-heavy: 974 of the 1,238 founder-minutes (79%) landed in the first three days, and 58 of the 91 reversals happened on Jul 28 alone.

What I found, specifically

• I added functionality to ralph-loop.sh so that if a subagent is running, it waits for it to finish before kill
• BUT, if it’s the last e2e agent, it stops and waits for it to commit and run the test suite (full tests take about 30s)
• It passed — 0 killed build workers over 72 dispatches after it landed, versus 10 per day before
• The plan cap sat inside the working distribution — p85 at 94% of the cap — so every kill threw away a finished ~10-agent research fan-out that the next iteration then re-derived (~40% of the 116 research workers were re-runs). I raised it 1.5× → 4× as a pure hang backstop. Passed: one fire in the next 11 plan iterations, at 6,187 s
• Measuring the startup tax — is the dead time before a first commit real, or an artifact? Should I split into shorter issues? Rejected through measurement
• Demoting the retrieval sweeps to Sonnet and keeping only the judgment layer on Opus: agent-hours per planned issue fell 1.29 → 1.02

I saved everything in a shared.jsonl ledger, which also got saved logs from every session where it needed answers from me, as that’s where a lot of my time was spent (mainly /ralph-arch and /ralph-create-issues). I felt that there are oftentimes answers that can be taken easily if it’s framed in the right way. When I write the right follow-up questions, the AI can easily see, based on the answers it gives, what the best recommendation is, and I needed data to see how to design a solution to this.

I used this in order to tweak the /decision-escalation skill and the instructions for how it should present questions and what kind of information it should fetch beforehand.

§9 — /ralph-design · JUL 25–27

Around this time, it was also clear that the UI design is an important aspect. Often times, as many people know, once you start implementing the UI, you realize that the spec is not complete. In addition, when a non-ui-focused AI wants to make UI decisions, I feel it often makes the pretty bad ones. These two facts in combination sometimes gives crappy results.

This thought, coupled with the fact that I was thinking my partner needs to be able to help impact a product (since I don’t have any industry experience and since my partner will do most customer meetings in the future as well), led me to think there needs to be a way to better include the UI design before the TDD loop implements it. There are a lot of assumptions that happen there and a lot of things that arise at that point.

I used my fantastic research instructions to figure out what some possible solutions to this are. I think the main criterion that things fell on was the synchronization back and forth between projects. There are a lot of products out there, like Lovable, base44, Claude Design, etc., but all of them failed at syncing between the design tool and the product.

What I did instead was to set up a flow where /ralph-design checks out a visual prototype branch, which I or my co-founder can sit in. In there, we use a design skill that basically just does frontend and is not allowed to do any backend instructions whatsoever. It can be used both for:

• design brainstorming
• any tasks that need design

I added a flag to all of the issues where it says ***Design:*** ⬜ pending, and if the issue needs any type of design, it also needs a design pass beforehand. Similarly to the decisions in the architecture, these design decisions are then locked down.

flowchart TD
    classDef human   fill:#fde2b3,stroke:#d98c00,color:#5a3d00
    classDef system  fill:#cfe3ff,stroke:#2f6fd0,color:#0b2c5c
    classDef trigger fill:#eeeeee,stroke:#888888,color:#333333
    classDef done    fill:#d9f2d9,stroke:#3a9c3a,color:#14501a
    classDef header  fill:#eaeaea,stroke:#bbbbbb,color:#111111

    subgraph AUTH[" "]
        direction TB
        Ahdr["Authoring — human present"]:::header
        Ahdr --> PP[["/ralph-product-planning"]]:::human
        PP --> US[["/ralph-find-user-stories"]]:::human
        US --> PR[["/ralph-prio"]]:::human
        PR --> DG{"Story touches UI?"}:::trigger
        DG -- "Yes — flagged Design: ⬜ pending" --> DS[["/ralph-design — live on a throwaway proto branch"]]:::human
        DS --> DN["Design decisions locked in design-nodes/"]:::done
        DN --> AR[["/ralph-arch"]]:::human
        DG -- No --> AR
        AR --> CI[["/ralph-create-issues"]]:::human
    end

    subgraph LOOP[" "]
        direction TB
        Lhdr["The loop — no human"]:::header
        Lhdr --> TP[["/ralph-tdd-plan"]]:::system
        TP --> TDD[["/ralph-tdd"]]:::system
        TDD --> DONE["Done gate — green tests + commit referencing #N"]:::done
    end

    CI -- "agent-ready" --> TP

The design gate: a story that touches UI gets a /ralph-design sitting on a throwaway branch before /ralph-arch will take it. Amber = human in the loop, blue = the loop on its own, green = locked or done.

§10 SCRIPTS

Throughout, I always try to identify what type of deterministic scripts to run to help the AI not fuck up, and the most important ones are around the ralph-loop. There’s a lot about how it works, but I also added some scripts for it to figure out what to do next. I also added, for example, planning-status.sh, where it kind of deterministically checks where each issue is in the pipeline. It fetches both GitHub issues and things in the milestone user stories that are preliminary user stories that are prioed, the ones that are waiting for design, arch, etc. This helped it figure out each skill what to work on next without me needing to say it, I could just call each skill and they got told to take next in line if no user input was specified.

planning-status.sh — every story’s place in the pipeline and which skill it is waiting on, computed deterministically from GitHub and the planning docs. Here the queues are drained: M1–M4 shipped, M5 up next.

A few other ones worth mentioning:

• track-claim.sh — a lock built out of git, so I could run /ralph-arch, /ralph-design and /ralph-create-issues in several clones at once without them stepping on each other. Each track file has a claim line at the top, and if two runs grab the same line the loser’s push conflicts and it just aborts.
• check-decision-edges.py --trace — hunts down a decision instead of grepping for it. Every locked decision is one long line, ~2.6 KB on average, so a grep for T19-D2 dumped 57 KB of paragraphs into context, while --trace gives back just the definition in 3.1 KB and can walk what overrode it from there.
• docker-vm-janitor.sh — cleans up the Docker mess the loop leaves behind, once per closed issue. Every iteration ends in a SIGKILL so agents never get to clean up after themselves, and they also spin up stuff nobody asked for — on one VM I found six stray Postgres containers just sitting there.

§11 ADDING PI AND QWEN3.8 · JUL 31 → AUG 5

So, around this time, when the agents started chugging, I was starting to reach my weekly usage limits, pretty quickly, and wanted to add Pi because I also want to try out some different models. I had this research already made up in the research folder, and I went through a port here.

I analyzed what’s needed to run the ralph-loop, made a plan, and just let it iterate. It wasn’t more complicated than that. It took a bit of time, like a day for the port, but in the end everything worked as expected.

The goal was to be able to start the docker image in the same way as before, and type pi, and have everything in feature parity without chance for drift. To achieve this about 10 extensions were written, including root-traversal CLAUDE.md reading, statusline, subagents, etc.

When trying it out, it was instantly a bit different. For example, the OpenAI plans don’t show thinking in the same way as Anthropic. When Qwen came out, Qwen 3.8, it was really, really cool to try out. It had the same thinking trace so you could see and adjust it. But after analyzing a full issue throughput, it did not hit the mark. Specifically, between the consolidated output, and the final issue plan, mistakes got made. When those mistakes later were uncovered by the ralph-tdd-worker it chose to make its OWN decisions around it where it should clearly have invoked /ralph-arch-missing-stop and parked the issue.

It was analyzed and maybe it was a bit in the wording that could, in some way, be interpreted to be a liiiittle bit ambiguous, but not really. Claude models had never had a problem with it, the Qwen model did, so it kind of went off the rails and started to take its own architecture decisions inside the ralph-tdd-worker work. As such, I used it mostly for other things, like research, experiments, changes to the loop, etc, and kept the main loop running in Claude Code.

§12 FINAL PRODUCT VERSION

One can upload leases in different shapes and formats, have it extract data, review where the data comes from, and get an overview of the leases. It’s easily extendable with more lease versions, chat, different models, and data to present. You can check it in motion here: the demo on Google Drive

This is how the system design ended up looking:

flowchart TD
    classDef human   fill:#fde2b3,stroke:#d98c00,color:#5a3d00
    classDef system  fill:#cfe3ff,stroke:#2f6fd0,color:#0b2c5c
    classDef trigger fill:#eeeeee,stroke:#888888,color:#333333
    classDef done    fill:#d9f2d9,stroke:#3a9c3a,color:#14501a
    classDef header  fill:#eaeaea,stroke:#bbbbbb,color:#111111

    U(["A user, in the browser"]):::human

    subgraph VM[" "]
        Vhdr["One VM · Docker Compose · six containers"]:::header
        FE["<b>frontend</b><br>React + Vite<br>upload · review & correct · portfolio"]:::system
        PLAT["<b>platform API</b><br>FastAPI<br>documents · corrections · portfolio · notice deadlines"]:::system
        PG[("<b>Postgres 18</b><br>leases · fields · corrections · job queue<br>— everything is saved here")]:::done
        OBJ[("<b>object-store volume</b><br>the original PDFs")]:::done
        MIN["<b>mining worker</b><br>reads the PDF, classifies the form,<br>pins every value to a box on the page"]:::system
        INF["<b>inference</b><br>reads the PDF, asks the model<br>stateless — saves nothing"]:::system
        PW["<b>platform-worker</b><br>turns the result into lease fields"]:::system
    end

    CL["<b>Claude</b><br>Opus extracts · Haiku re-ranks"]:::trigger


    Vhdr ~~~ FE
    U --> FE
    FE -- "1 · upload — later, review & correct" --> PLAT
    PLAT -- "2 · lease row + job" --> PG
    PLAT -- "2 · PDF" --> OBJ
    PG -- "3 · ping → mining<br>takes the job" --> MIN
    MIN -- "4 · what does this lease say?" --> INF
    INF <--> CL
    INF -- "5 · value + quote + page" --> MIN
    MIN -- "6 · grounded fields + a job" --> PG
    PG -- "7 · ping → worker<br>takes the job" --> PW
    PW -- "8 · fields saved" --> PG

Each blue box is its own Docker container, and so is Postgres; Docker Compose starts all six from one file, on one VM.

§13 NEXT STEPS

One thing I personally noticed was that I was spending most of my time answering questions from the loops, about branching choices. A lot of architecture choices. And I was getting a bit of decision fatigue. The terminal UI is not information heavy friendly, lots of bullet lists, etc. So I thought I would try to make this into a better looking UI for me.

I created, but never integrated, a decision UI, which the AI could use instead of asking things through the TUI. When writing this I realize it SOUNDS a bit strange. Why not design around the skill to the extent that it can take the decisions itself? And the short answer is that I.. didn’t trust it. I tried different ways to give it context and instructions but it so often made decisions I didn’t really agree with. Even though most of the architecture was clear, there were tradeoffs, and so many times, like always when dealing with AI, I felt it made the wrong choice.

I think it was a combination of a lot of context to go through, while having the headspace to make the right judgement calls. So, what I opted to do, was to make a UI which made it easier for me to make the right decisions.

I made it focus on the information I most often asked the agent: how it changed UX, technical impact, size of build, and any tech debt it might incur. The idea is that it served it in a static page (that I later could reach on my phone, such that I could advance also /ralph-arch while not at the computer) and acted on the input.

Above the decisions, what the agent assumed while working. Leave the right ones, flip a wrong one and it comes back as something to discuss. The amber one also names the earlier decision it overrules.

One decision on the page. Options are columns, judged on the same rows; blue is the agent’s case, green is my pick. The winning cell per row is computed separately from the recommendation, so it can disagree with it — here build time goes to doing nothing. The question is a real /ralph-arch ask from Track 21; the batch around it was written for this screenshot, since no real sitting ever went through the page.

But, it was never wired in. An experiment to run in a different issue.

§14 WHAT WENT GREAT, AND WHAT I WOULD DO DIFFERENTLY

All in all this went pretty good, I would say the quality and the speed of development of new ideas and features went fast. It felt stable. Things.. just happened, and architecture aligned. I think the initial research was a good reason for this, a great foundation to have as reference, since I did not have a good reference project to look at.

Another great thing, which I have taken with me to other projects, is the logs. Every subagent, every run, now gets hard numbers, which makes improvements quite effective. Also the “dont kill the loop if it’s running a subagent”, with a soft and a hard time cap, has proven great.

What did not work so great was decision fatigue with all the questions. The more the repo grew, the more context was needed, the longer things took to research, and the more I needed to have multiple fan-out and reconcile situations going, resulting in longer rounds, harder to multitask effectively, and more information given for every decision. Next time, I will progress to a more effective decision process, with better UI, that is more mindful to the developer focus.

One more thing, a reflection that I had from earlier project, and that I have heard from more and more people in the industry: when code is free, a small scope is needed. I built many things I would not even have tried building without AI. Was it fun? Yes. But damn that’s a lot of tokens wasted!

§15 CLOSING

When I started this ralph-loop I had a loop that had worked for a simpler system. This project would entail many modules, with complicated architecture, and with technologies I had not touched beforehand. In five weeks, I was able to build a working version where one could upload leases in different shapes and formats, have it extract and review data, and get an overview of the leases.

We started doing customer interviews when people got back from vacation and unfortunately we could not find a problem real estate companies wanted to pay for. The budget holders had bigger problems, and rental admin was.. while time consuming, not something that moved the needle enough. Vacancies, opex, legal questions, were often top of mind. After about 20-30 interviews over the course of a couple of weeks, me and my partner decided to take a step back and rethink.

Since then I have made many more projects, and tried codex hands off, but I miss the claude hooks, and the versatility the terminal gives you. I guess maybe this is what EMACS or VIM people felt in the past :) I’ve also coded by hand a bit again and it’s surprisingly refreshing.

Anywho anyhow. I am looking for work! AI Implementation Engineer, AI Solution Engineer, or Forward Deployed Engineer. Hit me up!