Dominik and Openspider: AI Coding interview

Right now, release 1.0, planned for November 17. OpenSpider covers Screaming Frog’s feature set, but is open source (Apache-2.0) and built for agents from the start: CLI, MCP server, and desktop app all use the same Rust core.

Development is clean-room. The agents only get our own specification with verifiable acceptance criteria. They never see the reference product, and every test carries the spec ID it covers.

On performance, AI coding is strong when you first let the agents measure. Two examples from the last few days:

Write path: An agent reworked the writer onto a channel with larger SQLite transactions. Large crawls take around 30% less time as a result. Link score: The analysis now runs in memory instead of via SQL. For one million URLs it dropped from 113 to 23 seconds.

Without hard budgets, though, it quickly becomes arbitrary. The detail pane in the live crawl has a budget of 150 ms and now sits at a median of 34 ms, measured on real hardware in the release build.

Models with separate roles:

Claude Opus 5.5: orchestration and build OpenAI Codex with GPT-6.1 Sol: build packages in loops on a Hetzner server Claude Sonnet 5.5: independent acceptance tests, only from the spec, without the builder’s history Fable 5.1: review before every merge

The separation pays off. The independent test agent found corruption of crawl files that had been in the code since the first milestone: a File::open/close in the owner process released SQLite’s POSIX locks.

Harness:

Claude Code as the hub with many parallel sub-agents, each in its own Git worktree built and tested on the server; the CI matrix for macOS, Windows, and Linux runs in GitHub Actions, including signing and notarization 17 Rust crates, around 2,500 commits in four weeks

Tech stack:

Rust with Tokio SQLite in WAL mode, one file per crawl Chromium via CDP for rendering Tauri v2 with React and TypeScript Playwright in Chromium and WebKit

MCP is a product feature for us: OpenSpider ships its own MCP server that you can connect with one click to Claude Desktop, Claude Code, Codex, or Cursor.

Around 14.5 billion tokens in just under four weeks, counted from the session logs, as of November 9. Of that, a good 11 billion go to Claude (Opus 5.5, Sonnet 5.5, Fable 5.1) and a good 3 billion to Codex with GPT-6.1 Sol. The number sounds huge, but around 94% of it is cache reads. The agents keep reading the same context: spec, code, reports. The models actually generated “only” around 50 million output tokens. From that came a good 2,500 commits and around 380,000 lines of Rust.

My own effort is 50% of my free time. I barely write code myself. I decide what gets built, clarify questions from the spec, test on real hardware, approve releases, and occasionally look over the code. The agents run almost around the clock for that, including at night and on weekends.

On costs, I work with subscriptions (Claude and Codex €200 packages) instead of the API. On top of that come a build server at Hetzner for around €300 per month, the Apple developer account, and code signing for Windows via Azure. Billed through the API, it would have been many times that. For me that is the real lever: with subscription and cache, an agent setup of this size pays off even for an open-source project.

I have been developing software for over 25 years, most of that time for the web. I got into AI coding relatively early, with Cursor at least a year and a half ago. I was among the first 10,000 users. After that I tried GitHub Copilot and was also early with Claude Code, back then still only as a CLI.

After the web came my first Apple app, and with OpenSpider I am now building a complete desktop application for the first time. That is a deliberate choice: I want to see how far AI carries across very different tech stacks, from web through mobile to desktop. And OpenSpider, with its Rust core, signed installers for macOS and Windows, and its own MCP server, has been the toughest test of that so far.

“Developing hardware” can mean many things. I have already built small robots myself and adapted them with AI, for example a media box for my daughter. There, hardware, software, and AI blur together anyway. Agents designing their own chips, on the other hand, I do not see yet — but that is also outside my expertise. In principle I see no major difference from software: there, too, models are already improving themselves. With hardware it simply becomes physical, and you need specialized machines.

At OpenSpider I see the limits today less in the code than at four places around it:

What agents cannot see, they cannot verify. A signed desktop app on real macOS and Windows can only be automated to a limited extent. To this day, no agent can remotely control the update click in the release app under Windows. For that I sit down myself at two machines. “Green” often only means “green on Linux.” Our agents build on a Linux server. A test that passes under Linux can fail under Windows on a filename with a colon. You only see that in the platform matrix of the CI. Agents quietly soften tests. If you ask an agent to make flaky timing tests robust, it likes to raise thresholds or lower the load until nothing is really checked anymore. That is why other models cross-check for us, and we deliberately inject faults to see whether a test still catches them. The bottleneck is orchestration, not writing. With many agents in parallel, disk space, CI runtimes, and conflicting assignments became a problem. They write code faster than you can cleanly integrate it.

Deciding, prioritizing, and accepting on real hardware remains human work for me.