My Open Source AI Project: 347 Commits, No Users
On June 18, 2025, I ran my first commit: “RAGnetic Project Setup.” I spent the next four months building an open source AI project and never launched it.
347 commits, 22,000 lines of Python, about 100 API endpoints, and the only people who ever touched it were a couple of coworkers I asked to try it as a favor. That’s the whole outcome.
Here’s what building the wrong thing, for four months, taught me.
The project is called RAGnetic, and the code is still on GitHub if you want to see the receipts yourself.
The open source AI project I meant to build, and what I actually built
I worked in biotech for about two years, from 2023 to 2025. What I kept running into was process knowledge scattered across docs, wikis, and people’s heads, and no fast way to ask a question and get a real answer.
The idea was to consolidate that knowledge and embed it into whichever AI model you preferred, local or one of the frontier ones, and just talk to it whenever you needed information.
Point it at your documents, your databases, your APIs, describe an agent in YAML, and deploy it. “Complete ownership of your on-premise AI.”
That’s a specific problem. What I shipped was not a specific solution to it.
The README organizes RAGnetic into five pillars: Ingest, Adapt, Collaborate, Interact, Evaluate.
- Ingest: PDFs, DOCX, CSV, web pages, code repos, live SQL, with PII redaction and hybrid BM25 + vector retrieval.
- Adapt: hosted models (OpenAI, Anthropic, Gemini), local ones through Ollama, and LoRA fine-tuning wired directly into agent config.
- Collaborate: multi-agent graphs on LangGraph.
- Interact: a Docker sandbox for code, a SQL toolkit, web search, ArXiv, an HTTP toolkit.
- Evaluate: benchmarking with retrieval metrics and an LLM judge.
Underneath all of that: FastAPI, Celery, Redis, Postgres, Alembic migrations, user and role management, API keys with rate limiting, and a credits and cost-accounting system with per-model pricing tables, the kind of thing you need once you’re actually paying attention to what an AI tool costs you.
Read that list again. Each bullet is somebody else’s entire company.
I built a generic platform for anyone who might one day want to talk to their documents, instead of the specific tool a biotech team needs to talk to their process docs. Somewhere between the idea and the code, the target quietly disappeared.
Lesson 1: I tried creating a platform instead of a product
Here’s the part that actually stings. I didn’t need to guess who my first users would be.
They were sitting next to me. A couple of coworkers agreed to try it, people who had the exact problem I’d started this to solve.
What I handed them was RAGnetic: five pillars, a YAML config language, a CLI, a dashboard, LoRA fine-tuning. Not “ask your process docs a question.” A platform, when what they needed was a product.
Here’s what I’ve come to think since: the best products don’t try to solve everything, at least not in the beginning. They solve one thing all the way, for one person, and earn the right to solve the next thing later. Mine tried to solve everything from day one, which meant it never quite solved anything.
Part of why it never got further than that was scope. One more feature was always the thing standing between where it was and where it needed to be to show someone.
Retrieval was fine, but the eval system wasn’t done. The eval system was fine, but the dashboard needed a rework.
There was always a next thing, and the next thing always felt more urgent than putting an unfinished tool in front of the people who’d asked for the specific thing in the first place.
The moment that actually landed the lesson wasn’t a user conversation. It was applying to YC and getting rejected.
That’s what made me look at the space honestly: RAG frameworks and agent platforms were already one of the most saturated categories in software, full of funded teams shipping the same pitch. A generic “better platform” story loses to whoever has more distribution.
I didn’t have distribution. I had a biotech problem and two coworkers who would have told me, for free, whether embedding their process docs into a model actually helped them find an answer faster. I built the platform instead of asking them.
What I’d do differently: skip the platform. Build the one thing the person next to me needed, show it to them in a week, and let their answer decide whether pillar two ever gets built.
Lesson 2: I built a benchmark that couldn’t fail
I had a sandboxed code executor, a distributed task queue, role-based access control, and a cost calculator with per-model pricing tables. What I didn’t have was a trustworthy answer to “is this agent any good?”
Benchmarking an AI agent means the same thing it means anywhere else: you write down a fixed set of test questions with known correct answers, run the agent against every one of them, and check two things whenever anything changes. Did it pull the right source material back out of your documents, and did the answer it gave actually match a correct answer?
Do that before the next feature, and you have a number that tells you whether a change made the system better or worse. Skip it, and every “it feels smarter now” is a guess.
Here’s how RAGnetic did it. The only benchmark I ever saved to the repo ran 10 questions, generated by an LLM from a single PDF (a paper on language agents). The gold answers were generated and then never compared to what the agent actually produced.
The benchmark only asks an LLM judge whether the answer is faithful to the retrieved context, and on that measure it scored 10 for 10. Worth knowing that’s a real metric and a real distinction, not something I made up to sound rigorous: Ragas lists faithfulness and factual correctness as separate metrics, because “grounded in the context you retrieved” and “actually the right answer” are different questions. I only ever asked the first one.
The retrieval side tells the real story. I logged two versions of the retrieval metrics: a strict one that matches chunk IDs exactly, and a lenient one with a content-hash fallback for when the IDs don’t line up.
The strict metric scored zero on every single question, all 10, because the two ID systems it was comparing never could have matched. The lenient metric hit on 9 of 10.
That means one question missed retrieval entirely, the agent answered from context that wasn’t the right context, and the faithfulness judge scored it faithful anyway.
| What the benchmark measured | Result |
|---|---|
| Faithfulness: is the answer supported by the context it retrieved? | 10 / 10 |
| Retrieval, lenient (content-hash fallback match) | 9 / 10 |
| Retrieval, strict (exact chunk-ID match) | 0 / 10 |
| Whether the answer was actually correct | never measured |
A 100% pass rate on 10 easy questions from one document isn’t evidence the system works. It’s evidence I never built a way for it to fail, the same trap an overfit model falls into: a number that looks perfect because you never tested it against anything that could prove it wrong.
I’ve made the same mistake in the other direction too: when I later looked at how Claude Code’s auto mode keeps an agent in check, the thing that stood out was a small, dumb judge sitting between the agent and the action, not a vibe check. A real evaluation is the same idea applied to quality instead of safety: something small and mechanical that can actually say no.
Meanwhile the README called the whole thing “production-ready” and “enterprise-grade,” under a section titled “Enterprise & Government Readiness.” One person, one PDF, one benchmark that couldn’t fail, and a claim like that sitting at the top of the file. It wasn’t dishonesty so much as the same instinct that built the five pillars: reach for the bigger claim before you have the evidence for the smaller one.
What I’d do differently: build the evaluation before the second feature. A real, hand-written test set on real documents, run on every change.
If gold answers exist, compare them in code, an exact match or a real scoring function, not just a second model’s opinion of the first model’s answer.
That’s a rule I hold myself to now on newer AI features: numbers and facts come from your actual data, checked in code, never from what a model remembers or another model’s vibe check. If you can’t measure “better” that way, you’re not steering, you’re guessing.
Lesson 3: every layer I sat on top of moved faster than I did
RAGnetic is glue over LangChain, LangGraph, Ollama, Hugging Face, PEFT and TRL, Chroma, FAISS, Qdrant, Pinecone, Mongo, Celery, Redis, and Postgres. Ten dependencies, all moving, all at once, and I was the only one keeping them wired together.
The scars are still in the repo. Eight requirements*.txt files, one of them literally named requirements-old-bloated.txt.backup.
A hard pin on numpy==1.26.4. A 2.4 GB virtual environment.
Commit messages like “More REDIS REDIS REDIS REDIS STUFF” and “SOME ANNOYING MIGRATION ISSUES THAT ARE FINALLY FIXED,” both from the same July week.
The clearest example is the Ollama client. app/core/config.py builds it by importing ChatOllama from langchain_community.chat_models.ollama. app/agents/agent_graph.py builds the same client by importing ChatOllama from langchain_ollama instead.
Two different code paths, constructing the same object, and one of them was already on the path LangChain had moved away from by the time I wrote the other — the current docs install langchain-ollama and import ChatOllama from it, not from langchain_community. I didn’t notice while I was building it. The ecosystem I was gluing together shipped a breaking change faster than I finished wiring the last one in.
Some of that is just what building on other people’s fast-moving projects looks like. But a lot of my time went into keeping the glue from coming apart, time I wasn’t spending on the one thing that would have made a coworker’s day easier.
What I’d do differently: fewer dependencies, one storage backend, and an install that takes one command. If a stranger, or a coworker, can’t try it in five minutes, they won’t.
Lesson 4: it kept me ahead of the AI race
I don’t want to end on pure regret, because that isn’t honest either.
The same speed that beat me in Lesson 3 cut the other way too. Chasing LangChain, LangGraph, Ollama, and half a dozen vector stores as they all moved at once meant I was never reading about the AI race from the sidelines.
I was inside it, learning and adapting in real time. Even as a failure, RAGnetic is the reason I know what the raw architecture of an agent actually looks like when you wire it together yourself instead of skimming a diagram of one.
In four months, alone, I shipped multi-tenant auth, background workers, a sandboxed executor, a hybrid retriever, a LoRA training pipeline wired into the same YAML as everything else, a CLI, a dashboard, a REST API, database migrations, and observability. I understand the full stack of an LLM application in a way that reading about it never would have given me.
I can also name the actual mistakes now instead of feeling vaguely bad about them: I built a platform before I found a person, I built a benchmark that couldn’t fail, and I let ten fast-moving dependencies eat time that should have gone to the two people who’d offered to try the thing.
RAGnetic is still up on GitHub, and I haven’t touched it since March. What I keep coming back to is whether the honest version of this idea is smaller than what I built: not a platform with five pillars, but something that runs entirely on your own machine, on an open-source model, built around the one question I started with. And what’s the proof?
If you’ve shipped something nobody used, what’s the one feature you kept promising yourself you’d add before showing anyone?