🛠️ board game chat

Board Game Chat is a web app that allows you to ask rules questions when playing a board game.

(It's a work-in-progress. Currently, Chrome-only. And the production version is broken for some reason.)

This is yet another app that exists to scratch a specific itch of my own, in this case: having to look up rules clarifications mid-game, particularly for rules-heavy games. (The specific motivator was Frosthaven, hence why it's the only one implemented in the app so far.)

It is effectively a Chat-GPT-like interface on top of a RAG system.

Apart from solving a problem for myself, it was an opportunity to understand RAG systems by building one from scratch.

☙

RAGs involve an LLM with access to documents via a search system.

They involve a pipeline of steps, all needing to work decently for the final outcome to be decent:

• parse PDFs of documents
  • tried several, two column layouts mess many systems up
  • ended up using Google Gemini 2 Flash
• split those documents up into chunks
  • currently naively splitting on pages, rather than semantic sections
• index those chunks (classically, and as vector embeddings)
• convert a user's query into related queries (for vector search and classic text search)
  • don't need to be particularly smart for this, but speed is nice, so using LLAMA running on Cerebras
• return results for those queries
  • combining traditional and vector results is tricky
  • currently doing it naively
  • should use a reranker
• analyze the results and summarize an answer
  • often the weakest link, because it requires proper "reasoning," which the models are bad at
    • particularly, because in games, specific terms have strict in-game meanings rather than fuzzy meaning
    • in the app, I explicitly show what was fed into the model, so that if the "summary" goofed, we would still have something useful (effectively making the app a rules search engine)
  • often gets fed bad input and has not enough to work with to come up with an answer
    • using an agent / MCP-enabled approach would allow it to re-query for more information

While I was working on this, every weekend when we would play and a rules question came up, we would race to see what was faster: the app, or a human with a rulebook. The app was often faster (not always!), but would often make mistakes. But, a few times, it would find a rule that my friend missed (the kinds of questions we would typically ask involved the intersection of 2 or 3 rules). With enough work, I think it'd be possible to make the app faster, more comprehensive, and "reason" sufficiently well.

By biggest take-away is that LLMs now make it very easy to implement "semantic search" (where "cat" is similar to "dog" because they are both "pets"), but still tricky to implement good overall search.

☙

LLMs are a moving target. At the start of this project, the context windows of LLMs weren't large enough to fit an entire rulebook and related FAQs, but in 2025, you could fit everything in (often with better results, but not always). Lately, we've just been using a Google Notebook with the rulebook and FAQ uploaded, and it works well enough, obviating this project almost entirely. I'll still likely poke around to understand the limitations of the underlying models better.

Future

• add a reranker (based on some readings, it's a critically important step)
• try an "agent" like approach, where the orchestrator (running a proper "reasoning" model) can choose to run queries more than once
• rewrite with Datalevin (which now includes vector search and text search)
  • (currently "rolling my own db" by using Nippy to persist to files)
• experiment with putting the whole rulebook in context
  • how are the results? how's the speed?