I built out askthecanon.com this weekend, a semantic search over 100 public-domain books (from the Gutenberg project). You ask a question in plain language and get the passages that mean that, cited by author, title, and chapter. I wanted a local, no-generative-AI solution: a retrieval engine using Hugging Face embeddings and NumPy that returns original passages, no full vector database (yet), no API calls at query time.

Why Ask the Canon?

I am already finding timeless wisdom using it myself (some of the best apps come from "scratching your own itch"), and I hope it offers a breath of fresh air in a world that seems to be dominated by AI-generated content and quick summaries.

My itch was wanting to read the originals, not an AI-generated summary, but also recognizing I don't have time and focus to read through a whole work (although it's still my aim, I see deep value in it). What if we can meet somewhere in the middle?

Most of what we wrestle with is not new: fear, ambition, grief, how to deal with people who wrong us. A chatbot will distill what the canon says about any of it in seconds, in one smooth, agreeable, slightly forgettable voice. Sometimes that's enough. Often I'd rather read the actual sentence Marcus Aurelius or Francis Bacon wrote and sit with it.

Thoreau, in Walden, on what real reading asks of you:

"Most men have learned to read to serve a paltry convenience ... but of reading as a noble intellectual exercise they know little or nothing; yet this only is reading, in a high sense, not that which lulls us as a luxury and suffers the nobler faculties to sleep the while, but what we have to stand on tip-toe to read and devote our most alert and wakeful hours to."

Ask the Canon does only that: it points a plain-language question at a hand-picked shelf of public-domain books and returns the real passages that answer it, cited down to the chapter. It never writes a word of its own, so nothing is invented and nothing is misattributed.

Ask the Canon card answering "what is the value of reading classic books?" with a passage from Thoreau's Walden: shall I hear the name of Plato and never read his book?

A result, rendered by the app's own "share as image" feature. Thoreau made my argument in 1854.

This is the first of three posts on how it's built. This one is the engine: how you go from a folder of messy text files to ranked, cited answers without reaching for the heavy infrastructure everyone assumes you need.

The default is over-engineered

Reach for "semantic search" and the stock answer is a vector database (Pinecone, Weaviate, pgvector) plus an embeddings API you call on every query.

That's the right shape at a billion vectors. At personal scale, tens of thousands of passages, it buys you operational weight and a network round-trip you don't need.

The whole corpus here is 79,292 passages across 100 books. As a float32 matrix at 768 dimensions that's about 240 MB, small enough to load once and keep resident. Once it's in RAM, "find the most similar passage" is a matrix multiply, and np.argsort over the result. That's the entire search:

def embed(texts: list[str]) -> np.ndarray:
    return _model().encode(texts, normalize_embeddings=True)

def retrieve(query: str, vectors: np.ndarray, k: int = 5) -> list[tuple[int, float]]:
    scores = vectors @ embed([query])[0]
    top = np.argsort(scores)[::-1][:k]
    return [(int(i), float(scores[i])) for i in top]

Because the vectors are L2-normalized at embed time (normalize_embeddings=True), the dot product vectors @ query is cosine similarity. No similarity function to import, no index to tune. One @.

I later refactored that bare multiply into a _scores() helper so the index can ship as float16, halving its memory, see this commit.

What embed() actually does

I've been waving at embed() as if turning text into numbers is obvious. It isn't, so here's the honest version at the level you need to follow the rest.

all-mpnet-base-v2 is a pretrained neural network (a sentence-transformer). Hand it a string and it hands back a fixed list of 768 floating-point numbers, always 768, whether the input is a single word or a full paragraph. That list is the vector. Each number is a learned feature, a coordinate on some axis the model discovered in training. None of them map to anything a human named ("axis 412 = melancholy"), and they don't need to.

The one property that matters: the model was trained so texts with similar meaning land at nearby points and unrelated texts land far apart. "How do I face death without fear" and a passage from Meditations end up close; that same passage and a Kubernetes tutorial end up far. Everything downstream, the dot product and the ranking, just measures that distance.

And a chunk is one vector, not a matrix. The model does compute a vector per token internally, then pools them into a single 768-number summary of the whole chunk before it ever reaches my NumPy. Query in, one row out; chunk in, one row out; compare the two rows. If you want to pry the box open further, Jay Alammar's Illustrated Transformer walks through the token-level machinery, but you can build and reason about this whole engine without it.

Embed once, cache to disk

The trick that makes this cheap: you embed the corpus exactly once. The model never runs at query time except on the single short query string. I run a local all-mpnet-base-v2 model, so there's no API key and nothing leaves the machine.

Indexing a book writes the vectors straight to a .npy file next to the source:

def build_index(book_id: int, text: str) -> tuple[list[Chunk], np.ndarray]:
    chunks = chunk_text(text)
    vectors = embed([c.text for c in chunks])
    np.save(BOOKS_DIR / f"{book_id}.npy", vectors)
    return chunks, vectors

At startup, the per-book matrices stack into one library matrix with np.vstack, and that's what every query multiplies against.

Embedding is the only expensive step, and it happens offline, on my laptop, never on the server. This separation also makes the deploy boring: build the .npy files locally, rsync them to the droplet.

The server never loads the model to build anything; it only embeds the incoming query. (The model loading itself is lazy and gated on offline env vars so there's no hub round-trip, we'll see how in Part 2 of this series.)

Chunks carry their own citations

A vector store usually means a second store for metadata: which book, which chapter, where in the text. I don't have one. The citation rides along with the chunk.

When I split a book, I track Gutenberg's CHAPTER / BOOK / CANTO headings as I go and stamp each chunk with the section it fell in:

class Chunk(NamedTuple):
    label: str  # e.g. "BOOK XI — Chapter IX"
    text: str

So a result isn't a naked paragraph. It's Marcus Aurelius · Meditations — BOOK IV, reconstructed from data that lived in the chunk all along.

The whole "database" is four kinds of file, no server:

  • books/<id>.npy: the embeddings for one book
  • books/<id>.chunks.json: the passages and their chapter labels
  • books/<id>.meta.json: the title, author, and a short book summary
  • library.txt: the list of Gutenberg IDs I grow by hand (now committed to the repo, could split off as "config" later)

To add or remove books, I update library.txt and run sync to rebuild the index which I then rsync to the server. No database, no migrations, no schema, no API calls.

The bug hiding in the chunk size

Chunk size is the one knob that decides whether any of this works, and my first cut got it wrong in a "silent error" way.

all-mpnet-base-v2 reads at most 384 tokens, roughly 290 words, per input. Anything longer is truncated before a single number is computed.

My first chunk_text targeted 600-word chunks, and it checked the size after appending each paragraph, so 600 was a floor it always overshot. A 599-word chunk plus one more paragraph landed well past 1,000 words.

So the model embedded the opening third of each passage and silently dropped the rest. The text I stored and cited was the whole passage, but the vector ranking it represented only the first few hundred words. Search was answering on text it had never read, and a long passage's real subject often sat in the part that got cut.

Nothing errored. The results looked on point, but they were quietly wrong: ranking was decided by the opening of each passage and ignored the rest. The cited passages were often long, and search was not matching the later paragraphs that actually could have better answered the query.

The fix was two small changes. Drop the target to 250 words, a safe buffer under the token limit, and flush a chunk before a paragraph pushes it over, so target_words is a ceiling instead of a floor:

TARGET_WORDS = 250  # ~330-350 tokens, below mpnet's 384 max

para_words = len(para.split())
if current and words + para_words > target_words:
    chunks.append(Chunk(chunk_label, "\n\n".join(current)))
    current = current[-overlap:] if overlap else []
    words = sum(len(p.split()) for p in current)

Now every chunk is embedded in full. Matches got sharper and the cited passages are short enough to read at a glance. Smaller chunks also nearly tripled the corpus, from 29,435 passages to 79,292, which is why the matrix grew while the search improved.

One follow-on: tighter chunks shifted the score distribution (spotted doing some end-to-end testing with Claude), so I raised the noise threshold from 0.32 to 0.34 to keep false positives out.

When a vector database stops being overkill

A thoughtful skeptic would push back: this doesn't scale. Correct, and that's the point. A linear scan is O(n) per query. At 80k passages it's a few milliseconds; at 30 million it's not. The moment you outgrow a single machine's RAM, or need filtered queries, multi-tenant isolation, or sub-millisecond latency at scale, we'd go with a real vector database.

But as detailed in Build the Simplest Thing That Works I prefer to get something working fast and validate it first putting it in front of real users.

Two lines of NumPy sit at the root of this engine. Everything else, the cards, the PDF export, the off-domain rejection, is built on top of that.

Using Claude was awesome, both in terms of getting the vectorization off the ground and it matching a classic vibe design so well. But 100-plus commits in, I kept having to step in with my experience and judgment to make the right call and tune things.

This judgment call is the same one I keep coming back to: match the tool to the actual size of the problem, not the size you imagine. It's the kind of engineering judgment AI doesn't change. AI is an accelerator, not a compass, and it still needs you to point it.