Knowledge & ingestion

What happens between "upload" and "the assistant cited page 12" — formats, the pipeline, retrieval, and deletion.

Supported sources

Uploads can also go through the API — POST /v1/documents and POST /v1/documents/url (see the API reference).

The pipeline

Every source moves through four stages, visible live on the Add Sources page:

1. 1

   Extract

   Text is pulled out page by page (PDF) or from the cleaned article body (web pages — navigation and ads are stripped). Every piece is tagged with your workspace.

2. 2

   Chunk

   Text is split into overlapping passages of ~400 tokens with an 80-token overlap, so ideas that straddle a boundary aren't lost. This size is lab-measured against a golden dataset — not a guess.

3. 3

   Embed

   Each passage becomes a meaning vector using a sentence-embedding model, in parallel batches.

4. 4

   Index

   Vectors land in your workspace's own private namespace. No other workspace can ever retrieve them.

A document is Readywhen its last chunk is indexed — from that moment it's part of every answer's retrieval. If a stage fails, the document shows Failed with the actual error; delete it and re-add after fixing the source.

How retrieval works

Questions don't just do a similarity lookup. BrainStack runs hybrid retrieval: a dense vector search (meaning) and a BM25 keyword search (exact terms like error codes or product names) run side by side, and the two rankings are fused. That's why "ERR_4021" hits even when embeddings alone would miss it. The top passages become the numbered citations in the answer — every fact in an answer traces back to one.

Deleting a source

Removing a document from the Library deletes its indexed knowledge immediately and everywhere— the stored file, the chunks, and the vectors in your namespace. The next question simply can't draw on it. Existing answers keep their text, but their citations into that source no longer open.

Getting better answers

• • Prefer text-native PDFs over scans — extraction quality decides answer quality.
• • Split giant compilations into topical documents; retrieval ranks passages, and focused sources produce cleaner passages.
• • Index the page that CONTAINS the facts, not a landing page that links to them.