Storage: knowledge and data

Storage holds what the platform uses to answer and to automate: documents indexed for search and structured tables for queries. Everything stays in your installation's databases and storage.

Context types

Contexts are organized by workspace and have an owner, who can share them with other people and transfer ownership.

Knowledge contexts

Uploading files

Upload files or whole folders, up to 100 MB per file; each file keeps the folder's relative path. Files can be moved or renamed, and the original can be downloaded at any time.

Formats read

• Text: txt, md, csv, tsv, json, xml, html, and log.
• Documents: digital PDF, docx, xlsx, and pptx.
• Images: described by a local vision model so they become searchable.
• Audio and video: transcribed by a local transcription model.

Scanned PDFs without a text layer and the legacy Office formats (.doc, .xls, .ppt) do not have their text extracted.

Ingestion

Ingestion runs in the background, in the worker service:

upload
  -> original in object storage (MinIO or S3)
  -> task in the PostgreSQL queue
  -> text extraction
  -> chunking by document structure
  -> embeddings with the context's model
  -> lexical index in PostgreSQL and vectors in Milvus
  -> category and tags suggested by the chat model
  -> document indexed

Each document's indexing status is indexed (semantic and text search), lexical_only (text search only), or index_failed.

Hybrid search

Search combines vectors in Milvus with text search in PostgreSQL, merges both results with Reciprocal Rank Fusion, and reorders the best candidates with a rerank model. Retrieved chunks appear with their scores in chat and in flow traces.

Embedding model

Each context pins its own embedding model. Changing the model recomputes the embeddings and reindexes every document in the context. Embeddings and reranking use locally served models with an OpenAI-compatible API.

Vector inspector

The inspector runs hybrid search across all contexts, one context, or one file and shows, for each chunk, its text, lexical terms, dimensions, and vector norm.

Classification

Each document can carry a classification: public, internal, confidential, or restricted. The classification is an organizational label; access control is handled by workspace, sharing, and module.

Data contexts

Typed tables

Tables have columns of type text, integer, decimal, boolean, date, timestamp, and JSON, with a primary key, searchable columns, and a sensitivity label per column (public, internal, confidential, or personal).

Loading data

• CSV, TSV, or XLSX files up to 100 MB, in append or replace mode. In replace mode, the load goes into a staging table and the content is swapped only at the end, so a failed load never leaves the table empty.
• Manual row entry, up to 5,000 rows per operation.
• Each load shows the rejected lines and the reason for each one. Rejections also go into the governance irregularity register.

SQL queries

User SQL runs on a separate, read-only database role, isolated from the application's database: one SELECT statement at a time, with a statement timeout and up to 1,000 rows per result.

Ad-hoc SQL is off by default in every data context and is turned on with the Allow ad-hoc SQL option. While it is off, neither manual queries nor chat can query the context. Flow blocks work with it off, because their SQL is validated when the flow is published.

Use in flows

• Data query: reads rows with SQL from a table.
• Data write: writes rows, for example the structured output of an AI block.
• Load files: loads spreadsheets or CSVs from a Storage folder into a table.

See Flows and schedules.

Good practices

• Separate contexts by purpose and audience: sharing is done per context.
• Keep the same embedding model while a context is in use; changing it reindexes everything.
• Keep ad-hoc SQL off, as it is by default, in data contexts that should be reached only by published flows.