mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Garry TanandYMYD
f7d4f19124
Wave-assembled from PR #3774 by @paul-0320. Co-Authored-By: YMYD <paul@ymyd.co.kr>
109 lines
4.1 KiB
Markdown
109 lines
4.1 KiB
Markdown
# Multi-language full-text search
|
|
|
|
GBrain's keyword search arm uses Postgres full-text search (tsvector/tsquery).
|
|
The tokenizer language is configurable via the `GBRAIN_FTS_LANGUAGE`
|
|
environment variable. Default: `english`.
|
|
|
|
## How it works
|
|
|
|
Postgres text-search configurations control stemming and stop-word removal.
|
|
`GBRAIN_FTS_LANGUAGE` is read by `src/core/fts-language.ts` and applied on
|
|
both sides of the search:
|
|
|
|
- **Query side** — `websearch_to_tsquery('<lang>', $query)` in both engines
|
|
(Postgres and PGLite).
|
|
- **Write side** — the `update_page_search_vector` and
|
|
`update_chunk_search_vector` trigger functions that populate
|
|
`pages.search_vector` and `content_chunks.search_vector`.
|
|
|
|
The value is validated against `/^[a-z][a-z0-9_]*$/` before it is ever
|
|
interpolated into SQL (tsvector functions don't accept parameterized config
|
|
names). Invalid values fall back to `english` with a warning.
|
|
|
|
## Built-in languages
|
|
|
|
Set the env var to any configuration your Postgres instance ships:
|
|
|
|
```bash
|
|
export GBRAIN_FTS_LANGUAGE=portuguese
|
|
export GBRAIN_FTS_LANGUAGE=spanish
|
|
export GBRAIN_FTS_LANGUAGE=german
|
|
```
|
|
|
|
List what's available:
|
|
|
|
```sql
|
|
SELECT cfgname FROM pg_ts_config;
|
|
```
|
|
|
|
PGLite (the embedded default engine) ships the same built-in snowball
|
|
configurations as stock Postgres.
|
|
|
|
## First install vs. changing language later
|
|
|
|
On first install (or upgrade), the `configurable_fts_language` schema
|
|
migration reads `GBRAIN_FTS_LANGUAGE` and stamps the trigger functions with
|
|
that language. After the migration has run, changing the env var alone does
|
|
NOT retokenize existing rows — the migration shows as applied and is skipped.
|
|
Use the explicit command:
|
|
|
|
```bash
|
|
export GBRAIN_FTS_LANGUAGE=portuguese
|
|
gbrain reindex-search-vector --dry-run # preview: language + row counts
|
|
gbrain reindex-search-vector --yes # recreate triggers + backfill
|
|
```
|
|
|
|
The stamp survives later schema work: `initSchema()` — including the replay
|
|
behind `gbrain init --migrate-only` on every upgrade — applies the schema
|
|
template under the configured language, so it re-creates the trigger
|
|
functions as they already are instead of reverting them to `english`.
|
|
|
|
The command recreates both trigger functions under the new language and
|
|
backfills every existing `pages` and `content_chunks` row in batches,
|
|
streaming progress to stderr. It is idempotent: re-running with the same
|
|
language produces identical vectors. `--json` prints a machine-readable
|
|
result envelope but still requires `--yes` (or an interactive confirm).
|
|
|
|
No cache purge is needed. The resolved language is part of the query-cache
|
|
key, so rows written under the previous language are unreachable after the
|
|
switch — searches read the retokenized index immediately instead of being
|
|
served pre-switch results for up to `search.cache.ttl_seconds`. Switching
|
|
back reaches the original rows rather than rebuilding them.
|
|
|
|
## Recipe: accent-insensitive Portuguese (`pt_br`)
|
|
|
|
Brazilian Portuguese content often mixes accented and unaccented spellings
|
|
("São Paulo" vs "Sao Paulo"). Build a custom config that folds accents via
|
|
the `unaccent` extension, then stems with the portuguese snowball dictionary:
|
|
|
|
```sql
|
|
CREATE EXTENSION IF NOT EXISTS unaccent;
|
|
|
|
CREATE TEXT SEARCH CONFIGURATION pt_br (COPY = portuguese);
|
|
|
|
ALTER TEXT SEARCH CONFIGURATION pt_br
|
|
ALTER MAPPING FOR hword, hword_part, word
|
|
WITH unaccent, portuguese_stem;
|
|
```
|
|
|
|
Then point GBrain at it:
|
|
|
|
```bash
|
|
export GBRAIN_FTS_LANGUAGE=pt_br
|
|
gbrain reindex-search-vector --yes
|
|
```
|
|
|
|
Note: custom configurations require a real Postgres instance (e.g. the
|
|
Supabase engine). The config must exist BEFORE the migration or the reindex
|
|
command runs, or Postgres will reject the trigger recreation with
|
|
`text search configuration "pt_br" does not exist`.
|
|
|
|
## Caveats
|
|
|
|
- One language per brain: the setting is global to the database, not
|
|
per-source. Mixed-language brains should pick the dominant language (the
|
|
vector-search arm is language-agnostic and covers the rest).
|
|
- Keep `GBRAIN_FTS_LANGUAGE` set consistently in every environment that
|
|
writes to the brain (CLI shells, MCP server, cron jobs) — a writer without
|
|
the env var tokenizes new rows in `english` until the next reindex.
|