geary/sql/version-030.sql

--
-- Convert full-text search from FTS3/4 to FTS5
--

DROP TABLE IF EXISTS MessageSearchTable;

CREATE VIRTUAL TABLE MessageSearchTable USING fts5(
    body,
    attachments,
    subject,
    "from",
    receivers,
    cc,
    bcc,
    flags,

    tokenize="geary_tokeniser",
    prefix="2,4,6,8,10"
)
engine: Convert from SQLite FTS3/4 to FTS5 for full-text-search Add SQL migration that drops the old FTS4 MessageSearchTable table, re-create as a FTS5 table, clean up the column names a bit, and adds a flags column so unread/starred queries can be made fast. Define a SQLite FTS5 extension function `geary_matches()` to replace the FTS3 `offsets()` function which no longer exists in FTS5, based on Tracker's implementation. Update code to FTS5 conventions (docid -> rowid, etc), use new column names, populate and update the flags column as the email's flags change, and use new match function for getting matching tokens. Advanced searches are probably currently broken, these will be fixed by subsequent commits. 2020-11-04 00:55:13 +11:00			`--`
			`-- Convert full-text search from FTS3/4 to FTS5`
			`--`

			`DROP TABLE IF EXISTS MessageSearchTable;`

			`CREATE VIRTUAL TABLE MessageSearchTable USING fts5(`
			`body,`
			`attachments,`
			`subject,`
			`"from",`
			`receivers,`
			`cc,`
			`bcc,`
			`flags,`

ImapDb.Database: Register new ICU-based tokeniser for FTS The SQLite tokeniser does not deal with scripts that do not use spaces for word breaking (CJK, Thai, etc), thus searching in those languages does not work well. This adds a custom SQLite tokeniser based on ICU that breaks words for all languages supported by that library, and uses NFKC_Casefold normalisation to handle normalisation, case folding, and dropping of ignorable characters. Fixes #121 2020-11-13 08:41:08 +11:00			`tokenize="geary_tokeniser",`
engine: Convert from SQLite FTS3/4 to FTS5 for full-text-search Add SQL migration that drops the old FTS4 MessageSearchTable table, re-create as a FTS5 table, clean up the column names a bit, and adds a flags column so unread/starred queries can be made fast. Define a SQLite FTS5 extension function `geary_matches()` to replace the FTS3 `offsets()` function which no longer exists in FTS5, based on Tracker's implementation. Update code to FTS5 conventions (docid -> rowid, etc), use new column names, populate and update the flags column as the email's flags change, and use new match function for getting matching tokens. Advanced searches are probably currently broken, these will be fixed by subsequent commits. 2020-11-04 00:55:13 +11:00			`prefix="2,4,6,8,10"`
			`)`