Move SQL generation for FTS search from ImapDb.Account to SearchQuery.
Convert to use Geary.SearchQuery.Term instances to generate SQL, rather
than parsing the expression. Simplify the generated SQL substantially
and generate MATCH values that work with SQLite FTS5.
Add SQL migration that drops the old FTS4 MessageSearchTable table,
re-create as a FTS5 table, clean up the column names a bit, and adds a
flags column so unread/starred queries can be made fast.
Define a SQLite FTS5 extension function `geary_matches()` to replace
the FTS3 `offsets()` function which no longer exists in FTS5, based on
Tracker's implementation.
Update code to FTS5 conventions (docid -> rowid, etc), use new column
names, populate and update the flags column as the email's flags
change, and use new match function for getting matching tokens.
Advanced searches are probably currently broken, these will be fixed
by subsequent commits.
Although populating the search table had been broken up into batches
of 50 email, it was still search for and loading every single message
id in both the MessageTable and MessageSearchTable, doing a manual
join, and then updating the batch, for *each* batch, and in a RW
transaction.
Break this up so that the ids are loaded and joined only once, the
queries happens in a RO transaction, the manual join happens in a side
thread, leaving each RW transaction only having to load the messages
and update the search index for up to 50 messages.
Move SearchFolder and search EmailIdentifier implementation out of
ImapDb and into its own package. Decouple both from ImapDB, and improve
the implementation, fixing a few inefficiencies. Merge search
FolderProperties into the SearchFoldern implementation as an inner
class.
Merge SearchTerm into ImapDB.SearchQuery as an inner class and move the
outer class's source down a level, since it was the only file left in
the imap-db/search dir.
Two of my account's databases had duplicate inboxes which were not being
removed by the existing workaroud, this updates it so that they are.
I had removed the duplicate from a third account ages ago and it hasn't
come back since, so maybe the underlying problem is gone, but the
correct fix would be to move this workaround to a migration and add a
uniquness constraint on (parent_id, name).
Make ImapDb.Account::strip_removal_conditions use a single RO txn and
manually load flags from the message row when looking up email to filter
out, to keep overhead as low as possible.
This allows ImapDB unit tests to avoid running the process when it
really doesn't need to be run. Should fix anther criticial about
ImapDB being null (see previous commit).
Move args from open_async to ctor and use these to determine and store
DB file and attachment paths as properties. This allows constructing
the DB instance up front and markig it as non-nullable, simplifies
calling patterns and hence allows removing one use of the deprecated
Engine singleton, and by moving in the local data deletion
implementation from ImapEngine.GenericAccount, also allows removing a
static helper method.
Remove contact harvesting from DB version 005 (version 0.1.1), allowing
the implementation of both ContactStoreImpl and ImapDB.Database to be
cleaned up. Remove contact collection from ImapDB.Folder, reducing the
numer of Gee objects created when creating/merging messages. Finally,
remove the MessageAddresses object now that it is unused.
Prepare for doing composer autocomplete via queries by not loading all
contacts at startup in the engine. Update Geary.ContactStore API to that
end, and convert Geary.Account.get_contact_store accessor method to a
property.
find . -name '*.vala' -type f -exec sed -i 's/ *$//g' {} +
The following files was ignored:
test/client/composer/composer-web-view-test.vala
test/engine/util-html-test.vala
Fix#271
Make result stripping methods operate on the same collection so it's
easier to follow what is happening, and less objects need to be created
(especially when not stripping greedy stemming results).
Convert getters that look like properties into actual properties,
remove unused and redundant public and internal API, convert
implementation to use a tree that aucyally maintains references between
steps, rather than each creating a new list of path steps and
manipulating that.
Instead of each top-level IMAP folder being a FolderRoot object, then
children of that being FolderPath objects, this makes FolderRoot an
"empty" FolderPath, so that both top-level and descendant folders are
plain FolderPath objects. Aside from being more technically correct,
this means that empty namespace roots can now be used interchangably
with non-empty namespace roots (addressing issue #181), and custom
folder implementations no longer need to provide their own trivial,
custom FolderRoot.
To support this, a notion of an IMAP root and a local root have been
added from which all remote and local folder paths are now derived,
existing places that assume top-level == root have been fixed, and
unit tests have been added.
This splits off the sending part of SmtpOutboxFolder into a new
Smtp.ClientService class that uses the standard Folder APIs for managing
mail in the outbox, and makes SmtpOutboxFolder handle local mail
CRUD operations only. This enables removing SmtpOutboxFolder from
ImapDB.Account and managing it from GenericAccount instead.
Part three of a three part series.
This mostly aims to make the Geary.Attachment and ImapDB.Attachment
objects usable more generally for managing attachments, allowing
these to be instantiated once, persisted, and then reused, rather than
going through a number of representations (GMime, SQlite, Geary) and
having to be saved and re-loaded.
* src/engine/api/geary-attachment.vala (Attachment): Remove id property
and allow both file and filesize properties to be set after instances
are constructed. Update call sites.
* src/engine/api/geary-email.vala (Email): Remove get_attachment_by_id
since it unused.
* src/engine/imap-db/imap-db-attachment.vala (Attachment): Chase
Geary.Attachment API changes, move object-specific persistence code
into methods on the actual object class itself and modernise a
bit. Rename static methods to be a bit more terse. Update call sites
and add unit tests.
* src/engine/imap-db/imap-db-folder.vala (Folder): Rather than saving
attachments to the db then reloading them to add them to their email
objects, just instantiate Attachment instances once, save and then add
them.
* src/engine/imap-db/imap-db-gc.vala (GC): Replace custom SQL with
existing accessor for listing attachments.
* src/engine/util/util-stream.vala (MimeOutputStream): New adaptor class
for GMime streams to GIO output streams.
This stops generating account storage paths, and in particular the
location of attachment files, in a number of different places. Instead,
we determine the paths once, in ImapDb.Account and pass them around as
needed. This will help making the ImapDB classes unit-testable, and
in particular ImapDb.Folder by passing an instance of Db.Database,
rather than ImapDb.Database.
This makes both the open() and open_connection() methods on
Geary.DB.Database asynchronous, which allows the VersionedDatabase
open_background() and its hackery to be removed, and upgrades to be
performed asynchronously as well. It also adds a
exec_transaction_async() method to Connection, allowing an existing
object to also be used to establish an async transaction.
* src/engine/db/db-connection.vala (Connection): Add
exec_transaction_async method, update doc comments.
* src/engine/db/db-database.vala (Database): Make open and
open_connection async by executing SQLite code in a background thread,
update call sites. Move job management code out of
exec_transaction_async into a new internal add_async_job() method so it
can be used by Connection. Add unit tests.
* src/engine/db/db-transaction-async-job.vala (TransactionAsyncJob): Add
an optional internal connection method and make the job's cancellable
an internal property so a Connection instance can specify itself for
the transaction.
* src/engine/db/db-versioned-database.vala (VersionedDatabase): Remove
open_background() hack since open() is now async. Make version upgrade
hooks for derived classes async and update call sites. Use a
Nonblocking.Mutex rather than GLib mutex so upgrade exclusion works
asynchronously. Add unit tests.
* src/engine/imap-db/imap-db-database.vala (Database): Make database
upgrade methods async and execute SQL in async instructions now that
the bases classes support it. Add unit tests.
This commit makes the Imap.Account and Imap.Folder classes work somewhat
more like Imap.ClientSession, in that they have become higher-level
wrappers around ClientSession which come and go as the client session
does (i.e. as the connection to the IMAP server comes and goes). Further,
partly decouple account session lifecycle in ImapEngine.GenericAccount
and the folder session in ImapEngine.MinimalFolder from those objects
being opened/closed, so that sessions are created only when open /and/
the IMAP server is available, and disconnected on close /or/ when the
underlying connection goes away.
As a result, GenericAccount and MinimalFolder no longer claims a client
session on open and try to keep it forever. Instead if needed, they wait
for the server to become contactable.
This makes Geary much more robust in the face of changing network
connections - when working offline, resuming after sleep, and so on.
This allows simplifying the implementation and removing the dependency on
an instance of ImapDB.Account from ImapEngine.MinimalFolder.
* src/engine/imap-db/imap-db-account.vala (Account): Move
update_folder_status_async and update_folder_select_examine_async and
support code to Folder.
* src/engine/imap-db/imap-db-folder.vala (Folder): Rework moved methods
to pass in a Imap.FolderProperties instead of a complete Imap.Folder
and update call sites, use folder id to properly specify target DB row
when doing the update rather than inferring it from remote folder's
name. Update update_remote_selected_message_count to use same impl as
new methods, remove update_remote_status_message_count since it is
unused.
* src/engine/imap-engine/imap-engine-minimal-folder.vala (MinimalFolder):
Remove now unused ImapDB.Account instance, update subclasses and
constructor call sites.
This might fix an issue where folders are duplicated in FolderTable in
the db are duplicated.
* src/engine/imap-db/imap-db-account.vala (Account::clone_folder_async):
Check the folder hasn't already been loaded before attempting to clone
it.
* src/engine/imap-engine/imap-engine-generic-account.vala (Account): Only
connect/disconnect to Imap.Account's ready signal when opening/closing
the account so we don't queue a UpdateRemoteFolders op before loading
local folders, but do queue one after a LoadFolders if the remote is
already ready.
(Account::update_remote_folders): Reset the update timer before
queuing the op in case one has been already been recently queued.
This implements new cache for ContactListStore.
ContactListStore is created only once per account when account is opened
with GearyController::open_account. It is destroyed in
GearyController::close_account.
ContactListStoreCache class is introduced to manage ContactListStore
instances.
ComposerWidget receives ContactListStoreCache instance instead of
ContactListStore directly in constructor.
To increase performance, backwards compatibility breaking changes are
introduced to Engine API Geary.ContactStore. Signals:
* Gee.ContactStore::contact_added(Contact)
* Gee.ContactStore::contact_updated(Contact)
are replaced with batch equivalents:
* Gee::ContactStore::contacts_added(Gee.Collection<Contact>)
* Gee::ContactStore::contacts_updated(Gee.Collection<Contact>)
Geary.ComposerWidget::load_entry_completions is no longer async as it does
not involve time consuming ContactListStore creation.
CONTACT_MARKUP_NAME column is removed from ContactListStore as it used to keep
state about highlighted areas of text. This is not possible anymore as
ContactListStore is shared between multiple ComposerWidgets.
Highlight implementation has been moved to Geary.ContactEntryCompletion instead.
Additionally contacts_loaded signal is emitted from Geary.ImapEngine.GenericAccount
and propagated to Geary.Account
Geary.ContactListStore sort function is set upon receiving contacts_loaded
signal instead of after initial contacts are loaded. This speeds up Geary
startup for users with long contact lists.
* src/engine/imap-db/imap-db-account.vala
(Account::populate_search_table_batch_async): total_unindexed was wildy
inaccurate, since there may be been messages missing from MessageTable,
MessageSearchTable, or both. When clamped to 0, but when there actually
were messages to index, this would trigger an assert in the search
progress meter. So for now instead use something wildly overinflated
but guaranteed to not trigger the assert.
Bug 776654.
* src/engine/imap-db/imap-db-folder.vala (Folder): Add
REQUIRED_FTS_FIELDS constant that specifys what are the required fields
for FTS.
* src/engine/imap-db/imap-db-account.vala
(Account::populate_search_table_batch_async): Ensure rows returned meet
REQUIRED_FTS_FIELDS requirements.
* src/engine/api/geary-account-information.vala (AccountInformation): Add
an id property, replace the string email prop with a
RFC822.MailboxAddress primary_mailbox property. Fix all uses of `email`
to use either id or primary_mailbox instead. Tidy up the source a bit
while I'm here.
* src/engine/api/geary-engine.vala (Engine): Add ::get_account method
that throws an error when the account is not found.
(Engine::create_orphan_account): Updated to us `id` rather than `email`
to referr to an account.
Fixes Bug 767291
* src/engine/imap-db/imap-db-account.vala (Geary.ImapDB.Account): Handle
translation of "me" differently depending on if it is "to/cc/bcc:me" or
"from:me". Needed for DE at least.
* src/engine/imap-db/imap-db-account.vala (Geary.ImapDB.Account): Move
search term operator transation to a static initialiser so it isn't
have to be reconstrcuted for every term. Ensure english operator names
and values are also accepted as well as translations so if the User
Guide has not also been translated, the user can still use what the
docs say. Fix some bugs causing to:*, from:* and *:me to stop working.
