We've had reports on Matrix that existing batch transactions in
`create_or_merge_email_async` are taking up to 30s to run, which is
getting uncomfortably close to the 60s timeout.
Drop the batch size down from 25 to 10 to reduce the chance that this
will eventually fail with a "database is locked" error.
See #921 for context.
Although populating the search table had been broken up into batches
of 50 email, it was still search for and loading every single message
id in both the MessageTable and MessageSearchTable, doing a manual
join, and then updating the batch, for *each* batch, and in a RW
transaction.
Break this up so that the ids are loaded and joined only once, the
queries happens in a RO transaction, the manual join happens in a side
thread, leaving each RW transaction only having to load the messages
and update the search index for up to 50 messages.
Ensure all code paths that execute SQL are logging it if enabled, and
clean up the log formatting.
Add new `enable_result_logging` static property to control result
logging separately from SQL logging, so we can get SQL logged without
having to see email. Clean up result formatting, and keep track of the
current result row number and log that as context.
Since each context type already has access to the object that is its
context parent, don't bother with a stand-alone `logging_parent`
property, just have context types implement it and return the
appropriate object.
Ensure internal code can access a DatabaseConnection from context
objects to get access to connection-specific code, but make the
polymorphic context accessors internal so transactions can't access
them.
When constructing a new record, rather that getting the state object
from a source and hanging on to it, just keep a few salient details
and then release it, so we don't keep a ref to the source hanging
around. This also means we need to pre-fill well-known up front, adding
some overhead when not logging.
This fixes hitting resource limits like memory and file descriptors
when lots of objects are logged, as seen when logging is enabled for
the DB, and in particular for `DatabaseConnection` objects, since each
one represents an open SQLite database file and related state.
It also ruins the glorious future in which we could inspect logged
objects in the Inspector in the same way that Firefox allows you to
inspect objects in the JS console, but c'est la ve. The way forward
there is probably adding structured data to state objects, allowing
sources to provide a curated set of properties to log.
This isn't a good place to do it, but neither GLib's error model not the
engine's database model makes it possible to getting access to
transaction state otherwise at the moment.
Fix infinite recursion logging from DatabaseConnection caused by
`to_string` and Context implementing `to_logging_state` that calls that.
Make individual objects implement `Logging.Source` instead and have
them return appropriate, custom logging state individually.
Convert `Connection` into an interface, add two concrete implementations
that allow splitting up the database connection used generally, and the
connection passed to transactions. This allows limiting the API surface
that transactions have access to (so they can't e.g. create
sub-transactions) and perform transaction-specific work (e.g. better
logging when an error occurs).
The join used to determine how many messages should be left in the
folder was inefficient. The following query has also been performance
tested and found to be more efficient as a join (instead of a subquery).
Geary hasn't used compression for IMAP in a while now due to the
possibility of information leakage for secure connections, so remove
the command as well.
Add `cancelled_before_send` method to support explicitly cancelling the
command before it was sent, so that `wait_until_complete` does not end
up waiting forever when this happens. Call the new method from
`ClientConnection` as required.
Now that commands has a `should_send` cancellable property, check that
before queuing and before sending, so that cancelled commands never
actually get sent over the wire.
Since both submitting a command no longer requires a cancellable, and it
is desirable to avoid sending a queued command that has already been
cancelled beforehand, add a new `Command.should_send` Cancellable
property to specify if a command should (still) be sent or not, and stop
passing a cancellable to ClientSession when submitting commands.
Allow call sites to pass in existing cancellable objects, and thus also
add it it as a ctor property to the Command class and all subclasses.
Lastly, throw a cancelled exception in `wait_until_complete` if send
was cancelled so that the caller knows what happened.
Remove redundant cancellable argument from
`Imap.Client.command_transaction_async` and rename it to
`submit_command` to make it more obvious about what it does.
The IMAP protocol does not allow commands to be cancelled once sent over
the wire to the server. However `IMAP.Command` objects are currently
submitted to be send with a cancellable, which may be cancelled before
the response is received from the server. If this happens, the repsonse
to the command from the server is not waited for, and untagged data from
the cancelled command may end up being handled by the subsequent
command, which is bad.
After submitting a command to the connection, don't supply the given
cancellable to the command's wait method, so it hangs around to handle
any tagged and untagged responses as required.
