When messages have been appended and removed from a folder, Geary has
no choice but to do a full UID scan of the server to figure out
exactly what's changed. Prior code use a FETCH command, but SEARCH is
a better choice because there's less overhead in a SEARCH result: only
the UIDs are returned and they may be returned in a single line or in
batches. Also, there's no reason to return those results in a sorted
set, which was driving up transaction time, so an unsorted set is
returned.
On my account, a full normalization of All Mail went from 8s to 1s
with this change.
Geary now offers "true" deletion of messages in Gmail. It does this
by moving the offending email to the Trash folder and then expunging
them from the Trash.
Fortunately, this works with the composer without changes, meaning
discarded drafts won't sit in the Trash for 30 days.
When connecting to any IMAP server while the local user's locale is
configured to be Turkish, Geary will mis-parse many of the IMAP
server's responses, leading to essentially a failed connection due to
state issues and more.
The problem is that some of the parsing code was using g_utf8_strdown
to convert received text to lowercase to perform case-insensitive
string comparisons. Turkish has multiple letter I's (dotted and
dotless), and when the UTF-8 code transformed it to lowercase, a
different UTF-8 code point was selected than the English/ASCII 'i'.
The solution is to explicitly use ASCII variants of string
transformation, comparison, and hashing to ensure 7-bit operations are
used throughout the IMAP and RFC822 stack. Further commits will
follow that enforce this a bit more, but this commit is sufficient to
correct the problem for our Turkish users.
Although not completely solved, this reduces database pauses
(along with prior commit improving parallelization of database) by
how ImapDB.Folder accesses the database. In particular, one code
path required an INNER JOIN that simply took too long (~8s) on large
databases, locking the entire db in the process. Tests and hacking
showed that, annoyingly, manually performing the INNER JOIN by
intersecting the results of two SQL transactions was more efficient.
In addition, creating/merging emails into the database is done in
chunks with a small pause between each chunk to allow other tasks
the opportunity to get to the database. Create/merge is one of the
most expensive operations on the database.
This optimizes the search query itself, and also how we get the results:
instead of turning around and selecting the full row for each resulting
id, we simply pull out the id and internal date and that becomes the new
SearchEmailIdentifier. This new class also lets us more properly order
EmailIdentifiers everywhere else.
Folder normalization is now much simpler, faster, and requires
less resources than prior implementation. Normalization affects
the ReplayOperations, so an interface was changed here.
Both these problems stemmed from conversations holding messages from
in-folder and out-of-folder, and that all messages in the search
folder are considered out-of-folder.
This is squashed commit (sorry -- we'll get better about maintaining a
clean history in collaborative branches in the future!) of a massive
amount of work from Jim and myself.
* EmailIdentifiers for normal (i.e. not outbox, etc.) emails are the
same regardless of which folder they came from
* New EmailStore interface to manipulate messages that reside in any
folder, without having to care what folders are open
* Relevant places that manipulate emails (e.g. the toolbar) have been
updated to use the new EmailStore interface
* Conversation and ImplConversation have been smooshed together
* Many, many more items and bugfixes related to the above points
The prior ImapEngine.EmailPrefetcher pulled messages one at a time to
prevent hogging the connection with lots of data (when a lot of new mail is
detected). This patch simplifies some of the preparation code and
pulls the mail down in 128K batches, meaning multiple small messages are
pulled at once, meaning quicker availability without blocking the connection
entirely.
This patch solves the following memory/resource leak problems:
(a) Gee.TreeSet doesn't drop references when destroyed. Fixed by
using a subclass that clears the set when destroyed (exactly same
as patch made to Gee, however that won't be in distribution for
awhile.)
(b) Imap.ClientSession was holding refs to CommandResponses after
they'd been completed. They're now dropped.
(c) Imap.ClientSessionManager now has an open/close_async() (called
by Imap.Account.open/close_async()) that drops all ClientSessions.
(d) All classes in Engine (and some in the client) use Geary.BaseObject,
which uses a static map to track outstanding held references to
it. The table is dumped when Geary exits. Must be enabled with a
./configure flag.
Two outstanding memory leaks persist (one for Imap.ResponseCodes and
another when messages are selected/deselected), so this doesn't close
the ticket, but testing and use has shown these changes to make a huge
improvement on memory usage and reducing crashes.
This is to assist in completing ticket #5128, where because this
signal is fired multiple times although no state has changed (due
to Gtk.TreeSelection's "changed" signal wierdness) it's causing
the entire UI to update state unnecessarily.
This ticket may also assist in fixing #5327, but it does not close
that bug.
Previously, Geary could sometimes request elements of an email
message it already had in the database. Often this was due to
the client requesting information already present as well as one
additional field; the old implementation required all the requested
fields be pulled down.
Now Geary will work out on a message-by-message basis what is
present and what is not and request only missing fields for each
one. It will cluster messages missing the same fields into the
same request and it submits all requests simultaneously, to take
advantage of IMAP pipelining.
This work has some orthogonal benefits toward #5224, better
Dovecot support, by making the upper layers able to merge an email
using both locally-fetched data and remotely-fetched data.
The crash inside of HashMap appears to be due to a recent change
I made in Geary.Converstions to optimize loading large folders.
This reverts Geary back to returning copies of its conversation pools.
