The entire database module now uses Transactions in order to guarantee atomicity of all
operations. However, due to limitations in SQLHeavy and its implementation of async, we
can't use it (and SQLite's) transactional models. This patch introduces a rather hamhanded
transactional model where the entire database is locked for each request using a
NonblockingMutex. This is the safest approach, but certainly not an optimal one. When
SQLHeavy's code is in place, hopefully we can rip this out with a minimum of fuss.
This adds a new flag when listing messages, FAST. This indicates that the caller wants
messages that are immediately available to the Folder, avoiding a round-trip to the server
(or even disk) if possible. Not all folders will support FAST, but it can be used (as it is
now in the client) to quickly populate the message list and then initiate a connection in
the background to get the straight dope.
This commit finishes the second half of #3793 by detecting when messages have been deleted
(or moved out of) an open folder and notifying the system of the change. The nonblocking
synchronization primitives have been beefed up and may be broken out to a separate library
some day.
This commit also introduces the ReplayQueue, which replays events that occur on the server
locally. This class will probably become much more important as time goes on, perhaps used
to store user events that are replayed on the server as well.
This commit represents the first half of this ticket, as it only monitors additions (new
emails) in the folder. A second commit will follow for monitoring removals (deleted emails)
in the folder.
The commit for #3851 changed the semantics of how messages are stored in the local cache,
and not all the code was properly updated to recognize that, particularly in EngineFolder.
This patch also introduces EmailIdentifier, which allows for a mail message to be fetched by
a unique identifier rather than its position in the folder list. This is much more
efficient and less error-prone, and means that Email objects don't have to be updated if
their position changes (although there are still some questions in that area). For IMAP,
this means being able to use the message's UID.
Caused by a valac bug set off by a generic function designed to convert a Gee List to a Vala
array. Rather than fight it I've removed the function and added to FetchCommand a
constructor to build the command using a Gee Collection.
The SQLite Geary.Folder implementation now uses reference semantics, so when it's reselected
the same Folder object is used and in-memory state is consistent. Also found an off-by-one
error when first querying an IMAP Folder that caused the most recent new email not to be
downloaded the first time. Fixed as well in this commit.
Needed to rethink storage strategies as I researched this and realized that a true scarce database -- where the database is sparsely populated both in columns and rows -- is not feasible due to IMAP's UID rules. The strategy now means that the database rows are contiguous from the highest (newest) message to the oldest *requested by the user*. This is a better situation than having to download the UID for the entire folder.
This commit adds support for IMAP-specific properties, of which UIDValidity is crucial toward completing #3805. The additional code is to integrate these tables into the SQLite Geary backend and to make sure this information is requested from the IMAP server.
NOTE: This commit changes the database schema. Old databases will need to be blown away before running.
Fields added. However, it would be nice to use formatting to separate them from the body of the message. This is not easy with Gtk.TextView/Gtk.TextBuffer; see https://bugzilla.gnome.org/show_bug.cgi?id=59390. This may push us to move to Webkit earlier rather than later.
Problem was that the client fetches folder objects for the special folders (Inbox, Drafts, etc.) prior to fetching the full folder list and the Engine's fetch_folder_async() call wasn't prepared to handle fetches for folders not located in the local database.
Now supporting folder heirarchies. The client will now descend looking for subfolders. This task now opens up multiple outstanding requests to the Engine as well as exercises the database schema.
Closing this ticket opens the door to finishing #3692.
This commit introduces lazy loading of folder contents, which allows the Engine to report back email in chunks rather than all at once (which might require a round-trip to the server). This allows for the local database results to be returned to the caller right away while background I/O is occuring.
The code base is growing much faster than expected, faster than Shotwell it seems (not necessarily line count, but files and necessary organization of the library vs. Shotwell's initial flat directory). After some thought decided to move to a more standard Vala/GTK naming scheme of all lowercase with dashes for spaces starting with namespace (minus the "geary-", unless the class was in the topmost namespace). Three motivations:
1. Often confusing when working on code to see three "Folder.vala" in the gedit tabs: one IMAP, one SQLite, and one the interface definition.
2. This paves the way for waf integration, as right now we're held up using it because it barfs on projects with two files of the same name in different directories.
3. I find the CamelCase in the file browser becoming hard on the eyes, and this scheme seems a little more browsable.
This completes the heavy lifting of persisting messages locally. The strategy is that the local database may be sparsely populated, both in the availability of messages in a folder and the fields of a message that is partially stored. As data is pulled from the remote server it's always stored in the database. Future requests will always go to the database first, preventing unnecessary network traffic.
Also, this patch will detect when a message is stored in multiple folders on the server. The database uses soft links from the folder to the message, so the message is stored only once in the database. This technique relies heavily on the availability and validity of the Message-ID header, but we expect this to be reliable the vast majority of the time.
This iteration now stores headers locally and fetches them first before going to the network. Work done in the database to deal with IMAPisms. More work on the GMime bindings (couple of mistakes in prior commit).
Much of the API between the local and net stores had to be reworked for consistency as well as planning ahead for how messages will be retrieved and stored efficiently. This work also attempts to keep in mind that other mail sources (POP, etc.) may be required in the future, and hopefully can be added without major rework.
Engine now uses a master Folder object that synchronizes between the network and the local store. Also, the Geary.Folder interface was getting ugly to code when the implementation was more than trivial, so moved to standard getters for it.
This large diff represents a growth of the architecture to persist IMAP data as its downloaded. When listing folders, a local database is consulted first to immediately feed to the caller. In the background, network calls fetch the "real" list. The two are collated for differences which are reported to the caller via signals, who are then responsible for updating the user interface appropriately. No other synchronization work is represented in this diff.
Note that this breaks functionality: when a folder is selected, no messages appear in the message list. Fixing this requires more work, and this patch was already large enough. It's ticketed here: #3741
