Previously it was possible for the EXPUNGED server data's positional
addressing to be converted to the wrong EmailIdentifier, leading to
the wrong email being deleted, while the one the user asked for
was reported as missing as part of the ReplayQueue's local operation.
Problem was due to accumulation of messages marked for removal on
local side not being (eventually) removed from MessageLocationTable.
Can happen if folder is closed before EXPUNGE server data arrives,
or if Geary is shut down or crashes before same. Folder
normalization will deal with situation where locally removed
message was not actually removed on server.
Previously, Geary could sometimes request elements of an email
message it already had in the database. Often this was due to
the client requesting information already present as well as one
additional field; the old implementation required all the requested
fields be pulled down.
Now Geary will work out on a message-by-message basis what is
present and what is not and request only missing fields for each
one. It will cluster messages missing the same fields into the
same request and it submits all requests simultaneously, to take
advantage of IMAP pipelining.
This work has some orthogonal benefits toward #5224, better
Dovecot support, by making the upper layers able to merge an email
using both locally-fetched data and remotely-fetched data.
This also closes#5079, which I believe was caused by #4781 (and
could reproduce once isolated).
This commit may also close bug #5189, but need to verify further
on reporter's machine before closing.
These problems are all related to emails lacking a Message-ID in
their header, common with spam and mass-emailers.
This appears to be a timing issue between a message being deleted
and a message arriving at approx. the same time. The new event
handler dealing with incoming mail is not so strict and uses UID
addressing to be more generous in scooping up any new email than
it has in the local store.
*Believe* this to be fixed, but as this is so difficult to repro
and diagnose, there may still be some cases out there.
This rather large patch makes Geary.Conversations now responsible
for reestablishing a connection with the server if it ever drops
due to error. Once reestablished, Conversations will resynchronize
and report any new messages to the client. To the client, it's
invisible.
Working on this revealed problems with session teardown and Folder
state issues, as well as a memory leak. These are all fixed in
the patch.
There remains other connection timeout problems when the network is
unavailable, but will need to be addressed in a separate patch.
In addition, some of the work done in the old GenericImapFolder is
now broken out into separate components that monitor the folder
to do their work (the prefetcher and the email flags watcher). This
should make their code easier to maintain and understand.
This decouples the IMAP and Sqlite units from the Geary.Folder and
Geary.Account interface. Geary.EngineFolder will no longer deal
with generic local and remote folders, rather it is built knowing
it is dealing with IMAP.
This is necessary because of the IMAP engine changes coming, that
is dealing with servers that return server data out of order,
and even complete requests out of order.
This implements a background prefetch of all normalized messages
in the folder. It does not implement the chunked prefetch scheme
we discussed. It uses a simple priority scheme as well, only
pulling one message at a time, although it may be worthwhile
later implementing a pull-back system where the prefetch waits
until higher-priority traffic completes before continuing.
This incorporates much better duplicate detection in the local database, using both RFC-822 Message-ID as well as IMAP metadata (internaldate, RFC822 size) to determine if a message is already stored in the database. Very useful when a message is stored in multiple folders, or an already-downloaded message is returned to a folder it originated in (i.e. INBOX).
Also some minor fixes to listing email by EmailIdentifier which save a roundtrip to the server for certain edge cases.
Conversations now fully monitor the Folder for both additions and removals of messages.
This exposed a couple of bugs in the database code, which are fixed here as well. I also
used this opportunity to clean up some of the internal Conversations code, simplifying Node
management. #4333 will be a big boost here when implemented.
addressing.
Positional addressing is a nightmare for a lot of reasons, especially keeping positions
up-to-date as the Folder mutates. Now, positions are returned with the email but for
advisory reasons only, and do not keep up-to-date. It is expected that a client will use
positional addressing to "bootstrap" the application's data store, then use EmailIdentifier
addressing to traverse the Folder's contents.
Prior commit did not completely fix the problem in the case where a remote folder open took
a long time. This bulletproofs the solution, but does mean that there will be some
situations where FAST messages return with EmailLocations that radically change in the near
future. As the contract allows for any changes whatsoever, this is acceptable.
The entire database module now uses Transactions in order to guarantee atomicity of all
operations. However, due to limitations in SQLHeavy and its implementation of async, we
can't use it (and SQLite's) transactional models. This patch introduces a rather hamhanded
transactional model where the entire database is locked for each request using a
NonblockingMutex. This is the safest approach, but certainly not an optimal one. When
SQLHeavy's code is in place, hopefully we can rip this out with a minimum of fuss.
This adds a new flag when listing messages, FAST. This indicates that the caller wants
messages that are immediately available to the Folder, avoiding a round-trip to the server
(or even disk) if possible. Not all folders will support FAST, but it can be used (as it is
now in the client) to quickly populate the message list and then initiate a connection in
the background to get the straight dope.
This commit finishes the second half of #3793 by detecting when messages have been deleted
(or moved out of) an open folder and notifying the system of the change. The nonblocking
synchronization primitives have been beefed up and may be broken out to a separate library
some day.
This commit also introduces the ReplayQueue, which replays events that occur on the server
locally. This class will probably become much more important as time goes on, perhaps used
to store user events that are replayed on the server as well.
This commit represents the first half of this ticket, as it only monitors additions (new
emails) in the folder. A second commit will follow for monitoring removals (deleted emails)
in the folder.
The commit for #3851 changed the semantics of how messages are stored in the local cache,
and not all the code was properly updated to recognize that, particularly in EngineFolder.
This patch also introduces EmailIdentifier, which allows for a mail message to be fetched by
a unique identifier rather than its position in the folder list. This is much more
efficient and less error-prone, and means that Email objects don't have to be updated if
their position changes (although there are still some questions in that area). For IMAP,
this means being able to use the message's UID.
Caused by a valac bug set off by a generic function designed to convert a Gee List to a Vala
array. Rather than fight it I've removed the function and added to FetchCommand a
constructor to build the command using a Gee Collection.
The SQLite Geary.Folder implementation now uses reference semantics, so when it's reselected
the same Folder object is used and in-memory state is consistent. Also found an off-by-one
error when first querying an IMAP Folder that caused the most recent new email not to be
downloaded the first time. Fixed as well in this commit.
Needed to rethink storage strategies as I researched this and realized that a true scarce database -- where the database is sparsely populated both in columns and rows -- is not feasible due to IMAP's UID rules. The strategy now means that the database rows are contiguous from the highest (newest) message to the oldest *requested by the user*. This is a better situation than having to download the UID for the entire folder.
This commit adds support for IMAP-specific properties, of which UIDValidity is crucial toward completing #3805. The additional code is to integrate these tables into the SQLite Geary backend and to make sure this information is requested from the IMAP server.
NOTE: This commit changes the database schema. Old databases will need to be blown away before running.
Fields added. However, it would be nice to use formatting to separate them from the body of the message. This is not easy with Gtk.TextView/Gtk.TextBuffer; see https://bugzilla.gnome.org/show_bug.cgi?id=59390. This may push us to move to Webkit earlier rather than later.
Now supporting folder heirarchies. The client will now descend looking for subfolders. This task now opens up multiple outstanding requests to the Engine as well as exercises the database schema.
Closing this ticket opens the door to finishing #3692.
This commit introduces lazy loading of folder contents, which allows the Engine to report back email in chunks rather than all at once (which might require a round-trip to the server). This allows for the local database results to be returned to the caller right away while background I/O is occuring.
The code base is growing much faster than expected, faster than Shotwell it seems (not necessarily line count, but files and necessary organization of the library vs. Shotwell's initial flat directory). After some thought decided to move to a more standard Vala/GTK naming scheme of all lowercase with dashes for spaces starting with namespace (minus the "geary-", unless the class was in the topmost namespace). Three motivations:
1. Often confusing when working on code to see three "Folder.vala" in the gedit tabs: one IMAP, one SQLite, and one the interface definition.
2. This paves the way for waf integration, as right now we're held up using it because it barfs on projects with two files of the same name in different directories.
3. I find the CamelCase in the file browser becoming hard on the eyes, and this scheme seems a little more browsable.
2011-06-27 11:24:39 -07:00
Renamed from src/engine/sqlite/api/Folder.vala (Browse further)
