This should take care of the problem, exacerbated by multiple Inbox
folders being allowed in the root of the FolderTable. Much more
strict in this patch about folders named "Inbox".
Vector expansion was broken in that the unfulfilled map was being
populated with EmailIdentifiers lacking a message_id, only a UID
(from Imap.Folder). More rigorous unfulfilled map now in
AbstractListEmail that requires a UID, not an EmailIdentifier.
Also guarantees that the email is fetched exactly once per
transaction by mapping UID -> fields, then "flipping" the map
back to fields -> UIDs, maximizing the number of emails requested
per FETCH.
A recent optimization for EmailStore was to only open a connection
to the remote folder if a command came in requiring it. If an
ImapEngine.GenericFolder is opened for monitoring, however, no command
may come in requiring a remote connection, meaning no folder
normalization or unsolicited notifications arrive.
This patch "forces" the remote to open after a timeout, which is
cancelled if it's opened by a command.
Folder normalization is now much simpler, faster, and requires
less resources than prior implementation. Normalization affects
the ReplayOperations, so an interface was changed here.
Two problems: (1) Error was being thrown while mutex was held, causing
the SearchFolder to forever be locked. (2) INCOMPLETE_MESSAGE Error
was causing list operation to abort prematurely (which in turn was
due to #6604).
We were erroneously going into the database to find UIDs for messages we
just did vector expansion on, and thus hadn't yet recorded in the
database. With this patch, we use the UID from the EmailIdentifier
itself in that case.
Previously, the EmailIdentifier key's hash value was sometimes changing
during the execution of the function, so when someone later went to look
up the flags, the hash map was returning NULL. This way we delay
hashing the EmailIdentifier till right before it's used, so its hash
value can't change.
Both these problems stemmed from conversations holding messages from
in-folder and out-of-folder, and that all messages in the search
folder are considered out-of-folder.
The problem was due a timing issue in ClientConnection FSM; it was possible
for the machine to still be in the IDLE or IDLING state when ready to
SYNCHRONIZE. This is due to a timing hole where the server's response to
close IDLE (requested when the APPEND command was sent) didn't return in
time. This closes that timing hole and deals with SYNCHRONIZE occurring
while in IDLING/IDLE.
This is squashed commit (sorry -- we'll get better about maintaining a
clean history in collaborative branches in the future!) of a massive
amount of work from Jim and myself.
* EmailIdentifiers for normal (i.e. not outbox, etc.) emails are the
same regardless of which folder they came from
* New EmailStore interface to manipulate messages that reside in any
folder, without having to care what folders are open
* Relevant places that manipulate emails (e.g. the toolbar) have been
updated to use the new EmailStore interface
* Conversation and ImplConversation have been smooshed together
* Many, many more items and bugfixes related to the above points
We discovered a situation where, depending on the order email is
processed, it's possible for logically associated emails to form
two or more conversations. When an email is processed that belongs
in more than one conversation, its removal triggers this error (as
a Message-ID can be associated with one and only one conversation).
Bogus email headers, especially email that does not list a complete
References: list, aggravate this problem.
Charles discovered that some of the operations in ImapEngine.GenericFolder
were exiting before waiting for the command to complete, meaning the
async call returned while the operation was still running. This
has a broader problem if the command is executed (with a yield) and
then the folder is closed.
Account synchronizer now stops when it recognizes that all the mail in
the folder is present (but perhaps not completely downloaded) from
the server. The email prefetcher takes care of ther rest.
Also discovered a regression from last week's commit that prevented
vector expansion when using find_earliest_email_async(); fixed here
as well.
Encountered this morning -- list operation had unfulfilled emails
waiting for network operations but folder normalization determined
they were missing and used a writebehind to remove them from the
operation, but the loop to remove them explodes (can't do a remove
within a foreach). This loop solves that problem.
Two problems: (a) determination for vector expansion was not accounting
for INCLUDING_ID flag properly, and (b) messages added to the interior of
the vector during normalization (can happen when the database has a
gap in the middle of the vector) were being treated as appended (i.e. new)
messages, causing the ConversationMonitor to load all of them.
Forgot to deal with situation where a message in the remote folder
is removed that isn't stored in the local database. In that case,
the local position will be <= 0 and should be treated as a NOP
(although still signal that the total remote_count has changed).
Rather than attempt to be selective, there's enough changed here that we might as well
blow away the search index and let the indexer start afresh. Future tweaks to the
search index might need to be more selective.
