Commit graph

296 commits

Author SHA1 Message Date
Michael Gratton
f560707271 Geary.FtsSearchQuery: Fixes for email text disjunctions
Ensure OR is actually used to separate disjuncts, don't prefix match
non-stemmed terms when a stemmed version exists or when EXACT is
specified. Ensure column is correctly specified per disjunct.
2021-01-19 20:48:17 +11:00
Michael Gratton
20a4fd3ed2 Geary.ImapDb.SearchQuery: Rename to Geary.FtsSearchQuery
There's nothing IMAP-specific about the class, so move it to common
and rename to reflect what it is actually used for.
2021-01-19 20:48:17 +11:00
Michael Gratton
2ab7b2c39e Geary.ImapDb.SearchQuery: Handle queries with some/all negated terms
SQLite FTS5 doesn't allow all negated terms in matches, and since NOT
is a binary operator (rather than unary) negated terms should be listed
after any positive terms.
2021-01-19 20:48:17 +11:00
Michael Gratton
6ffbfcf5d4 Geary.ImapDb.SearchQuery: Handle folder and deleted message exclusions
Ensure designated excluded folders, folderless messages, and
marked-for-deletion messages are excluded from FTS search results.
2021-01-19 20:48:17 +11:00
Michael Gratton
6a614adf73 Geary.ImapDb.SearchQuery: Use expression to generate FTS5 queries
Move SQL generation for FTS search from ImapDb.Account to SearchQuery.
Convert to use Geary.SearchQuery.Term instances to generate SQL, rather
than parsing the expression. Simplify the generated SQL substantially
and generate MATCH values that work with SQLite FTS5.
2021-01-19 20:48:17 +11:00
Michael Gratton
4fe0d92147 engine: Convert from SQLite FTS3/4 to FTS5 for full-text-search
Add SQL migration that drops the old FTS4 MessageSearchTable table,
re-create as a FTS5 table, clean up the column names a bit, and adds a
flags column so unread/starred queries can be made fast.

Define a SQLite FTS5 extension function `geary_matches()` to replace
the FTS3 `offsets()` function which no longer exists in FTS5, based on
Tracker's implementation.

Update code to FTS5 conventions (docid -> rowid, etc), use new column
names, populate and update the flags column as the email's flags
change, and use new match function for getting matching tokens.

Advanced searches are probably currently broken, these will be fixed
by subsequent commits.
2021-01-19 20:48:17 +11:00
Michael Gratton
ec3057daf7 ConversationWebView: Fix plain text emails sometimes being too wide
Using `whitespace: pre-wrap` to format plain text email sometimes
causes additional width to be allocated by the plain text blocks that
then does not get used due to the constraints on the HTML element.
The allocated space remains however and hence an un-needed horizontal
scrollbar appears.

Using `break-spaces` instead seems to help since it allows breaks after
a space character, leading to the additional space not otherwise being
allocated.
2020-10-17 15:08:10 +11:00
Michael Gratton
4f9df7d74a engine: Remove in-tree unicodesn stemmer
Now that search is using libstemmer, the in-tree stemmer is no longer
needed and can be removed.
2020-10-14 20:05:15 +11:00
Michael Gratton
41be8693d4 Geary.Imap.ClientSession: Treat logout as disconnect
Convert `get_protocol_state` to an automatic property, so that rather
than requiring explcit signals for lifecycle events, a GObject notify
signal can be used instead.

Convert disconnect signal to a property so it can be accessed if needed.

Convert code listening to the disconnect signal to listen to notify
signals for `protocol_state` instead, and hence also treat the session
as disconnected when a logout is in progress.

Fixes #986
2020-09-27 17:38:10 +10:00
Michael Gratton
85e9046c71 Geary.Imap: Make command cancellable a property of the command object
Since both submitting a command no longer requires a cancellable, and it
is desirable to avoid sending a queued command that has already been
cancelled beforehand, add a new `Command.should_send` Cancellable
property to specify if a command should (still) be sent or not, and stop
passing a cancellable to ClientSession when submitting commands.

Allow call sites to pass in existing cancellable objects, and thus also
add it it as a ctor property to the Command class and all subclasses.

Lastly, throw a cancelled exception in `wait_until_complete` if send
was cancelled so that the caller knows what happened.

Remove redundant cancellable argument from
`Imap.Client.command_transaction_async` and rename it to
`submit_command` to make it more obvious about what it does.
2020-09-02 14:34:41 +10:00
Michael Gratton
87340a9675 Merge branch 'mjog/955-special-use-unavailable-at-startup' into 'mainline'
Geary.Account.Information: Fix special use folder not correctly stored

See merge request GNOME/geary!569
2020-08-31 04:17:32 +00:00
Michael Gratton
39f687fcd3 Geary.Account.Information: Fix special use folder not correctly stored
The special use map needs custom hash and equality functions to
work correctly. Add these and some unit tests.

See #995
2020-08-30 18:26:26 +10:00
Michael Gratton
a64412e696 RFC822MesageTest: Fix locale-sensitive test case
Use `ascii_down()` instead of `down()` so the test case is more robust
in the face of being run under different locales.

Fixes #950
2020-08-29 18:33:15 +10:00
Chris Heywood
4fdaaeacd9 Fix bug with message retention count during GC plus fix test case 2020-08-24 19:29:44 +10:00
Michael Gratton
3ddce74844 sql: Add db migration to clean up message_ids with extra whitespace 2020-08-23 12:58:17 +10:00
Michael Gratton
eb3a8b0fe4 Geary.RFC822.MessageID: Replace GMime parser with more lax parser
Support parsing message IDs with the form `<add-spec>` (per the RFC)
but also `(add-spec)` and `add-spec`.

Add unit tests.
2020-08-22 16:23:07 +10:00
Michael Gratton
4e50822674 Geary.RFC822.MessageID: Ensure RFC822 value does not include errant wtsp
`to_rfc822_string` must not include any leading/trailing white space
even if it was included in the `from_rfc822_string` constructor.
2020-08-22 11:57:12 +10:00
Michael Gratton
67a2d32a6c Merge branch 'mjog/dovecot-envelope-mailbox-quirk' into 'mainline'
Dovecot envelope mailbox quirk

See merge request GNOME/geary!546
2020-08-18 07:07:35 +00:00
Michael Gratton
31f10e2787 Geary.Imap: Add quirk for Envelope address structure placeholders
Some servers (e.g. Dovecot) use placeholder strings instead of the empty
string (e.g. "MISSING_DOMAIN") in FETCH Envelope responses when a
required address part (e.g. local part, domain) are empty.

This adds a quirk that can be enabled for such servers to check for the
placeholders and if found, replace them with empty strings.
2020-08-18 16:34:24 +10:00
Michael Gratton
757c0542dc Geary.Imap.ServerResponse: Add quirks property
Add a quirks object as a property so code that parses the response has
access to it. Simplify constructing server responses slightly and pass
the IMAP connection's quirks when doing so.
2020-08-18 16:34:24 +10:00
Michael Gratton
ebf7a8ad1d Geary.Imap: Update IMAP quirks based on server greeting rather type
Use server greeting to update IMAP quirks, so that if e.g. a GMail
account is configured as a generic account, it still gets the right
quirks.
2020-08-18 16:34:24 +10:00
Michael Gratton
0d283dfc72 Geary.RFC822.MailboxAddresses: Update append methods
Rename append method to concatenate_list since that's what it actually does. Add new method
for cat'ing a single mailbox, add methods for merging both a single mailbox and mailbox
list into a new list.
2020-08-18 16:32:26 +10:00
Michael Gratton
1c22eb4ebd Geary.RFC822.MailboxAddress: Handle empty mailbox and domain parts better
If an address does not have an `@`, or if the IMAP constructor is called
with empty mailbox or domain parts, then these properties will be empty.
This is not uncommon, especially on UNIX hosts where system accounts
send email.

Ensure this is handled correctly in the constructors and are
round-tripped correctly by `to_rfc822_address` and hence
`to_rfc822_string`.
2020-08-18 16:14:37 +10:00
Michael Gratton
c73888b2c4 Geary.RFC822.MessageIDList: Support merging message id lists
Add support for merging a list with a single id or a another list,
only appending the new id(s) if not already present. Add unit tests.
2020-08-13 19:52:12 +10:00
Michael Gratton
2957f15297 Geary.RFC822.MailboxAddresses: Update equality semantics
Although when processing mailbox addresses lists are effectively sets,
the ordering can be important for people. As such, make `equal_to`
comparisons require identical ordering, and add new `contains_all` for
order-independent comparisons.
2020-08-13 19:51:33 +10:00
Michael Gratton
2030b2dec7 test: Break out engine mock objects into their own name-space
Engine mocks don't need to be in the `Geary` namespace, and including
them there makes it difficult to use them in client tests, so put them
all in their own name-space and corresponding directory.
2020-08-13 19:51:33 +10:00
Michael Gratton
175c7dc7f2 Geary.RFC822.MessageIDList: Add unit tests for parsing RFC822 id lists 2020-08-13 17:55:07 +10:00
Michael Gratton
33801f22ef sql: Add migration for bad message-id data in the database 2020-08-13 15:50:07 +10:00
Michael Gratton
ec4c6d3989 Geary.RFC822.Message: Fix get_body() critical with multipart messages
Fixes a bad assumption MR !534 made about GMime structure and causing
a crash on sending multipart messages. Add unit tests to cover this
case.
2020-08-03 20:14:32 +10:00
Michael Gratton
e2f2b6cdab Geary.RFC822.Message: Add accessors for message headers and raw body
Allow accessing Header and Text representations of the message.
2020-07-30 12:53:11 +10:00
Michael Gratton
16efce0514 Fix build failure 2020-06-30 18:25:51 +10:00
Michael Gratton
cfbac77fa4 Merge branch 'mjog/unit-test-subproject' into 'mainline'
Unit test subproject and cleanup

See merge request GNOME/geary!517
2020-06-30 08:08:54 +00:00
Michael Gratton
3550c69723 Merge branch 'mjog/server-quirks' into 'mainline'
Server quirks

See merge request GNOME/geary!508
2020-06-30 07:50:05 +00:00
Michael Gratton
4d6dad1cfa Merge branch 'mjog/746-gmail-flag-quote-bug' into 'mainline'
GMail flag quote bug

Closes #746

See merge request GNOME/geary!507
2020-06-30 07:46:17 +00:00
Michael Gratton
0ae633d88f Update existing tests to work with ValaUnit 2020-06-30 17:31:07 +10:00
Chris Heywood
71262f0e79 Merge branch 'mainline' into remove-old-msgs-beyond-storage-pref 2020-05-17 12:03:30 +10:00
Michael Gratton
c54ff85ccd Geary.RFC822.Message: Update API for SMTP formatting
Remove `Message.without_bcc` ctor since we can now get GMime to exclude
BCC headers when serialising, avoiding the overhead of taking a complete
copy of the message just to strip BCCs.

Rename `get_network_buffer` to `get_rfc822_buffer` to be a bit more
explicit in that's the way to get an actual RFC822 formatted message.
Replace book args with flags, and take a protocol-specific approach
rather than feature-specific, because in the end you're either going to
want all the SMTP formatting quirks or none of them.

Remove custom dot-stuffing support from Smtp.ClientConnection, since it
is redundant.
2020-05-06 14:58:56 +10:00
Michael Gratton
26e76b6f4d Geary.RFC822Error: Rename so it's actually in the RFC822 namespace 2020-05-06 14:58:56 +10:00
Michael Gratton
288c78a93a Geary.RFC822.Message: Rework constructors and stocking from GMimeMessage
Make constructors use the Message.from_gmime where possible. Merge that
and stock_from_gmime so it's clear the stocking only happens in the
ctor. Update stocking to use high-level GMime APIs where available to
avoid re-parsing header values.
2020-05-06 14:58:56 +10:00
Michael Gratton
6e7631b8d3 Geary.RFC822: Clean up message data interfaces and classes
Split Geary.RFC822.MessageData interface up into DecodedMessageData and
EncodedMessageData so the difference between the two is clear and they
can't be used interchangeably.

Add `DecodedMessageData::to_rfc822_string` to provide a common interface
for round-tripping decoded data.

Update all classes to implement one of these and follow the same general
API patterns.
2020-05-06 14:58:56 +10:00
Michael Gratton
5b253cbee6 Geary.RFC822.MessageIdList: Update API to match MailboxAddresses
Make immutable, provide similar properties and accessors as
MailboxAddresses.
2020-05-06 14:58:56 +10:00
Michael Gratton
940781a379 test/engine: Make RFC822 tests use their own subdirectory
Ensures they match the same source layout as in the engine.
2020-05-06 14:58:56 +10:00
Michael Gratton
e0c1fb8a80 Geary.RFC822: Ensure various data constructors throw errors
Rather than just silently ignoring error conditions, throw RFC822
errors instead.
2020-05-06 14:58:56 +10:00
Michael Gratton
6d5f63692b Geary.Mime.ContentType: Rename ::deserialise to ::parse
Add unit tests.
2020-05-05 21:57:17 +10:00
Michael Gratton
d6d36768f9 Geary.Imap.Deserialiser: Handle reserved chars in response-text
RFC 3501 allows any kind of char except CRLF in `resp-text` after an
optional response code, so handle that.

Addresses another issue in #711
2020-05-04 10:54:20 +10:00
Michael Gratton
cff26e2501 Geary.Imap.Deserializer: Workaround GMail not quoting IMAP email flags
GMail doesn't seem to quote IMAP email flags when they contain spaces
or `]`, a reserved character under RFC 3501. We can't do anything
about the former, but we can work around the latter.

This patch adds a quirk allowing `]` in IMAP flags in general for
GMail accounts, and adds some additional unit tests to that end.
2020-05-04 10:53:31 +10:00
Michael Gratton
ddb3a899fb Geary.Imap: Add Quirks object to collect all IMAP service quirk config 2020-05-04 10:53:31 +10:00
Chris Heywood
4ff783b231 Merge branch 'mainline' into remove-old-msgs-beyond-storage-pref 2020-04-30 18:08:19 +10:00
Michael Gratton
991c9ef5d7 Geary.App.ConversationMonitor: Fix more races when starting/stopping
Ensure that if stopping the conversation monitor while it is waiting
for the folder to open that the folder isn't closed, and if an error
is thrown when opening the folder that it gets thrown by
::start_monitoring anyway.

Hopefully fixes #741
2020-04-24 15:30:04 +10:00
Michael Gratton
81a22175ab Geary.FolderMock: Convert to use new async mock calls. 2020-04-24 15:22:55 +10:00