IMAPx and message body search

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

IMAPx and message body search

Pete Biggs

Evo 3.36.3

Has something changed recently with searches?  A while ago I turned on
indexing on my Dovecot IMAP server and, as expected, it massively
increased the speed of "Message contains" searches. Recently though the
searches have returned to being deathly slow - Dovecot is still
indexing the mail.

I ran debugging output on the IMAP exchange and it looks like it is
stepping through each message rather than asking Dovecot to do the
search (I searched for "printer"):

A00076 OK [READ-WRITE] Select completed (0.001 + 0.000 secs).'
[imapx:A] I/O: 'A00078 UID SEARCH UID 147137 BODY printer'
[imapx:A] I/O: '* SEARCH
A00078 OK Search completed (0.020 + 0.000 + 0.019 secs).'
[imapx:A] I/O: 'A00079 UID SEARCH UID 147318 BODY printer'
[imapx:A] I/O: '* SEARCH
A00079 OK Search completed (0.018 + 0.000 + 0.017 secs).'
[imapx:A] I/O: 'A00080 UID SEARCH UID 146691 BODY printer'
[imapx:A] I/O: '* SEARCH
A00080 OK Search completed (0.018 + 0.000 + 0.017 secs).'
[imapx:A] I/O: 'A00081 UID SEARCH UID 146000 BODY printer'
[imapx:A] I/O: '* SEARCH

there's about 150k messages in the folder and at 0.017s per message, it
will take 42 mins to complete.  The CAPABILITY response on that account
is:

[imapx:A] I/O: 'A00015 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS
 ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSU
BJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-
EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SE
ARCH LIST-STATUS SPECIAL-USE BINARY MOVE] Logged in'

Is this an Evolution problem or is it something up with Dovecot - i.e.
is it advertising the correct capabilities that Evolution uses for
offloading the search? Dovecot was upgraded to 2.3.10 from 2.3.9 back
in March (this is on a CentOS 7 system), but I think the problem
started after that.

P.



_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: IMAPx and message body search

Patrick O'Callaghan
On Fri, 2020-07-24 at 23:22 +0100, Pete Biggs wrote:

> Evo 3.36.3
>
> Has something changed recently with searches?  A while ago I turned on
> indexing on my Dovecot IMAP server and, as expected, it massively
> increased the speed of "Message contains" searches. Recently though the
> searches have returned to being deathly slow - Dovecot is still
> indexing the mail.
>
> I ran debugging output on the IMAP exchange and it looks like it is
> stepping through each message rather than asking Dovecot to do the
> search (I searched for "printer"):
>
> A00076 OK [READ-WRITE] Select completed (0.001 + 0.000 secs).'
> [imapx:A] I/O: 'A00078 UID SEARCH UID 147137 BODY printer'
> [imapx:A] I/O: '* SEARCH
> A00078 OK Search completed (0.020 + 0.000 + 0.019 secs).'
> [imapx:A] I/O: 'A00079 UID SEARCH UID 147318 BODY printer'
> [imapx:A] I/O: '* SEARCH
> A00079 OK Search completed (0.018 + 0.000 + 0.017 secs).'
> [imapx:A] I/O: 'A00080 UID SEARCH UID 146691 BODY printer'
> [imapx:A] I/O: '* SEARCH
> A00080 OK Search completed (0.018 + 0.000 + 0.017 secs).'
> [imapx:A] I/O: 'A00081 UID SEARCH UID 146000 BODY printer'
> [imapx:A] I/O: '* SEARCH
>
> there's about 150k messages in the folder and at 0.017s per message, it
> will take 42 mins to complete.  The CAPABILITY response on that account
> is:
>
> [imapx:A] I/O: 'A00015 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS
>  ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSU
> BJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-
> EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SE
> ARCH LIST-STATUS SPECIAL-USE BINARY MOVE] Logged in'
>
> Is this an Evolution problem or is it something up with Dovecot - i.e.
> is it advertising the correct capabilities that Evolution uses for
> offloading the search? Dovecot was upgraded to 2.3.10 from 2.3.9 back
> in March (this is on a CentOS 7 system), but I think the problem
> started after that.

I access Gmail via IMAP (I don't sync the contents locally) and
searching is almost instantaneous, so that would seem to point to
Dovecot being the problem.

poc

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: IMAPx and message body search

Pete Biggs

> > [imapx:A] I/O: 'A00015 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS
> >  ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSU
> > BJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-
> > EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SE
> > ARCH LIST-STATUS SPECIAL-USE BINARY MOVE] Logged in'
> >
> > Is this an Evolution problem or is it something up with Dovecot - i.e.
> > is it advertising the correct capabilities that Evolution uses for
> > offloading the search? Dovecot was upgraded to 2.3.10 from 2.3.9 back
> > in March (this is on a CentOS 7 system), but I think the problem
> > started after that.
>
> I access Gmail via IMAP (I don't sync the contents locally) and
> searching is almost instantaneous, so that would seem to point to
> Dovecot being the problem.
>
Thanks. You mentioning Gmail reminded me that I still had a number of
other accounts active when I was doing the tests - I think I got the
CAPABILITY strings mixed up.  I turned the others off and this is the
only capability I'm getting now:

[imapx:A] I/O: 'A00002 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS
ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS
THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT
CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC
ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE
SNIPPET=FUZZY PREVIEW=FUZZY STATUS=SIZE LITERAL+ NOTIFY SPECIAL-USE]
Logged in'

So what triggers Evo to use server side searches rather than searching
every message individually?

P.


_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: IMAPx and message body search

Patrick O'Callaghan
On Sat, 2020-07-25 at 13:56 +0100, Pete Biggs wrote:

> > > [imapx:A] I/O: 'A00015 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS
> > >  ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSU
> > > BJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-
> > > EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SE
> > > ARCH LIST-STATUS SPECIAL-USE BINARY MOVE] Logged in'
> > >
> > > Is this an Evolution problem or is it something up with Dovecot - i.e.
> > > is it advertising the correct capabilities that Evolution uses for
> > > offloading the search? Dovecot was upgraded to 2.3.10 from 2.3.9 back
> > > in March (this is on a CentOS 7 system), but I think the problem
> > > started after that.
> >
> > I access Gmail via IMAP (I don't sync the contents locally) and
> > searching is almost instantaneous, so that would seem to point to
> > Dovecot being the problem.
> >
> Thanks. You mentioning Gmail reminded me that I still had a number of
> other accounts active when I was doing the tests - I think I got the
> CAPABILITY strings mixed up.  I turned the others off and this is the
> only capability I'm getting now:
>
> [imapx:A] I/O: 'A00002 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS
> ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS
> THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT
> CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC
> ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE
> SNIPPET=FUZZY PREVIEW=FUZZY STATUS=SIZE LITERAL+ NOTIFY SPECIAL-USE]
> Logged in'
>
> So what triggers Evo to use server side searches rather than searching
> every message individually?

No idea, but maybe compare that set with the capabilities when only the
Gmail IMAP account is enabled.

poc

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: IMAPx and message body search

Gnome Evolution - General mailing list
In reply to this post by Pete Biggs
On Fri, 2020-07-24 at 23:22 +0100, Pete Biggs wrote:
> Evo 3.36.3
>
> Has something changed recently with searches?

        Hi,
yes, there was made a change with searches recently, an aim to make
them quicker. It helped (I'd say) on most of the searches, but not on
all. The Evolution(-data-server) 3.36.4 has this fixed. It had been
submitted for stable 4 days ago in your Fedora 32. I guess the most of
the mirrors, if not all of them, have it available already.
        Bye,
        Milan

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: IMAPx and message body search

Pete Biggs

> yes, there was made a change with searches recently, an aim to make
> them quicker. It helped (I'd say) on most of the searches, but not on
> all. The Evolution(-data-server) 3.36.4 has this fixed. It had been
> submitted for stable 4 days ago in your Fedora 32. I guess the most of
> the mirrors, if not all of them, have it available already.
>

Yes, just done an update and got 3.36.4 - searches back to being very
quick on large folders.  Thanks.

P.


_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list