[ews] EWS throttling

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[ews] EWS throttling

Martin Kelly
Hi,

I'm new to Evolution, so I apologize if this topic has already been
covered. I'm using the EWS plugin and have noticed that, when syncing a
large amount of mail, it seems like the EWS plugin does not respond to
Exchange throttling messages. Specifically, I'm getting messages back
from the server with code ErrorServerBusy and a BackOffMilliseconds
value set. However, it seems like EWS is failing the transaction rather
than backing off. Note that I haven't done much code inspection (just
observed behavior and grepped for the string "BackOffMilliseconds" in
the sources), so I could be mistaken.

In any case, handling throttling would be very helpful for me when
processing a large mail sync. Is this difficult to add? Should I file a
bug report?

Here's more information about EWS throttling:
https://msdn.microsoft.com/en-us/library/office/jj945066(v=exchg.150).a
spx

Note that I'm using the Evolution debian package, version 3.22.6-1.

Thanks,
Martin
_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: [ews] EWS throttling

Milan Crha
On Tue, 2017-06-20 at 23:39 +0000, Martin Kelly wrote:
> Specifically, I'm getting messages back from the server with code
> ErrorServerBusy and a BackOffMilliseconds value set.

        Hi,
could you run evolution with EWS debugging on, to see the exact
response from the server, please? It can be done with:

   $ EWS_DEBUG=2 evolution

I'm wondering which policy the update exceeded and the error message
sometimes contains detailed information.

> However, it seems like EWS is failing the transaction
> rather than backing off.

Yes, can be. I hear about this for the first time, though it doesn't
mean much.

> In any case, handling throttling would be very helpful for me when
> processing a large mail sync.

What is the "large" here, like hundreds of messages from the last sync?
Even, it depends on the policy it exceeded.

> Is this difficult to add?

Possibly. Many things are done in a synchronous way, which means a
dedicated thread runs an operations and does not finish until it's
truly done. That "wait for result, eventually restart the query when
asked to" might be a little challenge to implement properly. I also
didn't check the code, it's only my first impression.

> Should I file a bug report?

Sure, you can. GNOME bugzilla, evolution-ews product:
https://bugzilla.gnome.org/enter_bug.cgi?product=evolution-ews

In case the policy has anything to do with number of connections, then
I'd try to turn off "Listen for server change notifications" in
Receiving Options tab of the mail account Properties. That option opens
another connection dedicated only for the notifications. Note that it
is not only about mail, the same applies to contacts, calendars, task
lists and memo lists - all of them can do an update at the same time,
from different processes.
        Bye,
        Milan
_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: [ews] EWS throttling

Martin Kelly
On Wed, 2017-06-21 at 07:43 +0200, Milan Crha wrote:

> On Tue, 2017-06-20 at 23:39 +0000, Martin Kelly wrote:
> > Specifically, I'm getting messages back from the server with code
> > ErrorServerBusy and a BackOffMilliseconds value set.
>
> Hi,
> could you run evolution with EWS debugging on, to see the exact
> response from the server, please? It can be done with:
>
>    $ EWS_DEBUG=2 evolution
>
I'm wondering which policy the update exceeded and the error message
> sometimes contains detailed information.
>

Sure, I've opened a bug and attached the log to it:

https://bugzilla.gnome.org/show_bug.cgi?id=784058

Unfortunately, I don't have the rest of the transaction logged.

> > However, it seems like EWS is failing the transaction
> > rather than backing off.
>
> Yes, can be. I hear about this for the first time, though it doesn't
> mean much.
>
> > In any case, handling throttling would be very helpful for me when
> > processing a large mail sync.
>
> What is the "large" here, like hundreds of messages from the last
> sync?
> Even, it depends on the policy it exceeded.
>
More than hundreds; thousands (importing a large amount of mail for the
first time). Something like 10,000 messages, I would guess.

> > Is this difficult to add?
>
> Possibly. Many things are done in a synchronous way, which means a
> dedicated thread runs an operations and does not finish until it's
> truly done. That "wait for result, eventually restart the query when
> asked to" might be a little challenge to implement properly. I also
> didn't check the code, it's only my first impression.
>
> > Should I file a bug report?
>
> Sure, you can. GNOME bugzilla, evolution-ews product:
> https://bugzilla.gnome.org/enter_bug.cgi?product=evolution-ews
>
> In case the policy has anything to do with number of connections,
> then
> I'd try to turn off "Listen for server change notifications" in
> Receiving Options tab of the mail account Properties. That option
> opens
> another connection dedicated only for the notifications. Note that it
> is not only about mail, the same applies to contacts, calendars, task
> lists and memo lists - all of them can do an update at the same time,
> from different processes.
> Bye,
> Milan
>
I will try this, thanks.
_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: [ews] EWS throttling

Milan Crha
On Wed, 2017-06-21 at 19:25 +0000, Martin Kelly wrote:
> <t:Value Name="BackOffMilliseconds">149545</t:Value>

        Hi,
thanks. I'm wondering what evolution-ews should do in such cases.
Getting this information to UI would not be the best thing, but seeing
that the server says to wait for almost two and a half minute it would
be pretty confusing for users to not see any update for that long
without any indication what there's going on in the background.

I'm moving to your bug report.
        Bye,
        Milan
_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: [ews] EWS throttling

Pete Biggs
On Thu, 2017-06-22 at 09:58 +0200, Milan Crha wrote:

> On Wed, 2017-06-21 at 19:25 +0000, Martin Kelly wrote:
> > <t:Value Name="BackOffMilliseconds">149545</t:Value>
>
> Hi,
> thanks. I'm wondering what evolution-ews should do in such cases.
> Getting this information to UI would not be the best thing, but seeing
> that the server says to wait for almost two and a half minute it would
> be pretty confusing for users to not see any update for that long
> without any indication what there's going on in the background.
>
I know it's not the idea that Evo is an Outlook clone, but I can assure
you that Outlook does just sit there with the "updating" progress bar
paused at times. Users just accept it because, well, it's Outlook and
it's sort of just how it is. I would be disappointed if Evolution were
to embrace that particular Outlook feature!  Can you give some form of
feedback saying something like "Download Paused by Server" and some
form of indication how long left to wait?

P.
_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: [ews] EWS throttling

Milan Crha
        Hi,

On Thu, 2017-06-22 at 10:02 +0100, Pete Biggs wrote:
> I would be disappointed if Evolution were to embrace that particular
> Outlook feature!

I agree.

> Can you give some form of feedback saying something like "Download
> Paused by Server" and some form of indication how long left to wait?

Yes, that's possible, for the message in the status bar, which can be
eventually cancelled by the user. There will be some caveats for sure,
but if some brave testers would be found...

        Bye,
        Milan
_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: [ews] EWS throttling

Martin Kelly
On Thu, 2017-06-22 at 11:40 +0200, Milan Crha wrote:

> Hi,
>
> On Thu, 2017-06-22 at 10:02 +0100, Pete Biggs wrote:
> > I would be disappointed if Evolution were to embrace that
> > particular
> > Outlook feature!
>
> I agree.
>
> > Can you give some form of feedback saying something like "Download
> > Paused by Server" and some form of indication how long left to
> > wait?
>
> Yes, that's possible, for the message in the status bar, which can be
> eventually cancelled by the user. There will be some caveats for
> sure,
> but if some brave testers would be found...
>
> Bye,

Another thing to note here is that, if we don't backoff, the user
experience is for the sync to error out with a cryptic message about
the server not being available. I don't think this is much better than
saying something about "server has stalled." I think the ideal thing is
to somehow proactively ask the server for policy and adjust accordingly
so that we never hit throttle limits. However, I don't know the
Exchange protocol and thus don't know if that kind of thing is
possible.
_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list