Migrating from Thunderbird via import without duplicates

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Migrating from Thunderbird via import without duplicates

Gnome Evolution - General mailing list
I understand how to import my mbox files from Thunderbird and I
understand how to setup my accounts on Evolution, but I'm not sure how
to do both without ending up with duplicates and essentially mitigating
the entire advantage of having local copies to import from.

---------------------------------------------------------------
Here is what I understand how to do now

1.) Import my "INBOX" mbox file to some new local folder (e.g. fooBox)

2.) Add existing mail account (e.g. [hidden email]) on Evolution.

3.) Move all mail from fooBox to [hidden email]->Inbox

4.) After step 2 is appears Evolution automatically syncs with the
bar.com server and downloads all the mail I had in fooBox already. After
step 3 now I have duplicates of everything and had to wait for all mail
to download from bar.com. I can remove these duplicates, but this makes
the import step seem useless. Is there any way to avoid downloading all
mail from the server and use the local copies, but only download those
not present in the local folder so as to save a bunch of bandwidth and
time?

---------------------------------------------------------------
Here is what I would expect to happen:

1.) Create a mail account (e.g. [hidden email]) on Evolution but have some
way to prevent it from automatically syncing

2.) Import my "INBOX" mbox file to Evolution in the [hidden email] Inbox folder

3.) Enable sync for the [hidden email] account

4.) Evolution checks the file list on bar.com and imports only those
that do not already exist in the local folder

---------------------------------------------------------------

Is there any way of doing the above?

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Gnome Evolution - General mailing list
On Tue, 2020-10-27 at 00:37 +0900, christopher via evolution-list
wrote:
> Is there any way of doing the above?

        Hi,
there basically isn't. Or it is, but it requires a lot of work, with
which it would be too easy to break internal things.

I suppose your server account is IMAP or any such, nothing stored
locally. Evolution (or better libcamel) stores a local copy of the
summary information (basically what you see in the message list) for
each folder of that account in some format, plus it stores also already
downloaded copies of the messages. Constructing the summary file from
an .mbox file is possible, but it's really complicated process and it
can break things. After all, those are private application data.

You'll be much safer letting Evolution to create the summary on its
own. Just add the remote account and that's all.
        Bye,
        Milan

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Gnome Evolution - General mailing list
Then what exactly is the use case for importing from Thunderbird as per
[the
manual](https://help.gnome.org/users/evolution/stable/import-apps-mozilla.html.en)?


As I have between 50 - 100 GB of mail to sync, with it all backed up
locally, I'm happy to try to break things in an effort to avoid days if
not weeks of re-syncing. Are there details on this process (warnings
heeded) anywhere?

I have access to both the mail server and local machine for all but my
gmail accounts. I'll test the process on less important demo accounts as
well.

I guess I could copy my local backups to the evolution maildir, ssh into
my mail server and rsync the maildirs. I'd hope it'd recognize there to
be no need to resync and duplicate everything at that point, but maybe
others have tried this with failure?

I have no idea how I'd approach this for the gmail accounts though as I
can't rsync or ssh into the gmail server. Any hints in the right
direction and I'm happy to tinker and break things! Thats why I have
backups! :)

Best regards,
Christopher

On 10/27/20 01:29, Milan Crha via evolution-list wrote:

> On Tue, 2020-10-27 at 00:37 +0900, christopher via evolution-list
> wrote:
>> Is there any way of doing the above?
> Hi,
> there basically isn't. Or it is, but it requires a lot of work, with
> which it would be too easy to break internal things.
>
> I suppose your server account is IMAP or any such, nothing stored
> locally. Evolution (or better libcamel) stores a local copy of the
> summary information (basically what you see in the message list) for
> each folder of that account in some format, plus it stores also already
> downloaded copies of the messages. Constructing the summary file from
> an .mbox file is possible, but it's really complicated process and it
> can break things. After all, those are private application data.
>
> You'll be much safer letting Evolution to create the summary on its
> own. Just add the remote account and that's all.
> Bye,
> Milan
>
> _______________________________________________
> evolution-list mailing list
> [hidden email]
> To change your list options or unsubscribe, visit ...
> https://mail.gnome.org/mailman/listinfo/evolution-list
_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Pete Biggs

> Then what exactly is the use case for importing from Thunderbird as per
> [the
> manual](https://help.gnome.org/users/evolution/stable/import-apps-mozilla.html.en)?
>

That is surely for importing to local folders, not IMAP folders

>
> As I have between 50 - 100 GB of mail to sync, with it all backed up
> locally, I'm happy to try to break things in an effort to avoid days if
> not weeks of re-syncing. Are there details on this process (warnings
> heeded) anywhere?

A serious question: if you are using IMAP, why do you need to "sync"
all your mail?  Evolution keeps a copy of any mail you read as a local
cache, there's no permanent store of the mail, it's not a duplicate of
your IMAP account.

>
> I have access to both the mail server and local machine for all but my
> gmail accounts. I'll test the process on less important demo accounts as
> well.

If you need a local store of *all* your mail, then you might like to
look at something like Offline IMAP - this will drag down all your mail
and keep it locally, then you point Evolution at that store rather than
the actual IMAP account.  You may be better looking at that to pre-
populate with your extant local copy.

>
> I guess I could copy my local backups to the evolution maildir, ssh into
> my mail server and rsync the maildirs. I'd hope it'd recognize there to
> be no need to resync and duplicate everything at that point, but maybe
> others have tried this with failure?

The good thing about standards is that there's so many to choose from.

I would strongly advise that you don't go fiddling around with the
internal Evolution data stores. Yes, it is Maildir, but the encoding of
the filenames and flags may not be consistent with what Evolution
thinks it wants.

Besides, the Maildir store is only for local accounts - the cache of
remote/IMAP accounts is not Maildir and is certainly not meant to be an
archival copy of the mail.

P.


_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Gnome Evolution - General mailing list
In reply to this post by Gnome Evolution - General mailing list
On Wed, 2020-10-28 at 00:31 +0900, christopher wrote:
> Then what exactly is the use case for importing from Thunderbird as
> per [the manual]

        Hi,
I do not think Thunderbird import imports also local caches, it imports
local accounts only (from the Mail part).

> As I have between 50 - 100 GB of mail to sync, with it all backed up
> locally, I'm happy to try to break things in an effort to avoid days
> if not weeks of re-syncing.

That feels like a lot of mails, or just few movies as attachments (just
kidding). Do you know how many mails this is? My largest IMAP account
has 498.162 messages at the moment, which is surely less than that
yours. I do not download all of the history locally, only the last
month is stored for the offline use, and even that not in all folders.

Note the initial synchronization doesn't download the messages, it only
downloads the headers (it depends also on the "Copy for offline
operations" option for the account and respective folders), thus it
should be significantly less data to be downloaded. Still, it takes
time and bandwidth.

>  Are there details on this process (warnings heeded) anywhere?

No, there are not, because nobody did it yet and should not do that.

I can think of only a single workaround:
a) mimic exact folder structure under On This Computer
b) import the respective .mbox files from Thunderbird to those folders
c) add an IMAP account
d) close Evolution while it is synchronizing the data
e) copy ~/.local/share/evolution/mail/local/folders.db file into
   the IMAP's directory ~/.cache/evolution/mail/<account-uid>/; it
   will overwrite the existing file
f) run Evolution

This only *tries to* not download the folder summary data, but the
format of the On This Computer and of the IMAP is not exactly the same,
there will be missing parts, which may or may not cause trouble. It
does not restore respective messages, the cache format is different.

I cannot tell whether it'll work or not. I cannot tell whether it'll
have any side effects or not. I cannot tell... It's all up to you. I
would not do this myself. I'd rather "waste" the bandwidth to have
things as they are supposed to be.

Note that the IMAP (and any other remote account) has the data stored
in ~/.cache/, which is a disposable place. When anything happens to the
data there, the mail provider will re-download it from the server.
        Bye,
        Milan

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Zan Lynx
On 10/27/2020 10:29 AM, Milan Crha via evolution-list wrote:
> That feels like a lot of mails, or just few movies as attachments (just
> kidding). Do you know how many mails this is? My largest IMAP account
> has 498.162 messages at the moment, which is surely less than that
> yours. I do not download all of the history locally, only the last
> month is stored for the offline use, and even that not in all folders.

I cannot speak for him but I saw he was using Gmail. Their IMAP system
duplicates messages and if you have a large number of folders, some of
which include the same messages plus "All Mail", you can easily end up
with a 5 or 6 times multiplier on message size.

This bothers me less since I tend to turn off local storage of IMAP and
limit it to the most basic caching of messages I intentionally read. I
can wait for a short message fetch. That helps a lot on Gmail too since
I rarely read the "All Mail" folder.

--
                 Knowledge is Power -- Power Corrupts
                         Study Hard -- Be Evil
_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Patrick O'Callaghan
On Tue, 2020-10-27 at 10:36 -0600, Zan Lynx wrote:

> On 10/27/2020 10:29 AM, Milan Crha via evolution-list wrote:
> > That feels like a lot of mails, or just few movies as attachments (just
> > kidding). Do you know how many mails this is? My largest IMAP account
> > has 498.162 messages at the moment, which is surely less than that
> > yours. I do not download all of the history locally, only the last
> > month is stored for the offline use, and even that not in all folders.
>
> I cannot speak for him but I saw he was using Gmail. Their IMAP system
> duplicates messages and if you have a large number of folders, some of
> which include the same messages plus "All Mail", you can easily end up
> with a 5 or 6 times multiplier on message size.

I use Gmail via IMAP and although what you say is true, I haven't
noticed any real-world impact. The vast majority of messages have only
one label (i.e. are in only one IMAP folder) and I don't download All
Mail anyway (in fact I don't sync locally at all). You can even
unsubscribe it if you like.

poc

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Gnome Evolution - General mailing list
On Tue, 2020-10-27 at 17:50 +0100, Patrick O'Callaghan wrote:
> I don't download All
> Mail anyway (in fact I don't sync locally at all). You can even
> unsubscribe it if you like.

        Hi,
for what it's worth, I have the All Mails disabled on the Gmail side
for IMAP synchronization (otherwise I'd have twice more messages).

Otherwise it's true, the messages are "duplicated" (it's in double
quotes, because Gmail de-duplicates by Message-ID) in IMAP folders
(labels) they are located in.
        Bye,
        Milan

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Ángel González
In reply to this post by Gnome Evolution - General mailing list
On 2020-10-27 at 17:29 +0100, Milan Crha wrote:
> On Wed, 2020-10-28 at 00:31 +0900, christopher wrote:
> >  Are there details on this process (warnings heeded) anywhere?
>
> No, there are not, because nobody did it yet and should not do that.
>
> I can think of only a single workaround:
> (...)

Maybe he could set up an IMAP server on localhost pointing to that
local copy, configure evolution to use that server, copying it to the
local cache (thus transferring the 100 GB on localhost interface), then
change the server configuration to the real server.

I would expect the folders.db to have important(?) differences (for
instance the message UIDs would all be different), but seems like it
should be slightly more compatible. Any opinion on that?


Best regards


_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Gnome Evolution - General mailing list
In reply to this post by Pete Biggs
Hi Pete (sorry for the missed reply-all last time),

Thank you for your reply. [And thank you to everyone else for your insight as well!]

>A serious question: if you are using IMAP, why do you need to "sync"
>all your mail? 

Because there is no backup on the server that I can restore from and if I press select all -> delete in Evolution all my mail goes to the ether without so much as a "are you sure?" dialog. This is obviously a worst case scenario, but at literally three (two if you count ctrl+a as one ;) ) quick keyboard clicks and no confirmation, its a real worry to me. There are various less severe scenarios where I'd like to have a backup of all my mail somewhere. Trusting the IMAP server to never fail is quite risky in my opinion. Do you have a better way to keep a backup than what I am suggesting?

Also I like to keep a local copy of all mail as I am offline quite often and need to have all work related email to reference various things from years past.

Another reason is the speed in which I can search for old mail. For example if I have a client I want to contact again, but don't remember their name, I can look up the project number and the mail thread will be right there within a second or two. Google's online interface is quick enough to do this at about the same speed, but as for the mail on my justhost.com account, that could take several minutes before I get a full list of results.

An even more recent example was when I started this thread. I added my [hidden email] account to Evolution, and copy pasted my local copies to the Inbox. I ended up with a bunch of duplicates and Evolution's "remove duplicates" was not able to remove them all for some reason. Now I have a total mess of an Inbox. The quickest way to fix this was to just delete everything and redump the local backup copies. How would you fix any large scale issue like this without a backup? 

>Evolution keeps a copy of any mail you read as a local
>cache, there's no permanent store of the mail, it's not a duplicate of
>your IMAP account.

Doesn't that depend on whether or not you enable the "Synchronize remote mail locally in all folders" option in Account Editor -> Receiving Options?

What I see locally is a mirror of what is on the mail server if I ssh into it after everything has synced. Am I overlooking something?

>If you need a local store of *all* your mail, then you might like to
>look at something like Offline IMAP - this will drag down all your mail
>and keep it locally, then you point Evolution at that store rather than
>the actual IMAP account.  You may be better looking at that to pre-
>populate with your extant local copy.

Thats an interesting idea, but how would I handle outgoing mail at that point? I'm not familiar with Offline IMAP, but if it is just syncing from the mail server (e.g. bar.com), and Evolution is just allowing me to view this locally sync'ed maildir, how would I use Evolution to reply to a message I am reading in this local maildir and have it send from my mail address of [hidden email]?

Ideally I'd like the following setup:

1.) Something like Offline IMAP or Evolution's "Synchronize remote mail locally in all folders" to sync to a local maildir
2.) Have an incremental backup of this maildir (e.g rsnapshot on cron) so that I have an actual backup and not just a mirror
3.) Send and receive emails through Evolution

I'm still not exactly sure how step three would look exactly. Keep it as a cache only imap account? Then pull a copy of the sent message from the mail server via step 1?

Maybe a better question to the entire community would be how do you keep regularly scheduled backups of your mail? For self-hosted servers its a no-brainer, but for gmail accounts or other cloud hosted accounts, I'm really at a loss as to why I cannot find a standard way of doing this.

Looking forward to your insights!

On Wed, Oct 28, 2020 at 1:21 AM Pete Biggs <[hidden email]> wrote:

> Then what exactly is the use case for importing from Thunderbird as per
> [the
> manual](https://help.gnome.org/users/evolution/stable/import-apps-mozilla.html.en)?
>

That is surely for importing to local folders, not IMAP folders

>
> As I have between 50 - 100 GB of mail to sync, with it all backed up
> locally, I'm happy to try to break things in an effort to avoid days if
> not weeks of re-syncing. Are there details on this process (warnings
> heeded) anywhere?

A serious question: if you are using IMAP, why do you need to "sync"
all your mail?  Evolution keeps a copy of any mail you read as a local
cache, there's no permanent store of the mail, it's not a duplicate of
your IMAP account.

>
> I have access to both the mail server and local machine for all but my
> gmail accounts. I'll test the process on less important demo accounts as
> well.

If you need a local store of *all* your mail, then you might like to
look at something like Offline IMAP - this will drag down all your mail
and keep it locally, then you point Evolution at that store rather than
the actual IMAP account.  You may be better looking at that to pre-
populate with your extant local copy.

>
> I guess I could copy my local backups to the evolution maildir, ssh into
> my mail server and rsync the maildirs. I'd hope it'd recognize there to
> be no need to resync and duplicate everything at that point, but maybe
> others have tried this with failure?

The good thing about standards is that there's so many to choose from.

I would strongly advise that you don't go fiddling around with the
internal Evolution data stores. Yes, it is Maildir, but the encoding of
the filenames and flags may not be consistent with what Evolution
thinks it wants.

Besides, the Maildir store is only for local accounts - the cache of
remote/IMAP accounts is not Maildir and is certainly not meant to be an
archival copy of the mail.

P.


_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Pete Biggs
On Thu, 2020-10-29 at 01:09 +0900, Bo Buckley wrote:
> Hi Pete (sorry for the missed reply-all last time),

Yeah, please try and do reply-to-list not reply-all!

>
> Thank you for your reply. [And thank you to everyone else for your insight as well!]
>
> >A serious question: if you are using IMAP, why do you need to "sync"
> >all your mail?  
>
> Because there is no backup on the server that I can restore from and
> if I press select all -> delete in Evolution all my mail goes to the
> ether without so much as a "are you sure?" dialog.

There should be a confirmatory popup on deletion - at least the first
time, you can tick "don't ask me again" though.

> This is obviously a worst case scenario, but at literally three (two
> if you count ctrl+a as one ;) ) quick keyboard clicks and no
> confirmation, its a real worry to me.

Yes, but mail is never deleted that easily. It will always, I think, be
put into a Trash first, then you have to explicitly expunge it to
permanently get rid of it.

>  There are various less severe scenarios where I'd like to have a
> backup of all my mail somewhere. Trusting the IMAP server to never
> fail is quite risky in my opinion. Do you have a better way to keep a
> backup than what I am suggesting?

I use ISP's and mail providers that make backups regularly. If yours
doesn't, then I might hazard a suggestion that you aren't using the
correct provider for your risk perception.

>
> Also I like to keep a local copy of all mail as I am offline quite
> often and need to have all work related email to reference various
> things from years past.

The local cache will provide that. But it's a cache and considered
expendable.  You also have to consider coherency between your local
copies and the IMAP server - what happens if you modify the local copy
when your offline - how do the changes get passed back to the server
and so on. It's not a trivial process.

>
> Another reason is the speed in which I can search for old mail. For
> example if I have a client I want to contact again, but don't
> remember their name, I can look up the project number and the mail
> thread will be right there within a second or two. Google's online
> interface is quick enough to do this at about the same speed, but as
> for the mail on my justhost.com account, that could take several
> minutes before I get a full list of results.

Again, that's an issue with your mail provider - I have all my mail
remote, albeit mostly on my own servers, and the mail is indexed when
it comes in and searching is stupidly fast - I can get search results
back from a 200,000 message folder in seconds.

>
> An even more recent example was when I started this thread. I added
> my [hidden email] account to Evolution, and copy pasted my local copies
> to the Inbox. I ended up with a bunch of duplicates and Evolution's
> "remove duplicates" was not able to remove them all for some reason.
> Now I have a total mess of an Inbox. The quickest way to fix this was
> to just delete everything and redump the local backup copies. How
> would you fix any large scale issue like this without a backup?  

That's the issue of playing around with Evolution's internal files. If
your mail is remote, only, then you won't get into that sort of
trouble. If the duplicates were only local (and hadn't got copied back
to the IMAP server), then you just delete the account and start again.
If things had got back to the server, then ask your ISP to restore from
a backup.

>
> >Evolution keeps a copy of any mail you read as a local
> >cache, there's no permanent store of the mail, it's not a duplicate of
> >your IMAP account.
>
> Doesn't that depend on whether or not you enable the "Synchronize
> remote mail locally in all folders" option in Account Editor ->
> Receiving Options?

AFAIK that's still just a cache - but it just pre-cache's it rather
than only caching messages as you read them. Others will probably
correct me.

>
> What I see locally is a mirror of what is on the mail server if I ssh
> into it after everything has synced. Am I overlooking something?

Where do you see it locally?

>
> >If you need a local store of *all* your mail, then you might like to
> >look at something like Offline IMAP - this will drag down all your mail
> >and keep it locally, then you point Evolution at that store rather than
> >the actual IMAP account.  You may be better looking at that to pre-
> >populate with your extant local copy.
>
> Thats an interesting idea, but how would I handle outgoing mail at
> that point?

How do you handle outgoing mail when you are offline now? It will be
the same - Offline IMAP basically makes your local copy the definitive
one and handles the syncing and coherency with the server. Everything
else is the same.

>
> Ideally I'd like the following setup:
>
> 1.) Something like Offline IMAP or Evolution's "Synchronize remote
> mail locally in all folders" to sync to a local maildir
> 2.) Have an incremental backup of this maildir (e.g rsnapshot on
> cron) so that I have an actual backup and not just a mirror
> 3.) Send and receive emails through Evolution
>
> I'm still not exactly sure how step three would look exactly. Keep it
> as a cache only imap account? Then pull a copy of the sent message
> from the mail server via step 1?

No, the local copy is the definitive one and you use it like any other
account - Offline IMAP ensures that the server version is kept in sync
with your local one: if Evolution puts a copy of a message in the Sent
folder locally, it will also appear in the Sent folder on the server.
If your mail server handles putting things in the Sent folder, then
that message will be synced to the local Sent folder.

>
> Maybe a better question to the entire community would be how do you
> keep regularly scheduled backups of your mail? For self-hosted
> servers its a no-brainer, but for gmail accounts or other cloud
> hosted accounts, I'm really at a loss as to why I cannot find a
> standard way of doing this.

Well gmail is "backed up" - or rather nothing is ever deleted or
removed and there are multiple copies in data centres around the world.
All other mail providers should also back up things for disaster
recovery and anything that uses AWS or O365 will be in a similar
position to gmail.

P.


_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Gnome Evolution - General mailing list
In reply to this post by Ángel González
On Tue, 2020-10-27 at 21:45 +0100, Ángel wrote:
> I would expect the folders.db to have important(?) differences (for
> instance the message UIDs would all be different), but seems like it
> should be slightly more compatible. Any opinion on that?

        Hi,
ah, right, the UIDs will be also different, thus what I suggest would
not work at all. (I thought of the extra data the IMAP folder stores in
the summary and forgot of the main important bit, the UID.)

Thanks for the correction.
        Bye,
        Milan

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Gnome Evolution - General mailing list
In reply to this post by Gnome Evolution - General mailing list
On Thu, 2020-10-29 at 01:09 +0900, Bo Buckley via evolution-list wrote:
> Ideally I'd like the following setup:
>
> 1.) Something like Offline IMAP or Evolution's "Synchronize remote
> mail locally in all folders" to sync to a local maildir
> 2.) Have an incremental backup of this maildir (e.g rsnapshot on
> cron) so that I have an actual backup and not just a mirror
> 3.) Send and receive emails through Evolution

        Hi,
the more I think of it the more I agree with Pete. To achieve what you
want you'll be the safest with the Offline IMAP (or the imapfw, which
seems to be a future replacement of it).

Then you can configure your account as before, just instead of having
IMAP as the Receiving Email server type you'll pick "Maildir-format
mail directories" and point it to the Offline IMAP directory. You setup
the SMTP as before, to your mail provider. If you had looked up the
details then just press "Back" in the New Mail Account Wizard and
change only the values, which need changes. You can also configure
send-only accounts, with "None" type for the Receiving Email part.

You should be able to configure Offline IMAP for multiple accounts,
storing data into multiple directories, separately. And when you've it
all under ~/Maildir, you can backup (rsync) all of them by one command.
        Bye,
        Milan

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Patrick O'Callaghan
In reply to this post by Pete Biggs
On Wed, 2020-10-28 at 18:25 +0000, Pete Biggs wrote:

> > Maybe a better question to the entire community would be how do you
> > keep regularly scheduled backups of your mail? For self-hosted
> > servers its a no-brainer, but for gmail accounts or other cloud
> > hosted accounts, I'm really at a loss as to why I cannot find a
> > standard way of doing this.
>
>
> Well gmail is "backed up" - or rather nothing is ever deleted or
>
> removed and there are multiple copies in data centres around the world.
>
> All other mail providers should also back up things for disaster
>
> recovery and anything that uses AWS or O365 will be in a similar
>
> position to gmail.

Obviously it's not theoretically impossible for Google to suddenly lose
all of its data centres, but I don't consider that to be something I
need to spend my time thinking about. Over ten years ago I persuaded my
university to switch to Google for handling mail, because the cost and
effort of doing it ourselves was rapidly overwhelming our meagre
resources and we simply couldn't keep up with the community's demands,
and of course for an educational institution G Suite (now Google
Workplace) was free.

There may be extra-technological reasons for not doing that, such as
regulatory requirements, suspicion of the provider etc., and of course
Google is not the only option, but in our circumstances I consider that
to have been an unqualified success.

poc

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Pete Biggs

> Obviously it's not theoretically impossible for Google to suddenly lose
> all of its data centres, but I don't consider that to be something I
> need to spend my time thinking about. Over ten years ago I persuaded my
> university to switch to Google for handling mail, because the cost and
> effort of doing it ourselves was rapidly overwhelming our meagre
> resources and we simply couldn't keep up with the community's demands,
> and of course for an educational institution G Suite (now Google
> Workplace) was free.
>
> There may be extra-technological reasons for not doing that, such as
> regulatory requirements, suspicion of the provider etc., and of course
> Google is not the only option, but in our circumstances I consider that
> to have been an unqualified success.
>
Yes, I think what I was trying to get across was risk management and
risk perception.  One can think of lots of scenarios where things can
go wrong, but it's weighing up the risk: i.e. likelihood and
consequences - I suspect the consequences of Google loosing all its
datacenters would be far more problematic than not having a copy of
your mail.

BTW, we too went through the process of outsourcing email, but Google
couldn't/wouldn't commit to housing the data in a certain jurisdiction
so our legal team said no.  If you ever want to meet risk adverse
people, just talk to the legal department of a UK university.

P.


_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Patrick O'Callaghan
On Thu, 2020-10-29 at 11:41 +0000, Pete Biggs wrote:
> BTW, we too went through the process of outsourcing email, but Google
> couldn't/wouldn't commit to housing the data in a certain jurisdiction
> so our legal team said no.  If you ever want to meet risk adverse
> people, just talk to the legal department of a UK university.

I hear you :-)

poc

_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Gnome Evolution - General mailing list
In reply to this post by Patrick O'Callaghan
POC,

Maybe I have just missed this the whole time using gmail, but in the
event in Evolution you did a Select All + Del. How would you recover? I
know Gmail keeps things in the trash for 30 days, so I guess this
what-if scenario would be easily noticed in this time period, but what
if you did something unknowingly and didn't notice until a few months
later?

One situation I couldn't recover from in Gmail recently without a backup
was the read/unread status of emails. I somehow messed up a sorting
filter in Thunderbird and marked everything as read mistakenly. I
organize what I need to still read by this attribute, so I would surely
forget to respond to several mails had I not had a recovery solution in
place. In this case I simply turned off the internet to my smartphone
and manually went in and marked them all unread again... A pain in the
ass. I tried restoring from my local backups, but it seems either gmail
was overriding my local changes on sync, or this information was simply
not stored in the mbox backups I had. Either way, I realized how flawed
my backup plan was after this, and thus the start of this thread.

I'm sure gmail has plenty of copies of your data, but do they actual
have backups (i.e. incremental, history reversible?) Without that, if
you make one mistake ALL the copies get mirrored and you're out of luck.

On 10/29/20 20:16, Patrick O'Callaghan wrote:

> On Wed, 2020-10-28 at 18:25 +0000, Pete Biggs wrote:
>>> Maybe a better question to the entire community would be how do you
>>> keep regularly scheduled backups of your mail? For self-hosted
>>> servers its a no-brainer, but for gmail accounts or other cloud
>>> hosted accounts, I'm really at a loss as to why I cannot find a
>>> standard way of doing this.
>>
>> Well gmail is "backed up" - or rather nothing is ever deleted or
>>
>> removed and there are multiple copies in data centres around the world.
>>
>> All other mail providers should also back up things for disaster
>>
>> recovery and anything that uses AWS or O365 will be in a similar
>>
>> position to gmail.
> Obviously it's not theoretically impossible for Google to suddenly lose
> all of its data centres, but I don't consider that to be something I
> need to spend my time thinking about. Over ten years ago I persuaded my
> university to switch to Google for handling mail, because the cost and
> effort of doing it ourselves was rapidly overwhelming our meagre
> resources and we simply couldn't keep up with the community's demands,
> and of course for an educational institution G Suite (now Google
> Workplace) was free.
>
> There may be extra-technological reasons for not doing that, such as
> regulatory requirements, suspicion of the provider etc., and of course
> Google is not the only option, but in our circumstances I consider that
> to have been an unqualified success.
>
> poc
>
> _______________________________________________
> evolution-list mailing list
> [hidden email]
> To change your list options or unsubscribe, visit ...
> https://mail.gnome.org/mailman/listinfo/evolution-list
_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Migrating from Thunderbird via import without duplicates

Patrick O'Callaghan
On Wed, 2020-11-04 at 21:01 +0900, christopher via evolution-list
wrote:
> POC,
>
> Maybe I have just missed this the whole time using gmail, but in the
> event in Evolution you did a Select All + Del. How would you recover? I
> know Gmail keeps things in the trash for 30 days, so I guess this
> what-if scenario would be easily noticed in this time period, but what
> if you did something unknowingly and didn't notice until a few months
> later?

Then I'd lose all my mail. I don't care. The probability of this
actually happening without me noticing it is a lot smaller than that of
my PC going up in flames or my backup disk suffering bitrot. I do keep
nightlybackups of my home directory of course, on an attached mirrored
disk. (I'm not a lunatic.) However I'm quite happy for Gmail to take
care of that side of things. That's a decision each of us has to make
for themselves.

> One situation I couldn't recover from in Gmail recently without a backup
> was the read/unread status of emails. I somehow messed up a sorting
> filter in Thunderbird and marked everything as read mistakenly. I
> organize what I need to still read by this attribute, so I would surely
> forget to respond to several mails had I not had a recovery solution in
> place. In this case I simply turned off the internet to my smartphone
> and manually went in and marked them all unread again... A pain in the
> ass. I tried restoring from my local backups, but it seems either gmail
> was overriding my local changes on sync, or this information was simply
> not stored in the mbox backups I had. Either way, I realized how flawed
> my backup plan was after this, and thus the start of this thread.

AFAIK Gmail pays no attention to your local copy. Evolution on the
other hand, does pay attention if you set it up that way, such as with
synced IMAP. And you can easily incorporate its local data in your
regular backup schedule.

> I'm sure gmail has plenty of copies of your data, but do they actual
> have backups (i.e. incremental, history reversible?) Without that, if
> you make one mistake ALL the copies get mirrored and you're out of luck.

Not for mail, to my knowledge (though they do keep generational copies
of files in Google Docs etc.). I don't care (see above).

poc


_______________________________________________
evolution-list mailing list
[hidden email]
To change your list options or unsubscribe, visit ...
https://mail.gnome.org/mailman/listinfo/evolution-list