Message deletion bug?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Message deletion bug?

Patrick O'Callaghan
[I tried this on 2.4.1 and 2.5.1 with the same result.]

First some background which may or may no be relevant. The way I handle
spam is for a server-side SpamAssassin process to file suspect messages
in a SPAM folder. For false negatives (i.e. undetected spam) I manually
move the offending messages to a subfolder of SPAM and periodically
process it by logging in to the server.

Processing involves a little shell script on the server which calls
sa-learn on the contents of the subfolder.

After that, I delete the contents of the subfolder using Evo. I then
expunge the subfolder to clean it out.

For curiousity, I just checked on the server and the last bunch of spam
had not been deleted. Nevertheless, Evo shows the folder as empty. I
made sure Evo was set to Show Deleted Messages, and Show Hidden
Messages. I also made sure there was nothing in the Search box.

I exited Evo and fired up Thunderbird. Sure enough, the messages are
still there. Go back to Evo and they don't appear.

I've never seen this happen before and am not sure I can reproduce it,
which is why I'm giving so much detail. Probably this has nothing to do
with spam processing as such. (Note that Evo doesn't know I consider
these messages to be spam, as I don't use the Junk controls).

It's worrying that an apparently deleted message is still present on the
server, *even after quitting and restarting Evo*. Is this another
caching bug or what?

poc

_______________________________________________
Evolution-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Message deletion bug?

ddougan
On Tue, 2005-11-22 at 15:12 -0400, Patrick O'Callaghan wrote:

> [I tried this on 2.4.1 and 2.5.1 with the same result.]
>
> First some background which may or may no be relevant. The way I handle
> spam is for a server-side SpamAssassin process to file suspect messages
> in a SPAM folder. For false negatives (i.e. undetected spam) I manually
> move the offending messages to a subfolder of SPAM and periodically
> process it by logging in to the server.
>
> Processing involves a little shell script on the server which calls
> sa-learn on the contents of the subfolder.
>
> After that, I delete the contents of the subfolder using Evo. I then
> expunge the subfolder to clean it out.

Patrick,

This sounds neat - it's bugged me for a while that I'm processing spam
in two places (I also have SA on my server). Would you mind posting your
script - I'd like to try it on my set-up?

Thanks,

Des

> For curiousity, I just checked on the server and the last bunch of spam
> had not been deleted. Nevertheless, Evo shows the folder as empty. I
> made sure Evo was set to Show Deleted Messages, and Show Hidden
> Messages. I also made sure there was nothing in the Search box.
>
> I exited Evo and fired up Thunderbird. Sure enough, the messages are
> still there. Go back to Evo and they don't appear.
>
> I've never seen this happen before and am not sure I can reproduce it,
> which is why I'm giving so much detail. Probably this has nothing to do
> with spam processing as such. (Note that Evo doesn't know I consider
> these messages to be spam, as I don't use the Junk controls).
>
> It's worrying that an apparently deleted message is still present on the
> server, *even after quitting and restarting Evo*. Is this another
> caching bug or what?
>
> poc
>
> _______________________________________________
> Evolution-list mailing list
> [hidden email]
> http://mail.gnome.org/mailman/listinfo/evolution-list
--

Des Dougan, Principal
Dougan Consulting Group

Ph: 604-980-2848       Email: des at DouganConsulting dot com    

        www.DouganConsulting.com

Design - Implementation - Support

_______________________________________________
Evolution-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Message deletion bug?

Karsten Bräckelmann-2

> This sounds neat - it's bugged me for a while that I'm processing spam
> in two places (I also have SA on my server). Would you mind posting your
> script - I'd like to try it on my set-up?

Speaking of server side SA filtering, there have been some notes posted
to this list some months ago to integrate this (at least 2 different
threads). Basically it boils down to client side spam[cd] and sa-learn
[1] scripts, that hit your server. Maybe these posts are helpful:

  http://mail.gnome.org/archives/evolution-list/2005-May/msg00030.html
  http://mail.gnome.org/archives/evolution-list/2005-May/msg00067.html

Have fun...

...guenther


[1] supporting at least --ham and --spam


--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

_______________________________________________
Evolution-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Message deletion bug?

Patrick O'Callaghan
In reply to this post by ddougan
On Tue, 2005-11-22 at 21:35 -0800, Des Dougan wrote:

> On Tue, 2005-11-22 at 15:12 -0400, Patrick O'Callaghan wrote:
> > [I tried this on 2.4.1 and 2.5.1 with the same result.]
> >
> > First some background which may or may no be relevant. The way I handle
> > spam is for a server-side SpamAssassin process to file suspect messages
> > in a SPAM folder. For false negatives (i.e. undetected spam) I manually
> > move the offending messages to a subfolder of SPAM and periodically
> > process it by logging in to the server.
> >
> > Processing involves a little shell script on the server which calls
> > sa-learn on the contents of the subfolder.
> >
> > After that, I delete the contents of the subfolder using Evo. I then
> > expunge the subfolder to clean it out.
>
> Patrick,
>
> This sounds neat - it's bugged me for a while that I'm processing spam
> in two places (I also have SA on my server). Would you mind posting your
> script - I'd like to try it on my set-up?

It's pretty specific to my situation, and almost embarrasingly simple,
but here goes:

<script>
#!/bin/bash
# Process spam

cd ~poc
cp /var/spool/imap/p/user/poc/SPAM/spam/*. SCRATCH/
chown poc.prof SCRATCH/*.
su -l -c "sa-learn --spam SCRATCH/*." poc
rm -f SCRATCH/*.
</script>

Notes:

1) The IMAP server is Cyrus, so messages are individual files and they
belong to the Cyrus user. This means you need root (or Cyrus) login on
the server and the script has to be run as root. However SA has to run
as *you* so it'll keep its hash tables specific to your spam. Therefore
you have to copy the messages and change their ownership, then change
personality to you to run SA.

2) I didn't bother including an option for --ham since SA hasn't given
me a false positive in over a year. YMMV.

3) You could change the "cp" to a "mv", but in any case you need to
resync Evo as it won't notice automatically (or maybe it will, opinions
vary). I don't do it that way as I'm keeping a corpus of old spam in a
separate folder and I use Evo to move messages into it. This also
applies to spam detected by SA, once I've looked it over for false
positives (increasingly a waste of time IMHO).

Anyway, best of luck :-)

poc

_______________________________________________
Evolution-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Message deletion bug?

Patrick O'Callaghan
In reply to this post by Karsten Bräckelmann-2
On Wed, 2005-11-23 at 15:00 +0100, guenther wrote:

> > This sounds neat - it's bugged me for a while that I'm processing spam
> > in two places (I also have SA on my server). Would you mind posting your
> > script - I'd like to try it on my set-up?
>
> Speaking of server side SA filtering, there have been some notes posted
> to this list some months ago to integrate this (at least 2 different
> threads). Basically it boils down to client side spam[cd] and sa-learn
> [1] scripts, that hit your server. Maybe these posts are helpful:
>
>   http://mail.gnome.org/archives/evolution-list/2005-May/msg00030.html
>   http://mail.gnome.org/archives/evolution-list/2005-May/msg00067.html

Yes, I'd seen these and they look like a good idea, but I worry about
certain details:

1) I don't like that you need to keep a local database and push it to
the server. For one thing, I use several installations of Evo and under
this scheme they'll learn slightly different things, so depending on
which one does the push behaviour might change. Probably not a biggie,
but still. Also, I don't want my 15000 users to be *able* to run rsync
to the server, at least not without a great deal of thought. I wonder if
IMAP POST could be twisted into doing it. Hmmm ...

2) I don't like any solution which is client-specific, but that's just
me, always thinking of the users (almost none of whom use Evo).

The basic problem is that every solution I've seen (including mine, see
recent message) requires some level of login access to the server. This
is simply a non-starter for a large user base such as ours. Note that
this isn't an Evo issue. IMHO it's basically a gap in mail access
protocols (IMAP etc.) which were designed before spam was a problem.
There are several proposals out there for complementary protocols for
mail administration, which would be the natural place for this stuff,
but none of them seem to be widespread.

poc

_______________________________________________
Evolution-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Message deletion bug?

ddougan
In reply to this post by Patrick O'Callaghan
On Wed, 2005-11-23 at 10:49 -0400, Patrick O'Callaghan wrote:

> On Tue, 2005-11-22 at 21:35 -0800, Des Dougan wrote:
> > On Tue, 2005-11-22 at 15:12 -0400, Patrick O'Callaghan wrote:
> > > [I tried this on 2.4.1 and 2.5.1 with the same result.]
> > >
> > > First some background which may or may no be relevant. The way I handle
> > > spam is for a server-side SpamAssassin process to file suspect messages
> > > in a SPAM folder. For false negatives (i.e. undetected spam) I manually
> > > move the offending messages to a subfolder of SPAM and periodically
> > > process it by logging in to the server.
> > >
> > > Processing involves a little shell script on the server which calls
> > > sa-learn on the contents of the subfolder.
> > >
> > > After that, I delete the contents of the subfolder using Evo. I then
> > > expunge the subfolder to clean it out.
> >
> > Patrick,
> >
> > This sounds neat - it's bugged me for a while that I'm processing spam
> > in two places (I also have SA on my server). Would you mind posting your
> > script - I'd like to try it on my set-up?
>
> It's pretty specific to my situation, and almost embarrasingly simple,
> but here goes:
>
> <script>
> #!/bin/bash
> # Process spam
>
> cd ~poc
> cp /var/spool/imap/p/user/poc/SPAM/spam/*. SCRATCH/
> chown poc.prof SCRATCH/*.
> su -l -c "sa-learn --spam SCRATCH/*." poc
> rm -f SCRATCH/*.
> </script>
>
> Notes:
>
> 1) The IMAP server is Cyrus, so messages are individual files and they
> belong to the Cyrus user. This means you need root (or Cyrus) login on
> the server and the script has to be run as root. However SA has to run
> as *you* so it'll keep its hash tables specific to your spam. Therefore
> you have to copy the messages and change their ownership, then change
> personality to you to run SA.
>
> 2) I didn't bother including an option for --ham since SA hasn't given
> me a false positive in over a year. YMMV.
>
> 3) You could change the "cp" to a "mv", but in any case you need to
> resync Evo as it won't notice automatically (or maybe it will, opinions
> vary). I don't do it that way as I'm keeping a corpus of old spam in a
> separate folder and I use Evo to move messages into it. This also
> applies to spam detected by SA, once I've looked it over for false
> positives (increasingly a waste of time IMHO).
>
> Anyway, best of luck :-)


Patrick,

Many thanks. I'll try to get some time this weekend to play with it.

Des
--

Des Dougan, Principal
Dougan Consulting Group

Ph: 604-980-2848       Email: des at DouganConsulting dot com    

        www.DouganConsulting.com

Design - Implementation - Support

_______________________________________________
Evolution-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Message deletion bug?

Karsten Bräckelmann-2
In reply to this post by Patrick O'Callaghan
On Wed, 2005-11-23 at 11:07 -0400, Patrick O'Callaghan wrote:

> On Wed, 2005-11-23 at 15:00 +0100, guenther wrote:
> > > This sounds neat - it's bugged me for a while that I'm processing spam
> > > in two places (I also have SA on my server). Would you mind posting your
> > > script - I'd like to try it on my set-up?
> >
> > Speaking of server side SA filtering, there have been some notes posted
> > to this list some months ago to integrate this (at least 2 different
> > threads). Basically it boils down to client side spam[cd] and sa-learn
> > [1] scripts, that hit your server. Maybe these posts are helpful:
> >
> >   http://mail.gnome.org/archives/evolution-list/2005-May/msg00030.html
> >   http://mail.gnome.org/archives/evolution-list/2005-May/msg00067.html
>
> Yes, I'd seen these and they look like a good idea, but I worry about
> certain details:
>
> 1) I don't like that you need to keep a local database and push it to
> the server.

I don't see the need for a local database.

Mark as Junk will pipe the entire mail into 'sa-learn --spam'. Now, a
custom sa-learn script just could pick up this mail and send it to a
centralized server -- the one running the "spam fighting tool" on it.
One possible solution for this "sending" is actually sending by mail to
a dedicated "user", attaching the mail to be learned. The originl user
account is known, so a script on the recipients ond can do the magic.


> For one thing, I use several installations of Evo and under
> this scheme they'll learn slightly different things, so depending on
> which one does the push behaviour might change. Probably not a biggie,
> but still. Also, I don't want my 15000 users to be *able* to run rsync
> to the server, at least not without a great deal of thought. I wonder if
> IMAP POST could be twisted into doing it. Hmmm ...
>
> 2) I don't like any solution which is client-specific, but that's just
> me, always thinking of the users (almost none of whom use Evo).
>
> The basic problem is that every solution I've seen (including mine, see
> recent message) requires some level of login access to the server.

Uhm, ever thought about sending a mail? This does not involve logging
in.

> This
> is simply a non-starter for a large user base such as ours. Note that
> this isn't an Evo issue. IMHO it's basically a gap in mail access
> protocols (IMAP etc.) which were designed before spam was a problem.
> There are several proposals out there for complementary protocols for
> mail administration, which would be the natural place for this stuff,
> but none of them seem to be widespread.

...guenther


--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

_______________________________________________
Evolution-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Message deletion bug?

Karsten Bräckelmann-2
In reply to this post by Patrick O'Callaghan

> First some background which may or may no be relevant. The way I handle
> spam is for a server-side SpamAssassin process to file suspect messages
> in a SPAM folder. For false negatives (i.e. undetected spam) I manually
> move the offending messages to a subfolder of SPAM and periodically
> process it by logging in to the server.
>
> Processing involves a little shell script on the server which calls
> sa-learn on the contents of the subfolder.
>
> After that, I delete the contents of the subfolder using Evo. I then
> expunge the subfolder to clean it out.
>
> For curiousity, I just checked on the server and the last bunch of spam
> had not been deleted. Nevertheless, Evo shows the folder as empty. I
> made sure Evo was set to Show Deleted Messages, and Show Hidden
> Messages. I also made sure there was nothing in the Search box.
>
> I exited Evo and fired up Thunderbird. Sure enough, the messages are
> still there. Go back to Evo and they don't appear.
>
> I've never seen this happen before and am not sure I can reproduce it,
> which is why I'm giving so much detail. Probably this has nothing to do
> with spam processing as such. (Note that Evo doesn't know I consider
> these messages to be spam, as I don't use the Junk controls).
>
> It's worrying that an apparently deleted message is still present on the
> server, *even after quitting and restarting Evo*. Is this another
> caching bug or what?

Finally getting back to the original post...

Puzzling. I never came across something like this. Sorry I can't tell
you more, dude... :/


It would be great, if you can keep an eye on this. If this ever is
reproducible, please file a bug report.

...guenther


--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

_______________________________________________
Evolution-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/evolution-list
Reply | Threaded
Open this post in threaded view
|

Re: Message deletion bug?

Patrick O'Callaghan
In reply to this post by Karsten Bräckelmann-2
On Sat, 2005-12-10 at 23:39 +0100, guenther wrote:

> On Wed, 2005-11-23 at 11:07 -0400, Patrick O'Callaghan wrote:
> > On Wed, 2005-11-23 at 15:00 +0100, guenther wrote:
> > > > This sounds neat - it's bugged me for a while that I'm processing spam
> > > > in two places (I also have SA on my server). Would you mind posting your
> > > > script - I'd like to try it on my set-up?
> > >
> > > Speaking of server side SA filtering, there have been some notes posted
> > > to this list some months ago to integrate this (at least 2 different
> > > threads). Basically it boils down to client side spam[cd] and sa-learn
> > > [1] scripts, that hit your server. Maybe these posts are helpful:
> > >
> > >   http://mail.gnome.org/archives/evolution-list/2005-May/msg00030.html
> > >   http://mail.gnome.org/archives/evolution-list/2005-May/msg00067.html
> >
> > Yes, I'd seen these and they look like a good idea, but I worry about
> > certain details:
> >
> > 1) I don't like that you need to keep a local database and push it to
> > the server.
>
> I don't see the need for a local database.
>
> Mark as Junk will pipe the entire mail into 'sa-learn --spam'. Now, a
> custom sa-learn script just could pick up this mail and send it to a
> centralized server -- the one running the "spam fighting tool" on it.
> One possible solution for this "sending" is actually sending by mail to
> a dedicated "user", attaching the mail to be learned. The originl user
> account is known, so a script on the recipients ond can do the magic.

Sure, as long as I can run such a script. Of course *I* can, since I own
the server, but Joe User can't because I don't give him Shell access so
he can't set it up. Meaning I'll have to set up a general-purpose
script, which is somewhat trickier. Sheer inertia has prevented me from
doing this so far.

O'Callaghan's Law: in computing, everything is possible and nothing is
easy :-)

poc

_______________________________________________
Evolution-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/evolution-list