Corrupt file- how does it work?

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

Peter Ceresole - 03 Feb 2007 10:15 GMT

Yesterday I had an incident with a corrupt file that crashed Eudora an
fetching mail. It did it twice (I wasn't sure what had happened the
first time). Both times it crashed Eudora and I had to rebuild the In
and Trash mailboxes. No great problem because all the mail that matters
is (1) backed up on a removable Firewire disk and (2) sorted into other
mailboxes.

I used my ISP's webmail service to check and delete all my incoming
mails on the server, and the problem has gone away.

This is about the third time it's happened in ten years- I'm really not
complaining. But why does it happen? I presume the corrupted mail had
some control codes in it. But surely emails are just bytes and are
handled neutrally? So how can they crash the mail client?

OS 10.3.9, pop3 account, Eudora 6.1.1 Light. And yes I know it's not the
latest, but it works extremely well for me, and the question stands.

Signature

Peter

Reply to this Message

Bill Cole - 03 Feb 2007 20:03 GMT

> Yesterday I had an incident with a corrupt file that crashed Eudora an
> fetching mail. It did it twice (I wasn't sure what had happened the
[quoted text clipped - 10 lines]
> some control codes in it. But surely emails are just bytes and are
> handled neutrally? So how can they crash the mail client?

That would be nice, but it sometimes does not quite work out.

I can't say for sure how your incident happened, but an example of what
might be possible is a classic buffer overflow: A MIME message or some
part of it may have a Content-Length header claiming one size, Eudora
sees that and uses it to allocate memory, but when the message is
actually read, there is more data there than expected, and when it is
all read into memory, it runs off the end of the allocated space into
something else, randomly. That sort of problem was sometimes hard to
catch on Classic MacOS, because often a too-short memory block would be
the latest one allocated and would be overflowed off into unallocated
space, harming nothing. On OS X memory allocation is radically
different, and the kernel is more likely (but by no means certain) to
notice an overflow as an attempt to write to memory that an application
should not be addressing at all, and stop it dead.

That's not a definitive explanation for your issue, and I can't even say
for sure that your version of Eudora has such bugs, but it is one of the
easier-to-explain examples of how malicious or corrupted input can cause
code to fail. In recent years there have been an ongoing stream of these
sorts of problems revealed in image-handling systems in Windows and
Quicktime that have made it possible in some cases to structure a
corrupt image in such a fashion that when the system is asked to render
an image, it either crashes the application or the system or in some
cases can be caused to run code carefully embedded in the supposed image
data. Since Eudora uses Quicktime to render embedded images in mail, it
is theoretically possible that a crashing Quicktime rendering bug like
that could hit Eudora.

Signature

Now where did I hide that website...

Reply to this Message

Peter Ceresole - 03 Feb 2007 20:19 GMT

> I can't say for sure how your incident happened, but an example of what
> might be possible is a classic buffer overflow

Thanks for that, Bill. Given that I'm not sure exactly which message was
causing the crash (I deleted four or five in the queue, all of which
were clearly spam) I shall never know. But your description of a buffer
overflow was interesting and instructive.

Signature

Peter

Reply to this Message

John H Meyers - 13 Feb 2007 03:35 GMT

> A MIME message or some part of it may have a Content-Length header
> claiming one size, Eudora sees that and uses it to allocate memory...

The following states that "Content-Length" is not defined
in any email standard (indeed, I can't find it anywhere
in any *current* RFC for email or MIME that I have yet searched);
why would Eudora then obey it to allocate memory?
(any product that believes *anything* externally generated
is foolhardy, more so if it believes something not even defined).

http://1997.webhistory.org/www.lists/www-talk.1993q4/0189.html
"the Content-Length header is not blessed by any standard.
In fact, there is no definition. The MIME working group
explicitly decided NOT to include such a header in MIME,
because it could not be made to work correctly over the
variety of mail transports in existence.
(The current popular use of Content-Length in RFC 822 based email
is both nonstandard and hazardous. At best it is useless information.)"

-[ ]-

Reply to this Message

Bill Cole - 14 Feb 2007 16:03 GMT

> > A MIME message or some part of it may have a Content-Length header
> > claiming one size, Eudora sees that and uses it to allocate memory...
>
> The following states that "Content-Length" is not defined
> in any email standard (indeed, I can't find it anywhere
> in any *current* RFC for email or MIME that I have yet searched);

The idea that a standard has to be defined in an RFC for software to use
it is quaint and backwards. I suggest you review the IETF standards for
what needs to happen ahead of the publication of an RFC as a standard.

See Usenet as an example of a working system that relied for over a
decade on mechanisms without formal specification other than working
code.

> why would Eudora then obey it to allocate memory?

Eudora is ancient at its core (in Internet Time frame of reference.)
That carries with it the potential for assumptions that fit the
realities of Mac users at UIUC in 1988 to survive in code used by a
wider range of users.

The origin of the Content-Length header in email is with mail delivery
in System V, and in a limited context where a delivery agent on a
trusted system removes existing Content-Length headers and generates
correct ones at delivery time, a MUA could trust and use that header.

Hypothetically, that's a sort of bug that could survive a long time in a
classic MacOS program. Even if it wasn't done to directly allocate
memory but simply trusted in some other way (e.g. stashed as the length
of a message in the ToC) that could come back later as a basis for
allocating memory.

> (any product that believes *anything* externally generated
> is foolhardy, more so if it believes something not even defined).

I think you need to read my post more carefully and in context of the
post I was responding to. I did not say that Eudora does precisely what
I described. I do not have access to the Eudora source. The original
question was:

: But surely emails are just bytes and are
: handled neutrally? So how can they crash the mail client?

Since Steve Dorner has not been seen here in years, any answer is
inherently hypothetical and speculative. I was simply trying to explain
one of the simplest possible ways for a corrupt message to be
mishandled. Another example of a mechanism for buffer overflow could be
trusting the length information provided by a POP3 or IMAP server, which
is less likely to be a problem and is not user-visible, so it might well
survive longer. In principle it remains a bad idea to trust that number
either, but doing so would be an easy error.

In any case, your statement is true but does not change the reality that
Eudora would not be at all special in trusting input that should not be
trusted. It is a chronic problem of software, and avoiding that class of
error perfectly is harder than it sounds. Demonstrably, eudora has over
the years had a number of cases where pathological messages have caused
it to crash or behave badly, and while that should never happen, it
does. Eudora is far from the only MUA to suffer from such problems (see
the various bits of MS malware^Wmailware as examples.)

> http://1997.webhistory.org/www.lists/www-talk.1993q4/0189.html

Did you notice that you are using a mailing list post regarding HTTP
from 1993 as a reference in a discussion of email in 2007?

Even in the context that it refers to, that post is incorrect today.
RFC2616 defines Content-Length for HTTP messages, where it is actually
critical for proper handling of some transactions.

A quick search of my archives indicates that various versions of Elm,
Mutt, Outlook or OE, possibly Agent 1.8, something that might have been
Notes v4, and either Vixie Cron or the local mailer on a number of
FreeBSD boxes circa 2001 all generated Content-Length headers for
text/plain mail. A bunch of spammers have liked the header over the
years as well, but that may be an artifact of mimicking some old version
of OE. The point here is that various pieces of mail software that at
least strive for legitimacy have found that non-standard header useful
or harmless.

Signature

Now where did I hide that website...

Reply to this Message

John H Meyers - 15 Feb 2007 03:41 GMT

[...]

Thanks for all the thoughtful, well-expressed points.

> Did you notice that you are using a mailing list post regarding HTTP
> from 1993 as a reference in a discussion of email in 2007?

Of course, so I searched all the *current* RFCs for any change,
and found no change, as mentioned.

> Even in the context that it refers to, that post is incorrect today.
> RFC2616 defines Content-Length for HTTP messages, where it is actually
> critical for proper handling of some transactions.

I'm aware of it existing (somewhat optionally now in 1.1) for HTTP,
but I think we're discussing SMTP/POP/IMAP here :)

> A quick search of my archives indicates that various versions of Elm,
> Mutt, Outlook or OE, possibly Agent 1.8, something that might have been
[quoted text clipped - 5 lines]
> least strive for legitimacy have found that non-standard header useful
> or harmless.

Our Solaris also creates it in local POP/IMAP server end-user mailboxes,
and then complains of numerous "mismatches" in half the mailboxes
(due to untoward events, no doubt, but still present to a large extent,
the frequency rising in proportion to mailbox file size,
especially among those electing to "leave mail on server").

I see it displayed by Eudora (Windows) in messages picked up via POP,
but given the fact that "mismatches" are so common,
I hope that Eudora doesn't pay it much attention,
or else very many people would be unable
to retrieve their mail properly using Eudora
from our own POP server.

Sorry to have appeared to have accused you
of saying that Eudora actually uses it;
of course you were only indicating a possibility,
of something that I hoped Eudora would not do,
for all these reasons,
and thanks for all the information and background
which you have added.

-[ ]-

Reply to this Message

John H Meyers - 15 Feb 2007 04:00 GMT

As to other things that might cause Eudora to flake out
during message retrieval, anti-virus activity is one
which has affected me -- while I was using Eudora v3
(under Windows), there was a time when some virus
was so common that it was continually being detected
during "new mail" checks, and because, in that version,
Eudora was directly appending new messages
to the Eudora "In" mailbox, my entire "In" mailbox
would continually cease to exist (snatched away by AV)
in the middle of Eudora's operations, causing
a lot of noise from Eudora (and a very hard time
for me to recover my accumulated mail,
even if only one day's worth).

The parsing of attachments (which Eudora separates
at once from arriving messages) could provide
some additional opportunity for mischief,
including the possibility of maliciously formed
improper file names (sometimes attempting to escape
into and overlay other areas of the file system),
and other nasties.

I hope I've redeemed myself from the sin
of having strayed across the border again
(from Windows country :) -- thanks, and best wishes.

-[ ]-

Reply to this Message

Bill Cole - 17 Feb 2007 03:41 GMT

> As to other things that might cause Eudora to flake out
> during message retrieval, anti-virus activity is one
[quoted text clipped - 9 lines]
> for me to recover my accumulated mail,
> even if only one day's worth).

A great point, and one that jogged my memory...

In 2004 my posting here prompted a developer from a major AV vendor to
write me asking for clues on how to safely fiddle with Eudora live, as
Eudora was transferring mail. I told him it was a bad idea, and he
confessed that there had been a conscious choice in dealing with Eudora
to accept sometimes losing a race and corrupting mailboxes in order to
obliterate viruses. At the time I knew of many reports of crashing by
Eudora with that AV product.

I'm not naming the vendor/product because I think they reconsidered that
choice.

> The parsing of attachments (which Eudora separates
> at once from arriving messages) could provide
[quoted text clipped - 3 lines]
> into and overlay other areas of the file system),
> and other nasties.

There's a hard problem there. AV software developers are used to the
idea that they have to act fast and firm with malware, obliterating it
before it does anything evil. A good thing in most cases, but when it
means whacking pieces out of the middle of Eudora mailboxes, a very
dangerous bit of aggressiveness.

Signature

Now where did I hide that website...

Reply to this Message

John H Meyers - 15 Feb 2007 22:48 GMT

More thoughts about "Content-length:"

o Since Eudora obtains (via POP protocol)
only one message body at a time from the POP server,
it really doesn't need any information about message length
to separate consecutive messages, which is the purpose
served by that header within "mbox format" files on the server
(in other words, if the server generates those, as ours does,
it's for its own use, not for a POP client's use).

o Any re-interpretation, formatting, or attachment separation
would certainly invalidate any original overall length for
the entire message which might have existed on the POP server
to begin with, so any literal copy of an original "Content-length:"
header for the entire message from the POP server might no longer be valid
for the message when locally stored by Eudora.

I don't see any such headers in downloaded messages
except when they existed on the POP server to begin with,
so it doesn't seem that Eudora needs or creates them itself,
or has any use for them at all -- one might speculate
that they could have some use in an entire, multi-message
mailbox file, if the Table of Contents were lost and
needed re-construction, but if that were needed,
one would expect Eudora to re-calculate and store them itself,
much as servers do, but which Eudora does not seem to be doing.

Although the original question was just to wonder
how incoming messages might hang or bother Eudora,
the side-issue of these headers came up,
so I thought it might be of interest
to see what else there might be to discover about it.

-[ ]-

Reply to this Message

Mac Forum / Applications / Eudora / February 2007

Corrupt file- how does it work?