 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Mac Forum / Applications / Eudora / April 2008Odysseus update, change in file storage
Hi all, For those of you not following the Odysseus forums (Odysseus being a project from Infinity Data Systems, attempting to recreate the features of Eudora), there are two pieces of news: 1. There have been a couple of betas, with basic functionality. Version 1.0 is promised for May 12. (Many of us are skeptical that this date will be kept, but what software is ever delivered on time?) See <http://www.infinitydatasystems.com/products/odysseus/index.html> 2. They are changing from a SQLite Hybrid Method (their initial plan, and what they are currently using in the betas) to a maildir method (for beta 0.5 and beyond), rather than the mbox that Eudora currently uses. See <http://www.infinitydatasystems.com/forums/viewtopic.php?f=3&t=223> Roberta  SignatureRoberta Millstein usenet@spamaway.rlm.net Remove "spamaway" to reply -- Posted via a free Usenet account from http://www.teranews.com Thanks for the info. I admit I'm not very technically minded, so I'm wondering what all of that means for practical use. Are you on the beta programme? If so how do you like it? SignatureCheers Martin > I admit I'm not very technically minded, so I'm wondering what all of > that means for practical use. Well, if you go to the thread (again, that's <http://www.infinitydatasystems.com/forums/viewtopic.php?f=2&t=223> ), you will see there is some disagreement on that point. But reading through it might give you a sense of whether the concerns raised would apply to you. > Are you on the beta programme? If so how do you like it? I am, but the last beta came out when I was away at a conference, and by the time I was able to turn my attention to it, there were already a gazillion messages to wade through. So, I'm going to wait for the next round. My general impression is that people are pleased with what they saw. Roberta SignatureRoberta Millstein usenet@spamaway.rlm.net Remove "spamaway" to reply -- Posted via a free Usenet account from http://www.teranews.com On Sun, 30 Mar 2008 18:45:45 -0500, Roberta Millstein wrote: > Odysseus is changing from a SQLite Hybrid Method (their initial plan, > and what they are currently using in the betas) to a maildir method > (for beta 0.5 and beyond), rather than the mbox that Eudora currently uses. "Maildir" format stores one message per file; I have heard that Odysseus will use one folder per mailbox. IMO, Maildir organization offers some value for mail _servers_, but is a very poor, inefficient (re most other functions) and wasteful choice for personal computers. When it comes time to list, view, archive, backup, copy, Spotlight or otherwise process a large accumulation of mail over time, in which you may have tens or hundreds of thousands of individual files, rather than collections of messages per file (like Mbox format), I think this will become apparent. The article below covers some aspects used only on servers, but this may help indicate _why_ it was developed primarily for servers, rather than for personal computers: http://en.wikipedia.org/wiki/Maildir Traditional Mbox format: http://en.wikipedia.org/wiki/Mbox -- > IMO, Maildir organization offers some value for mail _servers_, > but is a very poor, inefficient (re most other functions) [quoted text clipped - 5 lines] > of individual files, rather than collections of messages per file > (like Mbox format), I think this will become apparent. Not with modern operating systems. Certainly not with OS X. I wrote: > IMO, Maildir organization offers some value for mail _servers_, > but is a very poor, inefficient (re most other functions) [quoted text clipped - 5 lines] > of individual files, rather than collections of messages per file > (like Mbox format), I think this will become apparent. On Mon, 31 Mar 2008 19:33:14 -0500, Elmo wrote: > Not with modern operating systems. > > Certainly not with OS X. I'm glad to hear that processing hundreds of thousands of files during every search, backup, archiving, copying, etc., does not slow down OS X, nor even require any more storage space, but Odysseus is a cross-platform product, which has to run even on Windows, where it very considerably does :) The standard "zip" archive format ("Compressed folders" in Windows) is limited to 65535 total objects, for example. If Leopard is going to kill "Classic Eudora," then that's reason enough for me not to switch to a Mac at home :) -- > > Not with modern operating systems. > > [quoted text clipped - 3 lines] > during every search, backup, archiving, copying, etc., > does not slow down OS X, nor even require any more storage space, That's right. It doesn't. OS X has great file handling, and backup programs don't scan every file to see if they've changed--backup programs count on the OS to tell it that. > but Odysseus is a cross-platform product, > which has to run even on Windows, where it very considerably does :) Doesn't Windows have a good file system? > If Leopard is going to kill "Classic Eudora," > then that's reason enough for me not to switch to a Mac at home :) I haven't switched to Mac OS 10.5, because if it ain't broke I don't fix it. I wrote: > I'm glad to hear that processing hundreds of thousands of files > during every search, backup, archiving, copying, etc., > does not slow down OS X, nor even require any more storage space. On Mon, 31 Mar 2008 20:07:40 -0500, Elmo wrote: > That's right. It doesn't. OS X has great file handling, and backup > programs don't scan every file to see if they've changed -- backup > programs count on the OS to tell it that. Get a stopwatch and check that out -- even checking the "last mod date/time" (actually it should be "last inode change time") for hundreds of thousands of files can not be instantaneous, nor is checking for the inferior "archive flag" on hundreds of thousands of Windows files, nor is Winzip's matching all this against the existing catalog of a previous backup, when performing an "update," etc. >> but Odysseus is a cross-platform product, >> which has to run even on Windows, where it very considerably does :) > Doesn't Windows have a good file system? NTFS can't perform miracles, either :) There is an overhead per file in any processing, as well as an overhead per KB of processed content; file systems usually also allocate in units which leave some "slack" (unused space) in every file, plus space which contains the characteristics of every individual file, which makes a large collection of small files take more space (and more time to process in any way) than a collection of fewer combined files; for very small files, the overhead per file may be much more than either the space or the processing time for its content alone. > I haven't switched to Mac OS 10.5, > because if it ain't broke I don't fix it. And I'm not switching from XP to Vista, not only because XP ain't broke, but because Vista is :) Sooner or later, however, the vendor not only stops "supporting" the older OS, but somehow forces new software to start requiring it, and one day the pressure is too much to continue living with, although at different points for different folk. -- > > > Not with modern operating systems. > > > [quoted text clipped - 7 lines] > programs don't scan every file to see if they've changed--backup > programs count on the OS to tell it that. Which backup programs do that? Of the three with which I am familiar (Retrospect, Super Duper!, and Time Machine), only Time Machine uses functionality built in to the OS to decide which files to copy. That is why I can run Time Machine every hour. The others will take 10-15 minutes to scan my entire disk and its ~100,000 files, even if there are only 100 MB or so to backup. That is why I run these other programs at night. I don't it would be possible for more than one backup program to use OS features to determine which files have been modified. And even one program doing that will have problems maintaining more than one backup copy. I try to have at least 3 different backups of my computer, because in my experience backups are very fragile and easy to get wrong. > I haven't switched to Mac OS 10.5, because if it ain't broke I don't fix > it. So do you have any experience with Time Machine? SignatureJim Gibson > Doesn't Windows have a good file system? No it certainly does not! The Windows filing infrastructure is a pathetically limited system that we long suffering users have had inflicted upon us and which we've endured for years. Where would you have me to begin? Perhaps here: It has: - A pitifully limited number of file attributes - want an off-the-cuff list of what's missing? - No extensible or on-the-fly user/program/OS-defined attributing and other such metadata. - No data/attribute encapsulation (attributes exist as relationships within the O/S but many disappear the moment the file leaves the operating system). (Next time you see an email attachment, ask yourself how many attributes were lost when it left its previous filing system--or even ones before that.) - No facility for history /[or other]/ metadata--previous computers, locations, user info, apps etc.--is built into the filing structure. - No intrinsic/built-in encryption/authentication infrastructure is available. - Cannot properly handle files with filenames with greater than 254 CHR$ yet Internet Explorer can save them longer than that. A Mexican stand-off then exists and the file can't be moved or deleted. /[A truly great design faux pas.] / - Plain language can't always be used in filenames. For instance, reserved characters such as '?' or ':' cannot be used. This stupid and ancient anomaly is a throwback to the days of CP/M. /[Don't say there is no way around this as there is, moreover an example exists within XP to show how it can be done.] / - No intrinsic SQL-like database infrastructure. /(Whatever happened to WinFS? With at least a wishful thought of WinFS, even Microsoft has accepted that existing brain-dead filing system must go)./ - Bogs down to snail's pace or slower with a 1000+ files in directory, absolutely glacial when it has ten times this number or more. Such numbers are impracticably small for many applications. - No intrinsic CRC-like checking facility for data corruption is built into the file structure (a file can be damaged internally yet still appear OK to the Windows filing system). (Here's a typical (and good) example of the problem: large Eudora .MBX mail files are notoriously susceptible to file corruption. When say your computer crashes with an open 400MB, 1GB or bigger Eudora .MBX file then you've a pretty good chance that sections of the file will be corrupted and you'll never find out until it's too late. Moreover, maintenance utilities, or say programs based on the Eudora Automation Server, will effectively see this corruption as an EOF (end of file marker) and truncate the file at that point--voilà, your 1GB of messages is now, say, down to only 400MB and Eudora is none the wiser--nor are you! Yep, it's the Windoze apology for a filing system that's directly responsible for that corruption. ...And I've only just begun. Even the most lackadaisical types should have a modicum of concern about this awful neolithic throwback. /BTW, UNIX filing systems et al are hardly much better. / >:-o > > Doesn't Windows have a good file system? > > No it certainly does not! [rant deleted] Definitely off-topic for the Mac Eudora newsgroup. Roberta SignatureRoberta Millstein usenet@spamaway.rlm.net Remove "spamaway" to reply -- Posted via a free Usenet account from http://www.teranews.com > >> [quoted text clipped - 9 lines] > Definitely off-topic for the Mac Eudora newsgroup. > Not really. Clearly, you don't have--and I'd say never seen--gigabytes of damaged Eudora files. I've about 35GB of .MBX files awaiting rescue all damaged by Windows crashes and the failure of its antique filing system to cope with such situations. Any replacement for Eudora should: - be hardened against such failures (even Microsoft .PST files don't suffer such an ill fate). - have vastly better mail management utilities (presently, compacting a damaged Eudora mailbox will truncate it). (Do you reckon this to be a good idea?) - that the new mail filing system for any Eudora replacement should be open so that 3rd party maintenance/backup/rescue are easily made (Qualcomm never published the specifications of the .TOC index files, this only exacerbated the problem). Developing a mail package shouldn't just consist of counting its pretty features, therefore discussing Eudora data security and management is clearly a prime topic for this newsgroup. As is usual in the software industry, those that develop software are never the ones found mopping up the shrapnel on the bleeding edge of user-land. Failing to adequately consider BORING data management issues as a fundamental part of the development cycle is no longer a satisfactory response. Users are demanding more, and if developers don't adequately respond then you'll eventually find the legislators forcing the matter with warranty legislation and software lemon laws. Sorry if the boring bits hurt. > Roberta > > > Definitely off-topic for the Mac Eudora newsgroup. > > [quoted text clipped - 4 lines] > by Windows crashes and the failure of its antique filing system to cope > with such situations. So how does your response address, in any way, Roberta's comment that such discussion is definitely off-topic for the Mac Eudora newsgroup? > Developing a mail package shouldn't just consist of counting its pretty > features, therefore discussing Eudora data security and management is > clearly a prime topic for this newsgroup. The discussion at hand wasn't about Eudora data security and management; it was about the shortcomings of the Windows filing system. Given that, how does your response address, in any way, Roberta's comment that such discussion is definitely off-topic for the Mac Eudora newsgroup? > Sorry if the boring bits hurt. Sorry if actual relevance to the discussion at hand hurts. I bet you have all sorts of trouble communicating with others at all levels in your life. > So how does your response address, in any way, Roberta's comment that > such discussion is definitely off-topic for the Mac Eudora newsgroup? In this case I suggest you should all go to the comp.mail.odysseus.mac group. SignatureVenlig hilsen / Best regards Palle Jensen If you want to E-mail me then please remove the US President. > > So how does your response address, in any way, Roberta's comment that > > such discussion is definitely off-topic for the Mac Eudora newsgroup? > > In this case I suggest you should all go to the comp.mail.odysseus.mac group. As opposed to the comp.mail.eudora.mac group? When the a.shole insists that his rant on Windows and its filing system is on-topic for this group, how is running to another Macintosh group going to change anything? Answer: it won't. > As opposed to the comp.mail.eudora.mac group? Yessirs! > When the a.shole insists that his rant on Windows and its filing system > is on-topic for this group, how is running to another Macintosh group > going to change anything? Are you tired of listening to off-topic conversations then? Guess what I am, listening to talk about Odysseus in a mac/Eudora group. SignatureVenlig hilsen / Best regards Palle Jensen If you want to E-mail me then please remove the US President. > Are you tired of listening to off-topic conversations then? Guess what > I am, listening to talk about Odysseus in a mac/Eudora group. You're tired of talking about the Eudora replacement in a Eudora group? > > Are you tired of listening to off-topic conversations then? Guess what > > I am, listening to talk about Odysseus in a mac/Eudora group. > > You're tired of talking about the Eudora replacement in a Eudora group? Umm, my Eudora replacement is Mailsmith, but I guess that doesn´t make it on topic here ;-) > > > Are you tired of listening to off-topic conversations then? Guess what > > > I am, listening to talk about Odysseus in a mac/Eudora group. [quoted text clipped - 3 lines] > Umm, my Eudora replacement is Mailsmith, but I guess that doesn´t make > it on topic here ;-) Remind me again how MailSmith is a replacement for Eudora? Does it have: 1) the ability to edit received mails/subject lines 2) a unified "Who" column 3) A "Send again" command 4) Attachments extracted 5) Automatic distinguishing of Sent mails (by making them italic) when they are moved to another mailbox from the Out mailbox 6) A tabbed interface on the Win version (too bad the Mac one doesn't have this) and a trillion other features I can't remember right now? Prolly not, which is why I stick with Eudora and am hoping that Odysseus will have these and other Eudora features PDQ. I would say that makes Odysseus completely on-topic for this NG, and MailSmith off, generally speaking. > Remind me again how MailSmith is a replacement for Eudora? Does it have: > > 1) the ability to edit received mails/subject lines I don´t need that. > 2) a unified "Who" column Yes > 3) A "Send again" command Yes > 4) Attachments extracted Yes > 5) Automatic distinguishing of Sent mails (by making them italic) when > they are moved to another mailbox from the Out mailbox Not that I know of but since all my sent mail is in a separate folder I don´t care. > 6) A tabbed interface on the Win version (too bad the Mac one doesn't > have this) Again I don´t need that. > and a trillion other features I can't remember right now? Yes :-) > > Remind me again how MailSmith is a replacement for Eudora? Does it have: > > > > 1) the ability to edit received mails/subject lines > > I don´t need that. I did check out MailSmith as a possible alternative to Eudora. This is a missing killer feature. > > 2) a unified "Who" column > > Yes Except is doesn't do the right thing for sent mails moved to another mailbox. > > 3) A "Send again" command > [quoted text clipped - 9 lines] > Not that I know of but since all my sent mail is in a separate folder I > don´t care. Another missing killer feature. I want to keep all mails that relate to a topic, together. > > 6) A tabbed interface on the Win version (too bad the Mac one doesn't > > have this) > > Again I don´t need that. It gives me quick access to the things I am working on at this moment. > > and a trillion other features I can't remember right now? > > Yes :-) Three missing killer features. Obviously not a Eudora replacement and therefore OT. End of story. This correspondence is now closed - Editor. > > > Are you tired of listening to off-topic conversations then? Guess what > > > I am, listening to talk about Odysseus in a mac/Eudora group. [quoted text clipped - 3 lines] > Umm, my Eudora replacement is Mailsmith, but I guess that doesn´t make > it on topic here ;-) As long as I have been reading this group (a long time), there has been discussion of other Mac email software in comparison to Eudora. To me that is a far cry from a discussion of the Windows file system. So, as far as I am concerned, discussion of Odysseus and Mailsmith in comparison to Eudora are both legitimate. At some point if Odysseus really takes off, it might make sense to have a separate group for it, but we're nowhere near that point yet. (I don't know where the Mailsmith people hang out). Roberta SignatureRoberta Millstein usenet@spamaway.rlm.net Remove "spamaway" to reply ** Posted from http://www.teranews.com ** > > > > Are you tired of listening to off-topic conversations then? Guess what > > > > I am, listening to talk about Odysseus in a mac/Eudora group. [quoted text clipped - 14 lines] > > Roberta Mailsmith has not been updated for over 3 years. To me that seems as if it is DEAD. SignatureKeep you in the dark You know they all pretend Keep you in the dark And so it all began In article <kroner-30185-1EC88D.14390613042008@70-3-168-216.area5.spcsdns.net>, > Mailsmith has not been updated for over 3 years. > > To me that seems as if it is DEAD. Even with a beta as recent as April 12, 2008? Roberta SignatureRoberta Millstein usenet@spamaway.rlm.net Remove "spamaway" to reply ** Posted from http://www.teranews.com ** > In article > <kroner-30185-1EC88D.14390613042008@70-3-168-216.area5.spcsdns.net>, [quoted text clipped - 6 lines] > > Roberta They just woke up but to me that shows lack of interest and support. Therefore any new software can not have many expectations. 3 years to a beta...yea right SignatureKeep you in the dark You know they all pretend Keep you in the dark And so it all began > > > > Are you tired of listening to off-topic conversations then? Guess what > > > > I am, listening to talk about Odysseus in a mac/Eudora group. [quoted text clipped - 12 lines] > but we're nowhere near that point yet. (I don't know where the > Mailsmith people hang out). Well, BB runs a list but its fairly quiet. There's a beta which addresses most of my wants except editable mails. > You're tired of talking about the Eudora replacement in a Eudora group? Actually, yes. My Mac replaced a PC. That doesn't allow me to spam PC groups with Mac postings. SignatureVenlig hilsen / Best regards Palle Jensen If you want to E-mail me then please remove the US President. [...] > > [rant deleted] > > > > Definitely off-topic for the Mac Eudora newsgroup. > > > Not really. *plonk* SignatureSander Tekelenburg, <http://www.euronet.nl/~tekelenb/> Mac user: "Macs only have 40 viruses, tops!" PC user: "SEE! Not even the virus writers support Macs!" "Irritated User" has made many mis-statements and mis-directed complaints, but I'm sure is too busy foaming at the mouth to listen, or to be benefited by any time invested by others in supplying useful information. Bellyaching is commonly an inadequate response to self-caused ills, in attempt to assign responsibility elsewhere. -- > > > >> > >>> > >>> Doesn't Windows have a good file system? ~~~~~~~ > >>> > >> No it certainly does not! [quoted text clipped - 8 lines] > I've about 35GB of .MBX files awaiting rescue all damaged > by Windows crashes and the failure of its antique filing system ~~~~~~~ Comp.mail.eudora.mac. Mac-in-tosh. Macintosh, macintosh, macintosh. Made by that fruit company in Cupertino. Roberta SignatureRoberta Millstein usenet@spamaway.rlm.net Remove "spamaway" to reply -- Posted via a free Usenet account from http://www.teranews.com > --------------010608090609030207040302 > Content-Type: text/html; charset=ISO-8859-1 [quoted text clipped - 6 lines] > <title></title> > </head> Please do not, I repeat, NOT post HTML to Usenet. Thank you. hauke Btw: Are you sure you own the email address <irritated.user@whatever.com>? SignatureNow without signature. > I wrote: > [quoted text clipped - 22 lines] > The standard "zip" archive format ("Compressed folders" in Windows) > is limited to 65535 total objects, for example. And Eudora mailboxes are limited to 32767 messages. Maildir++ (which presumably is the variant any modern mail software would be using) does not put all messages in one directory, it splits them by mailbox. I think Elmo is overstating the ease with which MacOS X handles directories with very large numbers of files, but it does manage rather well in comparison to older versions using older versions of HFS or in comparison to FAT32 on Win2k (dunno about NTFS...) To put a random data point with real numbers on it, this is what my nightly disk clone said last night: | 11:22:12 PM | Info | Evaluated 1555307 items occupying 228.58 GB (174396 directories, 1340085 files, 40826 symlinks) | 11:22:12 PM | Info | Copied 43638 items totaling 0.95 GB (418 directories, 2396 files, 40824 symlinks) | 11:22:12 PM | Info | Cloned 224.93 GB of data in 1290 seconds at an effective transfer rate of 178.55 MB/s That's on a 2GHz Core Duo (iMac) with 2GB RAM running 10.4.11 (latest Tiger.) So, it is managing to stat well over 1000 filesystem nodes per second. It is easy to pick a side in the MUA storage format religious war and ignore the serious hardness of the problem in the modern world. Different people's priorities make different storage formats better for them, and the scaling demands of modern users make the shortcomings of each of the main choices intolerable for some slice of users. Personally, I like the idea of mixed formats: mbox for archives, something like Maildir++ for active mailboxes, and rich metadata indexes for all mailboxes so that you don't have to go mucking around inside the actual messages to keep status flags or search on key fields. I am influenced by the fact that I use IMAP and run Dovecot as my server, and it can support this sort of model. > If Leopard is going to kill "Classic Eudora," > then that's reason enough for me not to switch to a Mac at home :) Classic Eudora has a long list of issues on Tiger, they just are not fatal. It seems to me that the Leopard issues are also non-fatal, if a bit more annoying. SignatureNow where did I hide that website... > Maildir++ (which presumably is the variant any modern mail software > would be using) does not put all messages in one directory, it splits [quoted text clipped - 4 lines] > data point with real numbers on it, this is what my nightly disk clone > said last night: Thanks, Bill. I was hoping you'd weigh in on this, since you've discussed mail formats many times in the past and seem to know more about it than most of us. You confirmed my general impression; current OSes and computers make things better, but there are still performance issues to be reckoned with. > It is easy to pick a side in the MUA storage format religious war and > ignore the serious hardness of the problem in the modern world. > Different people's priorities make different storage formats better for > them, and the scaling demands of modern users make the shortcomings of > each of the main choices intolerable for some slice of users. And that's how the debate seems to be playing out on the IDS forums; we have different priorities, and so argue for different things. And some just don't really care much one way or the other. > Personally, I like the idea of mixed formats: mbox for archives, > something like Maildir++ for active mailboxes, and rich metadata indexes > for all mailboxes so that you don't have to go mucking around inside the > actual messages to keep status flags or search on key fields. I am > influenced by the fact that I use IMAP and run Dovecot as my server, and > it can support this sort of model. I wonder if IDS would consider something like that? Seems like a reasonable compromise. > Classic Eudora has a long list of issues on Tiger, they just are not > fatal. It seems to me that the Leopard issues are also non-fatal, if a > bit more annoying. They seem quite serious for some, and not at all for others. So far, I'm in the latter camp. *knocks head* Roberta SignatureRoberta Millstein usenet@spamaway.rlm.net Remove "spamaway" to reply -- Posted via a free Usenet account from http://www.teranews.com On Wed, 02 Apr 2008 23:10:29 -0500, Roberta Millstein wrote: > [Bill Cole] confirmed my general impression; > current OSes and computers make things better, > but there are still performance issues to be reckoned with. One point of misunderstanding may be in comparing a "local machine" based protective mechanism (e.g. "Time Machine") with the general issue of making a "portable" or "removable" archive (I was focusing on the issue of backup and archiving). I've probably mentioned that the "POP account" which I use with Eudora is at Gmail. Since all my incoming messages (and even outgoing messages, via "Bcc:") are stored there, it can be said that "zero time" is required by my "backup" operations, because each new message is simply added to the already-accumulated message store, which acts as my "backup." "Time machine" could be (not that I actually know its internals) simply noting which files have changed, now and then "checkpointing" the accumulated list, and paying no attention to those not receiving any new activity. What I'm referring to, however, is the operation of making a complete, _removable_ archive (or backup) of all my Eudora data; this always requires [re-]inspection of every single file, plus an entry in a "catalog" for every single file, etc., which creates an unavoidable "per file" overhead in various ways (including file "slack" space, compression inefficiency of so many small files vs. fewer but larger files, file descriptors for every individual file, directory entries and space for every individual file, opening and closing every individual input and output file during copying of an entire store, and so forth). Searching is another activity which is strained by per-file processing; the only way to _seem_ to compensate for that is by an extraordinary amount of up-front indexing, which only shifts the burden around, also requiring far more storage, plus a complex and more fragile system, where certain activities (e.g. editing messages) tax its coherence, and which also commonly have more limitations in searching (e.g. only "whole words" [like Google], "words beginning with" etc.) Even returning to the issue of local file storage, the NTFS (Windows) file system makes an estimate, based on the size of an entire disk, how large a "Master File Table" to originally allocate; if the system actually decides to create vastly many more (smaller) files than the anticipated distribution, that "MFT" may become fragmented when its initial allocation runs out, reducing overall efficiency (the MFT itself can not be "defragmented" during normal defragmentation); practically anything which goes to extremes will end up falling away from an optimum, and the "one message per file" strategy, apparently conceived for use on servers handling dynamic, temporary storage of transient message traffic, is pretty extreme when it comes to being used for a long-term, permanent repository, accumulating an enormous number of permanent messages. Is there any successful database system, by the way, which stores "one record per file"? -- > On Wed, 02 Apr 2008 23:10:29 -0500, Roberta Millstein wrote: > [quoted text clipped - 6 lines] > issue of making a "portable" or "removable" archive > (I was focusing on the issue of backup and archiving). John, there's no misunderstanding here. I'm referring to the discussion over at IDS, where the claim is that there are no performance issues with maildir that can be addressed within the email software and by improved computing power conjoined with modern OSes. I've been arguing just as you are, that it depends on how one is using one's computer. If all you're using is Time Machine, then things are golden and maildir is not an issue. But if you want to create a portable archive, or do a full backup, then maildir is going to suffer from performance problems. > "Time machine" could be (not that I actually know its internals) > simply noting which files have changed, now and then "checkpointing" > the accumulated list, and paying no attention to those not receiving > any new activity. Yes, that is more or less how it works. > What I'm referring to, however, is the operation of making > a complete, _removable_ archive (or backup) of all my Eudora data; [quoted text clipped - 6 lines] > opening and closing every individual input and output file during copying > of an entire store, and so forth). Yep, I'm with you. But it's not me that you have to convince. > Is there any successful database system, by the way, > which stores "one record per file"? An interesting question. Roberta SignatureRoberta Millstein usenet@spamaway.rlm.net Remove "spamaway" to reply -- Posted via a free Usenet account from http://www.teranews.com On Thu, 03 Apr 2008 13:09:46 -0500, Roberta Millstein wrote: JHM>> One point of misunderstanding may be in comparing a "local machine" ... > John, there's no misunderstanding here... > I'm with you. But it's not me that you have to convince. The misundertanding I meant was that Elmo's original assertions seemed based on the "local machine" direction of thought, and that the conflict between those assertions and my views could be resolved by illustrating the different context. If I happened to reply to the last post I saw and it was yours, that was just an artifact of my "FIFO newsgroup reading" habits (I can't even thread a needle :) >> Is there any successful database system, by the way, >> which stores "one record per file"? > An interesting question. I hoped so :) Best. > On Wed, 02 Apr 2008 23:10:29 -0500, Roberta Millstein wrote: > [quoted text clipped - 17 lines] > the accumulated list, and paying no attention to those not receiving > any new activity. FWIW, here is how it works: it hooks file system events to keep track of what data and metadata needs to be backed up, and does the backup in hourly batches that take no prep time because every event that results in a need to do actual backup work has already been recorded. Similar ideas have been around for quite a while, with things like the Veritas storage suite's 'snapshot' features. It is my understanding that a similar functionality is available on Windows when using NTFS, but that may just be on the server version. That sort of backup mustering that is done all of the time by software that is hooked into the filesystem layer is where the future is for all routine backup. Storage systems have become too large in general to let the traditional model of a backup client being scanned retrospectively for changes at backup time to survive. A growing number of OS's are implementing ways for userspace software to receive notification of filesystem changes, and that sort of capability can do away with the need for a filesystem scan ahead of an incremental backup; all you need is a resident backup client agent that gets notified on every filesystem change. It is also worth noting that such a model is not logically limited to live local backup media. Supposedly Time Machine works just fine on laptops that leave their TM disks behind for long periods and reconnect later, although of course the granularity of the backups suffers. > What I'm referring to, however, is the operation of making > a complete, _removable_ archive (or backup) of all my Eudora data; [quoted text clipped - 6 lines] > opening and closing every individual input and output file during copying > of an entire store, and so forth). To some degree that's a backup implementation effect. A full backup of a discrete filesystem (i.e. usually a disk partition or the logical equivalent) can be a lot faster than an incremental backup of just what's changed since the last backup, if you can afford the space and transfer bandwidth taken by full backups. With a backup approach that tracks changes as they happen rather than scanning for them retrospectively, most of the effort of figuring out what to back up is eliminated. > Searching is another activity which is strained by per-file processing; > the only way to _seem_ to compensate for that is by an extraordinary [quoted text clipped - 3 lines] > and which also commonly have more limitations in searching > (e.g. only "whole words" [like Google], "words beginning with" etc.) I think you are overstating this. Any mail storage model has to include indexing of the message fields that it presents to users in a message list, or the UI ends up painfully slow. Message content searching is actually a pretty complex issue in regards to how mail is stored. Search that is done inside a single-threaded MUA is made a little easier by having a few large files rather than many small ones because the need to open and close each file is avoided, but when you look at the issue in relationship to the modern world of multi-threaded apps on multi-core machines and search tools that work on whole systems (e.g. Spotlight, Google Desktop Search, etc.) the picture is less clear. One of the things Mac users sometimes complain about in Eudora is that Spotlight can't be used to find text in specific messages, and in fact there is no way to get that sort of search from Spotlight without a file-per-message model, and it is unlikely that any other system-wide search tool would be able to provide such a feature without file-per-message storage. > Even returning to the issue of local file storage, the NTFS (Windows) Windows is off-topic here. :) > file system makes an estimate, based on the size of an entire disk, > how large a "Master File Table" to originally allocate; [quoted text clipped - 8 lines] > when it comes to being used for a long-term, permanent repository, > accumulating an enormous number of permanent messages. You might find it interesting that the file-per-message Maildir++ model has been made popular primarily by the growing use of IMAP, where servers retain messages indefinitely, not transiently. The original Maildir model was developed as a way to simplify concurrent access to a message store without needing to have all software aware of a common locking mechanism. That may not strike you as relevant to a MUA mailstore, but with multiprocessor/multicore/hyperthreaded machines becoming the norm and desktop OS's long supporting multithreaded apps it is actually a significant issue. > Is there any successful database system, by the way, > which stores "one record per file"? It depends on your definition of "database" and "successful" I suppose. The traditional news spool layout has retained a fair number of users for decades (with the addition of overview indexing) in large part because many people have found the accessibility to be useful, but also because it avoids locking issues and allows multiple concurrent writers to be active in the same directory (i.e. newsgroup.) SignatureNow where did I hide that website... > FWIW, here is how [Time Machine] works... Thanks for the info. > That sort of backup mustering that is done all of the time by software > that is hooked into the filesystem layer is where the future is for all > routine backup... > A full backup of a discrete filesystem (i.e. usually a disk partition > or the logical equivalent) can be a lot faster than an incremental backup > of just what's changed since the last backup... > With a backup approach that tracks changes as they happen > rather than scanning for them retrospectively, most of the effort > of figuring out what to back up is eliminated. Coming back down to the small task at hand, for me, I just want an independent backup of my Eudora folder, and even if my entire computer can be backed up over gigabit ethernet to a Storage Area Network in the same time, I don't care :) This is _personal_ backup, to be put on a CD, or a USB stick, or "split" and emailed to myself at Gmail, not a trip to Mars :) So I use good old "zip folders," which is mighty universal and compatible. If Odysseus stores one message per file, the efficiency of my backup would sink faster than stocks in 1929, and in fact it won't even be possible to do it any more. >> Searching is another activity which is strained by per-file processing; >> the only way to _seem_ to compensate for that is by an extraordinary [quoted text clipped - 3 lines] >> and which also commonly have more limitations in searching >> (e.g. only "whole words" [like Google], "words beginning with" etc.) > I think you are overstating this. Any mail storage model has to include > indexing of the message fields that it presents to users in a message > list, or the UI ends up painfully slow. I have no indexes at all, other than my TOCs (which don't index anything for searching, except that "summaries" searches the TOC itself), and my searches are not only plenty fast enough, but I can search even for embedded strings within words, etc. Others who enable the "superfast X1 search" (Eudora v7 only, sorry) write all the time about their computer "churning" and barely responding while indexing (for rebuilding), about the huge indexes created, about the inability to search text embedded in longer words, about the indexes "breaking" fairly often, about losing the flexibility to "drop in" or re-arrange mailboxes and still keep searches working, etc. So I like my searches better -- primitive implementation, not "super fast," but lowest overhead in other ways, and also most flexible. With Odysseus' "one message per file," my searching would go down the drain, or require me to build a highly complex and "indexed" system for just a simple task. Even though I have ten years' mail on hand, Eudora's current "dumb" system works best for me. > Message content searching is actually a pretty complex issue in regards > to how mail is stored. Search that is done inside a single-threaded MUA [quoted text clipped - 9 lines] > other system-wide search tool would be able to provide such a feature > without file-per-message storage. I don't want a massive "system wide" tool for Eudora; I want Eudora's own tool, which does the job for me perfectly. "Lean and mean," less "big government bureaucracy" :) >> Even returning to the issue of local file storage, the NTFS (Windows).. > Windows is off-topic here. :) There was a specific question about whether Windows' NTFS was comparable, and Odysseus (the topic) is still cross-platform, and needs to suit all. > You might find it interesting that the file-per-message Maildir++ model > has been made popular primarily by the growing use of IMAP, where > servers retain messages indefinitely, not transiently. I did raise the point that it's more suitable for servers than for personal computers :) > The original Maildir model was developed as a way to simplify concurrent > access to a message store without needing to have all software aware of > a common locking mechanism. That may not strike you as relevant to a MUA > mailstore... Right :) > but with multiprocessor/multicore/hyperthreaded machines > becoming the norm and desktop OS's long supporting multithreaded apps > it is actually a significant issue. As I asked before, does this recommend the storage of database records in a "one record per file" manner? > The traditional news spool layout has retained a fair number of users > for decades (with the addition of overview indexing) in large part > because many people have found the accessibility to be useful, but also > because it avoids locking issues and allows multiple concurrent writers > to be active in the same directory (i.e. newsgroup.) And they need not worry about space allocation (not using the same physical blocks), or when they can re-use the space, how to assure that no two write requests write into the same place, etc.? There is always a "locking system" -- if you remove it from one place, you have to make it re-appear in another. Besides being again a _server_ issue, rather than a one-user-at-a-time personal computer issue, you could do exactly the same by building an "inner filing system" within a single OS file, which, in effect, is what databases do, so I'm just not persuaded that dumping the job onto the OS file system is a profound advance. The thought to move it all to the OS file level just doesn't grab me, but fortunately, I only "think small," and what I want is a personal email client that stands on its own, doesn't make a mountain (of individual files) out of what ain't the least broke, has fit the bill perfectly for all this time, and don't need fixing :) On the other hand, if we abandon the entire "mailbox" concept, in favor of a "views" concept, like Gmail or Opera's mail, then it becomes imperative to think of each item as independent, absolutely requiring an extensive and more complex indexing system (though not necessarily wasting so much space as "one message per file"). It's no longer Eudora, however, nor apparently even Odysseus. -- > > FWIW, here is how [Time Machine] works... > [quoted text clipped - 25 lines] > the efficiency of my backup would sink faster than stocks in 1929, > and in fact it won't even be possible to do it any more. I guess I made my point poorly... Your choice of a backup method for your mail is influenced by how the mail is stored. Other methods exist that fit the bazillion-little-file model better, and they are becoming more widespread in all sorts of environments precisely because the numbers of files that people want to keep backed up are exploding. On the Mac side, Time Machine is an example, but my Windows colleagues have told me that there's some similar thing on their side. Backup strategies traditionally have spent a lot of compute time in order to economize on bytes transferred and stored, but for desktop backup these days the real economy of that has changed. I can't say what exactly the Windows alternatives are, but I'm sure they must exist. > >> Searching is another activity which is strained by per-file processing; > >> the only way to _seem_ to compensate for that is by an extraordinary [quoted text clipped - 12 lines] > and my searches are not only plenty fast enough, but I can search > even for embedded strings within words, etc. Searches that are restricted to the header fields in the TOC use the TOC as an index. Meta-data fields that only exist in the TOC use the TOC as an index. > Others who enable the "superfast X1 search" (Eudora v7 only, sorry) > write all the time about their computer "churning" and barely responding [quoted text clipped - 11 lines] > for just a simple task. Even though I have ten years' mail on hand, > Eudora's current "dumb" system works best for me. Again, I think you are overstating the slowdown from doing unindexed searches of a large number of files. One of the reasons I use the 'tradspool' storage method (file per article) for most of my news server is so that I can easily do arbitrary content searches (usually using grep) across the whole spool (or in specific group subtrees) and find specific articles, and doing so is not particularly slow. > > Message content searching is actually a pretty complex issue in regards > > to how mail is stored. Search that is done inside a single-threaded MUA [quoted text clipped - 14 lines] > > "Lean and mean," less "big government bureaucracy" :) And file-per-message does not prevent raw unindexed searches of all of the message content by the MUA, if that's what one prefers. different people prefer different approaches, and where mbox makes finding a specific message outside of the MUA (or some other mbox-aware tool) impossible, file-per-message allows anything that can read a text file to do a search and find specific messages. > >> Even returning to the issue of local file storage, the NTFS (Windows).. > [quoted text clipped - 9 lines] > I did raise the point that it's more suitable for servers > than for personal computers :) I was trying to address your conjecture that it was designed to fit transient spools rather than permanent storage. > > The original Maildir model was developed as a way to simplify concurrent > > access to a message store without needing to have all software aware of [quoted text clipped - 9 lines] > As I asked before, does this recommend the storage of database records > in a "one record per file" manner? If you don't want to implement a database that has record-level locking, and want to allow access from other tools that don't know about such locking or some internal data structure, yes. > > The traditional news spool layout has retained a fair number of users > > for decades (with the addition of overview indexing) in large part [quoted text clipped - 5 lines] > physical blocks), or when they can re-use the space, how to assure > that no two write requests write into the same place, etc.? Right. Filesystem implementations make certain operations 'atomic' from the perspective of userspace software. > There is always a "locking system" -- if you remove it from one place, > you have to make it re-appear in another. Right, and with file-per-message most locking issues are pushed down into the filesystem so that userspace processes never see an inconsistent state. For example, moving a message between mailboxes is just a matter of changing which directory the file is linked into, and userspace code cannot catch that procedure in a half-done state. > Besides being again a _server_ issue, rather than a one-user-at-a-time > personal computer issue, One user at a time does not mean one task at a time. The clunky 'background' mail checking in Eudora is an example, and it is degraded by having to manipulate messages in mbox files rather than files in directories. Similar issues can occur outside of the MUA as well, since things like backup software and AV scanners can catch an mbox mid-write or out of sync with its TOC. > you could do exactly the same by building > an "inner filing system" within a single OS file, which, > in effect, is what databases do, so I'm just not persuaded > that dumping the job onto the OS file system is a profound advance. That's another approach, but doing that moves a MUA in the direction of Outlook, with its opaque .pst files that nothing but Outlook can touch. > The thought to move it all to the OS file level > just doesn't grab me, but fortunately, I only "think small," > and what I want is a personal email client that stands on its own, > doesn't make a mountain (of individual files) > out of what ain't the least broke, has fit the bill perfectly > for all this time, and don't need fixing :) And we come back to my original point: the different approaches to mail storage all have their own advantages and disadvantages. Maildir-like models may waste a little allocation block space and complicate filesystem structures, but they can make multi-threaded MUA's smoother, bind metadata tightly to the messages, and open up the mail store to other tools. On the other hand, mbox storage forces the MUA to keep a separate record of where messages sit in the file and per-message metadata, and while they reduce filesystem complexity and theoretically waste less allocated but not used space, in practice they usually are not actually compacted on every message delete, so they have wastage internally that is hard to see. Purpose-built database-like storage can be very fast, efficient with space, support multi-threaded access and provide neat tricks like 'views' but all of that comes at the cost of hiding the messages from other tools. > On the other hand, if we abandon the entire "mailbox" concept, > in favor of a "views" concept, like Gmail or Opera's mail, [quoted text clipped - 3 lines] > > It's no longer Eudora, however, nor apparently even Odysseus. SignatureNow where did I hide that website... > File-per-message does not prevent raw unindexed searches of all of > the message content by the MUA, if that's what one prefers. different > people prefer different approaches, and where mbox makes finding a > specific message outside of the MUA (or some other mbox-aware tool) > impossible, file-per-message allows anything that can read a text file > to do a search and find specific messages. I think that's a "red herring." Windows can use "handlers" to look inside of anything; for example, some searches in XP look inside "zip" files by default (much to the annoyance of many users, but one can obtain "tweaks" to turn that off); other OS searches (for text) bypass certain files by default, including Eudora's mailboxes -- but that can likewise be turned back on, and since Eudora's mailboxes are just text, it is perfectly possible to search within them for specific messages, with no further "awareness" of "mbox format" than what you see in front of your eyes in any text viewer. No tool that is unaware of the MUA and its file structure (or "handler" for access by external programs) will fully replace the MUA's own tools, no matter what, because it probably won't interpret HTML, nor MIME encoded embedded objects, nor meta-data about messages (e.g. status, labels, separate attachments, etc.) I'm too lazy to further address other points (do I hear a sigh of relief? :) -- > > File-per-message does not prevent raw unindexed searches of all of > > the message content by the MUA, if that's what one prefers. different [quoted text clipped - 6 lines] > > Windows can use "handlers" to look inside of anything; Completely irrelevant here. Frankly, I don't touch Windows when not paid to do so. I regret the fact that Qualcomm fell into the trap of porting Eudora to Windows, and my skepticism about Odysseus is grounded in the fact that it is targeting the Windows commercial MUA market. > for example, some searches in XP look inside "zip" files by default > (much to the annoyance of many users, but one can obtain "tweaks" [quoted text clipped - 4 lines] > with no further "awareness" of "mbox format" than what you see > in front of your eyes in any text viewer. You didn't read all of what I wrote carefully enough. Understandable, given my verbosity.... Sure, I can find any string inside a Eudora mailbox in Spotlight. It will happily tell me which of the 269 mailboxes in my mail archives hold some string, and that narrows my hunt to ~6MB per mailbox... So I can use Eudora to search a subset. Compare that to the Apple Mail experience. Spotlight will identify the individual message, because messages are files. One search kicks up each message with the search string, rather than a collection of mailboxes with thousands of messages each. > No tool that is unaware of the MUA and its file structure > (or "handler" for access by external programs) > will fully replace the MUA's own tools, no matter what, > because it probably won't interpret HTML, > nor MIME encoded embedded objects, nor meta-data about messages > (e.g. status, labels, separate attachments, etc.) Well, as far as the relevance here is concerned, even if one wanted to create a special mail handler for Spotlight, there is no way to define metadata in a handler that applies to a part of a file. In principle, metadata that is applicable to a whole file can be exposed to Spotlight, if there is some way to determine it from or attach it to the file. > I'm too lazy to further address other points (do I hear a sigh of relief? :) Yeah, well, I would find squabbling with me tiresome too.... SignatureNow where did I hide that website... > Sure, I can find any string inside a Eudora mailbox in Spotlight. > It will happily tell me which of the 269 mailboxes in my mail archives > hold some string, and that narrows my hunt to ~6MB per mailbox... > So I can use Eudora to search a subset. Or you can use any plain text viewer to find your string in that mailbox, and there's (are) your message(s), exactly as you would find below, no? And of course a little script can automate the whole thing. > Compare that to the Apple Mail experience. Spotlight will identify the > individual message, because messages are files. One search kicks up each > message with the search string, rather than a collection of mailboxes > with thousands of messages each. So will super-fast "X1" (or a script, as above) in that "other OS," and without needing to split one's mail store into hundreds of thousands of pieces to do it -- just like I'd rather have a CAT scan than a dissection :) I just can't wait to see multi-million record database searches all based on Spotlight -- perhaps better than Google? :) -- > > Sure, I can find any string inside a Eudora mailbox in Spotlight. > > It will happily tell me which of the 269 mailboxes in my mail archives [quoted text clipped - 3 lines] > Or you can use any plain text viewer to find your string in that mailbox, > and there's (are) your message(s), exactly as you would find below, no? From a user interaction perspective? Not at all. > And of course a little script can automate the whole thing. That's of interest to scripters, but that's a pretty small niche. SignatureNow where did I hide that website... On Tue, 15 Apr 2008 21:10:55 -0500: >> And of course a little script can automate the whole thing. > That's of interest to scripters, but that's a pretty small niche. Scripts are posted all the time, one person write, all others use :) -- > Eudora mailboxes are limited to 32767 messages. On Mac; fixed on Windows. In archives (e.g. zip) it's _files_ (mailboxes) which are counted, however, not messages, so multi-message mailboxes (even a mere 32767 messages each) raise the limit on "messages per archive" by a factor of tens of thousands of times as many, compared to the limit using "one message per file." The latter runs into a brick wall at 65535 objects per "classic" zip archive (limit removed by less universal "extended" formats tied to specific vendors, each of which "encourages" everyone else to adopt theirs :) http://schmidt.devlib.org/file-formats/zip-archive-file-format.html http://www.honeynet.org/scans/scan24/sol/pedram/reference/zip-archive-file-forma t.html -- > > Eudora mailboxes are limited to 32767 messages. > > On Mac; fixed on Windows. Which is off-topic in *this* newsgroup, but it is worth mentioning in a context that might be on-topic here that the TOC differences break cross-platform portability of Eudora mailboxes, so that if a former Windows user wants to bring very large Eudora mail archives over to Mac, they may need to be broken into smaller mailboxes. (and yes, I am one of those people who believe that 1994 was a tragic year for Eudora... ) > In archives (e.g. zip) it's _files_ (mailboxes) > which are counted, however, not messages, [quoted text clipped - 11 lines] > http://www.honeynet.org/scans/scan24/sol/pedram/reference/zip-archive-file-for > mat.html It is odd that both of those say nothing about a files/archive limit, when that's what you are talking about. And by the way, you are not quite correct. The definitive description of the Zip format is http://www.pkware.com/documents/casestudies/APPNOTE.TXT, which describes the Zip64 extensions to support largefiles and a 64-bit files/archive field. Zip64 is *NOT* legally encumbered, and there are free (public domain Java and BSD-licensed C) implementations available so that anyone who uses Zip and wants to get past the 64k limit but lacks adequate programming skills (e.g. the guys up in Redmond) can freely borrow the work done by their betters. I should add that I don't think that the *technical* hard limits on the number of messages in a mailbox (no matter how that is implemented) are really significant in a MUA to 99.9% of users, because the human factors kick in at much lower levels than 32k messages in one place and drive most people to subdivide even historical archives into smaller sets that they can work with and not feel lost in. Once in a while someone shows up here and moans about the 32k messages/mailbox problem, but it always seems to be someone trying to make Eudora into a data mining tool or some other sort of gross misapplication. For the most part, people do not want or need their MUA to present a collection of tens of thousands of messages to them in one flat list, so hard technical limits built into a storage model that end up only limiting that sort of use don't mean a lot pragmatically. SignatureNow where did I hide that website... >> > Eudora mailboxes are limited to 32767 messages. >> On Mac; fixed on Windows. > Which is off-topic in *this* newsgroup Odysseus (thread topic) is cross-platform, so its issues and comparisons may also be. > but it is worth mentioning in a > context that might be on-topic here that the TOC differences break > cross-platform portability of Eudora mailboxes What about the line endings (which, if they _have_ to be changed, necessarily also change relative message offsets in the TOC)? One might have wished that each program could have accepted all sensible line endings; would that have made compatibility possible? I don't know what removed the 32767 limit from Windows, but a user isn't asked to reformat TOCs, and I've seen someone claim that they can inter-operate all versions (alternately) at any time, on the same platform of course. > so that if a former Windows user wants to bring very large Eudora mail archives > over to Mac, they may need to be broken into smaller mailboxes. > > (and yes, I am one of those people who believe that 1994 was a tragic > year for Eudora... ) I don't know the inference, but you can enlighten me if you wish. >> In archives (e.g. zip) it's _files_ (mailboxes) >> which are counted, however, not messages, [quoted text clipped - 10 lines] >> http://schmidt.devlib.org/file-formats/zip-archive-file-format.html >> http://www.honeynet.org/scans/scan24/sol/pedram/reference/zip-archive-file-forma t.html > It is odd that both of those say nothing about a files/archive limit, > when that's what you are talking about. I listed those as places to find general information about zip file formats, not intended as related only to the last prior sentence. > And by the way, you are not quite correct. > > The definitive description of the Zip format is > http://www.pkware.com/documents/casestudies/APPNOTE.TXT, which describes > the Zip64 extensions to support largefiles and a 64-bit files/archive field. As far as I can see, BOTH pkware AND Winzip have "defined" their own _extensions_ and each has "invited" everyone else to adopt theirs, which is why I didn't make links to either one of them. Now, are both company's extensions compatible? I've thought not, but if they are, my next paragraphs are moot, and I'll be glad of it: Just because pkware was the earlier company does not mean that they can control everything that becomes of material released to public domain, any more than IBM controlled the DES encryption that they originated, etc. But I didn't set out to argue which is "right" or "better" -- only to mention that if "extended" formats are incompatible (and if Windows hasn't adopted either as built-in), then "extended" zip archives are tied (for most users who just want to use pre-packaged software) to whichever system they make them with (or others who decide to make themselves compatible, in case "7-Zip" etc. join any particular bandwagon, or perhaps work with both, joining the crowd who tend to say "I'll decode anything, but I'll only create mine" -- which is just like email, where it's "I'll import from any other program, but there's no export" :) > Zip64 is *NOT* legally encumbered, and there are free (public > domain Java and BSD-licensed C) implementations available so that anyone > who uses Zip and wants to get past the 64k limit but lacks adequate > programming skills (e.g. the guys up in Redmond) can freely borrow the > work done by their betters. Enjoy the crusade :) > I should add that I don't think that the *technical* hard limits on the > number of messages in a mailbox (no matter how that is implemented) are [quoted text clipped - 9 lines] > into a storage model that end up only limiting that sort of use don't > mean a lot pragmatically. I don't know where this is going, but my point, re Odysseus' file storage (which started as the topic :) is that if IDS stores only one message per file, the rather small limit of 65537 objects in the original zip format will require many people to go to a less universal "extended" format to make zip archives, while good old "Classic" Eudora's MBOX format, even if limited to 32767 messages per mailbox, could have expanded that limit (where "classical zip" format gives out) some tens of thousands of times over. In other words, my whole point was that IDS is making a poor choice, and I brought up this whole "zip business" (a common archiving method) to illustrate one of the many reasons why it's poor. "Remember Alice? -- this is a song about Alice!" [Arlo Guthrie - "Alice's Restaurant"] http://www.arlo.net/resources/lyrics/alices.shtml http://youtube.com/watch?v=5_7C0QGkiVo > >> > Eudora mailboxes are limited to 32767 messages. > [quoted text clipped - 11 lines] > What about the line endings (which, if they _have_ to be changed, > necessarily also change relative message offsets in the TOC)? Yes, if Windows mailboxes use CRLF (i.e. normal DOS/Windows linebreaking) instead of the bare CR used on the Mac, that would also break portability. > One might have wished that each program could have accepted > all sensible line endings; would that have made compatibility possible? In theory yes, but it is a harder thing to do than to postulate, since it would require a strategy to deal with edge cases in messages that use 8-bit encodings and non-text content. Eudora *could* have standardized on One True Format across platforms, and that would have been be a simpler choice. > I don't know what removed the 32767 limit from Windows, > but a user isn't asked to reformat TOCs, and I've seen someone claim > that they can inter-operate all versions (alternately) at any time, > on the same platform of course. On the Mac side it is a bit dumb: the MacOS Resource Manager puts a hard limit on the size of individual resources and the full resource fork, and the way Eudora lays out the TOC eats it all at 32k messages, largely filled with nulls. > > so that if a former Windows user wants to bring very large Eudora mail > > archives [quoted text clipped - 4 lines] > > I don't know the inference, but you can enlighten me if you wish. The year Eudora was ported to Windows (and the last year that the current source was published.) Competing with MS on their own turf is never a wise move. Eudora was the best Mac e-mail client and Qualcomm had adequate resources to keep it that way, had they resisted the urge to go after the PC side of the market. They spread their resources in a non-core area of their business too thin and flailed around trying to flank MS with ill-considered features that never really caught on. > >> In archives (e.g. zip) it's _files_ (mailboxes) > >> which are counted, however, not messages, [quoted text clipped - 29 lines] > and each has "invited" everyone else to adopt theirs, > which is why I didn't make links to either one of them. If you know of any technical spec for the WinZip extensions that are not part of the PKware "appnote" specification of the Zip file format, I would love to have a link. > Now, are both company's extensions compatible? > > I've thought not, but if they are, my next paragraphs are moot, My understanding from looking at the available documentation is that WinZip 9 and later have supported the PKWare 64-bit extensions completely and the differences between the PKWare and WinZip formats is in the encryption implementations. > and I'll be glad of it: I think you can be glad. > Just because pkware was the earlier company does not mean that they > can control everything that becomes of material released to public domain, > any more than IBM controlled the DES encryption that they originated, etc. It's hard to make that analogy between an algorithm invented with the specific goal of becoming a government standard and a file format invented for use in commercial software and made open by choice. PKWare defined the original format and published it in full and have supported the format as an evolving open standard that anyone can implement freely. > But I didn't set out to argue which is "right" or "better" -- only to mention > that if "extended" formats are incompatible (and if Windows [quoted text clipped - 14 lines] > > Enjoy the crusade :) No crusade involved, just noting that the problem you are citing is a problem of one vendor being behind the times in their implementation of an evolving open format. My real point that lead to this tangential diversion is that the problems specific to each of the options for how a MUA can store mail are limited to the ways particular subsets of users work with their mail and with their mail as part of their whole environment. The file-per-message approach certainly does work less well with some ways that people like to search and back up their mail. The mbox-file-per-logical-mailbox approach used by classical Eudora presents its own set of problems with other ways that other people prefer. There are alternatives that may partially address all of those problems at the cost of raising other issues such as transparency and complexity. I think that the bottom line for any new MUA is that because all of the solutions are imperfect, some compromise that addresses the problems of any simplistic model has to be made or else some significant slice of users won't use the new tool. One advantage Odysseus has over any existing MUA is that it has no legacy other than the loose affinity to the Eudora users, so they can pick an approach and tune it up in the early versions rather than be bound to some historical choice. SignatureNow where did I hide that website... > On Sun, 30 Mar 2008 18:45:45 -0500, Roberta Millstein wrote: > [quoted text clipped - 4 lines] > "Maildir" format stores one message per file; > I have heard that Odysseus will use one folder per mailbox. That was the original plan, but as I wrote above, they seem to have changed their minds. Roberta SignatureRoberta Millstein usenet@spamaway.rlm.net Remove "spamaway" to reply -- Posted via a free Usenet account from http://www.teranews.com |
Reply to this Message |
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.