Marc Perkel put forth on 1/12/2011 12:18 PM:

time sh -c "dd if=/dev/zero of=ddfile bs=8k count=2000000 && sync"

2000000+0 records in 2000000+0 records out 16384000000 bytes (16 GB) copied, 55.403 s, 296 MB/s

real 1m4.738s user 0m0.336s sys 0m20.199s

That's a horrible test case for a mail server, especially one using maildir storage. Streaming read/write b/w results are meaningless for mail I/O. You need a random I/O test such as bonnie++ or iozone to see your IOPS. We already know it'll be off the chart compared to mechanical drives though. Would still be neat to see the numbers. There's also the most realistic test, ironically, given the list on which you asked this question:

http://www.imapwiki.org/Benchmarking

:)

-- Stan

On 12.1.2011, at 21.15, Matt wrote:

I thought about doing this on my email server since its troubles are mostly disk I/O saturation but I was concerned about reliability. Have heard that after so many read/writes SSD will go bad.

There's no need to worry about that in any modern drives.

On 13.1.2011, at 19.37, Robert Brockway wrote:

Hi Timo. Wear levelling often isn't as good as is claimed on the box. Often wear levelling is only across subsets of the SSD not across the entire device.

I've seen several SSD drives fail in production after about 12 months of use, and this in low-write environments (eg, I log syslog to a remote syslog server). I'm refraining from deploying SSD for mail servers for now, much as I would love to.

How do they fail? Supposedly once a cell has reached its erase-limit it should become read-only. Maybe the failures had nothing to do with wearing?

One of the systems to fail was a firewall running off SSD.

SSD or CF?

It would appear it's also possible to damage some flash memory by powering off at the wrong moment? I had a router running on a nearly new SLC flash card and it kept suffering errors every 24 hours and perhaps it was filesystem corruption since it was kind of fixed when I rebooted. Then after a few more days it died completely, briefly I could repartition it and then an hour later I could no longer even get it detected to the OS and hence it appeared absolutely completely dead.

So that's a new 4GB SLC card, using around 500MB of it and a light writeable filesystem running pfsense (perhaps a few writes per minute) and it died inside a month... I don't have enough data to see if it died from wear or if I was just unlucky...

This is a cheap (ish) CF card though, not an SSD drive

Ed W

Matt put forth on 1/12/2011 1:15 PM:

I thought about doing this on my email server since its troubles are mostly disk I/O saturation but I was concerned about reliability. Have heard that after so many read/writes SSD will go bad. There are an awful lot of read/writes on an email server.

From: http://www.storagesearch.com/ssdmyths-endurance.html

"As a sanity check - I found some data from Mtron (one of the few SSD oems who do quote endurance in a way that non specialists can understand). In the data sheet for their 32G product - which incidentally has 5 million cycles write endurance - they quote the write endurance for the disk as "greater than 85 years assuming 100G / day erase/write cycles" - which involves overwriting the disk 3 times a day."

That was written in 2007. SSD flash cell life has increased substantially in the 3-4 year period since.

From a flash cell longevity standpoint, any decent SSD with wear leveling is going to easily outlive the typical server replacement cycle of 3-5 years, and far beyond that. Note that striping two such SSDs (RAID 0) will double the wear cycle life, and striping 4 SSDs will quadruple it, so that 85 years becomes 340+ years of wear life with a 4 SSD stripe (RAID 0).

Your misgivings about using SSDs are based on obsolete data from many years ago.

-- Stan

Miha Vrhovnik put forth on 1/13/2011 3:17 AM:

...

"As a sanity check - I found some data from Mtron (one of the few SSD oems who do quote endurance in a way that non specialists can understand). In the data sheet for their 32G product - which incidentally has 5 million cycles write endurance - they quote the write endurance for the disk as "greater than 85 years assuming 100G / day erase/write cycles" - which involves overwriting the disk 3 times a day."

That was written in 2007. SSD flash cell life has increased substantially in the 3-4 year period since.

Stan you are wrong on that...

You are correct. I was wrong. Let me fix that with a two word edit:

SSD life has increased substantially in the 3-4 year period since.

Overall useful life of an SSD (the whole device) has continued to increase substantially thanks to better wear leveling controllers and RAISE, even though individual flash cell life has continued to decrease as process geometries continue to decrease.

-- Stan

On Thu, 2011-01-13 at 20:19 +0100, Steve wrote:

I would not use MLC in a server environment. SLC has much better program/erase cycles per cell.

I wouldn't be overly worried about the underlying medium.

I'm more worried about the translation layer they use on top of it, to make it pretend to be spinning rust. It is essentially a file system, on top of which you are expected to layer another file system. Not particularly efficient, but at least TRIM addresses one of the biggest inefficiencies of that gratuitous extra layering.

The inefficiency is one thing, but it's the reliability that worries me. It's generally accepted that it takes at least 5 years for a file system implementation to truly reach maturity. And that's for open source code that you can debug, on a medium that you can access directly to do diagnosis and data recovery.

But what we're talking about here is a file system implemented inside a black box where you can't do any of that. And what's more, they keep changing it. Even if you manage to find some device that passes your testing, you may find that the next batch of the *same* device (from your point of view) actually contains completely different software *and* hardware if you take it apart.

These translation layers are almost always a complete pile of crap. Especially in the face of power failures, since they so often completely fail to implement basic data integrity features (the same kind of journalling features that also have to be implemented in the 'real' file system on top of this fake disk).

The best way to use flash is to have a file system that's *designed* for use on flash. The only problem with that is that it wouldn't work with DOS; you can't provide an INT 13h DISK BIOS handler to use it...

-- dwmw2

David Jonas put forth on 1/12/2011 6:37 PM:

I've been considering getting a pair of SSDs in raid1 for just the dovecot indexes. The hope would be to minimize the impact of pop3 users hammering the server. Proposed design is something like 2 drives (ssd or platter) for OS and logs, 2 ssds for indexes (soft raid1), 12 sata or sas drives in RAID5 or 6 (hw raid, probably 3ware) for maildirs. The indexes and mailboxes would be mirrored with drbd. Seems like the best of both worlds -- fast and lots of storage.

Let me get this straight. You're moving indexes to locally attached SSD for greater performance, and yet, you're going to mirror the indexes and store data between two such cluster hosts over a low bandwidth, high latency GigE network connection? If this is a relatively low volume environment this might work. But, if the volume is high enough that you're considering SSD for performance, I'd say using DRBD here might not be a great idea.

Anyone have any improvements on the design? Suggestions?

Yes. Go with a cluster filesystem such as OCFS or GFS2 and an inexpensive SAN storage unit that supports mixed SSD and spinning storage such as the Nexsan SATABoy with 2GB cache: http://www.nexsan.com/sataboy.php

Get the single FC controller model, two Qlogic 4Gbit FC PCIe HBAs, one for each cluster server. Attach the two servers to the two FC ports on the SATABoy controller. Unmask each LUN to both servers. This enabling the cluster filesystem.

Depending on the space requirements of your indexes, put 2 or 4 SSDs in a RAID0 stripe. RAID1 simply DECREASES the overall life of SSDs. SSDs don't have the failure modes of mechanical drives thus RAID'ing them is not necessary. You don't duplex your internal PCIe RAID cards do you? Same failure modes as SSDs.

Occupy the remaining 10 or 12 disk bays with 500GB SATA drives. Configure them as RAID10. RAID5/6 aren't suitable to substantial random write workloads such as mail and database. Additionally, rebuild times for parity RAID schemes (5/6) are up in the many hours, or even days category, and degraded performance of 5/6 is horrible. RAID10 rebuild times are a couple of hours and RAID10 suffers zero performance loss when a drive is down. Additionally, RAID10 can lose HALF the drives in the array as long as no two are both drives in a mirror pair. Thus, with a RAID10 of 10 disks, you could potentially lose 5 drives with no loss in performance. The probability of this is rare, but it demonstrates the point. With a 10 disk RAID 10 of 7.2k SATA drives, you'll have ~800 random read/write IOPS performance. That' may seem low, but that's an actual filesystem figure. The physical IOPS figure is double that, 1600. Since you'll have your indexes on 4 SSDs, and the indexes are where the bulk of IMAP IOPS take place (flags), you'll have over 50,000 random read/write IOPS.

Having both SSD and spinning drives in the same SAN controller eliminates the high latency low bandwidth link you were going to use with drbd. It also eliminates buying twice as many SSDs, PCIe RAID cards, and disks, one set for each cluster server. Total cost may end up being similar between the drbd and SAN based solutions, but you have significant advantages with the SAN solution beyond those already mentioned, such as using an inexpensive FC switch and attaching a D2D or tape backup host, installing the cluster filesystem software on it, and directly backing up the IMAP store while the cluster is online and running, or snapshooting it after doing a freeze at the VFS layer.

You should be able to acquire the FC SATABoy with all the drives and SSDs (depending on what size units you choose), plus the two server HBAs for $15-20k USD, maybe less. I've not purchased a unit with SSDs yet, only disk, and they've very reasonably priced compared to pretty much all other SAN arrays on the market. Nexsan's SSD pricing might be a little steep compared to Newegg, but the units they bundle are fully tested and certified with their array controllers. The performance is phenomenal for the price, but obviously there are higher performing units available.

-- Stan

David Jonas put forth on 1/14/2011 2:08 PM:

Raid10 is our normal go to, but giving up half the storage in this case seemed unnecessary. I was looking at SAS drives and it was getting pricy. I'll work SATA into my considerations.

That's because you're using the wrong equation for determining your disk storage needs. I posted a new equation on one of the lists a week or two ago. Performance and reliability are far more important now than total space. And today performance means transactional write IOPS not streaming reads. In today's world, specifically for transaction oriented applications (db and mail) smaller faster more expensive disks are less expensive in total ROI that big fat slow drives. The reason is that few if any organizations actually need 28TB (14 2TB Cavier Green drives--popular with idiots today) of mail storage in a single mail store. That's 50 years worth of mail storage for a 50,000 employee company, assuming your employees aren't allowed porn/video attachments, which which most aren't.

...

Having both SSD and spinning drives in the same SAN controller eliminates the high latency low bandwidth link you were going to use with drbd. It also eliminates buying twice as many SSDs, PCIe RAID cards, and disks, one set for each cluster server. Total cost may end up being similar between the drbd and SAN based solutions, but you have significant advantages with the SAN solution beyond those already mentioned, such as using an inexpensive FC switch and attaching a D2D or tape backup host, installing the cluster filesystem software on it, and directly backing up the IMAP store while the cluster is online and running, or snapshooting it after doing a freeze at the VFS layer.

As long as the SATAboy is reliable I can see it. Probably would be easier to sell to the higher ups too. They won't feel like they're buying everything twice.

Hit their website and look at their customer list and industry awards. They've won them all pretty much. Simple, reliable, inexpensive SAN storage arrays. No advanced features such as inbuilt snapshots and the like. Performance isn't the fastest on the market but it's far more than adequate. The performance per dollar ratio is very high. I've installed and used a SATABlade and SATABoy myself and they're extremely reliable, and plenty fast. Those were spinning models. I've not used SSDs in their chassis yet.

http://www.nexsan.com

You configure the controller and drives via a web interface over an ethernet port. There's a lot to love in the way Nexsan builds these things. At least, if you're a HardwareFreak like me.

-- Stan

...

The reason is that few if any organizations actually need 28TB (14 2TB Cavier Green drives--popular with idiots today) of mail storage in a single mail store. That's 50 years worth of mail storage for a 50,000 employee company, assuming your employees aren't allowed porn/video attachments, which which most aren't.

WTF? 28TB of mail storage for some is rather small. Good to see your still posting without a clue Stanley. Remember there is a bigger world out there from your tiny SOHO

I'm with you Noel.

We just bought 252TB of raw disk for about 5k users. Given, this is going in to Exchange on Netapp with multi-site database replication, so this cooks down to about 53TB of usable space with room for recovery databases, defragmentation, archives, etc, but still... 28TB is not much anymore.

Of course, Exchange has also gone in a different direction than folks have been indicating. 2010 has some pretty high memory requirements, but the actual IOPS demands are quite low compared to earlier versions. We're using 1TB 7200RPM SATA drives, and at the number of spindles we've got, combined with the cache in the controllers, expect to have quite a good bit of excess IOPS.

Even on the Dovecot side though - if you use the Director to group your users properly, and equip the systems with enough memory, disk should not be a bottleneck if you do anything reasonably intelligent. We support 12k concurrent IMAP users at ~.75 IOPS/user/sec. POP3, SMTP, and shell access on top of that is negligible.

I'm also surprised by the number of people trying to use DRBD to make local disk look like a SAN so they can turn around and put a cluster filesystem on it - with all those complex moving parts, how do you diagnose poor performance? Who is going to be able to support it if you get hit by a bus? Seems like folks would be better off building or buying a sturdy NFS server. Heck, even at larger budgets, if you're probably just going to end up with something that's essentially a clustered NFS server with a SAN behind it.

-Brad

Stan,

On 1/14/11 7:09 PM, "Stan Hoeppner" stan@hardwarefreak.com wrote:

The average size of an email worldwide today is less than 4KB, less than one typical filesystem block.

28TB / 4KB = 28,000,000,000,000 bytes / 4096 bytes = 6,835,937,500 = 6.8 billion emails / 5,000 users = 1,367,188 emails per user

6.8 billion emails is "not much anymore" for a 5,000 seat org?

You obviously don't live in the same world I do. Have you ever been part of a grant approval process and seen what kinds of files are exchanged, and with what frequency? Complied with retention and archival policies? Dealt with folks who won't (or can't) delete an message once they've received it?

Blithely applying some inexplicable figure you've pulled out of who-knows-where and extrapolating from that hardly constitutes prudent planning. We based our requirement on real numbers observed in our environment, expected growth, and our budget cycle. How do you plan? More blind averaging?

How much did that 252TB NetApp cost the university? $300k? $700k? Just a drop in the bucket right? Do you think that was a smart purchasing decision, given your state's $3.8 Billion deficit?

You're close, if a bit high with one of your guesses. Netapp is good to Education. Not that it matters - you know very little about the financial state of my institution or how capital expenditures work within my department's funding model.

I suppose I shouldn't be surprised though, you seem to be very skilled at taking a little bit of information and making a convincing-sounding argument about it... regardless of how much you actually know.

For comparison, as of Feb 2009, the entire digital online content of the Library of Congress was only 74TB. And you just purchased 252TB just for email for a 5,000 head count subsection of a small state university's population?

I work for central IS, so this is the first stage of a consolidated service offering that we anticipate may encompass all of our staff and faculty. We bought what we could with what we had, anticipating that usage will grow over time as individual units migrate off their existing infrastructure. Again, you're guessing and casting aspersions.

This is enterprise storage; I'm not sure that you know what this actually means either. With Netapp you generally lose on the order of 35-45% due to right-sizing, RAID, spares, and aggregate/volume/snapshot reserves. What's left will be carved up into LUNs and presented to the hosts.

1/3 of the available capacity is passive 3rd-site disaster-recovery. The remaining 2 sites each host both an active and a passive copy of each mail store; we design to be able to sustain a site outage without loss of service. Each site has extra space for several years of growth, database restores, and archival / records retention reserves.

That's how 16TB of active mail can end up requiring 252TB of raw disk. Doing things right can be expensive, but it's usually cheaper in the long run than doing it wrong. It's like looking into a whole other world for you, isn't it? No Newegg parts here...

-Brad

Brandon Davidson put forth on 1/14/2011 10:59 PM:

You obviously don't live in the same world I do. Have you ever been part of

Not currently, no, thankfully.

a grant approval process and seen what kinds of files are exchanged, and

I've never worked in the public sector, only private, so I've not dealt with the grant process, but I'm not totally ignorant of them either. I've assisted a couple of colleagues in the past with grant proposals. And yes, they can, and often do, suck.

with what frequency? Complied with retention and archival policies? Dealt with folks who won't (or can't) delete an message once they've received it?

I have, unfortunately, had to deal with regulatory compliance and some of the less than sane communications retention policies.

Blithely applying some inexplicable figure you've pulled out of who-knows-where and extrapolating from that hardly constitutes prudent planning.

Statistics are guidelines. As I'm not planning anything in this thread, I don't see how such non existent planning could be categorized as prudent or not. What I did do is simply make the case that 252TB seems bleeping outrageously high for 5k users, whether that entails email alone or every other kind of storage those 5k users need. If my math is correct, that's about 50GB/user, including your snapshot LUNs, etc.

We based our requirement on real numbers observed in our environment, expected growth, and our budget cycle.

You forgot to mention the 35-45% (mentioned below) gross storage loss due to inefficiencies in your chosen hardware vendor's platform/architecture. Over a third and almost half of the drive cost is entangled here is it not?

How do you plan? More blind averaging?

Ouija board.

You're close, if a bit high with one of your guesses. Netapp is good to Education.

Vendors with the largest profit margins (read: over priced) built into their products are those most willing and able to give big discounts to select customers.

Not that it matters - you know very little about the financial state of my institution or how capital expenditures work within my department's funding model.

That's true. I know nothing about the financial state of your institution. I didn't claim to. I simply know Oregon was/is facing a $3.8B deficit. Your institution is part of the state government budget. Thus, your institution's spending is part of that budget/deficit. That's simply fact. No?

I suppose I shouldn't be surprised though, you seem to be very skilled at taking a little bit of information and making a convincing-sounding argument about it... regardless of how much you actually know.

I know this: 252TB is bleeping ridiculously large for 5K seats at _any_ university, public or private, regardless of how much is wasted for "data management". Also, 34%-45% consumption of raw for any internal array functions/management is bleeping ridiculous. Is that your definition of "enterprise"? Massive required waste of raw capacity?

I work for central IS, so this is the first stage of a consolidated service offering that we anticipate may encompass all of our staff and faculty. We bought what we could with what we had, anticipating that usage will grow over time as individual units migrate off their existing infrastructure. Again, you're guessing and casting aspersions.

Guessing? Originally you stated that 252TB for email only, or specifically Exchange. You said nothing of a mass storage consolidation project:

Brad Davidson put forth on 1/14/2011 6:25 PM:

We just bought 252TB of raw disk for about 5k users. Given, this is going in to Exchange on Netapp

Casting aspersions? aspersion:

a : a false or misleading charge meant to harm someone's reputation <cast aspersions on her integrity>

What false or misleading charge did I make with the intention to harm your reputation Brad? I've merely made a technical argument for sane email retention policies, and against the need for 252TB for 5K users' email. I don't recall casting any aspersions.

This is enterprise storage; I'm not sure that you know what this actually means either. With Netapp you generally lose on the order of 35-45% due to right-sizing, RAID, spares, and aggregate/volume/snapshot reserves. What's left will be carved up into LUNs and presented to the hosts.

If your definition of "enterprise storage" is losing 34-45% of raw capacity for housekeeping chores, then I'll stick with my definition, and with Nexsan for my "enterprise" storage needs.

You didn't mention deduplication once yet. With Nexsan's DeDupe SG I cut my regulatory dictated on disk storage requirements in half, and Nexsan disk costs half as much as NetApp, for the same SATA disks. With Nexsan, my overall storage costs are less than half of a NetApp, for basically the same capability. The only "downside" is that I can't get all of the functionality in a single head controller--however in many ways this is actually an advantage. My total costs are still far lower than going with an FAS, CLARiiON, or HNAS+HCP. All software is integrated with no additional licensing costs. I have no restrictions on the number of SAN or CFS/CIFS hosts I can connect to a LUN or an exported filesystem.

Don't get me wrong--NetApp, EMC, and HDS all make great products with nice capabilities. However, you _really_ pay through the nose for it, and keep paying for it as you needs grow. With Nexsan I pay once and only once, unless/until I need more disks. No additional fees to unlock any capabilities already in the box. No restrictions.

1/3 of the available capacity is passive 3rd-site disaster-recovery. The remaining 2 sites each host both an active and a passive copy of each mail store; we design to be able to sustain a site outage without loss of service. Each site has extra space for several years of growth, database restores, and archival / records retention reserves.

Ok, so this 252TB of disk is actually spread out over multiple buildings with multiple FAS controllers, one in each building? And this 252TB isn't just for mail (and its safeguard data) as you previously stated?

That's how 16TB of active mail can end up requiring 252TB of raw disk. Doing things right can be expensive, but it's usually cheaper in the long run than doing it wrong. It's like looking into a whole other world for you, isn't it? No Newegg parts here...

There are many ways to "do things right". And I'd humbly suggest ballooning 16TB of mail (again ridiculously large) into 252TB of raw disk (you're still confusing us as to what is actually what in that 252TB) is not one of them. That's a 16x increase. Do you have deduplication and compression installed, enabled, and scheduled?

-- Stan

Stan Hoeppner stan@hardwarefreak.com wrote:

[...] The average size of an email worldwide today is less than 4KB, less than one typical filesystem block. [...]

Do not confuse "unix culture" of mostly plain text only email messages with "MS Junk" culture of overblown formatting with background images company logos as a few image files in every (internal) email.

-- [pl>en: Andrew] Andrzej Adam Filip : anfi@onet.eu Let thy maid servant be faithful, strong, and homely. -- Benjamin Franklin

Sven Hartge sven@svenhartge.de wrote:

Andrzej Adam Filip anfi@onet.eu wrote:

...

Stan Hoeppner stan@hardwarefreak.com wrote:

...

...

[...] The average size of an email worldwide today is less than 4KB, less than one typical filesystem block. [...]

...

Do not confuse "unix culture" of mostly plain text only email messages with "MS Junk" culture of overblown formatting with background images company logos as a few image files in every (internal) email.

I just did a rough analysis of the mail spool of my university (6.000 users, students and faculty staff, about 10 million mails) and the average mail size was at about 96KiB. Last year, this average was at 77KiB and in 2009 we were at 62KiB.

Mails the average size of 4KiB would then have been at a time when MIME was not yet invented, I believe. Somewhere in 1994.

Grüße, Sven.

I assume that in bigger organizations most mail stored in IMAP storage is internal. I also assume that size of typical mail in "unix/linux culture" and "MS culture" do differ. It may explain quite different experiences.

Could you elaborate about penetration by MS software/culture (especially about MS Exchange) in your university?

BTW I have seen a few (smaller) organizations with most (internal) mails below 4KB but remaining *huge* mails capable to very significantly influence average size. It makes me doubt about value of *bare* "average email size".

P.S. Anyway many organization are legally obliged to archive all emails.

-- [pl>en: Andrew] Andrzej Adam Filip : anfi@onet.eu Oh wearisome condition of humanity! Born under one law, to another bound. -- Fulke Greville, Lord Brooke

Sven Hartge put forth on 1/15/2011 9:29 AM:

Andrzej Adam Filip anfi@onet.eu wrote:

...

Stan Hoeppner stan@hardwarefreak.com wrote:

...

...

[...] The average size of an email worldwide today is less than 4KB, less than one typical filesystem block. [...]

...

Do not confuse "unix culture" of mostly plain text only email messages with "MS Junk" culture of overblown formatting with background images company logos as a few image files in every (internal) email.

I just did a rough analysis of the mail spool of my university (6.000 users, students and faculty staff, about 10 million mails) and the average mail size was at about 96KiB. Last year, this average was at 77KiB and in 2009 we were at 62KiB.

Mails the average size of 4KiB would then have been at a time when MIME was not yet invented, I believe. Somewhere in 1994.

No. You're doing a statistical mean. You need to be doing median. The reason should be obvious.

-- Stan

Andrzej Adam Filip put forth on 1/15/2011 4:02 AM:

Stan Hoeppner stan@hardwarefreak.com wrote:

...

[...] The average size of an email worldwide today is less than 4KB, less than one typical filesystem block. [...]

Do not confuse "unix culture" of mostly plain text only email messages with "MS Junk" culture of overblown formatting with background images company logos as a few image files in every (internal) email.

"average size of an email worldwide"

The bulk of all email is personal, not corporate: think Gmail, Hotmail, Yahoo, the 50k ISPs worldwide, etc. Average all of that together with the corporate mail (small percentage), and you're well under 4KB per message, especially considering the amount of SMS gatewaying going on with smart phones today. Most of those are one liners with the smtp header being 4 times the size of the body, with total message size being under 1KB.

-- Stan

Philipp Haselwarter put forth on 1/15/2011 8:32 PM:

,---- | More than 97% of all e-mails sent over the net are unwanted, according | to a Microsoft security report.[39] | | MAAWG estimates that 85% of incoming mail is "abusive email", as of the | second half of 2007. The sample size for the MAAWG's study was over 100 | million mailboxes.[40][41][42] | | Spamhaus estimates that 90% of incoming e-mail traffic is spam in North | America, Europe or Australasia.[43] By June 2008 96.5% of e-mail | received by businesses was spam.[18][unreliable source?] `----

I just have a tiny set of 4k spam mails, but they have an avg size of 39KB, ie well above 4KB.

This discussion has been in the context of _storing_ user email. The assumption is that an OP is smart/talented enough to get his spam filters/appliances killing 99% before it reaches intermediate storage or mailboxes. Thus, in the context of this discussion, the average size of a spam message is irrelevant, because we're talking about what goes into the mail store.

If you're storing significantly more than 1% of spam you need to get that under control before doing any kind of meaningful analysis of mail storage needs.

-- Stan

Am 16.01.2011 06:39, schrieb Noel Butler:

LOL this is just soooooooo funny., watching the no no no im right you're wrong, give up stanley, those on many lists are aware of your trolling, nobody cares about your lil SOHO world, this list contains many different sized orgs, and like someone else mentione,d the 4K email size is SO 1994, but, that about sums you up anyway.

On Sat, 2011-01-15 at 23:19 -0600, Stan Hoeppner wrote:

...

Philipp Haselwarter put forth on 1/15/2011 8:32 PM:

...

,---- | More than 97% of all e-mails sent over the net are unwanted, according | to a Microsoft security report.[39] | | MAAWG estimates that 85% of incoming mail is "abusive email", as of the | second half of 2007. The sample size for the MAAWG's study was over 100 | million mailboxes.[40][41][42] | | Spamhaus estimates that 90% of incoming e-mail traffic is spam in North | America, Europe or Australasia.[43] By June 2008 96.5% of e-mail | received by businesses was spam.[18][unreliable source?] `----

...

I just have a tiny set of 4k spam mails, but they have an avg size of 39KB, ie well above 4KB.

This discussion has been in the context of _storing_ user email. The assumption is that an OP is smart/talented enough to get his spam filters/appliances killing 99% before it reaches intermediate storage or mailboxes. Thus, in the context of this discussion, the average size of a spam message is irrelevant, because we're talking about what goes into the mail store.

If you're storing significantly more than 1% of spam you need to get that under control before doing any kind of meaningful analysis of mail storage needs.

the simple truth is SSD drives are fast, and it will/might be the future but mail people are conservative ( you might think about the hell loosing mail ), at present SSD drives tec is not ready for serving big mail stores. ( my opinion ) tec time is turning fast, so in whatever time we might all use only SSD drives then ,or not, we will see after all its not really a dovecot theme wich should lead to endless flames on this list

-- Best Regards

MfG Robert Schetterer

Germany/Munich/Bavaria

Btw, our average mailsize last we checked was 30KB. Thats a pretty good average as we're an ISP with a very wide user base. I think 4KB average is not a normal mail load.

Cor

On Sun, 2011-01-16 at 20:33 -0600, Stan Hoeppner wrote:

Cor Bosman put forth on 1/16/2011 5:34 PM:

...

Btw, our average mailsize last we checked was 30KB. Thats a pretty good average as we're an ISP with a very wide user base. I think 4KB average is not a normal mail load.

As another OP pointed out, some ISPs apparently have to deliver a lot of spam to mailboxen to avoid FPs, bumping up that average mail size considerably. Do you accept and deliver a lot of spam to user mailboxen?

Still assuming that 30K average is spam huh stanley, accept the fact you screwed up, your data is 1994, in fact I question its accuracy for even back then.

keep peddling, its funny to watch, thankfully its in the archives and google for any of your new prospective employers.

-------- Original-Nachricht --------

Datum: Sun, 16 Jan 2011 20:33:23 -0600 Von: Stan Hoeppner stan@hardwarefreak.com An: dovecot@dovecot.org Betreff: Re: [Dovecot] SSD drives are really fast running Dovecot

Cor Bosman put forth on 1/16/2011 5:34 PM:

...

Btw, our average mailsize last we checked was 30KB. Thats a pretty good average as we're an ISP with a very wide user base. I think 4KB average is not a normal mail load.

As another OP pointed out, some ISPs apparently have to deliver a lot of spam to mailboxen to avoid FPs, bumping up that average mail size considerably.

Spam does not bump the average mail size considerably. Average spam mails is way smaller then average normal mails. The reason for this is very simple: Spammers need to reach as many end users as possible. And they need to get those mails out as fastest as possible.

Do you accept and deliver a lot of spam to user mailboxen?

-- Stan

-- Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de

-------- Original-Nachricht --------

Datum: Mon, 17 Jan 2011 11:13:19 +0100 (CET) Von: Maarten Bezemer mcbdovecot@robuust.nl An: Dovecot Mailing List dovecot@dovecot.org Betreff: Re: [Dovecot] SSD drives are really fast running Dovecot

On Mon, 17 Jan 2011, Steve wrote:

...

Spam does not bump the average mail size considerably. Average spam mails is way smaller then average normal mails. The reason for this is very simple: Spammers need to reach as many end users as possible. And they need to get those mails out as fastest as possible.

Somewhat correct. Due to a lot of spamfilter setups skip messages above a certain size, we've seen an increase of such big messages. These affect the average quite severely.

An average, however, is only just that: an average. There may not even be 1 message that has exactly the average size...

When looking at last two weeks worth of spam that didn't come from obvious blacklisted sources, I see: 45 messages below 4KB (including quite some miserable failures that forgot to include a message body...) 127 messages above 8KB, of which only 14 above 20KB 940 messages between 4KB and 8KB

Yet, the _average_ was well above 8KB, due to a few 500KB+ messages.

You get 500Kb+ sized spam messages? That is not usual. I have not done any computation on my part but I remember seen last year (or so) a study showing that spam messages are usually below 64Kb.

Anyway... Why is this so ultra important how big spam messages are?

So, mean, median, or whatever, it's just lies, damn lies, and statistics.

-- Maarten

-- NEU: FreePhone - kostenlos mobil telefonieren und surfen! Jetzt informieren: http://www.gmx.net/de/go/freephone

-------- Original-Nachricht --------

Datum: Mon, 17 Jan 2011 13:45:51 +0100 Von: Giles Coochey giles@coochey.net An: dovecot@dovecot.org Betreff: Re: [Dovecot] SSD drives are really fast running Dovecot

On 17/01/2011 13:41, Steve wrote:

...

You get 500Kb+ sized spam messages? That is not usual. I have not done any computation on my part but I remember seen last year (or so) a study showing that spam messages are usually below 64Kb.

That can depend on what you clasify as SPAM. Many, 'newsletters' which you've been 'subscribed to' by negative option web-forms are considered SPAM by some, and those may contain PDF attachments of 500kb+

Welll.... I wrote about "usual" and those newsletter that you tag as Spam but have subscribed to them are definitely not the norm.

-- Best Regards,

Giles Coochey NetSecSpec Ltd NL T-Systems Mobile: +31 681 265 086 NL Mobile: +31 626 508 131 Gib Mobile: +350 5401 6693 Email/MSN/Live Messenger: giles@coochey.net Skype: gilescoochey

-- GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit gratis Handy-Flat! http://portal.gmx.net/de/go/dsl

On Mon, 17 Jan 2011, Steve wrote:

...

Von: Giles Coochey giles@coochey.net

That can depend on what you clasify as SPAM. Many, 'newsletters' which you've been 'subscribed to' by negative option web-forms are considered SPAM by some, and those may contain PDF attachments of 500kb+

Welll.... I wrote about "usual" and those newsletter that you tag as Spam but have subscribed to them are definitely not the norm.

I didn't count "newsletters" I subscribed to.. Always using traceable addresses for those. In this case, it was JPG spam with large pics. Some claiming to be LED lighting newsletter, others disguised as new year's greetings. But content showed something not quite related to LEDs or happy new year stuff :-P All these big spams are addressed to bogus addresses, and/or standard addresses like info@domain. Usually with info@some-other-domain in the From: header.

But these are my last 2 cents for this thread as it has been derailing for quite some time now ;-)

-- Maarten

Rick Romero put forth on 1/14/2011 8:29 PM:

And that's assuming a platter squeezing in 1TB of data at 7200RPMs doesn't get a comparable performance improvement to a higher rotational speed on a lower volume platter...

Size and density are irrelevant. Higher density will allow greater streaming throughput at the same spindle speed, _however_ this does nothing for seek performance. Streaming performance is meaningless for transaction servers. IOPS performance is critical for transaction servers. Seek performance equals IOPS performance. The _only_ way to increase mechanical disk IOPS is to increase the spindle speed the or the speed of the head actuator. If you've watched mechanical drive evolution for the past 20 years you've seen that actuator speed hasn't increased due to the physical properties of voice coil drive actuators.

Hell for the price of a single 250gb SSD drive, you can RAID 10 TEN 7200 RPM 500GB SATAs.

I think your pricing ratio is a bit off but we'll go with it. You'd get 50,000 4KB random IOPS from the SSD and only 750 IOPS from the RAID 10. The SSD could handle 67 times as many emails per second for 10 times the cost. Not a bad trade.

So while, yes, my 10 drive SATA RAID 10 ONLY performs 166MB/sec with a 'simplistic' dd test, In reality I just don't think Joe User is going to notice the difference between that and the superior performance of a single SSD drive when he POPs his 10 3k emails.

But Joe User _will_ notice a difference if this server with the RAID 10 mentioned above is supporting 5000 concurrent users, not just Joe. Responses will lag. With the SSD you can support 10000 concurrent users (assuming the rest of the hardware is up to that task and you have enough RAM) and responses for all of them will be nearly instantaneous. This is the difference SSD makes, and why it's worth the cost in many situations. However, doing so will require an email retention policy that doesn't allow unlimited storage--unless you can afford than much SSD capacity.

You can get 240,000 4k random IOPS and 1.9TB of capacity from two of these in a software RAID0 for $6,400 USD: http://www.newegg.com/Product/Product.aspx?Item=N82E16820227665

That's enough transactional IOPS throughput to support well over 50,000 concurrent IMAP users, probably far more. Of course this would require a server likely on the order of at least a single socket G34 AMD 12 core Magny Cours system w/2GHz cores, 128GB of RAM, and two free PCIe X4/X8 slots for the SSD cards, based on a board such as this SuperMicro: http://www.newegg.com/Product/Product.aspx?Item=N82E16813182240 (Actually this is the perfect board for running two of these RevoDrive X2 cards)

-- Stan

On Jan 15, 2011, at 6:30 AM, Charles Marcus wrote:

One thing we are looking at here (small 50+ userbase) is kind of a
'best of both worlds' setup - using SSD's (haven't decided yet to trust a
bare striped set or go with a 4 drive RAID10 - probably the latter so I can sleep at night) for the main OS and a limited amount of storage space per user (maildir) for active/recent email, then use another namespace with a much higher quota - I'm thinking about 10GB per user should
do in our environment - for 'slow' storage (cheap mechanical RAID10 setup) - ie, emails that are only accessed on occasion (mdbox).

Then, enforce a smallish per user quota (how much would depend on your particular environment, but I'm thinking something like 250 or maybe 500MB, since our users do get a lot of large attachments in the course of doing business) on their INBOX - & Sent, Drafts and Templates
folders too, but that's a question on my list of 'how to do' - how to easily place these 'special' folders on the 'fast' namespace, and all user created folders in the 'slow' namespace. It would be really nice if there were some kind of native way that dovecot could 'assign' the 'special' folders to the same namespace as the INBOX, and all other
user created folders to another...

Doing this will also help train users in proper email management - treating their INBOX just like they would a physical INBOX tray on
their desk. They wouldn't just let paper pile up there, why do so in their INBOX (because they 'can')? Ie, it should be something they should always strive to keep totally EMPTY. Of course this practically never happens, but the point is, they need to learn to make a decision once they are finished with it, and most importantly, take said action - either delete it, or file it.

Sounds like a great idea. I work with media companies where quotas can
be challenging.

-- Brad

On Mon, 2011-01-17 at 09:07 -0500, Charles Marcus wrote:

On 2011-01-15 9:30 AM, Charles Marcus wrote:

...

Then, enforce a smallish per user quota (how much would depend on your particular environment, but I'm thinking something like 250 or maybe 500MB, since our users do get a lot of large attachments in the course of doing business) on their INBOX - & Sent, Drafts and Templates folders too, but that's a question on my list of 'how to do' - how to easily place these 'special' folders on the 'fast' namespace, and all user created folders in the 'slow' namespace. It would be really nice if there were some kind of native way that dovecot could 'assign' the 'special' folders to the same namespace as the INBOX, and all other user created folders to another...

Timo - any chance you could comment on the best way to accomplish this - or if it is even possible right now? I'm hoping to start testing this in the next few weeks...

Well, you can have per-namespace quotas. But that of course means that the other namespace must have a prefix, so users would have to have something like:

• INBOX
• Drafts
• Sent
• bignamespace
  • work
  • etc.

Which wouldn't be very pretty. So if you then wanted to arbitrarily move mailboxes across different quota roots .. I'm not really even sure what would be a good way to configure that.

For INBOX only a separate namespace should be possible to implement without trouble though.

Quoting Stan Hoeppner stan@hardwarefreak.com:

...

Rick Romero put forth on 1/14/2011 8:29 PM:

...

And that's assuming a platter squeezing in 1TB of data at
7200RPMs doesn't get a comparable performance improvement to a higher rotational
speed on a lower volume platter...

Size and density are irrelevant.  Higher density will allow
greater streaming throughput at the same spindle speed, _however_ this does
nothing for seek performance.  Streaming performance is meaningless for
transaction servers. IOPS performance is critical for transaction servers.  Seek performance equals IOPS performance.  The _only_ way to increase mechanical disk IOPS is to increase the spindle speed the or the speed of the head
actuator.  If you've watched mechanical drive evolution for the past 20 years you've seen that actuator speed hasn't increased due to the physical properties
of voice coil drive actuators.

...

Hell for the price of a single 250gb SSD drive, you can RAID 10 TEN 7200 RPM 500GB SATAs.

I think your pricing ratio is a bit off but we'll go with it.  You'd get 50,000 4KB random IOPS from the SSD and only 750 IOPS from the RAID 10.  The SSD could handle 67 times as many emails per second for 10 times the cost.  Not a bad trade.

...

So while, yes, my 10 drive SATA RAID 10 ONLY performs 166MB/sec with a 'simplistic' dd test, In reality I just don't think Joe User is going to notice the difference between that and the superior performance of a single SSD drive when he POPs his 10 3k emails.

But Joe User _will_ notice a difference if this server with the RAID 10 mentioned above is supporting 5000 concurrent users, not just
Joe.  Responses will lag.  With the SSD you can support 10000 concurrent users
(assuming the rest of the hardware is up to that task and you have enough RAM) and responses for all of them will be nearly instantaneous.  This is the difference SSD makes, and why it's worth the cost in many situations.  However, doing so will require an email retention policy that doesn't allow unlimited storage--unless you can afford than much SSD capacity.

You can get 240,000 4k random IOPS and 1.9TB of capacity from two of these in a software RAID0 for $6,400 USD: http://www.newegg.com/Product/Product.aspx?Item=N82E16820227665

That's enough transactional IOPS throughput to support well over 50,000 concurrent IMAP users, probably far more.  Of course this would require a server likely on the order of at least a single socket G34 AMD 12 core
Magny Cours system w/2GHz cores, 128GB of RAM, and two free PCIe X4/X8 slots
for the SSD cards, based on a board such as this SuperMicro: http://www.newegg.com/Product/Product.aspx?Item=N82E16813182240 (Actually this is the perfect board for running two of these RevoDrive X2 cards)

I use pricewatch - so, yes, we may be talking refurb drives, but this is not an issue when you're saving enough money to just buy a few more of items you're already buying.

Also, if your filesystem is using 4k clusters, aren't you only using 1 random IOPS for a 4k email?   It just sounds to me like if you plan 'smarter', anyone can avoid the excessive costs of SSD and get 'end user similar' performance with commodity hardware.

Rick

On Sun, 2011-01-16 at 00:05 -0600, Stan Hoeppner wrote:

Using O_DIRECT with mbox files, the IOPS performance can be even greater. However, I don't know if this applies to Dovecot because AFAIK MMAP doesn't work well with O_DIRECT... ***Hay Timo, does/can Dovecot use Linux O_DIRECT for writing the mail files?

mmap doesn't matter, because mbox files aren't read with mmap. But I doubt it's a good idea to use O_DIRECT for mbox files, because even if it gives higher iops, you're using more iops because you keep re-reading the same data from disk since it's not cached to memory.

As for O_DIRECT writes.. I don't know if it's such a good idea either. If client is connected, it's often going to read the mail soon after it was written, so it's again a good idea that it stays in cache.

I once wrote a patch to free message contents from OS cache once the message was read entirely, because it probably wouldn't be read again. No one ever reported if it gave any better or worse performance. http://dovecot.org/patches/1.1/fadvise.diff

Quoting David Jonas djonas@vitalwerks.com:

I've been considering getting a pair of SSDs in raid1 for just the dovecot indexes.

While raid-1 is better than the raid-0 of the previous poster, do you really want to slow down your fast SDDs with software raid-1 on top of them?

The hope would be to minimize the impact of pop3 users hammering the server.

Proposed design is something like 2 drives (ssd or platter) for OS and logs

The OS is mostly cached, so any old drive should work. Logs are lots of writes, so a dedicated drive might be nice (no raid). Be sure to tune the OS for the logs...

2 ssds for indexes (soft raid1),

I'd probably just go with one drive, maybe have a spare as a cold- or
hot-spare in case of failure.

12 sata or sas drives in RAID5 or 6 (hw raid, probably 3ware) for maildirs. The

I'd say either raid-6 or raid-10 for this, depending on budget and size needs.

indexes and mailboxes would be mirrored with drbd. Seems like the best of both worlds -- fast and lots of storage.

drbd to where at what level? There was a some other discussion about this which basically said "Don't use drbd to mirror between VM guests" which I agree with. If you want to do this, use DRBD between VM servers (physical hosts) and not between VM guests (virtual hosts).

I do use a VM cluster with DRBD between the physical hosts, but not for mail services, and it works fine. Doing DRBD inside the virtual hosts though would not be good...

Does anyone run a configuration like this? How does it work for you?

No. I do 2 nodes, with DRBD between them, using GFS on them for both the mbox files and indexes... No virtualization at all... No SSD drives at all...

Anyone have any improvements on the design? Suggestions?

Only my advice about where to drbd if you are virtualizing, and what raid levels to use... But these are just my opinions and your milage may vary...

-- Eric Rostetter The Department of Physics The University of Texas at Austin

Go Longhorns!

El 13/01/11 17:01, David Woodhouse escribió:

On Wed, 2011-01-12 at 09:53 -0800, Marc Perkel wrote:

...

I just replaced my drives for Dovecot using Maildir format with a pair of Solid State Drives (SSD) in a raid 0 configuration. It's really really fast. Kind of expensive but it's like getting 20x the speed for 20x the price. I think the big gain is in the 0 seek time. You may find ramfs is even faster :) ramfs (tmpfs in linux-land) is useful for indexes. If you lose the indexes, they will created automatically the next time a user logs in.

 We are now trying zlib plugin to lower the number of iops to our 

maildir storage systems. We are using gzip (bzip2 increases a lot the latency). LZMA/xz seems interesting (high compression and rather good decompression speed) and lzo also seems interesting (blazing fast compression AND decompression, not much compression savings though)

 What kind of "tricks" do you use to lower the number of IOPs of 

your dovecot servers?

 Regards

 Javier

I hope you have backups.

Using XFS with delayed logging mount option (requires kernel 2.6.36 or later).

XFS has natively used delayed allocation for quite some time, coalescing multiple pending writes before pushing them into the buffer cache. This not only decreases physical IOPS, but it also decreases filesystem fragmentation by packing more files into each extent. Decreased fragmentation means fewer disk seeks required per file read, which also decreases physical IOPS. This also greatly reduces the wasted space typical of small file storage. Works very well with maildir, but also with the other mail storage formats.

What happens if you pull out the wrong cable in the rack, kernel lockup/oops, power failure, hot swap disk pulled, or something else which causes an unexpected loss of a few seconds of written data?

Surely your IOPs are hard limited by the number of fsyncs (and size of any battery backed ram)?

Ed W

On 17/01/2011 02:20, Stan Hoeppner wrote:

Ed W put forth on 1/16/2011 4:11 PM:

...

...

Using XFS with delayed logging mount option (requires kernel 2.6.36 or later).

XFS has natively used delayed allocation for quite some time, coalescing multiple pending writes before pushing them into the buffer cache. This not only decreases physical IOPS, but it also decreases filesystem fragmentation by packing more files into each extent. Decreased fragmentation means fewer disk seeks required per file read, which also decreases physical IOPS. This also greatly reduces the wasted space typical of small file storage. Works very well with maildir, but also with the other mail storage formats. What happens if you pull out the wrong cable in the rack, kernel lockup/oops, power failure, hot swap disk pulled, or something else which causes an unexpected loss of a few seconds of written data? Read the XFS FAQ. These questions have been answered hundreds of times since XFS was released in Irix in 1994. I'm not your personal XFS tutor.

Why the hostile reply?

The question was deeper than your response?

...

Surely your IOPs are hard limited by the number of fsyncs (and size of any battery backed ram)? Depends on how your applications are written and how often they call fsync. Do you mean BBWC? WRT delayed logging BBWC is mostly irrelevant. Keep in mind that for delayed logging to have a lot of metadata writes in memory someone, or many someones, must be doing something like an 'rm -rf' or equivalent on a large dir with many thousands of files. Even in this case, the processing is _very_ fast.

You have completely missed my point.

Your data isn't safe until it hits the disk. There are plenty of ways to spool data to ram rather than committing it, but they are all vulnerable to data loss until the data is written to disk.

You wrote: "filesystem metadata write operations are pushed almost entirely into RAM", but if the application requests an fsync then you still have to write it to disk? As such you are again limited by disk IO, which itself is limited by the performance of the device (and temporarily accelerated by any persistent write cache). Hence my point that your IOPs are generally limited by the number of fsyncs and any persistent write cache?

As I write this email I'm struggling with getting a server running again that has just been rudely powered down due to a UPS failing (power was fine, UPS failed...). This isn't such a rare event (IMHO) and hence I think we do need to assume that at some point every machine will suffer a rude and unexpected event which looses all in progress write cache. I have no complaints at XFS in general, but I think it's important that filesystem designers in general have give some thought to this event and recovering from it?

Please try not to be so hostile in your email construction - we aren't all idiots here, and even if we were, your writing style is not conducive to us wanting to learn from your apparent wealth of experience?

Regards

Ed W

On 20/01/2011 06:06, Stan Hoeppner wrote:

If you think the above is "hostile" you have lived a privileged and sheltered life, and I envy you. :) That isn't "hostile" but a combination of losing patience and being blunt. "Hostile" is "f--k you!". Obviously I wasn't being "hostile".

I'm living in the "Dovecot mailing list" which has historically been a very tolerant and forgiving place to learn? Do you mind if I continue to remain "sheltered"?

You're overreacting. Saying "I'm not your personal XFS tutor" is not being hostile. Heh, if you think that was hostile, go live on NANAE for a few days or a week and report back on what real hostility is. ;)

I for one don't want the tone of this list to deteriorate to "NANAE" levels....

There are plenty of lists and forums where you can get sarcastic answers from folks with more experience than ones self. Please lets try and keep the tone of this list as the friendly, helpful place it has been?

To offer just an *opinion*, being sarcastic (or just less than fully helpful) to "idiots" who "can't be bothered" to learn the basics before posting is rarely beneficial. Many simply leave and go elsewhere. Some do the spadework and become "experienced", but in turn they usually respond in the same sharp way to new "inexperienced" questions... The circle continues...

I find it helpful to always presume there is a reason I should respect the poster, despite what might look like a lazy question to me. Does someone with 10 years of experience in their of field deserve me to be sharp with them because they tried to skip a step and ask a "lazy question" without doing their own leg work? Only yesterday I was that dimwit having spent 5 hours applying the wrong patch to a kernel and wondering why it failed to build, until I finally asked their list and got a polite reply pointing out my very trivial mistake...

Lets assume everyone deserves some respect and take the time to answer the dim questions politely?

Oh well, pleading over. Good luck and genuinely thanks to Stan for spending his valuable time here. Here's hoping you will continue to do so, but also being nice to the dummies?

Regards

Ed W

On 1/20/11 12:06 AM -0600 Stan Hoeppner wrote:

This is amusing considering XFS is hands down the best filesystem available on any platform, including ZFS. Others are simply ignorant and repeat what they've heard without looking for current information.

Not to be overly brusque, but that's a laugh. The two "best" filesystems out there today are vxfs and zfs, for almost any enterprise workload that exists. I won't argue that xfs won't stand out for specific workloads such as sequential write, it might and I don't know quite enough about it to be sure, but for general workloads including a mail store zfs is leaps ahead. I'd include WAFL in the top 3 but it's only accessible via NFS. Well there is a SAN version but it doesn't really give you access to the best of the filesystem feature set (tradeoff for other features of the hardware).

Your pronouncement that others are simply ignorant is telling.

...

Your data isn't safe until it hits the disk. There are plenty of ways to spool data to ram rather than committing it, but they are all vulnerable to data loss until the data is written to disk.

The delayed logging code isn't a "ram spooler", although that is a mild side effect. Apparently I didn't explain it fully, or precisely. And keep in mind, I'm not the dev who wrote the code. So I'm merely repeating my recollection of the description from the architectural document and what was stated on the XFS list by the author, Dave Chinner of Red Hat. ... In my desire to be brief I didn't fully/correctly explain how delayed logging works. I attempted a simplified explanation that I thought most would understand. Here is the design document: http://oss.sgi.com/archives/xfs/2010-05/msg00329.html

I guess I understand your championing of it if you consider that a design document. That brief piece of email hardly describes it at all, and the performance numbers are pretty worthless (due to the caveat that barriers are disabled).

Given the paragraph in the "design document":

The best IO behaviour comes from the delayed logging version of XFS, with the lowest bandwidth and iops to sustain the highest performance. All the IO is to the log - no metadata is written to disk at all, which is the way this test should execute. As a result, the delayed logging code was the only configuration not limited by the IO subsystem - instead it was completely CPU bound (8 CPUs worth)...

it is indeed a "ram spooler", for metadata, which is a standard (and good) approach. That's not a side effect, that's the design. AFAICT from the brief description anyway.

This is guaranteed to lose data on power loss or drive failure.

Seriously, isn't it time this thread died a peaceful death. It has long since failed to to have any real relevance to Dovecot, except in the most extreme sense. It has evolved into a few testosterone poisoned individuals attempting to make this forum a theater for some mating ritual. If they seriously want to continue this convoluted thread, perhaps they would be so kind as to take it of-list and find a platform better suited for this public display. At the very least, I would hope that Timo might consider closing this thread. I know that Wietse would never have let this thread reach this point on the Postfix forum.

In any case, I am not creating a kill filter to dispense with it.

-- Jerry ✌ Dovecot.user@seibercom.net

Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header.

Jerry put forth on 1/21/2011 7:53 AM:

Seriously, isn't it time this thread died a peaceful death. It has long since failed to to have any real relevance to Dovecot, except in the most extreme sense. It has evolved into a few testosterone poisoned individuals attempting to make this forum a theater for some mating ritual. If they seriously want to continue this convoluted thread, perhaps they would be so kind as to take it of-list and find a platform better suited for this public display. At the very least, I would hope that Timo might consider closing this thread. I know that Wietse would never have let this thread reach this point on the Postfix forum.

In any case, I am not creating a kill filter to dispense with it.

I'm guilty as charged. Consider it dead. Sorry for the noise Jerry, everyone.

-- Stan

Sorry all. I responded before catching up to the end of the thread.

On 1/16/11 2:10 PM -0600 Stan Hoeppner wrote:

Using XFS with delayed logging mount option (requires kernel 2.6.36 or later). ... Using the delayed logging feature, filesystem metadata write operations are pushed almost entirely into RAM. Not only does this _dramatically_ decrease physical metadata write IOPS but it also increases metadata write performance by an order of magnitude. Really shines with maildir,

ext3 has the same "feature". It's fantastic for write performance, and especially for NFS, but for obvious reasons horrible for reliability. I'm sure XFS fixes the consistency problems of ext3, ie on power failure your fs is still consistent, but clearly this strategy is not good for a mail store, ie where you also care about not losing data.

I personally like ZFS with a SLOG for the sync writes. Plus you get the extra guarantees of zfs, ie it guarantees that your disk isn't lying to you.

Warren Baker put forth on 1/18/2011 2:53 AM:

On Monday, January 17, 2011, Stan Hoeppner stan@hardwarefreak.com wrote:

...

Cor Bosman put forth on 1/16/2011 5:34 PM:

...

Btw, our average mailsize last we checked was 30KB. Thats a pretty good average as we're an ISP with a very wide user base. I think 4KB average is not a normal mail load.

As another OP pointed out, some ISPs apparently have to deliver a lot of spam to mailboxen to avoid FPs, bumping up that average mail size considerably. Do you accept and deliver a lot of spam to user mailboxen?

At an ISP I worked at, we did a study (just over 2 years ago) on the average size of spam mail that was been delivered to the users. It worked out to an average size of between 8KB and 10KB. This was based on data over a period of 12months with an average of a 180 million mails per month been received. Legit mail was averaging at 32KB.

I doubt whether the sizes of spam has changed much.

What was the ratio of spam to ham you were delivering to user mailboxes?

-- Stan