[Dovecot] Persistence of UIDs
Hi everyone,
I have a question concerning UIDs. How persistant are they?
I am thinking about building some form of webmail specialized for some specific business purpose and I am thinking of building a sort of cache in a DB by storing the email addr, date, subject and UID for quick lookups and search of correspondance.
I am doing this because I am having issue with multiple people searching thru email folders that have 100k+ emails (which is another problem in itself, searches don't seem to scale well when folder goes above 60k emails).
So to come back to my question, can I store the UIDs and reuse those UIDs later on to obtain the body of the email??? Or can the UIDs change on the server and they will not be valid anymore?.
My setup is:
- dovecot 1.x (will migrate to 2.x soon)
- maildir
- everything stored on an intel 320 SSD (index and maildir folder)
Thanks,
-JD
Quoting Jean-Daniel Beaubien jd.beaubien@gmail.com:
I have a question concerning UIDs. How persistant are they?
[snip]
So to come back to my question, can I store the UIDs and reuse those UIDs later on to obtain the body of the email??? Or can the UIDs change on the server and they will not be valid anymore?.
You really need to read RFC 3501 (http://tools.ietf.org/html/rfc3501),
specifically section 2.3.1.1.
Short answer: UIDs will almost always be persistent, but you always
need to check UIDVALIDITY in the tiny chance that they may be
invalidated.
michael
On 25.1.2012, at 5.22, Jean-Daniel Beaubien wrote:
I have a question concerning UIDs. How persistant are they?
With Dovecot persistent enough. But as Michael said, check UIDVALIDITY.
I am thinking about building some form of webmail specialized for some specific business purpose and I am thinking of building a sort of cache in a DB by storing the email addr, date, subject and UID for quick lookups and search of correspondance.
Dovecot should already have such cache. If there are problems with that, I think it would be better to fix it on Dovecot's side rather than adding a second cache.
I am doing this because I am having issue with multiple people searching thru email folders that have 100k+ emails (which is another problem in itself, searches don't seem to scale well when folder goes above 60k emails).
Maybe enable fts-solr or fts-lucene? (Both work much better in v2.1.)
On Wed, Jan 25, 2012 at 7:45 AM, Timo Sirainen tss@iki.fi wrote:
On 25.1.2012, at 5.22, Jean-Daniel Beaubien wrote:
I have a question concerning UIDs. How persistant are they?
With Dovecot persistent enough. But as Michael said, check UIDVALIDITY.
I am thinking about building some form of webmail specialized for some specific business purpose and I am thinking of building a sort of cache in a DB by storing the email addr, date, subject and UID for quick lookups and search of correspondance.
Dovecot should already have such cache. If there are problems with that, I think it would be better to fix it on Dovecot's side rather than adding a second cache.
Very true. Has there been many search/index improvements since 1.0.9? I read thru the release notes but nothing jumped out at me.
I am doing this because I am having issue with multiple people searching thru email folders that have 100k+ emails (which is another problem in itself, searches don't seem to scale well when folder goes above 60k emails).
Maybe enable fts-solr or fts-lucene? (Both work much better in v2.1.)
I was under the impression that lucene was for full-text search. I'm just doing simple from/to field searches.
I will get a few numbers together about folder_size --> search time and I will post them tonight.
-jd
On 2012-01-25 8:34 AM, Jean-Daniel Beaubien jd.beaubien@gmail.com wrote:
On Wed, Jan 25, 2012 at 7:45 AM, Timo Sirainentss@iki.fi wrote:
Dovecot should already have such cache. If there are problems with that, I think it would be better to fix it on Dovecot's side rather than adding a second cache.
Very true. Has there been many search/index improvements since 1.0.9? I read thru the release notes but nothing jumped out at me.
Seriously??
1.0.9 is *very* old, and even no longer really supported.
Upgrade. Really. It isn't that hard. There is zero reason to stay on an unsupported version.
I am doing this because I am having issue with multiple people searching thru email folders that have 100k+ emails (which is another problem in itself, searches don't seem to scale well when folder goes above 60k emails).
Maybe enable fts-solr or fts-lucene? (Both work much better in v2.1.)
I was under the impression that lucene was for full-text search. I'm just doing simple from/to field searches.
I will get a few numbers together about folder_size --> search time and I will post them tonight.
Don't waste your time testing such an old and unsupported version, I'm sure Timo has no interest in any such numbers - *unless* you are planning on doing said tests on *both* the 1.0.9 version *and* the latest 2.0.x or 2.1 build and provide a *comparison* - *that* may be interesting...
--
Best regards,
Charles
On 25.1.2012, at 15.34, Jean-Daniel Beaubien wrote:
I am thinking about building some form of webmail specialized for some specific business purpose and I am thinking of building a sort of cache in a DB by storing the email addr, date, subject and UID for quick lookups and search of correspondance.
Dovecot should already have such cache. If there are problems with that, I think it would be better to fix it on Dovecot's side rather than adding a second cache.
Very true. Has there been many search/index improvements since 1.0.9? I read thru the release notes but nothing jumped out at me.
Disk I/O usage is the same probably, CPU usage is less in newer versions.
I am doing this because I am having issue with multiple people searching thru email folders that have 100k+ emails (which is another problem in itself, searches don't seem to scale well when folder goes above 60k emails).
Maybe enable fts-solr or fts-lucene? (Both work much better in v2.1.)
I was under the impression that lucene was for full-text search. I'm just doing simple from/to field searches.
In v2.1 from/to fields are also searched via FTS.
On Wed, Jan 25, 2012 at 8:47 AM, Timo Sirainen tss@iki.fi wrote:
On 25.1.2012, at 15.34, Jean-Daniel Beaubien wrote:
I am thinking about building some form of webmail specialized for some specific business purpose and I am thinking of building a sort of cache in a DB by storing the email addr, date, subject and UID for quick lookups and search of correspondance.
Dovecot should already have such cache. If there are problems with that, I think it would be better to fix it on Dovecot's side rather than adding a second cache.
Very true. Has there been many search/index improvements since 1.0.9? I read thru the release notes but nothing jumped out at me.
Disk I/O usage is the same probably, CPU usage is less in newer versions.
I am doing this because I am having issue with multiple people searching thru email folders that have 100k+ emails (which is another problem in itself, searches don't seem to scale well when folder goes above 60k emails).
Maybe enable fts-solr or fts-lucene? (Both work much better in v2.1.)
I was under the impression that lucene was for full-text search. I'm just doing simple from/to field searches.
In v2.1 from/to fields are also searched via FTS.
Ok, I managed to compile 2.1 rc5 on an old ubuntu 8.04 without any issue. However, the config file is giving me a bit of a hard time, I'll figure this part out tomorrow.
I'd just like to confirm that there is no risk to the actual mail data is ever something is badly configured when I start dovecot 2.1. I am managing this old server on my spare time for a friend, so I don't want to loose 2million+ emails and have to deal with those consequences :)
On 2012-01-25 8:40 PM, Jean-Daniel Beaubien jd.beaubien@gmail.com wrote:
I'd just like to confirm that there is no risk to the actual mail data is ever something is badly configured when I start dovecot 2.1. I am managing this old server on my spare time for a friend, so I don't want to loose 2million+ emails and have to deal with those consequences:)
There are *always* risks associated with things like this... maybe the chance is low, but no guarantees...
As always, it is *your* responsibility to *backup* *first*...
--
Best regards,
Charles
On 26.1.2012, at 14.03, Charles Marcus wrote:
On 2012-01-25 8:40 PM, Jean-Daniel Beaubien jd.beaubien@gmail.com wrote:
I'd just like to confirm that there is no risk to the actual mail data is ever something is badly configured when I start dovecot 2.1. I am managing this old server on my spare time for a friend, so I don't want to loose 2million+ emails and have to deal with those consequences:)
There are *always* risks associated with things like this... maybe the chance is low, but no guarantees...
Risks of some trouble, yes .. but you have to be highly creative if you want to accidentally lose any mails. I can't think of any way to do that without explicitly deleting files from filesystem or via IMAP/POP3 client.
participants (4)
-
Charles Marcus
-
Jean-Daniel Beaubien
-
Michael M Slusarz
-
Timo Sirainen