Hi,
I had enabled an option in dovecot. mail_attachment_dir = /var/mail/virtual/copymail/attachments
After a while I checked /var/mail/virtual and did some cleanup. I did not remember that copymail was specified in dovecot and erased it.
Oct 30 10:56:05 mx0 dovecot: imap(hidden): Error: file_istream.stat(/var/mail/virtual/copymail/attachments/6a/50/6a506530265ef7c9feb396410eaf6946036e9a79-b034401e794009503a0400002cb72ff6) failed: No such file or directory Oct 30 10:56:05 mx0 dovecot: imap(hidden): Error: istream-concat: Failed to get size of stream /var/mail/virtual/copymail/attachments/6a/50/6a506530265ef7c9feb396410eaf6946036e9a79-b034401e794009503a0400002cb72ff6 Oct 30 10:56:05 mx0 dovecot: imap(hidden): Error: read() failed: Invalid argument (FETCH for mailbox INBOX UID 196) Oct 30 10:56:05 mx0 dovecot: imap(hidden): Disconnected: Internal error occurred. Refer to server log for more information. [2012-10-30 10:56:05] in=150 out=950
I have Bacula and have restored most of the stuff, but obviously not all files. That is not too important. But I do not know, how to tell dovecot that it may "forget" about files that produce a "No such file or directory" error.
Can I do some "rescan/rebuild" in dovecot?
Kind regards
-Christian Rößner
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
On 30.10.2012, at 12.11, Christian Rößner wrote:
Oct 30 10:56:05 mx0 dovecot: imap(hidden): Error: file_istream.stat(/var/mail/virtual/copymail/attachments/6a/50/6a506530265ef7c9feb396410eaf6946036e9a79-b034401e794009503a0400002cb72ff6) failed: No such file or directory
I have Bacula and have restored most of the stuff, but obviously not all files. That is not too important. But I do not know, how to tell dovecot that it may "forget" about files that produce a "No such file or directory" error.
Can I do some "rescan/rebuild" in dovecot?
Currently you can't in any easy way. The easiest fix for now I think would be to write a script that reads through dbox files, parses the attachment metadata lines and recreates the missing files with the original size (e.g. sparse-0-filled). The dbox parsing can be done easily with:
doveadm dump m.1 | grep ^msg.ext-ref
The format is:
1*(<start offset> <byte count> <options> <ref>)
If the options="-" then the byte count is the final size. If options="B" then byte count is the base64-encoded size while the original file has to be base64-decoded size.
The format is:
1*(<start offset> <byte count> <options> <ref>)
If the options="-" then the byte count is the final size. If options="B" then byte count is the base64-encoded size while the original file has to be base64-decoded size.
Ok, so far I have "grep'ed" this here:
msg.ext-ref = 83713 1282212 B76 6a/50/6a506530265ef7c9feb396410eaf6946036e9a79-b034401e794009503a0400002cb72ff6 1443213 550635 B76 56/f2/56f25e225385902f3fc5185dc3d0103f59b34d14-b134401e794009503a0400002cb72ff6 1994019 477177 B76 c4/36/c436874b56cf3cd105e82f9243c7eac53c467f32-b234401e794009503a0400002cb72ff6 2561522 1075531 B76 77/af/77af1045a783308dbbf2f8a464c5136a0407e720-b334401e794009503a0400002cb72ff6 3715582 1195635 B76 99/33/99339b17a21ce052cd8f47f1d88c6e869cc1650b-b434401e794009503a0400002cb72ff6 4966686 715386 B76 fe/df/fedf23091720d3fa649af3bd6537e66304b8061a-b534401e794009503a0400002cb72ff6 5805913 788086 B76 ab/36/ab36f53a443f1855bc13caaba9e01e9464b2921f-b634401e794009503a0400002cb72ff6 6684258 906273 B76 10/70/1070d21039bc3f305bb948315a01344eefb2a465-b734401e794009503a0400002cb72ff6 7590707 204613 B76 39/44/394402c057791482f79351363f025ae0a7caf1b0-b834401e794009503a0400002cb72ff6 7795492 1349911 B76 41/bd/41bd01b4880065e5136cafbd1d191a1f8a1ead55-b934401e794009503a0400002cb72ff6 9271435 1504539 B76 c6/71/c671c1367e843741a2cc8f083a37231522d37640-ba34401e794009503a0400002cb72ff6 10877759 357555 B76 58/f5/58f582d2644025b843cf991f5cf783d27f9d90c9-bb34401e794009503a0400002cb72ff6 11826037 890683 B76 82/da/82dabbe06f269e7c79417db3b570246a648d2139-bc34401e794009503a0400002cb72ff6
msg.ext-ref = 118947 317624 B76 ad/9b/ad9be52e11433cd0337cda13bf0a458fd0fd948d-df905c0cd33d0950ae7800002cb72ff6 436770 139669 B76 78/15/781526d896a0530a5e76ebce65f2eb690d102dd3-e0905c0cd33d0950ae7800002cb72ff6 576610 457829 B76 61/3a/613a70c8515c572a04211fb0c63828d9c9acfb70-e1905c0cd33d0950ae7800002cb72ff6 1107667 410786 B76 7f/6b/7f6b7ee9b08a73600d98e8583aae343a90e76b96-e2905c0cd33d0950ae7800002cb72ff6 1611186 816686 B76 ff/ff/ffff9362c5356d8bedb17bd56edf0524bd0ae7b3-e3905c0cd33d0950ae7800002cb72ff6 2516232 643918 B76 4f/aa/4faa153fada5ceea79016cf2eadc1d05110f3f2e-e4905c0cd33d0950ae7800002cb72ff6 3291363 1036359 B76 e6/f3/e6f342bf28e8edfd3214666aaa52f0c067bae22b-e5905c0cd33d0950ae7800002cb72ff6 4418344 668813 B76 20/78/2078c98fb9bcadeeaa49bc38dc31548142fc71b1-e6905c0cd33d0950ae7800002cb72ff6 5154786 502218 B76 40/f4/40f4af3ad2077493caa34faabb201531609b50c4-e7905c0cd33d0950ae7800002cb72ff6 5782912 628591 B76 cc/a9/cca98a2a325f1be9a398d62890836cf11f267c4b-e8905c0cd33d0950ae7800002cb72ff6 6518382 526201 B76 17/47/1747a90b58c50c3d01da7f3a6601f7073cd5b163-e9905c0cd33d0950ae7800002cb72ff6 7140759 517776 B76 04/af/04afe7deb8e6ee99153433d2845da417e54cd042-ea905c0cd33d0950ae7800002cb72ff6 7769983 2317979 B76 05/13/0513bcfceff303125f233ad2c01c5ba2ed96c6a2-eb905c0cd33d0950ae7800002cb72ff6 10214312 3097649 B76 35/e4/35e46902b3e6473b9689a92acd71e58fb7165a8f-ec905c0cd33d0950ae7800002cb72ff6
msg.ext-ref = 75027 1291257 B76 b9/dc/b9dcd6899ae65e5c11b122d7bfc3be9fefc21024-5df010068b3f0950c27d00002cb72ff6 1441078 1131344 B76 f6/e6/f6e63f000d6501be472629747448057b122104c1-5ef010068b3f0950c27d00002cb72ff6 2572595 2218094 B76 93/96/9396c5eaeac2615119e55c67fa8f010332ba0fd3-5ff010068b3f0950c27d00002cb72ff6 4790862 2211695 B76 cc/a5/cca5607fb739306f3628a19575dc41432f74a22d-60f010068b3f0950c27d00002cb72ff6 7002730 2614603 B76 66/10/661002c8039997174e34b9ef31d0e693a556eebe-61f010068b3f0950c27d00002cb72ff6 9617506 2760312 B76 8c/65/8c656fe835af26c175337cd318daca8ae8e00369-62f010068b3f0950c27d00002cb72ff6 12377991 2341764 B76 19/c8/19c83e0bf1284e74e49feecaf95506266201551d-63f010068b3f0950c27d00002cb72ff6 15209343 406758 B76 b6/62/b66216837cc48422e22e7a9a22631f840a49ef78-64f010068b3f0950c27d00002cb72ff6 15616301 136877 B76 06/9f/069f5ab86dc9e8e9972f3f5c0dda03c1f3103730-65f010068b3f0950c27d00002cb72ff6 15753350 971075 B76 a7/7c/a77c36690ff0f0f774b82efaf15f93535ba027e9-66f010068b3f0950c27d00002cb72ff6 16849194 1197333 B76 4f/28/4f2881be6d0e8a7f53c0e226c0dbb148b05674c7-67f010068b3f0950c27d00002cb72ff6 18168424 850768 B76 92/72/9272e1ea7ceb79df6222686bf157f957fa9851c1-68f010068b3f0950c27d00002cb72ff6 19019393 135641 B76 60/fd/60fdcd7851c8f0a21f342aaafce9e49a3e00e1aa-69f010068b3f0950c27d00002cb72ff6 19155207 897179 B76 63/59/6359abf4f9e806e3990e0d6590e519924c838fa5-6af010068b3f0950c27d00002cb72ff6 20169966 1022612 B76 f8/65/f8654367f5df050d23565644e83c8c50abb69c39-6bf010068b3f0950c27d00002cb72ff6
But I did not understand the base64 explanation. Sorry :) For me it seems all "options" are B-prefixed. So they are all base64? But which value is now the size and how do I create the missing files now? Using dd? Can you give me an example from the output above? That would help me.
Thanks a lot
Kind regards
-Christian Rößner
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
On 30.10.2012, at 15.28, Christian Rößner wrote:
msg.ext-ref = 83713 1282212 B76 6a/50/6a506530265ef7c9feb396410eaf6946036e9a79-b034401e794009503a0400002cb72ff6
But I did not understand the base64 explanation. Sorry :) For me it seems all "options" are B-prefixed. So they are all base64? But which value is now the size and how do I create the missing files now? Using dd? Can you give me an example from the output above? That would help me.
They are all base64 yes, the B76 means that all the encoded lines will be 76 chars long. So the file size above needs to be 1282212, divided by 77 (76+LF) = 16652 full lines and 8 bytes over. Base64 encodes 3 byte blocks into 4 byte chars, so the original data has (16652*76+8)/4*3 = 949170 bytes (or 1-2 bytes less, but that makes no difference because it's padded anyway).
So if you create /attachments/6a/50/6a506530265ef7c9feb396410eaf6946036e9a79-b034401e794009503a0400002cb72ff6 that is 949170 bytes long, and do the same for the rest of the attachments, you should be able to read this mail without errors.
You can easily create the files without wasting space with: dd if=/dev/zero of=foo bs=1 seek=949169 count=1
Hi,
msg.ext-ref = 83713 1282212 B76 6a/50/6a506530265ef7c9feb396410eaf6946036e9a79-b034401e794009503a0400002cb72ff6
But I did not understand the base64 explanation. Sorry :) For me it seems all "options" are B-prefixed. So they are all base64? But which value is now the size and how do I create the missing files now? Using dd? Can you give me an example from the output above? That would help me.
They are all base64 yes, the B76 means that all the encoded lines will be 76 chars long. So the file size above needs to be 1282212, divided by 77 (76+LF) = 16652 full lines and 8 bytes over. Base64 encodes 3 byte blocks into 4 byte chars, so the original data has (16652*76+8)/4*3 = 949170 bytes (or 1-2 bytes less, but that makes no difference because it's padded anyway).
So if you create /attachments/6a/50/6a506530265ef7c9feb396410eaf6946036e9a79-b034401e794009503a0400002cb72ff6 that is 949170 bytes long, and do the same for the rest of the attachments, you should be able to read this mail without errors.
You can easily create the files without wasting space with: dd if=/dev/zero of=foo bs=1 seek=949169 count=1
Thanks. I have calculated both other files and recreated zero padded files. Now I am going to watch the log file and see, if errors are gone.
One last question: If the user now opens a mail, where the attachments are broken and he/she removes the mail, are the created hand-made files be removed automatically?
Thanks in advance
Kind regards
-Christian Rößner
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
On 30.10.2012, at 16.44, Christian Rößner wrote:
So if you create /attachments/6a/50/6a506530265ef7c9feb396410eaf6946036e9a79-b034401e794009503a0400002cb72ff6 that is 949170 bytes long, and do the same for the rest of the attachments, you should be able to read this mail without errors.
You can easily create the files without wasting space with: dd if=/dev/zero of=foo bs=1 seek=949169 count=1
Thanks. I have calculated both other files and recreated zero padded files. Now I am going to watch the log file and see, if errors are gone.
One last question: If the user now opens a mail, where the attachments are broken and he/she removes the mail, are the created hand-made files be removed automatically?
Yes.
participants (2)
-
Christian Rößner
-
Timo Sirainen