From dovecot at dovecot.org Tue Jul 1 13:06:03 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 01 Jul 2014 13:06:03 +0000 Subject: dovecot-2.2: doveadm fs delete: Added -n to usage text. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/2be0234cef8e changeset: 17547:2be0234cef8e user: Timo Sirainen date: Tue Jul 01 16:04:20 2014 +0300 description: doveadm fs delete: Added -n to usage text. diffstat: src/doveadm/doveadm-fs.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 40e128ff3537 -r 2be0234cef8e src/doveadm/doveadm-fs.c --- a/src/doveadm/doveadm-fs.c Tue Jul 01 00:10:21 2014 +0300 +++ b/src/doveadm/doveadm-fs.c Tue Jul 01 16:04:20 2014 +0300 @@ -387,7 +387,7 @@ { cmd_fs_copy, "fs copy", " " }, { cmd_fs_stat, "fs stat", " " }, { cmd_fs_metadata, "fs metadata", " " }, - { cmd_fs_delete, "fs delete", "[-R] " }, + { cmd_fs_delete, "fs delete", "[-R] [-n ] " }, { cmd_fs_iter, "fs iter", " " }, { cmd_fs_iter_dirs, "fs iter-dirs", " " }, }; From dovecot at dovecot.org Tue Jul 1 21:11:59 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 01 Jul 2014 21:11:59 +0000 Subject: dovecot-2.2: lib-http: Fixed bug in parsing of quoted string wit... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/3e0e4dd7fb65 changeset: 17548:3e0e4dd7fb65 user: Stephan Bosch date: Tue Jul 01 23:24:08 2014 +0300 description: lib-http: Fixed bug in parsing of quoted string with escape sequences. diffstat: src/lib-http/http-parser.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 2be0234cef8e -r 3e0e4dd7fb65 src/lib-http/http-parser.c --- a/src/lib-http/http-parser.c Tue Jul 01 16:04:20 2014 +0300 +++ b/src/lib-http/http-parser.c Tue Jul 01 23:24:08 2014 +0300 @@ -175,6 +175,7 @@ if (parser->cur >= parser->end || !http_char_is_text(*parser->cur)) return -1; str_append_c(str, *parser->cur); + parser->cur++; /* ERROR */ } else { From dovecot at dovecot.org Tue Jul 1 21:12:00 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 01 Jul 2014 21:12:00 +0000 Subject: dovecot-2.2: lib-http: Updated comments to RFC7230/RFC7231. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/9511372c7c18 changeset: 17549:9511372c7c18 user: Stephan Bosch date: Wed Jul 02 00:10:16 2014 +0300 description: lib-http: Updated comments to RFC7230/RFC7231. diffstat: src/lib-http/http-client-connection.c | 26 ++--- src/lib-http/http-client-request.c | 23 +++-- src/lib-http/http-client.c | 2 +- src/lib-http/http-date.c | 128 +++++++++++++++----------------- src/lib-http/http-header-parser.c | 16 ++- src/lib-http/http-message-parser.c | 132 ++++++++++++++++++++------------- src/lib-http/http-parser.c | 3 +- src/lib-http/http-parser.h | 3 +- src/lib-http/http-request-parser.c | 40 +++------ src/lib-http/http-response-parser.c | 47 ++++++----- src/lib-http/http-transfer-chunked.c | 22 ++--- src/lib-http/http-url.c | 30 +++---- 12 files changed, 237 insertions(+), 235 deletions(-) diffs (truncated from 908 to 300 lines): diff -r 3e0e4dd7fb65 -r 9511372c7c18 src/lib-http/http-client-connection.c --- a/src/lib-http/http-client-connection.c Tue Jul 01 23:24:08 2014 +0300 +++ b/src/lib-http/http-client-connection.c Wed Jul 02 00:10:16 2014 +0300 @@ -308,16 +308,14 @@ if (req->connect_tunnel) conn->tunneling = TRUE; - /* https://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-21; - Section 6.1.2.1: + /* RFC 7231, Section 5.1.1: Expect - Because of the presence of older implementations, the protocol allows - ambiguous situations in which a client might send "Expect: 100-continue" - without receiving either a 417 (Expectation Failed) or a 100 (Continue) - status code. Therefore, when a client sends this header field to an - origin server (possibly via a proxy) from which it has never seen a 100 - (Continue) status code, the client SHOULD NOT wait for an indefinite - period before sending the payload body. + o A client that sends a 100-continue expectation is not required to + wait for any specific length of time; such a client MAY proceed to + send the message body even if it has not yet received a response. + Furthermore, since 100 (Continue) responses cannot be sent through + an HTTP/1.0 intermediary, such a client SHOULD NOT wait for an + indefinite period before sending the message body. */ if (req->payload_sync && !conn->peer->seen_100_response) { i_assert(req->payload_chunked || req->payload_size > 0); @@ -551,13 +549,11 @@ if (conn->to_response != NULL) timeout_remove(&conn->to_response); - /* https://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-21 - Section 7.2: + /* RFC 7231, Section 6.2: - A client MUST be prepared to accept one or more 1xx status responses - prior to a regular response, even if the client does not expect a 100 - (Continue) status message. Unexpected 1xx status responses MAY be - ignored by a user agent. + A client MUST be able to parse one or more 1xx responses received + prior to a final response, even if the client does not expect one. A + user agent MAY ignore unexpected 1xx responses. */ if (req->payload_sync && response.status == 100) { if (conn->payload_continue) { diff -r 3e0e4dd7fb65 -r 9511372c7c18 src/lib-http/http-client-request.c --- a/src/lib-http/http-client-request.c Tue Jul 01 23:24:08 2014 +0300 +++ b/src/lib-http/http-client-request.c Wed Jul 02 00:10:16 2014 +0300 @@ -350,16 +350,18 @@ enum http_response_payload_type http_client_request_get_payload_type(struct http_client_request *req) { - /* https://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-23 - Section 3.3: + /* RFC 7230, Section 3.3: - The presence of a message body in a response depends on both the - request method to which it is responding and the response status code. - Responses to the HEAD request method never include a message body - because the associated response header fields, if present, indicate only - what their values would have been if the request method had been GET - 2xx (Successful) responses to CONNECT switch to tunnel mode instead of - having a message body (Section 4.3.6 of [Part2]). + The presence of a message body in a response depends on both the + request method to which it is responding and the response status code + (Section 3.1.2 of [RFC7230]). Responses to the HEAD request method + (Section 4.3.2 of [RFC7231]) never include a message body because the + associated response header fields (e.g., Transfer-Encoding, + Content-Length, etc.), if present, indicate only what their values + would have been if the request method had been GET (Section 4.3.1 of + [RFC7231]). 2xx (Successful) responses to a CONNECT request method + (Section 4.3.6 of [RFC7231]) switch to tunnel mode instead of having a + message body. */ if (strcmp(req->method, "HEAD") == 0) return HTTP_RESPONSE_PAYLOAD_TYPE_NOT_PRESENT; @@ -933,8 +935,7 @@ req->label = p_strdup_printf(req->pool, "[%s %s%s]", req->method, origin_url, req->target); - /* https://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-21 - Section-7.4.4 + /* RFC 7231, Section 6.4.4: -> A 303 `See Other' redirect status response is handled a bit differently. Basically, the response content is located elsewhere, but the original diff -r 3e0e4dd7fb65 -r 9511372c7c18 src/lib-http/http-client.c --- a/src/lib-http/http-client.c Tue Jul 01 23:24:08 2014 +0300 +++ b/src/lib-http/http-client.c Wed Jul 02 00:10:16 2014 +0300 @@ -54,7 +54,7 @@ http-client-connection: This is an actual connection to a server. Once a connection is ready to - handle requests, it claims a request from a host object. One connection can + handle requests, it claims a request from a queue object. One connection can service multiple hosts and one host can have multiple associated connections, possibly to different ips and ports. diff -r 3e0e4dd7fb65 -r 9511372c7c18 src/lib-http/http-date.c --- a/src/lib-http/http-date.c Tue Jul 01 23:24:08 2014 +0300 +++ b/src/lib-http/http-date.c Wed Jul 02 00:10:16 2014 +0300 @@ -7,75 +7,65 @@ #include -/* - Official specification is still RFC261, Section 3.3, but we anticipate - HTTPbis and use the draft Part 2, Section 5.1 as reference for our - parser: - - http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-20#section-5.1 +/* RFC 7231, Section 7.1.1.1: Date/Time Formats The defined syntax is as follows: - HTTP-date = rfc1123-date / obs-date + HTTP-date = IMF-fixdate / obs-date Preferred format: - rfc1123-date = day-name "," SP date1 SP time-of-day SP GMT - ; fixed length subset of the format defined in - ; Section 5.2.14 of [RFC1123] - day-name = %x4D.6F.6E ; "Mon", case-sensitive - / %x54.75.65 ; "Tue", case-sensitive - / %x57.65.64 ; "Wed", case-sensitive - / %x54.68.75 ; "Thu", case-sensitive - / %x46.72.69 ; "Fri", case-sensitive - / %x53.61.74 ; "Sat", case-sensitive - / %x53.75.6E ; "Sun", case-sensitive - date1 = day SP month SP year - ; e.g., 02 Jun 1982 - day = 2DIGIT - month = %x4A.61.6E ; "Jan", case-sensitive + IMF-fixdate = day-name "," SP date1 SP time-of-day SP GMT + ; fixed length/zone/capitalization subset of the format + ; see Section 3.3 of [RFC5322] + day-name = %x4D.6F.6E ; "Mon", case-sensitive + / %x54.75.65 ; "Tue", case-sensitive + / %x57.65.64 ; "Wed", case-sensitive + / %x54.68.75 ; "Thu", case-sensitive + / %x46.72.69 ; "Fri", case-sensitive + / %x53.61.74 ; "Sat", case-sensitive + / %x53.75.6E ; "Sun", case-sensitive + date1 = day SP month SP year + ; e.g., 02 Jun 1982 + day = 2DIGIT + month = %x4A.61.6E ; "Jan", case-sensitive / %x46.65.62 ; "Feb", case-sensitive - / %x4D.61.72 ; "Mar", case-sensitive - / %x41.70.72 ; "Apr", case-sensitive - / %x4D.61.79 ; "May", case-sensitive - / %x4A.75.6E ; "Jun", case-sensitive - / %x4A.75.6C ; "Jul", case-sensitive - / %x41.75.67 ; "Aug", case-sensitive - / %x53.65.70 ; "Sep", case-sensitive - / %x4F.63.74 ; "Oct", case-sensitive - / %x4E.6F.76 ; "Nov", case-sensitive - / %x44.65.63 ; "Dec", case-sensitive - year = 4DIGIT - GMT = %x47.4D.54 ; "GMT", case-sensitive - time-of-day = hour ":" minute ":" second - ; 00:00:00 - 23:59:59 - hour = 2DIGIT - minute = 2DIGIT - second = 2DIGIT - - The semantics of day-name, day, month, year, and time-of-day are the - same as those defined for the RFC 5322 constructs with the - corresponding name ([RFC5322], Section 3.3). + / %x4D.61.72 ; "Mar", case-sensitive + / %x41.70.72 ; "Apr", case-sensitive + / %x4D.61.79 ; "May", case-sensitive + / %x4A.75.6E ; "Jun", case-sensitive + / %x4A.75.6C ; "Jul", case-sensitive + / %x41.75.67 ; "Aug", case-sensitive + / %x53.65.70 ; "Sep", case-sensitive + / %x4F.63.74 ; "Oct", case-sensitive + / %x4E.6F.76 ; "Nov", case-sensitive + / %x44.65.63 ; "Dec", case-sensitive + year = 4DIGIT + GMT = %x47.4D.54 ; "GMT", case-sensitive + time-of-day = hour ":" minute ":" second + ; 00:00:00 - 23:59:60 (leap second) + hour = 2DIGIT + minute = 2DIGIT + second = 2DIGIT Obsolete formats: - obs-date = rfc850-date / asctime-date + obs-date = rfc850-date / asctime-date - rfc850-date = day-name-l "," SP date2 SP time-of-day SP GMT - date2 = day "-" month "-" 2DIGIT - ; day-month-year (e.g., 02-Jun-82) - day-name-l = %x4D.6F.6E.64.61.79 ; "Monday", case-sensitive - / %x54.75.65.73.64.61.79 ; "Tuesday", case-sensitive - / %x57.65.64.6E.65.73.64.61.79 ; "Wednesday", case-sensitive - / %x54.68.75.72.73.64.61.79 ; "Thursday", case-sensitive - / %x46.72.69.64.61.79 ; "Friday", case-sensitive - / %x53.61.74.75.72.64.61.79 ; "Saturday", case-sensitive - / %x53.75.6E.64.61.79 ; "Sunday", case-sensitive + rfc850-date = day-name-l "," SP date2 SP time-of-day SP GMT + date2 = day "-" month "-" 2DIGIT + ; e.g., 02-Jun-82 + day-name-l = %x4D.6F.6E.64.61.79 ; "Monday", case-sensitive + / %x54.75.65.73.64.61.79 ; "Tuesday", case-sensitive + / %x57.65.64.6E.65.73.64.61.79 ; "Wednesday", case-sensitive + / %x54.68.75.72.73.64.61.79 ; "Thursday", case-sensitive + / %x46.72.69.64.61.79 ; "Friday", case-sensitive + / %x53.61.74.75.72.64.61.79 ; "Saturday", case-sensitive + / %x53.75.6E.64.61.79 ; "Sunday", case-sensitive - asctime-date = day-name SP date3 SP time-of-day SP year - date3 = month SP ( 2DIGIT / ( SP 1DIGIT )) - ; month day (e.g., Jun 2) - + asctime-date = day-name SP date3 SP time-of-day SP year + date3 = month SP ( 2DIGIT / ( SP 1DIGIT )) + ; e.g., Jun 2 */ static const char *month_names[] = { @@ -257,14 +247,14 @@ } static int -http_date_parse_format_rfc1123(struct http_date_parser *parser) +http_date_parse_format_imf_fixdate(struct http_date_parser *parser) { /* - rfc1123-date = day-name "," SP date1 SP time-of-day SP GMT - ; fixed length subset of the format defined in - ; Section 5.2.14 of [RFC1123] + IMF-fixdate = day-name "," SP date1 SP time-of-day SP GMT + ; fixed length/zone/capitalization subset of the format + ; see Section 3.3 of [RFC5322] date1 = day SP month SP year - ; e.g., 02 Jun 1982 + ; e.g., 02 Jun 1982 Remaining: {...} SP day SP month SP year SP time-of-day SP GMT @@ -390,10 +380,10 @@ int i; /* - HTTP-date = rfc1123-date / obs-date - rfc1123-date = day-name "," SP date1 SP time-of-day SP GMT - ; fixed length subset of the format defined in - ; Section 5.2.14 of [RFC1123] + HTTP-date = IMF-fixdate / obs-date + IMF-fixdate = day-name "," SP date1 SP time-of-day SP GMT + ; fixed length/zone/capitalization subset of the format + ; see Section 3.3 of [RFC5322] obs-date = rfc850-date / asctime-date rfc850-date = day-name-l "," SP date2 SP time-of-day SP GMT asctime-date = day-name SP date3 SP time-of-day SP year @@ -414,7 +404,7 @@ return http_date_parse_format_rfc850(parser); } - /* rfc1123-date / asctime-date */ + /* IMF-fixdate / asctime-date */ for (i = 0; i < 7; i++) { if (strcmp(weekday_names[i], str_c(dayname)) == 0) { break; @@ -433,9 +423,9 @@ if (parser->cur[0] != ',') return -1; - /* rfc1123-date */ + /* IMF-fixdate */ parser->cur++; - return http_date_parse_format_rfc1123(parser); + return http_date_parse_format_imf_fixdate(parser); } bool http_date_parse(const unsigned char *data, size_t size, diff -r 3e0e4dd7fb65 -r 9511372c7c18 src/lib-http/http-header-parser.c --- a/src/lib-http/http-header-parser.c Tue Jul 01 23:24:08 2014 +0300 +++ b/src/lib-http/http-header-parser.c Wed Jul 02 00:10:16 2014 +0300 @@ -120,7 +120,8 @@ { const unsigned char *first; - /* field-content = *( HTAB / SP / VCHAR / obs-text ) + /* field-content = field-vchar [ 1*( SP / HTAB ) field-vchar ] + field-vchar = VCHAR / obs-text */ do { first = parser->cur; @@ -156,14 +157,19 @@ { int ret; - /* 'header' = *( header-field CRLF ) CRLF + /* RFC 7230, Section 3.2: Header Fields + From dovecot at dovecot.org Wed Jul 2 15:23:19 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:19 +0000 Subject: dovecot-2.2: lib: test-var-expand - use test_assert_idx() inside... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/1278fed4f2c9 changeset: 17550:1278fed4f2c9 user: Phil Carmody date: Wed Jul 02 18:21:23 2014 +0300 description: lib: test-var-expand - use test_assert_idx() inside loops It helps narrow down which test has failed. Signed-off-by: Phil Carmody diffstat: src/lib/test-var-expand.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diffs (27 lines): diff -r 9511372c7c18 -r 1278fed4f2c9 src/lib/test-var-expand.c --- a/src/lib/test-var-expand.c Wed Jul 02 00:10:16 2014 +0300 +++ b/src/lib/test-var-expand.c Wed Jul 02 18:21:23 2014 +0300 @@ -70,7 +70,7 @@ for (i = 0; i < N_ELEMENTS(tests); i++) { str_truncate(str, 0); var_expand(str, tests[i].in, table); - test_assert(strcmp(tests[i].out, str_c(str)) == 0); + test_assert_idx(strcmp(tests[i].out, str_c(str)) == 0, i); } test_end(); } @@ -91,11 +91,11 @@ test_begin("var_get_key_range"); for (i = 0; i < N_ELEMENTS(tests); i++) { var_get_key_range(tests[i].in, &idx, &size); - test_assert(tests[i].idx == idx); - test_assert(tests[i].size == size); + test_assert_idx(tests[i].idx == idx, i); + test_assert_idx(tests[i].size == size, i); if (tests[i].size == 1) - test_assert(tests[i].in[idx] == var_get_key(tests[i].in)); + test_assert_idx(tests[i].in[idx] == var_get_key(tests[i].in), i); } test_end(); } From dovecot at dovecot.org Wed Jul 2 15:23:20 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:20 +0000 Subject: dovecot-2.2: lib: test-lib - add unit tests for str_to_*() helpers Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/6c1e44033e60 changeset: 17551:6c1e44033e60 user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: lib: test-lib - add unit tests for str_to_*() helpers This doesn't test all the helpers, but ensures both signed and unsigned are tested, as are 32-bit and 64-bit cases. All the other helpers fall back onto using one of those cases. Unless uintmax_t is larger than 64 bits, in which case this needs a revisit. NOTE: This causes the following make check errors: test-strnum.c:35: Assert(#7) failed: ret == u64tests[i].ret test-strnum.c:35: Assert(#10) failed: ret == u64tests[i].ret test-strnum.c:37: Assert(#10) failed: val == u64tests[i].val str_to_uint64 ........................................................ : FAILED Corresponding to test cases: [7] = INVALID(18446744073709551616), This does not wrap-past-0 (become smaller) on multiply, but wraps-past-0 on addition. [10]= INVALID(20496382304121724020), This wraps-past-n (becomes larger) on multiply. Signed-off-by: Phil Carmody diffstat: src/lib/Makefile.am | 1 + src/lib/test-lib.c | 1 + src/lib/test-lib.h | 1 + src/lib/test-strnum.c | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 144 insertions(+), 0 deletions(-) diffs (178 lines): diff -r 1278fed4f2c9 -r 6c1e44033e60 src/lib/Makefile.am --- a/src/lib/Makefile.am Wed Jul 02 18:21:23 2014 +0300 +++ b/src/lib/Makefile.am Wed Jul 02 18:21:24 2014 +0300 @@ -304,6 +304,7 @@ test-str.c \ test-strescape.c \ test-strfuncs.c \ + test-strnum.c \ test-str-find.c \ test-str-sanitize.c \ test-time-util.c \ diff -r 1278fed4f2c9 -r 6c1e44033e60 src/lib/test-lib.c --- a/src/lib/test-lib.c Wed Jul 02 18:21:23 2014 +0300 +++ b/src/lib/test-lib.c Wed Jul 02 18:21:24 2014 +0300 @@ -37,6 +37,7 @@ test_str, test_strescape, test_strfuncs, + test_strnum, test_str_find, test_str_sanitize, test_time_util, diff -r 1278fed4f2c9 -r 6c1e44033e60 src/lib/test-lib.h --- a/src/lib/test-lib.h Wed Jul 02 18:21:23 2014 +0300 +++ b/src/lib/test-lib.h Wed Jul 02 18:21:24 2014 +0300 @@ -36,6 +36,7 @@ void test_str(void); void test_strescape(void); void test_strfuncs(void); +void test_strnum(void); void test_str_find(void); void test_str_sanitize(void); void test_time_util(void); diff -r 1278fed4f2c9 -r 6c1e44033e60 src/lib/test-strnum.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/lib/test-strnum.c Wed Jul 02 18:21:24 2014 +0300 @@ -0,0 +1,141 @@ +/* Copyright (c) 2014-2014 Dovecot authors, see the included COPYING file */ + +#include "test-lib.h" + +#include + +#define INVALID(n) { #n, -1, 0 } +#define VALID(n) { #n, 0, n } + +/* Assumes uintmax_t is 64 bit */ +static void test_str_to_u64(void) +{ + unsigned int i; + const struct { + const char *input; + int ret; + uintmax_t val; + } u64tests[] = { + INVALID(-1), + INVALID(foo), + VALID(0), + { "0 ", -1, 0 }, /* invalid as doesn't end with a '\0' */ + VALID(000000000000000000000000000000000000000000000000000000000000000), + { "000000000000000000000000000000000000000000000000000001000000001", 0, 1000000001 }, + { "18446744073709551615", 0, 18446744073709551615ULL }, + INVALID(18446744073709551616), + INVALID(20496382304121724010), /* 2^64*10/9 doesn't wrap */ + INVALID(20496382304121724017), /* 2^64*10/9 wraps only after addition */ + INVALID(20496382304121724020), /* 2^64*10/9 wraps on multiply*/ + }; + test_begin("str_to_uint64"); + for (i = 0; i < N_ELEMENTS(u64tests); ++i) { + uintmax_t val = 0xBADBEEF15BADF00D; + int ret = str_to_uint64(u64tests[i].input, &val); + test_assert_idx(ret == u64tests[i].ret, i); + if (ret == 0) + test_assert_idx(val == u64tests[i].val, i); + else + test_assert_idx(val == 0xBADBEEF15BADF00D, i); + } + test_end(); +} + +static void test_str_to_u32(void) +{ + unsigned int i; + const struct { + const char *input; + int ret; + uint32_t val; + } u32tests[] = { + VALID(0), + INVALID(-0), + VALID(4294967295), + INVALID(4294967296), + INVALID(4772185880), + INVALID(4772185884), + INVALID(4772185890), + }; + test_begin("str_to_uint32"); + for (i = 0; i < N_ELEMENTS(u32tests); ++i) { + uint32_t val = 0xDEADF00D; + int ret = str_to_uint32(u32tests[i].input, &val); + test_assert_idx(ret == u32tests[i].ret, i); + if (ret == 0) + test_assert_idx(val == u32tests[i].val, i); + else + test_assert_idx(val == 0xDEADF00D, i); + } + test_end(); +} + +/* Assumes long long is 64 bit, 2's complement */ +static void test_str_to_llong(void) +{ + unsigned int i; + const struct { + const char *input; + int ret; + long long val; + } i64tests[] = { + VALID(0), + VALID(-0), + INVALID(--0), + VALID(2147483648), + VALID(-2147483649), + VALID(9223372036854775807), + { "-9223372036854775808", 0, -9223372036854775807-1 }, + INVALID(9223372036854775808), + INVALID(-9223372036854775809), + }; + test_begin("str_to_llong"); + for (i = 0; i < N_ELEMENTS(i64tests); ++i) { + long long val = 123456789; + int ret = str_to_llong(i64tests[i].input, &val); + test_assert_idx(ret == i64tests[i].ret, i); + if (ret == 0) + test_assert_idx(val == i64tests[i].val, i); + else + test_assert_idx(val == 123456789, i); + } + test_end(); +} + +/* Assumes int is 32 bit, 2's complement */ +static void test_str_to_i32(void) +{ + unsigned int i; + const struct { + const char *input; + int ret; + int val; + } i32tests[] = { + VALID(0), + VALID(-0), + INVALID(--0), + VALID(2147483647), + VALID(-2147483648), + INVALID(2147483648), + INVALID(-2147483649), + }; + test_begin("str_to_int"); + for (i = 0; i < N_ELEMENTS(i32tests); ++i) { + int val = 123456789; + int ret = str_to_int(i32tests[i].input, &val); + test_assert_idx(ret == i32tests[i].ret, i); + if (ret == 0) + test_assert_idx(val == i32tests[i].val, i); + else + test_assert_idx(val == 123456789, i); + } + test_end(); +} + +void test_strnum(void) +{ + test_str_to_u64(); + test_str_to_u32(); + test_str_to_llong(); + test_str_to_i32(); +} From dovecot at dovecot.org Wed Jul 2 15:23:20 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:20 +0000 Subject: dovecot-2.2: lib: strnum - harden str_to_uintmax against overflows Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/1d873182d5a8 changeset: 17552:1d873182d5a8 user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: lib: strnum - harden str_to_uintmax against overflows The invalid number "20496382304121724020" (2^64*10/9) will be parsed as valid. 2049638230412172402 * 10 does not noticably wrap, it becomes 2049638230412172404 Do not perform operations which might wrap, and then try to detect the issue, just compare with the known fixed bounds before doing the multiplication. Signed-off-by: Phil Carmody diffstat: src/lib/strnum.c | 13 +++++++------ 1 files changed, 7 insertions(+), 6 deletions(-) diffs (29 lines): diff -r 6c1e44033e60 -r 1d873182d5a8 src/lib/strnum.c --- a/src/lib/strnum.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib/strnum.c Wed Jul 02 18:21:24 2014 +0300 @@ -84,18 +84,19 @@ int str_to_uintmax(const char *str, uintmax_t *num_r) { - uintmax_t next, n = 0; + uintmax_t n = 0; if (*str < '0' || *str > '9') return -1; for (; *str >= '0' && *str <= '9'; str++) { - next = n*10; - if (next < n) { - /* overflow */ - return -1; + if (n >= ((uintmax_t)-1 / 10)) { + if (n > (uintmax_t)-1 / 10) + return -1; + if ((uintmax_t)(*str - '0') > ((uintmax_t)-1 % 10)) + return -1; } - n = next + (*str - '0'); + n = n * 10 + (*str - '0'); } if (*str != '\0') return -1; From dovecot at dovecot.org Wed Jul 2 15:23:20 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:20 +0000 Subject: dovecot-2.2: lib-imap: number parsing simplification and hardenning Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/59b6893fbf65 changeset: 17553:59b6893fbf65 user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: lib-imap: number parsing simplification and hardenning The invalid string "4772185884" (2^32*10/9) will be misparsed as being valid. In uint32_t's, 477218588 * 10 + 4 = 477218588 Many large ranges have this issue, 477218588x-858993459x, 954437176x-... We have helper functions - use them. Signed-off-by: Phil Carmody diffstat: src/lib-imap/imap-url.c | 46 ++++++++-------------------------------------- 1 files changed, 8 insertions(+), 38 deletions(-) diffs (75 lines): diff -r 1d873182d5a8 -r 59b6893fbf65 src/lib-imap/imap-url.c --- a/src/lib-imap/imap-url.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib-imap/imap-url.c Wed Jul 02 18:21:24 2014 +0300 @@ -128,9 +128,6 @@ imap_url_parse_number(struct uri_parser *parser, const char *data, uint32_t *number_r) { - uint32_t number = 0; - const char *p = data; - /* [IMAP4] RFC3501, Section 9 * * number = 1*DIGIT @@ -138,23 +135,11 @@ * ; (0 <= n < 4,294,967,296) */ - if (i_isdigit(*p)) { - do { - uint32_t prev = number; - - number = number * 10 + (*p - '0'); - if (number < prev) { - parser->error = "IMAP number is too high"; - return -1; - } - p++; - } while (i_isdigit(*p)); - - if (*p == '\0') { - if (number_r != NULL) - *number_r = number; + if (i_isdigit(*data)) { + if (str_to_uint32(data, number_r) == 0) return 1; - } + parser->error = "IMAP number is too high"; + return -1; } parser->error = t_strdup_printf( @@ -166,29 +151,14 @@ imap_url_parse_offset(struct uri_parser *parser, const char *data, uoff_t *number_r) { - uoff_t number = 0; - const char *p = data; - /* Syntax for big (uoff_t) numbers. Not strictly IMAP syntax, but this is handled similarly for Dovecot IMAP FETCH BODY partial <.> implementation. */ - if (i_isdigit(*p)) { - do { - uoff_t prev = number; - - number = number * 10 + (*p - '0'); - if (number < prev) { - parser->error = "IMAP number is too high"; - return -1; - } - p++; - } while (i_isdigit(*p)); - - if (*p == '\0') { - if (number_r != NULL) - *number_r = number; + if (i_isdigit(*data)) { + if (str_to_uoff(data, number_r) == 0) return 1; - } + parser->error = "IMAP number is too high"; + return -1; } parser->error = t_strdup_printf( From dovecot at dovecot.org Wed Jul 2 15:23:20 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:20 +0000 Subject: dovecot-2.2: lib: strnum - add permissive partial-string integer... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/27006dc092d2 changeset: 17554:27006dc092d2 user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: lib: strnum - add permissive partial-string integer parser Not all strings we want to parse are already strtok'ed into separate pieces. Therefore add helpers which will read the integer, and return a pointer past the parsed integer. The previous helpers can be considered a special case which just follows up with a check that the '\0' has been reached. Showing a preference for const pointers generally, this does not try to mimic the non-const interface of strto{l,ul,ll,ull}(). Signed-off-by: Phil Carmody diffstat: src/lib/strnum.c | 14 +++++++++++--- src/lib/strnum.h | 10 ++++++++-- 2 files changed, 19 insertions(+), 5 deletions(-) diffs (61 lines): diff -r 59b6893fbf65 -r 27006dc092d2 src/lib/strnum.c --- a/src/lib/strnum.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib/strnum.c Wed Jul 02 18:21:24 2014 +0300 @@ -82,7 +82,7 @@ return 0; } -int str_to_uintmax(const char *str, uintmax_t *num_r) +int str_parse_uintmax(const char *str, uintmax_t *num_r, const char **endp_r) { uintmax_t n = 0; @@ -98,11 +98,19 @@ } n = n * 10 + (*str - '0'); } - if (*str != '\0') - return -1; + if (endp_r != NULL) + *endp_r = str; *num_r = n; return 0; } +int str_to_uintmax(const char *str, uintmax_t *num_r) +{ + const char *endp; + int ret = str_parse_uintmax(str, num_r, &endp); + if ((ret == 0) && (*endp != '\0')) + ret = -1; + return ret; +} int str_to_int(const char *str, int *num_r) { diff -r 59b6893fbf65 -r 27006dc092d2 src/lib/strnum.h --- a/src/lib/strnum.h Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib/strnum.h Wed Jul 02 18:21:24 2014 +0300 @@ -5,8 +5,13 @@ Stop when `end_char' is found from string. */ bool str_is_numeric(const char *str, char end_char) ATTR_PURE; -/* str_to_*() functions return 0 if string is valid number in valid range. - Otherwise -1 is returned and num_r is left untouched */ +/* str_to_*() functions return 0 if string is nothing more than a valid number + in valid range. Otherwise -1 is returned and num_r is left untouched + + str_parse_*() helpers do not require the number to be the entire string + and pass back the pointer just past a valid parsed integer in endp_r if + it is non-NULL. What is written to endp_r in error cases is undefined. +*/ int str_to_uint(const char *str, unsigned int *num_r) ATTR_WARN_UNUSED_RESULT; int str_to_ulong(const char *str, unsigned long *num_r) ATTR_WARN_UNUSED_RESULT; @@ -14,6 +19,7 @@ int str_to_uint32(const char *str, uint32_t *num_r) ATTR_WARN_UNUSED_RESULT; int str_to_uint64(const char *str, uint64_t *num_r) ATTR_WARN_UNUSED_RESULT; int str_to_uintmax(const char *str, uintmax_t *num_r) ATTR_WARN_UNUSED_RESULT; +int str_parse_uintmax(const char *str, uintmax_t *num_r, const char **endp_r) ATTR_WARN_UNUSED_RESULT ATTR_NULL(3); int str_to_int(const char *str, int *num_r) ATTR_WARN_UNUSED_RESULT; int str_to_long(const char *str, long *num_r) ATTR_WARN_UNUSED_RESULT; From dovecot at dovecot.org Wed Jul 2 15:23:20 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:20 +0000 Subject: dovecot-2.2: lib: test-strnum - tests for the new partial-string... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/85a44b4b121a changeset: 17555:85a44b4b121a user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: lib: test-strnum - tests for the new partial-string parser We can simplify the main tests by always testing whether an appended non-digit causes parsing to fail at the same time that we test it doesn't fail with the new more permissive helpers. Signed-off-by: Phil Carmody diffstat: src/lib/test-strnum.c | 18 +++++++++++++++++- 1 files changed, 17 insertions(+), 1 deletions(-) diffs (42 lines): diff -r 27006dc092d2 -r 85a44b4b121a src/lib/test-strnum.c --- a/src/lib/test-strnum.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib/test-strnum.c Wed Jul 02 18:21:24 2014 +0300 @@ -19,7 +19,6 @@ INVALID(-1), INVALID(foo), VALID(0), - { "0 ", -1, 0 }, /* invalid as doesn't end with a '\0' */ VALID(000000000000000000000000000000000000000000000000000000000000000), { "000000000000000000000000000000000000000000000000000001000000001", 0, 1000000001 }, { "18446744073709551615", 0, 18446744073709551615ULL }, @@ -37,6 +36,19 @@ test_assert_idx(val == u64tests[i].val, i); else test_assert_idx(val == 0xBADBEEF15BADF00D, i); + + /* This stanza totally assumes that uintmax_t == uint64_t */ + if (ret == 0) + T_BEGIN { + const char *longer = t_strconcat(u64tests[i].input, "x", NULL); + const char *endp; + ret = str_to_uintmax(longer, &val); + test_assert_idx(ret < 0, i); + ret = str_parse_uintmax(longer, &val, &endp); + test_assert_idx(ret == 0, i); + test_assert_idx(val == u64tests[i].val, i); + test_assert_idx(*endp == 'x', i); + } T_END; } test_end(); } @@ -134,6 +146,10 @@ void test_strnum(void) { + test_begin("test_strnum - size check"); + test_assert(sizeof(uintmax_t) == sizeof(uint64_t)); + test_end(); + /* If the above isn't true, then we do expect some failures possibly */ test_str_to_u64(); test_str_to_u32(); test_str_to_llong(); From dovecot at dovecot.org Wed Jul 2 15:23:25 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:25 +0000 Subject: dovecot-2.2: lib: strnum - add a permissive uoff_t parser Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/dfc55d17314a changeset: 17556:dfc55d17314a user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: lib: strnum - add a permissive uoff_t parser Functions like these are so cookie-cutter, we may as well use a macro. Note that signed helpers, if they ever appear, will need more care. Signed-off-by: Phil Carmody diffstat: src/lib/strnum.c | 11 +++++++++++ src/lib/strnum.h | 1 + 2 files changed, 12 insertions(+), 0 deletions(-) diffs (32 lines): diff -r 85a44b4b121a -r dfc55d17314a src/lib/strnum.c --- a/src/lib/strnum.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib/strnum.c Wed Jul 02 18:21:24 2014 +0300 @@ -112,6 +112,17 @@ return ret; } +#define STR_TO_U__TEMPLATE(name, type) \ +int name(const char *str, type *num_r, const char **endp_r) \ +{ \ + uintmax_t l; \ + if (str_parse_uintmax(str, &l, endp_r) < 0 || l > (type)-1)\ + return -1; \ + *num_r = l; \ + return 0; \ +} +STR_TO_U__TEMPLATE(str_parse_uoff, uoff_t) + int str_to_int(const char *str, int *num_r) { intmax_t l; diff -r 85a44b4b121a -r dfc55d17314a src/lib/strnum.h --- a/src/lib/strnum.h Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib/strnum.h Wed Jul 02 18:21:24 2014 +0300 @@ -30,6 +30,7 @@ int str_to_gid(const char *str, gid_t *num_r) ATTR_WARN_UNUSED_RESULT; int str_to_pid(const char *str, pid_t *num_r) ATTR_WARN_UNUSED_RESULT; int str_to_uoff(const char *str, uoff_t *num_r) ATTR_WARN_UNUSED_RESULT; +int str_parse_uoff(const char *str, uoff_t *num_r, const char **endp_r) ATTR_WARN_UNUSED_RESULT ATTR_NULL(3); int str_to_time(const char *str, time_t *num_r) ATTR_WARN_UNUSED_RESULT; /* Returns TRUE if str is a valid unsigned number that equals to num. */ From dovecot at dovecot.org Wed Jul 2 15:23:26 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:26 +0000 Subject: dovecot-2.2: imap: harden read_uoff_t() against overflow Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/9b9b6dc7dd9b changeset: 17557:9b9b6dc7dd9b user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: imap: harden read_uoff_t() against overflow Invalid strings like "20496382304121724029" (2^64*10/9) can be parsed as valid. Use the new helper. Change in error behaviour - previously overflows, if they were detected, caused *p to point to the digit causing the overflow. Now it's undefined. Current clients don't care about this difference, they just bail. Signed-off-by: Phil Carmody diffstat: src/imap/imap-fetch-body.c | 14 +------------- 1 files changed, 1 insertions(+), 13 deletions(-) diffs (24 lines): diff -r dfc55d17314a -r 9b9b6dc7dd9b src/imap/imap-fetch-body.c --- a/src/imap/imap-fetch-body.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/imap/imap-fetch-body.c Wed Jul 02 18:21:24 2014 +0300 @@ -208,19 +208,7 @@ becomes too big and wraps. */ static int read_uoff_t(const char **p, uoff_t *value) { - uoff_t prev; - - *value = 0; - while (**p >= '0' && **p <= '9') { - prev = *value; - *value = *value * 10 + (**p - '0'); - - if (*value < prev) - return -1; - - (*p)++; - } - return 0; + return str_parse_uoff(*p, value, p); } static int From dovecot at dovecot.org Wed Jul 2 15:23:26 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:26 +0000 Subject: dovecot-2.2: lib-http: test-http-url - make port number test cas... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/f21e0f0e42a3 changeset: 17558:f21e0f0e42a3 user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: lib-http: test-http-url - make port number test cases harder Exploit common parser weakeneses - out by one, and overflow detection failure. NOTE: causes make check to fail with the following error: http url invalid [13]: parse http://example.com:72817/index.html ..... : FAILED http url invalid [13] ................................................ : FAILED Signed-off-by: Phil Carmody diffstat: src/lib-http/test-http-url.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diffs (14 lines): diff -r 9b9b6dc7dd9b -r f21e0f0e42a3 src/lib-http/test-http-url.c --- a/src/lib-http/test-http-url.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib-http/test-http-url.c Wed Jul 02 18:21:24 2014 +0300 @@ -372,7 +372,9 @@ },{ .url = "http://example%00.com/index.html" },{ - .url = "http://example.com:65539/index.html" + .url = "http://example.com:65536/index.html" + },{ + .url = "http://example.com:72817/index.html" },{ .url = "http://example.com/settings/%00/" },{ From dovecot at dovecot.org Wed Jul 2 15:23:26 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:26 +0000 Subject: dovecot-2.2: lib: uri-util - harden uri_parse_port against overflow Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/659ea72a82aa changeset: 17559:659ea72a82aa user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: lib: uri-util - harden uri_parse_port against overflow The invalid input 72817 (2^16*10/9) is parsed as a valid value. 7281 * 10 + 7 = 72817 == 7281 (mod 2^16), so the prev check fails. Signed-off-by: Phil Carmody diffstat: src/lib/uri-util.c | 8 +++----- 1 files changed, 3 insertions(+), 5 deletions(-) diffs (25 lines): diff -r f21e0f0e42a3 -r 659ea72a82aa src/lib/uri-util.c --- a/src/lib/uri-util.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib/uri-util.c Wed Jul 02 18:21:24 2014 +0300 @@ -479,7 +479,7 @@ static int uri_parse_port(struct uri_parser *parser, struct uri_authority *auth) { - in_port_t port = 0; + unsigned long port = 0; int count = 0; /* RFC 3986: @@ -488,10 +488,8 @@ */ while (parser->cur < parser->end && i_isdigit(*parser->cur)) { - in_port_t prev = port; - - port = port * 10 + (in_port_t)(parser->cur[0] - '0'); - if (port < prev) { + port = port * 10 + (parser->cur[0] - '0'); + if (port > (in_port_t)-1) { parser->error = "Port number is too high"; return -1; } From dovecot at dovecot.org Wed Jul 2 15:23:26 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:26 +0000 Subject: dovecot-2.2: lib-http: test-http-url - add some tricky invalid n... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/a39ccb12c14f changeset: 17560:a39ccb12c14f user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: lib-http: test-http-url - add some tricky invalid numeric hostname URLs Try to get the numeric octet parser to fail. The RFCs specify that we should fall back onto parsing them as domain names instead, and hence the unexpected legitimacy of out-of-range numbers. NOTE: This causes make check to report the following error: http url valid [11]: http_url_parse(http://127.0.0.284/this/also/reverts/to/DNS) : ok test-http-url.c:328: Assert failed: urlp->have_host_ip == urlt->have_host_ip http url valid [11] .................................................. : FAILED Signed-off-by: Phil Carmody diffstat: src/lib-http/test-http-url.c | 20 ++++++++++++++++++++ 1 files changed, 20 insertions(+), 0 deletions(-) diffs (30 lines): diff -r 659ea72a82aa -r a39ccb12c14f src/lib-http/test-http-url.c --- a/src/lib-http/test-http-url.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib-http/test-http-url.c Wed Jul 02 18:21:24 2014 +0300 @@ -72,6 +72,26 @@ .host_name = "www.example.com", .enc_query = "question=What%20are%20you%20doing%3f&answer=Nothing." } },{ + /* These next 2 URLs don't follow the recommendations in + http://tools.ietf.org/html/rfc1034#section-3.5 and + http://tools.ietf.org/html/rfc3696 + However they satisfy the grammar in + http://tools.ietf.org/html/rfc1123#section-2 and + http://tools.ietf.org/html/rfc952 + so we should parse them. + */ + .url = "http://256.0.0.1/that/reverts/to/DNS", + .url_parsed = { + .path = "/that/reverts/to/DNS", + .host_name = "256.0.0.1" + } + },{ + .url = "http://127.0.0.284/this/also/reverts/to/DNS", + .url_parsed = { + .path = "/this/also/reverts/to/DNS", + .host_name = "127.0.0.284" + } + },{ .url = "http://www.example.com/#Status%20of%20development", .flags = HTTP_URL_ALLOW_FRAGMENT_PART, .url_parsed = { From dovecot at dovecot.org Wed Jul 2 15:23:26 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:26 +0000 Subject: dovecot-2.2: lib: uri-util - hardern uri_parse_dec_octet() again... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/439bba739c49 changeset: 17561:439bba739c49 user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: lib: uri-util - hardern uri_parse_dec_octet() against overflow Invalid input 284 (2^8*10/9) is incorrectly parsed as valid. 28 * 10 + 4 = 284 == 28 (mod 2^8), so the wrap detection fails. Signed-off-by: Phil Carmody diffstat: src/lib/uri-util.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-) diffs (24 lines): diff -r a39ccb12c14f -r 439bba739c49 src/lib/uri-util.c --- a/src/lib/uri-util.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib/uri-util.c Wed Jul 02 18:21:24 2014 +0300 @@ -258,7 +258,7 @@ uri_parse_dec_octet(struct uri_parser *parser, string_t *literal, uint8_t *octet_r) { - uint8_t octet = 0; + unsigned int octet = 0; int count = 0; /* RFC 3986: @@ -271,10 +271,8 @@ */ while (parser->cur < parser->end && i_isdigit(*parser->cur)) { - uint8_t prev = octet; - octet = octet * 10 + (parser->cur[0] - '0'); - if (octet < prev) + if (octet > 255) return -1; if (literal != NULL) From dovecot at dovecot.org Wed Jul 2 15:23:26 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:26 +0000 Subject: dovecot-2.2: lib: strnum - add permissive str_parse_uint() helper Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/2051de7285c4 changeset: 17562:2051de7285c4 user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: lib: strnum - add permissive str_parse_uint() helper Signed-off-by: Phil Carmody diffstat: src/lib/strnum.c | 1 + src/lib/strnum.h | 1 + 2 files changed, 2 insertions(+), 0 deletions(-) diffs (22 lines): diff -r 439bba739c49 -r 2051de7285c4 src/lib/strnum.c --- a/src/lib/strnum.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib/strnum.c Wed Jul 02 18:21:24 2014 +0300 @@ -122,6 +122,7 @@ return 0; \ } STR_TO_U__TEMPLATE(str_parse_uoff, uoff_t) +STR_TO_U__TEMPLATE(str_parse_uint, unsigned int) int str_to_int(const char *str, int *num_r) { diff -r 439bba739c49 -r 2051de7285c4 src/lib/strnum.h --- a/src/lib/strnum.h Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib/strnum.h Wed Jul 02 18:21:24 2014 +0300 @@ -14,6 +14,7 @@ */ int str_to_uint(const char *str, unsigned int *num_r) ATTR_WARN_UNUSED_RESULT; +int str_parse_uint(const char *str, unsigned int *num_r, const char **endp_r) ATTR_WARN_UNUSED_RESULT ATTR_NULL(3); int str_to_ulong(const char *str, unsigned long *num_r) ATTR_WARN_UNUSED_RESULT; int str_to_ullong(const char *str, unsigned long long *num_r) ATTR_WARN_UNUSED_RESULT; int str_to_uint32(const char *str, uint32_t *num_r) ATTR_WARN_UNUSED_RESULT; From dovecot at dovecot.org Wed Jul 2 15:23:26 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 15:23:26 +0000 Subject: dovecot-2.2: pop3: pop3-commands - harden integer parsers agains... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/2ed2ab04b63d changeset: 17563:2ed2ab04b63d user: Phil Carmody date: Wed Jul 02 18:21:24 2014 +0300 description: pop3: pop3-commands - harden integer parsers against integer overflow In get_msgnum(), the invalid input "4772185884" (2^32*10/9) would be parsed as being valid. In get_size(), the invalid input "204963823041217240178" (2^64*10/9) would be parsed as being valid. We have helpers now, so use them. Signed-off-by: Phil Carmody diffstat: src/pop3/pop3-commands.c | 20 ++++++-------------- 1 files changed, 6 insertions(+), 14 deletions(-) diffs (59 lines): diff -r 2051de7285c4 -r 2ed2ab04b63d src/pop3/pop3-commands.c --- a/src/pop3/pop3-commands.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/pop3/pop3-commands.c Wed Jul 02 18:21:24 2014 +0300 @@ -28,24 +28,20 @@ static const char *get_msgnum(struct client *client, const char *args, unsigned int *msgnum) { - unsigned int num, last_num; + unsigned int num; - num = 0; - while (*args != '\0' && *args != ' ') { + if (*args != '\0' && *args != ' ') { if (*args < '0' || *args > '9') { client_send_line(client, "-ERR Invalid message number: %s", args); return NULL; } - last_num = num; - num = num*10 + (*args - '0'); - if (num < last_num) { + if (str_parse_uint(args, &num, &args) < 0) { client_send_line(client, "-ERR Message number too large: %s", args); return NULL; } - args++; } if (num == 0 || num > client->messages_count) { @@ -72,24 +68,20 @@ static const char *get_size(struct client *client, const char *args, uoff_t *size) { - uoff_t num, last_num; + uoff_t num; - num = 0; - while (*args != '\0' && *args != ' ') { + if (*args != '\0' && *args != ' ') { if (*args < '0' || *args > '9') { client_send_line(client, "-ERR Invalid size: %s", args); return NULL; } - last_num = num; - num = num*10 + (*args - '0'); - if (num < last_num) { + if (str_parse_uoff(args, &num, &args) < 0) { client_send_line(client, "-ERR Size too large: %s", args); return NULL; } - args++; } while (*args == ' ') args++; From dovecot at dovecot.org Wed Jul 2 17:38:30 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 17:38:30 +0000 Subject: dovecot-2.2: lib: Added UINT64_SUM_OVERFLOWS() Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/d77d880c1385 changeset: 17564:d77d880c1385 user: Timo Sirainen date: Wed Jul 02 20:13:35 2014 +0300 description: lib: Added UINT64_SUM_OVERFLOWS() Maybe the unit tests are kind of unnecessary since the macro is so simple, but at least it's now a well tested simple macro :) diffstat: src/lib/bits.h | 3 +++ src/lib/test-bits.c | 24 ++++++++++++++++++++++++ 2 files changed, 27 insertions(+), 0 deletions(-) diffs (49 lines): diff -r 2ed2ab04b63d -r d77d880c1385 src/lib/bits.h --- a/src/lib/bits.h Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib/bits.h Wed Jul 02 20:13:35 2014 +0300 @@ -12,6 +12,9 @@ #include #include +#define UINT64_SUM_OVERFLOWS(a, b) \ + (a > (uint64_t)-1 - b) + size_t nearest_power(size_t num) ATTR_CONST; unsigned int bits_required8(uint8_t num) ATTR_CONST; diff -r 2ed2ab04b63d -r d77d880c1385 src/lib/test-bits.c --- a/src/lib/test-bits.c Wed Jul 02 18:21:24 2014 +0300 +++ b/src/lib/test-bits.c Wed Jul 02 20:13:35 2014 +0300 @@ -60,8 +60,32 @@ test_end(); } +static void test_sum_overflows(void) +{ +#define MAX64 (uint64_t)-1 + static const struct { + uint64_t a, b; + bool overflows; + } tests[] = { + { MAX64-1, 1, FALSE }, + { MAX64, 1, TRUE }, + { MAX64-1, 1, FALSE }, + { MAX64-1, 2, TRUE }, + { MAX64-1, MAX64-1, TRUE }, + { MAX64-1, MAX64, TRUE }, + { MAX64, MAX64, TRUE } + }; + unsigned int i; + + test_begin("UINT64_SUM_OVERFLOWS"); + for (i = 0; i < N_ELEMENTS(tests); i++) + test_assert(UINT64_SUM_OVERFLOWS(tests[i].a, tests[i].b) == tests[i].overflows); + test_end(); +} + void test_bits() { test_nearest_power(); test_bits_requiredXX(); + test_sum_overflows(); } From dovecot at dovecot.org Wed Jul 2 17:38:35 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 17:38:35 +0000 Subject: dovecot-2.2: quota: Moved some functions to quota-util.c Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/0d4de84a54f0 changeset: 17565:0d4de84a54f0 user: Timo Sirainen date: Wed Jul 02 20:34:43 2014 +0300 description: quota: Moved some functions to quota-util.c diffstat: src/plugins/quota/Makefile.am | 4 +- src/plugins/quota/quota-private.h | 12 +- src/plugins/quota/quota-util.c | 401 ++++++++++++++++++++++++++++++++++++ src/plugins/quota/quota.c | 421 +------------------------------------ 4 files changed, 426 insertions(+), 412 deletions(-) diffs (truncated from 959 to 300 lines): diff -r d77d880c1385 -r 0d4de84a54f0 src/plugins/quota/Makefile.am --- a/src/plugins/quota/Makefile.am Wed Jul 02 20:13:35 2014 +0300 +++ b/src/plugins/quota/Makefile.am Wed Jul 02 20:34:43 2014 +0300 @@ -29,7 +29,8 @@ quota-dirsize.c \ quota-maildir.c \ quota-plugin.c \ - quota-storage.c + quota-storage.c \ + quota-util.c quota_common_objects = \ quota.lo \ @@ -40,6 +41,7 @@ quota-maildir.lo \ quota-plugin.lo \ quota-storage.lo \ + quota-util.lo \ $(RQUOTA_XDR_LO) lib10_quota_plugin_la_SOURCES = $(quota_dist_sources) diff -r d77d880c1385 -r 0d4de84a54f0 src/plugins/quota/quota-private.h --- a/src/plugins/quota/quota-private.h Wed Jul 02 20:13:35 2014 +0300 +++ b/src/plugins/quota/quota-private.h Wed Jul 02 20:34:43 2014 +0300 @@ -150,8 +150,9 @@ after the first allocation is done, bytes_ceil is set to bytes_ceil2. */ uint64_t bytes_ceil, bytes_ceil2, count_ceil; - /* how many bytes/mails we are over quota (either *_ceil or *_over - is always zero) */ + /* How many bytes/mails we are over quota. Like *_ceil, these are set + only once and not updated by bytes_used/count_used. (Either *_ceil + or *_over is always zero.) */ uint64_t bytes_over, count_over; struct mail *tmp_mail; @@ -178,4 +179,11 @@ int64_t count_limit); int quota_count(struct quota_root *root, uint64_t *bytes_r, uint64_t *count_r); +int quota_root_parse_grace(struct quota_root_settings *root_set, + const char *value, const char **error_r); +bool quota_warning_match(const struct quota_warning_rule *w, + uint64_t bytes_before, uint64_t bytes_current, + uint64_t count_before, uint64_t count_current); +bool quota_transaction_is_over(struct quota_transaction_context *ctx, uoff_t size); + #endif diff -r d77d880c1385 -r 0d4de84a54f0 src/plugins/quota/quota-util.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/plugins/quota/quota-util.c Wed Jul 02 20:34:43 2014 +0300 @@ -0,0 +1,401 @@ +/* Copyright (c) 2005-2014 Dovecot authors, see the included COPYING file */ + +#include "lib.h" +#include "wildcard-match.h" +#include "quota-private.h" + +#include +#include + +#define QUOTA_DEFAULT_GRACE "10%" + +#define RULE_NAME_DEFAULT_FORCE "*" +#define RULE_NAME_DEFAULT_NONFORCE "?" + +struct quota_rule * +quota_root_rule_find(struct quota_root_settings *root_set, const char *name) +{ + struct quota_rule *rule; + + array_foreach_modifiable(&root_set->rules, rule) { + if (wildcard_match(name, rule->mailbox_mask)) + return rule; + } + return NULL; +} + +static struct quota_rule * +quota_root_rule_find_exact(struct quota_root_settings *root_set, + const char *name) +{ + struct quota_rule *rule; + + array_foreach_modifiable(&root_set->rules, rule) { + if (strcmp(rule->mailbox_mask, name) == 0) + return rule; + } + return NULL; +} + +static int +quota_rule_parse_percentage(struct quota_root_settings *root_set, + struct quota_rule *rule, + int64_t *limit, const char **error_r) +{ + int64_t percentage = *limit; + + if (percentage <= -100 || percentage >= UINT_MAX) { + *error_r = "Invalid percentage"; + return -1; + } + + if (rule == &root_set->default_rule) { + *error_r = "Default rule can't be a percentage"; + return -1; + } + + if (limit == &rule->bytes_limit) + rule->bytes_percent = percentage; + else if (limit == &rule->count_limit) + rule->count_percent = percentage; + else + i_unreached(); + return 0; +} + +static int quota_limit_parse(struct quota_root_settings *root_set, + struct quota_rule *rule, const char *unit, + uint64_t multiply, int64_t *limit, + const char **error_r) +{ + switch (i_toupper(*unit)) { + case '\0': + /* default */ + break; + case 'B': + multiply = 1; + break; + case 'K': + multiply = 1024; + break; + case 'M': + multiply = 1024*1024; + break; + case 'G': + multiply = 1024*1024*1024; + break; + case 'T': + multiply = 1024ULL*1024*1024*1024; + break; + case '%': + multiply = 0; + if (quota_rule_parse_percentage(root_set, rule, limit, + error_r) < 0) + return -1; + break; + default: + *error_r = t_strdup_printf("Unknown unit: %s", unit); + return -1; + } + *limit *= multiply; + return 0; +} + +static void +quota_rule_recalculate_relative_rules(struct quota_rule *rule, + int64_t bytes_limit, int64_t count_limit) +{ + if (rule->bytes_percent != 0) + rule->bytes_limit = bytes_limit * rule->bytes_percent / 100; + if (rule->count_percent != 0) + rule->count_limit = count_limit * rule->count_percent / 100; +} + +void quota_root_recalculate_relative_rules(struct quota_root_settings *root_set, + int64_t bytes_limit, + int64_t count_limit) +{ + struct quota_rule *rule; + struct quota_warning_rule *warning_rule; + + array_foreach_modifiable(&root_set->rules, rule) { + quota_rule_recalculate_relative_rules(rule, bytes_limit, + count_limit); + } + + array_foreach_modifiable(&root_set->warning_rules, warning_rule) { + quota_rule_recalculate_relative_rules(&warning_rule->rule, + bytes_limit, count_limit); + } + quota_rule_recalculate_relative_rules(&root_set->grace_rule, + bytes_limit, 0); + root_set->last_mail_max_extra_bytes = root_set->grace_rule.bytes_limit; + + if (root_set->set->debug && root_set->set->initialized) { + i_debug("Quota root %s: Recalculated relative rules with " + "bytes=%lld count=%lld. Now grace=%llu", root_set->name, + (long long)bytes_limit, (long long)count_limit, + (unsigned long long)root_set->last_mail_max_extra_bytes); + } +} + +static int +quota_rule_parse_limits(struct quota_root_settings *root_set, + struct quota_rule *rule, const char *limits, + const char *full_rule_def, + bool relative_rule, const char **error_r) +{ + const char **args, *key, *value, *error; + char *p; + uint64_t multiply; + int64_t *limit; + + args = t_strsplit(limits, ":"); + for (; *args != NULL; args++) { + multiply = 1; + limit = NULL; + + key = *args; + value = strchr(key, '='); + if (value == NULL) + value = ""; + else + key = t_strdup_until(key, value++); + + if (*value == '+') { + if (!relative_rule) { + *error_r = "Rule limit cannot have '+'"; + return -1; + } + value++; + } else if (*value != '-' && relative_rule) { + i_warning("quota root %s rule %s: " + "obsolete configuration for rule '%s' " + "should be changed to '%s=+%s'", + root_set->name, full_rule_def, + *args, key, value); + } + + if (strcmp(key, "storage") == 0) { + multiply = 1024; + limit = &rule->bytes_limit; + *limit = strtoll(value, &p, 10); + } else if (strcmp(key, "bytes") == 0) { + limit = &rule->bytes_limit; + *limit = strtoll(value, &p, 10); + } else if (strcmp(key, "messages") == 0) { + limit = &rule->count_limit; + *limit = strtoll(value, &p, 10); + } else { + *error_r = p_strdup_printf(root_set->set->pool, + "Unknown rule limit name: %s", key); + return -1; + } + + if (quota_limit_parse(root_set, rule, p, multiply, + limit, &error) < 0) { + *error_r = p_strdup_printf(root_set->set->pool, + "Invalid rule limit value '%s': %s", + *args, error); + return -1; + } + } + if (!relative_rule) { + if (rule->bytes_limit < 0) { + *error_r = "Bytes limit can't be negative"; + return -1; + } + if (rule->count_limit < 0) { + *error_r = "Count limit can't be negative"; + return -1; + } + } + return 0; +} + +int quota_root_add_rule(struct quota_root_settings *root_set, + const char *rule_def, const char **error_r) +{ + struct quota_rule *rule; + const char *p, *mailbox_mask; + int ret = 0; + + p = strchr(rule_def, ':'); + if (p == NULL) { + *error_r = "Invalid rule"; + return -1; + } + + /* : */ + mailbox_mask = t_strdup_until(rule_def, p++); + + rule = quota_root_rule_find_exact(root_set, mailbox_mask); + if (rule == NULL) { + if (strcmp(mailbox_mask, RULE_NAME_DEFAULT_NONFORCE) == 0) + rule = &root_set->default_rule; + else if (strcmp(mailbox_mask, RULE_NAME_DEFAULT_FORCE) == 0) { + rule = &root_set->default_rule; + root_set->force_default_rule = TRUE; + } else { + rule = array_append_space(&root_set->rules); + rule->mailbox_mask = strcasecmp(mailbox_mask, "INBOX") == 0 ? "INBOX" : + p_strdup(root_set->set->pool, mailbox_mask); + } + } + + if (strcmp(p, "ignore") == 0) { + rule->ignore = TRUE; + if (root_set->set->debug) { From dovecot at dovecot.org Wed Jul 2 17:38:35 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 17:38:35 +0000 Subject: dovecot-2.2: quota: Fixed quota_transaction_is_over() to handle ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/76d573ec5045 changeset: 17566:76d573ec5045 user: Timo Sirainen date: Wed Jul 02 20:36:49 2014 +0300 description: quota: Fixed quota_transaction_is_over() to handle "user is already over quota" case. If size=0 we didn't return failure. This change also fixes various potential integer overflows in the check. Added unit test for the function. diffstat: src/plugins/quota/Makefile.am | 20 ++++++++ src/plugins/quota/quota-util.c | 54 ++++++++++++++++++-- src/plugins/quota/test-quota-util.c | 92 +++++++++++++++++++++++++++++++++++++ 3 files changed, 159 insertions(+), 7 deletions(-) diffs (192 lines): diff -r 0d4de84a54f0 -r 76d573ec5045 src/plugins/quota/Makefile.am --- a/src/plugins/quota/Makefile.am Wed Jul 02 20:34:43 2014 +0300 +++ b/src/plugins/quota/Makefile.am Wed Jul 02 20:36:49 2014 +0300 @@ -5,6 +5,7 @@ AM_CPPFLAGS = \ -I$(top_srcdir)/src/lib \ + -I$(top_srcdir)/src/lib-test \ -I$(top_srcdir)/src/lib-master \ -I$(top_srcdir)/src/lib-dict \ -I$(top_srcdir)/src/lib-index \ @@ -105,3 +106,22 @@ rm -f $(top_builddir)/src/plugins/quota/rquota.x; \ fi; \ rm -f rquota_xdr.c rquota.h + +test_programs = \ + test-quota-util +noinst_PROGRAMS = $(test_programs) + +test_libs = \ + ../../lib-test/libtest.la \ + ../../lib/liblib.la +test_deps = $(noinst_LTLIBRARIES) $(test_libs) + +test_quota_util_SOURCES = test-quota-util.c +test_quota_util_LDADD = quota-util.o $(test_libs) +test_quota_util_DEPENDENCIES = quota-util.o $(test_deps) + +check: check-am check-test +check-test: all-am + for bin in $(test_programs); do \ + if ! $(RUN_TEST) ./$$bin; then exit 1; fi; \ + done diff -r 0d4de84a54f0 -r 76d573ec5045 src/plugins/quota/quota-util.c --- a/src/plugins/quota/quota-util.c Wed Jul 02 20:34:43 2014 +0300 +++ b/src/plugins/quota/quota-util.c Wed Jul 02 20:36:49 2014 +0300 @@ -391,11 +391,51 @@ bool quota_transaction_is_over(struct quota_transaction_context *ctx, uoff_t size) { - if ((ctx->count_used < 0 || - (uint64_t)ctx->count_used + 1 <= ctx->count_ceil) && - ((ctx->bytes_used < 0 && size <= ctx->bytes_ceil) || - (uint64_t)ctx->bytes_used + size <= ctx->bytes_ceil)) - return FALSE; - else - return TRUE; + if (ctx->count_used < 0) { + /* we've deleted some messages. we should be ok, unless we + were already over quota and still are after these + deletions. */ + if (ctx->count_over > 0) { + if (ctx->count_over > (uint64_t)-ctx->count_used + 1) + return TRUE; + } else { + return TRUE; + } + } else { + if (ctx->count_ceil < 1 || + ctx->count_ceil - 1 < (uint64_t)ctx->count_used) { + /* count limit reached */ + return TRUE; + } + } + + if (ctx->bytes_used < 0) { + const uint64_t bytes_deleted = (uint64_t)-ctx->bytes_used; + + /* we've deleted some messages. same logic as above. */ + if (ctx->bytes_over > 0) { + if (ctx->bytes_over > bytes_deleted) { + /* even after deletions we're over quota */ + return TRUE; + } + if (size > bytes_deleted - ctx->bytes_over) + return TRUE; + } else { + if (size > bytes_deleted && + size - bytes_deleted < ctx->bytes_ceil) + return TRUE; + } + } else if (size == 0) { + /* we need to explicitly test this case, since the generic + check would fail if user is already over quota */ + if (ctx->bytes_over > 0) + return TRUE; + } else { + if (ctx->bytes_ceil < size || + ctx->bytes_ceil - size < (uint64_t)ctx->bytes_used) { + /* bytes limit reached */ + return TRUE; + } + } + return FALSE; } diff -r 0d4de84a54f0 -r 76d573ec5045 src/plugins/quota/test-quota-util.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/plugins/quota/test-quota-util.c Wed Jul 02 20:36:49 2014 +0300 @@ -0,0 +1,92 @@ +/* Copyright (c) 2014 Dovecot authors, see the included COPYING file */ + +#include "lib.h" +#include "quota-private.h" +#include "test-common.h" + +struct test { + uint64_t limit, initial_size; + int64_t transaction_diff; + uint64_t new_size; + bool is_over; +}; + +static void test_quota_transaction_is_over(void) +{ +#define MAXU64 (uint64_t)-1 +#define MAXS64 9223372036854775807LL +#define MINS64 (-MAXS64 - 1LL) + static const struct test tests[] = { + /* first test only with new_size=1. these are used for both + count and bytes tests: */ + + /* limit, init, diff, new */ + { 1, 0, 0, 1, FALSE }, + { MAXU64, MAXU64, 0, 1, TRUE }, + { MAXU64, MAXU64-1, 0, 1, FALSE }, + { MAXU64, MAXU64-1, 1, 1, TRUE }, + + /* these are for bytes tests: */ + + /* limit, init, diff, new */ + { MAXU64, MAXU64, 0, 0, FALSE }, + { MAXU64, MAXU64-1, 1, 0, FALSE }, + { MAXU64-1, MAXU64, 1, 0, TRUE }, + { MAXU64-1, MAXU64, 0, 0, TRUE }, + { MAXU64-1, MAXU64, -1, 0, FALSE }, + { MAXU64, MAXU64, 0, 1, TRUE }, + { MAXU64, 0, 0, MAXU64, FALSE }, + { MAXU64, 1, 0, MAXU64, TRUE }, + { MAXU64, 0, 1, MAXU64, TRUE }, + { MAXU64-1, 0, 0, MAXU64, TRUE }, + { MAXU64-1, 0, 0, MAXU64-1, FALSE }, + { MAXU64-1, 1, 0, MAXU64-1, TRUE }, + { MAXU64-1, 1, -1, MAXU64-1, FALSE }, + { MAXU64, MAXU64, 0, MAXU64, TRUE }, + }; + struct quota_transaction_context ctx; + unsigned int i; + + test_begin("quota transcation is over (count)"); + for (i = 0; i < N_ELEMENTS(tests); i++) { + if (tests[i].new_size != 1) + continue; + + memset(&ctx, 0, sizeof(ctx)); + ctx.count_used = tests[i].transaction_diff; + if (tests[i].initial_size > tests[i].limit) + ctx.count_over = tests[i].initial_size - tests[i].limit; + else { + ctx.count_ceil = tests[i].limit - tests[i].initial_size; + i_assert(ctx.count_used < 0 || + (uint64_t)ctx.count_used <= ctx.count_ceil); /* test is broken otherwise */ + } + test_assert_idx(quota_transaction_is_over(&ctx, 0) == tests[i].is_over, i); + } + test_end(); + + test_begin("quota transcation is over (bytes)"); + for (i = 0; i < N_ELEMENTS(tests); i++) { + memset(&ctx, 0, sizeof(ctx)); + ctx.count_ceil = 1; + ctx.bytes_used = tests[i].transaction_diff; + if (tests[i].initial_size > tests[i].limit) + ctx.bytes_over = tests[i].initial_size - tests[i].limit; + else { + ctx.bytes_ceil = tests[i].limit - tests[i].initial_size; + i_assert(ctx.bytes_used < 0 || + (uint64_t)ctx.bytes_used <= ctx.bytes_ceil); /* test is broken otherwise */ + } + test_assert_idx(quota_transaction_is_over(&ctx, tests[i].new_size) == tests[i].is_over, i); + } + test_end(); +} + +int main(void) +{ + static void (*test_functions[])(void) = { + test_quota_transaction_is_over, + NULL + }; + return test_run(test_functions); +} From dovecot at dovecot.org Wed Jul 2 17:55:28 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 17:55:28 +0000 Subject: dovecot-2.2: test-quota-util: Link to quota-util.lo instead of .o Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/f2f9181f42b1 changeset: 17567:f2f9181f42b1 user: Timo Sirainen date: Wed Jul 02 20:53:46 2014 +0300 description: test-quota-util: Link to quota-util.lo instead of .o Hopefully fixes dependency tracking to work correctly? diffstat: src/plugins/quota/Makefile.am | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r 76d573ec5045 -r f2f9181f42b1 src/plugins/quota/Makefile.am --- a/src/plugins/quota/Makefile.am Wed Jul 02 20:36:49 2014 +0300 +++ b/src/plugins/quota/Makefile.am Wed Jul 02 20:53:46 2014 +0300 @@ -117,8 +117,8 @@ test_deps = $(noinst_LTLIBRARIES) $(test_libs) test_quota_util_SOURCES = test-quota-util.c -test_quota_util_LDADD = quota-util.o $(test_libs) -test_quota_util_DEPENDENCIES = quota-util.o $(test_deps) +test_quota_util_LDADD = quota-util.lo $(test_libs) +test_quota_util_DEPENDENCIES = quota-util.lo $(test_deps) check: check-am check-test check-test: all-am From dovecot at dovecot.org Wed Jul 2 20:42:45 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 02 Jul 2014 20:42:45 +0000 Subject: dovecot-2.2: lib-index: Recent idx->seq change in strmap forgot ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/d25443778208 changeset: 17568:d25443778208 user: Timo Sirainen date: Wed Jul 02 23:41:10 2014 +0300 description: lib-index: Recent idx->seq change in strmap forgot to initialize uid_lookup_seq in one place. diffstat: src/lib-index/mail-index-strmap.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diffs (20 lines): diff -r f2f9181f42b1 -r d25443778208 src/lib-index/mail-index-strmap.c --- a/src/lib-index/mail-index-strmap.c Wed Jul 02 20:53:46 2014 +0300 +++ b/src/lib-index/mail-index-strmap.c Wed Jul 02 23:41:10 2014 +0300 @@ -381,6 +381,8 @@ { const struct mail_index_record *rec; + i_assert(ctx->uid_lookup_seq > 0); + if (ctx->uid_lookup_seq > ctx->view->view->map->hdr.messages_count) { if (uid >= ctx->view->view->map->hdr.next_uid) { /* thread index has larger UIDs than what we've seen @@ -826,6 +828,7 @@ memset(&ctx, 0, sizeof(ctx)); ctx.view = view; + ctx.uid_lookup_seq = 1; /* create a map of old -> new index and remove records of expunged messages */ From dovecot at dovecot.org Thu Jul 3 08:38:39 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 08:38:39 +0000 Subject: dovecot-2.2: quota: Quota count tracking still was still incorre... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/6fcf8199b079 changeset: 17569:6fcf8199b079 user: Timo Sirainen date: Thu Jul 03 11:37:07 2014 +0300 description: quota: Quota count tracking still was still incorrect in over-quota conditions. diffstat: src/plugins/quota/quota-util.c | 6 +++--- src/plugins/quota/test-quota-util.c | 4 ++++ 2 files changed, 7 insertions(+), 3 deletions(-) diffs (32 lines): diff -r d25443778208 -r 6fcf8199b079 src/plugins/quota/quota-util.c --- a/src/plugins/quota/quota-util.c Wed Jul 02 23:41:10 2014 +0300 +++ b/src/plugins/quota/quota-util.c Thu Jul 03 11:37:07 2014 +0300 @@ -395,11 +395,11 @@ /* we've deleted some messages. we should be ok, unless we were already over quota and still are after these deletions. */ + const uint64_t count_deleted = (uint64_t)-ctx->count_used; + if (ctx->count_over > 0) { - if (ctx->count_over > (uint64_t)-ctx->count_used + 1) + if (count_deleted - 1 < ctx->count_over) return TRUE; - } else { - return TRUE; } } else { if (ctx->count_ceil < 1 || diff -r d25443778208 -r 6fcf8199b079 src/plugins/quota/test-quota-util.c --- a/src/plugins/quota/test-quota-util.c Wed Jul 02 23:41:10 2014 +0300 +++ b/src/plugins/quota/test-quota-util.c Thu Jul 03 11:37:07 2014 +0300 @@ -25,6 +25,10 @@ { MAXU64, MAXU64, 0, 1, TRUE }, { MAXU64, MAXU64-1, 0, 1, FALSE }, { MAXU64, MAXU64-1, 1, 1, TRUE }, + { MAXU64-1, MAXU64-1, 0, 1, TRUE }, + { MAXU64-1, MAXU64-1, -1, 1, FALSE }, + { MAXU64-2, MAXU64-1, -1, 1, TRUE }, + { MAXU64-2, MAXU64-1, -2, 1, FALSE }, /* these are for bytes tests: */ From dovecot at dovecot.org Thu Jul 3 09:46:32 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 09:46:32 +0000 Subject: dovecot-2.2: lib-imap: test_imap_url didn't compare port numbers Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/636b10a03e95 changeset: 17570:636b10a03e95 user: Phil Carmody date: Thu Jul 03 12:42:11 2014 +0300 description: lib-imap: test_imap_url didn't compare port numbers It only compare them when they were unset, and defaulting both to 0. Signed-off-by: Phil Carmody diffstat: src/lib-imap/test-imap-url.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 6fcf8199b079 -r 636b10a03e95 src/lib-imap/test-imap-url.c --- a/src/lib-imap/test-imap-url.c Thu Jul 03 11:37:07 2014 +0300 +++ b/src/lib-imap/test-imap-url.c Thu Jul 03 12:42:11 2014 +0300 @@ -650,7 +650,7 @@ test_out(t_strdup_printf("url->auth_type = %s", urlp->auth_type), strcmp(urlp->auth_type, urlt->auth_type) == 0); } - if (urlp->have_port) { + if (!urlp->have_port) { test_out("url->port = (unspecified)", urlp->have_port == urlt->have_port); } else { From dovecot at dovecot.org Thu Jul 3 09:46:33 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 09:46:33 +0000 Subject: dovecot-2.2: lib-test: test-common - add test_out_quiet() to red... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/36d07648cec8 changeset: 17571:36d07648cec8 user: Phil Carmody date: Thu Jul 03 12:42:11 2014 +0300 description: lib-test: test-common - add test_out_quiet() to reduce verbosity Like test_out() but only prints anything if success is false. This makes it quite much like test_assert(), except that it doesn't print the code fragment, it prints a custom string. However, it still counts as a test in the total count, unlike test_assert*()s. Signed-off-by: Phil Carmody diffstat: src/lib-test/test-common.c | 9 +++++++++ src/lib-test/test-common.h | 1 + 2 files changed, 10 insertions(+), 0 deletions(-) diffs (30 lines): diff -r 636b10a03e95 -r 36d07648cec8 src/lib-test/test-common.c --- a/src/lib-test/test-common.c Thu Jul 03 12:42:11 2014 +0300 +++ b/src/lib-test/test-common.c Thu Jul 03 12:42:11 2014 +0300 @@ -191,6 +191,15 @@ test_out_reason(name, success, NULL); } +void test_out_quiet(const char *name, bool success) +{ + if (success) { + total_count++; + return; + } + test_out(name, success); +} + void test_out_reason(const char *name, bool success, const char *reason) { int i = 0; diff -r 636b10a03e95 -r 36d07648cec8 src/lib-test/test-common.h --- a/src/lib-test/test-common.h Thu Jul 03 12:42:11 2014 +0300 +++ b/src/lib-test/test-common.h Thu Jul 03 12:42:11 2014 +0300 @@ -23,6 +23,7 @@ void test_end(void); void test_out(const char *name, bool success); +void test_out_quiet(const char *name, bool success); /* only prints failures */ void test_out_reason(const char *name, bool success, const char *reason) ATTR_NULL(3); From dovecot at dovecot.org Thu Jul 3 09:46:38 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 09:46:38 +0000 Subject: dovecot-2.2: lib-imap: test-imap-url - quieten successful sub-tests Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/55bf79d2e69d changeset: 17572:55bf79d2e69d user: Phil Carmody date: Thu Jul 03 12:42:11 2014 +0300 description: lib-imap: test-imap-url - quieten successful sub-tests Every sub-component of a URL doesn't need its own successful log, so use the only-print-on-error test_out_quiet() function instead. All failures are just as explicit as before. Signed-off-by: Phil Carmody diffstat: src/lib-imap/test-imap-url.c | 60 ++++++++++++++++++++++---------------------- 1 files changed, 30 insertions(+), 30 deletions(-) diffs (161 lines): diff -r 36d07648cec8 -r 55bf79d2e69d src/lib-imap/test-imap-url.c --- a/src/lib-imap/test-imap-url.c Thu Jul 03 12:42:11 2014 +0300 +++ b/src/lib-imap/test-imap-url.c Thu Jul 03 12:42:11 2014 +0300 @@ -630,119 +630,119 @@ valid_url_tests[i].url), urlp != NULL, error); if (urlp != NULL) { if (urlp->host_name == NULL || urlt->host_name == NULL) { - test_out(t_strdup_printf("url->host_name = %s", urlp->host_name), + test_out_quiet(t_strdup_printf("url->host_name = %s", urlp->host_name), urlp->host_name == urlt->host_name); } else { - test_out(t_strdup_printf("url->host_name = %s", urlp->host_name), + test_out_quiet(t_strdup_printf("url->host_name = %s", urlp->host_name), strcmp(urlp->host_name, urlt->host_name) == 0); } if (urlp->userid == NULL || urlt->userid == NULL) { - test_out(t_strdup_printf("url->userid = %s", urlp->userid), + test_out_quiet(t_strdup_printf("url->userid = %s", urlp->userid), urlp->userid == urlt->userid); } else { - test_out(t_strdup_printf("url->userid = %s", urlp->userid), + test_out_quiet(t_strdup_printf("url->userid = %s", urlp->userid), strcmp(urlp->userid, urlt->userid) == 0); } if (urlp->auth_type == NULL || urlt->auth_type == NULL) { - test_out(t_strdup_printf("url->auth_type = %s", urlp->auth_type), + test_out_quiet(t_strdup_printf("url->auth_type = %s", urlp->auth_type), urlp->auth_type == urlt->auth_type); } else { - test_out(t_strdup_printf("url->auth_type = %s", urlp->auth_type), + test_out_quiet(t_strdup_printf("url->auth_type = %s", urlp->auth_type), strcmp(urlp->auth_type, urlt->auth_type) == 0); } if (!urlp->have_port) { - test_out("url->port = (unspecified)", + test_out_quiet("url->port = (unspecified)", urlp->have_port == urlt->have_port); } else { - test_out(t_strdup_printf("url->port = %u", urlp->port), + test_out_quiet(t_strdup_printf("url->port = %u", urlp->port), urlp->have_port == urlt->have_port && urlp->port == urlt->port); } if (!urlp->have_host_ip) { - test_out("url->host_ip = (unspecified)", + test_out_quiet("url->host_ip = (unspecified)", urlp->have_host_ip == urlt->have_host_ip); } else { - test_out("url->host_ip = (valid)", + test_out_quiet("url->host_ip = (valid)", urlp->have_host_ip == urlt->have_host_ip); } if (urlp->mailbox == NULL || urlt->mailbox == NULL) { - test_out(t_strdup_printf("url->mailbox = %s", urlp->mailbox), + test_out_quiet(t_strdup_printf("url->mailbox = %s", urlp->mailbox), urlp->mailbox == urlt->mailbox); } else { - test_out(t_strdup_printf("url->mailbox = %s", urlp->mailbox), + test_out_quiet(t_strdup_printf("url->mailbox = %s", urlp->mailbox), strcmp(urlp->mailbox, urlt->mailbox) == 0); } - test_out(t_strdup_printf("url->uidvalidity = %u", urlp->uidvalidity), + test_out_quiet(t_strdup_printf("url->uidvalidity = %u", urlp->uidvalidity), urlp->uidvalidity == urlt->uidvalidity); - test_out(t_strdup_printf("url->uid = %u", urlp->uid), + test_out_quiet(t_strdup_printf("url->uid = %u", urlp->uid), urlp->uid == urlt->uid); if (urlp->section == NULL || urlt->section == NULL) { - test_out(t_strdup_printf("url->section = %s", urlp->section), + test_out_quiet(t_strdup_printf("url->section = %s", urlp->section), urlp->section == urlt->section); } else { - test_out(t_strdup_printf("url->section = %s", urlp->section), + test_out_quiet(t_strdup_printf("url->section = %s", urlp->section), strcmp(urlp->section, urlt->section) == 0); } - test_out(t_strdup_printf("url->partial = %"PRIuUOFF_T".%"PRIuUOFF_T, + test_out_quiet(t_strdup_printf("url->partial = %"PRIuUOFF_T".%"PRIuUOFF_T, urlp->partial_offset, urlp->partial_size), urlp->partial_offset == urlt->partial_offset && urlp->partial_size == urlt->partial_size); if (urlp->search_program == NULL || urlt->search_program == NULL) { - test_out(t_strdup_printf( + test_out_quiet(t_strdup_printf( "url->search_program = %s", urlp->search_program), urlp->search_program == urlt->search_program); } else { - test_out(t_strdup_printf( + test_out_quiet(t_strdup_printf( "url->search_program = %s", urlp->search_program), strcmp(urlp->search_program, urlt->search_program) == 0); } if (urlt->uauth_rumpurl != NULL) { if (urlp->uauth_rumpurl == NULL) { - test_out(t_strdup_printf( + test_out_quiet(t_strdup_printf( "url->uauth_rumpurl = %s", urlp->uauth_rumpurl), FALSE); } else { - test_out(t_strdup_printf( + test_out_quiet(t_strdup_printf( "url->uauth_rumpurl = %s", urlp->uauth_rumpurl), strcmp(urlp->uauth_rumpurl, urlt->uauth_rumpurl) == 0); } if (urlp->uauth_access_application == NULL || urlt->uauth_access_application == NULL) { - test_out(t_strdup_printf("url->uauth_access_application = %s", + test_out_quiet(t_strdup_printf("url->uauth_access_application = %s", urlp->uauth_access_application), urlp->uauth_access_application == urlt->uauth_access_application); } else { - test_out(t_strdup_printf("url->uauth_access_application = %s", + test_out_quiet(t_strdup_printf("url->uauth_access_application = %s", urlp->uauth_access_application), strcmp(urlp->uauth_access_application, urlt->uauth_access_application) == 0); } if (urlp->uauth_access_user == NULL || urlt->uauth_access_user == NULL) { - test_out(t_strdup_printf("url->uauth_access_user = %s", + test_out_quiet(t_strdup_printf("url->uauth_access_user = %s", urlp->uauth_access_user), urlp->uauth_access_user == urlt->uauth_access_user); } else { - test_out(t_strdup_printf("url->uauth_access_user = %s", + test_out_quiet(t_strdup_printf("url->uauth_access_user = %s", urlp->uauth_access_user), strcmp(urlp->uauth_access_user, urlt->uauth_access_user) == 0); } if (urlp->uauth_mechanism == NULL || urlt->uauth_mechanism == NULL) { - test_out(t_strdup_printf( + test_out_quiet(t_strdup_printf( "url->uauth_mechanism = %s", urlp->uauth_mechanism), urlp->uauth_mechanism == urlt->uauth_mechanism); } else { - test_out(t_strdup_printf( + test_out_quiet(t_strdup_printf( "url->uauth_mechanism = %s", urlp->uauth_mechanism), strcmp(urlp->uauth_mechanism, urlt->uauth_mechanism) == 0); } if (urlp->uauth_token == NULL || urlt->uauth_token == NULL) { - test_out(t_strdup_printf( + test_out_quiet(t_strdup_printf( "url->uauth_token = %s", urlp->uauth_token), urlp->uauth_token == urlt->uauth_token); } else { bool equal = urlp->uauth_token_size == urlt->uauth_token_size; size_t i; - test_out(t_strdup_printf( + test_out_quiet(t_strdup_printf( "url->uauth_token_size = %"PRIuSIZE_T, urlp->uauth_token_size), equal); @@ -753,7 +753,7 @@ break; } } - test_out(t_strdup_printf("url->uauth_token [index=%d]", (int)i), + test_out_quiet(t_strdup_printf("url->uauth_token [index=%d]", (int)i), equal); } } From dovecot at dovecot.org Thu Jul 3 09:46:38 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 09:46:38 +0000 Subject: dovecot-2.2: lib-imap: test-imap-url - cosmetic whitespace cleanup Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/140619879047 changeset: 17573:140619879047 user: Phil Carmody date: Thu Jul 03 12:44:50 2014 +0300 description: lib-imap: test-imap-url - cosmetic whitespace cleanup Only whitespace changes. All trailing space removed, reindented: $ grep '[[:space:]]$' src/lib-imap/test-imap-url.c $ git diff -w $ Signed-off-by: Phil Carmody diffstat: src/lib-imap/test-imap-url.c | 94 ++++++++++++++++++++++---------------------- 1 files changed, 47 insertions(+), 47 deletions(-) diffs (187 lines): diff -r 55bf79d2e69d -r 140619879047 src/lib-imap/test-imap-url.c --- a/src/lib-imap/test-imap-url.c Thu Jul 03 12:42:11 2014 +0300 +++ b/src/lib-imap/test-imap-url.c Thu Jul 03 12:44:50 2014 +0300 @@ -631,120 +631,120 @@ if (urlp != NULL) { if (urlp->host_name == NULL || urlt->host_name == NULL) { test_out_quiet(t_strdup_printf("url->host_name = %s", urlp->host_name), - urlp->host_name == urlt->host_name); + urlp->host_name == urlt->host_name); } else { test_out_quiet(t_strdup_printf("url->host_name = %s", urlp->host_name), - strcmp(urlp->host_name, urlt->host_name) == 0); + strcmp(urlp->host_name, urlt->host_name) == 0); } if (urlp->userid == NULL || urlt->userid == NULL) { test_out_quiet(t_strdup_printf("url->userid = %s", urlp->userid), - urlp->userid == urlt->userid); + urlp->userid == urlt->userid); } else { test_out_quiet(t_strdup_printf("url->userid = %s", urlp->userid), - strcmp(urlp->userid, urlt->userid) == 0); + strcmp(urlp->userid, urlt->userid) == 0); } if (urlp->auth_type == NULL || urlt->auth_type == NULL) { test_out_quiet(t_strdup_printf("url->auth_type = %s", urlp->auth_type), - urlp->auth_type == urlt->auth_type); + urlp->auth_type == urlt->auth_type); } else { test_out_quiet(t_strdup_printf("url->auth_type = %s", urlp->auth_type), - strcmp(urlp->auth_type, urlt->auth_type) == 0); + strcmp(urlp->auth_type, urlt->auth_type) == 0); } if (!urlp->have_port) { test_out_quiet("url->port = (unspecified)", - urlp->have_port == urlt->have_port); + urlp->have_port == urlt->have_port); } else { test_out_quiet(t_strdup_printf("url->port = %u", urlp->port), - urlp->have_port == urlt->have_port && urlp->port == urlt->port); + urlp->have_port == urlt->have_port && urlp->port == urlt->port); } if (!urlp->have_host_ip) { test_out_quiet("url->host_ip = (unspecified)", - urlp->have_host_ip == urlt->have_host_ip); + urlp->have_host_ip == urlt->have_host_ip); } else { test_out_quiet("url->host_ip = (valid)", - urlp->have_host_ip == urlt->have_host_ip); + urlp->have_host_ip == urlt->have_host_ip); } if (urlp->mailbox == NULL || urlt->mailbox == NULL) { test_out_quiet(t_strdup_printf("url->mailbox = %s", urlp->mailbox), - urlp->mailbox == urlt->mailbox); + urlp->mailbox == urlt->mailbox); } else { test_out_quiet(t_strdup_printf("url->mailbox = %s", urlp->mailbox), - strcmp(urlp->mailbox, urlt->mailbox) == 0); + strcmp(urlp->mailbox, urlt->mailbox) == 0); } test_out_quiet(t_strdup_printf("url->uidvalidity = %u", urlp->uidvalidity), - urlp->uidvalidity == urlt->uidvalidity); + urlp->uidvalidity == urlt->uidvalidity); test_out_quiet(t_strdup_printf("url->uid = %u", urlp->uid), - urlp->uid == urlt->uid); + urlp->uid == urlt->uid); if (urlp->section == NULL || urlt->section == NULL) { test_out_quiet(t_strdup_printf("url->section = %s", urlp->section), - urlp->section == urlt->section); + urlp->section == urlt->section); } else { test_out_quiet(t_strdup_printf("url->section = %s", urlp->section), - strcmp(urlp->section, urlt->section) == 0); + strcmp(urlp->section, urlt->section) == 0); } test_out_quiet(t_strdup_printf("url->partial = %"PRIuUOFF_T".%"PRIuUOFF_T, - urlp->partial_offset, urlp->partial_size), - urlp->partial_offset == urlt->partial_offset && - urlp->partial_size == urlt->partial_size); + urlp->partial_offset, urlp->partial_size), + urlp->partial_offset == urlt->partial_offset && + urlp->partial_size == urlt->partial_size); if (urlp->search_program == NULL || urlt->search_program == NULL) { test_out_quiet(t_strdup_printf( - "url->search_program = %s", urlp->search_program), - urlp->search_program == urlt->search_program); + "url->search_program = %s", urlp->search_program), + urlp->search_program == urlt->search_program); } else { test_out_quiet(t_strdup_printf( - "url->search_program = %s", urlp->search_program), - strcmp(urlp->search_program, urlt->search_program) == 0); + "url->search_program = %s", urlp->search_program), + strcmp(urlp->search_program, urlt->search_program) == 0); } if (urlt->uauth_rumpurl != NULL) { if (urlp->uauth_rumpurl == NULL) { test_out_quiet(t_strdup_printf( - "url->uauth_rumpurl = %s", urlp->uauth_rumpurl), FALSE); + "url->uauth_rumpurl = %s", urlp->uauth_rumpurl), FALSE); } else { test_out_quiet(t_strdup_printf( - "url->uauth_rumpurl = %s", urlp->uauth_rumpurl), - strcmp(urlp->uauth_rumpurl, urlt->uauth_rumpurl) == 0); + "url->uauth_rumpurl = %s", urlp->uauth_rumpurl), + strcmp(urlp->uauth_rumpurl, urlt->uauth_rumpurl) == 0); } if (urlp->uauth_access_application == NULL || - urlt->uauth_access_application == NULL) { + urlt->uauth_access_application == NULL) { test_out_quiet(t_strdup_printf("url->uauth_access_application = %s", - urlp->uauth_access_application), - urlp->uauth_access_application == urlt->uauth_access_application); + urlp->uauth_access_application), + urlp->uauth_access_application == urlt->uauth_access_application); } else { test_out_quiet(t_strdup_printf("url->uauth_access_application = %s", - urlp->uauth_access_application), - strcmp(urlp->uauth_access_application, - urlt->uauth_access_application) == 0); + urlp->uauth_access_application), + strcmp(urlp->uauth_access_application, + urlt->uauth_access_application) == 0); } if (urlp->uauth_access_user == NULL || - urlt->uauth_access_user == NULL) { + urlt->uauth_access_user == NULL) { test_out_quiet(t_strdup_printf("url->uauth_access_user = %s", - urlp->uauth_access_user), - urlp->uauth_access_user == urlt->uauth_access_user); + urlp->uauth_access_user), + urlp->uauth_access_user == urlt->uauth_access_user); } else { test_out_quiet(t_strdup_printf("url->uauth_access_user = %s", - urlp->uauth_access_user), - strcmp(urlp->uauth_access_user, - urlt->uauth_access_user) == 0); + urlp->uauth_access_user), + strcmp(urlp->uauth_access_user, + urlt->uauth_access_user) == 0); } if (urlp->uauth_mechanism == NULL || urlt->uauth_mechanism == NULL) { test_out_quiet(t_strdup_printf( - "url->uauth_mechanism = %s", urlp->uauth_mechanism), - urlp->uauth_mechanism == urlt->uauth_mechanism); + "url->uauth_mechanism = %s", urlp->uauth_mechanism), + urlp->uauth_mechanism == urlt->uauth_mechanism); } else { test_out_quiet(t_strdup_printf( "url->uauth_mechanism = %s", urlp->uauth_mechanism), - strcmp(urlp->uauth_mechanism, urlt->uauth_mechanism) == 0); + strcmp(urlp->uauth_mechanism, urlt->uauth_mechanism) == 0); } if (urlp->uauth_token == NULL || urlt->uauth_token == NULL) { test_out_quiet(t_strdup_printf( - "url->uauth_token = %s", urlp->uauth_token), - urlp->uauth_token == urlt->uauth_token); + "url->uauth_token = %s", urlp->uauth_token), + urlp->uauth_token == urlt->uauth_token); } else { bool equal = urlp->uauth_token_size == urlt->uauth_token_size; size_t i; test_out_quiet(t_strdup_printf( - "url->uauth_token_size = %"PRIuSIZE_T, urlp->uauth_token_size), - equal); + "url->uauth_token_size = %"PRIuSIZE_T, urlp->uauth_token_size), + equal); if (equal) { for (i = 0; i < urlp->uauth_token_size; i++) { @@ -754,7 +754,7 @@ } } test_out_quiet(t_strdup_printf("url->uauth_token [index=%d]", (int)i), - equal); + equal); } } } @@ -1014,7 +1014,7 @@ if (urlp != NULL) { const char *urlnew = imap_url_create(urlp); test_out(t_strdup_printf - ("create %s", urlnew), strcmp(url, urlnew) == 0); + ("create %s", urlnew), strcmp(url, urlnew) == 0); } test_end(); From dovecot at dovecot.org Thu Jul 3 11:38:53 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 11:38:53 +0000 Subject: dovecot-2.2: fts: If we detect corrupted fts expunge log, unlink... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/9c6643daae98 changeset: 17574:9c6643daae98 user: Timo Sirainen date: Thu Jul 03 14:37:08 2014 +0300 description: fts: If we detect corrupted fts expunge log, unlink it. This avoids the same error repeating forever. diffstat: src/plugins/fts/fts-expunge-log.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diffs (15 lines): diff -r 140619879047 -r 9c6643daae98 src/plugins/fts/fts-expunge-log.c --- a/src/plugins/fts/fts-expunge-log.c Thu Jul 03 12:44:50 2014 +0300 +++ b/src/plugins/fts/fts-expunge-log.c Thu Jul 03 14:37:08 2014 +0300 @@ -459,6 +459,11 @@ *_ctx = NULL; + if (ctx->corrupted) { + if (unlink(ctx->log->path) < 0 && errno != ENOENT) + i_error("unlink(%s) failed: %m", ctx->log->path); + } + if (ctx->input != NULL) i_stream_unref(&ctx->input); i_free(ctx); From dovecot at dovecot.org Thu Jul 3 11:56:21 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 11:56:21 +0000 Subject: dovecot-2.2: virtual: Recent flags dropping wasn't working as in... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/abf049058125 changeset: 17575:abf049058125 user: Timo Sirainen date: Thu Jul 03 14:54:43 2014 +0300 description: virtual: Recent flags dropping wasn't working as intended. In the old code '+' meant that \Recent flags were dropped also when the virtual mailbox was EXAMINEd. SELECTing a mailbox always dropped \Recent flags regardless of the '+' flag. What should have happened (and does in new code) is that the \Recent flags are dropped only on SELECT and only if '+' flag is set. diffstat: src/plugins/virtual/virtual-storage.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r 9c6643daae98 -r abf049058125 src/plugins/virtual/virtual-storage.c --- a/src/plugins/virtual/virtual-storage.c Thu Jul 03 14:37:08 2014 +0300 +++ b/src/plugins/virtual/virtual-storage.c Thu Jul 03 14:54:43 2014 +0300 @@ -156,8 +156,8 @@ i_assert(bbox->box == NULL); - if (bbox->clear_recent) - flags |= MAILBOX_FLAG_DROP_RECENT; + if (!bbox->clear_recent) + flags &= ~MAILBOX_FLAG_DROP_RECENT; mailbox = bbox->name; ns = mail_namespace_find(user->namespaces, mailbox); From dovecot at dovecot.org Thu Jul 3 12:28:21 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 12:28:21 +0000 Subject: dovecot-2.2: lib-storage: mailbox_get_metadata() now opens the m... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/af665736536c changeset: 17576:af665736536c user: Timo Sirainen date: Thu Jul 03 15:26:32 2014 +0300 description: lib-storage: mailbox_get_metadata() now opens the mailbox only if it's necessary. diffstat: src/lib-storage/index/index-status.c | 16 +++++++++++----- 1 files changed, 11 insertions(+), 5 deletions(-) diffs (32 lines): diff -r abf049058125 -r af665736536c src/lib-storage/index/index-status.c --- a/src/lib-storage/index/index-status.c Thu Jul 03 14:54:43 2014 +0300 +++ b/src/lib-storage/index/index-status.c Thu Jul 03 15:26:32 2014 +0300 @@ -375,6 +375,17 @@ enum mailbox_metadata_items items, struct mailbox_metadata *metadata_r) { + /* handle items that don't require opening the mailbox */ + if ((items & MAILBOX_METADATA_BACKEND_NAMESPACE) != 0) { + metadata_r->backend_ns_prefix = ""; + metadata_r->backend_ns_type = + mailbox_list_get_namespace(box->list)->type; + items &= ~MAILBOX_METADATA_BACKEND_NAMESPACE; + } + if (items == 0) + return 0; + + /* handle items that require opening the mailbox */ if (!box->opened) { if (mailbox_open(box) < 0) return -1; @@ -392,10 +403,5 @@ get_metadata_cache_fields(box, metadata_r); if ((items & MAILBOX_METADATA_PRECACHE_FIELDS) != 0) get_metadata_precache_fields(box, metadata_r); - if ((items & MAILBOX_METADATA_BACKEND_NAMESPACE) != 0) { - metadata_r->backend_ns_prefix = ""; - metadata_r->backend_ns_type = - mailbox_list_get_namespace(box->list)->type; - } return 0; } From dovecot at dovecot.org Thu Jul 3 13:09:02 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 13:09:02 +0000 Subject: dovecot-2.2: lib: DLLIST*_REMOVE*() no longer breaks the linked ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/46e47e4082d8 changeset: 17577:46e47e4082d8 user: Timo Sirainen date: Thu Jul 03 16:07:09 2014 +0300 description: lib: DLLIST*_REMOVE*() no longer breaks the linked list if we try to remove item that doesn't exist there. Hopefully there wasn't any code that actually did this, but it's safer this way anyway. Perhaps it could be even made to assert-crash if it happens. diffstat: src/lib/llist.h | 19 +++++++++---------- src/lib/test-llist.c | 13 +++++++++++++ 2 files changed, 22 insertions(+), 10 deletions(-) diffs (84 lines): diff -r af665736536c -r 46e47e4082d8 src/lib/llist.h --- a/src/lib/llist.h Thu Jul 03 15:26:32 2014 +0300 +++ b/src/lib/llist.h Thu Jul 03 16:07:09 2014 +0300 @@ -13,10 +13,10 @@ DLLIST_PREPEND_FULL(list, item, prev, next) #define DLLIST_REMOVE_FULL(list, item, prev, next) STMT_START { \ - if ((item)->prev == NULL) \ + if ((item)->prev != NULL) \ + (item)->prev->next = (item)->next; \ + else if ((*list) == item) \ *(list) = (item)->next; \ - else \ - (item)->prev->next = (item)->next; \ if ((item)->next != NULL) { \ (item)->next->prev = (item)->prev; \ (item)->next = NULL; \ @@ -61,16 +61,15 @@ DLLIST2_INSERT_AFTER_FULL(head, tail, after, item, prev, next) #define DLLIST2_REMOVE_FULL(head, tail, item, prev, next) STMT_START { \ - if ((item)->prev == NULL) \ + if ((item)->prev != NULL) \ + (item)->prev->next = (item)->next; \ + else if (*(head) == item) \ *(head) = (item)->next; \ - else \ - (item)->prev->next = (item)->next; \ - if ((item)->next == NULL) \ - *(tail) = (item)->prev; \ - else { \ + if ((item)->next != NULL) { \ (item)->next->prev = (item)->prev; \ (item)->next = NULL; \ - } \ + } else if ((*tail) == item) \ + *(tail) = (item)->prev; \ (item)->prev = NULL; \ } STMT_END diff -r af665736536c -r 46e47e4082d8 src/lib/test-llist.c --- a/src/lib/test-llist.c Thu Jul 03 15:26:32 2014 +0300 +++ b/src/lib/test-llist.c Thu Jul 03 16:07:09 2014 +0300 @@ -12,6 +12,7 @@ static void test_dllist(void) { struct dllist *head = NULL, *l4, *l3, *l2, *l1; + struct dllist empty = { NULL, NULL }; l4 = t_new(struct dllist, 1); l3 = t_new(struct dllist, 1); @@ -48,6 +49,11 @@ test_assert(head == l1); test_assert(l1->prev == NULL && l1->next == l3); test_assert(l3->prev == l1 && l3->next == NULL); + /* removal of an entry not in the list shouldn't cause the list to break */ + DLLIST_REMOVE(&head, &empty); + test_assert(head == l1); + test_assert(l1->prev == NULL && l1->next == l3); + test_assert(l3->prev == l1 && l3->next == NULL); /* remove last two */ DLLIST_REMOVE(&head, l1); DLLIST_REMOVE(&head, l3); @@ -59,6 +65,7 @@ static void test_dllist2(void) { struct dllist *head = NULL, *tail = NULL, *l4, *l3, *l2, *l1; + struct dllist empty = { NULL, NULL }; l4 = t_new(struct dllist, 1); l3 = t_new(struct dllist, 1); @@ -111,6 +118,12 @@ test_assert(head == l1 && tail == l3); test_assert(l1->prev == NULL && l1->next == l3); test_assert(l3->prev == l1 && l3->next == NULL); + /* removal of an entry not in the list shouldn't cause the list to break */ + DLLIST2_REMOVE(&head, &tail, &empty); + test_assert(head == l1); + test_assert(head == l1 && tail == l3); + test_assert(l1->prev == NULL && l1->next == l3); + test_assert(l3->prev == l1 && l3->next == NULL); /* remove last two */ DLLIST2_REMOVE(&head, &tail, l1); DLLIST2_REMOVE(&head, &tail, l3); From dovecot at dovecot.org Thu Jul 3 14:31:39 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 14:31:39 +0000 Subject: dovecot-2.2: lib-index: Index cache could have kept too many ind... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/8944983786cc changeset: 17578:8944983786cc user: Timo Sirainen date: Thu Jul 03 17:29:58 2014 +0300 description: lib-index: Index cache could have kept too many indexes open. If a lot of indexes were allocated and then later on they were opened and closed, the alloc-cache simply kept all the indexes open even after they should have been closed. diffstat: src/lib-index/mail-index-alloc-cache.c | 64 +++++++++++++++++++++++++++------ src/lib-index/mail-index-alloc-cache.h | 1 + src/lib-index/mail-index.c | 2 + 3 files changed, 55 insertions(+), 12 deletions(-) diffs (156 lines): diff -r 46e47e4082d8 -r 8944983786cc src/lib-index/mail-index-alloc-cache.c --- a/src/lib-index/mail-index-alloc-cache.c Thu Jul 03 16:07:09 2014 +0300 +++ b/src/lib-index/mail-index-alloc-cache.c Thu Jul 03 17:29:58 2014 +0300 @@ -33,6 +33,7 @@ static MODULE_CONTEXT_DEFINE_INIT(mail_index_alloc_cache_index_module, &mail_index_module_register); static struct mail_index_alloc_cache_list *indexes = NULL; +static unsigned int indexes_cache_references_count = 0; static struct timeout *to_index = NULL; static struct mail_index_alloc_cache_list * @@ -57,10 +58,23 @@ } static void +mail_index_alloc_cache_list_unref(struct mail_index_alloc_cache_list *list) +{ + i_assert(list->referenced); + i_assert(indexes_cache_references_count > 0); + + indexes_cache_references_count--; + mail_index_close(list->index); + list->referenced = FALSE; +} + +static void mail_index_alloc_cache_list_free(struct mail_index_alloc_cache_list *list) { + i_assert(list->refcount == 0); + if (list->referenced) - mail_index_close(list->index); + mail_index_alloc_cache_list_unref(list); mail_index_free(&list->index); i_free(list->mailbox_path); i_free(list); @@ -153,29 +167,34 @@ return match->index; } -static void destroy_unrefed(bool all) +static bool destroy_unrefed(unsigned int min_destroy_count) { struct mail_index_alloc_cache_list **list, *rec; + bool destroyed = FALSE; bool seen_ref0 = FALSE; for (list = &indexes; *list != NULL;) { rec = *list; if (rec->refcount == 0 && - (all || rec->destroy_time <= ioloop_time)) { + (min_destroy_count > 0 || rec->destroy_time <= ioloop_time)) { *list = rec->next; + destroyed = TRUE; mail_index_alloc_cache_list_free(rec); + if (min_destroy_count > 0) + min_destroy_count--; } else { if (rec->refcount == 0) seen_ref0 = TRUE; - if (all && rec->index->open_count == 1 && + if (min_destroy_count > 0 && + rec->index->open_count == 1 && rec->referenced) { /* we're the only one keeping this index open. we might be here, because the caller is deleting this mailbox and wants its indexes to be closed. so close it. */ - rec->referenced = FALSE; - mail_index_close(rec->index); + destroyed = TRUE; + mail_index_alloc_cache_list_unref(rec); } list = &(*list)->next; } @@ -183,12 +202,13 @@ if (!seen_ref0 && to_index != NULL) timeout_remove(&to_index); + return destroyed; } static void ATTR_NULL(1) index_removal_timeout(void *context ATTR_UNUSED) { - destroy_unrefed(FALSE); + destroy_unrefed(0); } void mail_index_alloc_cache_unref(struct mail_index **_index) @@ -223,7 +243,7 @@ void mail_index_alloc_cache_destroy_unrefed(void) { - destroy_unrefed(TRUE); + destroy_unrefed(UINT_MAX); } void mail_index_alloc_cache_index_opened(struct mail_index *index) @@ -240,9 +260,29 @@ list->index_dir_dev = st.st_dev; } } - if (list != NULL && !list->referenced) { - /* keep it referenced for ourself */ - list->referenced = TRUE; - index->open_count++; +} + +void mail_index_alloc_cache_index_closing(struct mail_index *index) +{ + struct mail_index_alloc_cache_list *list = + MAIL_INDEX_ALLOC_CACHE_CONTEXT(index); + + i_assert(index->open_count > 0); + if (index->open_count > 1 || list == NULL) + return; + + if (list->referenced) { + /* we're closing our referenced index */ + return; } + while (indexes_cache_references_count > INDEX_CACHE_MAX) { + if (!destroy_unrefed(1)) { + /* our cache is full already, don't keep more */ + return; + } + } + /* keep the index referenced for caching */ + indexes_cache_references_count++; + list->referenced = TRUE; + index->open_count++; } diff -r 46e47e4082d8 -r 8944983786cc src/lib-index/mail-index-alloc-cache.h --- a/src/lib-index/mail-index-alloc-cache.h Thu Jul 03 16:07:09 2014 +0300 +++ b/src/lib-index/mail-index-alloc-cache.h Thu Jul 03 17:29:58 2014 +0300 @@ -11,5 +11,6 @@ /* internal: */ void mail_index_alloc_cache_index_opened(struct mail_index *index); +void mail_index_alloc_cache_index_closing(struct mail_index *index); #endif diff -r 46e47e4082d8 -r 8944983786cc src/lib-index/mail-index.c --- a/src/lib-index/mail-index.c Thu Jul 03 16:07:09 2014 +0300 +++ b/src/lib-index/mail-index.c Thu Jul 03 17:29:58 2014 +0300 @@ -631,6 +631,8 @@ void mail_index_close(struct mail_index *index) { i_assert(index->open_count > 0); + + mail_index_alloc_cache_index_closing(index); if (--index->open_count > 0) return; From dovecot at dovecot.org Thu Jul 3 14:46:06 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 14:46:06 +0000 Subject: dovecot-2.2: lib-index: Fixed error handling in mail_index_open() Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/092137bd092b changeset: 17579:092137bd092b user: Timo Sirainen date: Thu Jul 03 17:40:37 2014 +0300 description: lib-index: Fixed error handling in mail_index_open() diffstat: src/lib-index/mail-index.c | 24 +++++++++++++----------- 1 files changed, 13 insertions(+), 11 deletions(-) diffs (63 lines): diff -r 8944983786cc -r 092137bd092b src/lib-index/mail-index.c --- a/src/lib-index/mail-index.c Thu Jul 03 17:29:58 2014 +0300 +++ b/src/lib-index/mail-index.c Thu Jul 03 17:40:37 2014 +0300 @@ -26,6 +26,8 @@ struct mail_index_module_register mail_index_module_register = { 0 }; +static void mail_index_close_nonopened(struct mail_index *index); + struct mail_index *mail_index_alloc(const char *dir, const char *prefix) { struct mail_index *index; @@ -559,8 +561,6 @@ if (index->open_count > 0) { if ((ret = mail_index_open_opened(index, flags)) <= 0) { /* doesn't exist and create flag not used */ - index->open_count++; - mail_index_close(index); } return ret; } @@ -591,8 +591,7 @@ of the index files */ if ((ret = mail_index_open_files(index, flags)) <= 0) { /* doesn't exist and create flag not used */ - index->open_count++; - mail_index_close(index); + mail_index_close_nonopened(index); return ret; } index->open_count++; @@ -628,14 +627,8 @@ index->lock_type = F_UNLCK; } -void mail_index_close(struct mail_index *index) +static void mail_index_close_nonopened(struct mail_index *index) { - i_assert(index->open_count > 0); - - mail_index_alloc_cache_index_closing(index); - if (--index->open_count > 0) - return; - i_assert(!index->syncing); i_assert(index->views == NULL); @@ -652,6 +645,15 @@ index->indexid = 0; } +void mail_index_close(struct mail_index *index) +{ + i_assert(index->open_count > 0); + + mail_index_alloc_cache_index_closing(index); + if (--index->open_count == 0) + mail_index_close_nonopened(index); +} + int mail_index_unlink(struct mail_index *index) { const char *path; From dovecot at dovecot.org Thu Jul 3 14:46:06 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 14:46:06 +0000 Subject: dovecot-2.2: virtual: Never keep more than specified number of p... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/251747c3afe5 changeset: 17580:251747c3afe5 user: Timo Sirainen date: Thu Jul 03 17:44:32 2014 +0300 description: virtual: Never keep more than specified number of physical mailboxes open. This should make virtual mailboxes work for users who have a a ton of mailboxes with a ton of mails. Earlier code would likely have failed either with "Too many open files" or crashed with "Out of memory". You can change the max number of open mailboxes with: plugin { virtual_max_open_mailboxes = 64 } The default is 64. diffstat: src/plugins/virtual/virtual-mail.c | 4 +- src/plugins/virtual/virtual-save.c | 4 +- src/plugins/virtual/virtual-storage.c | 160 +++++++++++++++++++++++++++++++-- src/plugins/virtual/virtual-storage.h | 22 ++++ src/plugins/virtual/virtual-sync.c | 78 +++++++++------ 5 files changed, 221 insertions(+), 47 deletions(-) diffs (truncated from 534 to 300 lines): diff -r 092137bd092b -r 251747c3afe5 src/plugins/virtual/virtual-mail.c --- a/src/plugins/virtual/virtual-mail.c Thu Jul 03 17:40:37 2014 +0300 +++ b/src/plugins/virtual/virtual-mail.c Thu Jul 03 17:44:32 2014 +0300 @@ -108,11 +108,13 @@ bbox = virtual_backend_box_lookup(mbox, vmail->cur_vrec.mailbox_id); vmail->cur_backend_mail = backend_mail_find(vmail, bbox->box); if (vmail->cur_backend_mail == NULL) { - if (mailbox_open(bbox->box) < 0) { + if (virtual_backend_box_open(mbox, bbox) < 0) { virtual_box_copy_error(mail->box, bbox->box); return -1; } (void)virtual_mail_set_backend_mail(mail, bbox); + } else { + virtual_backend_box_accessed(mbox, bbox); } vmail->cur_lost = !mail_set_uid(vmail->cur_backend_mail, vmail->cur_vrec.real_uid); diff -r 092137bd092b -r 251747c3afe5 src/plugins/virtual/virtual-save.c --- a/src/plugins/virtual/virtual-save.c Thu Jul 03 17:40:37 2014 +0300 +++ b/src/plugins/virtual/virtual-save.c Thu Jul 03 17:44:32 2014 +0300 @@ -36,7 +36,7 @@ i_assert(ctx->backend_save_ctx == NULL); i_assert(ctx->open_errstr == NULL); - if (mailbox_open(mbox->save_bbox->box) < 0) { + if (virtual_backend_box_open(mbox, mbox->save_bbox) < 0) { errstr = mailbox_get_last_error(mbox->save_bbox->box, &ctx->open_error); ctx->open_errstr = i_strdup(errstr); @@ -45,6 +45,8 @@ virtual_transaction_get(_t, mbox->save_bbox->box); ctx->backend_save_ctx = mailbox_save_alloc(backend_trans); } + } else { + virtual_backend_box_accessed(mbox, mbox->save_bbox); } return _t->save_ctx; } diff -r 092137bd092b -r 251747c3afe5 src/plugins/virtual/virtual-storage.c --- a/src/plugins/virtual/virtual-storage.c Thu Jul 03 17:40:37 2014 +0300 +++ b/src/plugins/virtual/virtual-storage.c Thu Jul 03 17:44:32 2014 +0300 @@ -4,6 +4,7 @@ #include "array.h" #include "ioloop.h" #include "str.h" +#include "llist.h" #include "mkdir-parents.h" #include "unlink-directory.h" #include "index-mail.h" @@ -20,6 +21,8 @@ #include #include +#define VIRTUAL_DEFAULT_MAX_OPEN_MAILBOXES 64 + extern struct mail_storage virtual_storage; extern struct mailbox virtual_mailbox; extern struct virtual_mailbox_vfuncs virtual_mailbox_vfuncs; @@ -70,6 +73,24 @@ return &storage->storage; } +static int +virtual_storage_create(struct mail_storage *_storage, + struct mail_namespace *ns ATTR_UNUSED, + const char **error_r) +{ + struct virtual_storage *storage = (struct virtual_storage *)_storage; + const char *value; + + value = mail_user_plugin_getenv(_storage->user, "virtual_max_open_mailboxes"); + if (value == NULL) + storage->max_open_mailboxes = VIRTUAL_DEFAULT_MAX_OPEN_MAILBOXES; + else if (str_to_uint(value, &storage->max_open_mailboxes) < 0) { + *error_r = "Invalid virtual_max_open_mailboxes setting"; + return -1; + } + return 0; +} + static void virtual_storage_get_list_settings(const struct mail_namespace *ns ATTR_UNUSED, struct mailbox_list_settings *set) @@ -145,9 +166,9 @@ return -1; } -static int virtual_backend_box_open(struct virtual_mailbox *mbox, - struct virtual_backend_box *bbox, - enum mailbox_flags flags) +static int virtual_backend_box_alloc(struct virtual_mailbox *mbox, + struct virtual_backend_box *bbox, + enum mailbox_flags flags) { struct mail_user *user = mbox->storage->storage.user; struct mail_namespace *ns; @@ -190,7 +211,7 @@ bboxes = array_get(&mbox->backend_boxes, &count); for (i = 0; i < count; ) { - ret = virtual_backend_box_open(mbox, bboxes[i], flags); + ret = virtual_backend_box_alloc(mbox, bboxes[i], flags); if (ret <= 0) { if (ret < 0) break; @@ -236,6 +257,119 @@ return &mbox->box; } +void virtual_backend_box_sync_mail_unset(struct virtual_backend_box *bbox) +{ + struct mailbox_transaction_context *trans; + + if (bbox->sync_mail != NULL) { + trans = bbox->sync_mail->transaction; + mail_free(&bbox->sync_mail); + (void)mailbox_transaction_commit(&trans); + } +} + +static bool +virtual_backend_box_close_any_except(struct virtual_mailbox *mbox, + struct virtual_backend_box *except_bbox) +{ + struct virtual_backend_box *bbox; + + /* first try to close a mailbox without any transactions. + we'll also skip any mailbox that has notifications enabled (ideally + these would be handled by mailbox list index) */ + for (bbox = mbox->open_backend_boxes_head; bbox != NULL; bbox = bbox->next_open) { + i_assert(bbox->box->opened); + + if (bbox != except_bbox && + bbox->box->transaction_count == 0 && + bbox->box->notify_callback == NULL) { + i_assert(bbox->sync_mail == NULL); + virtual_backend_box_close(mbox, bbox); + return TRUE; + } + } + + /* next try to close a mailbox that has sync_mail, but no + other transactions */ + for (bbox = mbox->open_backend_boxes_head; bbox != NULL; bbox = bbox->next_open) { + if (bbox != except_bbox && + bbox->sync_mail != NULL && + bbox->box->transaction_count == 1 && + bbox->box->notify_callback == NULL) { + virtual_backend_box_sync_mail_unset(bbox); + i_assert(bbox->box->transaction_count == 0); + virtual_backend_box_close(mbox, bbox); + return TRUE; + } + } + return FALSE; +} + +void virtual_backend_box_opened(struct virtual_mailbox *mbox, + struct virtual_backend_box *bbox) +{ + /* the backend mailbox was already opened. if we didn't get here + from virtual_backend_box_open() we may need to close a mailbox */ + while (mbox->backends_open_count > mbox->storage->max_open_mailboxes && + virtual_backend_box_close_any_except(mbox, bbox)) + ; + + mbox->backends_open_count++; + DLLIST2_APPEND_FULL(&mbox->open_backend_boxes_head, + &mbox->open_backend_boxes_tail, bbox, + prev_open, next_open); +} + +int virtual_backend_box_open(struct virtual_mailbox *mbox, + struct virtual_backend_box *bbox) +{ + i_assert(!bbox->box->opened); + + /* try to keep the number of open mailboxes below the threshold + before opening the mailbox */ + while (mbox->backends_open_count >= mbox->storage->max_open_mailboxes && + virtual_backend_box_close_any_except(mbox, bbox)) + ; + + if (mailbox_open(bbox->box) < 0) + return -1; + virtual_backend_box_opened(mbox, bbox); + return 0; +} + +void virtual_backend_box_close(struct virtual_mailbox *mbox, + struct virtual_backend_box *bbox) +{ + i_assert(bbox->box->opened); + + if (bbox->search_result != NULL) + mailbox_search_result_free(&bbox->search_result); + + if (bbox->search_args != NULL && + bbox->search_args_initialized) { + mail_search_args_deinit(bbox->search_args); + bbox->search_args_initialized = FALSE; + } + i_assert(mbox->backends_open_count > 0); + mbox->backends_open_count--; + + DLLIST2_REMOVE_FULL(&mbox->open_backend_boxes_head, + &mbox->open_backend_boxes_tail, bbox, + prev_open, next_open); + mailbox_close(bbox->box); +} + +void virtual_backend_box_accessed(struct virtual_mailbox *mbox, + struct virtual_backend_box *bbox) +{ + DLLIST2_REMOVE_FULL(&mbox->open_backend_boxes_head, + &mbox->open_backend_boxes_tail, bbox, + prev_open, next_open); + DLLIST2_APPEND_FULL(&mbox->open_backend_boxes_head, + &mbox->open_backend_boxes_tail, bbox, + prev_open, next_open); +} + static void virtual_mailbox_close_internal(struct virtual_mailbox *mbox) { struct virtual_backend_box **bboxes; @@ -243,23 +377,18 @@ bboxes = array_get_modifiable(&mbox->backend_boxes, &count); for (i = 0; i < count; i++) { - if (bboxes[i]->search_result != NULL) - mailbox_search_result_free(&bboxes[i]->search_result); - if (bboxes[i]->box == NULL) continue; - if (bboxes[i]->search_args != NULL && - bboxes[i]->search_args_initialized) { - mail_search_args_deinit(bboxes[i]->search_args); - bboxes[i]->search_args_initialized = FALSE; - } + if (bboxes[i]->box->opened) + virtual_backend_box_close(mbox, bboxes[i]); mailbox_free(&bboxes[i]->box); if (array_is_created(&bboxes[i]->sync_outside_expunges)) array_free(&bboxes[i]->sync_outside_expunges); array_free(&bboxes[i]->sync_pending_removes); array_free(&bboxes[i]->uids); } + i_assert(mbox->backends_open_count == 0); } static int @@ -343,16 +472,19 @@ struct virtual_backend_box *const *bboxes; unsigned int i, count; struct mailbox_status status; + bool opened; mbox->have_guids = TRUE; mbox->have_save_guids = TRUE; bboxes = array_get(&mbox->backend_boxes, &count); for (i = 0; i < count; i++) { + opened = bboxes[i]->box->opened; if (mailbox_get_status(bboxes[i]->box, 0, &status) < 0) { virtual_box_copy_error(&mbox->box, bboxes[i]->box); return -1; } + i_assert(bboxes[i]->box->opened == opened); if (!status.have_guids) mbox->have_guids = FALSE; if (!status.have_save_guids) @@ -432,7 +564,7 @@ to wait for changes and avoid opening all mailboxes here. */ array_foreach(&mbox->backend_boxes, bboxp) { - if (mailbox_open((*bboxp)->box) < 0) { + if (virtual_backend_box_open(mbox, *bboxp) < 0) { /* we can't report error in here, so do it later */ (*bboxp)->open_failed = TRUE; continue; @@ -548,7 +680,7 @@ .v = { NULL, virtual_storage_alloc, - NULL, + virtual_storage_create, index_storage_destroy, NULL, virtual_storage_get_list_settings, diff -r 092137bd092b -r 251747c3afe5 src/plugins/virtual/virtual-storage.h --- a/src/plugins/virtual/virtual-storage.h Thu Jul 03 17:40:37 2014 +0300 +++ b/src/plugins/virtual/virtual-storage.h Thu Jul 03 17:44:32 2014 +0300 @@ -51,6 +51,8 @@ /* List of mailboxes while a virtual mailbox is being opened. Used to track loops. */ ARRAY_TYPE(const_string) open_stack; + + unsigned int max_open_mailboxes; From dovecot at dovecot.org Thu Jul 3 16:02:24 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 16:02:24 +0000 Subject: dovecot-2.2: lib-storage: Minor parsing fix to namespace types: ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/ee568c83319d changeset: 17581:ee568c83319d user: Timo Sirainen date: Thu Jul 03 19:00:46 2014 +0300 description: lib-storage: Minor parsing fix to namespace types: Use strcmp() instead of strncmp(). diffstat: src/lib-storage/mail-namespace.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diffs (19 lines): diff -r 251747c3afe5 -r ee568c83319d src/lib-storage/mail-namespace.c --- a/src/lib-storage/mail-namespace.c Thu Jul 03 17:44:32 2014 +0300 +++ b/src/lib-storage/mail-namespace.c Thu Jul 03 19:00:46 2014 +0300 @@ -94,12 +94,12 @@ ns = i_new(struct mail_namespace, 1); ns->refcount = 1; ns->user = user; - if (strncmp(ns_set->type, "private", 7) == 0) { + if (strcmp(ns_set->type, "private") == 0) { ns->owner = user; ns->type = MAIL_NAMESPACE_TYPE_PRIVATE; - } else if (strncmp(ns_set->type, "shared", 6) == 0) + } else if (strcmp(ns_set->type, "shared") == 0) ns->type = MAIL_NAMESPACE_TYPE_SHARED; - else if (strncmp(ns_set->type, "public", 6) == 0) + else if (strcmp(ns_set->type, "public") == 0) ns->type = MAIL_NAMESPACE_TYPE_PUBLIC; else { *error_r = t_strdup_printf("Unknown namespace type: %s", From dovecot at dovecot.org Thu Jul 3 16:05:29 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 16:05:29 +0000 Subject: dovecot-2.2: lib-storage: Move "shared" storage name to mail-sto... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/9bb653472f16 changeset: 17582:9bb653472f16 user: Timo Sirainen date: Thu Jul 03 19:03:59 2014 +0300 description: lib-storage: Move "shared" storage name to mail-storage-private.h where it can be used. diffstat: src/lib-storage/index/shared/shared-storage.c | 2 +- src/lib-storage/index/shared/shared-storage.h | 2 -- src/lib-storage/mail-namespace.c | 2 +- src/lib-storage/mail-storage-private.h | 2 ++ 4 files changed, 4 insertions(+), 4 deletions(-) diffs (48 lines): diff -r ee568c83319d -r 9bb653472f16 src/lib-storage/index/shared/shared-storage.c --- a/src/lib-storage/index/shared/shared-storage.c Thu Jul 03 19:00:46 2014 +0300 +++ b/src/lib-storage/index/shared/shared-storage.c Thu Jul 03 19:03:59 2014 +0300 @@ -356,7 +356,7 @@ } struct mail_storage shared_storage = { - .name = SHARED_STORAGE_NAME, + .name = MAIL_SHARED_STORAGE_NAME, .class_flags = 0, /* unknown at this point */ .v = { diff -r ee568c83319d -r 9bb653472f16 src/lib-storage/index/shared/shared-storage.h --- a/src/lib-storage/index/shared/shared-storage.h Thu Jul 03 19:00:46 2014 +0300 +++ b/src/lib-storage/index/shared/shared-storage.h Thu Jul 03 19:03:59 2014 +0300 @@ -1,8 +1,6 @@ #ifndef SHARED_STORAGE_H #define SHARED_STORAGE_H -#define SHARED_STORAGE_NAME "shared" - struct shared_storage { struct mail_storage storage; union mailbox_list_module_context list_module_ctx; diff -r ee568c83319d -r 9bb653472f16 src/lib-storage/mail-namespace.c --- a/src/lib-storage/mail-namespace.c Thu Jul 03 19:00:46 2014 +0300 +++ b/src/lib-storage/mail-namespace.c Thu Jul 03 19:03:59 2014 +0300 @@ -162,7 +162,7 @@ mixed %% usage, but still allows for specifying a shared namespace to an explicit location without any %% */ ns->flags |= NAMESPACE_FLAG_NOQUOTA | NAMESPACE_FLAG_NOACL; - driver = "shared"; + driver = MAIL_SHARED_STORAGE_NAME; } else { driver = NULL; } diff -r ee568c83319d -r 9bb653472f16 src/lib-storage/mail-storage-private.h --- a/src/lib-storage/mail-storage-private.h Thu Jul 03 19:00:46 2014 +0300 +++ b/src/lib-storage/mail-storage-private.h Thu Jul 03 19:03:59 2014 +0300 @@ -17,6 +17,8 @@ /* Block size when read()ing message (header and) body. */ #define MAIL_READ_FULL_BLOCK_SIZE IO_BLOCK_SIZE +#define MAIL_SHARED_STORAGE_NAME "shared" + struct mail_storage_module_register { unsigned int id; }; From dovecot at dovecot.org Thu Jul 3 16:11:59 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 16:11:59 +0000 Subject: dovecot-2.2: acl: Compile fix caused by previous change Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/ec22ade1f101 changeset: 17583:ec22ade1f101 user: Timo Sirainen date: Thu Jul 03 19:10:33 2014 +0300 description: acl: Compile fix caused by previous change diffstat: src/plugins/acl/acl-shared-storage.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 9bb653472f16 -r ec22ade1f101 src/plugins/acl/acl-shared-storage.c --- a/src/plugins/acl/acl-shared-storage.c Thu Jul 03 19:03:59 2014 +0300 +++ b/src/plugins/acl/acl-shared-storage.c Thu Jul 03 19:10:33 2014 +0300 @@ -89,7 +89,7 @@ const char *name; i_assert(ns->type == MAIL_NAMESPACE_TYPE_SHARED); - i_assert(strcmp(storage->name, SHARED_STORAGE_NAME) == 0); + i_assert(strcmp(storage->name, MAIL_SHARED_STORAGE_NAME) == 0); if (ioloop_time < alist->last_shared_add_check + SHARED_NS_RETRY_SECS) { /* already added, don't bother rechecking */ From dovecot at dovecot.org Thu Jul 3 16:13:39 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 16:13:39 +0000 Subject: dovecot-2.2: lib-storage: Added mail_namespace_is_shared_user_ro... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/7a60541913e9 changeset: 17584:7a60541913e9 user: Timo Sirainen date: Thu Jul 03 19:12:02 2014 +0300 description: lib-storage: Added mail_namespace_is_shared_user_root() and used it where useful. Most importantly this should fix a crash in ACL plugin where type=shared namespace was used without any kind of per-user prefix/location (i.e. it probably should have been a type=public namespace instead). diffstat: src/lib-storage/list/mailbox-list-subscriptions.c | 3 +-- src/lib-storage/mail-namespace.c | 21 +++++++++++++++++++-- src/lib-storage/mail-namespace.h | 5 +++++ src/plugins/acl/acl-mailbox.c | 3 +-- 4 files changed, 26 insertions(+), 6 deletions(-) diffs (74 lines): diff -r ec22ade1f101 -r 7a60541913e9 src/lib-storage/list/mailbox-list-subscriptions.c --- a/src/lib-storage/list/mailbox-list-subscriptions.c Thu Jul 03 19:10:33 2014 +0300 +++ b/src/lib-storage/list/mailbox-list-subscriptions.c Thu Jul 03 19:12:02 2014 +0300 @@ -61,8 +61,7 @@ /* When listing shared namespace's subscriptions, we need to autocreate all the visible child namespaces. their subscriptions are listed later. */ - if (ns != NULL && ns->type == MAIL_NAMESPACE_TYPE_SHARED && - (ns->flags & NAMESPACE_FLAG_AUTOCREATED) == 0) { + if (ns != NULL && mail_namespace_is_shared_user_root(ns)) { /* we'll need to get the namespace autocreated. one easy way is to just ask to join a reference and pattern */ diff -r ec22ade1f101 -r 7a60541913e9 src/lib-storage/mail-namespace.c --- a/src/lib-storage/mail-namespace.c Thu Jul 03 19:10:33 2014 +0300 +++ b/src/lib-storage/mail-namespace.c Thu Jul 03 19:12:02 2014 +0300 @@ -662,8 +662,7 @@ ns = mail_namespace_find_mask(namespaces, mailbox, 0, 0); i_assert(ns != NULL); - if (ns->type == MAIL_NAMESPACE_TYPE_SHARED && - (ns->flags & NAMESPACE_FLAG_AUTOCREATED) == 0) { + if (mail_namespace_is_shared_user_root(ns)) { /* see if we need to autocreate a namespace for shared user */ if (strchr(mailbox, mail_namespace_get_sep(ns)) != NULL) return mail_namespace_find_shared(ns, mailbox); @@ -751,3 +750,21 @@ } return NULL; } + +bool mail_namespace_is_shared_user_root(struct mail_namespace *ns) +{ + struct mail_storage *const *storagep; + + if (ns->type != MAIL_NAMESPACE_TYPE_SHARED) + return FALSE; + if ((ns->flags & NAMESPACE_FLAG_AUTOCREATED) != 0) { + /* child of the shared root */ + return FALSE; + } + /* if we have driver=shared storage, we're a real shared root */ + array_foreach(&ns->all_storages, storagep) { + if (strcmp((*storagep)->name, MAIL_SHARED_STORAGE_NAME) == 0) + return TRUE; + } + return FALSE; +} diff -r ec22ade1f101 -r 7a60541913e9 src/lib-storage/mail-namespace.h --- a/src/lib-storage/mail-namespace.h Thu Jul 03 19:10:33 2014 +0300 +++ b/src/lib-storage/mail-namespace.h Thu Jul 03 19:12:02 2014 +0300 @@ -150,4 +150,9 @@ void mail_namespace_finish_list_init(struct mail_namespace *ns, struct mailbox_list *list); +/* Returns TRUE if this is the root of a type=shared namespace that is actually + used for accessing shared users' mailboxes (as opposed to marking a + type=public namespace "wrong"). */ +bool mail_namespace_is_shared_user_root(struct mail_namespace *ns); + #endif diff -r ec22ade1f101 -r 7a60541913e9 src/plugins/acl/acl-mailbox.c --- a/src/plugins/acl/acl-mailbox.c Thu Jul 03 19:10:33 2014 +0300 +++ b/src/plugins/acl/acl-mailbox.c Thu Jul 03 19:12:02 2014 +0300 @@ -567,8 +567,7 @@ return; } - if (box->list->ns->type == MAIL_NAMESPACE_TYPE_SHARED && - (box->list->ns->flags & NAMESPACE_FLAG_AUTOCREATED) == 0) { + if (mail_namespace_is_shared_user_root(box->list->ns)) { /* this is the root shared namespace, which itself doesn't have any existing mailboxes. */ return; From dovecot at dovecot.org Thu Jul 3 16:19:13 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 16:19:13 +0000 Subject: dovecot-2.2: openssl: optionally disable TLS compression Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/cea292767b95 changeset: 17585:cea292767b95 user: Phil Carmody date: Thu Jul 03 19:17:16 2014 +0300 description: openssl: optionally disable TLS compression Make ssl compression optional, but enabled by default. Other ssl options might be tweakable in the future, so have a single ssl_options string, and explode it into individual flags. (Compare postfix configuration.) Based on an idea by Andreas Schulze Signed-off-by: Phil Carmody diffstat: src/lib-master/master-service-ssl-settings.c | 22 +++++++++++++++++++++- src/lib-master/master-service-ssl-settings.h | 7 +++++++ src/lib-master/master-service-ssl.c | 1 + src/lib-ssl-iostream/iostream-openssl-context.c | 8 ++++++-- src/lib-ssl-iostream/iostream-ssl.h | 1 + src/login-common/ssl-proxy-openssl.c | 10 +++++++--- 6 files changed, 43 insertions(+), 6 deletions(-) diffs (154 lines): diff -r 7a60541913e9 -r cea292767b95 src/lib-master/master-service-ssl-settings.c --- a/src/lib-master/master-service-ssl-settings.c Thu Jul 03 19:12:02 2014 +0300 +++ b/src/lib-master/master-service-ssl-settings.c Thu Jul 03 19:17:16 2014 +0300 @@ -28,6 +28,7 @@ DEF(SET_BOOL, ssl_require_crl), DEF(SET_BOOL, verbose_ssl), DEF(SET_BOOL, ssl_prefer_server_ciphers), + DEF(SET_STR, ssl_options), /* parsed as a string to set bools */ SETTING_DEFINE_LIST_END }; @@ -49,7 +50,8 @@ .ssl_verify_client_cert = FALSE, .ssl_require_crl = TRUE, .verbose_ssl = FALSE, - .ssl_prefer_server_ciphers = FALSE + .ssl_prefer_server_ciphers = FALSE, + .ssl_options = "", }; const struct setting_parser_info master_service_ssl_setting_parser_info = { @@ -98,6 +100,24 @@ *error_r = "ssl_verify_client_cert set, but ssl_ca not"; return FALSE; } + + /* Now explode the ssl_options string into individual flags */ + /* First set them all to defaults */ + set->parsed_opts.compression = TRUE; + + /* Then modify anything specified in the string */ + const char **opts = t_strsplit_spaces(set->ssl_options, ", "); + const char *opt; + while ((opt = *opts++) != NULL) { + if (strcasecmp(opt, "no_compression") == 0) { + set->parsed_opts.compression = FALSE; + } else { + *error_r = t_strdup_printf("ssl_options: unknown flag: '%s'", + opt); + return FALSE; + } + } + return TRUE; #endif } diff -r 7a60541913e9 -r cea292767b95 src/lib-master/master-service-ssl-settings.h --- a/src/lib-master/master-service-ssl-settings.h Thu Jul 03 19:12:02 2014 +0300 +++ b/src/lib-master/master-service-ssl-settings.h Thu Jul 03 19:17:16 2014 +0300 @@ -13,10 +13,17 @@ const char *ssl_protocols; const char *ssl_cert_username_field; const char *ssl_crypto_device; + const char *ssl_options; + bool ssl_verify_client_cert; bool ssl_require_crl; bool verbose_ssl; bool ssl_prefer_server_ciphers; + + /* These are derived from ssl_options, not set directly */ + struct { + bool compression; + } parsed_opts; }; extern const struct setting_parser_info master_service_ssl_setting_parser_info; diff -r 7a60541913e9 -r cea292767b95 src/lib-master/master-service-ssl.c --- a/src/lib-master/master-service-ssl.c Thu Jul 03 19:12:02 2014 +0300 +++ b/src/lib-master/master-service-ssl.c Thu Jul 03 19:17:16 2014 +0300 @@ -120,6 +120,7 @@ ssl_set.verbose = set->verbose_ssl; ssl_set.verify_remote_cert = set->ssl_verify_client_cert; ssl_set.prefer_server_ciphers = set->ssl_prefer_server_ciphers; + ssl_set.compression = set->parsed_opts.compression; if (ssl_iostream_context_init_server(&ssl_set, &service->ssl_ctx, &error) < 0) { diff -r 7a60541913e9 -r cea292767b95 src/lib-ssl-iostream/iostream-openssl-context.c --- a/src/lib-ssl-iostream/iostream-openssl-context.c Thu Jul 03 19:12:02 2014 +0300 +++ b/src/lib-ssl-iostream/iostream-openssl-context.c Thu Jul 03 19:17:16 2014 +0300 @@ -499,12 +499,16 @@ const struct ssl_iostream_settings *set, const char **error_r) { + long ssl_ops = SSL_OP_NO_SSLv2 | + (SSL_OP_ALL & ~SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS); + ctx->pool = pool_alloconly_create("ssl iostream context", 4096); /* enable all SSL workarounds, except empty fragments as it makes SSL more vulnerable against attacks */ - SSL_CTX_set_options(ctx->ssl_ctx, SSL_OP_NO_SSLv2 | - (SSL_OP_ALL & ~SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS)); + if (!set->compression) + ssl_ops |= SSL_OP_NO_COMPRESSION; + SSL_CTX_set_options(ctx->ssl_ctx, ssl_ops); #ifdef SSL_MODE_RELEASE_BUFFERS SSL_CTX_set_mode(ctx->ssl_ctx, SSL_MODE_RELEASE_BUFFERS); #endif diff -r 7a60541913e9 -r cea292767b95 src/lib-ssl-iostream/iostream-ssl.h --- a/src/lib-ssl-iostream/iostream-ssl.h Thu Jul 03 19:12:02 2014 +0300 +++ b/src/lib-ssl-iostream/iostream-ssl.h Thu Jul 03 19:17:16 2014 +0300 @@ -18,6 +18,7 @@ bool verify_remote_cert; /* neither/both */ bool require_valid_cert; /* stream-only */ bool prefer_server_ciphers; + bool compression; }; /* Returns 0 if ok, -1 and sets error_r if failed. The returned error string diff -r 7a60541913e9 -r cea292767b95 src/login-common/ssl-proxy-openssl.c --- a/src/login-common/ssl-proxy-openssl.c Thu Jul 03 19:12:02 2014 +0300 +++ b/src/login-common/ssl-proxy-openssl.c Thu Jul 03 19:17:16 2014 +0300 @@ -101,6 +101,7 @@ const char *protocols; bool verify_client_cert; bool prefer_server_ciphers; + bool compression; }; static int extdata_index; @@ -640,6 +641,7 @@ login_set->auth_ssl_require_client_cert || login_set->auth_ssl_username_from_cert; lookup_ctx.prefer_server_ciphers = set->ssl_prefer_server_ciphers; + lookup_ctx.compression = set->parsed_opts.compression; ctx = hash_table_lookup(ssl_servers, &lookup_ctx); if (ctx == NULL) @@ -1011,11 +1013,12 @@ { X509_STORE *store; STACK_OF(X509_NAME) *xnames = NULL; - /* enable all SSL workarounds, except empty fragments as it makes SSL more vulnerable against attacks */ - SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL & - ~SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS); + long ssl_ops = SSL_OP_ALL & ~SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS; + if (!set->parsed_opts.compression) + ssl_ops |= SSL_OP_NO_COMPRESSION; + SSL_CTX_set_options(ssl_ctx, ssl_ops); #ifdef SSL_MODE_RELEASE_BUFFERS SSL_CTX_set_mode(ssl_ctx, SSL_MODE_RELEASE_BUFFERS); @@ -1286,6 +1289,7 @@ login_set->auth_ssl_require_client_cert || login_set->auth_ssl_username_from_cert; ctx->prefer_server_ciphers = ssl_set->ssl_prefer_server_ciphers; + ctx->compression = ssl_set->parsed_opts.compression; ctx->ctx = ssl_ctx = SSL_CTX_new(SSLv23_server_method()); if (ssl_ctx == NULL) From dovecot at dovecot.org Thu Jul 3 16:29:15 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 16:29:15 +0000 Subject: dovecot-2.2: fts-lucene: Delay initialization to fix assert-cras... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/120b2a1645d8 changeset: 17586:120b2a1645d8 user: Timo Sirainen date: Thu Jul 03 19:27:45 2014 +0300 description: fts-lucene: Delay initialization to fix assert-crash with mbox diffstat: src/plugins/fts-lucene/fts-backend-lucene.c | 57 +++++++++++++++++++++------- 1 files changed, 42 insertions(+), 15 deletions(-) diffs (135 lines): diff -r cea292767b95 -r 120b2a1645d8 src/plugins/fts-lucene/fts-backend-lucene.c --- a/src/plugins/fts-lucene/fts-backend-lucene.c Thu Jul 03 19:17:16 2014 +0300 +++ b/src/plugins/fts-lucene/fts-backend-lucene.c Thu Jul 03 19:27:45 2014 +0300 @@ -126,14 +126,35 @@ return &backend->backend; } +static void fts_backend_lucene_real_init(struct lucene_fts_backend *backend) +{ + struct fts_lucene_user *fuser = + FTS_LUCENE_USER_CONTEXT(backend->backend.ns->user); + const char *path; + + if (backend->index != NULL) + return; + + /* initialize this path lazily, because with mbox format the get_path() + is overridden by the mbox code, but it hasn't had a chance to do + that yet in fts_backend_lucene_init(). */ + path = mailbox_list_get_root_forced(backend->backend.ns->list, + MAILBOX_LIST_PATH_TYPE_INDEX); + + backend->dir_path = i_strconcat(path, "/"LUCENE_INDEX_DIR_NAME, NULL); + backend->index = lucene_index_init(backend->dir_path, + backend->backend.ns->list, + &fuser->set); + + path = t_strconcat(backend->dir_path, "/"LUCENE_EXPUNGE_LOG_NAME, NULL); + backend->expunge_log = fts_expunge_log_init(path); +} + static int fts_backend_lucene_init(struct fts_backend *_backend, const char **error_r) { - struct lucene_fts_backend *backend = - (struct lucene_fts_backend *)_backend; struct fts_lucene_user *fuser = FTS_LUCENE_USER_CONTEXT(_backend->ns->user); - const char *path; if (fuser == NULL) { /* invalid settings */ @@ -142,15 +163,6 @@ } /* fts already checked that index exists */ - path = mailbox_list_get_root_forced(_backend->ns->list, - MAILBOX_LIST_PATH_TYPE_INDEX); - - backend->dir_path = i_strconcat(path, "/"LUCENE_INDEX_DIR_NAME, NULL); - backend->index = lucene_index_init(backend->dir_path, - _backend->ns->list, &fuser->set); - - path = t_strconcat(backend->dir_path, "/"LUCENE_EXPUNGE_LOG_NAME, NULL); - backend->expunge_log = fts_expunge_log_init(path); return 0; } @@ -159,8 +171,10 @@ struct lucene_fts_backend *backend = (struct lucene_fts_backend *)_backend; - lucene_index_deinit(backend->index); - fts_expunge_log_deinit(&backend->expunge_log); + if (backend->index != NULL) + lucene_index_deinit(backend->index); + if (backend->expunge_log != NULL) + fts_expunge_log_deinit(&backend->expunge_log); i_free(backend->dir_path); i_free(backend); } @@ -176,6 +190,8 @@ struct fts_index_header hdr; uint32_t set_checksum; + fts_backend_lucene_real_init(backend); + if (fts_index_get_header(box, &hdr)) { set_checksum = fts_lucene_settings_checksum(&fuser->set); if (!fts_index_have_compatible_settings(_backend->ns->list, @@ -210,6 +226,8 @@ i_assert(!backend->updating); + fts_backend_lucene_real_init(backend); + ctx = i_new(struct lucene_fts_backend_update_context, 1); ctx->ctx.backend = _backend; ctx->mime_parts = fuser->set.mime_parts; @@ -425,7 +443,8 @@ struct lucene_fts_backend *backend = (struct lucene_fts_backend *)_backend; - lucene_index_close(backend->index); + if (backend->index != NULL) + lucene_index_close(backend->index); return 0; } @@ -434,6 +453,8 @@ struct lucene_fts_backend *backend = (struct lucene_fts_backend *)_backend; + fts_backend_lucene_real_init(backend); + if (lucene_index_rescan(backend->index) < 0) return -1; return lucene_index_optimize(backend->index); @@ -445,6 +466,8 @@ (struct lucene_fts_backend *)_backend; int ret; + fts_backend_lucene_real_init(backend); + ret = lucene_index_expunge_from_log(backend->index, backend->expunge_log); if (ret == 0) { @@ -465,6 +488,8 @@ (struct lucene_fts_backend *)_backend; int ret; + fts_backend_lucene_real_init(backend); + if (fts_backend_select(backend, box) < 0) return -1; T_BEGIN { @@ -532,6 +557,8 @@ (struct lucene_fts_backend *)_backend; int ret; + fts_backend_lucene_real_init(backend); + T_BEGIN { HASH_TABLE_TYPE(wguid_result) guids; From dovecot at dovecot.org Thu Jul 3 16:36:33 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 16:36:33 +0000 Subject: dovecot-2.2: Compile fix for old systems without SSL_OP_NO_COMPR... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/4c4db2d76137 changeset: 17587:4c4db2d76137 user: Timo Sirainen date: Thu Jul 03 19:34:57 2014 +0300 description: Compile fix for old systems without SSL_OP_NO_COMPRESSION diffstat: src/lib-ssl-iostream/iostream-openssl-context.c | 2 ++ src/login-common/ssl-proxy-openssl.c | 3 +++ 2 files changed, 5 insertions(+), 0 deletions(-) diffs (29 lines): diff -r 120b2a1645d8 -r 4c4db2d76137 src/lib-ssl-iostream/iostream-openssl-context.c --- a/src/lib-ssl-iostream/iostream-openssl-context.c Thu Jul 03 19:27:45 2014 +0300 +++ b/src/lib-ssl-iostream/iostream-openssl-context.c Thu Jul 03 19:34:57 2014 +0300 @@ -506,8 +506,10 @@ /* enable all SSL workarounds, except empty fragments as it makes SSL more vulnerable against attacks */ +#ifdef SSL_OP_NO_COMPRESSION if (!set->compression) ssl_ops |= SSL_OP_NO_COMPRESSION; +#endif SSL_CTX_set_options(ctx->ssl_ctx, ssl_ops); #ifdef SSL_MODE_RELEASE_BUFFERS SSL_CTX_set_mode(ctx->ssl_ctx, SSL_MODE_RELEASE_BUFFERS); diff -r 120b2a1645d8 -r 4c4db2d76137 src/login-common/ssl-proxy-openssl.c --- a/src/login-common/ssl-proxy-openssl.c Thu Jul 03 19:27:45 2014 +0300 +++ b/src/login-common/ssl-proxy-openssl.c Thu Jul 03 19:34:57 2014 +0300 @@ -1016,8 +1016,11 @@ /* enable all SSL workarounds, except empty fragments as it makes SSL more vulnerable against attacks */ long ssl_ops = SSL_OP_ALL & ~SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS; + +#ifdef SSL_OP_NO_COMPRESSION if (!set->parsed_opts.compression) ssl_ops |= SSL_OP_NO_COMPRESSION; +#endif SSL_CTX_set_options(ssl_ctx, ssl_ops); #ifdef SSL_MODE_RELEASE_BUFFERS From dovecot at dovecot.org Thu Jul 3 17:29:55 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 17:29:55 +0000 Subject: dovecot-2.2: lmtp: Removed code that attempts to deduplicate mai... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/51274bf2a47d changeset: 17588:51274bf2a47d user: Timo Sirainen date: Thu Jul 03 20:28:16 2014 +0300 description: lmtp: Removed code that attempts to deduplicate mail files by copying them between user mailboxes. This sometimes started failing if the mail that was being used for copying was deleted by the user. There's no good way for lmtp code to fix that situation. If deduplication is needed, it could be implemented in a more generic way inside mailbox_copy() where after initial copy it would store the destination struct mail to src_mail->last_copy_dest_mail. If another mail is copied, the last_copy_dest_mail could be attempted to be used for the copying and if that doesn't work it would fallback to regular copying. This should probably be attempted only for lda/lmtp processes as it would just cause extra overhead for others. diffstat: src/lmtp/client.h | 3 -- src/lmtp/commands.c | 69 ++++++++++------------------------------------------ 2 files changed, 14 insertions(+), 58 deletions(-) diffs (171 lines): diff -r 4c4db2d76137 -r 51274bf2a47d src/lmtp/client.h --- a/src/lmtp/client.h Thu Jul 03 19:34:57 2014 +0300 +++ b/src/lmtp/client.h Thu Jul 03 20:28:16 2014 +0300 @@ -29,9 +29,6 @@ struct mail *raw_mail; - struct mail_user *dest_user; - struct mail *first_saved_mail; - unsigned int mail_body_7bit:1; unsigned int mail_body_8bitmime:1; }; diff -r 4c4db2d76137 -r 51274bf2a47d src/lmtp/commands.c --- a/src/lmtp/commands.c Thu Jul 03 19:34:57 2014 +0300 +++ b/src/lmtp/commands.c Thu Jul 03 20:28:16 2014 +0300 @@ -614,7 +614,7 @@ static int client_deliver(struct client *client, const struct mail_recipient *rcpt, - struct mail *src_mail, struct mail_deliver_session *session) + struct mail_deliver_session *session) { struct mail_deliver_context dctx; struct mail_storage *storage; @@ -622,6 +622,7 @@ const struct mail_storage_settings *mail_set; struct lda_settings *lda_set; struct mail_namespace *ns; + struct mail_user *dest_user; struct setting_parser_context *set_parser; void **sets; const char *line, *error, *username; @@ -649,7 +650,7 @@ i_set_failure_prefix("lmtp(%s, %s): ", my_pid, username); if (mail_storage_service_next(storage_service, rcpt->service_user, - &client->state.dest_user) < 0) { + &dest_user) < 0) { client_send_line(client, ERRSTR_TEMP_MAILBOX_FAIL, rcpt->address); return -1; @@ -657,18 +658,18 @@ sets = mail_storage_service_user_get_set(rcpt->service_user); lda_set = sets[1]; settings_var_expand(&lda_setting_parser_info, lda_set, client->pool, - mail_user_var_expand_table(client->state.dest_user)); + mail_user_var_expand_table(dest_user)); memset(&dctx, 0, sizeof(dctx)); dctx.session = session; dctx.pool = session->pool; dctx.set = lda_set; dctx.session_id = client->state.session_id; - dctx.src_mail = src_mail; + dctx.src_mail = client->state.raw_mail; dctx.src_envelope_sender = client->state.mail_from; - dctx.dest_user = client->state.dest_user; + dctx.dest_user = dest_user; if (*dctx.set->lda_original_recipient_header != '\0') { - dctx.dest_addr = mail_deliver_get_address(src_mail, + dctx.dest_addr = mail_deliver_get_address(dctx.src_mail, dctx.set->lda_original_recipient_header); } if (dctx.dest_addr == NULL) @@ -678,19 +679,12 @@ !client->lmtp_set->lmtp_save_to_detail_mailbox) dctx.dest_mailbox_name = "INBOX"; else { - ns = mail_namespace_find_inbox(dctx.dest_user->namespaces); + ns = mail_namespace_find_inbox(dest_user->namespaces); dctx.dest_mailbox_name = t_strconcat(ns->prefix, rcpt->detail, NULL); } - dctx.save_dest_mail = array_count(&client->state.rcpt_to) > 1 && - client->state.first_saved_mail == NULL; - if (mail_deliver(&dctx, &storage) == 0) { - if (dctx.dest_mail != NULL) { - i_assert(client->state.first_saved_mail == NULL); - client->state.first_saved_mail = dctx.dest_mail; - } client_send_line(client, "250 2.0.0 <%s> %s Saved", rcpt->address, client->state.session_id); ret = 0; @@ -717,10 +711,11 @@ rcpt->address); ret = -1; } + mail_user_unref(&dest_user); return ret; } -static bool client_deliver_next(struct client *client, struct mail *src_mail, +static bool client_deliver_next(struct client *client, struct mail_deliver_session *session) { const struct mail_recipient *rcpts; @@ -730,15 +725,13 @@ rcpts = array_get(&client->state.rcpt_to, &count); while (client->state.rcpt_idx < count) { ret = client_deliver(client, &rcpts[client->state.rcpt_idx], - src_mail, session); + session); i_set_failure_prefix("lmtp(%s): ", my_pid); client->state.rcpt_idx++; if (ret == 0) return TRUE; /* failed. try the next one. */ - if (client->state.dest_user != NULL) - mail_user_unref(&client->state.dest_user); } return FALSE; } @@ -815,51 +808,17 @@ client_input_data_write_local(struct client *client, struct istream *input) { struct mail_deliver_session *session; - struct mail *src_mail; - uid_t old_uid, first_uid = (uid_t)-1; + uid_t old_uid; if (client_open_raw_mail(client, input) < 0) return; session = mail_deliver_session_init(); old_uid = geteuid(); - src_mail = client->state.raw_mail; - while (client_deliver_next(client, src_mail, session)) { - if (client->state.first_saved_mail == NULL || - client->state.first_saved_mail == src_mail) - mail_user_unref(&client->state.dest_user); - else { - /* use the first saved message to save it elsewhere too. - this might allow hard linking the files. */ - client->state.dest_user = NULL; - src_mail = client->state.first_saved_mail; - first_uid = geteuid(); - i_assert(first_uid != 0); - } - } + while (client_deliver_next(client, session)) + ; mail_deliver_session_deinit(&session); - if (client->state.first_saved_mail != NULL) { - struct mail *mail = client->state.first_saved_mail; - struct mailbox_transaction_context *trans = mail->transaction; - struct mailbox *box = trans->box; - struct mail_user *user = box->storage->user; - - /* just in case these functions are going to write anything, - change uid back to user's own one */ - if (first_uid != old_uid) { - if (seteuid(0) < 0) - i_fatal("seteuid(0) failed: %m"); - if (seteuid(first_uid) < 0) - i_fatal("seteuid() failed: %m"); - } - - mail_free(&mail); - mailbox_transaction_rollback(&trans); - mailbox_free(&box); - mail_user_unref(&user); - } - if (old_uid == 0) { /* switch back to running as root, since that's what we're practically doing anyway. it's also important in case we From dovecot at dovecot.org Thu Jul 3 17:43:37 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 17:43:37 +0000 Subject: dovecot-2.2: acl: Create struct acl_mailbox also for shared root... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/5f21c78cd7b2 changeset: 17589:5f21c78cd7b2 user: Timo Sirainen date: Thu Jul 03 20:42:08 2014 +0300 description: acl: Create struct acl_mailbox also for shared root namespace mailboxes. This fixes crashes where imap_acl code attempts to access ACLs for nonexistent mailboxes inside shared root namespace. Alternatively the imap_acl plugin could have checked the nonexistence of ACLs but this is probably easier and more guaranteed to work. diffstat: src/plugins/acl/acl-mailbox.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diffs (29 lines): diff -r 51274bf2a47d -r 5f21c78cd7b2 src/plugins/acl/acl-mailbox.c --- a/src/plugins/acl/acl-mailbox.c Thu Jul 03 20:28:16 2014 +0300 +++ b/src/plugins/acl/acl-mailbox.c Thu Jul 03 20:42:08 2014 +0300 @@ -561,6 +561,7 @@ struct acl_mailbox_list *alist = ACL_LIST_CONTEXT(box->list); struct mailbox_vfuncs *v = box->vlast; struct acl_mailbox *abox; + bool ignore_acls = (box->flags & MAILBOX_FLAG_IGNORE_ACLS) != 0; if (alist == NULL) { /* ACLs disabled */ @@ -570,7 +571,7 @@ if (mail_namespace_is_shared_user_root(box->list->ns)) { /* this is the root shared namespace, which itself doesn't have any existing mailboxes. */ - return; + ignore_acls = TRUE; } abox = p_new(box->pool, struct acl_mailbox, 1); @@ -582,7 +583,7 @@ mailbox_get_name(box)); v->free = acl_mailbox_free; - if ((box->flags & MAILBOX_FLAG_IGNORE_ACLS) == 0) { + if (ignore_acls) { abox->acl_enabled = TRUE; v->is_readonly = acl_is_readonly; v->exists = acl_mailbox_exists; From dovecot at dovecot.org Thu Jul 3 18:57:10 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 18:57:10 +0000 Subject: dovecot-2.2: lib: Added o_stream_flush_parent_if_needed() for wr... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/5740cb57d02c changeset: 17590:5740cb57d02c user: Timo Sirainen date: Thu Jul 03 21:54:52 2014 +0300 description: lib: Added o_stream_flush_parent_if_needed() for wrapper ostreams. diffstat: src/lib/ostream-private.h | 4 ++++ src/lib/ostream.c | 17 +++++++++++++++++ 2 files changed, 21 insertions(+), 0 deletions(-) diffs (40 lines): diff -r 5f21c78cd7b2 -r 5740cb57d02c src/lib/ostream-private.h --- a/src/lib/ostream-private.h Thu Jul 03 20:42:08 2014 +0300 +++ b/src/lib/ostream-private.h Thu Jul 03 21:54:52 2014 +0300 @@ -50,5 +50,9 @@ size_t block_size); void o_stream_copy_error_from_parent(struct ostream_private *_stream); +/* This should be called before sending data to parent stream. It makes sure + that the parent stream's output buffer doesn't become too large. + Returns 1 if more data can be safely added, 0 if not, -1 if error. */ +int o_stream_flush_parent_if_needed(struct ostream_private *_stream); #endif diff -r 5f21c78cd7b2 -r 5740cb57d02c src/lib/ostream.c --- a/src/lib/ostream.c Thu Jul 03 20:42:08 2014 +0300 +++ b/src/lib/ostream.c Thu Jul 03 21:54:52 2014 +0300 @@ -451,6 +451,23 @@ o_stream_close(dest); } +int o_stream_flush_parent_if_needed(struct ostream_private *_stream) +{ + if (o_stream_get_buffer_used_size(_stream->parent) >= IO_BLOCK_SIZE) { + /* we already have quite a lot of data in parent stream. + unless we can flush it, don't add any more to it or we + could keep wasting memory by just increasing the buffer + size all the time. */ + if (o_stream_flush(_stream->parent) < 0) { + o_stream_copy_error_from_parent(_stream); + return -1; + } + if (o_stream_get_buffer_used_size(_stream->parent) >= IO_BLOCK_SIZE) + return 0; + } + return 1; +} + static int o_stream_default_flush(struct ostream_private *_stream) { int ret; From dovecot at dovecot.org Thu Jul 3 18:57:11 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 18:57:11 +0000 Subject: dovecot-2.2: lib-compression: Compression ostreams may have caus... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/00b3aa3842b7 changeset: 17591:00b3aa3842b7 user: Timo Sirainen date: Thu Jul 03 21:55:31 2014 +0300 description: lib-compression: Compression ostreams may have caused parent ostream to use too much memory. diffstat: src/lib-compression/ostream-bzlib.c | 2 ++ src/lib-compression/ostream-lzma.c | 2 ++ src/lib-compression/ostream-zlib.c | 3 +++ 3 files changed, 7 insertions(+), 0 deletions(-) diffs (37 lines): diff -r 5740cb57d02c -r 00b3aa3842b7 src/lib-compression/ostream-bzlib.c --- a/src/lib-compression/ostream-bzlib.c Thu Jul 03 21:54:52 2014 +0300 +++ b/src/lib-compression/ostream-bzlib.c Thu Jul 03 21:55:31 2014 +0300 @@ -113,6 +113,8 @@ if (zstream->flushed) return 0; + if ((ret = o_stream_flush_parent_if_needed(&zstream->ostream)) <= 0) + return ret; if ((ret = o_stream_zlib_send_outbuf(zstream)) <= 0) return ret; diff -r 5740cb57d02c -r 00b3aa3842b7 src/lib-compression/ostream-lzma.c --- a/src/lib-compression/ostream-lzma.c Thu Jul 03 21:54:52 2014 +0300 +++ b/src/lib-compression/ostream-lzma.c Thu Jul 03 21:55:31 2014 +0300 @@ -119,6 +119,8 @@ if (zstream->flushed) return 0; + if ((ret = o_stream_flush_parent_if_needed(&zstream->ostream)) <= 0) + return ret; if ((ret = o_stream_zlib_send_outbuf(zstream)) <= 0) return ret; diff -r 5740cb57d02c -r 00b3aa3842b7 src/lib-compression/ostream-zlib.c --- a/src/lib-compression/ostream-zlib.c Thu Jul 03 21:54:52 2014 +0300 +++ b/src/lib-compression/ostream-zlib.c Thu Jul 03 21:55:31 2014 +0300 @@ -181,6 +181,9 @@ if (zstream->flushed) return 0; + + if ((ret = o_stream_flush_parent_if_needed(&zstream->ostream)) <= 0) + return ret; if (!zstream->header_sent) { if (o_stream_zlib_send_gz_header(zstream) < 0) return -1; From dovecot at dovecot.org Thu Jul 3 19:18:33 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 19:18:33 +0000 Subject: dovecot-2.2: acl: Oops, ignore_acls check was reversed. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/a372391290a3 changeset: 17592:a372391290a3 user: Timo Sirainen date: Thu Jul 03 22:16:48 2014 +0300 description: acl: Oops, ignore_acls check was reversed. diffstat: src/plugins/acl/acl-mailbox.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 00b3aa3842b7 -r a372391290a3 src/plugins/acl/acl-mailbox.c --- a/src/plugins/acl/acl-mailbox.c Thu Jul 03 21:55:31 2014 +0300 +++ b/src/plugins/acl/acl-mailbox.c Thu Jul 03 22:16:48 2014 +0300 @@ -583,7 +583,7 @@ mailbox_get_name(box)); v->free = acl_mailbox_free; - if (ignore_acls) { + if (!ignore_acls) { abox->acl_enabled = TRUE; v->is_readonly = acl_is_readonly; v->exists = acl_mailbox_exists; From dovecot at dovecot.org Thu Jul 3 19:19:20 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 19:19:20 +0000 Subject: dovecot-2.2: acl: Global ACLs now support "patterns with spaces ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/40dc5774aa69 changeset: 17593:40dc5774aa69 user: Timo Sirainen date: Thu Jul 03 22:17:49 2014 +0300 description: acl: Global ACLs now support "patterns with spaces inside quotes" diffstat: src/plugins/acl/acl-global-file.c | 73 ++++++++++++++++++++++++++++---------- 1 files changed, 53 insertions(+), 20 deletions(-) diffs (124 lines): diff -r a372391290a3 -r 40dc5774aa69 src/plugins/acl/acl-global-file.c --- a/src/plugins/acl/acl-global-file.c Thu Jul 03 22:16:48 2014 +0300 +++ b/src/plugins/acl/acl-global-file.c Thu Jul 03 22:17:49 2014 +0300 @@ -4,6 +4,7 @@ #include "array.h" #include "ioloop.h" #include "istream.h" +#include "strescape.h" #include "wildcard-match.h" #include "acl-api-private.h" #include "acl-global-file.h" @@ -62,20 +63,65 @@ return strcmp(r1->vpattern, r2->vpattern); } +struct acl_global_file_parse_ctx { + struct acl_global_file *file; + ARRAY(struct acl_global_parse_rights) parse_rights; +}; + +static int +acl_global_file_parse_line(struct acl_global_file_parse_ctx *ctx, + const char *line, const char **error_r) +{ + struct acl_global_parse_rights *pright; + const char *p, *vpattern; + + if (*line == '"') { + line++; + if (str_unescape_next(&line, &vpattern) < 0) { + *error_r = "Missing '\"'"; + return -1; + } + if (line[0] != ' ') { + *error_r = "Expecting space after '\"'"; + return -1; + } + line++; + } else { + p = strchr(line, ' '); + if (p == NULL) { + *error_r = "Missing ACL rights"; + return -1; + } + if (p == line) { + *error_r = "Empty ACL pattern"; + return -1; + } + vpattern = t_strdup_until(line, p); + line = p + 1; + } + + pright = array_append_space(&ctx->parse_rights); + pright->vpattern = p_strdup(ctx->file->rights_pool, vpattern); + return acl_rights_parse_line(line, ctx->file->rights_pool, + &pright->rights, error_r); +} + static int acl_global_file_read(struct acl_global_file *file) { - ARRAY(struct acl_global_parse_rights) parse_rights; + struct acl_global_file_parse_ctx ctx; struct acl_global_parse_rights *pright; struct acl_global_rights *right; struct istream *input; - const char *line, *p, *error, *prev_vpattern; + const char *line, *error, *prev_vpattern; unsigned int linenum = 0; int ret = 0; array_clear(&file->rights); p_clear(file->rights_pool); - i_array_init(&parse_rights, 32); + memset(&ctx, 0, sizeof(ctx)); + ctx.file = file; + i_array_init(&ctx.parse_rights, 32); input = i_stream_create_file(file->path, (size_t)-1); while ((line = i_stream_read_next_line(input)) != NULL) { @@ -83,20 +129,7 @@ if (line[0] == '\0' || line[0] == '#') continue; T_BEGIN { - p = strchr(line, ' '); - if (p == NULL) { - error = "Missing ACL rights"; - ret = -1; - } else if (p == line) { - error = "Empty ACL pattern"; - ret = -1; - } else { - pright = array_append_space(&parse_rights); - pright->vpattern = p_strdup_until(file->rights_pool, - line, p++); - ret = acl_rights_parse_line(p, file->rights_pool, - &pright->rights, &error); - } + ret = acl_global_file_parse_line(&ctx, line, &error); if (ret < 0) { i_error("Global ACL file %s line %u: %s", file->path, linenum, error); @@ -113,10 +146,10 @@ i_stream_destroy(&input); /* sort all parsed rights */ - array_sort(&parse_rights, acl_global_parse_rights_cmp); + array_sort(&ctx.parse_rights, acl_global_parse_rights_cmp); /* combine identical patterns into same structs */ prev_vpattern = ""; right = NULL; - array_foreach_modifiable(&parse_rights, pright) { + array_foreach_modifiable(&ctx.parse_rights, pright) { if (right == NULL || strcmp(prev_vpattern, pright->vpattern) != 0) { right = array_append_space(&file->rights); @@ -126,7 +159,7 @@ array_append(&right->rights, &pright->rights, 1); } - array_free(&parse_rights); + array_free(&ctx.parse_rights); return ret; } From dovecot at dovecot.org Thu Jul 3 19:25:35 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 03 Jul 2014 19:25:35 +0000 Subject: dovecot-2.2: mbox: If GUID lookup fails because mbox_min_index_s... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/0343869d47cf changeset: 17594:0343869d47cf user: Timo Sirainen date: Thu Jul 03 22:24:05 2014 +0300 description: mbox: If GUID lookup fails because mbox_min_index_size skipped indexes, say so in error message. diffstat: src/lib-storage/index/mbox/mbox-storage.c | 11 +++++++++-- 1 files changed, 9 insertions(+), 2 deletions(-) diffs (22 lines): diff -r 40dc5774aa69 -r 0343869d47cf src/lib-storage/index/mbox/mbox-storage.c --- a/src/lib-storage/index/mbox/mbox-storage.c Thu Jul 03 22:17:49 2014 +0300 +++ b/src/lib-storage/index/mbox/mbox-storage.c Thu Jul 03 22:24:05 2014 +0300 @@ -618,9 +618,16 @@ static int mbox_mailbox_get_guid(struct mbox_mailbox *mbox, guid_128_t guid_r) { + const char *errstr; + if (mail_index_is_in_memory(mbox->box.index)) { - mail_storage_set_error(mbox->box.storage, MAIL_ERROR_NOTPOSSIBLE, - "Mailbox GUIDs are not permanent without index files"); + errstr = "Mailbox GUIDs are not permanent without index files"; + if (mbox->storage->set->mbox_min_index_size != 0) { + errstr = t_strconcat(errstr, + " (mbox_min_index_size is non-zero)", NULL); + } + mail_storage_set_error(mbox->box.storage, + MAIL_ERROR_NOTPOSSIBLE, errstr); return -1; } if (mbox_sync_header_refresh(mbox) < 0) From dovecot at dovecot.org Fri Jul 4 08:50:07 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 04 Jul 2014 08:50:07 +0000 Subject: dovecot-2.2: lib-dict: Removed NFS flushing from dict-file backend. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/b2f3b9f5d7cc changeset: 17595:b2f3b9f5d7cc user: Timo Sirainen date: Fri Jul 04 11:48:27 2014 +0300 description: lib-dict: Removed NFS flushing from dict-file backend. diffstat: src/lib-dict/dict-file.c | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diffs (16 lines): diff -r 0343869d47cf -r b2f3b9f5d7cc src/lib-dict/dict-file.c --- a/src/lib-dict/dict-file.c Thu Jul 03 22:24:05 2014 +0300 +++ b/src/lib-dict/dict-file.c Fri Jul 04 11:48:27 2014 +0300 @@ -111,7 +111,11 @@ if (dict->fd == -1) return TRUE; - nfs_flush_file_handle_cache(dict->path); + /* Disable NFS flushing for now since it can cause unnecessary + problems and there's no easy way for us to know here if + mail_nfs_storage=yes. In any case it's pretty much an unsupported + setting nowadays. */ + /*nfs_flush_file_handle_cache(dict->path);*/ if (nfs_safe_stat(dict->path, &st1) < 0) { i_error("stat(%s) failed: %m", dict->path); return FALSE; From dovecot at dovecot.org Fri Jul 4 10:18:29 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 04 Jul 2014 10:18:29 +0000 Subject: dovecot-2.2: lib-storage: If mail body reading failed, the error... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/9f482e0fc965 changeset: 17596:9f482e0fc965 user: Timo Sirainen date: Fri Jul 04 13:16:01 2014 +0300 description: lib-storage: If mail body reading failed, the error message may have contained only minimal errno string. Even though the istream could have had a much better internal error message. So show it. diffstat: src/lib-storage/index/index-mail.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diffs (16 lines): diff -r b2f3b9f5d7cc -r 9f482e0fc965 src/lib-storage/index/index-mail.c --- a/src/lib-storage/index/index-mail.c Fri Jul 04 11:48:27 2014 +0300 +++ b/src/lib-storage/index/index-mail.c Fri Jul 04 13:16:01 2014 +0300 @@ -898,8 +898,10 @@ errno = mail->data.stream->stream_errno; mail_storage_set_critical(mail->mail.mail.box->storage, - "read(%s) failed: %m (uid=%u)", - i_stream_get_name(mail->data.stream), mail->mail.mail.uid); + "read(%s) failed: %s (uid=%u)", + i_stream_get_name(mail->data.stream), + i_stream_get_error(mail->data.stream), + mail->mail.mail.uid); return -1; } From dovecot at dovecot.org Fri Jul 4 10:18:29 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 04 Jul 2014 10:18:29 +0000 Subject: dovecot-2.2: When creating istream-error, give an error string w... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/82c9a9b7e6e6 changeset: 17597:82c9a9b7e6e6 user: Timo Sirainen date: Fri Jul 04 13:16:59 2014 +0300 description: When creating istream-error, give an error string whenever possible. diffstat: src/lib-fs/fs-posix.c | 2 +- src/lib-imap-storage/imap-msgpart.c | 2 +- src/lib/iostream-temp.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diffs (36 lines): diff -r 9f482e0fc965 -r 82c9a9b7e6e6 src/lib-fs/fs-posix.c --- a/src/lib-fs/fs-posix.c Fri Jul 04 13:16:01 2014 +0300 +++ b/src/lib-fs/fs-posix.c Fri Jul 04 13:16:59 2014 +0300 @@ -349,7 +349,7 @@ struct istream *input; if (file->fd == -1 && fs_posix_open(file) < 0) - input = i_stream_create_error(errno); + input = i_stream_create_error_str(errno, "%s", fs_last_error(_file->fs)); else input = i_stream_create_fd(file->fd, max_buffer_size, FALSE); i_stream_set_name(input, _file->path); diff -r 9f482e0fc965 -r 82c9a9b7e6e6 src/lib-imap-storage/imap-msgpart.c --- a/src/lib-imap-storage/imap-msgpart.c Fri Jul 04 13:16:01 2014 +0300 +++ b/src/lib-imap-storage/imap-msgpart.c Fri Jul 04 13:16:59 2014 +0300 @@ -400,7 +400,7 @@ virtual_skip -= cache->virtual_pos; } if (message_skip_virtual(input, virtual_skip, &cr_skipped) < 0) { - errinput = i_stream_create_error(errno); + errinput = i_stream_create_error_str(errno, "%s", i_stream_get_error(input)); i_stream_set_name(errinput, i_stream_get_name(input)); i_stream_unref(&input); return errinput; diff -r 9f482e0fc965 -r 82c9a9b7e6e6 src/lib/iostream-temp.c --- a/src/lib/iostream-temp.c Fri Jul 04 13:16:01 2014 +0300 +++ b/src/lib/iostream-temp.c Fri Jul 04 13:16:59 2014 +0300 @@ -255,7 +255,7 @@ tstream->dupstream_start_offset; fd = dup(i_stream_get_fd(tstream->dupstream)); if (fd == -1) - input = i_stream_create_error(errno); + input = i_stream_create_error_str(errno, "dup() failed: %m"); else { input2 = i_stream_create_fd_autoclose(&fd, max_buffer_size); i_stream_seek(input2, abs_offset); From dovecot at dovecot.org Fri Jul 4 11:03:32 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 04 Jul 2014 11:03:32 +0000 Subject: dovecot-2.2: lib-storage: Log mail istream read failures in one ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/0902eded882f changeset: 17598:0902eded882f user: Timo Sirainen date: Fri Jul 04 14:01:53 2014 +0300 description: lib-storage: Log mail istream read failures in one place. Also handle ENOENT errors by checking if the mail has already been expunged, and if so don't log an error, just return "mail is already expunged" error to client. diffstat: src/lib-storage/index/index-mail-binary.c | 8 ++- src/lib-storage/index/index-mail-headers.c | 6 +-- src/lib-storage/index/index-mail.c | 51 +++++++++++++++++++---------- src/lib-storage/index/index-mail.h | 2 + 4 files changed, 41 insertions(+), 26 deletions(-) diffs (140 lines): diff -r 82c9a9b7e6e6 -r 0902eded882f src/lib-storage/index/index-mail-binary.c --- a/src/lib-storage/index/index-mail-binary.c Fri Jul 04 13:16:59 2014 +0300 +++ b/src/lib-storage/index/index-mail-binary.c Fri Jul 04 14:01:53 2014 +0300 @@ -116,7 +116,8 @@ if (ctx->input->stream_errno != 0) { errno = ctx->input->stream_errno; mail_storage_set_critical(ctx->mail->box->storage, - "read(%s) failed: %m", i_stream_get_name(ctx->input)); + "read(%s) failed: %s", i_stream_get_name(ctx->input), + i_stream_get_error(ctx->input)); return -1; } @@ -407,8 +408,9 @@ "Invalid data in MIME part"); } else { mail_storage_set_critical(_mail->box->storage, - "read(%s) failed: %m", - i_stream_get_name(cache->input)); + "read(%s) failed: %s", + i_stream_get_name(cache->input), + i_stream_get_error(cache->input)); } mail_storage_free_binary_cache(_mail->box->storage); binary_streams_free(&ctx); diff -r 82c9a9b7e6e6 -r 0902eded882f src/lib-storage/index/index-mail-headers.c --- a/src/lib-storage/index/index-mail-headers.c Fri Jul 04 13:16:59 2014 +0300 +++ b/src/lib-storage/index/index-mail-headers.c Fri Jul 04 14:01:53 2014 +0300 @@ -489,11 +489,7 @@ message_parse_header(stream, NULL, hdr_parser_flags, imap_envelope_parse_callback, mail); if (stream->stream_errno != 0) { - errno = stream->stream_errno; - mail_storage_set_critical(mail->mail.mail.box->storage, - "read(%s) failed: %m (uid=%u)", - i_stream_get_name(mail->data.stream), - mail->mail.mail.uid); + index_mail_stream_log_failure_for(mail, stream); return -1; } mail->data.save_envelope = FALSE; diff -r 82c9a9b7e6e6 -r 0902eded882f src/lib-storage/index/index-mail.c --- a/src/lib-storage/index/index-mail.c Fri Jul 04 13:16:59 2014 +0300 +++ b/src/lib-storage/index/index-mail.c Fri Jul 04 14:01:53 2014 +0300 @@ -855,11 +855,7 @@ (i_stream_read(parser_input) == -1 && !i_stream_have_bytes_left(parser_input))); } else { - errno = parser_input->stream_errno; - mail_storage_set_critical(mail->mail.mail.box->storage, - "mail parser: read(%s, box=%s) failed: %m", - i_stream_get_name(parser_input), - mail->mail.mail.box->vname); + index_mail_stream_log_failure_for(mail, parser_input); ret = -1; } i_stream_unref(&parser_input); @@ -891,18 +887,41 @@ return 0; } +static void index_mail_stream_log_failure(struct index_mail *mail) +{ + index_mail_stream_log_failure_for(mail, mail->data.stream); +} + int index_mail_stream_check_failure(struct index_mail *mail) { if (mail->data.stream->stream_errno == 0) return 0; + index_mail_stream_log_failure(mail); + return -1; +} - errno = mail->data.stream->stream_errno; - mail_storage_set_critical(mail->mail.mail.box->storage, - "read(%s) failed: %s (uid=%u)", - i_stream_get_name(mail->data.stream), - i_stream_get_error(mail->data.stream), - mail->mail.mail.uid); - return -1; +void index_mail_stream_log_failure_for(struct index_mail *mail, + struct istream *input) +{ + struct mail *_mail = &mail->mail.mail; + + i_assert(input->stream_errno != 0); + + errno = input->stream_errno; + if (errno == ENOENT) { + /* was the mail just expunged? we could get here especially if + external attachments are used and the attachment is deleted + before we've opened the file. */ + mail_index_refresh(_mail->box->index); + if (mail_index_is_expunged(_mail->transaction->view, _mail->seq)) { + mail_set_expunged(_mail); + return; + } + } + mail_storage_set_critical(_mail->box->storage, + "read(%s) failed: %s (uid=%u, box=%s)", + i_stream_get_name(input), i_stream_get_error(input), + _mail->uid, mailbox_get_vname(_mail->box)); } static int index_mail_parse_body(struct index_mail *mail, @@ -1001,9 +1020,7 @@ if (message_get_header_size(data->stream, &data->hdr_size, &has_nuls) < 0) { - mail_storage_set_critical(_mail->box->storage, - "read(%s) failed: %m", - i_stream_get_name(data->stream)); + index_mail_stream_log_failure(mail); return -1; } data->hdr_size_set = TRUE; @@ -1027,9 +1044,7 @@ if (message_get_body_size(data->stream, &data->body_size, &has_nuls) < 0) { - mail_storage_set_critical(_mail->box->storage, - "read(%s) failed: %m", - i_stream_get_name(data->stream)); + index_mail_stream_log_failure(mail); return -1; } data->body_size_set = TRUE; diff -r 82c9a9b7e6e6 -r 0902eded882f src/lib-storage/index/index-mail.h --- a/src/lib-storage/index/index-mail.h Fri Jul 04 13:16:59 2014 +0300 +++ b/src/lib-storage/index/index-mail.h Fri Jul 04 14:01:53 2014 +0300 @@ -232,6 +232,8 @@ enum mail_fetch_field field); int index_mail_opened(struct mail *mail, struct istream **stream); int index_mail_stream_check_failure(struct index_mail *mail); +void index_mail_stream_log_failure_for(struct index_mail *mail, + struct istream *input); struct index_mail *index_mail_get_index_mail(struct mail *mail); bool index_mail_get_cached_uoff_t(struct index_mail *mail, From dovecot at dovecot.org Fri Jul 4 11:15:58 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 04 Jul 2014 11:15:58 +0000 Subject: dovecot-2.2: lib-storage: istream-mail updates mail->expunged if... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/c95577fbbe5a changeset: 17599:c95577fbbe5a user: Timo Sirainen date: Fri Jul 04 14:14:21 2014 +0300 description: lib-storage: istream-mail updates mail->expunged if it notices ENOENT from parent stream. diffstat: src/lib-storage/index/index-mail.c | 16 ++++++++++------ src/lib-storage/index/index-mail.h | 1 + src/lib-storage/index/istream-mail.c | 7 ++++++- 3 files changed, 17 insertions(+), 7 deletions(-) diffs (66 lines): diff -r 0902eded882f -r c95577fbbe5a src/lib-storage/index/index-mail.c --- a/src/lib-storage/index/index-mail.c Fri Jul 04 14:01:53 2014 +0300 +++ b/src/lib-storage/index/index-mail.c Fri Jul 04 14:14:21 2014 +0300 @@ -900,6 +900,13 @@ return -1; } +void index_mail_refresh_expunged(struct mail *mail) +{ + mail_index_refresh(mail->box->index); + if (mail_index_is_expunged(mail->transaction->view, mail->seq)) + mail_set_expunged(mail); +} + void index_mail_stream_log_failure_for(struct index_mail *mail, struct istream *input) { @@ -907,16 +914,13 @@ i_assert(input->stream_errno != 0); - errno = input->stream_errno; - if (errno == ENOENT) { + if (input->stream_errno == ENOENT) { /* was the mail just expunged? we could get here especially if external attachments are used and the attachment is deleted before we've opened the file. */ - mail_index_refresh(_mail->box->index); - if (mail_index_is_expunged(_mail->transaction->view, _mail->seq)) { - mail_set_expunged(_mail); + index_mail_refresh_expunged(_mail); + if (_mail->expunged) return; - } } mail_storage_set_critical(_mail->box->storage, "read(%s) failed: %s (uid=%u, box=%s)", diff -r 0902eded882f -r c95577fbbe5a src/lib-storage/index/index-mail.h --- a/src/lib-storage/index/index-mail.h Fri Jul 04 14:01:53 2014 +0300 +++ b/src/lib-storage/index/index-mail.h Fri Jul 04 14:14:21 2014 +0300 @@ -234,6 +234,7 @@ int index_mail_stream_check_failure(struct index_mail *mail); void index_mail_stream_log_failure_for(struct index_mail *mail, struct istream *input); +void index_mail_refresh_expunged(struct mail *mail); struct index_mail *index_mail_get_index_mail(struct mail *mail); bool index_mail_get_cached_uoff_t(struct index_mail *mail, diff -r 0902eded882f -r c95577fbbe5a src/lib-storage/index/istream-mail.c --- a/src/lib-storage/index/istream-mail.c Fri Jul 04 14:01:53 2014 +0300 +++ b/src/lib-storage/index/istream-mail.c Fri Jul 04 14:14:21 2014 +0300 @@ -84,8 +84,13 @@ doesn't have the body */ return -1; } - if (stream->istream.stream_errno != 0) + if (stream->istream.stream_errno != 0) { + if (stream->istream.stream_errno == ENOENT) { + /* update mail's expunged-flag if needed */ + index_mail_refresh_expunged(mstream->mail); + } return -1; + } if (i_stream_mail_try_get_cached_size(mstream) && mstream->expected_size > stream->istream.v_offset + size) { i_stream_mail_set_size_corrupted(mstream, size); From dovecot at dovecot.org Fri Jul 4 11:17:42 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 04 Jul 2014 11:17:42 +0000 Subject: dovecot-2.2: lib-storage: Bodystructure parsing flags weren't up... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/cea7c4540ed0 changeset: 17600:cea7c4540ed0 user: Timo Sirainen date: Fri Jul 04 14:15:41 2014 +0300 description: lib-storage: Bodystructure parsing flags weren't updated correctly on error conditions. This fixes an assert-crash sometimes when mail parsing failed. diffstat: src/lib-storage/index/index-mail-headers.c | 8 ++++++- src/lib-storage/index/index-mail.c | 31 ++++++++++++++++++----------- 2 files changed, 26 insertions(+), 13 deletions(-) diffs (104 lines): diff -r c95577fbbe5a -r cea7c4540ed0 src/lib-storage/index/index-mail-headers.c --- a/src/lib-storage/index/index-mail-headers.c Fri Jul 04 14:14:21 2014 +0300 +++ b/src/lib-storage/index/index-mail-headers.c Fri Jul 04 14:15:41 2014 +0300 @@ -298,7 +298,11 @@ T_BEGIN { index_mail_parse_header_finish(mail); } T_END; - data->save_bodystructure_header = FALSE; + if (data->save_bodystructure_header) { + i_assert(!data->save_bodystructure_body || + data->parser_ctx != NULL); + data->save_bodystructure_header = FALSE; + } return; } @@ -430,6 +434,8 @@ mail); } else { /* just read the header */ + i_assert(!data->save_bodystructure_body || + data->parser_ctx != NULL); message_parse_header(data->stream, &data->hdr_size, hdr_parser_flags, index_mail_parse_header_cb, mail); diff -r c95577fbbe5a -r cea7c4540ed0 src/lib-storage/index/index-mail.c --- a/src/lib-storage/index/index-mail.c Fri Jul 04 14:14:21 2014 +0300 +++ b/src/lib-storage/index/index-mail.c Fri Jul 04 14:15:41 2014 +0300 @@ -850,7 +850,10 @@ if (parser_input->stream_errno == 0 || parser_input->stream_errno == EPIPE) { /* EPIPE = input already closed. allow the caller to - decide if that is an error or not. */ + decide if that is an error or not. (for example we + could be coming here from IMAP APPEND when IMAP + client has closed the connection too early. we + don't want to log an error in that case.) */ i_assert(!success || (i_stream_read(parser_input) == -1 && !i_stream_have_bytes_left(parser_input))); @@ -867,8 +870,16 @@ } mail->data.parts = NULL; mail->data.parsed_bodystructure = FALSE; + if (mail->data.save_bodystructure_body) + mail->data.save_bodystructure_header = TRUE; return -1; } + if (mail->data.save_bodystructure_body) { + mail->data.parsed_bodystructure = TRUE; + mail->data.save_bodystructure_header = FALSE; + mail->data.save_bodystructure_body = FALSE; + i_assert(mail->data.parts != NULL); + } if (mail->data.no_caching) { /* if we're here because we aborted parsing, don't get any @@ -954,11 +965,6 @@ ret = index_mail_stream_check_failure(mail); if (index_mail_parse_body_finish(mail, field, TRUE) < 0) ret = -1; - if (ret == 0 && data->save_bodystructure_body) { - data->save_bodystructure_body = FALSE; - data->parsed_bodystructure = TRUE; - i_assert(data->parts != NULL); - } i_stream_seek(data->stream, old_offset); return ret; @@ -1090,8 +1096,11 @@ data->save_bodystructure_header = TRUE; data->save_bodystructure_body = TRUE; (void)get_cached_parts(mail); - if (index_mail_parse_headers(mail, NULL) < 0) + if (index_mail_parse_headers(mail, NULL) < 0) { + data->save_bodystructure_header = TRUE; return -1; + } + i_assert(data->parser_ctx != NULL); } if (index_mail_parse_body(mail, field) < 0) @@ -1309,6 +1318,8 @@ MAIL_FETCH_MESSAGE_PARTS); } mail->data.parser_input = NULL; + if (mail->data.save_bodystructure_body) + mail->data.save_bodystructure_header = TRUE; } if (data->filter_stream != NULL) i_stream_unref(&data->filter_stream); @@ -1751,11 +1762,7 @@ mail->data.save_date = ioloop_time; } - if (index_mail_parse_body_finish(mail, 0, success) == 0) { - mail->data.save_bodystructure_body = FALSE; - mail->data.parsed_bodystructure = TRUE; - i_assert(mail->data.parts != NULL); - } + (void)index_mail_parse_body_finish(mail, 0, success); } static void index_mail_drop_recent_flag(struct mail *mail) From dovecot at dovecot.org Fri Jul 4 11:17:43 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 04 Jul 2014 11:17:43 +0000 Subject: dovecot-2.2: imap: If FETCH fails because mail has already been ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/d0d9f79a71eb changeset: 17601:d0d9f79a71eb user: Timo Sirainen date: Fri Jul 04 14:16:16 2014 +0300 description: imap: If FETCH fails because mail has already been expunged, don't log an error. diffstat: src/imap/imap-fetch-body.c | 16 ++++++++++++---- 1 files changed, 12 insertions(+), 4 deletions(-) diffs (49 lines): diff -r cea7c4540ed0 -r d0d9f79a71eb src/imap/imap-fetch-body.c --- a/src/imap/imap-fetch-body.c Fri Jul 04 14:15:41 2014 +0300 +++ b/src/imap/imap-fetch-body.c Fri Jul 04 14:16:16 2014 +0300 @@ -27,17 +27,24 @@ unsigned int binary_size:1; }; -static void fetch_read_error(struct imap_fetch_context *ctx) +static void fetch_read_error(struct imap_fetch_context *ctx, + const char **disconnect_reason_r) { struct imap_fetch_state *state = &ctx->state; - errno = state->cur_input->stream_errno; + if (state->cur_input->stream_errno == ENOENT) { + if (state->cur_mail->expunged) { + *disconnect_reason_r = "Mail expunged while it was being FETCHed"; + return; + } + } mail_storage_set_critical(state->cur_mail->box->storage, "read(%s) failed: %s (FETCH %s for mailbox %s UID %u)", i_stream_get_name(state->cur_input), i_stream_get_error(state->cur_input), state->cur_human_name, mailbox_get_vname(state->cur_mail->box), state->cur_mail->uid); + *disconnect_reason_r = "FETCH read() failed"; } static const char *get_body_name(const struct imap_fetch_body_data *body) @@ -85,6 +92,7 @@ static int fetch_stream_continue(struct imap_fetch_context *ctx) { struct imap_fetch_state *state = &ctx->state; + const char *disconnect_reason; off_t ret; o_stream_set_max_buffer_size(ctx->client->output, 0); @@ -97,8 +105,8 @@ if (state->cur_offset != state->cur_size) { /* unfinished */ if (state->cur_input->stream_errno != 0) { - fetch_read_error(ctx); - client_disconnect(ctx->client, "FETCH failed"); + fetch_read_error(ctx, &disconnect_reason); + client_disconnect(ctx->client, disconnect_reason); return -1; } if (!i_stream_have_bytes_left(state->cur_input)) { From dovecot at dovecot.org Fri Jul 4 11:19:33 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 04 Jul 2014 11:19:33 +0000 Subject: dovecot-2.2: lmtp: Small code cleanup Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/16d30ea1e692 changeset: 17602:16d30ea1e692 user: Timo Sirainen date: Fri Jul 04 14:17:54 2014 +0300 description: lmtp: Small code cleanup diffstat: src/lmtp/client.h | 1 - src/lmtp/commands.c | 21 ++++++--------------- 2 files changed, 6 insertions(+), 16 deletions(-) diffs (55 lines): diff -r d0d9f79a71eb -r 16d30ea1e692 src/lmtp/client.h --- a/src/lmtp/client.h Fri Jul 04 14:16:16 2014 +0300 +++ b/src/lmtp/client.h Fri Jul 04 14:17:54 2014 +0300 @@ -16,7 +16,6 @@ const char *session_id; const char *mail_from; ARRAY(struct mail_recipient) rcpt_to; - unsigned int rcpt_idx; unsigned int data_end_idx; diff -r d0d9f79a71eb -r 16d30ea1e692 src/lmtp/commands.c --- a/src/lmtp/commands.c Fri Jul 04 14:16:16 2014 +0300 +++ b/src/lmtp/commands.c Fri Jul 04 14:17:54 2014 +0300 @@ -715,25 +715,17 @@ return ret; } -static bool client_deliver_next(struct client *client, - struct mail_deliver_session *session) +static void client_deliver_mails(struct client *client, + struct mail_deliver_session *session) { const struct mail_recipient *rcpts; - unsigned int count; - int ret; + unsigned int i, count; rcpts = array_get(&client->state.rcpt_to, &count); - while (client->state.rcpt_idx < count) { - ret = client_deliver(client, &rcpts[client->state.rcpt_idx], - session); + for (i = 0; i < count; i++) { + (void)client_deliver(client, &rcpts[i], session); i_set_failure_prefix("lmtp(%s): ", my_pid); - - client->state.rcpt_idx++; - if (ret == 0) - return TRUE; - /* failed. try the next one. */ } - return FALSE; } static void client_rcpt_fail_all(struct client *client) @@ -815,8 +807,7 @@ session = mail_deliver_session_init(); old_uid = geteuid(); - while (client_deliver_next(client, session)) - ; + client_deliver_mails(client, session); mail_deliver_session_deinit(&session); if (old_uid == 0) { From dovecot at dovecot.org Fri Jul 4 11:19:56 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 04 Jul 2014 11:19:56 +0000 Subject: dovecot-2.2: config: parser - trivial error message typo Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/aee2ce74afa8 changeset: 17603:aee2ce74afa8 user: Phil Carmody date: Fri Jul 04 14:18:25 2014 +0300 description: config: parser - trivial error message typo Signed-off-by: Phil Carmody diffstat: src/config/config-parser.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 16d30ea1e692 -r aee2ce74afa8 src/config/config-parser.c --- a/src/config/config-parser.c Fri Jul 04 14:17:54 2014 +0300 +++ b/src/config/config-parser.c Fri Jul 04 14:18:25 2014 +0300 @@ -669,7 +669,7 @@ } } if (*line != '{') { - *value_r = "Expecting '='"; + *value_r = "Expecting '{'"; return CONFIG_LINE_TYPE_ERROR; } } From dovecot at dovecot.org Fri Jul 4 11:50:52 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 04 Jul 2014 11:50:52 +0000 Subject: dovecot-2.2: lib: failures - cosmetic write_full cleanup Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/189f97de7ffa changeset: 17604:189f97de7ffa user: Phil Carmody date: Fri Jul 04 14:48:44 2014 +0300 description: lib: failures - cosmetic write_full cleanup Error message should have a trailing newline. Use the POSIX macro for stderr's file number, rather than its numeric value. Signed-off-by: Phil Carmody diffstat: src/lib/failures.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (21 lines): diff -r aee2ce74afa8 -r 189f97de7ffa src/lib/failures.c --- a/src/lib/failures.c Fri Jul 04 14:18:25 2014 +0300 +++ b/src/lib/failures.c Fri Jul 04 14:48:44 2014 +0300 @@ -457,7 +457,7 @@ if (*fd != STDERR_FILENO) { if (close(*fd) < 0) { - str = t_strdup_printf("close(%d) failed: %m", *fd); + str = t_strdup_printf("close(%d) failed: %m\n", *fd); (void)write_full(STDERR_FILENO, str, strlen(str)); } } @@ -516,7 +516,7 @@ str = t_strdup_printf("\001%c%s %s=%s\n", LOG_TYPE_OPTION+1, my_pid, key, value); - (void)write_full(2, str, strlen(str)); + (void)write_full(STDERR_FILENO, str, strlen(str)); } void i_set_failure_prefix(const char *prefix_fmt, ...) From dovecot at dovecot.org Fri Jul 4 12:35:05 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 04 Jul 2014 12:35:05 +0000 Subject: dovecot-2.2: lib: istream-tee wasn't returning data correctly al... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/dffc6dd2be0d changeset: 17605:dffc6dd2be0d user: Timo Sirainen date: Fri Jul 04 15:33:12 2014 +0300 description: lib: istream-tee wasn't returning data correctly always. This fixes an assert-crash in istream-tee.c. (Hopefully it was always assert-crashing instead of returning corrupted data.) diffstat: src/lib/istream-tee.c | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diffs (30 lines): diff -r 189f97de7ffa -r dffc6dd2be0d src/lib/istream-tee.c --- a/src/lib/istream-tee.c Fri Jul 04 14:48:44 2014 +0300 +++ b/src/lib/istream-tee.c Fri Jul 04 15:33:12 2014 +0300 @@ -35,6 +35,7 @@ old_used = tstream->istream.pos - tstream->istream.skip; tstream->istream.buffer = data; + i_assert(tstream->istream.istream.v_offset >= tee->input->v_offset); tstream->istream.skip = tstream->istream.istream.v_offset - tee->input->v_offset; i_assert(tstream->istream.skip + old_used <= size); @@ -155,14 +156,17 @@ } else { /* there's still some data available from parent */ i_assert(last_high_offset < input->v_offset + size); + tee_streams_update_buffer(tstream->tee); i_assert(stream->pos < size); - stream->buffer = data; } i_assert(stream->buffer == data); ret = size - stream->pos; i_assert(ret > 0); stream->pos = size; + + i_assert(stream->istream.v_offset + (stream->pos - stream->skip) == + input->v_offset + size); return ret; } From dovecot at dovecot.org Sun Jul 6 16:10:33 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sun, 06 Jul 2014 16:10:33 +0000 Subject: dovecot-2.2: virtual: Fixed assert-crashes where trying to open ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/f1d29866e147 changeset: 17606:f1d29866e147 user: Timo Sirainen date: Sun Jul 06 19:08:59 2014 +0300 description: virtual: Fixed assert-crashes where trying to open an already opened backend mailbox. diffstat: src/plugins/virtual/virtual-mail.c | 6 +++--- src/plugins/virtual/virtual-save.c | 6 +++--- src/plugins/virtual/virtual-storage.c | 3 ++- 3 files changed, 8 insertions(+), 7 deletions(-) diffs (58 lines): diff -r dffc6dd2be0d -r f1d29866e147 src/plugins/virtual/virtual-mail.c --- a/src/plugins/virtual/virtual-mail.c Fri Jul 04 15:33:12 2014 +0300 +++ b/src/plugins/virtual/virtual-mail.c Sun Jul 06 19:08:59 2014 +0300 @@ -108,14 +108,14 @@ bbox = virtual_backend_box_lookup(mbox, vmail->cur_vrec.mailbox_id); vmail->cur_backend_mail = backend_mail_find(vmail, bbox->box); if (vmail->cur_backend_mail == NULL) { - if (virtual_backend_box_open(mbox, bbox) < 0) { + if (!bbox->box->opened && + virtual_backend_box_open(mbox, bbox) < 0) { virtual_box_copy_error(mail->box, bbox->box); return -1; } (void)virtual_mail_set_backend_mail(mail, bbox); - } else { - virtual_backend_box_accessed(mbox, bbox); } + virtual_backend_box_accessed(mbox, bbox); vmail->cur_lost = !mail_set_uid(vmail->cur_backend_mail, vmail->cur_vrec.real_uid); mail->expunged = vmail->cur_lost || vmail->cur_backend_mail->expunged; diff -r dffc6dd2be0d -r f1d29866e147 src/plugins/virtual/virtual-save.c --- a/src/plugins/virtual/virtual-save.c Fri Jul 04 15:33:12 2014 +0300 +++ b/src/plugins/virtual/virtual-save.c Sun Jul 06 19:08:59 2014 +0300 @@ -36,7 +36,8 @@ i_assert(ctx->backend_save_ctx == NULL); i_assert(ctx->open_errstr == NULL); - if (virtual_backend_box_open(mbox, mbox->save_bbox) < 0) { + if (!mbox->save_bbox->box->opened && + virtual_backend_box_open(mbox, mbox->save_bbox) < 0) { errstr = mailbox_get_last_error(mbox->save_bbox->box, &ctx->open_error); ctx->open_errstr = i_strdup(errstr); @@ -45,9 +46,8 @@ virtual_transaction_get(_t, mbox->save_bbox->box); ctx->backend_save_ctx = mailbox_save_alloc(backend_trans); } - } else { - virtual_backend_box_accessed(mbox, mbox->save_bbox); } + virtual_backend_box_accessed(mbox, mbox->save_bbox); return _t->save_ctx; } diff -r dffc6dd2be0d -r f1d29866e147 src/plugins/virtual/virtual-storage.c --- a/src/plugins/virtual/virtual-storage.c Fri Jul 04 15:33:12 2014 +0300 +++ b/src/plugins/virtual/virtual-storage.c Sun Jul 06 19:08:59 2014 +0300 @@ -564,7 +564,8 @@ to wait for changes and avoid opening all mailboxes here. */ array_foreach(&mbox->backend_boxes, bboxp) { - if (virtual_backend_box_open(mbox, *bboxp) < 0) { + if (!(*bboxp)->box->opened && + virtual_backend_box_open(mbox, *bboxp) < 0) { /* we can't report error in here, so do it later */ (*bboxp)->open_failed = TRUE; continue; From dovecot at dovecot.org Mon Jul 7 10:17:05 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 07 Jul 2014 10:17:05 +0000 Subject: dovecot-2.2: lib: i_stream_read_copy_from_parent() needs to upda... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/1a38a85d952e changeset: 17607:1a38a85d952e user: Timo Sirainen date: Mon Jul 07 13:15:33 2014 +0300 description: lib: i_stream_read_copy_from_parent() needs to update access counter also when returning -2 This finishes the 467a4d19f873 fix. diffstat: src/lib/istream.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diffs (15 lines): diff -r f1d29866e147 -r 1a38a85d952e src/lib/istream.c --- a/src/lib/istream.c Sun Jul 06 19:08:59 2014 +0300 +++ b/src/lib/istream.c Mon Jul 07 13:15:33 2014 +0300 @@ -207,8 +207,10 @@ if (pos > stream->pos) ret = 0; else do { - if ((ret = i_stream_read(stream->parent)) == -2) + if ((ret = i_stream_read(stream->parent)) == -2) { + i_stream_update(stream); return -2; + } stream->istream.stream_errno = stream->parent->stream_errno; stream->istream.eof = stream->parent->eof; From dovecot at dovecot.org Mon Jul 7 10:25:57 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 07 Jul 2014 10:25:57 +0000 Subject: dovecot-2.2: lib-storage: Minor code cleanup to istream-mail. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/6b512716b37e changeset: 17608:6b512716b37e user: Timo Sirainen date: Mon Jul 07 13:24:22 2014 +0300 description: lib-storage: Minor code cleanup to istream-mail. eof=TRUE shouldn't be possible with ret=-2, so this just makes it clearer what the code's intention is. diffstat: src/lib-storage/index/istream-mail.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 1a38a85d952e -r 6b512716b37e src/lib-storage/index/istream-mail.c --- a/src/lib-storage/index/istream-mail.c Mon Jul 07 13:15:33 2014 +0300 +++ b/src/lib-storage/index/istream-mail.c Mon Jul 07 13:24:22 2014 +0300 @@ -78,7 +78,7 @@ i_stream_mail_set_size_corrupted(mstream, size); return -1; } - } else if (ret < 0 && stream->istream.eof) { + } else if (ret == -1 && stream->istream.eof) { if (!mstream->input_has_body) { /* trying to read past the header, but this stream doesn't have the body */ From dovecot at dovecot.org Mon Jul 7 13:23:34 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 07 Jul 2014 13:23:34 +0000 Subject: dovecot-2.2: lib-index: Don't update log_file_tail_offset unnece... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/16d4cf2c0d65 changeset: 17609:16d4cf2c0d65 user: Timo Sirainen date: Mon Jul 07 16:21:08 2014 +0300 description: lib-index: Don't update log_file_tail_offset unnecessarily. Update it only if we're already writing to transaction log anyway or if we're required to update the offset because mail_index_sync_commit() has increased it past non-external transactions (this is especially important with mdbox map index). diffstat: src/lib-index/mail-index-sync.c | 7 ++++- src/lib-index/mail-index-transaction-export.c | 8 +----- src/lib-index/mail-index-transaction-private.h | 1 + src/lib-index/mail-index-transaction-update.c | 1 + src/lib-index/mail-transaction-log-append.c | 27 ++++++++++++++++++----- src/lib-index/mail-transaction-log-file.c | 16 +------------ src/lib-index/mail-transaction-log.h | 5 +++- src/lib-index/test-mail-transaction-log-append.c | 2 +- 8 files changed, 37 insertions(+), 30 deletions(-) diffs (162 lines): diff -r 6b512716b37e -r 16d4cf2c0d65 src/lib-index/mail-index-sync.c --- a/src/lib-index/mail-index-sync.c Mon Jul 07 13:24:22 2014 +0300 +++ b/src/lib-index/mail-index-sync.c Mon Jul 07 16:21:08 2014 +0300 @@ -751,9 +751,12 @@ mail_transaction_log_set_mailbox_sync_pos(ctx->index->log, seq, offset); /* If tail offset has changed, make sure it gets written to - transaction log. */ - if (ctx->last_tail_offset != offset) + transaction log. do this only if we're required to changes. */ + if (ctx->last_tail_seq != seq || + ctx->last_tail_offset < offset) { ctx->ext_trans->log_updates = TRUE; + ctx->ext_trans->tail_offset_changed = TRUE; + } } static bool mail_index_sync_want_index_write(struct mail_index *index) diff -r 6b512716b37e -r 16d4cf2c0d65 src/lib-index/mail-index-transaction-export.c --- a/src/lib-index/mail-index-transaction-export.c Mon Jul 07 13:24:22 2014 +0300 +++ b/src/lib-index/mail-index-transaction-export.c Mon Jul 07 16:21:08 2014 +0300 @@ -476,12 +476,8 @@ &null4, 4); } - /* Update the tail offsets only when committing the sync transaction. - Other transactions may not know the latest tail offset and might - end up shrinking it. (Alternatively the shrinking tail offsets could - just be ignored, which would probably work fine too.) */ - append_ctx->append_sync_offset = t->sync_transaction; - + append_ctx->index_sync_transaction = t->sync_transaction; + append_ctx->tail_offset_changed = t->tail_offset_changed; append_ctx->want_fsync = (t->view->index->fsync_mask & change_mask) != 0 || (t->flags & MAIL_INDEX_TRANSACTION_FLAG_FSYNC) != 0; diff -r 6b512716b37e -r 16d4cf2c0d65 src/lib-index/mail-index-transaction-private.h --- a/src/lib-index/mail-index-transaction-private.h Mon Jul 07 13:24:22 2014 +0300 +++ b/src/lib-index/mail-index-transaction-private.h Mon Jul 07 16:21:08 2014 +0300 @@ -93,6 +93,7 @@ unsigned int index_deleted:1; unsigned int index_undeleted:1; unsigned int commit_deleted_index:1; + unsigned int tail_offset_changed:1; /* non-extension updates. flag updates don't change this because they may be added and removed, so be sure to check that the updates array is non-empty also. */ diff -r 6b512716b37e -r 16d4cf2c0d65 src/lib-index/mail-index-transaction-update.c --- a/src/lib-index/mail-index-transaction-update.c Mon Jul 07 13:24:22 2014 +0300 +++ b/src/lib-index/mail-index-transaction-update.c Mon Jul 07 16:21:08 2014 +0300 @@ -101,6 +101,7 @@ t->index_undeleted = FALSE; t->log_updates = FALSE; t->log_ext_updates = FALSE; + t->tail_offset_changed = FALSE; } void mail_index_transaction_set_log_updates(struct mail_index_transaction *t) diff -r 6b512716b37e -r 16d4cf2c0d65 src/lib-index/mail-transaction-log-append.c --- a/src/lib-index/mail-transaction-log-append.c Mon Jul 07 13:24:22 2014 +0300 +++ b/src/lib-index/mail-transaction-log-append.c Mon Jul 07 16:21:08 2014 +0300 @@ -122,9 +122,22 @@ buffer_t buf; unsigned char update_data[sizeof(*u) + sizeof(offset)]; - if (file->max_tail_offset == file->sync_offset) { - if (ctx->output->used == 0 && - file->saved_tail_offset == file->max_tail_offset) { + if (!ctx->index_sync_transaction) { + /* this is a non-syncing transaction. update the tail offset + only if we're already writing something else to transaction + log anyway. */ + i_assert(!ctx->tail_offset_changed); + /* FIXME: For now we never do this update, because it would + cause errors about shrinking tail offsets with old Dovecot + versions. This is anyway just an optimization, so it doesn't + matter all that much if we don't do it here. Finish this + in v2.3. */ + /*if (ctx->output->used == 0)*/ + return; + } else if (file->max_tail_offset == file->sync_offset) { + /* we're synced all the way to tail offset, so this sync + transaction can also be included in the same tail offset. */ + if (ctx->output->used == 0 && !ctx->tail_offset_changed) { /* nothing to write here after all (e.g. all unchanged flag updates were dropped by export) */ return; @@ -137,6 +150,10 @@ file->max_tail_offset += ctx->output->used + sizeof(*hdr) + sizeof(*u) + sizeof(offset); ctx->sync_includes_this = TRUE; + } else { + /* This is a syncing transaction. Since we're finishing a sync, + we may need to update the tail offset even if we don't have + anything else to do. */ } offset = file->max_tail_offset; @@ -189,9 +206,7 @@ buffer_delete(ctx->output, 0, boundary_size); } - if (ctx->append_sync_offset) - log_append_sync_offset_if_needed(ctx); - + log_append_sync_offset_if_needed(ctx); if (log_buffer_write(ctx) < 0) return -1; file->sync_highest_modseq = ctx->new_highest_modseq; diff -r 6b512716b37e -r 16d4cf2c0d65 src/lib-index/mail-transaction-log-file.c --- a/src/lib-index/mail-transaction-log-file.c Mon Jul 07 13:24:22 2014 +0300 +++ b/src/lib-index/mail-transaction-log-file.c Mon Jul 07 16:21:08 2014 +0300 @@ -957,20 +957,8 @@ sizeof(tail_offset)); if (tail_offset < file->saved_tail_offset) { - if (file->sync_offset < file->saved_tail_sync_offset) { - /* saved_tail_offset was already set in header, - but we still had to resync the file to find - modseqs. ignore this record. */ - return 1; - } - mail_index_set_error(file->log->index, - "Transaction log file %s seq %u: " - "log_file_tail_offset update shrank it " - "(%u vs %"PRIuUOFF_T" " - "sync_offset=%"PRIuUOFF_T")", - file->filepath, file->hdr.file_seq, - tail_offset, file->saved_tail_offset, - file->sync_offset); + /* ignore shrinking tail offsets */ + return 1; } else { file->saved_tail_offset = tail_offset; if (tail_offset > file->max_tail_offset) diff -r 6b512716b37e -r 16d4cf2c0d65 src/lib-index/mail-transaction-log.h --- a/src/lib-index/mail-transaction-log.h Mon Jul 07 13:24:22 2014 +0300 +++ b/src/lib-index/mail-transaction-log.h Mon Jul 07 16:21:08 2014 +0300 @@ -176,7 +176,10 @@ uint64_t new_highest_modseq; unsigned int transaction_count; - unsigned int append_sync_offset:1; + /* same as mail_index_transaction->sync_transaction */ + unsigned int index_sync_transaction:1; + /* same as mail_index_transaction->tail_offset_changed */ + unsigned int tail_offset_changed:1; unsigned int sync_includes_this:1; unsigned int want_fsync:1; }; diff -r 6b512716b37e -r 16d4cf2c0d65 src/lib-index/test-mail-transaction-log-append.c --- a/src/lib-index/test-mail-transaction-log-append.c Mon Jul 07 13:24:22 2014 +0300 +++ b/src/lib-index/test-mail-transaction-log-append.c Mon Jul 07 16:21:08 2014 +0300 @@ -97,7 +97,7 @@ test_begin("transaction log append: append_sync_offset only"); test_assert(mail_transaction_log_append_begin(log->index, 0, &ctx) == 0); - ctx->append_sync_offset = TRUE; + ctx->index_sync_transaction = TRUE; file->max_tail_offset = 123; test_assert(mail_transaction_log_append_commit(&ctx) == 0); From dovecot at dovecot.org Wed Jul 9 07:57:20 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 09 Jul 2014 07:57:20 +0000 Subject: dovecot-2.2: lib-http: Made sure that connections that are still... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/320a224ad353 changeset: 17610:320a224ad353 user: Stephan Bosch date: Wed Jul 09 10:55:27 2014 +0300 description: lib-http: Made sure that connections that are still connecting to the server aren't marked as idle. diffstat: src/lib-http/http-client-connection.c | 80 +++++++++++++++++++--------------- 1 files changed, 45 insertions(+), 35 deletions(-) diffs (149 lines): diff -r 16d4cf2c0d65 -r 320a224ad353 src/lib-http/http-client-connection.c --- a/src/lib-http/http-client-connection.c Mon Jul 07 16:21:08 2014 +0300 +++ b/src/lib-http/http-client-connection.c Wed Jul 09 10:55:27 2014 +0300 @@ -45,6 +45,8 @@ */ static void http_client_connection_input(struct connection *_conn); +static void +http_client_connection_disconnect(struct http_client_connection *conn); unsigned int http_client_connection_count_pending(struct http_client_connection *conn) @@ -89,12 +91,11 @@ struct http_client_connection *conn = *_conn; struct http_client_request **req; - conn->connected = FALSE; - conn->closing = TRUE; - http_client_connection_debug(conn, "Server explicitly closed connection"); + http_client_connection_disconnect(conn); + array_foreach_modifiable(&conn->request_wait_list, req) { if ((*req)->state < HTTP_REQUEST_STATE_FINISHED) http_client_request_resubmit(*req); @@ -117,9 +118,8 @@ http_client_connection_debug(conn, "Aborting connection: %s", error); - conn->connected = FALSE; - conn->closing = TRUE; - + http_client_connection_disconnect(conn); + array_foreach_modifiable(&conn->request_wait_list, req) { i_assert((*req)->submitted); http_client_request_error(*req, status, error); @@ -156,8 +156,7 @@ http_client_connection_debug(conn, "Aborting connection with temporary error: %s", error); - conn->connected = FALSE; - conn->closing = TRUE; + http_client_connection_disconnect(conn); http_client_connection_retry_requests(conn, status, error); http_client_connection_unref(_conn); @@ -168,6 +167,10 @@ { http_client_connection_debug(conn, "Idle connection timed out"); + /* cannot get here unless connection was established at some point */ + i_assert(conn->connect_succeeded); + + http_client_connection_disconnect(conn); http_client_connection_unref(&conn); } @@ -175,7 +178,8 @@ { unsigned int timeout, count; - if (array_is_created(&conn->request_wait_list) && + if (conn->connected && + array_is_created(&conn->request_wait_list) && array_count(&conn->request_wait_list) == 0 && conn->incoming_payload == NULL && conn->client->set.max_idle_time_msecs > 0) { @@ -1102,6 +1106,37 @@ conn->refcount++; } +static void +http_client_connection_disconnect(struct http_client_connection *conn) +{ + conn->closing = TRUE; + conn->connected = FALSE; + + if (conn->connect_request != NULL) + http_client_request_abort(&conn->connect_request); + + if (conn->incoming_payload != NULL) { + /* the stream is still accessed by lib-http caller. */ + i_stream_remove_destroy_callback(conn->incoming_payload, + http_client_payload_destroyed); + } + + connection_disconnect(&conn->conn); + + if (conn->io_req_payload != NULL) + io_remove(&conn->io_req_payload); + if (conn->to_requests != NULL) + timeout_remove(&conn->to_requests); + if (conn->to_connect != NULL) + timeout_remove(&conn->to_connect); + if (conn->to_input != NULL) + timeout_remove(&conn->to_input); + if (conn->to_idle != NULL) + timeout_remove(&conn->to_idle); + if (conn->to_response != NULL) + timeout_remove(&conn->to_response); +} + void http_client_connection_unref(struct http_client_connection **_conn) { struct http_client_connection *conn = *_conn; @@ -1117,19 +1152,7 @@ http_client_connection_debug(conn, "Connection destroy"); - conn->closing = TRUE; - conn->connected = FALSE; - - if (conn->connect_request != NULL) - http_client_request_abort(&conn->connect_request); - - if (conn->incoming_payload != NULL) { - /* the stream is still accessed by lib-http caller. */ - i_stream_remove_destroy_callback(conn->incoming_payload, - http_client_payload_destroyed); - } - - connection_disconnect(&conn->conn); + http_client_connection_disconnect(conn); if (array_is_created(&conn->request_wait_list)) { /* abort all pending requests */ @@ -1151,19 +1174,6 @@ if (conn->ssl_iostream != NULL) ssl_iostream_unref(&conn->ssl_iostream); connection_deinit(&conn->conn); - - if (conn->io_req_payload != NULL) - io_remove(&conn->io_req_payload); - if (conn->to_requests != NULL) - timeout_remove(&conn->to_requests); - if (conn->to_connect != NULL) - timeout_remove(&conn->to_connect); - if (conn->to_input != NULL) - timeout_remove(&conn->to_input); - if (conn->to_idle != NULL) - timeout_remove(&conn->to_idle); - if (conn->to_response != NULL) - timeout_remove(&conn->to_response); /* remove this connection from the list */ conn_arr = &conn->peer->conns; From dovecot at dovecot.org Wed Jul 9 12:37:28 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 09 Jul 2014 12:37:28 +0000 Subject: dovecot-2.2: lib-storage: Added struct mail_user.session_id Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/23f0f5f6a365 changeset: 17611:23f0f5f6a365 user: Timo Sirainen date: Wed Jul 09 15:20:59 2014 +0300 description: lib-storage: Added struct mail_user.session_id diffstat: src/lib-storage/index/shared/shared-storage.c | 1 + src/lib-storage/mail-storage-service.c | 3 +++ src/lib-storage/mail-user.h | 1 + 3 files changed, 5 insertions(+), 0 deletions(-) diffs (42 lines): diff -r 320a224ad353 -r 23f0f5f6a365 src/lib-storage/index/shared/shared-storage.c --- a/src/lib-storage/index/shared/shared-storage.c Wed Jul 09 10:55:27 2014 +0300 +++ b/src/lib-storage/index/shared/shared-storage.c Wed Jul 09 15:20:59 2014 +0300 @@ -256,6 +256,7 @@ owner = mail_user_alloc(userdomain, user->set_info, user->unexpanded_set); owner->autocreated = TRUE; + owner->session_id = p_strdup(owner->pool, user->session_id); if (mail_user_init(owner, &error) < 0) { if (!owner->nonexistent) { mailbox_list_set_critical(list, diff -r 320a224ad353 -r 23f0f5f6a365 src/lib-storage/mail-storage-service.c --- a/src/lib-storage/mail-storage-service.c Wed Jul 09 10:55:27 2014 +0300 +++ b/src/lib-storage/mail-storage-service.c Wed Jul 09 15:20:59 2014 +0300 @@ -681,6 +681,8 @@ mail_user_unref(&mail_user); return -1; } + mail_user->session_id = + p_strdup(mail_user->pool, user->input.session_id); if ((user->flags & MAIL_STORAGE_SERVICE_FLAG_NO_NAMESPACES) == 0) { if (mail_namespaces_init(mail_user, error_r) < 0) { mail_user_unref(&mail_user); @@ -1088,6 +1090,7 @@ user->input = *input; user->input.userdb_fields = NULL; user->input.username = p_strdup(user_pool, username); + user->input.session_id = p_strdup(user_pool, input->session_id); user->user_info = user_info; user->flags = flags; diff -r 320a224ad353 -r 23f0f5f6a365 src/lib-storage/mail-user.h --- a/src/lib-storage/mail-user.h Wed Jul 09 10:55:27 2014 +0300 +++ b/src/lib-storage/mail-user.h Wed Jul 09 15:20:59 2014 +0300 @@ -23,6 +23,7 @@ uid_t uid; gid_t gid; const char *service; + const char *session_id; struct ip_addr *local_ip, *remote_ip; const char *auth_token, *auth_user; From dovecot at dovecot.org Wed Jul 9 12:37:28 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 09 Jul 2014 12:37:28 +0000 Subject: dovecot-2.2: lib-fs: Added fs_settings.username and .session_id Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/3f84f9cb7460 changeset: 17612:3f84f9cb7460 user: Timo Sirainen date: Wed Jul 09 15:24:21 2014 +0300 description: lib-fs: Added fs_settings.username and .session_id diffstat: src/lib-fs/fs-api.h | 6 ++++++ src/lib-storage/mailbox-list.c | 2 ++ 2 files changed, 8 insertions(+), 0 deletions(-) diffs (28 lines): diff -r 23f0f5f6a365 -r 3f84f9cb7460 src/lib-fs/fs-api.h --- a/src/lib-fs/fs-api.h Wed Jul 09 15:20:59 2014 +0300 +++ b/src/lib-fs/fs-api.h Wed Jul 09 15:24:21 2014 +0300 @@ -70,6 +70,12 @@ }; struct fs_settings { + /* Username and session ID are mainly used for debugging/logging, + but may also be useful for other purposes if they exist (they + may be NULL). */ + const char *username; + const char *session_id; + /* Dovecot instance's base_dir */ const char *base_dir; /* Directory where temporary files can be created at any time diff -r 23f0f5f6a365 -r 3f84f9cb7460 src/lib-storage/mailbox-list.c --- a/src/lib-storage/mailbox-list.c Wed Jul 09 15:20:59 2014 +0300 +++ b/src/lib-storage/mailbox-list.c Wed Jul 09 15:24:21 2014 +0300 @@ -1825,6 +1825,8 @@ ssl_set.ca_file = list->mail_set->ssl_client_ca_file; memset(&fs_set, 0, sizeof(fs_set)); + fs_set.username = list->ns->user->username; + fs_set.session_id = list->ns->user->session_id; fs_set.temp_file_prefix = mailbox_list_get_global_temp_prefix(list); fs_set.base_dir = list->ns->user->set->base_dir; fs_set.temp_dir = list->ns->user->set->mail_temp_dir; From dovecot at dovecot.org Wed Jul 9 12:57:41 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 09 Jul 2014 12:57:41 +0000 Subject: dovecot-2.2: lib-fs: Added username and session_id to struct fs ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/27d269bbbdf0 changeset: 17613:27d269bbbdf0 user: Timo Sirainen date: Wed Jul 09 15:55:59 2014 +0300 description: lib-fs: Added username and session_id to struct fs directly. diffstat: src/lib-fs/fs-api-private.h | 2 ++ src/lib-fs/fs-api.c | 4 ++++ 2 files changed, 6 insertions(+), 0 deletions(-) diffs (33 lines): diff -r 3f84f9cb7460 -r 27d269bbbdf0 src/lib-fs/fs-api-private.h --- a/src/lib-fs/fs-api-private.h Wed Jul 09 15:24:21 2014 +0300 +++ b/src/lib-fs/fs-api-private.h Wed Jul 09 15:55:59 2014 +0300 @@ -59,6 +59,8 @@ struct fs_vfuncs v; char *temp_path_prefix; + char *username, *session_id; + struct fs_settings set; string_t *last_error; diff -r 3f84f9cb7460 -r 27d269bbbdf0 src/lib-fs/fs-api.c --- a/src/lib-fs/fs-api.c Wed Jul 09 15:24:21 2014 +0300 +++ b/src/lib-fs/fs-api.c Wed Jul 09 15:55:59 2014 +0300 @@ -35,6 +35,8 @@ fs_deinit(&fs); return -1; } + fs->username = i_strdup(set->username); + fs->session_id = i_strdup(set->session_id); *fs_r = fs; return 0; } @@ -142,6 +144,8 @@ fs->name, fs->files_open_count); } + i_free(fs->username); + i_free(fs->session_id); i_free(fs->temp_path_prefix); T_BEGIN { fs->v.deinit(fs); From dovecot at dovecot.org Thu Jul 10 12:42:06 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 10 Jul 2014 12:42:06 +0000 Subject: dovecot-2.2: lib-fs: Added FS_PROPERTY_COPY_METADATA and fs_set_... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/2fada3a76d10 changeset: 17614:2fada3a76d10 user: Timo Sirainen date: Thu Jul 10 15:40:08 2014 +0300 description: lib-fs: Added FS_PROPERTY_COPY_METADATA and fs_set_metadata() tracking. diffstat: src/lib-fs/fs-api-private.h | 4 +++- src/lib-fs/fs-api.c | 10 ++++++++++ src/lib-fs/fs-api.h | 5 ++++- 3 files changed, 17 insertions(+), 2 deletions(-) diffs (91 lines): diff -r 27d269bbbdf0 -r 2fada3a76d10 src/lib-fs/fs-api-private.h --- a/src/lib-fs/fs-api-private.h Wed Jul 09 15:55:59 2014 +0300 +++ b/src/lib-fs/fs-api-private.h Thu Jul 10 15:40:08 2014 +0300 @@ -75,7 +75,6 @@ struct istream *seekable_input; struct istream *pending_read_input; - bool write_pending; const struct hash_method *write_digest_method; void *write_digest; @@ -86,6 +85,9 @@ struct fs_file *copy_src; struct istream *copy_input; struct ostream *copy_output; + + unsigned int write_pending; + unsigned int metadata_changed:1; }; struct fs_lock { diff -r 27d269bbbdf0 -r 2fada3a76d10 src/lib-fs/fs-api.c --- a/src/lib-fs/fs-api.c Wed Jul 09 15:55:59 2014 +0300 +++ b/src/lib-fs/fs-api.c Thu Jul 10 15:40:08 2014 +0300 @@ -242,6 +242,7 @@ { if (file->fs->v.set_metadata != NULL) T_BEGIN { file->fs->v.set_metadata(file, key, value); + file->metadata_changed = TRUE; } T_END; } @@ -472,6 +473,8 @@ T_BEGIN { ret = file->fs->v.write_stream_finish(file, TRUE); } T_END; + if (ret != 0) + file->metadata_changed = FALSE; return ret; } @@ -482,6 +485,8 @@ T_BEGIN { ret = file->fs->v.write_stream_finish(file, TRUE); } T_END; + if (ret != 0) + file->metadata_changed = FALSE; return ret; } @@ -493,6 +498,7 @@ T_BEGIN { (void)file->fs->v.write_stream_finish(file, FALSE); } T_END; + file->metadata_changed = FALSE; } void fs_write_set_hash(struct fs_file *file, const struct hash_method *method, @@ -628,6 +634,8 @@ T_BEGIN { ret = src->fs->v.copy(src, dest); } T_END; + if (ret < 0 || errno != EAGAIN) + dest->metadata_changed = FALSE; return ret; } @@ -638,6 +646,8 @@ T_BEGIN { ret = dest->fs->v.copy(NULL, dest); } T_END; + if (ret < 0 || errno != EAGAIN) + dest->metadata_changed = FALSE; return ret; } diff -r 27d269bbbdf0 -r 2fada3a76d10 src/lib-fs/fs-api.h --- a/src/lib-fs/fs-api.h Wed Jul 09 15:55:59 2014 +0300 +++ b/src/lib-fs/fs-api.h Thu Jul 10 15:40:08 2014 +0300 @@ -22,7 +22,10 @@ when its children are deleted. */ FS_PROPERTY_DIRECTORIES = 0x80, FS_PROPERTY_WRITE_HASH_MD5 = 0x100, - FS_PROPERTY_WRITE_HASH_SHA256 = 0x200 + FS_PROPERTY_WRITE_HASH_SHA256 = 0x200, + /* fs_copy() will copy the metadata if fs_set_metadata() hasn't + been explicitly called. */ + FS_PROPERTY_COPY_METADATA = 0x400 }; enum fs_open_mode { From dovecot at dovecot.org Thu Jul 10 13:01:59 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 10 Jul 2014 13:01:59 +0000 Subject: dovecot-2.2: lib: test-istream-concat - test only concat, not si... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/9280679be21e changeset: 17615:9280679be21e user: Phil Carmody date: Thu Jul 10 15:59:53 2014 +0300 description: lib: test-istream-concat - test only concat, not simultanious limit streams Test just concat functionality in this unit test. Simultanious access of limit streams can be tested elsewhere. Without the fix in: 31efe2d04793 lib: istream-concat read() returned -2 too early. The failure previously seen in test-istream-concat would be still reproducable: test-istream-concat.c:84: Assert failed: size >= TEST_MAX_BUFFER_SIZE istream concat random ................................................ : FAILED test: random seed #1 was 1403118493 Signed-off-by: Phil Carmody diffstat: src/lib/test-istream-concat.c | 9 ++------- 1 files changed, 2 insertions(+), 7 deletions(-) diffs (35 lines): diff -r 2fada3a76d10 -r 9280679be21e src/lib/test-istream-concat.c --- a/src/lib/test-istream-concat.c Thu Jul 10 15:40:08 2014 +0300 +++ b/src/lib/test-istream-concat.c Thu Jul 10 15:59:53 2014 +0300 @@ -51,7 +51,7 @@ static bool test_istream_concat_random(void) { - struct istream **streams, *input, *input1, *input2; + struct istream **streams, *input; const unsigned char *data; unsigned char *w_data; size_t size = 0; @@ -72,12 +72,8 @@ input = i_stream_create_concat(streams); i_stream_set_max_buffer_size(input, TEST_MAX_BUFFER_SIZE); - input1 = i_stream_create_limit(input, (uoff_t)-1); - input2 = i_stream_create_limit(input, (uoff_t)-1); - i_stream_unref(&input); for (i = 0; i < 1000; i++) { - input = i % 2 == 0 ? input1 : input2; if (rand() % 3 == 0) { i_stream_seek(input, rand() % offset); } else { @@ -101,8 +97,7 @@ } for (i = 0; i < stream_count; i++) i_stream_unref(&streams[i]); - i_stream_unref(&input1); - i_stream_unref(&input2); + i_stream_unref(&input); return !test_has_failed(); } From dovecot at dovecot.org Thu Jul 10 13:02:04 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 10 Jul 2014 13:02:04 +0000 Subject: dovecot-2.2: lib: test-istream-concat - add simultanious limit s... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/4583a0c2e368 changeset: 17616:4583a0c2e368 user: Phil Carmody date: Thu Jul 10 15:59:53 2014 +0300 description: lib: test-istream-concat - add simultanious limit streams back into the mix Return to the previous complex nested stream-type case. 1 time in 10, it tests the simple case of: test_stream \ test_stream -}- concat_stream test_stream / (2-11 of these) 9 times in 10, it tests this configuration instead: test_stream \ / limit_stream test_stream -}- concat_stream {- limit_stream test_stream / \ limit_stream (2-11 of these) (1-9 of these) Since 31efe2d04793 lib: istream-concat read() returned -2 too early., all tests pass every time. Signed-off-by: Phil Carmody diffstat: src/lib/test-istream-concat.c | 20 +++++++++++++++----- 1 files changed, 15 insertions(+), 5 deletions(-) diffs (49 lines): diff -r 9280679be21e -r 4583a0c2e368 src/lib/test-istream-concat.c --- a/src/lib/test-istream-concat.c Thu Jul 10 15:59:53 2014 +0300 +++ b/src/lib/test-istream-concat.c Thu Jul 10 15:59:53 2014 +0300 @@ -51,11 +51,11 @@ static bool test_istream_concat_random(void) { - struct istream **streams, *input; + struct istream **streams, *concat, **limits = NULL; const unsigned char *data; unsigned char *w_data; size_t size = 0; - unsigned int i, j, offset, stream_count, data_len; + unsigned int i, j, offset, stream_count, data_len, simult; stream_count = (rand() % TEST_MAX_ISTREAM_COUNT) + 2; streams = t_new(struct istream *, stream_count + 1); @@ -70,10 +70,18 @@ streams[i] = NULL; i_assert(offset > 0); - input = i_stream_create_concat(streams); - i_stream_set_max_buffer_size(input, TEST_MAX_BUFFER_SIZE); + concat = i_stream_create_concat(streams); + i_stream_set_max_buffer_size(concat, TEST_MAX_BUFFER_SIZE); + + simult = rand() % TEST_MAX_ISTREAM_COUNT; + if (simult > 0) { + limits = t_new(struct istream *, simult); + for (i = 0; i < simult; i++) + limits[i] = i_stream_create_limit(concat, (uoff_t)-1); + } for (i = 0; i < 1000; i++) { + struct istream *input = (simult == 0) ? concat : limits[rand() % simult]; if (rand() % 3 == 0) { i_stream_seek(input, rand() % offset); } else { @@ -97,7 +105,9 @@ } for (i = 0; i < stream_count; i++) i_stream_unref(&streams[i]); - i_stream_unref(&input); + for (i = 0; i < simult; i++) + i_stream_unref(&limits[i]); + i_stream_unref(&concat); return !test_has_failed(); } From dovecot at dovecot.org Thu Jul 10 13:02:05 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 10 Jul 2014 13:02:05 +0000 Subject: dovecot-2.2: lib: test-istream-tee - verify _read returns correc... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/f87c9ca897a5 changeset: 17617:f87c9ca897a5 user: Phil Carmody date: Thu Jul 10 15:59:53 2014 +0300 description: lib: test-istream-tee - verify _read returns correct values after _set_size() Previously, only an increase of 1 in the size was tested. This ensures that 0 and numbers > 1 are also tested. Also add _idx to the asserts, so we know where in the loop it failed. Signed-off-by: Phil Carmody diffstat: src/lib/test-istream-tee.c | 18 +++++++++++------- 1 files changed, 11 insertions(+), 7 deletions(-) diffs (42 lines): diff -r 4583a0c2e368 -r f87c9ca897a5 src/lib/test-istream-tee.c --- a/src/lib/test-istream-tee.c Thu Jul 10 15:59:53 2014 +0300 +++ b/src/lib/test-istream-tee.c Thu Jul 10 15:59:53 2014 +0300 @@ -15,7 +15,7 @@ { struct istream *test_input, *child_input[CHILD_COUNT]; struct tee_istream *tee; - unsigned int i, len; + unsigned int i, len, delta; test_input = test_istream_create(str); test_istream_set_max_buffer_size(test_input, TEST_BUF_SIZE); @@ -26,19 +26,23 @@ child_input[i] = tee_i_stream_create_child(tee); test_istream_set_allow_eof(test_input, FALSE); - for (len = 1; len < TEST_BUF_SIZE; len++) { + delta = 1; + for (len = 1; len < TEST_BUF_SIZE; len += delta) { test_istream_set_size(test_input, len); for (i = 0; i < CHILD_COUNT; i++) { - test_assert(i_stream_read(child_input[i]) == 1); - test_assert(!tee_i_stream_child_is_waiting(child_input[i])); - test_assert(i_stream_read(child_input[i]) == 0); - test_assert(!tee_i_stream_child_is_waiting(child_input[i])); + test_assert_idx(i_stream_read(child_input[i]) == delta, len); + test_assert_idx(!tee_i_stream_child_is_waiting(child_input[i]), len); + test_assert_idx(i_stream_read(child_input[i]) == 0, len); + test_assert_idx(!tee_i_stream_child_is_waiting(child_input[i]), len); } + delta = rand() % 32; /* may stand still */ + if(delta > TEST_BUF_SIZE - len) + delta = 1; } test_istream_set_size(test_input, len); for (i = 0; i < CHILD_COUNT; i++) { - test_assert(i_stream_read(child_input[i]) == 1); + test_assert(i_stream_read(child_input[i]) == delta); test_assert(i_stream_read(child_input[i]) == -2); test_assert(!tee_i_stream_child_is_waiting(child_input[i])); } From dovecot at dovecot.org Thu Jul 10 13:02:05 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 10 Jul 2014 13:02:05 +0000 Subject: dovecot-2.2: lib: test-istream-tee - more randomisation to the t... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/cba6877471ca changeset: 17618:cba6877471ca user: Phil Carmody date: Thu Jul 10 15:59:53 2014 +0300 description: lib: test-istream-tee - more randomisation to the tests Signed-off-by: Phil Carmody diffstat: src/lib/test-istream-tee.c | 12 ++++++++---- 1 files changed, 8 insertions(+), 4 deletions(-) diffs (35 lines): diff -r f87c9ca897a5 -r cba6877471ca src/lib/test-istream-tee.c --- a/src/lib/test-istream-tee.c Thu Jul 10 15:59:53 2014 +0300 +++ b/src/lib/test-istream-tee.c Thu Jul 10 15:59:53 2014 +0300 @@ -47,23 +47,27 @@ test_assert(!tee_i_stream_child_is_waiting(child_input[i])); } - for (len++; len <= TEST_STR_LEN; len++) { + delta = 1; + while ((len += delta) <= TEST_STR_LEN) { test_istream_set_size(test_input, len); for (i = 0; i < CHILD_COUNT; i++) { test_assert(i_stream_read(child_input[i]) == -2); test_assert(!tee_i_stream_child_is_waiting(child_input[i])); } for (i = 0; i < CHILD_COUNT-1; i++) { - i_stream_skip(child_input[i], 1); + i_stream_skip(child_input[i], delta); test_assert(i_stream_read(child_input[i]) == 0); test_assert(tee_i_stream_child_is_waiting(child_input[i])); } - i_stream_skip(child_input[i], 1); + i_stream_skip(child_input[i], delta); for (i = 0; i < CHILD_COUNT; i++) { - test_assert(i_stream_read(child_input[i]) == 1); + test_assert(i_stream_read(child_input[i]) == delta); test_assert(i_stream_read(child_input[i]) == -2); test_assert(!tee_i_stream_child_is_waiting(child_input[i])); } + delta = rand() % 31 + 1; /* mustn't stand still */ + if(delta > TEST_STR_LEN - len) + delta = 1; } for (i = 0; i < CHILD_COUNT-1; i++) { From dovecot at dovecot.org Thu Jul 10 13:02:05 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 10 Jul 2014 13:02:05 +0000 Subject: dovecot-2.2: lib: test-istream-tee - randomise which tee stream ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/4f56f24b5bb2 changeset: 17619:4f56f24b5bb2 user: Phil Carmody date: Thu Jul 10 15:59:53 2014 +0300 description: lib: test-istream-tee - randomise which tee stream lags behind the others Just in case there's something special about the start or the end of the list of children, make each file be the one that lags behind the others. Signed-off-by: Phil Carmody diffstat: src/lib/test-istream-tee.c | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diffs (26 lines): diff -r cba6877471ca -r 4f56f24b5bb2 src/lib/test-istream-tee.c --- a/src/lib/test-istream-tee.c Thu Jul 10 15:59:53 2014 +0300 +++ b/src/lib/test-istream-tee.c Thu Jul 10 15:59:53 2014 +0300 @@ -49,17 +49,20 @@ delta = 1; while ((len += delta) <= TEST_STR_LEN) { + unsigned int lagger = rand() % CHILD_COUNT; test_istream_set_size(test_input, len); for (i = 0; i < CHILD_COUNT; i++) { test_assert(i_stream_read(child_input[i]) == -2); test_assert(!tee_i_stream_child_is_waiting(child_input[i])); } - for (i = 0; i < CHILD_COUNT-1; i++) { + for (i = 0; i < CHILD_COUNT; i++) { + if (i == lagger) + continue; i_stream_skip(child_input[i], delta); test_assert(i_stream_read(child_input[i]) == 0); test_assert(tee_i_stream_child_is_waiting(child_input[i])); } - i_stream_skip(child_input[i], delta); + i_stream_skip(child_input[lagger], delta); for (i = 0; i < CHILD_COUNT; i++) { test_assert(i_stream_read(child_input[i]) == delta); test_assert(i_stream_read(child_input[i]) == -2); From dovecot at dovecot.org Thu Jul 10 15:19:35 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 10 Jul 2014 15:19:35 +0000 Subject: dovecot-2.2: lib: Compiler warning fixes for 32bit systems Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/a366ae9b182f changeset: 17620:a366ae9b182f user: Timo Sirainen date: Thu Jul 10 18:17:44 2014 +0300 description: lib: Compiler warning fixes for 32bit systems diffstat: src/lib/test-istream-tee.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diffs (30 lines): diff -r 4f56f24b5bb2 -r a366ae9b182f src/lib/test-istream-tee.c --- a/src/lib/test-istream-tee.c Thu Jul 10 15:59:53 2014 +0300 +++ b/src/lib/test-istream-tee.c Thu Jul 10 18:17:44 2014 +0300 @@ -30,7 +30,7 @@ for (len = 1; len < TEST_BUF_SIZE; len += delta) { test_istream_set_size(test_input, len); for (i = 0; i < CHILD_COUNT; i++) { - test_assert_idx(i_stream_read(child_input[i]) == delta, len); + test_assert_idx(i_stream_read(child_input[i]) == (int)delta, len); test_assert_idx(!tee_i_stream_child_is_waiting(child_input[i]), len); test_assert_idx(i_stream_read(child_input[i]) == 0, len); test_assert_idx(!tee_i_stream_child_is_waiting(child_input[i]), len); @@ -42,7 +42,7 @@ test_istream_set_size(test_input, len); for (i = 0; i < CHILD_COUNT; i++) { - test_assert(i_stream_read(child_input[i]) == delta); + test_assert(i_stream_read(child_input[i]) == (int)delta); test_assert(i_stream_read(child_input[i]) == -2); test_assert(!tee_i_stream_child_is_waiting(child_input[i])); } @@ -64,7 +64,7 @@ } i_stream_skip(child_input[lagger], delta); for (i = 0; i < CHILD_COUNT; i++) { - test_assert(i_stream_read(child_input[i]) == delta); + test_assert(i_stream_read(child_input[i]) == (int)delta); test_assert(i_stream_read(child_input[i]) == -2); test_assert(!tee_i_stream_child_is_waiting(child_input[i])); } From dovecot at dovecot.org Thu Jul 10 15:33:03 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 10 Jul 2014 15:33:03 +0000 Subject: dovecot-2.2: lmtp: Remove <> from Delivered-To: header. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/70486f364ce0 changeset: 17621:70486f364ce0 user: Timo Sirainen date: Thu Jul 10 18:31:10 2014 +0300 description: lmtp: Remove <> from Delivered-To: header. This annoyingly changes Dovecot behavior in the middle of v2.2.x series, but the earlier value was definitely wrong.. Perhaps we still need to provide a setting for this, but that's pretty annoying as well. diffstat: src/lmtp/commands.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r a366ae9b182f -r 70486f364ce0 src/lmtp/commands.c --- a/src/lmtp/commands.c Thu Jul 10 18:17:44 2014 +0300 +++ b/src/lmtp/commands.c Thu Jul 10 18:31:10 2014 +0300 @@ -856,7 +856,7 @@ str_printfa(str, "Return-Path: <%s>\r\n", client->state.mail_from); if (rcpt_to != NULL) - str_printfa(str, "Delivered-To: <%s>\r\n", rcpt_to); + str_printfa(str, "Delivered-To: %s\r\n", rcpt_to); } str_printfa(str, "Received: from %s", client->lhlo); From dovecot at dovecot.org Thu Jul 10 21:22:30 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 10 Jul 2014 21:22:30 +0000 Subject: dovecot-2.2: lib: file_*_lock() with flock() timeouts should hav... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/abc7ebe1dad9 changeset: 17622:abc7ebe1dad9 user: Timo Sirainen date: Fri Jul 11 00:20:41 2014 +0300 description: lib: file_*_lock() with flock() timeouts should have returned errno=EAGAIN (Instead of EINTR.) diffstat: src/lib/file-lock.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 70486f364ce0 -r abc7ebe1dad9 src/lib/file-lock.c --- a/src/lib/file-lock.c Thu Jul 10 18:31:10 2014 +0300 +++ b/src/lib/file-lock.c Fri Jul 11 00:20:41 2014 +0300 @@ -127,6 +127,7 @@ if (errno == EWOULDBLOCK || errno == EINTR) { /* a) locked by another process, b) timeouted */ + errno = EAGAIN; return 0; } i_error("flock(%s) locking failed for file %s: %m", From dovecot at dovecot.org Fri Jul 11 08:16:16 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 11 Jul 2014 08:16:16 +0000 Subject: dovecot-2.2: lib: ioloop-epoll didn't correctly check if there w... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/410f263e6fc3 changeset: 17623:410f263e6fc3 user: Timo Sirainen date: Fri Jul 11 11:14:41 2014 +0300 description: lib: ioloop-epoll didn't correctly check if there were any IO events. Alternatively we could have checked for array_count(&ctx->events) > ctx->deleted_count, but this code is a bit more understandable. This change doesn't actually fix any proper bugs, it just causes the process to crash instead of going to infinite wait loop. diffstat: src/lib/ioloop-epoll.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r abc7ebe1dad9 -r 410f263e6fc3 src/lib/ioloop-epoll.c --- a/src/lib/ioloop-epoll.c Fri Jul 11 00:20:41 2014 +0300 +++ b/src/lib/ioloop-epoll.c Fri Jul 11 11:14:41 2014 +0300 @@ -177,7 +177,7 @@ msecs = io_loop_get_wait_time(ioloop, &tv); events = array_get_modifiable(&ctx->events, &events_count); - if (events_count > 0) { + if (ioloop->io_files != NULL) { ret = epoll_wait(ctx->epfd, events, events_count, msecs); if (ret < 0 && errno != EINTR) i_fatal("epoll_wait(): %m"); From dovecot at dovecot.org Fri Jul 11 09:11:36 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 11 Jul 2014 09:11:36 +0000 Subject: dovecot-2.2: lib: Added fd=-1 assert to i_close_fd() macro. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/94cdc5bcb308 changeset: 17624:94cdc5bcb308 user: Timo Sirainen date: Fri Jul 11 12:10:02 2014 +0300 description: lib: Added fd=-1 assert to i_close_fd() macro. This way we'll see clearly where it fails, instead of just seeing assert in close_keep_errno() without an easy way to see where it crashed. diffstat: src/lib/macros.h | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 410f263e6fc3 -r 94cdc5bcb308 src/lib/macros.h --- a/src/lib/macros.h Fri Jul 11 11:14:41 2014 +0300 +++ b/src/lib/macros.h Fri Jul 11 12:10:02 2014 +0300 @@ -205,6 +205,7 @@ #endif #define i_close_fd(fd) STMT_START { \ + i_assert(*fd != -1); \ if (unlikely(close_keep_errno(fd) < 0)) \ i_error("close(%d[%s:%d]) failed: %m", \ *(fd), __FILE__, __LINE__); \ From dovecot at dovecot.org Fri Jul 11 11:18:53 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 11 Jul 2014 11:18:53 +0000 Subject: dovecot-2.2: lib-http: Debug log now includes where HTTP request... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/b162e39909d7 changeset: 17625:b162e39909d7 user: Timo Sirainen date: Fri Jul 11 14:17:17 2014 +0300 description: lib-http: Debug log now includes where HTTP requests spent their time on. diffstat: src/lib-http/http-client-connection.c | 11 ++++++++--- src/lib-http/http-client-private.h | 3 +++ src/lib-http/http-client-request.c | 2 ++ 3 files changed, 13 insertions(+), 3 deletions(-) diffs (67 lines): diff -r 94cdc5bcb308 -r b162e39909d7 src/lib-http/http-client-connection.c --- a/src/lib-http/http-client-connection.c Fri Jul 11 12:10:02 2014 +0300 +++ b/src/lib-http/http-client-connection.c Fri Jul 11 14:17:17 2014 +0300 @@ -400,7 +400,9 @@ i_assert(conn->incoming_payload != NULL); i_assert(conn->conn.io == NULL); - http_client_connection_debug(conn, "Response payload stream destroyed"); + http_client_connection_debug(conn, + "Response payload stream destroyed (%u ms after initial response)", + timeval_diff_msecs(&ioloop_timeval, &req->response_time)); /* caller is allowed to change the socket fd to blocking while reading the payload. make sure here that it's switched back. */ @@ -548,6 +550,7 @@ http_client_connection_unref(&conn); return; } + req->response_time = ioloop_timeval; /* Got some response; cancel response timeout */ if (conn->to_response != NULL) @@ -584,8 +587,10 @@ } http_client_connection_debug(conn, - "Got %u response for request %s", - response.status, http_client_request_label(req)); + "Got %u response for request %s (took %u ms + %u ms in queue)", + response.status, http_client_request_label(req), + timeval_diff_msecs(&req->response_time, &req->sent_time), + timeval_diff_msecs(&req->sent_time, &req->submit_time)); /* make sure connection output is unlocked if 100-continue failed */ if (req->payload_sync && !conn->payload_continue) diff -r 94cdc5bcb308 -r b162e39909d7 src/lib-http/http-client-private.h --- a/src/lib-http/http-client-private.h Fri Jul 11 12:10:02 2014 +0300 +++ b/src/lib-http/http-client-private.h Fri Jul 11 14:17:17 2014 +0300 @@ -71,6 +71,9 @@ struct ostream *payload_output; struct timeval release_time; + struct timeval submit_time; + struct timeval sent_time; + struct timeval response_time; unsigned int attempts; unsigned int redirects; diff -r 94cdc5bcb308 -r b162e39909d7 src/lib-http/http-client-request.c --- a/src/lib-http/http-client-request.c Fri Jul 11 12:10:02 2014 +0300 +++ b/src/lib-http/http-client-request.c Fri Jul 11 14:17:17 2014 +0300 @@ -433,6 +433,7 @@ void http_client_request_submit(struct http_client_request *req) { req->client->pending_requests++; + req->submit_time = ioloop_timeval; http_client_request_do_submit(req); http_client_request_debug(req, "Submitted"); @@ -724,6 +725,7 @@ iov[2].iov_len = 2; req->state = HTTP_REQUEST_STATE_PAYLOAD_OUT; + req->sent_time = ioloop_timeval; o_stream_cork(output); if (o_stream_sendv(output, iov, N_ELEMENTS(iov)) < 0) { *error_r = t_strdup_printf("write(%s) failed: %s", From dovecot at dovecot.org Fri Jul 11 11:29:37 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 11 Jul 2014 11:29:37 +0000 Subject: dovecot-2.2: auth: If userdb prefetch is configured wrong, retur... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/535562da7b29 changeset: 17626:535562da7b29 user: Timo Sirainen date: Fri Jul 11 14:27:52 2014 +0300 description: auth: If userdb prefetch is configured wrong, return temporary failure instead of "user not known" diffstat: src/auth/userdb-prefetch.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diffs (16 lines): diff -r b162e39909d7 -r 535562da7b29 src/auth/userdb-prefetch.c --- a/src/auth/userdb-prefetch.c Fri Jul 11 14:17:17 2014 +0300 +++ b/src/auth/userdb-prefetch.c Fri Jul 11 14:27:52 2014 +0300 @@ -25,8 +25,10 @@ auth_request_log_error(auth_request, AUTH_SUBSYS_DB, "passdb didn't return userdb entries"); } - } else if (!auth_request->userdb_lookup || - auth_request->set->debug) { + callback(USERDB_RESULT_INTERNAL_FAILURE, auth_request); + return; + } + if (!auth_request->userdb_lookup || auth_request->set->debug) { /* more userdbs, they may know the user */ auth_request_log_debug(auth_request, AUTH_SUBSYS_DB, "passdb didn't return userdb entries, " From dovecot at dovecot.org Fri Jul 11 13:51:48 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 11 Jul 2014 13:51:48 +0000 Subject: dovecot-2.2: mailbox_list_index=yes: Don't update INBOX's STATUS... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/fd0616d553b0 changeset: 17627:fd0616d553b0 user: Timo Sirainen date: Fri Jul 11 16:49:35 2014 +0300 description: mailbox_list_index=yes: Don't update INBOX's STATUS information to index. This should reduce disk I/O diffstat: src/lib-storage/list/mailbox-list-index-status.c | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diffs (34 lines): diff -r 535562da7b29 -r fd0616d553b0 src/lib-storage/list/mailbox-list-index-status.c --- a/src/lib-storage/list/mailbox-list-index-status.c Fri Jul 11 14:27:52 2014 +0300 +++ b/src/lib-storage/list/mailbox-list-index-status.c Fri Jul 11 16:49:35 2014 +0300 @@ -23,6 +23,12 @@ struct index_list_storage_module index_list_storage_module = MODULE_CONTEXT_INIT(&mail_storage_module_register); +/* Never update the STATUS information for INBOX. INBOX is almost always opened + anyway, so this just causes extra writes. (Although this could be useful if + somebody has a lot of other users' shared INBOXes.) */ +#define MAILBOX_IS_NEVER_IN_INDEX(box) \ + ((box)->inbox_any) + static int index_list_open_view(struct mailbox *box, struct mail_index_view **view_r, uint32_t *seq_r) @@ -33,6 +39,8 @@ uint32_t seq; int ret; + if (MAILBOX_IS_NEVER_IN_INDEX(box)) + return 0; if (mailbox_list_index_refresh(box->list) < 0) return -1; @@ -389,6 +397,8 @@ if (ilist->syncing || ilist->updating_status) return 0; + if (MAILBOX_IS_NEVER_IN_INDEX(box)) + return 0; /* refresh the mailbox list index once. we can't do this again after locking, because it could trigger list syncing. */ From dovecot at dovecot.org Mon Jul 21 07:55:19 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 21 Jul 2014 07:55:19 +0000 Subject: dovecot-2.2: rawlog: Added -i parameter to include the remote IP... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/fbf434ad2485 changeset: 17628:fbf434ad2485 user: Timo Sirainen date: Mon Jul 21 10:53:19 2014 +0300 description: rawlog: Added -i parameter to include the remote IP address in the filename. diffstat: src/util/rawlog.c | 33 ++++++++++++++++++++++----------- 1 files changed, 22 insertions(+), 11 deletions(-) diffs (106 lines): diff -r fd0616d553b0 -r fbf434ad2485 src/util/rawlog.c --- a/src/util/rawlog.c Fri Jul 11 16:49:35 2014 +0300 +++ b/src/util/rawlog.c Mon Jul 21 10:53:19 2014 +0300 @@ -28,7 +28,8 @@ RAWLOG_FLAG_LOG_INPUT = 0x01, RAWLOG_FLAG_LOG_OUTPUT = 0x02, RAWLOG_FLAG_LOG_TIMESTAMPS = 0x04, - RAWLOG_FLAG_LOG_BOUNDARIES = 0X10 + RAWLOG_FLAG_LOG_BOUNDARIES = 0x10, + RAWLOG_FLAG_LOG_IP_IN_FILENAME = 0x20 }; struct rawlog_proxy { @@ -212,16 +213,23 @@ return 1; } -static void proxy_open_logs(struct rawlog_proxy *proxy, const char *path) +static void proxy_open_logs(struct rawlog_proxy *proxy, const char *path, + const char *ip_addr) { const char *fname, *timestamp; + string_t *path_prefix; int fd; timestamp = t_strflocaltime("%Y%m%d-%H%M%S", time(NULL)); + path_prefix = t_str_new(128); + str_printfa(path_prefix, "%s/", path); + if (ip_addr != NULL && + (proxy->flags & RAWLOG_FLAG_LOG_IP_IN_FILENAME) != 0) + str_printfa(path_prefix, "%s-", ip_addr); + str_printfa(path_prefix, "%s-%s.", timestamp, dec2str(getpid())); if ((proxy->flags & RAWLOG_FLAG_LOG_INPUT) != 0) { - fname = t_strdup_printf("%s/%s-%s.in", path, timestamp, - dec2str(getpid())); + fname = t_strdup_printf("%s.in", str_c(path_prefix)); fd = open(fname, O_CREAT|O_EXCL|O_WRONLY, 0600); if (fd == -1) { i_error("rawlog_open: creat(%s): %m", fname); @@ -232,8 +240,7 @@ } if ((proxy->flags & RAWLOG_FLAG_LOG_OUTPUT) != 0) { - fname = t_strdup_printf("%s/%s-%s.out", path, timestamp, - dec2str(getpid())); + fname = t_strdup_printf("%s.out", str_c(path_prefix)); fd = open(fname, O_CREAT|O_EXCL|O_WRONLY, 0600); if (fd == -1) { i_error("rawlog_open: creat(%s): %m", fname); @@ -247,7 +254,8 @@ static struct rawlog_proxy * rawlog_proxy_create(int client_in_fd, int client_out_fd, int server_fd, - const char *path, enum rawlog_flags flags) + const char *path, const char *ip_addr, + enum rawlog_flags flags) { struct rawlog_proxy *proxy; @@ -273,7 +281,7 @@ proxy->flags = flags; proxy->prev_lf_in = proxy->prev_lf_out = TRUE; - proxy_open_logs(proxy, path); + proxy_open_logs(proxy, path, ip_addr); return proxy; } @@ -338,7 +346,7 @@ dec2str(getppid()))); ioloop = io_loop_create(); - (void)rawlog_proxy_create(0, 1, sfd[0], path, flags); + (void)rawlog_proxy_create(0, 1, sfd[0], path, getenv("IP"), flags); io_loop_run(ioloop); io_loop_destroy(&ioloop); @@ -354,7 +362,7 @@ int c; master_service = master_service_init("rawlog", 0, - &argc, &argv, "+f:obt"); + &argc, &argv, "+f:obit"); while ((c = master_getopt(master_service)) > 0) { switch (c) { case 'f': @@ -368,6 +376,9 @@ case 'b': flags |= RAWLOG_FLAG_LOG_BOUNDARIES; break; + case 'i': + flags |= RAWLOG_FLAG_LOG_IP_IN_FILENAME; + break; case 't': flags |= RAWLOG_FLAG_LOG_TIMESTAMPS; break; @@ -379,7 +390,7 @@ argv += optind; if (argc < 1) - i_fatal("Usage: rawlog [-f in|out] [-b] [-t] "); + i_fatal("Usage: rawlog [-f in|out] [-i] [-b] [-t] "); master_service_init_log(master_service, "rawlog: "); master_service_init_finish(master_service); From dovecot at dovecot.org Mon Jul 21 07:56:04 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 21 Jul 2014 07:56:04 +0000 Subject: dovecot-2.2: lib-http: Added initial support for server-side HTT... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/edcbd75b76ba changeset: 17629:edcbd75b76ba user: Stephan Bosch date: Mon Jul 21 10:54:05 2014 +0300 description: lib-http: Added initial support for server-side HTTP API. diffstat: TODO | 17 + src/lib-http/Makefile.am | 13 +- src/lib-http/http-server-connection.c | 884 ++++++++++++++++++++++++++++++++++ src/lib-http/http-server-private.h | 202 +++++++ src/lib-http/http-server-request.c | 190 +++++++ src/lib-http/http-server-response.c | 376 ++++++++++++++ src/lib-http/http-server.c | 70 ++ src/lib-http/http-server.h | 93 +++ 8 files changed, 1842 insertions(+), 3 deletions(-) diffs (truncated from 1900 to 300 lines): diff -r fbf434ad2485 -r edcbd75b76ba TODO --- a/TODO Mon Jul 21 10:53:19 2014 +0300 +++ b/TODO Mon Jul 21 10:54:05 2014 +0300 @@ -319,3 +319,20 @@ - general - things break if next_uid gets to 2^32 + + - lib-http: + - Client: + - Handle HTTP/1.0 servers properly: + -> Transfer-Encoding is not allowed + - Implement support for priority/deadline-based scheduling. + Much like: https://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html + - Allow handling non-idempotent requests specially + (no automatic retry, block pipeline) + - Implement support for `Range:' requests. + - Implement optional round-robin request scheduling for when + host has multiple IPs. + - Server: + - Implement API structure for virtual hosts and resources. This way, + multiple services can coexist independently on the same HTTP server. + - Implement support for `Range:' requests. + - Review compliance with RFC 7230 and RFC 7231 diff -r fbf434ad2485 -r edcbd75b76ba src/lib-http/Makefile.am --- a/src/lib-http/Makefile.am Mon Jul 21 10:53:19 2014 +0300 +++ b/src/lib-http/Makefile.am Mon Jul 21 10:54:05 2014 +0300 @@ -4,7 +4,8 @@ -I$(top_srcdir)/src/lib \ -I$(top_srcdir)/src/lib-test \ -I$(top_srcdir)/src/lib-dns \ - -I$(top_srcdir)/src/lib-ssl-iostream + -I$(top_srcdir)/src/lib-ssl-iostream \ + -I$(top_srcdir)/src/lib-master libhttp_la_SOURCES = \ http-date.c \ @@ -23,7 +24,11 @@ http-client-peer.c \ http-client-queue.c \ http-client-host.c \ - http-client.c + http-client.c \ + http-server-response.c \ + http-server-request.c \ + http-server-connection.c \ + http-server.c headers = \ http-date.h \ @@ -38,7 +43,9 @@ http-response.h \ http-response-parser.h \ http-client-private.h \ - http-client.h + http-client.h \ + http-server-private.h \ + http-server.h pkginc_libdir=$(pkgincludedir) pkginc_lib_HEADERS = $(headers) diff -r fbf434ad2485 -r edcbd75b76ba src/lib-http/http-server-connection.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/lib-http/http-server-connection.c Mon Jul 21 10:54:05 2014 +0300 @@ -0,0 +1,884 @@ +/* Copyright (c) 2013-2014 Dovecot authors, see the included COPYING file */ + +#include "lib.h" +#include "llist.h" +#include "array.h" +#include "str.h" +#include "ioloop.h" +#include "istream.h" +#include "ostream.h" +#include "connection.h" +#include "iostream-rawlog.h" +#include "iostream-ssl.h" +#include "master-service.h" +#include "master-service-ssl.h" +#include "http-date.h" +#include "http-request-parser.h" + +#include "http-server-private.h" + +/* + * Logging + */ + +static inline void +http_server_connection_debug(struct http_server_connection *conn, + const char *format, ...) ATTR_FORMAT(2, 3); + +static inline void +http_server_connection_debug(struct http_server_connection *conn, + const char *format, ...) +{ + va_list args; + + if (conn->server->set.debug) { + + va_start(args, format); + i_debug("http-server: conn %s: %s", + http_server_connection_label(conn), t_strdup_vprintf(format, args)); + va_end(args); + } +} + +static inline void +http_server_connection_error(struct http_server_connection *conn, + const char *format, ...) ATTR_FORMAT(2, 3); + +static inline void +http_server_connection_error(struct http_server_connection *conn, + const char *format, ...) +{ + va_list args; + + va_start(args, format); + i_error("http-server: conn %s: %s", + http_server_connection_label(conn), t_strdup_vprintf(format, args)); + va_end(args); +} + +static inline void +http_server_connection_client_error(struct http_server_connection *conn, + const char *format, ...) ATTR_FORMAT(2, 3); + +static inline void +http_server_connection_client_error(struct http_server_connection *conn, + const char *format, ...) +{ + va_list args; + + va_start(args, format); + i_info("http-server: conn %s: %s", + http_server_connection_label(conn), t_strdup_vprintf(format, args)); + va_end(args); +} + + + +/* + * Connection + */ + +static void http_server_connection_input(struct connection *_conn); + +static void +http_server_connection_update_stats(struct http_server_connection *conn) +{ + if (conn->conn.input != NULL) + conn->stats.input = conn->conn.input->v_offset; + if (conn->conn.output != NULL) + conn->stats.output = conn->conn.output->offset; +} + +const struct http_server_stats * +http_server_connection_get_stats(struct http_server_connection *conn) +{ + http_server_connection_update_stats(conn); + return &conn->stats; +} + +static void +http_server_connection_input_halt(struct http_server_connection *conn) +{ + if (conn->conn.io != NULL) + io_remove(&conn->conn.io); +} + +static void +http_server_connection_input_resume(struct http_server_connection *conn) +{ + if (conn->conn.io == NULL && !conn->input_broken && !conn->close_indicated) { + conn->conn.io = io_add(conn->conn.fd_in, IO_READ, + http_server_connection_input, &conn->conn); + } +} + +static void +http_server_connection_idle_timeout(struct http_server_connection *conn) +{ + http_server_connection_client_error(conn, "Disconnected for inactivity"); + http_server_connection_close(&conn, "Disconnected for inactivity"); +} + +static void +http_server_connection_timeout_stop(struct http_server_connection *conn) +{ + if (conn->to_idle != NULL) + timeout_remove(&conn->to_idle); +} + +static void +http_server_connection_timeout_start(struct http_server_connection *conn) +{ + if (conn->to_idle == NULL && conn->server->set.max_client_idle_time_msecs > 0) { + conn->to_idle = timeout_add(conn->server->set.max_client_idle_time_msecs, + http_server_connection_idle_timeout, conn); + } +} + +static void +http_server_connection_timeout_reset(struct http_server_connection *conn) +{ + if (conn->to_idle != NULL) + timeout_reset(conn->to_idle); +} + +static void http_server_connection_ready(struct http_server_connection *conn) +{ + struct stat st; + + if (conn->server->set.rawlog_dir != NULL && + stat(conn->server->set.rawlog_dir, &st) == 0) { + iostream_rawlog_create(conn->server->set.rawlog_dir, + &conn->conn.input, &conn->conn.output); + } + + conn->http_parser = http_request_parser_init + (conn->conn.input, &conn->server->set.request_limits); + o_stream_set_flush_callback(conn->conn.output, + http_server_connection_output, conn); +} + +static void http_server_connection_destroy(struct connection *_conn) +{ + struct http_server_connection *conn = + (struct http_server_connection *)_conn; + + conn->closed = TRUE; + http_server_connection_unref(&conn); +} + +static void http_server_payload_finished(struct http_server_connection *conn) +{ + timeout_remove(&conn->to_input); + http_server_connection_input_resume(conn); +} + +static void +http_server_payload_destroyed_timeout(struct http_server_connection *conn) +{ + http_server_connection_input(&conn->conn); +} + +static void http_server_payload_destroyed(struct http_server_request *req) +{ + struct http_server_connection *conn = req->conn; + int stream_errno; + + i_assert(conn != NULL); + i_assert(conn->request_queue_tail == req || + req->state >= HTTP_SERVER_REQUEST_STATE_FINISHED); + i_assert(conn->conn.io == NULL); + + http_server_connection_debug(conn, "Request payload stream destroyed"); + + /* caller is allowed to change the socket fd to blocking while reading + the payload. make sure here that it's switched back. */ + net_set_nonblock(conn->conn.fd_in, TRUE); + + stream_errno = conn->incoming_payload->stream_errno; + conn->incoming_payload = NULL; + + /* handle errors in transfer stream */ + if (req->response == NULL && stream_errno != 0 && + conn->conn.input->stream_errno == 0) { + switch (stream_errno) { + case EMSGSIZE: + conn->input_broken = TRUE; + http_server_connection_client_error(conn, + "Client sent excessively large request"); + http_server_request_fail(req, 413, "Payload Too Large", TRUE); + return; + case EIO: + conn->input_broken = TRUE; + http_server_connection_client_error(conn, + "Client sent invalid request payload"); + http_server_request_fail(req, 400, "Bad Request", conn->input_broken); + return; + default: + break; + } + } + + if (req->state < HTTP_SERVER_REQUEST_STATE_PROCESSING) { + /* finished reading request */ + req->state = HTTP_SERVER_REQUEST_STATE_PROCESSING; + if (req->response != NULL && req->response->submitted) + http_server_request_submit_response(req); + } + + /* input stream may have pending input. make sure input handler + gets called (but don't do it directly, since we get get here + somewhere from the API user's code, which we can't really know what + state it is in). this call also triggers sending the next response if + necessary. */ + if (!conn->input_broken) { + conn->to_input = From dovecot at dovecot.org Mon Jul 21 08:02:44 2014 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 21 Jul 2014 08:02:44 +0000 Subject: dovecot-2.2: lib-http: Fixed linking test programs. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/e07f504f1040 changeset: 17630:e07f504f1040 user: Timo Sirainen date: Mon Jul 21 11:00:59 2014 +0300 description: lib-http: Fixed linking test programs. diffstat: src/lib-http/Makefile.am | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diffs (35 lines): diff -r edcbd75b76ba -r e07f504f1040 src/lib-http/Makefile.am --- a/src/lib-http/Makefile.am Mon Jul 21 10:54:05 2014 +0300 +++ b/src/lib-http/Makefile.am Mon Jul 21 11:00:59 2014 +0300 @@ -127,11 +127,15 @@ libhttp.la \ ../lib-dns/libdns.la \ ../lib-ssl-iostream/libssl_iostream.la \ + ../lib-master/libmaster.la \ + ../lib-settings/libsettings.la \ $(test_libs) test_http_client_DEPENDENCIES = \ libhttp.la \ ../lib-dns/libdns.la \ ../lib-ssl-iostream/libssl_iostream.la \ + ../lib-master/libmaster.la \ + ../lib-settings/libsettings.la \ $(test_deps) test_http_server_SOURCES = test-http-server.c @@ -140,11 +144,15 @@ libhttp.la \ ../lib-dns/libdns.la \ ../lib-ssl-iostream/libssl_iostream.la \ + ../lib-master/libmaster.la \ + ../lib-settings/libsettings.la \ $(test_libs) test_http_server_DEPENDENCIES = \ libhttp.la \ ../lib-dns/libdns.la \ ../lib-ssl-iostream/libssl_iostream.la \ + ../lib-master/libmaster.la \ + ../lib-settings/libsettings.la \ $(test_deps) check: check-am check-test From pigeonhole at rename-it.nl Wed Jul 23 17:35:08 2014 From: pigeonhole at rename-it.nl (pigeonhole at rename-it.nl) Date: Wed, 23 Jul 2014 19:35:08 +0200 Subject: dovecot-2.2-pigeonhole: lib-sieve: Implemented support for gener... Message-ID: details: http://hg.rename-it.nl/dovecot-2.2-pigeonhole/rev/8e2832c044a5 changeset: 1883:8e2832c044a5 user: Stephan Bosch date: Wed Jul 23 19:28:07 2014 +0200 description: lib-sieve: Implemented support for generic Sieve storages. Doveadm-sieve plugin may be broken at this point; not tested so far. diffstat: TODO | 32 +- configure.ac | 4 +- src/Makefile.am | 1 - src/lib-sieve/Makefile.am | 19 +- src/lib-sieve/plugins/include/cmd-include.c | 54 +- src/lib-sieve/plugins/include/ext-include-binary.c | 18 +- src/lib-sieve/plugins/include/ext-include-common.c | 81 +- src/lib-sieve/plugins/include/ext-include-common.h | 12 +- src/lib-sieve/sieve-binary.h | 2 +- src/lib-sieve/sieve-common.h | 8 + src/lib-sieve/sieve-error.c | 43 +- src/lib-sieve/sieve-error.h | 4 + src/lib-sieve/sieve-script-dict.c | 3 - src/lib-sieve/sieve-script-file.h | 31 - src/lib-sieve/sieve-script-private.h | 80 +- src/lib-sieve/sieve-script.c | 620 ++++--- src/lib-sieve/sieve-script.h | 101 +- src/lib-sieve/sieve-storage-private.h | 244 +++ src/lib-sieve/sieve-storage-sync.c | 131 + src/lib-sieve/sieve-storage.c | 1078 +++++++++++++++ src/lib-sieve/sieve-storage.h | 188 ++ src/lib-sieve/sieve.c | 236 +-- src/lib-sieve/sieve.h | 19 +- src/lib-sieve/storage/Makefile.am | 3 + src/lib-sieve/storage/dict/Makefile.am | 13 + src/lib-sieve/storage/dict/sieve-dict-script.c | 315 ++++ src/lib-sieve/storage/dict/sieve-dict-storage.c | 185 ++ src/lib-sieve/storage/dict/sieve-dict-storage.h | 68 + src/lib-sieve/storage/file/Makefile.am | 18 + src/lib-sieve/storage/file/sieve-file-script-sequence.c | 239 +++ src/lib-sieve/storage/file/sieve-file-script.c | 757 ++++++++++ src/lib-sieve/storage/file/sieve-file-storage-active.c | 428 +++++ src/lib-sieve/storage/file/sieve-file-storage-list.c | 125 + src/lib-sieve/storage/file/sieve-file-storage-quota.c | 123 + src/lib-sieve/storage/file/sieve-file-storage-save.c | 452 ++++++ src/lib-sieve/storage/file/sieve-file-storage.c | 681 +++++++++ src/lib-sieve/storage/file/sieve-file-storage.h | 177 ++ src/lib-sievestorage/Makefile.am | 21 - src/lib-sievestorage/sieve-storage-list.c | 115 - src/lib-sievestorage/sieve-storage-list.h | 29 - src/lib-sievestorage/sieve-storage-private.h | 69 - src/lib-sievestorage/sieve-storage-quota.c | 159 -- src/lib-sievestorage/sieve-storage-quota.h | 28 - src/lib-sievestorage/sieve-storage-save.c | 459 ------ src/lib-sievestorage/sieve-storage-save.h | 39 - src/lib-sievestorage/sieve-storage-script.c | 740 ---------- src/lib-sievestorage/sieve-storage-script.h | 34 - src/lib-sievestorage/sieve-storage.c | 744 ---------- src/lib-sievestorage/sieve-storage.h | 58 - src/managesieve/Makefile.am | 4 +- src/managesieve/cmd-deletescript.c | 10 +- src/managesieve/cmd-getscript.c | 6 +- src/managesieve/cmd-havespace.c | 3 +- src/managesieve/cmd-listscripts.c | 4 +- src/managesieve/cmd-putscript.c | 22 +- src/managesieve/cmd-renamescript.c | 9 +- src/managesieve/cmd-setactive.c | 22 +- src/managesieve/managesieve-client.c | 50 +- src/managesieve/managesieve-quota.c | 2 +- src/plugins/Makefile.am | 5 +- src/plugins/doveadm-sieve/Makefile.am | 4 +- src/plugins/doveadm-sieve/doveadm-sieve-plugin.c | 51 +- src/plugins/lda-sieve/lda-sieve-plugin.c | 282 ++- src/sieve-tools/sievec.c | 3 +- src/testsuite/testsuite-binary.c | 1 - src/testsuite/testsuite-script.c | 1 - tests/extensions/include/execute.svtest | 30 +- 67 files changed, 6201 insertions(+), 3396 deletions(-) diffs (truncated from 11144 to 300 lines): diff -r 92405f753f6a -r 8e2832c044a5 TODO --- a/TODO Thu Jun 26 23:25:43 2014 +0200 +++ b/TODO Wed Jul 23 19:28:07 2014 +0200 @@ -1,12 +1,6 @@ Current activities: -* Make the sieve storage a base class with (possibly) various implementations, - just like mail-storage. This aims to provide support for alternate types - of script storage like LDAP or SQL database. - - Implement read/write script storages for using ManageSieve with dict - database - - Implement infrastructure for loading a sequence of global scripts from - a database (for sieve_before/sieve_after). +* Implement LDAP Sieve script storage for read-only access. Parallel plugin-based efforts (on hold at the moment): @@ -17,8 +11,21 @@ Next (mostly in order of descending priority/precedence): * Implement index extension +* Implement message modification and extraction API in order to: + - Implement replace, enclose, foreverypart, mime and extracttext extensions * Add normalize() method to comparators to normalize the string before matching (for efficiency). +* Properly implement Sieve internationalization support (utf-8 handling), + currently it is not complete: + - Make this implementation fully conform section 2.7.2 of RFC5228 (Comparisons + Across Character Sets). + - Verify validity of utf8 where necessary. + - Implement comparator-i;unicode-casemap. +* Add support for stream matching for handling large values, e.g. from the body + extension. +* Clean up doveadm sync implementation: + - Mostly comments referring specifically to the file storage type. + - API can be polished a lot as well. * Improve error handling. - Implement dropping errors in the user's mailbox as a mail message. * Further develop regex extension and update it to the latest draft: @@ -50,12 +57,6 @@ fails rather than an implicit keep. - Add a method to implicitly pass environment variables such as SENDER and RECIPIENT through the script socket service. -* Properly implement Sieve internationalization support (utf-8 handling), - currently it is not complete: - - Make this implementation fully conform section 2.7.2 of RFC5228 (Comparisons - Across Character Sets). - - Verify validity of utf8 where necessary. - - Implement comparator-i;unicode-casemap. * Make testsuite much more exhaustive: - Add support for testing the content of result actions - Test as many error/warning/info conditions as possible. @@ -79,10 +80,6 @@ * Implement extlists extension as a plugin * Enotify extension: detect use of variable values extracted from the message that are used in the method argument. RFC reports this as a security issue. -* Add support for stream matching for handling large values, e.g. from the body - extension. -* Implement message modification and extraction API in order to: - - Implement replace, enclose, foreverypart, mime and extracttext extensions * Provide a solution for mail_get_headers_utf8 reparsing the whole message each time it is called (header and address test; Timo might provide solution from within Dovecot) @@ -91,7 +88,6 @@ * Implement IMAP plugin for IMAPSieve support: - Requires some sort of Sieve transaction support. - - Requires (IMAP) METADATA support in Dovecot. - This may include support for manually running a script on a set of messages through IMAP (no specification for something like this is available; we will have to provide our own) diff -r 92405f753f6a -r 8e2832c044a5 configure.ac --- a/configure.ac Thu Jun 26 23:25:43 2014 +0200 +++ b/configure.ac Wed Jul 23 19:28:07 2014 +0200 @@ -101,6 +101,9 @@ src/Makefile src/lib-sieve/Makefile src/lib-sieve/util/Makefile +src/lib-sieve/storage/Makefile +src/lib-sieve/storage/file/Makefile +src/lib-sieve/storage/dict/Makefile src/lib-sieve/plugins/Makefile src/lib-sieve/plugins/vacation/Makefile src/lib-sieve/plugins/subaddress/Makefile @@ -126,7 +129,6 @@ src/lib-sieve/plugins/vnd.dovecot/Makefile src/lib-sieve/plugins/vnd.dovecot/debug/Makefile src/lib-sieve-tool/Makefile -src/lib-sievestorage/Makefile src/lib-managesieve/Makefile src/plugins/Makefile src/plugins/doveadm-sieve/Makefile diff -r 92405f753f6a -r 8e2832c044a5 src/Makefile.am --- a/src/Makefile.am Thu Jun 26 23:25:43 2014 +0200 +++ b/src/Makefile.am Wed Jul 23 19:28:07 2014 +0200 @@ -2,7 +2,6 @@ sieve_subdirs = \ lib-sieve \ - lib-sievestorage \ plugins \ lib-sieve-tool \ sieve-tools \ diff -r 92405f753f6a -r 8e2832c044a5 src/lib-sieve/Makefile.am --- a/src/lib-sieve/Makefile.am Thu Jun 26 23:25:43 2014 +0200 +++ b/src/lib-sieve/Makefile.am Wed Jul 23 19:28:07 2014 +0200 @@ -1,4 +1,4 @@ -SUBDIRS = util plugins +SUBDIRS = util storage plugins dovecot_pkglib_LTLIBRARIES = libdovecot-sieve.la @@ -43,11 +43,17 @@ cmp-i-ascii-casemap.c if BUILD_UNFINISHED +unfinished_storages = unfinished_plugins = \ $(extdir)/metadata/libsieve_ext_metadata.la endif -# These are not actual plugins just yet... +strgdir = $(top_builddir)/src/lib-sieve/storage +storages = \ + $(strgdir)/file/libsieve_storage_file.la \ + $(strgdir)/dict/libsieve_storage_dict.la \ + $(unfinished_storages) + extdir = $(top_builddir)/src/lib-sieve/plugins plugins = \ $(extdir)/vacation/libsieve_ext_vacation.la \ @@ -73,12 +79,14 @@ $(unfinished_plugins) libdovecot_sieve_la_DEPENDENCIES = \ + $(storages) \ $(plugins) \ $(top_builddir)/src/lib-sieve/util/libsieve_util.la \ $(LIBDOVECOT_LDA_DEPS) \ $(LIBDOVECOT_STORAGE_DEPS) \ $(LIBDOVECOT_DEPS) libdovecot_sieve_la_LIBADD = \ + $(storages) \ $(plugins) \ $(top_builddir)/src/lib-sieve/util/libsieve_util.la \ $(LIBDOVECOT_LDA) \ @@ -91,8 +99,8 @@ sieve-smtp.c \ sieve-lexer.c \ sieve-script.c \ - sieve-script-file.c \ - sieve-script-dict.c \ + sieve-storage.c \ + sieve-storage-sync.c \ sieve-ast.c \ sieve-binary.c \ sieve-binary-file.c \ @@ -137,7 +145,8 @@ sieve-lexer.h \ sieve-script.h \ sieve-script-private.h \ - sieve-script-file.h \ + sieve-storage.h \ + sieve-storage-private.h \ sieve-ast.h \ sieve-binary.h \ sieve-binary-private.h \ diff -r 92405f753f6a -r 8e2832c044a5 src/lib-sieve/plugins/include/cmd-include.c --- a/src/lib-sieve/plugins/include/cmd-include.c Thu Jun 26 23:25:43 2014 +0200 +++ b/src/lib-sieve/plugins/include/cmd-include.c Wed Jul 23 19:28:07 2014 +0200 @@ -6,6 +6,7 @@ #include "sieve-common.h" #include "sieve-script.h" +#include "sieve-storage.h" #include "sieve-ast.h" #include "sieve-code.h" #include "sieve-extensions.h" @@ -213,8 +214,9 @@ struct sieve_ast_argument *arg = cmd->first_positional; struct cmd_include_context_data *ctx_data = (struct cmd_include_context_data *) cmd->data; + struct sieve_storage *storage; struct sieve_script *script; - const char *script_location, *script_name; + const char *script_name; enum sieve_error error = SIEVE_ERROR_NONE; int ret; @@ -247,30 +249,41 @@ return FALSE; } - script_location = ext_include_get_script_location - (this_ext, ctx_data->location, script_name); - if ( script_location == NULL ) { - sieve_argument_validate_error(valdtr, arg, - "include: %s location for included script '%s' is unavailable " - "(contact system administrator for more information)", - ext_include_script_location_name(ctx_data->location), - str_sanitize(script_name, 80)); + storage = ext_include_get_script_storage + (this_ext, ctx_data->location, script_name, &error); + if ( storage == NULL ) { + // FIXME: handle ':optional' in this case + if (error == SIEVE_ERROR_NOT_FOUND) { + sieve_argument_validate_error(valdtr, arg, + "include: %s location for included script `%s' is unavailable " + "(contact system administrator for more information)", + ext_include_script_location_name(ctx_data->location), + str_sanitize(script_name, 80)); + } else { + sieve_argument_validate_error(valdtr, arg, + "include: failed to access %s location for included script `%s' " + "(contact system administrator for more information)", + ext_include_script_location_name(ctx_data->location), + str_sanitize(script_name, 80)); + } return FALSE; } /* Create script object */ - script = sieve_script_create - (this_ext->svinst, script_location, script_name, - sieve_validator_error_handler(valdtr), &error); + script = sieve_storage_get_script + (storage, script_name, &error); + if ( script == NULL ) + return FALSE; - ret = 0; - if ( script != NULL ) - ret = sieve_script_open(script, &error); - - if ( script == NULL || ret < 0 ) { + ret = sieve_script_open(script, &error); + if ( ret < 0 ) { if ( error != SIEVE_ERROR_NOT_FOUND ) { - if ( script != NULL ) - sieve_script_unref(&script); + sieve_argument_validate_error(valdtr, arg, + "failed to access included %s script '%s': %s", + ext_include_script_location_name(ctx_data->location), + str_sanitize(script_name, 80), + sieve_script_get_last_error_lcase(script)); + sieve_script_unref(&script); return FALSE; /* Not found */ @@ -295,8 +308,7 @@ "included %s script '%s' does not exist", ext_include_script_location_name(ctx_data->location), str_sanitize(script_name, 80)); - if ( script != NULL ) - sieve_script_unref(&script); + sieve_script_unref(&script); return FALSE; } } diff -r 92405f753f6a -r 8e2832c044a5 src/lib-sieve/plugins/include/ext-include-binary.c --- a/src/lib-sieve/plugins/include/ext-include-binary.c Thu Jun 26 23:25:43 2014 +0200 +++ b/src/lib-sieve/plugins/include/ext-include-binary.c Wed Jul 23 19:28:07 2014 +0200 @@ -7,6 +7,7 @@ #include "sieve-common.h" #include "sieve-error.h" #include "sieve-script.h" +#include "sieve-storage.h" #include "sieve-binary.h" #include "sieve-generator.h" #include "sieve-interpreter.h" @@ -277,7 +278,7 @@ struct sieve_binary_block *inc_block = NULL; unsigned int location, flags; string_t *script_name; - const char *script_location; + struct sieve_storage *storage; struct sieve_script *script; enum sieve_error error; int ret; @@ -314,24 +315,24 @@ } /* Can we find the script dependency ? */ - script_location = ext_include_get_script_location - (ext, location, str_c(script_name)); - if ( script_location == NULL ) { + storage = ext_include_get_script_storage + (ext, location, str_c(script_name), NULL); + if ( storage == NULL ) { /* No, recompile */ + // FIXME: handle ':optional' in this case return FALSE; } /* Can we open the script dependency ? */ - script = sieve_script_create - (ext->svinst, script_location, str_c(script_name), NULL, &error); + script = sieve_storage_get_script + (storage, str_c(script_name), &error); if ( script == NULL ) { From pigeonhole at rename-it.nl Wed Jul 23 17:36:52 2014 From: pigeonhole at rename-it.nl (pigeonhole at rename-it.nl) Date: Wed, 23 Jul 2014 19:36:52 +0200 Subject: dovecot-2.2-pigeonhole: lib-sieve: Created sieve LDAP storage dr... Message-ID: details: http://hg.rename-it.nl/dovecot-2.2-pigeonhole/rev/c01b68b0c6b1 changeset: 1884:c01b68b0c6b1 user: Stephan Bosch date: Wed Jul 23 19:36:00 2014 +0200 description: lib-sieve: Created sieve LDAP storage driver. diffstat: INSTALL | 158 +- TODO | 6 +- configure.ac | 87 + doc/example-config/conf.d/90-sieve.conf | 77 +- doc/example-config/sieve-ldap.conf | 74 + doc/script-location-ldap.txt | 64 + pigeonhole-config.h.in | 3 + src/lib-sieve/Makefile.am | 1 + src/lib-sieve/sieve-storage.c | 1 + src/lib-sieve/storage/Makefile.am | 3 +- src/lib-sieve/storage/ldap/Makefile.am | 32 + src/lib-sieve/storage/ldap/sieve-ldap-db.c | 1380 ++++++++++++++ src/lib-sieve/storage/ldap/sieve-ldap-db.h | 140 + src/lib-sieve/storage/ldap/sieve-ldap-script.c | 314 +++ src/lib-sieve/storage/ldap/sieve-ldap-storage-settings.c | 155 + src/lib-sieve/storage/ldap/sieve-ldap-storage.c | 215 ++ src/lib-sieve/storage/ldap/sieve-ldap-storage.h | 104 + 17 files changed, 2718 insertions(+), 96 deletions(-) diffs (truncated from 3023 to 300 lines): diff -r 8e2832c044a5 -r c01b68b0c6b1 INSTALL --- a/INSTALL Wed Jul 23 19:28:07 2014 +0200 +++ b/INSTALL Wed Jul 23 19:36:00 2014 +0200 @@ -64,6 +64,60 @@ configuration files are provided in the doc/example-config directory of this package. +Sieve Interpreter - Script Locations +------------------------------------ + +The Sieve interpreter can retrieve Sieve scripts from several types of +locations. The default `file' location type is a local filesystem path pointing +to a Sieve script file or a directory containing multiple Sieve script files. +More complex setups can use other location types such as `ldap' or `dict' to +fetch Sieve scripts from remote databases. + +All settings that specify the location of one ore more Sieve scripts accept the +following syntax: + +location = [:]path[;