Making progress :)

I'll document some obtracles I found, to make it easier for the next one implementing push with Dovecot and Lua.

First I tried with my usual Alpine based container, but Alpine seems not to build the Lua stuff for Dovecot :(

So I moved to an Ubuntu 18.04 based container and the official Dovecot CE repo:

FROM ubuntu:18.04 RUN apt-get update &&
apt-get install -y apt-transport-https gpg curl &&
curl https://repo.dovecot.org/DOVECOT-REPO-GPG | gpg --import && \ gpg --export ED409DA1 > /etc/apt/trusted.gpg.d/dovecot.gpg && \ echo "deb https://repo.dovecot.org/ce-2.3-latest/ubuntu/bionic bionic main" > /etc/apt/sources.list.d/dovecot.list &&
apt-get update &&
bash -c "apt-get install -y dovecot-{core,imapd,sqlite,managesieved,sieve,pop3d,lmtpd,submissiond,lua} lua-socket" CMD [ "/usr/sbin/dovecot","-F","-c","/etc/dovecot/dovecot.conf" ]

I had to install lua-socket, which is used by the example script and not required by dovecot-lua, which is ok, you just need to know.

Using Aki's code snippet as user=<metadata> lead to an other error:

Aug 03 14:54:15 Error: doveadm: lua: /usr/share/lua/5.2/socket/url.lua:31: bad argument #1 to 'gsub' (string expected, got nil) Aug 03 14:54:15 Error: lmtp(<username> 38): lmtp-server: conn 10.9.94.14:42092 [1]: rcpt <username>: lua: /usr/share/lua/5.2/socket/url.lua:31: bad argument #1 to 'gsub' (string expected, got nil)

I'm now skipping the notification, if no metadata is set, like the OX driver does:

function dovecot_lua_notify_event_message_new(ctx, event) -- check if there is a push token registered if (ctx.meta == nil or ctx.meta == '') then return end

Currently looking into the following questions:

• can I get the rfc 5423 type of event somehow (obviously I can set it on the event myself depending of the function called)
• looking at the example code, it looks like it can be called for multiple messages, when does that happen (LMTP send more then one)
• why is the mailbox status put into an other structure and send with a different notifiction
• does anyone have a code snippet to send a JSON encoded message (probably easy to figure out looking at Lua docu)

Ralf

Am 03.08.20 um 11:56 schrieb Ralf Becker:

Thanks Aki, I'll check it out :)

Am 03.08.20 um 11:40 schrieb Aki Tuomi:

...

...

On 03/08/2020 12:31 Ralf Becker rb@egroupware.org wrote:

We're currently using the OX push driver, which is straight forward (simple web hook) and allows to store (multiple) push tokens of our webmailer direct in mailbox metadata.

Only drawback is that it only supports new arriving mails in the INBOX, even mails moved via Sieve to other folders are NOT reported.

Therefore we updated now to Dovecot 2.3(.10.1) to also get mails moved by user or Sieve scripts, deleted mails or flag changes.

As far as I read the example Lua scripts and (a little) the Dovecot C code, the nice indirection of using mailbox metadata to a) enable push and b) store push tokens (modify reported user attribute with them) does NOT exist in the Lua driver by default.

So my questions is: how can I access mailbox metadata from within the Lua script, to make eg. the example script behave like the OX driver with user_from_metadata set?

I'm happy to update the Lua examples, if that's any help ...

Ralf

-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0 Actually it does exist:

https://doc.dovecot.org/admin_manual/lua/#mail_user.metadata_get

or

https://doc.dovecot.org/admin_manual/lua/#object-mailbox

mailbox:metadata_get()

You get both of these objects from the push notification data, you just have to keep them in the context state. (See the example scripts)

function dovecot_lua_notify_begin_txn(user) local meta = user:metadata_get("/private/key") return {messages={}, ep=user:plugin_getenv("push_lua_url"), username=user.username, meta=meta} end

Aki

-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0

https://doc.dovecot.org/configuration_manual/push_notification/#message-even...

dovecot_lua_notify_event_flags_set(context, {name, mailbox, uid,
uid_validity, flags, keywords_set})

--> dovecot_lua_notify_event_flags_set(context, {name, mailbox, uid, uid_validity, flags, keywords})

Called when message flags or keywords are set. flags is a bitmask. keywords_set is a table of strings of the keywords set by the event.

dovecot_lua_notify_event_flags_clear(context, {name, mailbox, uid,
uid_validity, flags, keywords_clear, keywords_old})

--> dovecot_lua_notify_event_flags_clear(context, {name, mailbox, uid, uid_validity, flags, flags_old, keywords, keywords_old})

Ralf

-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0

In case someone is interested this is the final version of the script:

-- To use -- -- plugin { -- push_notification_driver = lua:file=/etc/dovecot/dovecot-push.lua -- push_lua_url = https://Bearer:<push-token>@<egroupware-domain>/egroupware/push -- } -- -- server is sent a PUT message with JSON body like push_notification_driver = ox:url= user_from_metadata -- plus additionally the events MessageAppend, MessageExpunge, FlagsSet and FlagsClear -- MessageTrash and MessageRead are ignored, so are empty FlagSet/Clear or FlagSet NonJunk (TB) -- local http = require "socket.http" local ltn12 = require "ltn12" -- luarocks install json-lua local json = require "JSON" function table_get(t, k, d) return t[k] or d end function script_init() return 0 end function dovecot_lua_notify_begin_txn(user) local meta = user:metadata_get("/private/vendor/vendor.dovecot/http-notify") if (meta == nil or meta:sub(1,5) ~= "user=") then meta = nil; else meta = meta:sub(6) end return {user=user, event=dovecot.event(), ep=user:plugin_getenv("push_lua_url"), messages={}, meta=meta} end function dovecot_lua_notify_event_message_new(ctx, event) -- check if there is a push token registered if (ctx.meta == nil) then return end -- get mailbox status local mbox = ctx.user:mailbox(event.mailbox) mbox:sync() local status = mbox:status(dovecot.storage.STATUS_RECENT, dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES) mbox:free() table.insert(ctx.messages, { user = ctx.meta, ["imap-uidvalidity"] = event.uid_validity, ["imap-uid"] = event.uid, folder = event.mailbox, event = event.name, from = event.from, subject = event.subject, snippet = event.snippet, unseen = status.unseen, messages = status.messages }) end function dovecot_lua_notify_event_message_append(ctx, event) dovecot_lua_notify_event_message_new(ctx, event) end -- ignored, as FlagSet flags=[\Seen] is sent anyway too -- function dovecot_lua_notify_event_message_read(ctx, event) -- dovecot_lua_notify_event_message_expunge(ctx, event) -- end -- ignored, as most MUA nowadays expunge immediatly -- function dovecot_lua_notify_event_message_trash(ctx, event) -- dovecot_lua_notify_event_message_expunge(ctx, event) -- end function dovecot_lua_notify_event_message_expunge(ctx, event) -- check if there is a push token registered if (ctx.meta == nil) then return end -- get mailbox status local mbox = ctx.user:mailbox(event.mailbox) mbox:sync() local status = mbox:status(dovecot.storage.STATUS_RECENT, dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES) mbox:free() -- agregate multiple Expunge (or Trash or Read) if (#ctx.messages == 1 and ctx.messages[1].user == ctx.meta and ctx.messages[1].folder == event.mailbox and ctx.messages[1]["imap-uidvalidity"] == event.uid_validity and ctx.messages[1].event == event.name) then if (type(ctx.messages[1]["imap-uid"]) ~= 'table') then ctx.messages[1]["imap-uid"] = {ctx.messages[1]["imap-uid"]} end table.insert(ctx.messages[1]["imap-uid"], event.uid) ctx.messages[1].unseen = status.unseen ctx.messages[1].messages = status.messages return; end table.insert(ctx.messages, { user = ctx.meta, ["imap-uidvalidity"] = event.uid_validity, ["imap-uid"] = event.uid, folder = event.mailbox, event = event.name, unseen = status.unseen, messages = status.messages }) end function dovecot_lua_notify_event_flags_set(ctx, event) -- check if there is a push token registered if (ctx.meta == nil or (#event.flags == 0 and #event.keywords == 0) or -- ignore TB sends it empty (#event.keywords == 1 and event.keywords[1] == "NonJunk")) -- ignore TB NonJunk then return end local status = nil; if (#event.flags == 1 and event.flags[1] == "\\Seen") then -- get mailbox status local mbox = ctx.user:mailbox(event.mailbox) mbox:sync() status = mbox:status(dovecot.storage.STATUS_RECENT, dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES) mbox:free() end -- agregate multiple FlagSet if (#ctx.messages == 1 and ctx.messages[1].user == ctx.meta and ctx.messages[1].folder == event.mailbox and ctx.messages[1]["imap-uidvalidity"] == event.uid_validity and ctx.messages[1].event == event.name and arrayEqual(ctx.messages[1].flags, event.flags) and arrayEqual(ctx.messages[1].keywords, event.keywords)) then if (type(ctx.messages[1]["imap-uid"]) ~= 'table') then ctx.messages[1]["imap-uid"] = {ctx.messages[1]["imap-uid"]} end table.insert(ctx.messages[1]["imap-uid"], event.uid) if (status ~= nil) then ctx.messages[1].unseen = status.unseen end return; end table.insert(ctx.messages, { user = ctx.meta, ["imap-uidvalidity"] = event.uid_validity, ["imap-uid"] = event.uid, folder = event.mailbox, event = event.name, flags = event.flags, keywords = event.keywords, unseen = status.unseen }) end function arrayEqual(t1, t2) if (#t1 ~= #t2) then return false end if (#t1 == 1 and t1[1] == t2[1]) then return true end return json:encode(t1) == json:encode(t2) end function dovecot_lua_notify_event_flags_clear(ctx, event) -- check if there is a push token registered AND something to clear (TB sends it empty) if (ctx.meta == nil or (#event.flags == 0 and #event.keywords == 0)) then return end table.insert(ctx.messages, { user = ctx.meta, ["imap-uidvalidity"] = event.uid_validity, ["imap-uid"] = event.uid, folder = event.mailbox, event = event.name, flags = event.flags, ["flags-old"] = event.flags_old, keywords = event.keywords, ["keywords-old"] = event.keywords_old }) end function dovecot_lua_notify_end_txn(ctx) -- report all states for i,msg in ipairs(ctx.messages) do local e = dovecot.event(ctx.event) e:set_name("lua_notify_mail_finished") reqbody = json:encode(msg) e:log_debug(ctx.ep .. " - sending " .. reqbody) res, code = http.request({ method = "PUT", url = ctx.ep, source = ltn12.source.string(reqbody), headers={ ["content-type"] = "application/json; charset=utf-8", ["content-length"] = tostring(#reqbody) } }) e:add_int("result_code", code) e:log_info("Mail notify status " .. tostring(code)) end end

It's also in our Github repo: https://raw.githubusercontent.com/EGroupware/swoolepush/master/doc/dovecot-p...

Ralf

-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0