How to access mailbox metadata in Lua push driver
We're currently using the OX push driver, which is straight forward (simple web hook) and allows to store (multiple) push tokens of our webmailer direct in mailbox metadata.
Only drawback is that it only supports new arriving mails in the INBOX, even mails moved via Sieve to other folders are NOT reported.
Therefore we updated now to Dovecot 2.3(.10.1) to also get mails moved by user or Sieve scripts, deleted mails or flag changes.
As far as I read the example Lua scripts and (a little) the Dovecot C code, the nice indirection of using mailbox metadata to a) enable push and b) store push tokens (modify reported user attribute with them) does NOT exist in the Lua driver by default.
So my questions is: how can I access mailbox metadata from within the Lua script, to make eg. the example script behave like the OX driver with user_from_metadata set?
I'm happy to update the Lua examples, if that's any help ...
Ralf
-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
Actually it does exist:
https://doc.dovecot.org/admin_manual/lua/#mail_user.metadata_get
or
https://doc.dovecot.org/admin_manual/lua/#object-mailbox
mailbox:metadata_get()
You get both of these objects from the push notification data, you just have to keep them in the context state. (See the example scripts)
function dovecot_lua_notify_begin_txn(user) local meta = user:metadata_get("/private/key") return {messages={}, ep=user:plugin_getenv("push_lua_url"), username=user.username, meta=meta} end
Aki
Making progress :)
I'll document some obtracles I found, to make it easier for the next one implementing push with Dovecot and Lua.
First I tried with my usual Alpine based container, but Alpine seems not to build the Lua stuff for Dovecot :(
So I moved to an Ubuntu 18.04 based container and the official Dovecot CE repo:
FROM ubuntu:18.04
RUN apt-get update &&
apt-get install -y apt-transport-https gpg curl &&
curl https://repo.dovecot.org/DOVECOT-REPO-GPG | gpg --import && \ gpg --export ED409DA1 > /etc/apt/trusted.gpg.d/dovecot.gpg && \ echo "deb https://repo.dovecot.org/ce-2.3-latest/ubuntu/bionic bionic main" > /etc/apt/sources.list.d/dovecot.list &&
apt-get update &&
bash -c "apt-get install -y
dovecot-{core,imapd,sqlite,managesieved,sieve,pop3d,lmtpd,submissiond,lua}
lua-socket"
CMD [ "/usr/sbin/dovecot","-F","-c","/etc/dovecot/dovecot.conf" ]
I had to install lua-socket, which is used by the example script and not required by dovecot-lua, which is ok, you just need to know.
Using Aki's code snippet as user=<metadata> lead to an other error:
Aug 03 14:54:15 Error: doveadm: lua: /usr/share/lua/5.2/socket/url.lua:31: bad argument #1 to 'gsub' (string expected, got nil) Aug 03 14:54:15 Error: lmtp(<username> 38): lmtp-server: conn 10.9.94.14:42092 [1]: rcpt <username>: lua: /usr/share/lua/5.2/socket/url.lua:31: bad argument #1 to 'gsub' (string expected, got nil)
I'm now skipping the notification, if no metadata is set, like the OX driver does:
function dovecot_lua_notify_event_message_new(ctx, event) -- check if there is a push token registered if (ctx.meta == nil or ctx.meta == '') then return end
Currently looking into the following questions:
- can I get the rfc 5423 type of event somehow (obviously I can set it on the event myself depending of the function called)
- looking at the example code, it looks like it can be called for multiple messages, when does that happen (LMTP send more then one)
- why is the mailbox status put into an other structure and send with a different notifiction
- does anyone have a code snippet to send a JSON encoded message (probably easy to figure out looking at Lua docu)
Ralf
Am 03.08.20 um 11:56 schrieb Ralf Becker:
-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
Some answers to my questions, a first version of my script and more questions ;) Am 03.08.20 um 18:15 schrieb Ralf Becker:
still no idea, maybe Ake? I noticed that some events have the same uid-validity, are the from a single transaction, eg. I delete my Trash?
these two I managed to solve im my current version of the script, which also support now all message event types: -- To use -- -- plugin { -- push_notification_driver = lua:file=/etc/dovecot/dovecot-push.lua -- push_lua_url = http://push.notification.server/handler -- } -- -- server is sent a PUT message with JSON body like push_notification_driver = ox:url=<push_lua_url> user_from_metadata -- local http = require "socket.http" local ltn12 = require "ltn12" -- luarocks install json-lua local json = require "JSON" function table_get(t, k, d) return t[k] or d end function script_init() return 0 end function dovecot_lua_notify_begin_txn(user) local meta = user:metadata_get("/private/vendor/vendor.dovecot/http-notify") return {user=user, event=dovecot.event(), ep=user:plugin_getenv("push_lua_url"), messages={}, meta=meta} end function dovecot_lua_notify_event_message_new(ctx, event) -- check if there is a push token registered if (ctx.meta == nil or ctx.meta == '') then return end -- get mailbox status local mbox = ctx.user:mailbox(event.mailbox) mbox:sync() local status = mbox:status(dovecot.storage.STATUS_RECENT, dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES) mbox:free() table.insert(ctx.messages, { user = ctx.meta, ["imap-uidvalidity"] = event.uid_validity, ["imap-uid"] = event.uid, folder = event.mailbox, event = event.name, from = event.from, subject = event.subject, snippet = event.snippet, unseen = status.unseen }) end function dovecot_lua_notify_event_message_append(ctx, event) dovecot_lua_notify_event_message_new(ctx, event) end function dovecot_lua_notify_event_message_read(ctx, event) -- check if there is a push token registered if (ctx.meta == nil or ctx.meta == '') then return end -- get mailbox status local mbox = ctx.user:mailbox(event.mailbox) mbox:sync() local status = mbox:status(dovecot.storage.STATUS_RECENT, dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES) mbox:free() table.insert(ctx.messages, { user = ctx.meta, ["imap-uidvalidity"] = event.uid_validity, ["imap-uid"] = event.uid, folder = event.mailbox, event = event.name, unseen = status.unseen }) end function dovecot_lua_notify_event_message_trash(ctx, event) dovecot_lua_notify_event_message_read(ctx, event) end function dovecot_lua_notify_event_message_expunge(ctx, event) dovecot_lua_notify_event_message_read(ctx, event) end function dovecot_lua_notify_event_flags_set(ctx, event) -- check if there is a push token registered if (ctx.meta == nil or ctx.meta == '') then return end table.insert(ctx.messages, { user = ctx.meta, ["imap-uidvalidity"] = event.uid_validity, ["imap-uid"] = event.uid, folder = event.mailbox, event = event.name, flags = event.flags, ["keywords-set"] = event.keywords_set }) end function dovecot_lua_notify_event_flags_clear(ctx, event) -- check if there is a push token registered if (ctx.meta == nil or ctx.meta == '') then return end table.insert(ctx.messages, { user = ctx.meta, ["imap-uidvalidity"] = event.uid_validity, ["imap-uid"] = event.uid, folder = event.mailbox, event = event.name, flags = event.flags, ["keywords-clear"] = event.keywords_clear, ["keywords-old"] = event.keywords_old }) end function dovecot_lua_notify_end_txn(ctx) -- report all states for i,msg in ipairs(ctx.messages) do local e = dovecot.event(ctx.event) e:set_name("lua_notify_mail_finished") reqbody = json:encode(msg) e:log_debug(ctx.ep .. " - sending " .. reqbody) res, code = http.request({ method = "PUT", url = ctx.ep, source = ltn12.source.string(reqbody), headers={ ["content-type"] = "application/json; charset=utf-8", ["content-length"] = tostring(#reqbody) } }) e:add_int("result_code", code) e:log_info("Mail notify status " .. tostring(code)) end end This leads to a couple more questions ;) - is there a way (eg. return value) to stop event processing already in dovecot_lua_notify_begin_txn - sometimes multiple events are generated, eg. when I read an email: {"event":"FlagsClear","flags":[],"folder":"INBOX/eGroupWare/calconnect","imap-uid":2275,"imap-uidvalidity":1499767470,"keywords-old":[],"user":"user=5::42;***"} {"event":"FlagsSet","flags":["\\Seen"],"folder":"INBOX/eGroupWare/calconnect","imap-uid":2275,"imap-uidvalidity":1499767470,"user":"user=5::42;***"} {"event":"MessageRead","folder":"INBOX/eGroupWare/calconnect","imap-uid":2275,"imap-uidvalidity":1499767470,"unseen":0,"user":"user=5::42;***"} Ralf -- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
https://doc.dovecot.org/configuration_manual/push_notification/#message-even...
dovecot_lua_notify_event_flags_set(context, {name, mailbox, uid,
uid_validity, flags, keywords_set})--> dovecot_lua_notify_event_flags_set(context, {name, mailbox, uid, uid_validity, flags, keywords})
Called when message flags or keywords are set. flags is a bitmask. keywords_set is a table of strings of the keywords set by the event.
dovecot_lua_notify_event_flags_clear(context, {name, mailbox, uid,
uid_validity, flags, keywords_clear, keywords_old})--> dovecot_lua_notify_event_flags_clear(context, {name, mailbox, uid, uid_validity, flags, flags_old, keywords, keywords_old})
Ralf
-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
Because these events also apply for more than just LMTP. You could be storing bunch of mails with IMAP APPEND.
- why is the mailbox status put into an other structure and send with a different notifiction
The LUA code very faitfully implements the C API for push notifications in Dovecot.
- does anyone have a code snippet to send a JSON encoded message (probably easy to figure out looking at Lua docu) these two I managed to solve im my current version of the script, which also support now all message event types:
<snip7>
This leads to a couple more questions ;) - is there a way (eg. return value) to stop event processing already in dovecot_lua_notify_begin_txn
There is no really good way to do this. You can error out in begin_txn, but that will emit error. The only doable way, right now, is to maybe add key {disabled=True} to your context, and then add if ctx.disabled: return elsewhere.
This is what happens when you read unseen mail.
Aki
In case someone is interested this is the final version of the script:
-- To use -- -- plugin { -- push_notification_driver = lua:file=/etc/dovecot/dovecot-push.lua -- push_lua_url = https://Bearer:<push-token>@<egroupware-domain>/egroupware/push -- } -- -- server is sent a PUT message with JSON body like push_notification_driver = ox:url=<push_lua_url> user_from_metadata -- plus additionally the events MessageAppend, MessageExpunge, FlagsSet and FlagsClear -- MessageTrash and MessageRead are ignored, so are empty FlagSet/Clear or FlagSet NonJunk (TB) -- local http = require "socket.http" local ltn12 = require "ltn12" -- luarocks install json-lua local json = require "JSON" function table_get(t, k, d) return t[k] or d end function script_init() return 0 end function dovecot_lua_notify_begin_txn(user) local meta = user:metadata_get("/private/vendor/vendor.dovecot/http-notify") if (meta == nil or meta:sub(1,5) ~= "user=") then meta = nil; else meta = meta:sub(6) end return {user=user, event=dovecot.event(), ep=user:plugin_getenv("push_lua_url"), messages={}, meta=meta} end function dovecot_lua_notify_event_message_new(ctx, event) -- check if there is a push token registered if (ctx.meta == nil) then return end -- get mailbox status local mbox = ctx.user:mailbox(event.mailbox) mbox:sync() local status = mbox:status(dovecot.storage.STATUS_RECENT, dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES) mbox:free() table.insert(ctx.messages, { user = ctx.meta, ["imap-uidvalidity"] = event.uid_validity, ["imap-uid"] = event.uid, folder = event.mailbox, event = event.name, from = event.from, subject = event.subject, snippet = event.snippet, unseen = status.unseen, messages = status.messages }) end function dovecot_lua_notify_event_message_append(ctx, event) dovecot_lua_notify_event_message_new(ctx, event) end -- ignored, as FlagSet flags=[\Seen] is sent anyway too -- function dovecot_lua_notify_event_message_read(ctx, event) -- dovecot_lua_notify_event_message_expunge(ctx, event) -- end -- ignored, as most MUA nowadays expunge immediatly -- function dovecot_lua_notify_event_message_trash(ctx, event) -- dovecot_lua_notify_event_message_expunge(ctx, event) -- end function dovecot_lua_notify_event_message_expunge(ctx, event) -- check if there is a push token registered if (ctx.meta == nil) then return end -- get mailbox status local mbox = ctx.user:mailbox(event.mailbox) mbox:sync() local status = mbox:status(dovecot.storage.STATUS_RECENT, dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES) mbox:free() -- agregate multiple Expunge (or Trash or Read) if (#ctx.messages == 1 and ctx.messages[1].user == ctx.meta and ctx.messages[1].folder == event.mailbox and ctx.messages[1]["imap-uidvalidity"] == event.uid_validity and ctx.messages[1].event == event.name) then if (type(ctx.messages[1]["imap-uid"]) ~= 'table') then ctx.messages[1]["imap-uid"] = {ctx.messages[1]["imap-uid"]} end table.insert(ctx.messages[1]["imap-uid"], event.uid) ctx.messages[1].unseen = status.unseen ctx.messages[1].messages = status.messages return; end table.insert(ctx.messages, { user = ctx.meta, ["imap-uidvalidity"] = event.uid_validity, ["imap-uid"] = event.uid, folder = event.mailbox, event = event.name, unseen = status.unseen, messages = status.messages }) end function dovecot_lua_notify_event_flags_set(ctx, event) -- check if there is a push token registered if (ctx.meta == nil or (#event.flags == 0 and #event.keywords == 0) or -- ignore TB sends it empty (#event.keywords == 1 and event.keywords[1] == "NonJunk")) -- ignore TB NonJunk then return end local status = nil; if (#event.flags == 1 and event.flags[1] == "\\Seen") then -- get mailbox status local mbox = ctx.user:mailbox(event.mailbox) mbox:sync() status = mbox:status(dovecot.storage.STATUS_RECENT, dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES) mbox:free() end -- agregate multiple FlagSet if (#ctx.messages == 1 and ctx.messages[1].user == ctx.meta and ctx.messages[1].folder == event.mailbox and ctx.messages[1]["imap-uidvalidity"] == event.uid_validity and ctx.messages[1].event == event.name and arrayEqual(ctx.messages[1].flags, event.flags) and arrayEqual(ctx.messages[1].keywords, event.keywords)) then if (type(ctx.messages[1]["imap-uid"]) ~= 'table') then ctx.messages[1]["imap-uid"] = {ctx.messages[1]["imap-uid"]} end table.insert(ctx.messages[1]["imap-uid"], event.uid) if (status ~= nil) then ctx.messages[1].unseen = status.unseen end return; end table.insert(ctx.messages, { user = ctx.meta, ["imap-uidvalidity"] = event.uid_validity, ["imap-uid"] = event.uid, folder = event.mailbox, event = event.name, flags = event.flags, keywords = event.keywords, unseen = status.unseen }) end function arrayEqual(t1, t2) if (#t1 ~= #t2) then return false end if (#t1 == 1 and t1[1] == t2[1]) then return true end return json:encode(t1) == json:encode(t2) end function dovecot_lua_notify_event_flags_clear(ctx, event) -- check if there is a push token registered AND something to clear (TB sends it empty) if (ctx.meta == nil or (#event.flags == 0 and #event.keywords == 0)) then return end table.insert(ctx.messages, { user = ctx.meta, ["imap-uidvalidity"] = event.uid_validity, ["imap-uid"] = event.uid, folder = event.mailbox, event = event.name, flags = event.flags, ["flags-old"] = event.flags_old, keywords = event.keywords, ["keywords-old"] = event.keywords_old }) end function dovecot_lua_notify_end_txn(ctx) -- report all states for i,msg in ipairs(ctx.messages) do local e = dovecot.event(ctx.event) e:set_name("lua_notify_mail_finished") reqbody = json:encode(msg) e:log_debug(ctx.ep .. " - sending " .. reqbody) res, code = http.request({ method = "PUT", url = ctx.ep, source = ltn12.source.string(reqbody), headers={ ["content-type"] = "application/json; charset=utf-8", ["content-length"] = tostring(#reqbody) } }) e:add_int("result_code", code) e:log_info("Mail notify status " .. tostring(code)) end end
It's also in our Github repo: https://raw.githubusercontent.com/EGroupware/swoolepush/master/doc/dovecot-p...
Ralf
-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
participants (2)
-
Aki Tuomi
-
Ralf Becker