Rebuilding SIS attachment links from log

[Daniel Miller]

That works brilliantly!  I revised my script now to take advantage of 
yours -

#!/bin/bash

# These variables need to be customized for your particular installation
LISTFILE='/var/mail/files.fail'
ATTACHMENT_STORAGE_BASE='/var/files/attachments'

# These variables are based on current Dovecot behaviour and should not 
require changing
HASH_FOLDER='hashes'

function usage
{
     echo "Dovecot Single-Instance-Storage Attachment Repair"
     echo "usage: dovesisfix [-d] [-t] [-v] [-h]"
     echo "   -t | --test-only   perform logfile analysis and show steps 
to be taken without any on-disk modification"
     echo "   -v | --verbose     provide verbose messages at each step"
     echo "   -d | --debug               provide additional debug messages"
     echo "   -h | --help                this screen"
}

while [ "$1" != "" ]; do
     case $1 in
         -d | --debug )          DEBUG=1
                                 VERBOSE=1
                                 ;;
         -t | --test-only )      TESTMODE=1
                                 ;;
         -v | --verbose )        VERBOSE=1
                                 ;;
         -h | --help )           usage
                                 exit
                                 ;;
         * )                     usage
                                 exit 1
     esac
     shift
done

while read -r LINE
do
     # Now extract the aa/bb/ prefix, the base attachment file name, and 
user hash
     CATEGORY_PATH="${LINE:0:5}"
     BASE_HASH="${LINE:6:40}"
     USER_HASH="${LINE:47:32}"

ATTACH_SOURCE="$ATTACHMENT_STORAGE_BASE/$CATEGORY_PATH/$HASH_FOLDER/$BASE_HASH"
ATTACH_TARGET="$ATTACHMENT_STORAGE_BASE/$CATEGORY_PATH/$BASE_HASH-$USER_HASH"

     # If in debug/verbose mode show operation about to occur
     if [ "$VERBOSE" = 1 ]; then
         echo "The file $ATTACH_SOURCE must be linked to $ATTACH_TARGET"
     fi

     # Verify that source exists
     if [ ! -f "$ATTACH_SOURCE" ]; then
         echo "ERROR: File $ATTACH_SOURCE does not exist. You must 
restore this from a backup and run this utility again."
     fi
     # This is a Good Thing.
     if [ "$DEBUG" = 1 ]; then
         echo "The file $ATTACH_SOURCE appears to be a valid file."
     fi

     # Check if user link mysteriously reappeared
     if [ -f "$ATTACH_TARGET" ]; then
         echo "INFO: File $ATTACH_TARGET exists. This may mean the fault 
has been previously corrected. Clearing/rotating the logfile $LOGFILE is 
appropriate now."
         continue
     fi

     # Prepare to create user link
     LINK_LINE="$ATTACH_SOURCE $ATTACH_TARGET"
     if [ "$DEBUG" = 1 ]; then
         echo "About to execute command: ln $LINK_LINE"
     fi

     # If test mode, do nothing
     if [ "$TESTMODE" = 1 ]; then
         continue
     fi

     # There's probably more tests I could/should do - but I don't know how
     # So...if we're not in test mode...time to do it to it.
     LINK_CREATED=$(ln $LINK_LINE)
     if [ "$VERBOSE" = 1 ]; then
         echo "Repair result for $ATTACH_TARGET - $LINK_CREATED"
     fi

done < "$LISTFILE"





On 3/4/2015 9:29 AM, Hardy Flor wrote:
> I use this script for review. For a correct result Dovecot should not 
> run.
>
> #!/bin/bash
>
> attdir="/var/files/attachments"
> maildir="/var/mail"
>
> cd "$attdir" || exit 1
> find -type f -printf "%P\n" | grep -v "hashes" | sort -u 
> >"$maildir/attachments.s"
> cd "$maildir" || exit 2
> rm "$maildir/files.ok" "$maildir/files.fail" 2>/dev/null
> for f in $(find -type f -name "m.*" -printf "%P\n"); do
>   doveadm dump -t dbox "$f" | egrep "^msg.ext-ref" | while read z; do
>     set -- $z
>     while [ -n "$1" ]; do
>       if [[ $1 == */* ]]; then
>         test -r "$attdir/$1" && echo "$1" >>"$maildir/files.ok" || 
> echo "$1" >>"$maildir/files.fail"
>       fi
>       shift
>     done
>   done
> done
> sort -u "$maildir/files.ok" >"$maildir/files.s"
> diff -Nu "$maildir/attachments.s" "$maildir/files.s" | tee 
> "$maildir/files.diff" | egrep "^\-"
>
>
>
> Am 03.03.2015 um 22:08 schrieb Daniel Miller:
>> This seems simple enough...I'm just not script wizard.  If someone 
>> can throw together a starting point I can test and tweak it from 
>> there.  It seems to me:
>>
>> 1.  Read /var/mail/mail.err or specified logfile
>> 2.  For each "failed: 
>> read(/var/mail/attachments/aa/bb/attachmentHash-userHash" line,
>>     a. Confirm /var/mail/attachments/aa/bb/hashes/attachmentHash exists
>>         i. If attachmentHash is missing display such for possible 
>> backup searching.
>>     b. create link attachmentHash-userHash to hashes/attachmentHash
>> 3.  Continue to end of file
>>
>> Can this be done via "pure" BASH?  Need sed/awk as well?
>>

-- 
Daniel L. Miller, VP - Engineering, SET
AM Fire & Electronic Services, Inc. [AMFES]
dmiller at amfes.com 702-312-5276