WindfluechterNet Blog

Ok, my last version was a pure Bash solution: working, but slow. There were some comments how to improve the performance and I decided finally to reimplement the second script as Python script.
The Bash script didn’t finish within a day. The Python script ends after 1-2 hours in my test scenario. So, here are the scripts again:

make-lsLR.sh – call this regularly (cron) to create the needed files that are stored in /root/. Of course you can alter the location easily and exclude other directories from being scanned. [download make-lsLR.sh]
[code]
#!/bin/bash
#
# Usage: ./make-lsLR.sh
#
# Purpose:
# to make a file that is parseable for recovering
# a filled /lost+found directory by parsing
# filesize, md5sum, permisions and path+filename
#
# Author: Ingo Juergensman – http://blog.windfluechter.net
# License: GPL v2, see http://gnu.org for details.
#
# first: get all directories
nice -15 find / -path /sys -prune -o
-path /proc -prune -o
-path /var/lib/backuppc -prune -o
-path /var/spool/squid -prune -o
-type d -printf “%U:%G %#m ” -print > /root/ls-md5sum-dirs.txt

exit 0

# next: get all relevant information
nice -15 find / -path /sys -prune -o
-path /proc -prune -o
-path /var/lib/backuppc -prune -o
-path /var/spool/squid -prune -o
-type f -printf “%s %U:%G %#m ”
-exec nice -15 md5sum {} ; | tr -s ” ” > /root/ls-md5sum-files.txt
[/code]

check_lost+found.py – The second script is to be run when your fsck managed to mess up with your files and stored them into lost+found directory. It takes 3 arguments: 1) the source directory where your messed up lost+found directory is, 2) the target directory to which the data will be saved and 3) a switch to actually make it happen instead of a dry-run. [download check_lostfound.py.sh]
[code]
#!/usr/bin/python
#
# usage: check_lostfound.py [make_it_so]
#
# Purpose: to find files in lost+found and trying to restore
# original files by comparing ls-md5sum-files.txt (generated by
# make-lsLR.sh
# Option make_it_so cause the data actually being written/moved
# whereas the script runs in dry mode per default.
#
# Author: Ingo Juergensman – http://blog.windfluechter.net
# License: GPL v2, see http://gnu.org for details.
#

from string import *
import sys, os
from os.path import *
import string

if len(sys.argv)<>4:
print “usage: “+sys.argv[0]+” ”
sys.exit(1)
else:
mnt = sys.argv[1]
target = sys.argv[2]
mode = sys.argv[3]

dirs=”/root/ls-md5sum-dirs.txt”
source=”/root/ls-md5sum-files.txt”
lost=”/root/lostfound-files.txt”
print “Creating list of files in %s/lost+found” % mnt
cmd = ‘find %s/lost+found -type f -printf “%%s %%U:%%G %%#m ” -exec nice -15 md5sum {} ; > /root/lostfound-files.txt’ % mnt
os.system(cmd)

d = open(dirs, ‘r+’)
f = open(source, ‘r+’)
l = open(lost, ‘r+’)

sfiles={}
lfiles=[]

# create the missing directories first
print “Creating missing directories in %s” % target
for entry in d:
ugid=string.split(entry, ” “, 2)[0]
perm=string.split(entry, ” “, 2)[1]
pfad=”%s%s” % (target,string.split(replace(entry, ”
“, “”), ” “, 2)[2:][0])
res = isdir(pfad)
if (os.path.exists(pfad) and os.path.isdir(pfad)):
print “%s exists… ” % pfad
else:
cmd = “%s %s – mkdir %s” % (ugid, perm, pfad)
print cmd
try:
if mode == “make_it_so”:
os.makedirs(pfad)
uid, gid = string.split(ugid, “:”)
os.chown(pfad, int(uid), int(gid))
os.chmod(pfad, int(perm))
except:
print “%s exists… ” % pfad

# now parse /root/ls-md5sum-files.txt to get
# md5sum and /path/to/filename pairs
for line in f:
line=replace(line, ”
“, “”)
#size1 = split(line)[0]
#ugid1 = split(line)[1]
#perm1 = split(line)[2]
md5s1 = string.split(line)[3]
#path1 = split(line)[4]
sfiles[md5s1] = strip(str(string.split(line,” “, 4)[4:][0]))

# next: do the same to the files in lost+found
for line in l:
size2 = string.split(line)[0]
ugid2 = string.split(line)[1]
perm2 = string.split(line)[2]
md5s2 = string.split(line)[3]
path2 = strip(str(string.split(replace(line, ”
“, “”),” “, 4)[4:][0]))
s = “%s %s %s %s %s” % (md5s2, ugid2, perm2, size2, path2)
lfiles.append(s)

# finally look at lost+found and copy the files
# to the appropriate place. Instead of copying
# the files can be moved as well, but to copy is
# safer in case of mistakes or errors
for lf in lfiles:
md5s = string.split(lf)[0]
lfile = string.split(lf, ” “, 4)[4:][0]
if sfiles.has_key(md5s):
if os.path.exists(target+sfiles[md5s]):
pass
else:
targetfile = sfiles[md5s]
print “restoring %s%s ” % (target, targetfile) #lfile
cmd = ‘cp -p “%s” “%s%s”‘ % (lfile, target, targetfile)
if mode == “make_it_so”:
os.system(cmd)
del sfiles[md5s]

[/code]

I’ve chosen to copy the files to a different place instead of moving them within the same filesystem to their original place for safety reasons. Primary goal is to retrieve the files from lost+found, not to replace a full featured backup and restore application. Because of this the script doesn’t handle hard- nor symlinks correctly. It just copy files.

Of course there’s still room for improvements, like handling hard-/symlinks correctly or using inode number instead of md5sums to move data back to its prior location. But it works for me[tm] well enough in this way, so I’m satisfied so far. You’re welcome, though, to improve this piece of ugliness if you like.

Maybe someone else finds this usefull as well. Use it on your own risk, of course. 🙂