I came across a small annoying problem when upgrading my Debian headless 'server' one time : dpkg failed to remount read-write enabled my /boot partition or it was out of space and consequently the update of grub/initramfs did not work, letting my old server unable to reboot by itself. (yes I know, I could have confgured /etc/apt/apt.conf to work around the read-only /boot partition).

Since I'm really a lazy guy and I had some free time, I tried to find a way other than to simply reconnect the headless server to a screen, keyboard and mouse(?) in order to fix GRUB and the /boot partition.

I chose to use a small LiveCD, as the one provided by Slitaz, to debug the server remotely (by the network). The basic idea is to put the LiveCD into the CD tray, wait for the OS to launch a pre-configured sshd server where I would be able to log-in to gain root access to my Debian headless broken server.

The following instructions are derived from the very good Slitaz cookbook and handbook, available on the official website. More specifically, the hacking steps are from the Handbook/Hack a LiveCD page.

Here is how I have created the slitaz-rescue LiveCD.

Getting a minimal LiveCD ISO

You can directly download a stable or cooking version of Slitaz. Personally, I went for a minimal "flavor" called slitaz-2.0-base available from this page.

Mount and copy the ISO, we will modify some files inside.

mkdir iso 
fuseiso slitaz-2.0-base.iso iso 
cp -av iso rootCD 
fusermount -u iso

Then uncompress the initramfs image:

mkdir rootfs && cd rootfs 
su -c 'lzma -S "gz" -dc ../rootCD/boot/rootfs.gz | cpio -id'

And chroot inside. As root, type :

$> ls 
rootCD rootfs slitaz-2.0-base.iso 
$> cp /etc/resolv.conf rootfs/etc/resolv.conf 
$> cp /etc/hosts rootfs/etc/hosts 
$> mount -o bind /dev rootfs/dev 
$> mount -o bind /proc rootfs/proc 
$> chroot rootfs /bin/ash

Here come the real work. First, enable the SSH server, named dropbear, in the /etc/rcS.conf. Use vi or nano to edit the file from inside the slitaz chroot. Second, disable the "hwconf.sh" script to avoid the prompt for sound and display settings. You should have something like this at the end:

# Daemons to start at boot time. SliTaz only provides a few daemons: firewall, 

# Web server (lighttpd), SSH server (dropbear) and rsyncd, so boot order is 

# not really important, but dbus/hald should be started before slim. 

RUN_DAEMONS="dbus dropbear hald firewall lighttpd slim" 


# Initialization scripts to run at boot time. Boot order is important, 

# bootopts.sh (boot options) must start first, then you are free to choose. 

# Note that the local.sh script exists to let you quickly add some local startup 

# commands. 

RUN_SCRIPTS="bootopts.sh i18n.sh network.sh local.sh" 

# hwconf.sh disabled

Exit the chroot environment, add you public SSH key in the common user "tux" $HOME, and finally replace the isolinux.cfg default configuration file by the 'en.cfg to avoid the keyboard configuration prompt.

mkdir -p rootfs/etc/skel/.ssh 

cp ~myaccount/.ssh/id_rsa.pub rootfs/etc/skel/.ssh/authorized_keys 

cp rootCD/boot/isolinux/{en,isolinux}.cfg

Rebuild the ISO

... without forgetting to unmount /dev and /proc before !

umount rootfs/dev 
umount rootfs/proc 
cd rootfs 
su -c 'find . -print | cpio -o -H newc | lzma -z >../rootCD/boot/rootfs.gz'

And eventually generate a new .iso file.

genisoimage -R -o slitaz-rescueCD.iso \ 
	-b boot/isolinux/isolinux.bin \ 
	-c boot/isolinux/boot.cat \ 
	-no-emul-boot -boot-load-size 4 \ 
	-input-charset iso8859-1 \ 
	-boot-info-table \ 
	-V Slitaz-RescueCD \ 
	rootCD

Test it !

If you wish, you can test your ISO with QEMU, uml-utils{tunctl} and a local DHCP server as follow :

su -c 'tunctl -u <myuser> 
su -c 'ifconfig tap0 192.168.66.1/24 up' 
su -c 'dhcpd tap0' 
qemu -net nic -net tap,ifname=tap0,script=no,downscript=no -cdrom slitaz-rescueCD.iso &
ssh -i rescue_id_rsa tux@192.168.66.10

N.B.: this will work if you add to your dhcpd.conf the following subnet declaration :

subnet 192.168.66.0 netmask 255.255.255.0 { 
	option routers 192.168.66.1; 
	option broadcast-address 192.168.66.255; 
	range 192.168.66.10 192.168.66.100; 
}

The final rescueCD ISO file is attached to this post. Beware that you must have the right ssh rescueCD private key to get access to the LiveCD dropbear server.

/DS

P.S.: There is a million of other possible solutions. You could have just changed the tux password on the LiveCD albeit I don't know how to do this. You could have used a bootp/pxe server to autoboot from the network, add avahi into the LiveCD to retrieve the server IP more easily ... or you could just connect a video card, a screen and a keyboard.