Finally, I submitted my GSoC final evaluation. In GSoC terms, the project is over. After some starting problems (GSoC starts parallel to the end of the semester in Germany, and I was also finishing my Bachelor thesis in the first two weeks), everything went more or less well.
I finished the project to some extent. Everything I could do without touching other tools too much is done. The two scripts work well (and the tests on my real NetBSD machines will follow this week), though they would be nicer if some other tools (namely, mtree(8), etcupdate(8) and pkg_info(8)) had some additional functionality.
I came out with two scripts:
sysbackup. You can see the results in the repository.
A small description of the term set in NetBSD: If you install NetBSD, you unpack several sets. Sets differ in size, but everyone with less than 100MB. They are named like
sysupdate will check for updates or update your system. If you start it with command
check, it will download mtree specifications from a NetBSD server and compare them to either the system one's or the file system. If there are differences, an update is proposed, as well as 3rd party packages that are depending on libraries which would be changed are shown.
Then, you can issue the command
fetch, which will fetch the needed setfiles from a NetBSD mirror. You can also directly say
sysupdate will unpack the fetched setfiles to a staging directory, backup your old system (see below for
sysbackup) and then copy these setfiles over to your real system, proposing an update of 3rd party packages.
If there is also a kernel update, the kernel will be copied to
/netbsd (or something else you specify) and your
/boot.cfg will be changed to be able to boot the old kernel (being copied before overwriting it).
This is not the most efficient method. You could also have tars containing just the diffs, instead of downloading all the data from one set. But this way, there is no need to change anything about the release management (which would entail a discussion lasting longer than GSoC), it just needs access to one NetBSD mirror to extract the mtree files and hash and sign them.
Additionally, sysupdate features the
install command, which lets you install a range of sets you feed (e.g., when you built them yourself).
sysbackup was the second, “optional” part of the project. It has two modes: soft and hard. A soft backup needs a list of files which should be backed up. Then, these files are tared up and the backup will be noted in a database.
Hard backups will require knowledge about the tool before the installation of NetBSD, as it requires a very specific setup. A hard backup is a copy of the file system to another one (different partition or hard disk). Then, this partition is made bootable. As the boot order cannot be seen by a running operating system, you have to reboot from that different partition or hard disk yourself by choosing the right one in your BIOS, Firmware or whatever you use if you want to boot the old one.
As it would be very inefficient to backup everything, you have to specify excludes and sharemounts.
An exclude is a file system (or directory) which will not be copied to the new system, and won't be mounted there.
/usr/pkg is a good example if you make a hard backup before jumping to a new major release – as packages are usually bound to a specific base system version.
A sharemount is a file system (not a directory!) which should be shared across different NetBSD versions.
/usr/pkgsrc, or even better,
/srv are good examples, since they are version-independent.
Unfortunately, you cannot specify a simple directory. You have to have your own file system for a sharemount, since determining the correct order of mounting with symlinks, nullfs mounts etc. make it impossible to resolve the exact position of a directory in POSIX sh.
So you have to think about how you partition your hard disk at the installation if you want to use sysbackup!
sysbackup also has the commands
status to check the backups you have and the dates,
clean to clean old entries in the database which do not exist anymore, and
rollback to roll back to a specific backup (which won't work for hard backups yet).
I have to see how to continue. I need a small addition to test(1) to commit, and then I have to publish the scripts for public review, improve (or even abandon) them and maybe, eventually commit them. Until then, I will also write a more thorough tutorial for the NetBSD wiki and add more polish to the manpages (I think, I forgot an XXX somewhere).
There are some parts missing and it's not as functional as it could be, but these would require major modifications in other tools. For the beginning, I wanted to stay as unobstrusive as possible. But these will come in over time. If the scripts get committed, I'd like to do that before 7.0, and maybe place it for the old ones in pkgsrc.
I have to talk to jmmv, who wrote a similar tool named
sysupgrade earlier, but with some problems imho (I wrote about that before GSoC beginning). I also have to talk to releng about the creation of the files I need on a server. I can surely do that for releases myself, but it would have some delay (the creation takes several hours), and for NetBSD-current packages on http://nyftp.netbsd.org it would not be possible.
But let's see. Maybe other developers will refuse the script, or my mentor (Brett Lymn, blymn) is unhappy with me. Whom I would like to thank much btw for the assistance during GSoC!