Wednesday, June 25, 2014

GSoC week 5

Nofsync
I created another implementation of nofsync plugin (disables fsync(), makes it much faster), this time in python as DNF plugin that disables fsyncing in the YumDB. It is a small bit slower than the C library using LD_PRELOAD, because it doesn't eliminate fsyncs made from scriptlets (by gtk-update-icon-cache and such). But it's much simpler from packaging perspective (mock can stay noarch) and could be actually upstreamable (in dnf), because there are some other use cases, where you don't try to recover from hardware failure anyway - for example anaconda. If the power goes down, you probably don't try to resume existing installation. And this could make it faster (nofsync makes package installation approximately 3 times faster).
To compare the two implementations, set either
config_opts['nofsync'] = 'python'
or
config_opts['nofsync'] = 'ld_preload'
Default is python, to disable it, set the option to something else (empty string)

LVM support
Last week I implemented base for LVM plugin for mock using regular snapshots. This week I rewrote the plugin to use LVM Thin snapshots, which offer better performance, flexibility and share the space with the original volume and other snapshots, therefore don't waste much space. I created basic commands that can be used to manipulate the snapshots.
Example workflow:
I'll try to demonstrate how building different packages can be faster with LVM plugin. Let's repeat the configuration options necessary to set it up:
config_opts['plugin_conf']['root_cache_enable'] = False
config_opts['plugin_conf']['lvm_root_enable'] = True
config_opts['plugin_conf']['lvm_root_opts'] = {
    'volume_group': 'my-volume-group',

}
You can now also specify 'mount_options', which will be passed to -o option of mount. To set size to larger than the default 2GB, use for example 'size': '4G' (it is passed to lvcreate's -L option, so it can be any string lvcreate will understand). Now let's initialize it:
$ mock --init
Mock will now create thin pool with given size, create a logical volume in it, mount it and install the base packages into it. After the initialization is done, it creates a new snapshot named 'postinit', which will be then used to rollback changes during --clean (which is by default also executed as part of --rebuild). Now try to install some packages you often use for building your own packages. I'm a Java packager and almost every Java package in Fedora requires maven-local to build.
$ mock --install maven-local
But now since I want to rebuild more Java packages, I'd like to make snapshot of the buildroot.
$ mock --snapshot mvn
This creates a new snapshot of the current state and sets it as the default. We can list snapshots of current buildroot with --list-snapshots command (the default snapshot is prefixed with asterisk)
$ mock --list-snapshots
Snapshots for mock-devel:
  postinit
* mvn


So let's rebuild something
$ mock --rebuild jetty-9.2.1-1.fc21.src.rpm
$ mock --rebuild jetty-schemas-3.1-3.fc21.src.rpm
Because the 'mvn' snapshot was set as the default, it means that each clean executed as part of the rebuild command didn't return to the state in 'postinit', but to the state in 'mvn' snapshot. And that was the reason we wanted LVM support in the first place - it didn't have to install 300+MB of maven-local's dependencies again (with original mock, this would probably take more than 3 minutes) but still the buildroot was cleaned of the packages pulled in by previous build. We could then install some additional packages, for example eclipse, and make a snapshot that can be used to build eclipse plugins.
Now let's pretend there has been an update to my 'rnv' package, which is in C and doesn't use maven-local.
$ mock --rollback-to postinit
$ mock --list-snapshots
  mvn
* postinit
Now 'postinit' snapshot was set as default and buildroot has been restored to the state it was in when 'postinit' snapshot was taken (after initialization, no maven-local there). The 'mvn' snapshot is retained and we can switch back again using --rollback-to mvn.
So now I can rebuild my hypothetical rnv update. If I decide that I don't need the 'mvn' snapshot anymore, I can remove it with
$ mock --remove-snapshot mvn
You cannot remove 'postinit' snapshot. To remove all logical volumes belonigng to the buildroot, use mock --scrub lvm
 
So that's it. You can create as many snapshots as you want (and snapshots of snapshots) and keep a hierarchy of them to build packages that have different sets of BuildRequires.
Few more details:
  • The real snapshot names passed to LVM commands have root name prefixed to avoid clashes with other buildroots or volumes that don't belong to mock at all. It also checks whether the snapshots belong to mock's thinpool.
  • The volume group needs to be provided by user, mock won't create one. It won't touch anything else besides the thinpool, so it should be quite safe if it uses the same volume group as you system (I have it like that).
  • The command names suck. I know. I'll try to provide short options for them.
  • If you try the version in my jenkins repository, everything is renamed to xmock including the command - to allow it to exist alongside original mock.

No comments:

Post a Comment