This week I've been mostly continuing the parts I started the week before. I performed more drastical refactoring - splitting the main monolithic class (Root) that does basically everything except few specific features that are delegated to other modules. I moved rest of the buildroot-building code to Buildroot class I created before for this purpose. The state logging and plugin loading/calling was decoupled to separate classes. The package management code resides in PackageManager class and Yum/Dnf subclasses. I renamed the former Root class to Commands which now does just what the name suggests - executes commands, such as build, buildsrpm, in buildroot. I've adapted plugins to the new model and customized the initialization to prepare field for the LVM backend, which will of course be implemented as a plugin (but doing that without hardcoding some parts into core will require enhancing the plugin API).
Jenkins
I've requested a Jenkins project for my improved version of mock which is now available at http://jenkins.cloud.fedoraproject.org/job/mock/ and provides built packages (and repository) at http://jenkins.cloud.fedoraproject.org/job/mock/ws/RPMS. I build it using the spec file in the mock source tree but before building I inject a script into %prep which replaces all occurences of 'mock' with 'xmock' (including filenames). Thus, the package can be installed alongside upstream mock without any conflicts. Just everything is renamed to xmock - the binary, the config directory, also the group. So feel free to test it :-)
(But be careful, mock is running as root. If I make a mistake it may have consequences on your system)
DNF
I've already got feedback from my mentor (Mikolaj Izdebski). The bad news is that dnf installroot support doesn't work for him althought it works perfectly fine for me. The problem is that for him, dnf doesn't load the config from the installroot and uses the system-wide one. I've read the corresponding part of dnf's source code and I don't see any reason why it might fail to find the configuration. If installroot is defined and the config file within is readable (tested with access(2)) it will be used. I'm yet to find why it doesn't work for him.
Interactive output
I've modified mock to print output of building and package management commands. One thing my mentor suggested was to get more output from dnf/yum. Currently, neither dnf nor yum output anything when they're synchronizing repos or downloading packages, only when installing. The reason is that they check whether the output is a terminal (with os.isatty()), which is not - in case of mock, it's a pipe. I had to trick it to them to think the output is a tty. I solved it using a pseudoterminal instead of a pipe and now os.isatty() happily returns True in the child processes and dnf now prints everything including the progressbars when downloading. But since it uses carriage retruns and backspaces to erase parts of already printed output it also introduced lot of mess to the logs, because in text files those characters aren't interpreted the same way as on terminal. So I also had to modify the output logging to get rid of these.
Note: to get reasonable output from yum/dnf, it's debuglevel has to be set to at least 2 in the config (current default is 1)
Skipping package verification
Another feature he requested was speeding up mock by skipping verification of packages when installing. Neither yum nor dnf have an option to disable those, but since they're written in Python, there's always a hackish way to accomplish what you need. There are two solutions to this problem: 1. create a plugin that modifies yum/dnf to not verify packages 2. create a wrapper module that will modify them. I chose number 1 and implemented it for dnf for now. I created a simple plugin that, once loaded, rebinds the dnf's method verify_transaction to a no-op lambda. Then I just copy it into buildroot and inject the plugin path to dnf.conf. It can be toggled with config option 'noverify' (enabled by default).
Other
Also, I fixed some corner case behavior. Now it doesn't fail when you delete the /var/lib/mock directory. Previously it recreated it with wrong permissions making it hard to detect, why it failed and how to make it work again (user had to manually set the setgid bit on the directory). Also it prints warning when incorrectly executed as regular user without setuid wrapper (Previously it printed OSError: Success).
Future
I've been exploring the possibility of mock executing commands in a contained environment and possibly not doing most of it's work as root. I've discovered some interesting things about Linux namespaces that might quite change the way mock works. I will try to make a follow up post about this soon.
Jenkins
I've requested a Jenkins project for my improved version of mock which is now available at http://jenkins.cloud.fedoraproject.org/job/mock/ and provides built packages (and repository) at http://jenkins.cloud.fedoraproject.org/job/mock/ws/RPMS. I build it using the spec file in the mock source tree but before building I inject a script into %prep which replaces all occurences of 'mock' with 'xmock' (including filenames). Thus, the package can be installed alongside upstream mock without any conflicts. Just everything is renamed to xmock - the binary, the config directory, also the group. So feel free to test it :-)
(But be careful, mock is running as root. If I make a mistake it may have consequences on your system)
DNF
I've already got feedback from my mentor (Mikolaj Izdebski). The bad news is that dnf installroot support doesn't work for him althought it works perfectly fine for me. The problem is that for him, dnf doesn't load the config from the installroot and uses the system-wide one. I've read the corresponding part of dnf's source code and I don't see any reason why it might fail to find the configuration. If installroot is defined and the config file within is readable (tested with access(2)) it will be used. I'm yet to find why it doesn't work for him.
Interactive output
I've modified mock to print output of building and package management commands. One thing my mentor suggested was to get more output from dnf/yum. Currently, neither dnf nor yum output anything when they're synchronizing repos or downloading packages, only when installing. The reason is that they check whether the output is a terminal (with os.isatty()), which is not - in case of mock, it's a pipe. I had to trick it to them to think the output is a tty. I solved it using a pseudoterminal instead of a pipe and now os.isatty() happily returns True in the child processes and dnf now prints everything including the progressbars when downloading. But since it uses carriage retruns and backspaces to erase parts of already printed output it also introduced lot of mess to the logs, because in text files those characters aren't interpreted the same way as on terminal. So I also had to modify the output logging to get rid of these.
Note: to get reasonable output from yum/dnf, it's debuglevel has to be set to at least 2 in the config (current default is 1)
Skipping package verification
Another feature he requested was speeding up mock by skipping verification of packages when installing. Neither yum nor dnf have an option to disable those, but since they're written in Python, there's always a hackish way to accomplish what you need. There are two solutions to this problem: 1. create a plugin that modifies yum/dnf to not verify packages 2. create a wrapper module that will modify them. I chose number 1 and implemented it for dnf for now. I created a simple plugin that, once loaded, rebinds the dnf's method verify_transaction to a no-op lambda. Then I just copy it into buildroot and inject the plugin path to dnf.conf. It can be toggled with config option 'noverify' (enabled by default).
Other
Also, I fixed some corner case behavior. Now it doesn't fail when you delete the /var/lib/mock directory. Previously it recreated it with wrong permissions making it hard to detect, why it failed and how to make it work again (user had to manually set the setgid bit on the directory). Also it prints warning when incorrectly executed as regular user without setuid wrapper (Previously it printed OSError: Success).
Future
I've been exploring the possibility of mock executing commands in a contained environment and possibly not doing most of it's work as root. I've discovered some interesting things about Linux namespaces that might quite change the way mock works. I will try to make a follow up post about this soon.
No comments:
Post a Comment