GSoC 2014, Week #1: Off to a good start

Hello everybody,

This year I am working with the Gentoo organization again on a quest to improve the overlay manager, layman.

Before I go into this week's success let me give you a little understanding about what layman is.

Essentially, layman is a tool that manages overlays that can contain experimental or third-party packages for Gentoo, while integrating these overlays into the main system. It can also serve as a manager for version control repositories.

To further embed the idea, imagine you are running a Gentoo system and want to add one of these third party overlays so you can install one of their packages because it's not on Gentoo's main distribution.

You would first install layman on your system and then run:

layman -L:

         * triquetra                 [Git       ] (git://git.o.g.o/user/triquetra.git, http://git.o.g.o/gitroot/user/triquetra.git, git+ssh://git@git.o.g.o/user/triquetra.git                               )
         * tryton                    [Mercurial ] (http://www.tryton.org/hg/tryton-overlay/                                                                                                                  )
         * turbogears2               [Git       ] (git://git.o.g.o/proj/turbogears2.git, http://git.o.g.o/gitroot/proj/turbogears2.git, git+ssh://git@git.o.g.o/proj/turbogears2.git                         )
         * twitch153                 [Git       ] (git://github.com/twitch153/ebuilds.git                                                                                                                    )
         * ub0rlay                   [Git       ] (git://repo.or.cz/ub0rlay.git                                                                                                                              )

This would give you a list of all the available overlays that you can install. I just snagged a piece out to show you.

They are listed by name, repo type, and repo url.

From here you'll notice that I have an overlay on there! So I want to add that overlay to inflate my already large ego by calling layman again:

        layman -a twitch153

After this is accomplished you install the packages that are on my repository.


However, this is not a crash course on layman, but instead it is a post on the progress that I've made so far throughout this week along with some of the knowledge I acquired by working on this project.

1.) python2.7/python3.x compatibility:

   
      The current version of layman on the portage tree only has support for python2.7. To add longevity into the life of layman it needed to have python3.x compatibility. So over the span of a few days I added support by doing the following:

•   Running a python tool called 2to3, this will point all things (plus maybe even more than you'd need) to get a python program up and running for python3.x. By running this I was able to get some of the little stuff out of the way, but more needed to be done.

• Migration of optparse to argparse:

       In order to parse out the command line arguments you give python a class was created called optparse. However, in python3.6, it is going to be deprecated, so for the sake of longevity migration to another class which is almost identical to optparse was necessary. argparse is pretty similar aside from a few minute details that I noticed (there are probably more, but in layman's case, I noticed these):
    - Option groups: Managing and handling option groups with argparser was much easier than with optparser, simply assign a variable to the add_argument_group(groupname):

          actions = self.parser.add_argument_group('<Actions>')

 and add an argument to the group:

        actions.add_argument('-a', '--add', ...)

for length reasons I'm not going to go over everything, but I will say that argparser is a nice alteration to optparse and it was much easier to migrate than I thought it'd be. So two thumbs up there!

• SSL-Fetch code migration:

     The ssl-fetch is a python library created in order to conveniently download things with verified ssl connections.

    The migration was simple enough, since the code was originally from layman, but made to be a little more broad so other packages could use it as well. This was added to allow ssl downloads of overlays for layman's remote database as well as tar archived overlays.

   The source code for ssl-fetch can be found here:
       https://github.com/dol-sen/ssl-fetch

 2.) Repos.conf support has been added:

In any Gentoo system, if you look at a file located in /etc/portage/repos.conf/gentoo.conf

You will find a configuration file that looks very similar to this format:

[DEFAULT]
main-repo = gentoo

[gentoo]
location = /usr/portage
sync-type = rsync
sync-uri = rsync://rsync.gentoo.org/gentoo-portage

However, this config type is not currently supported by layman and instead there is a file in /var/lib/layman/ called make.conf, its contents are the directories of every overlay installed by layman and looks like this:

PORTDIR_OVERLAY="
/var/lib/layman/twitch153
$PORTDIR_OVERLAY
"
This tells portage which sources the make.conf in /var/lib/layman that there is an overlay at this directory that also holds packages for portage.

repos.conf support has been added to allow layman to write to a config file that follows the same format of gentoo.conf. For example, if I had repos.conf support and installed the twitch153 overlay it would be added to a file called /etc/portage/repos.conf/layman.conf and look like so:

[twitch153]
sync-type = None
auto-sync = No
sync-uri = git://github.com/twitch153/ebuilds.git
location = /var/lib/layman/twitch153


This support was added by making use of the configparser library. When an overlay object is passed down to the reposconf.py file it will break the overlay object into the pieces it needs (overlay name, sync-type, auto-sync, sync-uri, and the location in which it is installed) and then it will add those values to the config file layman.conf via a ConfigParser object.

ex.)

repo_config = ConfigParser.ConfigParser()

repo_config.add_section(overlay.name)
repo_config.set(overlay.name, 'location', <installed location>)
...

return write()

This simple to use interface allows the code to become easy to read and maintainable for anyone who is unfamiliar with layman's codebase.

However, now layman has more than one configuration option: make.conf, and repos.conf. How does layman know which config option to execute?
This leads us into the next topic...

3.) RepoConfManager:

In order to handle multiple config type plugins such as make.conf and repos.conf a class was created that would interface with the databasing portion of layman, repoconfmanager.py.

This portion of the work I did throughout this week was something I typically don't do. Through a good portion of my coding career I have done my own python coding in a procedural style so I typically don't work a lot in python classes. That is going to change for sure, but that doesn't mean that what I've learned isn't new to me.

So how does RepoConfManager handle multiple config types? Well, you need initialized functions of the available configuration objects:

        import layman.reposconf as reposconf
        import layman.makeconf as makeconf

This code imports the classes to allow me to create objects of that type.

Then (this is what I was really impressed by), a python dictionary is created with these values:

    modules = {'make.conf': (makeconf, 'ConfigHandler'), 'repos.conf': (reposconf, 'ConfigHandler')}

and it is then run through a for loop of the given config types.

        for types in conf_types:
            config = getattr(modules[types][0], modules[types][1])(config, overlays)

this beautiful loop creates a config object of any associated modules of that type by associating the object string, and the class name with the module types in a tuple. From there you can continue with any function that the object holds.

There are a few things about this method of instantiating an object that are great:

1. It allows the code to be easily maintainable.
2. It allows for custom config type plugins. 

Although no support for custom config type plugins is added yet, I'm sure before the end of this summer there will be another post announcing the addition of that feature ;)

So what am I currently working on? What new improvements should you expect to see next week?

One thing that I spent that last day or two working on is this:

4.) URL Updating:

What is it?

Well, remember how layman has to grab the repositories from an overlay list? When the overlay is installed on the system the source url is recorded to a file called installed.xml, along with a few other locations in which this link is recorded.

Why does this matter?

The source url is necessary in order to sync the contents of the overlay and keep them up to date. In certain cases though something can happen that creates a situation where your recorded local source url is not the same as the remote source url.

When this happens it needs to be corrected. Currently layman's code supports notifying a user if their local source differs, so why shouldn't it be corrected for them instead? That's exactly what I have been working on and plan to continue working on into next week.

I have added support for URL updating of git repositories, but as of right now that's all I've accomplished. That leaves me with about 9 more repository types to add URL updating for...

To end this post I'll give you a list what to expect for next week:

•  Continuing to refine code already in layman.
• Adding url update support for more overlay types.
• Hopefully custom config type plugin support for RepoConfManager.

To anyone interested the source code and my commits can be found on
git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.


  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [2]https://github.com/twitch153/layman/tree/gsoc2014


I'll see you guys next week!

With regards,
    Devan Franchini (twitch153)