GSoC 2014, Week #2: Slight woes

Hello readers,

This week started with a bumpy start.

Essentially, any horrible thing that could happen with my main desktop, ended up happening. All short from a micro-nuke dropping on the computer.

So that ate up about one and a half days of possible work that I could've put toward layman while I worked on trying to diagnose whether it was a dead CPU or a motherboard issue. I ended up getting a CPU/Motherboard combo to save time diagnosing and ensuring that I'll have a working main desktop by next week. Fingers crossed that nothing else happens and I throw something...

On a brighter note, a lot of work was done this week and I am not really far behind on my timeline at all (thanks to a few days of working longer than usual).

Although the list of what was accomplished isn't as long as last week (for a multitude of reasons) I accomplished what I set out to accomplish.

The following is what I accomplished this week:

1.) Added a re-add overlay command line option:

      In my last post I showed you how to add an overlay using layman, this new command line option will do something similar (hopefully self-explanatory as well).

2.) Added overlay type checking:

       In the event that your repository changes overlay types (ie. from git to cvs) layman will have to handle that. Previously no existing code in layman was set to handle this event. Now, when syncing overlays via the command line option: layman -S or layman -s <overlay>, layman will check the overlay type of the overlay on your local machine and the remote overlay that goes by the same name.

      Under that circumstance, layman will give a warning to the user, notifying them of the change and will then proceed to re-add the overlay with the code provided by the re-add function.

3.) Added overlay URL updating capabilities for all supported repository types:

      Adding the support for this was a fun activity that gave me a little insight on how layman handles different overlay actions as well as the methods that one could take when updating the repository:

      To start, I'll begin by explaining the inheritance of the classes and how this allows them to interact in a neat fashion.

Below, I will depict the inheritance tree for you:

                                     Overlay
                                           |
                                      Source
                                           |
                                   Overlay type

So for example, we want to add my overlay to a local machine with layman (<sarcasm>because I don't do much to the ebuilds it'll ensure longterm stability! </sarcasm>) we will invoke layman as so:

        layman -a twitch153

Hopefully you remember this. Now I'm going to explain in further detail as to how it all works (not entirely, I'm kind of lying to you).

The overlay object will call it's add function, which goes through each available source and attempts to add them by calling source.add():

     def add(self, base):
            res = 1
            first_s = True
            for s in self.sources:
                if not first_s:
                    self.output.info("\nTrying next source of listed sources...", 4)
                try:
                    res = s.add(base)
                    if res == 0:
                        # Worked, throw other sources away
                        self.sources = [s]
                        break
                except Exception as error:
                    self.output.warn(str(error), 4)
                first_s = False
            return res

This then traverses into the source class where you look to see what happens:

    def add(self, base):
        '''Add the overlay.'''

        mdir = path([base, self.parent.name])

        if os.path.exists(mdir):
            self.output.error('Directory ' + mdir +
                ' already exists. Will not overwrite its contents!')
            return False

        os.makedirs(mdir)
        return True

layman will check to see if the path already exists and to prompt the user if it does. So where does the actual action of syncing the overlay come in?

The trick behind it is by looking back at the Overlay class:

    res = s.add(base)

This doesn't really go straight to the source class. In fact, if we were to look at this in a level type scheme, the Overlay class would be level 1, and the Overlay_type class would be your level 2. So what about source? Well, source would be considered your level 1.5.

The overlay_type class inherits all the functions given to it by the Source class:

    class GitOverlay(OverlaySource)

Doing this allows any overlay type to make use of the code in the Source class while also executing its own add() function.

After all of this, what does that have to do with URL updating? Now that you understand the inheritance behind each overlay_type it allows us to make sense of how each overlay type can cleanly have its own updating methods.

The overlay class has a list of available overlay types that allow for a custom method of URL updating, these available types (Bzr, Cvs, Git, Mercurial, and Subversion) all run their own separate functions to update their URLs.

Bzr: To update the repository URL in a Bzr overlay you need to run the bind command:

    bzr bind <source_url>

Cvs: To update the repository URL in a Cvs overlay you need to echo the new repostiory URL into the CVS/Root file of the overlay, and then (if the repository location has changed) you need to replace the old repository location with the new one.

Git:  By running git remote set-url <remote name> <new_url> <old_url> you can update the URL from which git will pull from.

Mercurial: You can update the URL for a mercurial overlay by replacing the old url with the new url in the .hg/hgrc of the overlay. In layman, we make use of sed to do this:

     sed -i 's/oldurl/newurl/' <overlay_location>/.hg/hgrc

Subversion: To update the src-url of an svn overlay you simply need to run svn switch --relocate, followed by the old_url and then the new_url.

For all other overlay types, layman syncs the overlay by grabbing the source url from the installed.xml file in /var/lib/layman/, so when the overlay class recognizes an that an overlay type isn't in the list of supported types, it will simply set the source of the object to one of the new sources reported by the remote database and re-sync the overlay with the newly reported source url.

4.) Polished code already in layman:

A little self explanatory as well, but I cleaned up some of my mess along the way this week as well so you guys can all enjoy clean and bug free (hopefully...) code!

Goals for next week:

• Continue to refine the code already in layman - This goal will be ongoing throughout the entire GSoC and will never stop throughout my entire life. I will always have to go back and clean up my messes.
• Add updates to the DTD to add IRC channel information for overlays in the repository.xml.
• Hopefully add custom config type plugin support (it's being put off...I know).

To anyone interested the source code and my commits can be found on
git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.


  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [2]https://github.com/twitch153/layman/tree/gsoc2014

See you guys next week!

With regards,
    Devan Franchini (twitch153) 

GSoC 2014, Week #1: Off to a good start

Hello everybody,

This year I am working with the Gentoo organization again on a quest to improve the overlay manager, layman.

Before I go into this week's success let me give you a little understanding about what layman is.

Essentially, layman is a tool that manages overlays that can contain experimental or third-party packages for Gentoo, while integrating these overlays into the main system. It can also serve as a manager for version control repositories.

To further embed the idea, imagine you are running a Gentoo system and want to add one of these third party overlays so you can install one of their packages because it's not on Gentoo's main distribution.

You would first install layman on your system and then run:

layman -L:

         * triquetra                 [Git       ] (git://git.o.g.o/user/triquetra.git, http://git.o.g.o/gitroot/user/triquetra.git, git+ssh://git@git.o.g.o/user/triquetra.git                               )
         * tryton                    [Mercurial ] (http://www.tryton.org/hg/tryton-overlay/                                                                                                                  )
         * turbogears2               [Git       ] (git://git.o.g.o/proj/turbogears2.git, http://git.o.g.o/gitroot/proj/turbogears2.git, git+ssh://git@git.o.g.o/proj/turbogears2.git                         )
         * twitch153                 [Git       ] (git://github.com/twitch153/ebuilds.git                                                                                                                    )
         * ub0rlay                   [Git       ] (git://repo.or.cz/ub0rlay.git                                                                                                                              )

This would give you a list of all the available overlays that you can install. I just snagged a piece out to show you.

They are listed by name, repo type, and repo url.

From here you'll notice that I have an overlay on there! So I want to add that overlay to inflate my already large ego by calling layman again:

        layman -a twitch153

After this is accomplished you install the packages that are on my repository.


However, this is not a crash course on layman, but instead it is a post on the progress that I've made so far throughout this week along with some of the knowledge I acquired by working on this project.

1.) python2.7/python3.x compatibility:

   
      The current version of layman on the portage tree only has support for python2.7. To add longevity into the life of layman it needed to have python3.x compatibility. So over the span of a few days I added support by doing the following:

•   Running a python tool called 2to3, this will point all things (plus maybe even more than you'd need) to get a python program up and running for python3.x. By running this I was able to get some of the little stuff out of the way, but more needed to be done.

• Migration of optparse to argparse:

       In order to parse out the command line arguments you give python a class was created called optparse. However, in python3.6, it is going to be deprecated, so for the sake of longevity migration to another class which is almost identical to optparse was necessary. argparse is pretty similar aside from a few minute details that I noticed (there are probably more, but in layman's case, I noticed these):
    - Option groups: Managing and handling option groups with argparser was much easier than with optparser, simply assign a variable to the add_argument_group(groupname):

          actions = self.parser.add_argument_group('<Actions>')

 and add an argument to the group:

        actions.add_argument('-a', '--add', ...)

for length reasons I'm not going to go over everything, but I will say that argparser is a nice alteration to optparse and it was much easier to migrate than I thought it'd be. So two thumbs up there!

• SSL-Fetch code migration:

     The ssl-fetch is a python library created in order to conveniently download things with verified ssl connections.

    The migration was simple enough, since the code was originally from layman, but made to be a little more broad so other packages could use it as well. This was added to allow ssl downloads of overlays for layman's remote database as well as tar archived overlays.

   The source code for ssl-fetch can be found here:
       https://github.com/dol-sen/ssl-fetch

 2.) Repos.conf support has been added:

In any Gentoo system, if you look at a file located in /etc/portage/repos.conf/gentoo.conf

You will find a configuration file that looks very similar to this format:

[DEFAULT]
main-repo = gentoo

[gentoo]
location = /usr/portage
sync-type = rsync
sync-uri = rsync://rsync.gentoo.org/gentoo-portage

However, this config type is not currently supported by layman and instead there is a file in /var/lib/layman/ called make.conf, its contents are the directories of every overlay installed by layman and looks like this:

PORTDIR_OVERLAY="
/var/lib/layman/twitch153
$PORTDIR_OVERLAY
"
This tells portage which sources the make.conf in /var/lib/layman that there is an overlay at this directory that also holds packages for portage.

repos.conf support has been added to allow layman to write to a config file that follows the same format of gentoo.conf. For example, if I had repos.conf support and installed the twitch153 overlay it would be added to a file called /etc/portage/repos.conf/layman.conf and look like so:

[twitch153]
sync-type = None
auto-sync = No
sync-uri = git://github.com/twitch153/ebuilds.git
location = /var/lib/layman/twitch153


This support was added by making use of the configparser library. When an overlay object is passed down to the reposconf.py file it will break the overlay object into the pieces it needs (overlay name, sync-type, auto-sync, sync-uri, and the location in which it is installed) and then it will add those values to the config file layman.conf via a ConfigParser object.

ex.)

repo_config = ConfigParser.ConfigParser()

repo_config.add_section(overlay.name)
repo_config.set(overlay.name, 'location', <installed location>)
...

return write()

This simple to use interface allows the code to become easy to read and maintainable for anyone who is unfamiliar with layman's codebase.

However, now layman has more than one configuration option: make.conf, and repos.conf. How does layman know which config option to execute?
This leads us into the next topic...

3.) RepoConfManager:

In order to handle multiple config type plugins such as make.conf and repos.conf a class was created that would interface with the databasing portion of layman, repoconfmanager.py.

This portion of the work I did throughout this week was something I typically don't do. Through a good portion of my coding career I have done my own python coding in a procedural style so I typically don't work a lot in python classes. That is going to change for sure, but that doesn't mean that what I've learned isn't new to me.

So how does RepoConfManager handle multiple config types? Well, you need initialized functions of the available configuration objects:

        import layman.reposconf as reposconf
        import layman.makeconf as makeconf

This code imports the classes to allow me to create objects of that type.

Then (this is what I was really impressed by), a python dictionary is created with these values:

    modules = {'make.conf': (makeconf, 'ConfigHandler'), 'repos.conf': (reposconf, 'ConfigHandler')}

and it is then run through a for loop of the given config types.

        for types in conf_types:
            config = getattr(modules[types][0], modules[types][1])(config, overlays)

this beautiful loop creates a config object of any associated modules of that type by associating the object string, and the class name with the module types in a tuple. From there you can continue with any function that the object holds.

There are a few things about this method of instantiating an object that are great:

1. It allows the code to be easily maintainable.
2. It allows for custom config type plugins. 

Although no support for custom config type plugins is added yet, I'm sure before the end of this summer there will be another post announcing the addition of that feature ;)

So what am I currently working on? What new improvements should you expect to see next week?

One thing that I spent that last day or two working on is this:

4.) URL Updating:

What is it?

Well, remember how layman has to grab the repositories from an overlay list? When the overlay is installed on the system the source url is recorded to a file called installed.xml, along with a few other locations in which this link is recorded.

Why does this matter?

The source url is necessary in order to sync the contents of the overlay and keep them up to date. In certain cases though something can happen that creates a situation where your recorded local source url is not the same as the remote source url.

When this happens it needs to be corrected. Currently layman's code supports notifying a user if their local source differs, so why shouldn't it be corrected for them instead? That's exactly what I have been working on and plan to continue working on into next week.

I have added support for URL updating of git repositories, but as of right now that's all I've accomplished. That leaves me with about 9 more repository types to add URL updating for...

To end this post I'll give you a list what to expect for next week:

•  Continuing to refine code already in layman.
• Adding url update support for more overlay types.
• Hopefully custom config type plugin support for RepoConfManager.

To anyone interested the source code and my commits can be found on
git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.


  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [2]https://github.com/twitch153/layman/tree/gsoc2014


I'll see you guys next week!

With regards,
    Devan Franchini (twitch153)