Coding in the Big Peep's world: August 2014

Tuesday, August 19, 2014

GSoC 2014, Week #13: All good things must come to an end

Hello everyone,

As you might already know, this is the last post for my Google Summer of Code endeavour for this year. That said, that doesn't mean that this last week was spent doing nothing.

1.) Pull Requests, merged:

All my previous pull requests for features that I wanted to add to layman have been merged with master. This includes the overlay-maker utility, squashfs-support, the portage sync plug-in, the ability to disable and enable a repository, modular overlay types, and more!

2.) Goodbye G-Common:

G-common was an overlay type that was supported by Gentoo. It was decided that g-common was no longer going to be supported because no one (that we can tell) uses it. Typically you wouldn't see this type of thing happening, but g-common was replaced by g-sorcery. Which is the latest variant of that tool.

I personally have no experience with either VCS, but I have spoken to the maintainer of g-sorcery.

3.) Modular Configuration types:

Remember the RepoConfManager class? The class that handles all the repo configuration configuring has now been modified to support config type modules. So what does that mean? It means if a user has a configuration method that they want to support for layman they only need to add the module to layman's codebase and add the option to their layman.cfg.

The same can be said for layman overlay types. If a user would like layman to support an overlay type that's not currently supported then all they need to do is add the module. If it works well, you could then submit a bug report and submit a patch with the new module and it might be added to layman on the next release.

4.) Documentation changes:

With new features comes new documentation. This should always be the case, always. So users can be aware of the latest and greatest things that have been done to their newest release. In preparation for a new release, the man pages have been modified, the layman,cfg has been modified, and the version has been bumped to 2.2.0!

5.) Code refining:

As I had said previously, you will never stop refining your code. I will bug squash till the day I die and I'll love every second of it too. After the documentation changes were made, the live ebuild (9999) for layman was updated as well. So users could now install (and test) the latest codebase for layman and return feedback to me to allow me to improve upon my code, or fix mistakes previously missed. A few bugs here and there have been found and squashed throughout the span of this week.

Goals for next week:

Go to School, get my education on.
Enjoy this last week of Summer.

I would like to say thank you to everyone who assisted me throughout the span of this program and I thoroughly enjoyed myself working on layman, in an effort to contribute to the Gentoo community. As I said above, the layman-9999 ebuild is available for people to test, and I welcome all feedback!

With regards,
Devan Franchini (twitch153)

Sunday, August 10, 2014

GSoC 2014, Week #12: Starting to wrap everything

Hello faithful readers!

As the title says, things are beginning to get wrapped up nicely with a little bow and everything.

A lot of time this week was spent on trying to get all my pull requests pushed to layman's master branch. Two notable ones (in my opinion) that got merged into the master branch were the overlay-maker tool and the portage sync plug-in. However, a few things needed to be changed to get these features up to snuff, particularly the overlay-maker.

1.) Overlay maker auto-completion improvements:

    Recall previously that I discussed that the overlay-maker has auto-completion for source URLs? Well the code for that was admittedly very horrid to go through. Not very clean, not very easily maintained.

In order to improve upon that, I took the original concept and changed the implementation almost completely.

    Originally, if you were to give the overlay-maker the source URL of "git://git.overlays.gentoo.org/proj/layman.git" a series of string modifications were made to turn that source into two other source URLs along with the homepage and two separate feeds.

    To do this, there was a function called _set_gentoo_info() or something very similar to that. Sounds a little complex, but not really. Right? Right. I would agree with you. But imagine if you wanted to add support for another source URL for auto-completion of information? And then another? If we go under the assumption that the _set_gentoo_info() function was roughly 45-50 lines of code and we wanted to add two more source URLs for support then that's an additional 45-50 lines of code per source mirror!

    That's what we call "spaghettified code" and it's not pretty, and not always necessary.

    So for the sake of space, maintainability, and pride in your ability to code you want to take something like that down from 135-150 lines to 60 lines of code. Maintain the idea of modifying the URLs and feeds, and the homepage based on a specific URL mirror such as github, gentoo, or bitbucket. Then realize what differs from one github URL to another github URL. Those are your variables.

    After that it's as simple as creating a template constant that includes the auto-completion information for you. Once that's implemented then it's as simple as breaking up the URL into the variables you need and then perform interpolation on the template to get the correct overlay information.

   To anyone interested in seeing that concept in action then you should take a peak at: https://github.com/twitch153/layman/commit/7537f0b6216c2cbb13e2059edd9be0ed8e19a097

2.) Overlay-maker CLI interface:

Up until now whenever a user wanted to tell the overlay-maker that they wanted to have auto-completion support for their overlay then they had to specify on every overlay creation. That can get tedious and sometimes annoying if you're working on adding a large amount of overlays to an xml. In favor of ease of use and not wanting users to curse my name at the end of their keyboard I gave users the ability to simply specify this via command line arguments.

You can now do the following:

List all mirrors supported for auto-completion.
Tell the overlay-maker you only want the bare basics for the overlay(s) you are creating.
Select a mirror for auto-completion when creating your overlay.
Override certain checks when creating an overlay such as only saving the overlay.xml in your "overlay_defs" location and giving an overlay a name of an overlay that already exists.

After both these changes were made to the overlay-maker then it was time that the pull-request was merged into master. It's only a matter of time before a new release is available to users (wink, wink).

3.) Merged test-suite and updated-updater:

Both of these branches were merged to master along with the other two.

    The changes made to the test-suite were simple things like modifying what gets checked to assure that the test is working out exactly as it should.

   The changes made to updated-updater were also minimal. Now updater.py will check to see which conf_types are enabled and will then create the proper configuration files based on that.

    This brought the pull-request count down from 7 to 3! Which just shows how much closer we are to completion.

4.) Added layman-mounter tool:

    Recall in the past (we seem to do that a lot recently) that I added squashfs overlay support to layman. Well, it turns out that the method I used to add the support by unpacking the archive wasn't what my mentor had in mind (whoops!). After a little bit of discussion we came to the agreement that we wanted to add squashfs support to layman so that it would mount it read-only.

    The benefit of this is that you save space on your system. The drawback is that once you reboot your computer the overlay is no longer mounted.

   To address that drawback a utility script was created called "layman-mounter". This sounds a little...odd for a tool. What will this do? Well, it's not that odd. Really. I swear. And it does the following:

lists all mountable overlays.
lists the status of all mountable overlays (mounted/unmounted).
Can mount overlay(s).
Can unmount overlay(s).

The beauty of that is that a user can then have layman-mounter invoked on every reboot to mount all your overlays. Or only select ones. The squashfs overlay code was also changed to reflect the behavior of being mounted instead of extracted. So look for this handy tool on the next release.

Be aware that this method of squashfs overlay support does require you to have CONFIG_SQUASHFS enabled in your kernel though.

Goals for next week:

Work on getting the last three pull-requests merged with layman's master branch.
Add modular configuration plug-ins.
Refine any code necessary in order to get a new release out before the end of the program.

As always, to anyone interested the source code and my commits can be found on git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.

[1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
[2]https://github.com/twitch153/layman/tree/gsoc2014

With regards,
Devan Franchini (twitch153)

Friday, August 1, 2014

GSoC 2014, Week #11: Code Refinement at its finest

Hello again everyone,

This week was a fairly busy week due to my birthday and all the celebration that revolved around that.

Don't worry though! I didn't completely slack off. Work was accomplished.

1.) Modular overlay type plug-ins:

    If you can recall from my previous posts, I explained a little about the SquashFS overlay type as well as making the code for both Tar and SquashFS overlay types common in one class called ArchiveOverlay. If any of that doesn't sound familiar, no worries. I'll explain as we go along. But knowing of that will help you understand a bit better of what I'm about to discuss.

    This week I implemented a system in layman that will interact with modules. This commit: https://github.com/twitch153/layman/commit/0d4523513464c9bce836308d085995cb8a24644b is the module plug-in system and essentially what it does is look in a specified directory and find all available plug-ins. From there it will become a container of sorts that will obtain the class of any module, by name.

    This logic was then implemented into layman's overlay types. Recall wanting to install the overlay "twitch153". You would invoke layman as so: layman -a twitch153 and it would install (as long as you have git installed on your system). So you need to have git on your system and layman just magically takes care of it! Vioala!

    Well, in reality, you need git on your system because layman is informed that the type of the overlay is "git". The Overlay class will then find the proper class associated with "git", which in turn would be "GitOverlay". From there it would go to that class' add() function to see what steps would be necessary to add the overlay.

    As always, we strive to know more about why things happen. So why oh why am I explaining this to you and wasting the precious time you could be spending looking at lolcats and looking at other people's pictures of their food on their Facebook? Simple. Now that the module plug-in system has been implemented the Overlay class no longer needs to keep track of the available overlay type classes. This makes the main code much more maintainable, as well as making the addition of a new overlay type (such as squashfs) as simple as adding it to the correct directory and configuring it correctly.

This also has another benefit to it. If this gets implemented into the main version of layman (which I would hope it does), it would make it possible to split out the modules into their own separate packages. This follows the Gentoo mentality of having choice. In previous versions of layman (even the current 2.1.0), you have specific USE flags that determine which VCSs you would add support for in layman. So if you were to install layman with USE='-darcs' you would not emerge the darcs version control system as a dependency. You would, however still have the code in layman that handles darcs overlays.

This goes against the mentality of choice and forces users to have something on their system that they might never use. This is why the nature of modular overlay-type plug-ins is a great thing.

That's all that was accomplished this week. I do hope to work a little more next week to try to pick up some of the slack I left behind because of my birthday. But hey, if you can't celebrate your own birthday then I feel bad for you.

Goals for next week:

Work on anything I need to in order to get the overlay-maker and the sync-plugin into the master branch of layman.
Discuss splitting the layman overlay-type plug-ins.
Refine layman's codebase.