GSoC 2014, Week #6: Hit the mid point

You know, they say that the point of no return is when you've gone so far that turning back just seems impossible.

Typically that sounds pretty dismal. But in this case it's the exact opposite!

This week has been a week of good fortune for me and this is why:

1.) I passed my GSoC midterm evaluations! Which means we've hit the midpoint for the GSoC program.

2.) My main computer is alive again! My computer (a.k.a Buff-Kitty) is finally alive after me destroying tons of computer parts (I don't want to talk about it...). It's been awhile but I can confidently say you were missed, Buff-Kitty.

In celebration of new life, a lolcat shall be posted:

You sure do, Buff-Kitty. You sure do.

Enough of that though. Onto the goodies of the week. So what's new?

Recall what I said my plans for this week were:

• Revamp the overlay definition generator script.
• Polish up the portage sync module.
• Begin adding squashfs overlay support. 

 So what was accomplished this week? To all of you who were ecstatic about having squashfs overlay support well I have good news for you. The good news is that you'll get it...eventually. But not this week. Because I was being doing a complete overhaul on the overlay definition generator script and polishing up the portage sync module to add some nifty new features to it.

Let's start with the sync plugin module since it didn't get an entirely new overhaul.

 1.) Portage Sync Module:

  Recall in my previous post how I explained that the sync module made use of two variable methods of running: the API method, and the subprocess method. By default, the sync plugin should run via the API method, however, if something occurs that prevents this from happening then portage will fall back on the subprocess method. Although to be honest with you I don't know what could happen to prevent you from running a program's API yet, still be able to run the subprocess. That's an aside though.

  "Okay, I remember all of this and I remember that it interacts via a file in your repos.conf folder, big deal, what's new?" Nice to see you're just as impatient as I am. So I'll just cut to the chase. You can tell portage to sync a repo that doesn't exist. Just because the repo is listed in the repos.conf config file it doesn't necessarily mean that the repository is installed. This would cause a failure to sync the repo (for obvious reasons).

  Fear not! For a new feature to the API variant of the plugin has been added. This feature will check the configuration for the repository in the repos.conf config file and create a new overlay definition using the newly revamped overlay-maker class. After creating the new overlay definition.xml file it will then install the new repo for you. Then when you go to sync the repo again, it will successfully sync the repo (for obvious reasons).

2.) Overlay-Maker class:

  Previously this was just a simple script to prompt users for overlay information and then to pipe it into a script. Throughout this week that has changed and the overlay-maker has been turned into a little more than that. By making use of the new overlay-maker class you can make layman overlay definitions and write them to a desired file via a script, or an interactive command line interface. Further more the overlay-maker adds auto-completion support for overlays that have mirrors from github.com or from git.overlays.gentoo.org.

  If given a source URL that is from either of those two locations the script will automatically fill out the rest of the available sources for that overlay along with the homepage and feeds available.

  It is possible that before the end of this GSoC that the overlay-maker will be able to be ran through an array of subparser commands from layman, however even if I am not able to get it accomplished prior to the end of the program, I do hope to get it done so that users can add overlay definitions simply by doing something like:

    layman repo --add twitch153

  This would then bring up the interactive interface to add the overlay by the name of twitch153 and add it to a file of your choice. It would also make it easier for developers maintaining large repository.xml files. If they wanted to complete remove a repo from a large repository.xml file and this method of interaction was possible they could run:

    layman repo --location=repository.xml --del twitch153

  And have the peace of mind of knowing it is gone for good. The other positive aspect of this is that you could then create a script or a cron to add/remove repos from a .xml file. The possibilities are exciting for sure.

  For all of you layman users, during this upcoming week or weekend a new release candidate for layman is going to the portage tree and will be available for all to install and test. So enjoy that and I encourage feedback about it including things you would like to see in the next RC of layman (because there will be another one before the end of this GSoC...hint, hint).

Goals for next week:

• Add finishing touches to portage sync module.
• Continue to work on overlay-maker class.
• Hopefully begin adding squashfs overlay support.

As always, to anyone interested the source code and my commits can be found on git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.

  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [2]https://github.com/twitch153/layman/tree/gsoc2014

Hope everyone is enjoying their summers!

With regards,
    Devan Franchini (twitch153)

GSoC 2014, Week #5: Midterm evals approaching

Hello,

     For everyone who hasn't done a Google Summer of Code or known anyone who has I'll explain a little bit about it shortly.

     For the past ten years, Google has offered a program worldwide that allows students the opportunity to get a stipend to write code for open source organizations. This is Google Summer of Code. In order to get in to the Summer of Code you need to supply a proposal which can consist of your project proposal and a time line that you expect to go by in order to break things into manageable pieces. If you get accepted you then begin your summer project!

    It doesn't mean everything is in the clear just because you were accepted though. Throughout the Summer of Code you have to go through two evaluations:

• The Midterm Evaluation (June 23rd)
• The Final Evaluation      (August 22nd)

Essentially, these are both evaluations from your project mentor that confirm you are keeping up with your proposed time line. If you do not pass either of these, you don't get to fully finish the Summer of Code. You will be removed and you will cry yourself to sleep thinking about what could've been! (Tragic, right?)

    Never fear though, not only am I keeping up with my proposed time line, but I'm actually ahead of schedule (I also hope to stay ahead of schedule).

This following week the following work has been done for layman:

1.) Minimal changes to documentation and code:

    This week's minimal changes include changes to the manual page formatting to make it a little cleaner when being displayed. As well as changes to the way both config.py and overlay.py handle dictionary logic.

    For those of you who have never coded in python allow me to briefly explain what a dictionary is:

    A python dictionary is a data structure that associates a value with a key, not uncommon to that of an actual dictionary. To grab a value from a dictionary in python you can do a few things, but the most common way of grabbing a specific value from a dictionary is by simply grabbing its value as so:

    dictionary['<key>']

    With that slight crash course on python dictionaries allow me to explain the problem with layman's previous dictionary logic: In some cases layman's code would check dictionaries for values by doing something like this -

        if dictionary['key']:
            ....

    This logic has been tested by me in both py2.7 and py3.x and neither times would anything like this actually end up with anything other than KeyError: 'value' if the value wasn't in the dictionary. Part of me believes that this logic might have worked in previous versions of python but I haven't tested it to confirm. My implemented solution was to change the logic to something like this:

        if 'value' in dictionary:
            ....

This will prevent the code from causing runtime errors and will neatly check to see if the value exists in the keys of the dictionary.

2.) Beginning addition of layman portage sync plugin module:

    Going on the idea that if you're using layman then the chances of you using Gentoo are extremely high. Furthermore, the chances of you using portage is just as high, if not higher.

    In my previous posts I explained how to add a layman overlay, but did I ever mention how to sync the overlay? If I did, then...oh well. Under the assumption that I never explained how to sync layman overlays I will explain now. You can either sync all the installed overlays with the command: layman -S or you can sync an individual overlay with the command: layman -s <overlay>. Simple, no?

    This doesn't take into account that you also sync your main portage overlay via emerge though. So when you want to sync secondary overlays you'll have to do sync them via layman and then your main via emerge. If you're anything like me you sync portage very often and regularly forget to sync your layman overlays. So throughout this week I've been in the process of doing two things, the first of which is working on adding a portage sync plugin module for layman.

How this is done:

    Although the current version of portage does not support sync modules, it is in the works to have a version of portage that does support sync modules. So that's why I'm making one for layman. So until that version of portage becomes available on the portage tree I'll lazily say you don't need to know how I added the module. However, I will explain slightly how I created the module itself.

    You need two things:

• The __init__.py
• The module.

    In the __init__.py you need to specify the config_class and the module specifications. To see both, you can look at my source code to get a better look, this part isn't complicated. Even for someone who has never coded before.

    In the module you need to create your config class and have it subclass the SyncBase class. From there you need to specify your __init__() function, your new() function, and your _sync() function. That is the bare minimum requirements for a portage sync plugin.

The layman sync plugin uses two different methods to be ran:

1. Subprocess method
2. API method

Firstly, we'll discuss the more common and simplistic of the two, the subprocess method. Portage already has a built in function to execute processes and this is call the portage.process.spawn_bash() function. This function will also return an exit code upon completion of the process. It's really quite beautiful.

For a user to add a sync plugin it'd be as simple as adding the execution of the subprocess for portage to execute.

Secondly, we'll discuss the API method of the layman sync plugin module. This method is slightly more complex than the previous one but is considerably less complex than some other things.

To make use of layman's API and not require a subproccess to be executed, we added this second config_class. This class creates an instance of layman's API and executes it's built-in sync function directly. Multiple checks have been added to ensure the API will be instantiated when necessary in order to become less resource intensive. Why is that?

Well, consider you have 5 overlays installed on your system. Okay, 5 isn't that many. But if you have portage syncing your layman overlays and say you have 50 overlays instead of 5, it would have to create an instance of the layman API 50 times when it might not be necessary.

3.) Overlay definition generator script in the works:

    If you were to look into layman's manual you would see that you can actually create your own overlay definitions in one or many .xml files and layman will pick them up if they are in the correct directory (default is: /etc/layman/overlays, but can be changed via command line or config setting. Multiple examples are also included in the overlay to guide users into making their own overlay.xml definitions.

    For the ease of use for users and developers both, a tool is being created that allows you create an overlay definition via a script that can be ran in the safety and comfort of your terminals.

4.) A note to all Gentoo users:

    This week we also released a test version of layman into the wild for users to hunt bugs with. So for any brave soul that is willing to test this and report back feedback of any kind, it'd be greatly appreciated.

    To install it on your system, simply install the -9999 version of layman in the portage tree and enjoy! Don't be afraid, it shouldn't bite you.

Goals for next week:

• Revamp the overlay definition generator script.
• Polish up the portage sync module.
• Begin adding squashfs overlay support.

As always, to anyone interested the source code and my commits can be found on git.overlays.gentoo.org/gitweb/[1][1] or on github.com[2] on layman's gsoc2014 branch.


  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [1][1] Because my main computer has been down I haven't been able to push
            my latest commits to this repo.
  [2]https://github.com/twitch153/layman/tree/gsoc2014

GSoC 2014, Week #4: The wheels keep spinning

Hello new and old readers all the same,

    Typically I make my posts every Friday but due other pressing matters unrelated to my coding adventures, I was unable to present my weekly achievements for the week.

    Because I know you guys are all very curious about my computer (because you all love it as much as I do, let's be honest), I tested my old motherboard with my new parts and it was a no-go. So I returned my new motherboard I purchased to Newegg and I should expect a new one within the next week or two. We all know how long these things can take...anyway, I'll be sure to keep all of you updated on how that goes.

    Now on to the good stuff, the main course of the post: What's been done to layman?

1.) Implementation of the "with" statement to open files:

    Sometimes it's great to explain things to people, when the changes you make to your code are very specific to your codebase, in this particular case, it isn't necessarily like that and the implementation is quite common. So common in fact, that I shamelessly found a great link that explains exactly what the "with" statement is used for: http://effbot.org/zone/python-with-statement.htm. The author of this page does an exceptional job to explain why this was implemented in later versions of python.

    The short of it: this was implemented in many places in the codebase of layman to make the opening and closing of a file that layman makes use of "safer", in case something doesn't work correctly for whatever reason, which will prevent leaving unused resources on your system open. So that's something everyone can enjoy!

2.) Replaced "subpath" with "branch":

    If anyone has made use of the subpath variable you're in luck! We removed it from layman's latest code! Why are you in luck? Well, because we replaced it with the branch variable. It was discussed between myself and my mentor (Gentoo developer, dol-sen) that using "subpath" would be mildly misleading for the general behaviour of how this variable is going to work. As most VCS's (version control systems) are git based in one way or another we decided using branch just made more sense. While I am currently in a waiting process for my previously mentioned .dtd patch to be submitted, hopefully everything will work out alright before we push out layman's upcoming release candidate

    Along with this change came the addition of alterations to layman's manual page that comes equipped with some good examples of how to make use of this fancy new variable.

3.) Added module branch support:

    Okay, so we have this fancy new variable we called branch. Big deal. How is this variable going to help me? Well, since you're all dying to know (and haven't figured it out already...) that variable can be used in an overlay.xml file. Once layman catches that this variable has a value, the following VCS types can make use of this variable for branch support:

• Git: When a branch name is given to an overlay that is git based, layman will append the following to the cloning process of the git repo - "-b <branch name>". This allows the user to make custom overlay files that specify repositories and supply them with a particular branch, which is useful if you want the same repository under multiple aliases to be handled by layman.
• Mercurial: Similarly with git, when a mercurial overlay has a branch defined, layman will append the following to the args for mercurial when cloning - "-r <branch name>" which is actually having the user specify a specific revision for mercurial when cloning that repository.
• CVS & Tar: For both CVS and Tar overlay types which previously made use of the "subpath" variable, the behaviour of defining a branch in either of their repo definitions will remain the same and the branch will be treated as a subpath for the repo. This note has also been added to layman's manual page.

Note to all readers: if you think I'm doing something incorrect or think there is some branch love I can give to other repository types, I encourage patches. Of course they'll be tested and everything beforehand, but I will accept any patch and test it to be applied to layman for the greater good of the community.

4.) Refined test code:

    When installing layman you can run tests to ensure the test cases work in accordance with how layman claims it does on your system. layman currently supports both doctest testing, and non-doctest testing via a file called external.py (I can only assume this represents "external testing"). This week I worked on refining the external test code in two ways:

1. I added compatibility for python3.x by changing the "print " declarations to "print()". 
2. One of the test cases is a Unicode test to ensure layman's output is unicode. However, python3.x will not explicitly decode strings when printing them out if they are encoded. So changes were made to ensure that the output from layman when running this test case would remain the same in both python2.7 and python3.x by implicitly decoding the string output.

5.) Adds file:// url support for tar overlays.

    However, when running the test cases I ran into a testing failure that struck me as odd. Until I realized it was partially my fault.

    This test ensures that tar overlays behave correctly by running the add() sync() and delete() functions for a temp created tar overlay. By making use of the tempfile library that python provides you can make....temporary files! (Who would've guessed?) This is particularly useful when doing test cases and just whenever you need to make use of temporary files such as possibly cache and the sort.

    Firstly, this test case initially failed due to the fact that I accidentally removed the tempfile import from the tar.py file (my bad!).

    To my surprise though, the test continued to fail, and it was no longer something that I caused! So why was it failing? Well, thanks to layman's output I was able to trace it down to tar.py's _extract() function. Tar overlays never put into consideration if the user was actually specifying a file:// url instead of an actual url that layman could download a tar from. But this test made use of a file:// temporary.tar.bz2. So I needed to add support for that in tar.py. It was simple enough. If the user specifies file://<tar_name> the just remove the "file://" section from the name and use tar to extract it.

    The test continued to fail though...but why? Where is it failing? By inserting some debug print statements (mostly saying "Are we here yet?!") I was able to see that layman was actually deleting the file after extracting it. Which makes sense if you consider it wasn't supporting local tar overlays, why keep a tar you can just download again on sync? But if we're keeping this tar overlay locally on the machine then that's just no good.

    To fix this issue a boolean was created called "clean_tar". In _extract() it will first check this boolean to see if it is True. If it is, then layman will get the okay to delete the file and purge it from this cruel, cruel, virtual world. For local files, this variable is hard-coded to False, and layman will never have permission to clean up local tar files unless the user themselves modify the code (modifying open source code?! What's wrong with you!?). Fear not though! Users do have some choice in their lives when it comes to this variable. Users can modify the configuration option: clean_tar to either yes or no. If yes, layman will delete downloaded tar overlay files after extracting them, and obviously the opposite effect will happen if users set the option to no; giving them the choice to keep downloaded tar overlays on their system to collect precious hard drive space.

Goals for next week:

• Push out a new release candidate for layman (along with possibly changelog?)
• Begin migrating from a docstring tests to a full test suite.

To anyone interested the source code and my commits can be found on
git.overlays.gentoo.org/gitweb/[1][1] or on github.com[2] on layman's gsoc2014 branch.


  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [1][1] Because my main computer has been down I haven't been able to push
            the commits to this repo. My mentor dol-sen should be pushing these to
            this repo some time this upcoming week.
  [2]https://github.com/twitch153/layman/tree/gsoc2014

 See you next week!

With regards,
    Devan Franchini (twitch153) 

GSoC 2014, Week #3: Going good

Hello again everyone,

This week went fairly smooth. I got my computer parts in and to my despair there is an issue with the motherboard. So! Hopefully this Sunday I'll be able to see if I can't solve this problem and have my main computer alive and functional once and for all!

Not much can be listed as being accomplished but a decent amount of work was done this week regarding layman:

1.) Refined current code:

       I'm sure you're all thinking: "But Devan, didn't you do that last week? How much code could you possibly refine before it's perfect?". Well, forget that. Because code can always be refine in some way. Especially with something as expansive as layman. Throughout this week if I had to think about it I would probably say that I used git far more than I did anything else throughout this week.

      To avoid dragging out ideas and beating them into your brain to the point where your eyes glaze over I'll spare you most of the details on my git adventures and cover the main things that I've been getting familiar with throughout this GSoC involving git.

• Using git rebase: Although I know git rebase is used for far more than I use it for, I have been using git rebase to handle my commits. If I broke something in a previous commit I would find the necessary SHA of the commit before the commit I wanted to repair and then I would run: git rebase -i <SHA>, once run it will bring up a text editor that will list all the commits up to that SHA. If I wanted to reword a commit I would just replace pick with reword, if I wanted to repair my mistakes from any commit I would replace pick with edit. This would also allow me to reword a commit if I so desired by using the command git commit --amend. After finishing all my repairs I would run git rebase --continue which will take me to my next edit, reword,... until it hit pick which will just continue on until you successfully rebase. After doing that you have to push all your commits to the branch by running git push --force and you'll have fixed your commits and pushed them to your remote repository!

•  Breaking up a commit into two commits: So say I made a commit that I thought at the time was good but then I realized it'd be better as two separate commits. Well, I would rebase and edit the commit I wanted to break up into two (or more). Once you get to that commit you should run git reset HEAD~ which will reset your current HEAD to the one prior to that commit. So now your changes are "unrecorded". You can then continue to make your changes and break them up into multiple commits as you would normally. Once you're all done simple run git rebase --continue, and feel good about yourself for doing a job well done!

By making use of these two topics I've been fixing up and refining the code without having to make different commits admitting I messed up (I mean, come on! Who wants to admit that they forgot an indentation here or there on multiple occasions?).

2.) Updated the repository.dtd file:

       Although my understanding and experience with XML is slim to none, I can at least explain what was done here and why. If you look here you will see the overlay where the layman overlay repositories get their XML definitions. The structure of the repositories.xml is defined in its .dtd file. The changes I made to the repository.dtd include changes to add information for IRC channels, and overlay branches.

3.) Documentation updates:

       If you're using layman then it's extremely likely that you're also using Linux. More specifically, Gentoo. So you should know that if you run man layman you'll get the manual for layman which includes pretty much all you need to know about how layman works, and the configuration options you can modify to do certain things.

      In preparation for the new release of layman, the documentation has been modified to reflect the changes I've been adding to layman throughout these past weeks and some cleaning up for the man page as well.

Goals for next week:

• Finish adding branch support for overlay installations (testing included).
• Get the patch I created for the .dtd modifications to be applied to the repository.
• Prepare for a new layman release.

To anyone interested the source code and my commits can be found on
git.overlays.gentoo.org/gitweb/[1][1] or on github.com[2] on layman's gsoc2014 branch.


  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [1][1] Because my main computer has been down I haven't been able to push
            the commits to this repo.
  [2]https://github.com/twitch153/layman/tree/gsoc2014

 See you next week!

With regards,
    Devan Franchini (twitch153)