GSoC 2014, Week #4: The wheels keep spinning

Hello new and old readers all the same,

    Typically I make my posts every Friday but due other pressing matters unrelated to my coding adventures, I was unable to present my weekly achievements for the week.

    Because I know you guys are all very curious about my computer (because you all love it as much as I do, let's be honest), I tested my old motherboard with my new parts and it was a no-go. So I returned my new motherboard I purchased to Newegg and I should expect a new one within the next week or two. We all know how long these things can take...anyway, I'll be sure to keep all of you updated on how that goes.

    Now on to the good stuff, the main course of the post: What's been done to layman?

1.) Implementation of the "with" statement to open files:

    Sometimes it's great to explain things to people, when the changes you make to your code are very specific to your codebase, in this particular case, it isn't necessarily like that and the implementation is quite common. So common in fact, that I shamelessly found a great link that explains exactly what the "with" statement is used for: http://effbot.org/zone/python-with-statement.htm. The author of this page does an exceptional job to explain why this was implemented in later versions of python.

    The short of it: this was implemented in many places in the codebase of layman to make the opening and closing of a file that layman makes use of "safer", in case something doesn't work correctly for whatever reason, which will prevent leaving unused resources on your system open. So that's something everyone can enjoy!

2.) Replaced "subpath" with "branch":

    If anyone has made use of the subpath variable you're in luck! We removed it from layman's latest code! Why are you in luck? Well, because we replaced it with the branch variable. It was discussed between myself and my mentor (Gentoo developer, dol-sen) that using "subpath" would be mildly misleading for the general behaviour of how this variable is going to work. As most VCS's (version control systems) are git based in one way or another we decided using branch just made more sense. While I am currently in a waiting process for my previously mentioned .dtd patch to be submitted, hopefully everything will work out alright before we push out layman's upcoming release candidate

    Along with this change came the addition of alterations to layman's manual page that comes equipped with some good examples of how to make use of this fancy new variable.

3.) Added module branch support:

    Okay, so we have this fancy new variable we called branch. Big deal. How is this variable going to help me? Well, since you're all dying to know (and haven't figured it out already...) that variable can be used in an overlay.xml file. Once layman catches that this variable has a value, the following VCS types can make use of this variable for branch support:

• Git: When a branch name is given to an overlay that is git based, layman will append the following to the cloning process of the git repo - "-b <branch name>". This allows the user to make custom overlay files that specify repositories and supply them with a particular branch, which is useful if you want the same repository under multiple aliases to be handled by layman.
• Mercurial: Similarly with git, when a mercurial overlay has a branch defined, layman will append the following to the args for mercurial when cloning - "-r <branch name>" which is actually having the user specify a specific revision for mercurial when cloning that repository.
• CVS & Tar: For both CVS and Tar overlay types which previously made use of the "subpath" variable, the behaviour of defining a branch in either of their repo definitions will remain the same and the branch will be treated as a subpath for the repo. This note has also been added to layman's manual page.

Note to all readers: if you think I'm doing something incorrect or think there is some branch love I can give to other repository types, I encourage patches. Of course they'll be tested and everything beforehand, but I will accept any patch and test it to be applied to layman for the greater good of the community.

4.) Refined test code:

    When installing layman you can run tests to ensure the test cases work in accordance with how layman claims it does on your system. layman currently supports both doctest testing, and non-doctest testing via a file called external.py (I can only assume this represents "external testing"). This week I worked on refining the external test code in two ways:

1. I added compatibility for python3.x by changing the "print " declarations to "print()". 
2. One of the test cases is a Unicode test to ensure layman's output is unicode. However, python3.x will not explicitly decode strings when printing them out if they are encoded. So changes were made to ensure that the output from layman when running this test case would remain the same in both python2.7 and python3.x by implicitly decoding the string output.

5.) Adds file:// url support for tar overlays.

    However, when running the test cases I ran into a testing failure that struck me as odd. Until I realized it was partially my fault.

    This test ensures that tar overlays behave correctly by running the add() sync() and delete() functions for a temp created tar overlay. By making use of the tempfile library that python provides you can make....temporary files! (Who would've guessed?) This is particularly useful when doing test cases and just whenever you need to make use of temporary files such as possibly cache and the sort.

    Firstly, this test case initially failed due to the fact that I accidentally removed the tempfile import from the tar.py file (my bad!).

    To my surprise though, the test continued to fail, and it was no longer something that I caused! So why was it failing? Well, thanks to layman's output I was able to trace it down to tar.py's _extract() function. Tar overlays never put into consideration if the user was actually specifying a file:// url instead of an actual url that layman could download a tar from. But this test made use of a file:// temporary.tar.bz2. So I needed to add support for that in tar.py. It was simple enough. If the user specifies file://<tar_name> the just remove the "file://" section from the name and use tar to extract it.

    The test continued to fail though...but why? Where is it failing? By inserting some debug print statements (mostly saying "Are we here yet?!") I was able to see that layman was actually deleting the file after extracting it. Which makes sense if you consider it wasn't supporting local tar overlays, why keep a tar you can just download again on sync? But if we're keeping this tar overlay locally on the machine then that's just no good.

    To fix this issue a boolean was created called "clean_tar". In _extract() it will first check this boolean to see if it is True. If it is, then layman will get the okay to delete the file and purge it from this cruel, cruel, virtual world. For local files, this variable is hard-coded to False, and layman will never have permission to clean up local tar files unless the user themselves modify the code (modifying open source code?! What's wrong with you!?). Fear not though! Users do have some choice in their lives when it comes to this variable. Users can modify the configuration option: clean_tar to either yes or no. If yes, layman will delete downloaded tar overlay files after extracting them, and obviously the opposite effect will happen if users set the option to no; giving them the choice to keep downloaded tar overlays on their system to collect precious hard drive space.

Goals for next week:

• Push out a new release candidate for layman (along with possibly changelog?)
• Begin migrating from a docstring tests to a full test suite.

To anyone interested the source code and my commits can be found on
git.overlays.gentoo.org/gitweb/[1][1] or on github.com[2] on layman's gsoc2014 branch.


  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [1][1] Because my main computer has been down I haven't been able to push
            the commits to this repo. My mentor dol-sen should be pushing these to
            this repo some time this upcoming week.
  [2]https://github.com/twitch153/layman/tree/gsoc2014

 See you next week!

With regards,
    Devan Franchini (twitch153)