GSoC 2014, Week #9: Testing, testing, 1, 2, 3...

Hello everyone!

Good news! My mentor, Brian Dolbec (aka, dol-sen) has returned from the hospital! He has to take things slow but at least we know he made it out alive. Best wishes for him and I hope everything goes well for him.

In an attempt to make things as short as possible here is what has been accomplished this week:

1.) Improvements to overlay-maker class:

1. Support for bitbucket.org mirror auto-fill info has been added. When a user now gives the overlay-maker class a bitbucket.org source URL it will now try to determine which VCS type it is, as well as the possible feed and homepage URLs.
2. In addition to adding the support for bitbucket.org auto-fill info, the function necessary to auto-fill the info for source URLs has been cleaned up for improved maintainability.
3. The get_input() and get_ans() functions were improved upon, to allow for a colorful interface with users, similar to the output messages of layman. This was done for overlay uniformity and to just be...pretty.
4. Message output for the overlay-maker class has been migrated from your run-of-the-mill print() functions to the output suite that layman has readily available. Again, this brings a uniform look to the interactive sessions.
5. The get_ans() and get_input() functions were moved to utils.py. This allows them to be where they should belong, as well as making them readily available to other classes that can make use of them (ex: db.py).

2.) Added disable and enable functionality to repos.conf files:

    Do you all remember in my last post where I mentioned the ability to disable and enable a repository in the make.conf files? Well, I discussed it over with dol-sen and we figured the best way to "disable" a repo for repos.conf files is to simply comment it out.

    Because that is the case, adding support was easy and is now readily available to be merged onto the master git branch,

3.) Began migration from doctests to test suite:

    Essentially a doctest is an in ".py" file test that was run via the python command line. It it a test that can be run to ensure that the function that is being tested, actually works. The drawback of doctests (in my humble opinion) is that they should belong in a separate place aside from the actual ".py" file that is being tested.

    If we're moving the tests to a separate place, why not just move them from doctests to the actual implementation of the test?

    That's what I aim to accomplish in this upcoming week (and what I've been working on this week).

    To see a full list of tests added to the test suite, look at the test-suite branch on my layman github repo.

 Goals for next week:

•  Continue working on migrating docstring tests to a full test suite.

As always, to anyone interested the source code and my commits can be found on git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.

  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [2]https://github.com/twitch153/layman/tree/gsoc2014

With regards,
    Devan Franchini (twitch153)

GSoC 2014, Week #8: Squashing some bugs

Hello everyone!

This week has been good in making leeway with open bugs for layman!

To all of you who are unaware of what bugs are (no, not the squirmy wormy ones), a bug is essentially a report regarding a certain program or package or pretty much whatever. It can cover a vast amount of things.

If you are curious to see the open bug for layman, follow this link here:
https://bugs.gentoo.org/buglist.cgi?quicksearch=layman&list_id=2406998

But I'm getting ahead of myself, I'm going to begin by discussing what has been accomplished this week for layman.

1.) Centralize archive overlay type code:

    Last week I added support for squashfs overlay types for layman and it struck me: "Hrm, the code for both tar overlays and squashfs overlays are super similar. So similar, in fact that the differences are less than the similarities. I bet I could make a class that all archive types get their main functions from". So that's exactly what I did!

    I created the ArchiveOverlay class which took all the similar code from both archive overlay types and brought it together for any archive overlay type to grab from. It's like a goody bag of archive functions.

   So now any archive overlay type will now subclass from the ArchiveOverlay instead of the SourceOverlay class, which the ArchiveOverlay class subclasses from. It's a large hierarchy that is as follows:

                             <archive overlay type being installed>
                                                           |
                                                     Overlay
                                                           |
                                          <archive overlay type>
                                                           |
                                                ArchiveOverlay
                                                           |
                                                 SourceOverlay
                                                           |
                                              Overlay installed
                                                           |
                                                         ???
                                                           |
                                                       Profit!

    When you go to install a tar overlay layman will know it is a tar overlay type. The Overlay class will then call the class associated with that type, being tar. The class associate with tar is...TarOverlay! So the TarOverlay class is called, and it wants to call the add() function. However, if you looked at the source code you'd see that the TarOverlay class doesn't have an add() function. So where does it get it from? The ArchiveOverlay class that it subclasses.

   The ArchiveOverlay class requires you to extract the archive, that the TarOverlay class has in its own class, that's what makes this class different from say...the SquashfsOverlay class. The way you extract the archive is different.

   So the TarOverlay class will call the ArchiveOverlay class' add() function which in turn will make use of the TarOverlay class' extract() function. The ArchiveOverlay class' add() function isn't complete though, it will make use of the add() function in the SourceOverlay class, to do anything else that the ArchiveOverlay class' function didn't cover. After this is complete you can enjoy your newly installed archive overlay and feel happy that you didn't have to do anything more than: layman -a <overlay>

2.) Adding multiple description field support:

    If you look at this bug report here: https://bugs.gentoo.org/show_bug.cgi?id=465730 you will see that layman currently did not handle multiple description fields when there was the possibility to do so (mostly due to it being translated into different languages and whatnot). This week I added that support to the Overlay class.

    I personally believe it needs to be fine-tuned in a few ways. I think I will wait until dol-sen's recovery to fully tackle this bug though. In all honesty it would work fine now as it is, but would not take into account that lang attribute of the description xml tag.

    In accordance with the changes being made to the Overlay class to support multiple description fields, the layman sync plugin and the overlay-maker class both also have been modified to reflect the newly added support. The overlay-maker class does not currently add more than one description to the overlay being created though. A function has been added to grab the description though and can be used to prompt the user for multiple description fields if we decide that would best.

3.) Slight additions to overlay-maker class:

    A few changes have been made to the overlay-maker class.

1. github,com and git.overlays.gentoo.org https:// URLs are now supported (why they weren't before, I have no idea...I blame the media).
2. On a successful write, the user is now notified of where the xml file is being written to.

    Nothing massively crazy or huge, but helpful nonetheless.  I have hopes that the layman sync plugin module and the overlay-maker class will both be merged to layman's master branch soon. But no promises.

4.) Adds interactive session for unofficial overlay installs:

     https://bugs.gentoo.org/show_bug.cgi?id=315359 hoped to add more of a warning to users that are not installing official overlays. I added that feature which I also made configurable. So now a user will be prompted when installing unofficial layman overlays. They will be asked if they would like to continue even though the overlay is unofficial and if they respond with "yes" then they will continue on with the installation.

    This feature, as I said previously, will be configurable and will be able to be turned on or off with the "check_official" configuration option. The config option name might be changed in the near future and if it does you guys will be the first to know. Well, unless someone checks the source code and it's on there before I report it to you.

5.) CLI improvements:

    CLI or (command-line interface) is the method of calling a command through the command-line and give it certain command line arguments. You've seen this in layman -a <overlay> the command line arguments here are the -a flag and the <overlay> name.

    Keeping that in mind, know that if you previously wanted to install multiple layman overlays, you'd have to do it as so: layman -a <overlay 1> -a <overlay 2>. But to be honest that's not really a natural way of doing so. Over this week I changed the way layman handled certain actions such as --add or --delete. So now instead of having to call the flag multiple times you can simply append multiple overlay names to the actions, as so: layman -a <overlay 1> <overlay 2>.

    This was done by removing the action parameter of the argparser add_argument function, and replacing it with nargs = '+'. Doing this changes the behaviour of how the flag gathers its information and will allow users to perform layman actions in a more natural manner.

    Another minimal change has been made to the command line arguments as well. I have taken the liberty of renaming the --setup_help flag to --setup-help as it should have been originally. Whether I made this mistake or not, I'm not aware. But it's fixed now, which is good.

6.) Ability to enable or disable overlays:

    Imagine you're spending the night in a hotel that just so happens to have really crappy internet (it's not like they don't have multiple people using the internet all the time, all day long or anything...) and you want to prevent portage from accessing the ebuilds that a certain overlay has installed.

    So how would you tackle this problem? You could go ahead and delete the overlay. That would work, sure. Unfortunately, you'd have to download the overlay again after that's done. Which, given your current situation, would suck. Or you could just have portage "forget" the overlay exists.

    How? With layman's new ability to disable and then enable the overlay. Preface: this currently only works if you use the make.conf repo configuration method.

    If you were to look in your make.conf (located by default) in /var/lib/layman, you would see the following:

                                          PORTDIR_OVERLAY="

                                          <overlay locations>

                                          $PORTDIR_OVERLAY

                                          "

    With the ability to enable or disable an overlay, this had to be changed. It's a rather simple and elegant solution:

                                          ENABLED="

                                          <overlay 1 location>

                                          <overlay 2 location>

                                          "

                                          DISABLED="

                                          "

                                          PORTDIR_OVERLAY="

                                          $ENABLED

                                          $PORTDIR_OVERLAY

                                          "

    So now, portage will only acknowledge the overlays listed in the ENABLED var and ignore the overlays in the DISABLED var. When calling layman -D <overlay 1> the overlay location of <overlay 1> will then be moved to the DISABLED var in the make.conf. Which you can re-enabled by invoking the layman -E <overlay 1> command. I hope to incorporate this into the repos.conf repo configuration method as well but how layman should go about doing so needs to be discussed.

7.) Small fixes made to codebase:

    Two small fixes were made to layman's codebase.

1. Improvements of the way layman handled the success or failure of a write to a repo config method has been fixed. layman will now take into account the fact that a user can be invoking more than one repo config method and one may fail while the other succeeds. Previously this would lead to a false positive. That behaviour has been fixed.
2. layman previously falsely informed a user that an overlay was not supported if they ran layman -v -i <overlay> and the overlay was actually supported. This was because layman was unpacking the values from the Info() function and unpacked the boolean for "supported" in the position of the boolean for "official".

 Goals for next week:

•  Begin working on migrating docstring tests to a full test suite.
• This goal has been allotted to take 2 weeks so don't expect a massively fulfilling report next week. 

As always, to anyone interested the source code and my commits can be found on git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.

  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [2]https://github.com/twitch153/layman/tree/gsoc2014

With regards,
    Devan Franchini (twitch153)

GSoC 2014, Week #7: Going at a fine pace

Hello everyone!

This week has gone pretty well. I took yesterday off to spend the 4th of July with my family and I had some good times, but now is the time that I inform all you lovely readers what was accomplished throughout this week regarding layman.

A word beforehand, my mentor Brian Dolbec (dol-sen) is currently in the hospital and will be there at least until July 13th. I just wanted him to know that my best wishes go out to him and I hope the best for him.

While dol-sen is in the hospital recovering I will be bugging my other mentor (quantumsummers) a lot more than I bet he would hope for because I need the green light on my pull requests I have for certain features that I'll be elaborating more on in this post.

1.) Portage sync module plugin:

    The portage sync module plugin for layman should be ready to be merged with the master git branch. I required more seasoned eyes to skim through it and let me know that it is before merging it. The only change made to it throughout this week was that I added fetching of the remote list to layman by default when running any of the actions in the sync module.

2.) Overlay-maker class:

    This week has included a two changes for the overlay-maker class, one of which was made that directly affect the user.

    1.) The old method of branch tags has been removed:
            I previously added branch support for multiple overlay types.
    In order to declare an overlay had a branch you would simply add the
    <branch></branch> tags to indicate that the overlay uses this "branch".
    This has been changed. When you are creating an overlay.xml file now you
    can specify the branch for multiple source tags. ex.) You have an overlay
    located at http://github.com/twitch153/ebuilds.git and it uses the branch
    experimental, but you have another source of the same repo at
    git://foobar.com/twitch153/repo.git and it uses the branch unofficial. With
    the previous nature of the <branch></branch> xml tag you couldn't
    specify both tags for both sources and you'd be left sad and full of hate
    and anger toward the dev that made the decision to make it such a static
    thing. With the new nature of the branch info though you could do this:

        <source type="git" branch="experimental">http://github.com/twitch153/ebuilds.git</source>
        <source type="git" branch="unofficial">git://foobar.com/twitch153/repo.git</source>

     and that would work perfectly well! Of course, with this new change comes the need to modify the .dtd file again, so I'll have to create a patch and run that by the maintainers of the .dtd file once again. That'll be done once these changes get merged to the master branch.

    2.) Proper feed support was added for the overlay-maker class and
    overlay dictionaries:
        If a user was to run through an interactive overlay maker session and
    specified the feeds to the overlay and wrote the overlay to the file...it
    wouldn't work! Why? Well, it's simple really. The overlay-maker class is an
    interactive session that will create a python dictionary of overlay
    elements. However, due to my clear lack of perfection I used an incorrect
    key for the overlay feeds and therefore the overlay class would totally
    overlook the feeds when creating an overlay object to be written to an
    xml.

3.) Squashfs overlay support:

   For those of you unaware of what squashfs is: it is an archive-like file system. There are absolutely tons of different things about it that I could list but I lack the proper knowledge to continue on by saying anything other than it is like an archiving type similar to tar or zip.  So, in our vast abundance of knowledge we already know that layman already supports tar overlays so is squashfs really all that different? The answer is an emphatic no. It is my own belief that you could probably go even further and say that adding any sort of archive overlay support to layman would be so similar that you might want to make an archive class where all archive overlay types would get their main functions from and go from there based on the little differences that they do have.

    However, this week I added both squashfs overlay support and an external test to see if it works in practice and not just on paper (hint: it does).

Goals for next week:

• Improve on the archive overlay types and possibly make an archive class to centralize the main functions necessary for each type.
• Discuss with quantumsummers about moving some of my work to the master branch.
• Focus on squashing some bugs for layman.

As always, to anyone interested the source code and my commits can be found on git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.

  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [2]https://github.com/twitch153/layman/tree/gsoc2014

With regards,
    Devan Franchini (twitch153) 

GSoC 2014, Week #6: Hit the mid point

You know, they say that the point of no return is when you've gone so far that turning back just seems impossible.

Typically that sounds pretty dismal. But in this case it's the exact opposite!

This week has been a week of good fortune for me and this is why:

1.) I passed my GSoC midterm evaluations! Which means we've hit the midpoint for the GSoC program.

2.) My main computer is alive again! My computer (a.k.a Buff-Kitty) is finally alive after me destroying tons of computer parts (I don't want to talk about it...). It's been awhile but I can confidently say you were missed, Buff-Kitty.

In celebration of new life, a lolcat shall be posted:

You sure do, Buff-Kitty. You sure do.

Enough of that though. Onto the goodies of the week. So what's new?

Recall what I said my plans for this week were:

• Revamp the overlay definition generator script.
• Polish up the portage sync module.
• Begin adding squashfs overlay support. 

 So what was accomplished this week? To all of you who were ecstatic about having squashfs overlay support well I have good news for you. The good news is that you'll get it...eventually. But not this week. Because I was being doing a complete overhaul on the overlay definition generator script and polishing up the portage sync module to add some nifty new features to it.

Let's start with the sync plugin module since it didn't get an entirely new overhaul.

 1.) Portage Sync Module:

  Recall in my previous post how I explained that the sync module made use of two variable methods of running: the API method, and the subprocess method. By default, the sync plugin should run via the API method, however, if something occurs that prevents this from happening then portage will fall back on the subprocess method. Although to be honest with you I don't know what could happen to prevent you from running a program's API yet, still be able to run the subprocess. That's an aside though.

  "Okay, I remember all of this and I remember that it interacts via a file in your repos.conf folder, big deal, what's new?" Nice to see you're just as impatient as I am. So I'll just cut to the chase. You can tell portage to sync a repo that doesn't exist. Just because the repo is listed in the repos.conf config file it doesn't necessarily mean that the repository is installed. This would cause a failure to sync the repo (for obvious reasons).

  Fear not! For a new feature to the API variant of the plugin has been added. This feature will check the configuration for the repository in the repos.conf config file and create a new overlay definition using the newly revamped overlay-maker class. After creating the new overlay definition.xml file it will then install the new repo for you. Then when you go to sync the repo again, it will successfully sync the repo (for obvious reasons).

2.) Overlay-Maker class:

  Previously this was just a simple script to prompt users for overlay information and then to pipe it into a script. Throughout this week that has changed and the overlay-maker has been turned into a little more than that. By making use of the new overlay-maker class you can make layman overlay definitions and write them to a desired file via a script, or an interactive command line interface. Further more the overlay-maker adds auto-completion support for overlays that have mirrors from github.com or from git.overlays.gentoo.org.

  If given a source URL that is from either of those two locations the script will automatically fill out the rest of the available sources for that overlay along with the homepage and feeds available.

  It is possible that before the end of this GSoC that the overlay-maker will be able to be ran through an array of subparser commands from layman, however even if I am not able to get it accomplished prior to the end of the program, I do hope to get it done so that users can add overlay definitions simply by doing something like:

    layman repo --add twitch153

  This would then bring up the interactive interface to add the overlay by the name of twitch153 and add it to a file of your choice. It would also make it easier for developers maintaining large repository.xml files. If they wanted to complete remove a repo from a large repository.xml file and this method of interaction was possible they could run:

    layman repo --location=repository.xml --del twitch153

  And have the peace of mind of knowing it is gone for good. The other positive aspect of this is that you could then create a script or a cron to add/remove repos from a .xml file. The possibilities are exciting for sure.

  For all of you layman users, during this upcoming week or weekend a new release candidate for layman is going to the portage tree and will be available for all to install and test. So enjoy that and I encourage feedback about it including things you would like to see in the next RC of layman (because there will be another one before the end of this GSoC...hint, hint).

Goals for next week:

• Add finishing touches to portage sync module.
• Continue to work on overlay-maker class.
• Hopefully begin adding squashfs overlay support.

As always, to anyone interested the source code and my commits can be found on git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.

  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [2]https://github.com/twitch153/layman/tree/gsoc2014

Hope everyone is enjoying their summers!

With regards,
    Devan Franchini (twitch153)

GSoC 2014, Week #5: Midterm evals approaching

Hello,

     For everyone who hasn't done a Google Summer of Code or known anyone who has I'll explain a little bit about it shortly.

     For the past ten years, Google has offered a program worldwide that allows students the opportunity to get a stipend to write code for open source organizations. This is Google Summer of Code. In order to get in to the Summer of Code you need to supply a proposal which can consist of your project proposal and a time line that you expect to go by in order to break things into manageable pieces. If you get accepted you then begin your summer project!

    It doesn't mean everything is in the clear just because you were accepted though. Throughout the Summer of Code you have to go through two evaluations:

• The Midterm Evaluation (June 23rd)
• The Final Evaluation      (August 22nd)

Essentially, these are both evaluations from your project mentor that confirm you are keeping up with your proposed time line. If you do not pass either of these, you don't get to fully finish the Summer of Code. You will be removed and you will cry yourself to sleep thinking about what could've been! (Tragic, right?)

    Never fear though, not only am I keeping up with my proposed time line, but I'm actually ahead of schedule (I also hope to stay ahead of schedule).

This following week the following work has been done for layman:

1.) Minimal changes to documentation and code:

    This week's minimal changes include changes to the manual page formatting to make it a little cleaner when being displayed. As well as changes to the way both config.py and overlay.py handle dictionary logic.

    For those of you who have never coded in python allow me to briefly explain what a dictionary is:

    A python dictionary is a data structure that associates a value with a key, not uncommon to that of an actual dictionary. To grab a value from a dictionary in python you can do a few things, but the most common way of grabbing a specific value from a dictionary is by simply grabbing its value as so:

    dictionary['<key>']

    With that slight crash course on python dictionaries allow me to explain the problem with layman's previous dictionary logic: In some cases layman's code would check dictionaries for values by doing something like this -

        if dictionary['key']:
            ....

    This logic has been tested by me in both py2.7 and py3.x and neither times would anything like this actually end up with anything other than KeyError: 'value' if the value wasn't in the dictionary. Part of me believes that this logic might have worked in previous versions of python but I haven't tested it to confirm. My implemented solution was to change the logic to something like this:

        if 'value' in dictionary:
            ....

This will prevent the code from causing runtime errors and will neatly check to see if the value exists in the keys of the dictionary.

2.) Beginning addition of layman portage sync plugin module:

    Going on the idea that if you're using layman then the chances of you using Gentoo are extremely high. Furthermore, the chances of you using portage is just as high, if not higher.

    In my previous posts I explained how to add a layman overlay, but did I ever mention how to sync the overlay? If I did, then...oh well. Under the assumption that I never explained how to sync layman overlays I will explain now. You can either sync all the installed overlays with the command: layman -S or you can sync an individual overlay with the command: layman -s <overlay>. Simple, no?

    This doesn't take into account that you also sync your main portage overlay via emerge though. So when you want to sync secondary overlays you'll have to do sync them via layman and then your main via emerge. If you're anything like me you sync portage very often and regularly forget to sync your layman overlays. So throughout this week I've been in the process of doing two things, the first of which is working on adding a portage sync plugin module for layman.

How this is done:

    Although the current version of portage does not support sync modules, it is in the works to have a version of portage that does support sync modules. So that's why I'm making one for layman. So until that version of portage becomes available on the portage tree I'll lazily say you don't need to know how I added the module. However, I will explain slightly how I created the module itself.

    You need two things:

• The __init__.py
• The module.

    In the __init__.py you need to specify the config_class and the module specifications. To see both, you can look at my source code to get a better look, this part isn't complicated. Even for someone who has never coded before.

    In the module you need to create your config class and have it subclass the SyncBase class. From there you need to specify your __init__() function, your new() function, and your _sync() function. That is the bare minimum requirements for a portage sync plugin.

The layman sync plugin uses two different methods to be ran:

1. Subprocess method
2. API method

Firstly, we'll discuss the more common and simplistic of the two, the subprocess method. Portage already has a built in function to execute processes and this is call the portage.process.spawn_bash() function. This function will also return an exit code upon completion of the process. It's really quite beautiful.

For a user to add a sync plugin it'd be as simple as adding the execution of the subprocess for portage to execute.

Secondly, we'll discuss the API method of the layman sync plugin module. This method is slightly more complex than the previous one but is considerably less complex than some other things.

To make use of layman's API and not require a subproccess to be executed, we added this second config_class. This class creates an instance of layman's API and executes it's built-in sync function directly. Multiple checks have been added to ensure the API will be instantiated when necessary in order to become less resource intensive. Why is that?

Well, consider you have 5 overlays installed on your system. Okay, 5 isn't that many. But if you have portage syncing your layman overlays and say you have 50 overlays instead of 5, it would have to create an instance of the layman API 50 times when it might not be necessary.

3.) Overlay definition generator script in the works:

    If you were to look into layman's manual you would see that you can actually create your own overlay definitions in one or many .xml files and layman will pick them up if they are in the correct directory (default is: /etc/layman/overlays, but can be changed via command line or config setting. Multiple examples are also included in the overlay to guide users into making their own overlay.xml definitions.

    For the ease of use for users and developers both, a tool is being created that allows you create an overlay definition via a script that can be ran in the safety and comfort of your terminals.

4.) A note to all Gentoo users:

    This week we also released a test version of layman into the wild for users to hunt bugs with. So for any brave soul that is willing to test this and report back feedback of any kind, it'd be greatly appreciated.

    To install it on your system, simply install the -9999 version of layman in the portage tree and enjoy! Don't be afraid, it shouldn't bite you.

Goals for next week:

• Revamp the overlay definition generator script.
• Polish up the portage sync module.
• Begin adding squashfs overlay support.

As always, to anyone interested the source code and my commits can be found on git.overlays.gentoo.org/gitweb/[1][1] or on github.com[2] on layman's gsoc2014 branch.


  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [1][1] Because my main computer has been down I haven't been able to push
            my latest commits to this repo.
  [2]https://github.com/twitch153/layman/tree/gsoc2014

GSoC 2014, Week #4: The wheels keep spinning

Hello new and old readers all the same,

    Typically I make my posts every Friday but due other pressing matters unrelated to my coding adventures, I was unable to present my weekly achievements for the week.

    Because I know you guys are all very curious about my computer (because you all love it as much as I do, let's be honest), I tested my old motherboard with my new parts and it was a no-go. So I returned my new motherboard I purchased to Newegg and I should expect a new one within the next week or two. We all know how long these things can take...anyway, I'll be sure to keep all of you updated on how that goes.

    Now on to the good stuff, the main course of the post: What's been done to layman?

1.) Implementation of the "with" statement to open files:

    Sometimes it's great to explain things to people, when the changes you make to your code are very specific to your codebase, in this particular case, it isn't necessarily like that and the implementation is quite common. So common in fact, that I shamelessly found a great link that explains exactly what the "with" statement is used for: http://effbot.org/zone/python-with-statement.htm. The author of this page does an exceptional job to explain why this was implemented in later versions of python.

    The short of it: this was implemented in many places in the codebase of layman to make the opening and closing of a file that layman makes use of "safer", in case something doesn't work correctly for whatever reason, which will prevent leaving unused resources on your system open. So that's something everyone can enjoy!

2.) Replaced "subpath" with "branch":

    If anyone has made use of the subpath variable you're in luck! We removed it from layman's latest code! Why are you in luck? Well, because we replaced it with the branch variable. It was discussed between myself and my mentor (Gentoo developer, dol-sen) that using "subpath" would be mildly misleading for the general behaviour of how this variable is going to work. As most VCS's (version control systems) are git based in one way or another we decided using branch just made more sense. While I am currently in a waiting process for my previously mentioned .dtd patch to be submitted, hopefully everything will work out alright before we push out layman's upcoming release candidate

    Along with this change came the addition of alterations to layman's manual page that comes equipped with some good examples of how to make use of this fancy new variable.

3.) Added module branch support:

    Okay, so we have this fancy new variable we called branch. Big deal. How is this variable going to help me? Well, since you're all dying to know (and haven't figured it out already...) that variable can be used in an overlay.xml file. Once layman catches that this variable has a value, the following VCS types can make use of this variable for branch support:

• Git: When a branch name is given to an overlay that is git based, layman will append the following to the cloning process of the git repo - "-b <branch name>". This allows the user to make custom overlay files that specify repositories and supply them with a particular branch, which is useful if you want the same repository under multiple aliases to be handled by layman.
• Mercurial: Similarly with git, when a mercurial overlay has a branch defined, layman will append the following to the args for mercurial when cloning - "-r <branch name>" which is actually having the user specify a specific revision for mercurial when cloning that repository.
• CVS & Tar: For both CVS and Tar overlay types which previously made use of the "subpath" variable, the behaviour of defining a branch in either of their repo definitions will remain the same and the branch will be treated as a subpath for the repo. This note has also been added to layman's manual page.

Note to all readers: if you think I'm doing something incorrect or think there is some branch love I can give to other repository types, I encourage patches. Of course they'll be tested and everything beforehand, but I will accept any patch and test it to be applied to layman for the greater good of the community.

4.) Refined test code:

    When installing layman you can run tests to ensure the test cases work in accordance with how layman claims it does on your system. layman currently supports both doctest testing, and non-doctest testing via a file called external.py (I can only assume this represents "external testing"). This week I worked on refining the external test code in two ways:

1. I added compatibility for python3.x by changing the "print " declarations to "print()". 
2. One of the test cases is a Unicode test to ensure layman's output is unicode. However, python3.x will not explicitly decode strings when printing them out if they are encoded. So changes were made to ensure that the output from layman when running this test case would remain the same in both python2.7 and python3.x by implicitly decoding the string output.

5.) Adds file:// url support for tar overlays.

    However, when running the test cases I ran into a testing failure that struck me as odd. Until I realized it was partially my fault.

    This test ensures that tar overlays behave correctly by running the add() sync() and delete() functions for a temp created tar overlay. By making use of the tempfile library that python provides you can make....temporary files! (Who would've guessed?) This is particularly useful when doing test cases and just whenever you need to make use of temporary files such as possibly cache and the sort.

    Firstly, this test case initially failed due to the fact that I accidentally removed the tempfile import from the tar.py file (my bad!).

    To my surprise though, the test continued to fail, and it was no longer something that I caused! So why was it failing? Well, thanks to layman's output I was able to trace it down to tar.py's _extract() function. Tar overlays never put into consideration if the user was actually specifying a file:// url instead of an actual url that layman could download a tar from. But this test made use of a file:// temporary.tar.bz2. So I needed to add support for that in tar.py. It was simple enough. If the user specifies file://<tar_name> the just remove the "file://" section from the name and use tar to extract it.

    The test continued to fail though...but why? Where is it failing? By inserting some debug print statements (mostly saying "Are we here yet?!") I was able to see that layman was actually deleting the file after extracting it. Which makes sense if you consider it wasn't supporting local tar overlays, why keep a tar you can just download again on sync? But if we're keeping this tar overlay locally on the machine then that's just no good.

    To fix this issue a boolean was created called "clean_tar". In _extract() it will first check this boolean to see if it is True. If it is, then layman will get the okay to delete the file and purge it from this cruel, cruel, virtual world. For local files, this variable is hard-coded to False, and layman will never have permission to clean up local tar files unless the user themselves modify the code (modifying open source code?! What's wrong with you!?). Fear not though! Users do have some choice in their lives when it comes to this variable. Users can modify the configuration option: clean_tar to either yes or no. If yes, layman will delete downloaded tar overlay files after extracting them, and obviously the opposite effect will happen if users set the option to no; giving them the choice to keep downloaded tar overlays on their system to collect precious hard drive space.

Goals for next week:

• Push out a new release candidate for layman (along with possibly changelog?)
• Begin migrating from a docstring tests to a full test suite.

To anyone interested the source code and my commits can be found on
git.overlays.gentoo.org/gitweb/[1][1] or on github.com[2] on layman's gsoc2014 branch.


  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [1][1] Because my main computer has been down I haven't been able to push
            the commits to this repo. My mentor dol-sen should be pushing these to
            this repo some time this upcoming week.
  [2]https://github.com/twitch153/layman/tree/gsoc2014

 See you next week!

With regards,
    Devan Franchini (twitch153) 

GSoC 2014, Week #3: Going good

Hello again everyone,

This week went fairly smooth. I got my computer parts in and to my despair there is an issue with the motherboard. So! Hopefully this Sunday I'll be able to see if I can't solve this problem and have my main computer alive and functional once and for all!

Not much can be listed as being accomplished but a decent amount of work was done this week regarding layman:

1.) Refined current code:

       I'm sure you're all thinking: "But Devan, didn't you do that last week? How much code could you possibly refine before it's perfect?". Well, forget that. Because code can always be refine in some way. Especially with something as expansive as layman. Throughout this week if I had to think about it I would probably say that I used git far more than I did anything else throughout this week.

      To avoid dragging out ideas and beating them into your brain to the point where your eyes glaze over I'll spare you most of the details on my git adventures and cover the main things that I've been getting familiar with throughout this GSoC involving git.

• Using git rebase: Although I know git rebase is used for far more than I use it for, I have been using git rebase to handle my commits. If I broke something in a previous commit I would find the necessary SHA of the commit before the commit I wanted to repair and then I would run: git rebase -i <SHA>, once run it will bring up a text editor that will list all the commits up to that SHA. If I wanted to reword a commit I would just replace pick with reword, if I wanted to repair my mistakes from any commit I would replace pick with edit. This would also allow me to reword a commit if I so desired by using the command git commit --amend. After finishing all my repairs I would run git rebase --continue which will take me to my next edit, reword,... until it hit pick which will just continue on until you successfully rebase. After doing that you have to push all your commits to the branch by running git push --force and you'll have fixed your commits and pushed them to your remote repository!

•  Breaking up a commit into two commits: So say I made a commit that I thought at the time was good but then I realized it'd be better as two separate commits. Well, I would rebase and edit the commit I wanted to break up into two (or more). Once you get to that commit you should run git reset HEAD~ which will reset your current HEAD to the one prior to that commit. So now your changes are "unrecorded". You can then continue to make your changes and break them up into multiple commits as you would normally. Once you're all done simple run git rebase --continue, and feel good about yourself for doing a job well done!

By making use of these two topics I've been fixing up and refining the code without having to make different commits admitting I messed up (I mean, come on! Who wants to admit that they forgot an indentation here or there on multiple occasions?).

2.) Updated the repository.dtd file:

       Although my understanding and experience with XML is slim to none, I can at least explain what was done here and why. If you look here you will see the overlay where the layman overlay repositories get their XML definitions. The structure of the repositories.xml is defined in its .dtd file. The changes I made to the repository.dtd include changes to add information for IRC channels, and overlay branches.

3.) Documentation updates:

       If you're using layman then it's extremely likely that you're also using Linux. More specifically, Gentoo. So you should know that if you run man layman you'll get the manual for layman which includes pretty much all you need to know about how layman works, and the configuration options you can modify to do certain things.

      In preparation for the new release of layman, the documentation has been modified to reflect the changes I've been adding to layman throughout these past weeks and some cleaning up for the man page as well.

Goals for next week:

• Finish adding branch support for overlay installations (testing included).
• Get the patch I created for the .dtd modifications to be applied to the repository.
• Prepare for a new layman release.

To anyone interested the source code and my commits can be found on
git.overlays.gentoo.org/gitweb/[1][1] or on github.com[2] on layman's gsoc2014 branch.


  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [1][1] Because my main computer has been down I haven't been able to push
            the commits to this repo.
  [2]https://github.com/twitch153/layman/tree/gsoc2014

 See you next week!

With regards,
    Devan Franchini (twitch153)