GSoC 2014, Week #10: Testing, Testing, 4, 5, 6...

Welcome back to all my long term readers, and to anyone new...welcome! Hope you enjoy this blog as much as I enjoy writing for it.

Unbeknown to me previously, my mentor had surgery while he was in the hospital. I was actually quite shocked to hear that, considering his condition previously. I just wanted to take the time here to say that I hope he has a good, and painless recovery. In the very least, I hope he is able to fully recover.

This week, the following has been accomplished:

1.) More docstring migrations to test suite:

   If you've been keeping up with my posts you'll know that this week is the second week that I've been working on migrating from docstring tests in layman's code, to a full test suite.

   Good news is, this is my second week and I should be wrapping it up at a good number of 13 tests.

   Although this means little to your average user, the tests are a necessary part of assuring that the code works precisely as it says it does.

   The keyword of the day is quality assurance, or simply put: QA. QA is HUGE in the Gentoo Developer community. It's like the law of the land, metaphorically speaking. As I am part of the amazing Gentoo developer community I too firmly believe in QA and its necessity in our code. That doesn't mean I don't believe people make mistakes though. That's why these tests exist, to confirm that everything is working as it should.

2.) New layman release:

   To those of you who have been waiting for an official version of layman to be released that includes all the amazing work that has been done up until...roughly my fifth week or so working on it, well wait no more! layman is now officially at version 2.1.0 (OoooOOoooh AaaAaaAAaah).

   A handful of things had to be done previously to allow this release to be done, including uploading the source to sourceforge.com, as well as updating the CHANGES file to allow users to see what has been changed since the last release. Nothing spectacular or amazing, but still a very necessary part of the development cycle.

   It's almost neglectful to just push out a new version of something without telling people what's new about it. Although, this is just my own opinion on that.

   So fire up your CPU and get to installing layman-2.1.0, and be sure to give me some proper feedback on things I've done to make your life easier/harder. Especially if I made it harder due to some unforeseen bug in my code.

   I apologize about the short post, fingers crossed that next week I can give the post just a little bit more substance.

Goals for next week:

• Begin making layman VCS types modular.
• Polish any code already in layman's codebase.

As always, to anyone interested the source code and my commits can be found on git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.

  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [2]https://github.com/twitch153/layman/tree/gsoc2014

With regards,
    Devan Franchini (twitch153)

GSoC 2014, Week #9: Testing, testing, 1, 2, 3...

Hello everyone!

Good news! My mentor, Brian Dolbec (aka, dol-sen) has returned from the hospital! He has to take things slow but at least we know he made it out alive. Best wishes for him and I hope everything goes well for him.

In an attempt to make things as short as possible here is what has been accomplished this week:

1.) Improvements to overlay-maker class:

1. Support for bitbucket.org mirror auto-fill info has been added. When a user now gives the overlay-maker class a bitbucket.org source URL it will now try to determine which VCS type it is, as well as the possible feed and homepage URLs.
2. In addition to adding the support for bitbucket.org auto-fill info, the function necessary to auto-fill the info for source URLs has been cleaned up for improved maintainability.
3. The get_input() and get_ans() functions were improved upon, to allow for a colorful interface with users, similar to the output messages of layman. This was done for overlay uniformity and to just be...pretty.
4. Message output for the overlay-maker class has been migrated from your run-of-the-mill print() functions to the output suite that layman has readily available. Again, this brings a uniform look to the interactive sessions.
5. The get_ans() and get_input() functions were moved to utils.py. This allows them to be where they should belong, as well as making them readily available to other classes that can make use of them (ex: db.py).

2.) Added disable and enable functionality to repos.conf files:

    Do you all remember in my last post where I mentioned the ability to disable and enable a repository in the make.conf files? Well, I discussed it over with dol-sen and we figured the best way to "disable" a repo for repos.conf files is to simply comment it out.

    Because that is the case, adding support was easy and is now readily available to be merged onto the master git branch,

3.) Began migration from doctests to test suite:

    Essentially a doctest is an in ".py" file test that was run via the python command line. It it a test that can be run to ensure that the function that is being tested, actually works. The drawback of doctests (in my humble opinion) is that they should belong in a separate place aside from the actual ".py" file that is being tested.

    If we're moving the tests to a separate place, why not just move them from doctests to the actual implementation of the test?

    That's what I aim to accomplish in this upcoming week (and what I've been working on this week).

    To see a full list of tests added to the test suite, look at the test-suite branch on my layman github repo.

 Goals for next week:

•  Continue working on migrating docstring tests to a full test suite.

As always, to anyone interested the source code and my commits can be found on git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.

  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [2]https://github.com/twitch153/layman/tree/gsoc2014

With regards,
    Devan Franchini (twitch153)

GSoC 2014, Week #8: Squashing some bugs

Hello everyone!

This week has been good in making leeway with open bugs for layman!

To all of you who are unaware of what bugs are (no, not the squirmy wormy ones), a bug is essentially a report regarding a certain program or package or pretty much whatever. It can cover a vast amount of things.

If you are curious to see the open bug for layman, follow this link here:
https://bugs.gentoo.org/buglist.cgi?quicksearch=layman&list_id=2406998

But I'm getting ahead of myself, I'm going to begin by discussing what has been accomplished this week for layman.

1.) Centralize archive overlay type code:

    Last week I added support for squashfs overlay types for layman and it struck me: "Hrm, the code for both tar overlays and squashfs overlays are super similar. So similar, in fact that the differences are less than the similarities. I bet I could make a class that all archive types get their main functions from". So that's exactly what I did!

    I created the ArchiveOverlay class which took all the similar code from both archive overlay types and brought it together for any archive overlay type to grab from. It's like a goody bag of archive functions.

   So now any archive overlay type will now subclass from the ArchiveOverlay instead of the SourceOverlay class, which the ArchiveOverlay class subclasses from. It's a large hierarchy that is as follows:

                             <archive overlay type being installed>
                                                           |
                                                     Overlay
                                                           |
                                          <archive overlay type>
                                                           |
                                                ArchiveOverlay
                                                           |
                                                 SourceOverlay
                                                           |
                                              Overlay installed
                                                           |
                                                         ???
                                                           |
                                                       Profit!

    When you go to install a tar overlay layman will know it is a tar overlay type. The Overlay class will then call the class associated with that type, being tar. The class associate with tar is...TarOverlay! So the TarOverlay class is called, and it wants to call the add() function. However, if you looked at the source code you'd see that the TarOverlay class doesn't have an add() function. So where does it get it from? The ArchiveOverlay class that it subclasses.

   The ArchiveOverlay class requires you to extract the archive, that the TarOverlay class has in its own class, that's what makes this class different from say...the SquashfsOverlay class. The way you extract the archive is different.

   So the TarOverlay class will call the ArchiveOverlay class' add() function which in turn will make use of the TarOverlay class' extract() function. The ArchiveOverlay class' add() function isn't complete though, it will make use of the add() function in the SourceOverlay class, to do anything else that the ArchiveOverlay class' function didn't cover. After this is complete you can enjoy your newly installed archive overlay and feel happy that you didn't have to do anything more than: layman -a <overlay>

2.) Adding multiple description field support:

    If you look at this bug report here: https://bugs.gentoo.org/show_bug.cgi?id=465730 you will see that layman currently did not handle multiple description fields when there was the possibility to do so (mostly due to it being translated into different languages and whatnot). This week I added that support to the Overlay class.

    I personally believe it needs to be fine-tuned in a few ways. I think I will wait until dol-sen's recovery to fully tackle this bug though. In all honesty it would work fine now as it is, but would not take into account that lang attribute of the description xml tag.

    In accordance with the changes being made to the Overlay class to support multiple description fields, the layman sync plugin and the overlay-maker class both also have been modified to reflect the newly added support. The overlay-maker class does not currently add more than one description to the overlay being created though. A function has been added to grab the description though and can be used to prompt the user for multiple description fields if we decide that would best.

3.) Slight additions to overlay-maker class:

    A few changes have been made to the overlay-maker class.

1. github,com and git.overlays.gentoo.org https:// URLs are now supported (why they weren't before, I have no idea...I blame the media).
2. On a successful write, the user is now notified of where the xml file is being written to.

    Nothing massively crazy or huge, but helpful nonetheless.  I have hopes that the layman sync plugin module and the overlay-maker class will both be merged to layman's master branch soon. But no promises.

4.) Adds interactive session for unofficial overlay installs:

     https://bugs.gentoo.org/show_bug.cgi?id=315359 hoped to add more of a warning to users that are not installing official overlays. I added that feature which I also made configurable. So now a user will be prompted when installing unofficial layman overlays. They will be asked if they would like to continue even though the overlay is unofficial and if they respond with "yes" then they will continue on with the installation.

    This feature, as I said previously, will be configurable and will be able to be turned on or off with the "check_official" configuration option. The config option name might be changed in the near future and if it does you guys will be the first to know. Well, unless someone checks the source code and it's on there before I report it to you.

5.) CLI improvements:

    CLI or (command-line interface) is the method of calling a command through the command-line and give it certain command line arguments. You've seen this in layman -a <overlay> the command line arguments here are the -a flag and the <overlay> name.

    Keeping that in mind, know that if you previously wanted to install multiple layman overlays, you'd have to do it as so: layman -a <overlay 1> -a <overlay 2>. But to be honest that's not really a natural way of doing so. Over this week I changed the way layman handled certain actions such as --add or --delete. So now instead of having to call the flag multiple times you can simply append multiple overlay names to the actions, as so: layman -a <overlay 1> <overlay 2>.

    This was done by removing the action parameter of the argparser add_argument function, and replacing it with nargs = '+'. Doing this changes the behaviour of how the flag gathers its information and will allow users to perform layman actions in a more natural manner.

    Another minimal change has been made to the command line arguments as well. I have taken the liberty of renaming the --setup_help flag to --setup-help as it should have been originally. Whether I made this mistake or not, I'm not aware. But it's fixed now, which is good.

6.) Ability to enable or disable overlays:

    Imagine you're spending the night in a hotel that just so happens to have really crappy internet (it's not like they don't have multiple people using the internet all the time, all day long or anything...) and you want to prevent portage from accessing the ebuilds that a certain overlay has installed.

    So how would you tackle this problem? You could go ahead and delete the overlay. That would work, sure. Unfortunately, you'd have to download the overlay again after that's done. Which, given your current situation, would suck. Or you could just have portage "forget" the overlay exists.

    How? With layman's new ability to disable and then enable the overlay. Preface: this currently only works if you use the make.conf repo configuration method.

    If you were to look in your make.conf (located by default) in /var/lib/layman, you would see the following:

                                          PORTDIR_OVERLAY="

                                          <overlay locations>

                                          $PORTDIR_OVERLAY

                                          "

    With the ability to enable or disable an overlay, this had to be changed. It's a rather simple and elegant solution:

                                          ENABLED="

                                          <overlay 1 location>

                                          <overlay 2 location>

                                          "

                                          DISABLED="

                                          "

                                          PORTDIR_OVERLAY="

                                          $ENABLED

                                          $PORTDIR_OVERLAY

                                          "

    So now, portage will only acknowledge the overlays listed in the ENABLED var and ignore the overlays in the DISABLED var. When calling layman -D <overlay 1> the overlay location of <overlay 1> will then be moved to the DISABLED var in the make.conf. Which you can re-enabled by invoking the layman -E <overlay 1> command. I hope to incorporate this into the repos.conf repo configuration method as well but how layman should go about doing so needs to be discussed.

7.) Small fixes made to codebase:

    Two small fixes were made to layman's codebase.

1. Improvements of the way layman handled the success or failure of a write to a repo config method has been fixed. layman will now take into account the fact that a user can be invoking more than one repo config method and one may fail while the other succeeds. Previously this would lead to a false positive. That behaviour has been fixed.
2. layman previously falsely informed a user that an overlay was not supported if they ran layman -v -i <overlay> and the overlay was actually supported. This was because layman was unpacking the values from the Info() function and unpacked the boolean for "supported" in the position of the boolean for "official".

 Goals for next week:

•  Begin working on migrating docstring tests to a full test suite.
• This goal has been allotted to take 2 weeks so don't expect a massively fulfilling report next week. 

As always, to anyone interested the source code and my commits can be found on git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.

  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [2]https://github.com/twitch153/layman/tree/gsoc2014

With regards,
    Devan Franchini (twitch153)

GSoC 2014, Week #7: Going at a fine pace

Hello everyone!

This week has gone pretty well. I took yesterday off to spend the 4th of July with my family and I had some good times, but now is the time that I inform all you lovely readers what was accomplished throughout this week regarding layman.

A word beforehand, my mentor Brian Dolbec (dol-sen) is currently in the hospital and will be there at least until July 13th. I just wanted him to know that my best wishes go out to him and I hope the best for him.

While dol-sen is in the hospital recovering I will be bugging my other mentor (quantumsummers) a lot more than I bet he would hope for because I need the green light on my pull requests I have for certain features that I'll be elaborating more on in this post.

1.) Portage sync module plugin:

    The portage sync module plugin for layman should be ready to be merged with the master git branch. I required more seasoned eyes to skim through it and let me know that it is before merging it. The only change made to it throughout this week was that I added fetching of the remote list to layman by default when running any of the actions in the sync module.

2.) Overlay-maker class:

    This week has included a two changes for the overlay-maker class, one of which was made that directly affect the user.

    1.) The old method of branch tags has been removed:
            I previously added branch support for multiple overlay types.
    In order to declare an overlay had a branch you would simply add the
    <branch></branch> tags to indicate that the overlay uses this "branch".
    This has been changed. When you are creating an overlay.xml file now you
    can specify the branch for multiple source tags. ex.) You have an overlay
    located at http://github.com/twitch153/ebuilds.git and it uses the branch
    experimental, but you have another source of the same repo at
    git://foobar.com/twitch153/repo.git and it uses the branch unofficial. With
    the previous nature of the <branch></branch> xml tag you couldn't
    specify both tags for both sources and you'd be left sad and full of hate
    and anger toward the dev that made the decision to make it such a static
    thing. With the new nature of the branch info though you could do this:

        <source type="git" branch="experimental">http://github.com/twitch153/ebuilds.git</source>
        <source type="git" branch="unofficial">git://foobar.com/twitch153/repo.git</source>

     and that would work perfectly well! Of course, with this new change comes the need to modify the .dtd file again, so I'll have to create a patch and run that by the maintainers of the .dtd file once again. That'll be done once these changes get merged to the master branch.

    2.) Proper feed support was added for the overlay-maker class and
    overlay dictionaries:
        If a user was to run through an interactive overlay maker session and
    specified the feeds to the overlay and wrote the overlay to the file...it
    wouldn't work! Why? Well, it's simple really. The overlay-maker class is an
    interactive session that will create a python dictionary of overlay
    elements. However, due to my clear lack of perfection I used an incorrect
    key for the overlay feeds and therefore the overlay class would totally
    overlook the feeds when creating an overlay object to be written to an
    xml.

3.) Squashfs overlay support:

   For those of you unaware of what squashfs is: it is an archive-like file system. There are absolutely tons of different things about it that I could list but I lack the proper knowledge to continue on by saying anything other than it is like an archiving type similar to tar or zip.  So, in our vast abundance of knowledge we already know that layman already supports tar overlays so is squashfs really all that different? The answer is an emphatic no. It is my own belief that you could probably go even further and say that adding any sort of archive overlay support to layman would be so similar that you might want to make an archive class where all archive overlay types would get their main functions from and go from there based on the little differences that they do have.

    However, this week I added both squashfs overlay support and an external test to see if it works in practice and not just on paper (hint: it does).

Goals for next week:

• Improve on the archive overlay types and possibly make an archive class to centralize the main functions necessary for each type.
• Discuss with quantumsummers about moving some of my work to the master branch.
• Focus on squashing some bugs for layman.

As always, to anyone interested the source code and my commits can be found on git.overlays.gentoo.org/gitweb/[1] or on github.com[2] on layman's gsoc2014 branch.

  [1]http://git.overlays.gentoo.org/gitweb/?p=proj/layman.git;a=shortlog;h=refs/heads/gsoc2014
  [2]https://github.com/twitch153/layman/tree/gsoc2014

With regards,
    Devan Franchini (twitch153)