Writings

Friday will be my last day at IBM. It's been an incredible run here, but, as they say, all good things.... I've had a pretty amazing run at IBM. From the Sydney Olympics to OpenStack, with tons of really cool projects in between. I got to call the Linux Technology Center my home over the last 13 years, an organization that lead the charge in getting IBM involved in Open Source projects far and wide. A place where I had to chance to work with many of my Open Source heroes. And an organization that dramatically changed the culture of development at IBM, very much for the better. And starting Monday, I'm going to do much of that again, in my new role at Samsung Research America's Open Source Group. Over the past year Samsung has been building up the OSG very much along the lines of IBM's Linux Technology Center or Intel's Open Source Technology Center: a center of competency for Open Source skills within the company, and a pool of extremely talented upstream developers across a range of projects. I'm extremely excited to be a part of that. I've been telling a few folks over the last couple of weeks, and because they are FAQs, here are the first few questions everyone jumps to: Are you still going to work on OpenStack? Yes. My role in the OpenStack community, and the amount of time I'm spending upstream won't change in any real way. Honestly, I'll probably have more time upstream. So I'll be in Atlanta and Paris, have plenty of time for my TC and QA PTL duties, and be slinging code, ideas, and debugging the gate just like I am today. Are you moving? No. We really like where we live, the Hudson Valley is definitely home, and moving is something that was never on the table. So I'll become a full time work from home person. I've got a great home office, will be running the woodstove during the cold months, and go for long mid day bike rides over the Walkway during the warm ones. There are tons of unknowns are part of this new adventure, but all my interactions with the new team so far have made me extremely confident and excited that it will be a great fit. I'll be out in San Jose all next week doing initial orientation and team sync, so if you happen to be in the area, let me know.

For the week of Monday Oct 20th to Sunday Oct 27th (partial, it's still Sunday morning).

• 34894 - # of test jobs run by the CI system
• 25316 - # of devstack clouds created by the CI system
  • 8254 - # of large ops runs (devstack + fake virt driver + limitted tempest test to drive it)
  • 940 - # of swift functional runs (devstack + swift functional tests)
  • 16122 - # of runs that do some level of devstack + tempest with libvirt qemu
• 508536 - # of qemu guests started (successfully) in the devstack clouds
• 128 - Max # of libvirt qemu guests started (successfully) in a single test run

Things that you figure out when you are playing with elastic search on a Sunday morning. One of the surprises for me was how much we use devstack outside of the base tempest run use case, and that our max # of guests spawned in a run is now over 100. Update: Clark Boylan correctly pointed out that our guests are 2nd level, on platforms that don't support nested kvm, and thus are libvirt/qemu guests not libvirt/kvm guests.

As with many OpenStack core reviewers, my review queue can be completely overwhelming, often 300 - 400 active reviews that I have +2 / -2 authority on. It's really easy to get discouraged on a list that big. Fortunately there are ways to trim that down. Gerrit provides a simple query language to select which reviews you see, using the query bar in the top right of the page: The way this works is by adding criteria into the search box, which by default is ANDed together to get the final results. In the process these queries change the URL for Gerrit, so you can bookmark the resultant queries for easy access later. Restricting to Single Project (and pulling your own stuff) This query is basically what you get when you click on a project link:

status:open project:openstack/tempest

Nothing special, but you can go one step further by removing yourself from the list of reviews:

status:open project:openstack/tempest -owner:sdague@linux.vnet.ibm.com

This also demonstrates that we can have both positive criteria and negative criteria. Little Lost Projects (don't loose the little ones) In addition to having +2 on nova, devstack, tempest, I've got it on a bunch of smaller projects, which I often forget I need to go review. You can build a single query that has all your little lost projects in a single list:

status:open (project:openstack-dev/hacking OR project:openstack-dev/grenade)

No Objections You can also filter based on votes in the various columns. It's not nearly as detailed as I'd like, but it is still useful. I have a basic query for No Objections on most projects that I review which looks something like this:

status:open project:openstack/tempest -Verified-1 -CodeReview-1 -CodeReview-2

This removes all reviews that have a current -1 in Verified column, and a -1 or -2 in the CodeReview column. So patches with negative feedback are dropped from view. The top of your review list may contain patches that haven't cleared CI yet, but that's easy to see. There might also be Jenkins -2 reviews in this list, but gate failed merges can usually use extra eyes. I consider this a base list of patches that there is no reason I shouldn't be reviewing them. Potential Merges I'm typically up and at my computer at 7am EST, which is often a very slow time for zuul. So one of the things I look for is code that only requires one more +2 to go to merge on projects like Nova. Many of these are easy to review fixes, and clear the decks before the queue gets busy in the afternoon.

status:open -Verified-1 CodeReview+2 -CodeReview-1 -CodeReview-2 (project:openstack/nova OR project:openstack/python-novaclient)

Like the last one, we are filtering out all patches with negative feedback, but also requiring that there is an active +2 on the patch. I also make sure to do this for both nova and python-novaclient, which often gets lost in the noise. Lost Patches Especially in Nova it's easy for a patch to get lost, as there are so many of them. I define lost as a patch that's passed CI, but has no feedback in code review.

status:open -Verified-1 Verified+1 -CodeReview+2 -CodeReview+1 -CodeReview-1 -CodeReview-2 (project:openstack/nova OR project:openstack/python-novaclient) branch:master

These patches are often from newer folks on the project, and as such often need more time, so I typically only go after lost patches if I know I can set aside a solid hour on them. However I try hard to get to this query at least once a week, to make sure things don't get fully lost, as a -1 will give the patch originator feedback to work on, and a +2 will make it far more likely to get the attention of other core reviewers when they are looking for mergable code. Experimenting with your own The gerrit query language is somewhat limited (full docs are online), so it can't do everything I'd like, but even just these few slices make it easier to be able to get into a certain mindset for reviewing different slices of code. I have a toolbar folder full of bookmarks for these slices on different projects to do just that. If you have other gerrit queries you regularly use, please leave a comment. Would love to see the ways other folks optimize gerrit for their workload.

One of the the things I was testing on this vacation was getting a foreign sim card to work on my Verizon Samsung Galaxy S3. LTE phones on Verizon all take sim cards now, and as of the 4.1 update for the S3 it's supposedly unlocked as a world phone, at least the internet largely said so. As a dry run for the Hong Kong OpenStack summit I wanted to figure out if this was true or not by trying it in Canada. Adventure #1: finding a Rogers Rogers is one of the big telcos in Canada, and has a pay as you go plan with data. This seemed to be the best bet to figure this all out. Finding an actual Rogers though, turned out harder than expected. The address we had for Saint John didn't have anything obvious, and unlike the states, there wasn't a cell phone store in every little town. That meant that by the time we actually found a Rogers it was about 3 days into Canada, in Truro, in the mall. The last time I did this was with a pre-smart phone in Germany, where their were telco stores in every transit hub, so I hadn't actually expected the stores to be that sparse. Adventure #2: the sim Apparently the way I read everything online was only about half of what's actually going on with Rogers. The prepaid plans existed, were a little differently structured than I expected. The Rogers rep was very skeptical that a Verizon phone was going to take a Rogers sim, but they popped one out anyway, we rebooted the phone, and it didn't lock out. It just showed a little funny icon in the notification area that there was a non-verizon sim in there.   This isn't actually a problem, and you can long hold on it, realize the Setup Wizard is the app causing it, and kill it. Then you don't need to look at it until a reboot, or about a week (it came back later for me which is how I got the screen shot). Some time on the phone from the Rogers office and I had a Halifax number, a 500 MB data allowance, and some credit for overruns on phone and texting. It cost me about $60 CAD. The cell phone was on their network, Rogers sent me a few texts with my number, and off we went to Cape Breton. Adventure #3: data On the drive to Cape Breton, I realized there was no data service on the phone. That was kind of the point of all this, to have data. Of course without internet, it was hard to debug. Also, complicating things, was the fact that Cape Breton is a bit sparse on cell coverage. Which meant even if I could figure out a fix, it wasn't really testable up there. A couple days in I started searching to figure out what the deal was, and eventually I got to the bottom of it. While the Rogers sim did everything correct to get on the phone network, and the radios all worked for that, to get on the data network you need to define an APN. Pre Android 4.0 there were APIs for this. Post Android 4.0 there are not, however on the S3 you can manually create an APN. There is this good app in the market called Offline SIM APN Database which has all these settings, and lets you copy and paste them easily. A couple of minutes later you'll have defined an APN. After that, reboot, it doesn't really like to take APN definitions live. So when we were leaving Cape Breton headed to Halifax, on the highway, back in civilization I finally had data on the phone. Just let it prefer Global mode and it was working. However, pretty quickly I noticed it was edge only. Adventure #4: edge Edge is basically the GSM version of 1xRTT (for people that live in CDMA land), and is slow. More searching basically led me to the fact that this was as good as it was getting on the S3. While the S3 was eventually opened up to be a global phone, it didn't start that way, and it wasn't really a design point for the Verizon version which added the 700 Mhz LTE radio. So the silicon doesn't have 2100 Mhz, which is basically where most GSM telcos implement HSPA, their 3G. So we were on Edge for the whole trip where we had data. Things that work fine on edge (even though they are slow): Google Maps, Foursquare, Facebook, Gmail Things that get goofy on edge: Untappd, Accuweather (both seem way to data hunger, and not very happy if things reset connections). I turned off sync while on the Rogers sim. Given the slowness of data this was very helpful for battery life, and for not having something else in the way when I wanted the data connection. Conclusion: it works At the end of the day, this all worked. Not having 3G was annoying, but fine. From what I can tell the S4 has all the radios to do 3G on GSM, as do some of the more recent Motorola phones on Verizon. Alternatives: Verizon International Plans When I finally figured out the APN issue, I also found my way to Verizon's international plans. They aren't actually all that bad, especially if you are trapped on edge, so data isn't going to be used all that heavily. And even more so with Canada, which after you add the international plan, all the calls and texts are free. I think that for future Canada trips (especially if we're there for something shorter) we'll just enable that on our phones instead of doing the sim route. However, on this trip the sim route was a learning experience as much as anything else. Update (Jan 4 2014): Hong Kong Because this post is getting a bunch of new finds on "the google", I wanted to add a little more data. I went to Hong Kong a couple months after this. There I was able to get a 7 day unlimitted data sim for ~ $12 US. They set it up in the kiosk and I was online in about 5 minutes. In Hong Kong I actually had 3G (HSPA+). So realize that 3G bands vary by country, and whether you'll get 3G on your phone is actually hard to completely figure out in advance. Also, I'm on Verizon stock firmware, and I've had friends have difficulty with this on Cyanogen mod on Verizon phones.

Note: everything else about the vacation is going to be positive, but I needed to mini rant about this one. One of the unfortunate things about leaving the country is returning. Not because it's not great to come home, but because of the stark contrast between border crossing outbound, and border crossing inbound. I've done this border crossing enough that when going to Canada I plan that entering Canada adds 15 mins to the drive, entering the US adds 1 hour to the drive. Here's how it went this time. Entering Canada No line of cars, pull up to security checkpoint, stop, hand over our passports. Guard: hey folks, where are you going today? Me: Vacationing in Nova Scotia for the next couple of weeks. Guard: Any Tabaco or Alcohol? Me: Half a bottle of Port. Guard: ok, but no Tabaco or Firearms? Me: Nope. Guard: how long you going to be here? Me: About two weeks. Guard: Ok, enjoy the vacation. Elapsed time: 3 minutes   Crossing back to the US Wait in line for 10 minutes in line of cars. Get to booth, stop, hand over our passports. Guard: hey folks, where you headed? Me: home to Poughkeepsie, NY, back from a couple weeks vacation. Guard: how long... oh you said a couple of weeks. Me: yep, about two weeks. Guard: any alcohol or tabacco? Me: yes, we're bringing by alcohol from a distillery in Canada Guard is typing away with passports, not actually paying attention. Guard: what was that, sorry? Me: we've got some alcohol from Canada. Guard: how much and what kind? Me: 3L, some vodka, rum, a couple other types. Guard: that's from a Canadian distillery right, not Cuban? Me: yep, from a microdistillery down in Luneburg. Guard: ok, any Tabaco? Me: no Guard: can you roll down your back window? Me: sure Guard: you folks bringing any produce back into the states? Susan: I think we've got some celery and a pepper in the cooler. Guard: the celery is ok, but we'll need to take the pepper. Susan: ok, if it matters they both actually came from the US. Guard: doesn't matter miss. Where is this? Me: in the cooler in the back. Guard: can you pop the trunk sir? Me: should be open. Guard goes to back, opens it, struggles with the cooler a little bit. Me: would you like any help? Guard: no, sir, stay in the car. Guard gets back to the guard station, elicit pepper that's been travelling with us for two weeks in hand. Guard goes back to typing on computer. Guard: miss, did you loose you passport at some point? Susan: yes, my bag was stolen in India back in 2000-2001 with my passport. Guard: (type type type) ok, we'll need you folks to pull to the side up there. Please pull into bay #2 and go inside. We pull into bay #2, there is a big sign that says "wait for guard". Me: hmmm... so do we wait, or go in? Susan: I don't know, lets wait, they'll tell us what they want. A couple of minutes pass, Guard hands off passports to Guard #2 chats to him a bit, then walks over to our car. Guard: folks, I need you to go inside. Me: no problem. Go inside, come up to desk. Guard #2 and Guard #3 start typing at things and looking at passports and us. Guard #3 seems to have gotten the role of checking out our passports. Guard #3: so miss, you lost your passport? Susan: yes, my bag was stolen in India, back in 2000, so I had to get a replacement. Guard #3: ok, and this passport is a renewal? Susan: yes. Guard #3: so this would be your third passport, so to speak? Susan: correct. Guard #3: when were you folks last in Canada? Me: we were just trying to sort that out. I think I was last here in Ottawa back around 2007. Susan: I think my last time was 2005 when we did a Vancouver trip. Guard #3: what about 2008, 2009? Me: huh, no I don't think we were here then. Guard #3 is now clearly looking at our other travel records, but he didn't actually tell us he was interested in anything beyond Canada. Guard #3: where else did you travel in 2009? Me: ... I think that might have been India, or maybe that was the year before. We were in Germany around then as well. (honestly, we travel enough, that I have a hard time keeping the order of trips sorted, especially being 4 years ago). Guard #1 comes back, Guard #3 and #1 start trying to find something on the computer. Some linke. Guard #3 asks what he's actually supposed to do. Guard #1 says go to some system, plug in her (Susan's) name and birthday. Guard #1: hey folks, can you take a seat for a moment. Note, there aren't any seats, but there is a stone window sill on the other side of the room. We go and sit there. There is typing, and more typing. Guard #1 comes and goes. At some point I think Guard #1 had to log into something with his credentials because Guard #3 couldn't get into the system they needed. About 10 minutes pass, Susan and I chit chatting during it. Guard #3 calls us up. Guard #3: miss, what is your mother's maiden name? Susan responds with her mom's full maiden name, first and last. Guard #3: what was that? (clearly he was expecting only last name) Susan repeats. Guard #3: ok. (Hands back our passports) you folks are free to go. We walk out the door and head out. Total elapsed time: 40 minutes Note, this isn't the first time that Susan's gotten pulled aside because of the stolen passport, though they only seemed to start doing it in 2008 (back from India, where they didn't want to let her get on the plane without a second form of ID), and 2009 (on our trip from Germany where Susan got swept off into a back room for 30 minutes after we landed). My guess is they implemented a new software system that flags more people. I'd have thought that after a few of these conversations they'd have annotated the records, but apparently our tax dollars hard at work means that they really like talking to her on every border crossing. Especially given that the stolen passport wouldn't even have valid dates on it any more, because it was 12 years ago that it was stolen. Realistically this was still less than the hour I'd allocated for the crossing. But it just always frustrates me that border crossing back to our own country is such a dreadful experience, and a think I always loath at the end of the trip.

Julien Danjou, the project technical lead for the OpenStack Ceilometer project, had some choice words to say about github pull requests, which resonates very strongly with me:

The pull-request system looks like an incredible easy way to contribute to any project hosted on Github. You're a click away to send your contribution to any software. But the problem is that any worthy contribution isn't an effort of a single click. Doing any proper and useful contribution to a software is never done right the first time. There's a dance you will have to play. A slowly rhythmed back and forth between you and the software maintainer or team. You'll have to dance it until your contribution is correct and can be merged. But as a software maintainer, not everybody is going to follow you on this choregraphy, and you'll end up with pull-request you'll never get finished unless you wrap things up yourself. So the gain in pull-requests here, isn't really bigger than a good bug report in most cases. This is where the social argument of Github isn't anymore. As soon as you're talking about projects bigger than a color theme for your favorite text editor, this feature is overrated.

After working on OpenStack for the last year, I'm completely spoiled by our workflow and how it enables developer productivity. Recently I went back to just using git without gerrit to try to work on a 4 person side project, and it literally felt like developing in a thick sea of tar. A system like Gerrit, and pre-merge interactive reviews, lets you build project culture quickly (it's possible to do it other ways, but I've seen gerrit really facilitate it). The onus is on the contributors to get it right before it's merged, and they get the feedback to get a patch done the right way. Coherent project culture is one of the biggest factors in attaining project velocity, as then everyone is working towards the same goals, with the same standards.

Saturday was a rainy day, so I decided to deal with switching out the root disk on my home server with an SSD that I purchased a couple weeks ago. It's part of my quest to get all the root disks of my machines off spinning media. My home server was a build from parts machine, that's long enough in the tooth that it won't boot from USB. So I found a stack of CDRs upstairs, of equally dubious age, burned an Ubuntu 12.10 server iso, and started the install. Things were chugging along quite well until the installer was supposed to install additional packages. Then it bombed out (I blame the ancient CDRs). I was able to get it to at least install grub, and get the thing to boot back onto the network. What I found myself with was a super minimal install. It didn't yet have a normal sources.list, it didn't have openssh-server, it didn't have ssh client even, it didn't have any of the normal even minimal server install tools. I had about 30 minutes of manual to typing to get the base apt repo in, and get me so I could ssh in from upstairs to drive the rest of the process. Boostrapping a Puppet Master This is the machine that's my puppet master. I had a copy of the oldroot over in one of my software raid arrays, so the moment I got that mounted, I dumped over the /etc/puppet this machine should have, and tried to just puppet my way up the rest of the way. I'd been on a month long kick to puppetize my home infrastructure, so this was a promising direction. It turns out puppet up from nothing is a little harder when you are the puppetmaster, and dnsserver for the network as well. :) So it was about another 30 minutes of manually installing what was needed to get my puppetmaster started. Once that was up, I managed to get the first puppet agent run in, and it was epic. 45 minutes chugging away pulling down all the policies I needed, applying packages and configs, all the kind of magic that prevented me from spending my whole day trying to figure out how I had this server setup before. It also showed me where my policy had holes. I've got xfs filesystems now, so xfsprogs need to be in the base case. My libvirt setup didn't actually install kvm, but in the super minimal install, that wasn't there. I hadn't gotten around to managing my openvpn server yet, that's in there now. If I was to do it again... One thing I really need is both a puppet and puppetmaster bootstrapping script. Using puppet to manage your puppetmaster is cool and all, but there is a bit of snake eating it's own tail to get you started that required a little more manual command slinging than I liked. But, had I not had so much of my server policy encoded in puppet, I'd still be typing commands now to get that box up and running. So I'm sold on the whole process, even for a smallish IT environment, like a few home servers and remote guests.

Last night our MHVLUG meeting was a Desktop Shootout, where people showed off their Linux desktop environments what they liked and what they didn't. I went last, presenting Unity, and got a few good questions and comments about how I did things.

White listing the Tray

I have a number of apps (RedShift GUI and an IBM firewall thing being prime examples) that minimize to the system tray. Under default Unity this means there is no UI for them. So I've done a broad white listing all all applications in the tray. It definitely looks a little uglier now, but at least it works. :)

Alt Tab

The default alt tab behavior drives me nuts, because it collapses 2 chrome windows into a single item. So you can't quickly use it to flip between 2 browser windows. This is provided by the Unity compiz plugin. Fortunately you can just load up one of the other compiz switchers instead. Run ccsm (Compiz Config Settings Manager) and scroll down to the Window Manangement section. You'll want to enable "Application Switcher" and configure Next Window (All windows) key stroke there. If there are other questions you think off, please just ask in comments, and I'll update the post with the answers.

There is a pretty interesting look at the CEO of Barnes & Noble this week in the NY Times. It shows how much of a David and Goliath fight B&N is in for, with 1% of the valuation of Amazon who they are trying to compete with. I have very mixed feelings about Amazon, and continue to have mixed feelings about my kindle, and the closed nature of the device. But I'm becoming less and less a fan of the book publishers. They seem to just be missing the point that their old pricing model, and scarcity model, doesn't work any more. Their insistence on pricing control dramatically makes me buy less ebooks. An unlendable ebook has an intrinsic value of $5 or less to me. They are priced typically at 3 times that, which has made me a frequent buyer of used hardcover from ... Amazon, where no one other than Amazon is making any money on it. If ebooks came without DRM, so I was sure I'd still be able to reread it in 4 decades, or could lend my mom & dad the book once I was done with it, then the current $10 - $15 range would be something I'd be fine with. Though I expect I'd still purchase more dollars worth of books over all if they were priced closer to $5. And then, there is the scarcity issue. Richard Wiseman, an established author, couldn't get his book Paranormality published by any of the american publishers because it says ghosts aren't real. American publishers are so focused on cranking out supernatural to their readers, that they block out anything that calls that into question. Failing to get an american traditional publisher, he self published on Amazon and Apple in ebook form. All of which makes the book publishers look, feel, smell, a lot like other big media, and completely out of touch with what their paying audience is interested in.

If you have a website, or have any creative input into a website, this is a book that is a must read. When people come to your website, they are looking for something. And the number one lesson is don't make them think, make it obvious. Through repeated examples, Krug will show you sites that look nice, but that completely confuse their users, and how he would correct them. You will immediately want to redo your site navigation after reading this. And you'll have a much cleaner overall look once you are done. Buy this book, read it, and make your little corner of the inter webs a better place.