I am about to start a big project for work, and I wanted to get the latest tools on my machine because I know I won’t have the opportunity for a while. Unfortunately, my install of Kubuntu Hardy 8.04 is starting to feel behind-the-times and the latest versions of my applications are getting increasingly hard to install as they depend on newer libraries. However, I have been very nervous to upgrade. I love KDE 3.5; it is a nearly perfect mix of stability, configurability, and usability; perhaps I’ve just grown very used to it. I avoided upgrading to the unfinished KDE 4.0, and have been following KDE4 closely while waiting for it to get complete enough not to drive me crazy. It looks almost there. So last week, when the project start got delayed yet again, I tackled upgrading to the freshly released Alpha 6 of Kubuntu Karmic.

Now I understand that alpha-quality software is not for production use. I also understand that the jump from KDE3 to KDE4 is a big one. However, Karmic is coming along nicely, and Alpha 6 looks almost there. My logic went like this: get the reinstall out of the way, upgrade to ext4, get used to KDE4, get the configuration file merging headache out of the way, and get everything working in the KDE4 version of my most important apps. Then it should be easy to aptitude dist-upgrade through the betas to the release even in the middle of a firestorm.

I expected this project would take a day or two, and it went more or less according to plan, but the experience was more painful than I expected. Perhaps my notes might help out others.

Before upgrading I did some due diligence. I checked the bug database, and didn’t see anything that would obviously effect me. I reviewed the development mailing list logs. Finally, I hung out on IRC for a couple of days and got opinions on the stability of the development branch. At that point I had collected enough courage to jump in.

Impressions of Karmic

For me the big change from Hardy to Karmic is KDE4. However, I did notice:

• The installer is really fast and easy to use.
• The system boots far faster.
• There is no more grub screen by default.
• I have a bunch of new folders in my home directory that I don’t need and don’t want: Documents, Pictures, Videos, Templates. The Desktop folder is being deprecated. And now they make me use a Download folder (instead of sticking it in Desktop). I’m sure this can all be configured away.

Impressions of KDE4

Summary

It is usable, but not yet as good as 3.5.

Plasma

My first impression of plasma was overwhelmingly negative. I could appreciate the configurability, but found it exceptionally painful to get things organized in a way that supported my preferred workflow. I find plasma’s interaction between panels, activities, dashboards, and virtual desktops to be too complex—especially on my dual monitor setup. It’s a lot simpler with the “one activity per virtual desktop” setting introduced in KDE 4.3, but it still isn’t intuitive. Riddle-me this: 3 virtual desktops across 2 monitors with 1 separate dashboard equals 8 activity panes. It also isn’t obvious what will happen when I disconnect a display. I suspect some of the confusion is related to inconsistent behavior as a result of bugginess.

I also find plasma to be slower than KDE 3.5. Again, it is mostly the effect of zooming out to configure my activities and desktops (the slowness contributes to the feeling of bugginess). It doesn’t seem as snappy and responsive. On the other hand, there are lots of nice effects and they run a lot faster than equivalent effects under the compiz setup I played with previously (and then turned off). The effects are well integrated into system settings and much easier to navigate and tweak than under compiz.

Perhaps the problem with plasma is just having non-intuitive defaults. Things got easier once I deciphered the nomenclature (the weirdest is how they call the “edit plasmoid bubble” a “cashew”; while figuring out the desktop I started feeling an overwhelming urge to get a snack). I started over by deleting all my activities and I let the “one activity per virtual desktop” button set everything up for me. I think I had gotten myself into a bad state while playing around. Now that I’m actually able to use the setup, plasma is really growing on me. It does make it easy to do things that were previously hard to conceive.

I think the paradigm can be just as useful and a lot simpler by merging the idea of activities and virtual desktops. There should then be an option for “virtual desktop per monitor” or “virtual desktop across all monitors”.

Scorecard

Problems with Clean Solutions

• Kopete fails to connect to Google Talk. Workaround: aptitude install qca-tls.
• KNetworkManager doesn’t like WPA on my access point. Workaround: use wicd. Bug#: 434342
• Default weather widgets won’t search for my city. Workaround: use plasma-widget-weatherforecast. Bug#: 434254
• The change monitor button (Fn-F7, a.k.a. the video button) didn’t work on my ThinkPad T61. This is the same as previous version of Ubuntu (but the monitor naming changed a little bit). Thinkwiki has a solution. Essentially, drop the script into /usr/local/bin, and create a file in /etc/acpi/events/ibmvideobtn that says:
  event=ibm/hotkey HKEY 00000080 00001007
action=/usr/local/bin/thinkpad-fn-f7
  A copy of the script is here.

Problems with Ugly Workarounds

• Logging out crashes X and takes the keyboard with it. Workaround: reboot. Bug#: 428662
• Can’t unlock screen. Workaround: run killall kscreenlocker from a virtual terminal. Bug#: 434276
• Spaces break bash tab completion. Workaround: prepend command with a slash (). Bug#: 419509
• Kopete won’t connect to jabber through a CNAME. Workaround: Use the A record or IP address. Bug#: 434214
• Kmail can’t pass options to GVim. Workaround: use gvim -f %f as the command and lose the options. Bug#: 434180
• Weird cups problems where cupsctl wants a password on bootup, and then reports “unathorized”. This is easy to ignore, but I got another authorization error while trying to install one of my printers—it reported my password as incorrect. It turns out that temporarily disabling apparmor with /etc/init.d/apparmor stop allowed me to get the task done. Debian Bug#: 543468.
• Corner actions in SystemSettings→Desktop→Screen Edges conflicts with the ones in SystemSettings→Desktop→Screen Saver→Advanced Options. I didn’t log this one.

Annoying Missing Features

• Severity 1: It does not appear to be possible to get the Application Launcher to display under the mouse pointer through a click on the background. I use this all the time, and it drives me crazy to have to move my mouse all the way to the panel on the edge of my dual-monitor widescreen display. Typing it makes it sound petty, but it drives me nuts.
• Severity 4: Konsole doesn’t save a default window size anymore. Whatever size my last closed window has is the size the next one will open with. It’s annoying because I like my terminals to be exactly half of my screen, but sometimes I will adjust it for a specific task. Now every time I adjust it I have to fiddle it back.

Summary

The good news is that in my judgment KDE 4.3 has only one really annoying missing feature. That means that KDE 4.3 is pretty feature complete in comparison to KDE 3.5. Now that I’ve got a usable system, I’m really happy with it. Though this is definitely a usable alpha, I am anxiously looking forward to the final release of Karmic Koala. Hopefully it will include fixes to some of these problems.

Getting Back to Work

Of course the moment I pass the no-turning-back point of the upgrade, I got notified of the start of my project. That means that messing with the alpha has put me behind schedule. I’d better stop submitting bug reports and buckle-down to put my shiny new tools to work.

Banjo Six Months Later

Banjo is a good foundation for a blog engine, but it needs some work. It is certainly quicker to implement than writing your own blog engine, but it is not as finished as I had hoped. I fixed a number of bugs, and the banjo maintainer was nice enough to give me write access to contribute my changes back. However, my goal was to work with a community where I could learn from experienced Django developers, and there isn’t much of a community around Banjo right now.

Discovering Byteflow

When fireant came across my blog post on Banjo, he asked me why I wasn’t using Byteflow. My answer was that I had not heard of Byteflow. I started looking into it, and it looked like a good alternative to Banjo.

My first impression of Byteflow was very similar to Banjo: lots of features, lots of potential, but it looked like a dead project. The server wasn’t responding well, the homepage news was out of date, and the mailing lists were full of spam.

Then I looked at the IRC logs and found an active community that mostly speaks an Eastern European language (Russian or Ukrainian, based on the location of the authors). When I asked a couple of questions in English, I got very friendly and helpful responses from piranha, the main Byteflow author. Not only are the Byteflow developers active, but they are open to contributions. I helped clean up the newsgroup spam, and updated the homepage to reflect the release last spring. And as I adapted Byteflow to the specifics of my installation, they accepted my patches and got them into the code base in the same day.

So far Byteflow has exceeded my expectations. Not only has it been a reliable blog engine (for the last two days anyway), it also incorporates many of the features I was hoping to implement* in Banjo. Most of all, I’ve been learning a lot from working with the Byteflow developers (piranha already showed me a few cool Mercurial tricks). That is my biggest justification for spending time on a Django blog instead of using a mature product. I’ve been the only tech guy at my family startup for the past year, and I’ve needed to find a project where I can keep my skills fresh by working with others.

* I’m finally an OpenID provider! After almost ten years, I signed up for a Slashdot account just to test it out.

Setting up Byteflow

This week I had two free days between work projects, so I decided to migrate my blog to Byteflow. The installation went very smoothly—the best I have experienced with a Django app.

Dependencies

Byteflow has no dependencies that needed to be filled outside of a typical package management system.

Base (all from Aptitude):

• Debian Lenny
• Postgres
• Django 1.1 (1.0 should work)
• Python 2.5.2
• Apache
• mod_python*

* I’m convinced that I should switch to mod_wsgi, but Byteflow works fine with my existing mod_python install.

From Aptitude:

• python-psycopg2 2.0.7-4
• python-openid 2.2.1-2
• python-beautifulsoup 3.0.7-1
• python-imaging 1.1.6-3

Installation

In the past the Byteflow project has announced releases, but I don’t see them packaged for download anywhere. It would be preferable to have a known functional configuration. Since it has been a couple of weeks without a series of commits to the Mercurial repository, I decided to have faith that it’s in working condition and I cloned the repo.

hg clone http://hg.piranha.org.ua/byteflow/

It is a small repo and a clean codebase. Everything worked out of the box by copying settings_local.py.template to setting_local.py, editing it to point to a clean database, running syncdb, and pointing Apache to the Byteflow settings.py. The default template came up, and a test post through the admin interface appeared.

Very easy, very slick.

Configuring Byteflow as an App

I like to keep code maintained by other people strictly separate from code I maintain. It was as easy to do that with Byteflow as with most Django apps, but did require some tweaking. The basic idea is to keep Byteflow in my shared_django_libs directory (I’ll call this PROJECT_ROOT), and have the apps Apache sees in a completely different directory tree (I’ll call my personal site REPO_ROOT).

I’ll define a couple of variables to make this easier to follow:

• PROJECT_ROOT: my clone of the Byteflow repository.
• REPO_ROOT: the repository where I keep my site.
• WEB_ROOT: the Apache web root where I keep static files, .htaccess files, and other stuff serve-able by Apache. The location is REPO_ROOT/WEB_ROOT.
• STATIC_ROOT: the directory where Django serves static files. The location is REPO_ROOT/WEB_ROOT/static.
• STATIC_URL: the URL where Apache can see static files. It is SITE_NAME + ‘static’.
• MEDIA_ROOT: The Byteflow settings.py differentiates between media files and static files. I treat them the same, so my MEDIA_ROOT = STATIC_ROOT.
• MEDIA_URL: As described above, my MEDIA_URL = STATIC_URL.
• SITE_ROOT: the directory containing my settings.py and other local Django code. It’s at REPO_ROOT/SITE_ROOT.
• THEMES_DIR: The directory where my local themes live. It is SITE_ROOT/THEMES_DIR.

1. Copy the default settings.py into your app (not settings_local.py).
2. Edit your settings.py as follows:
   • At the top of the file, just before the lines where it adds ‘apps’ and ‘compat’ to the sys.path, set PROJECT_ROOT to point at your byteflow install. Then add this line:
     sys.path.insert(0, os.path.join(os.path.dirname(PROJECT_ROOT)))
   • Add variables for SITE_ROOT and WEB_ROOT. Then use those variables to define STATIC_ROOT, MEDIA_ROOT, and THEMES_DIR.
   • Remove the loading of settings_local.py.
3. Create a file containing your SECRET_KEY, and point your settings.py to it (I used the SITE_ROOT variable).
4. Add these lines at the bottom of your urls.py, where it can catch any URLs you don’t already deal with:
   import byteflow.urls
urlpatterns += byteflow.urls.urlpatterns
5. Copy from PROJECT_ROOT/static to STATIC_ROOT these directories: css, img, js.

That’s all I remember it taking. Now Byteflow plays happily with everything I’m already running.

Creating a Custom Theme

Of course the default theme doesn’t look like _my_ ugly blog. To get my distinctive look, I had to create a custom theme. It took a while to get everything looking right, but the process wasn’t really difficult. It mostly involved me merging the important parts of Byteflow’s default base.html into my existing template, and looking at the other Byteflow themes to figure out how to make changes to the Byteflow pages.

• In THEMES_DIR, create a directory with the name of the theme you specified in your settings.py
• It needs a base.html in that directory. My base.html includes the contents of PROJECT_ROOT/templates/base.html, header.html, and footer.html.
• You can over-ride any template in PROJECT_ROOT/templates by having a template with the same name and directory structure in your theme. For example, I changed the way a blog post looks by copying post_entry.html from PROJECT_ROOT/templates/blog into THEMES_DIR/my_theme/blog, and editing it. You can also override templatetags by copying them from PROJECT_ROOT/templates/templatetags into THEMES_DIR/my_theme/templatetags. I changed the way the dates looked by stealing the datelinks,html from one of the included Byteflow templates. (Thanks for the tip on that one, piranha.)
• Create a directory in STATIC_ROOT with your theme name, and then you can override the contents of css, img, and js. Took my existing CSS file, called it main.css, and then added a couple of entries to change the look of Byteflow generated elements.

Importing the Data from Banjo

Since I only had 30 or so posts to migrate, I figured I’d just do it by hand to gain experience with Byteflow’s interface and the various markup languages. It also gave me the chance to correct some typos and formatting problems introduced during the last two migrations and the use of Banjo’s unpredictable markup engine. Though I fixed a lot, unfortunately I suspect I added some new typos.

Before I installed Byteflow, I did a pg_dump on my blog database. Then I dropped and recreated the database for Byteflow. I then used that dump to enter each post through the Admin interface.

I couldn’t bear to lose the comments on my previous blog (all two of them), but the Admin UI doesn’t allow comment creation. So I submitted the comments through the blog interface, and edited the DB to have the original timestamps. Unbeknownst to me, Byteflow sent out an email for each comment alerting the original submitter that they had just created an account on my site. One of them contacted me, worried about being blamed for something they didn’t do. If you decide to do a similar migration, disable Django’s email sending ability until you complete the migration.

Problems with reStructered Text

I couldn’t use reStructured Text out of the box. I was getting weird errors that perfectly matched the bug reported here: http://code.djangoproject.com/ticket/6681

The workaround discussed there worked for me. All you need to do is comment out this line in django/contrib/admindocs/utils.py: docutils.parsers.rst.roles.DEFAULT_INTERPRETED_ROLE = 'cmsreference'

With that change, a was able to use reStructured Text on two posts, but the third had strange docutils errors showing up in my page. I didn’t have time to turn that in to a reproducible bug report, so I just switched to Markdown.

Impressions

Banjo features not in Byteflow

There are a few things I preferred in Banjo:

• Byteflow has no concept of a “Post Summary” and a “Post Excerpt”. The entire post shows up on the home page. I prefer this, but some people might not.
• Byteflow has a good tagging engine, but it has no concept of hierarchical Categories. Personally, I like Categories better than tags, but I’m willing to live without them for now (I’m emulating them with tags).
• Byteflow has no concept of a post update time. This also doesn’t bother me, but some might care.
• Byteflow can not configure the URL to a blog post in the same way as Banjo. I set Banjo up to by YYYY/slug, but byteflow insists on YYYY/MM/DD/slug. Given how rarely I post, I don’t need such a complex URL structure.

Best things about Byteflow

There is a lot to praise, but at the top of my list is the very helpful developer community (as represented by piranha). I appreciated their willingness to accept my patches and allow me to contribute.

The code is well organized and easy to modify.

I also like the way it handles comment spam. When a person posts a comment, a site account is automatically generated and an email is sent to the comment author. The comment is not viewable until it has been approved by either the site administrator or through the email sent to the comment author. In the future the author can log in to post without needing any moderation. This scheme should eliminate the majority of comment spam which is produced by people using non-legitimate email addresses. However, anonymity is sacrificed. I’m not sure if I prefer it to reCAPTCHA—that will take some thought.

Annoyances

There are a couple of design decisions that bother me a little bit, like the lack of admin control over comments and how subscriptions are handled, but these are pretty minor.

I worry about having problems with my data during upgrade, but the author is working on South integration so I’m hoping that is addressed soon.

The Byteflow.su project server responds to me only intermittently, but I’m starting to suspect it’s DNS related and might be on my end.

I’m not sure I understand the authentication system. The default template has some buttons related to authentication that mystify me (“Close”?). But it appears to be working okay.

On the whole, I have very few annoyances with Byteflow. The only big one I had (not being able to store themes outside of the byteflow root), was quickly addressed when they incorporated my patch.

Conclusion

As usual, the migration took longer than I expected, but it was less than two days of work (including research into Byteflow, documentation work, contributing changes back, playing with the CSS of my theme, and importing data). Manually importing and formatting posts took half of a long day. A clean setup without all the additional activities would take just a couple of hours.

I like how the Byteflow authors run their project. The pace of development has slowed recently because piranha is happy with the current feature set and has largely moved on to other projects. However Byteflow is not abandoned; piranha still uses it as his primary blogging platform. Since he is readily available on IRC and active on the mailing list, it is easy to discuss the architecture with him and make contributions. I am betting that I’ll be happy with Byteflow for a long time to come.

Summary: Herein is shown how to work around the problems in Excel 2007 to create a multi-series, multi-axis, dynamic bar chart.

First, I’ll air my complaints about Microsoft Office 2007. I am not a fan of Microsoft Office, and I find the ribbon UI to be a lamentable step away from usability. Though I can appreciate that it makes some operations more discoverable to the new user, it makes complex uses frustrating as the selection of buttons and menus is constantly changing. The pretty pictures take up more of my precious vertical real estate than a menu would, and I can’t drag the ribbon to the side like I could with a toolbar (that one at least should be fixable). I also dislike the toolbar icons in the title-bar; they are further away, and I frequently have trouble dragging a window around as the status bar is mostly icons and thus non-selectable. The 2003 UI that OpenOffice copied is not great, but I don’t think the 2007 changes are improvements. I look forward to the day when my clients will save the money and let me work in OpenOffice all the time.

With that out of my system, I can explain my recent wrestle with Excel 2007. My client asked me to create a multi-series, multi-axis, dynamic range bar chart. My first reaction to the Excel 2007 charts was amazement—the default charts are very nice and simple to create. Like most of the “improvements” in Excel 2007, they look good until you buy the product and try to get some real work done. Then you realize that they broke all the power tools in order to make the initial experience glitzy. These charts were surprisingly difficult, which is as much due to the buggy application as to an awkward interaction.

The request was to create a bar chart with team member names along the bottom. Each team member would have two columns tracking different types of data: total number of actions, and total value of those actions. The total number of actions would be the value along the left vertical axis, and the total value would be along the right vertical axis. They want to be able to add and remove team members, and have the chart update dynamically.

So you can see what I’m talking about, here is a screen shot of the finished chart:

Steps

Get the data you want into a table:

Three columns: team member name, number of actions, value of actions.

Create a standard multi-series bar chart:

1. Ribbon→Insert→Column Chart→2D Clustered Column
2. Drag it where you want it.
3. Right click on it and select “Select Data”
4. Highlight the columns you want to include, plus two additional blank columns.
   • The additional blank columns will be used to space the populated columns so that the two axes don’t overlap.
5. In the “Select Data Source” dialog, make sure that the data categories (team member names) are the “Horizontal Axis Labels” and that the two sets of data values, plus the blank columns, are the “Legend Entries (Series)”. Order the blank columns (blank series) to be between the two sets of data. Then click OK.
   • Excel will order the series correctly if you put the blank columns between your populated columns in the table. If you choose to then hide the columns, you will need to select “Hidden and Empty Cells” in the “Select Data Source” and click on “Show data in hidden rows and columns”. “Show empty cells as Gaps” should also be selected. We skip this step because when we put in the named ranges we can select a blank series anywhere we want, eliminating the need for hidden columns.
6. You can manipulate most chart components by clicking directly on the chart to select the item, then right clicking and selecting “Format “. When that is not possible, such as for the blank series, go to the Ribbon→Layout and on the left side there is a drop down which displays the various chart components allowing you to select them and manipulate their attributes by clicking on “Format Selection”.
   1. Select Series 1 (Num Actions). Confirm that the series is being plotted on the Primary Axis.
   2. Select Series 2 (Blank Series). Confirm that it is also being plotted on the Primary Axis.
   3. Select Series 3 (Blank Series). Change it to be plotted on the Secondary Axis.
   4. Select Series 4 (Action Value). Change it to be plotted on the Secondary Axis.
      • If the columns overlap in any way, then you have the blank series in the wrong relationship to the populated series. The primary axis needs a populated series on the left (Series 1) and a blank series on the right (Series 2). The secondary axis then needs a populated series on the right (Series 4) and a blank series on the left (Series 3). You can change the number of the series looking at the formula bar while that series is selected. It is the last number in the SERIES formula.

Clean up the chart formatting

• This chart will need lots of horizontal room, so I move the labels underneath the chart by selecting the legend as a whole, right clicking, selecting “Format Legend” and changing the position to “Bottom”. I also like to widen the legend to take up the whole bottom, so that the names are closer to their respective axes.
• Select the legend items for the blank series, and press the Delete key to remove them from the legend. Your first click should select the legend as a whole, then additional clicks will tell Excel you want the item within the legend. You can also right click on select “Delete” (not “Delete Series”).
• The total amounts can be pretty large, so I select the right axis, choose “Format”, and change the “Display Units:” to Millions.

Make the chart dynamic

1. Ribbon→Formulas→Name Manager→New
2. The Name must not start with a “C”. I was starting all my named ranges with “Chart_” to differentiate them from the lists I use for data validation. Excel would accept the range, and then not let me edit the chart ever again. Sometimes after save and re-open the chart would be unreliable. And it was not dynamic. After much experimenting and Googling I tried changing the prefix to “Graph_” and suddenly everything is functional and reliable. Arg.
3. The Scope needs to be the sheet name (in my case “Team Unassigned”). When I used the Workbook as the scope, my SERIES formula would give a really generic error about something not being right. I don’t think it was able to find the named series. Changing the scope to the sheet name solved that error. This weird behavior, coupled with the buggy behavior listed above, gave me nightmares of confusion and anger.
4. It is in the “Refers to” box that we get dynamic. We use the OFFSET formula to create our range. This formula gets long, and you must use the mouse when you want to move within the box, as the arrow keys will affect the table and mess everything up (insert curse here). The OFFSET formula is (reference, row offset, column offset, [height], [width]). The reference is the first cell in our range. Since our reference point is within our range, the offsets are both zero. The width should be 1. The smarts is in the height, which we dynamically calculate by counting the populated cells in the source range with COUNTA. Since my cells are populated from another sheet, COUNTA will see them all as having a value. So I need to subtract COUNTBLANK to get the actual number of cells in the charted range. The formula I used is:
   =OFFSET('Team Unassigned'!$A$22,0,0, COUNTA('Team Unassigned'!$A$22:$A$340) — COUNTBLANK('Team Unassigned'!$A$22:$A$340),1)
   • If you have any blank cells in your charted range, Excel will give a very generic error and refuse to graph anything. Double check your formula that it only includes cells with chartable data.
   • Your other named ranges can be offsets from your first one, with formulas like this one:
     =OFFSET('Team Unassigned'!Graph_Unassigned_Name,0,1).
   • Whenever you want to use your named range, you use "'<sheet_name>'!<named_range_name>". I only use one named range for both blank series, and I define it to be an unused columns on the far right of my data, like so:
     Graph_Unassigned_Blank =OFFSET('Team Unassigned'!Graph_Unassigned_Name,0,26).
     This will use column AA for the range. My fourth named range is Graph_Unassigned_Value, selecting the total value of actions.
5. Change each series definition in the chart to use the dynamic named ranges. This is where if you have any problems with your named ranges you’ll get really vague and non-helpful errors. You can see an example of to do this in the above image.
   • Select the Series, and edit the SERIES formula in the formula bar to use the appropriate named ranges. The SERIES formula is (label_name, range_for_categories, range_for_values, series_num). My first series is:
     =SERIES('Team Unassigned'!$B$21,'Team Unassigned'!Graph_Unassigned_Name, 'Team Unassigned'!Graph_Unassigned_Num,1).
     I leave off the label_name for the blank series:
     =SERIES(,'Team Unassigned'!Graph_Unassigned_Name,'Team Unassigned'!Graph_Unassigned_Blank, 2).
     The only difference between the two blank series is the series numbers. The final series definition only has to change the last named-range to the new value range (Graph_Unassigned_Value in my case). You can use the arrow keys to move between the series definitions.

And that is all there is to it . Once you know how it’s done, it isn’t too bad.