Migration Assistant cannot be trusted

So my rMBP arrived Thursday morning and I started a migration over LAN that night, before hitting the bed. It took 9 hours (i.e. until Friday morning) to drop to “1 minute remaining”. On the rMBP, Migration Assistant told me to plug in an ethernet cable to make the migration go faster. “But you don’t have a built-in ethernet jack anymore!” The irony.

And then it decided to hang at “1 minute remaining” for the remainder of yesterday. (WTF x 1)

So I said “fuck it” and aborted the migration. For the record Migration Assistant allows you to cleanly abort a migration, by pressing Cmd-Q. To see if I can speed up the migration, I removed the HDD from the old machine and put it in a USB 3.0 enclosure. I restarted a migration over USB. This time Migration Assistant tells me it will take about 2 hours. Much better.

And then the unthinkable happened:

At around the “37 minute remaining” mark, Migration Assistant stopped copying files and pretended it was done. (WTF x 2)

And it gave me a warning message to this effect: a user’s UID had to be reassigned from 502 to 501 and because the user also has a relocated home directory, I may need to manually correct the permissions after the migration. (WTF x 3)

Even a seasoned Mac user like myself find this warning message a little cryptic. “Thanks for not even telling me the username or relocated home directory path in question!” (WTF x 4)

I knew something was wrong, so I started digging into the newly migrated home directory…

“Where is my music? Where are my photos? Why am I missing random files and directories?” (WTF x 5)

And I’m not even talking about special files and directories that may have application-specific or machine-specific semantics – e.g. I can *somewhat* understand if you don’t completely mirror “~/Library” because that’s mostly profiles and settings, which can have application-specific semantics. And by “semantics” I really mean stupidities like the use of absolute paths where relative paths would be more appropriate. Instead, I’m just talking about ordinary files and folders: zip files, text files, etc.

At this point I said “fuck it” for a second time and just rsync’ed all the missing ordinary files and directories.

Setting Environment Variables Locally for Django Development

While reading the book “Two Scoops of Django“, I learned many useful things about Django. One of the really useful tips in chapter 5 of the book is the isolation of secrets (e.g. API keys and database passwords) from the source code repository, through the use of environment variables. The concept is great, but I think there is a better way for setting the environment variables on your local development machine.

The book says to put the environment variables in .bashrc, .profile, or .bash_profile, or to put them in the bin/activate script of the virtualenv. This approach is not ideal, for the following reasons:

  1. Using .bashrc and friends means that your API keys and database passwords are always loaded whenever you use the shell, even when you don’t mean to work on your Django project;
  2. In the long run, it contaminates your shell environment with useless environment variables as you work on more and more projects;
  3. At the same time, you will likely be forced to prefix the environment variable names with your project names, to prevent conflicts, making the names longer than necessary;
  4. You have to manually clean up your .bashrc (or its equivalent) after your project’s end-of-life.
  5. You have to use different syntaxes as you migrate between Linux/Unix/Mac and Windows

Even if you go with bin/activate, you are still exposing your API keys and database passwords to the shell, and you have to clean them up in the deactivate function. It’s troublesome and mistake-prone. This is actually a consequence of how virtualenv is doing it wrong.

So here is my approach, which I believe is an improvement:

  1. In the site-packages directory of your virtualenv (something like “lib/python2.7/site-packages/”), define a new mysecrets.py module. The content of this module is like this:
    # `secrets` **must** be a simple dictionary.
    # i.e. no logic to programmatically generate keys or
    # values.
    secrets = {
        'postgresql_username': 'bob',
        'postgresql_password': '132!^a8!#)',
        # etc
  2. Then, in the settings/local.py module of your Django project, import this mysecrets.py module and use the secrets function:
    from .base import *
    import mysecrets
    SECRETS = mysecrets.secrets
    # The rest of the usual content in local.py…

Now the SECRETS dictionary is even accessible through the django.conf.settings object.

This approach deals with the problems above while still achieving the effect of isolating secrets from the source code repository:

  • It is shell-agnostic because you are using Python to declare environment variables, and you certainly already know Python since you are working on a Django project;
  • You don’t leak your secrets through your shell;
  • You don’t contaminate your shell environment with useless variables;
  • You don’t have to prepend your variable names with your project name;
  • You don’t have to manually edit .bashrc (or its equivalent) in order to clean up, after your project’s end-of-life. You would probably delete your project’s virtualenv anyway.
  • You don’t have to edit the deactivate function in bin/activate to clean up the environment.


@pydanny is concerned that this approach may degenerate into the local_settings.py anti-pattern. While his concern is understandable, I perceive it as a matter of good sense and self-discipline. Like I said in this thread in the Hong Kong Python User Group, anybody who has read @pydanny and @pyaudrey‘s Two Scoops of Django should understand the intention of isolating secrets while keeping all the real logic under version control. The module is purposely named mysecrets.py; anybody who adds anything other than secrets (e.g. API keys and database passwords) into this module simply fails English and should not be programming in any programming language at all. Like the old saying goes: we are all consenting adults.

An even safer approach would involve writing Windows INI-style config files and using Python’s ConfigParser to parse them, as suggested by Jimmy Wong. While this avoids the degeneration into the local_settings.py anti-pattern, it also entails knowing and using an extra syntax (INI-style config file syntax), however trivial it is. It’s a trade-off that I am hesitant to take. I’m lazy this way.

Comments are most welcome.

Update 2

I don’t remember why I originally wrote `secrets` as a function. In retrospec, it should be just a dictionary. It would look more declarative and be less prone to degenerate into the local_settings.py anti-pattern. I’ve updated the code snippets above.

PHP Kiddies

Yesterday, my friend Jawaad showed me an interesting piece of ugly PHP code. I reiterated the fact that PHP sucks and that it remains trolling the web application development landscape only because a bunch of incompetent, self-proclaimed web development gurus are feeding it. I thought I should provide some evidence to support my claim, so here I present you a tip of the iceberg:


Notice how many groups are found in the above search, and notice how I have already filtered the results to the “Internet & Technology” category. I wonder how many more groups I can find in the “Computers & Hardware” and “Cyberculture” categories. A few pages later…


Now notice how I’m on page 6 of the search results. I have tried to look into the following pages of search results, just to make sure that I do not exaggerate. However, I got tired by the time I clicked to page 25. Now, get creative, interpolate, and extrapolate. If you don’t believe me, do the search yourself. I’m not making this shit up.

These would be the same people who don’t understand one of the simple rules in the etiquette of using support forums: “Search before you post a question!” In other words, these are just a bunch of wanna-be-web-developer kiddies.

Things I would like to see in “Office 14″

1. Unicode VBA

Seriously, Microsoft, what part of “internationalisation” and “Unicode” did you not understand? Unlike your majority of US-based coders, we Asians often need to work with multiple languages. Just Japanese+Chinese+English is enough to completely break the programming experience in Office 2007. Yes, it was 2006, but that’s no excuse to stick with code-page stupidity when the Apple and Linux people had moved almost anything-GUI to Unicode already.

Nobody wants to switch regional settings and reboot the computer just to set a couple of fucking button captions in VBA, m’kay?

2. One, Unified Set of Widgets

Just who decided to have Form Controls and ActiveX Controls? Tell you what, chances are that, if a geek like me can’t be bothered to learn about two different sets of widgets, most users can’t be bothered either.

3. A Real-Programmer-Style VBA Reference

A bunch of disconnected explanations about Ranges and shit is not what I call a “reference.” Just because you call it “VBA” and “macro” programming doesn’t give you an excuse to disregard types. You may document the VBA object model as though type-safety does not matter, but the VBA runtime would still give me a big, obvious “Your types don’t match. You are stupid. And I won’t tell you what type you gave me either. Now fuck off and get lost.” when I don’t give it the type it expects because, hey, you guys never told me what exact type that method returned.

Why Does Extreme Programming Work?

Here is a piece of cynical thought: Extreme Programming works not only because of its inherent traits, but also because of some interesting side effect.

Let’s start with two simple questions: how many times have you been interrupted in your work today; be it by your boss, your superior officer, or one of your colleagues? What were you doing when you experienced the interruptions?

Those of you who, most of the time, work solo in your cubicle probably experience the most interruptions. And you were probably staring at the monitor, trying to make sense out of some important piece of code or technical document.

The reason you get interrupted is because other people do not believe that you are doing actual work, regardless of how legitimate your endeavour was. Think about it: when you are staring at the monitor, most likely the font is too small for a bystander to read with ease. If he cannot read with ease, he won’t bother with reading and will not be convinced that you are busy. Therefore he assumes that it’s ok to bother you.

Now, have you ever noticed how, sometimes, your boss walks towards you, only to realise that you are on the phone with somebody business-related, so she walks away within 5 seconds without bothering you? The reason your boss walks away is because she can hear all kinds of jargon in your conversation, so they assume (rightly) that you are onto something important.

I have come to hypothesise that one gets the fewest useless, unscheduled interruptions if he picks a working methodology that involves lots of talking, thereby letting uninvolved parties hear lots of jargon. When people hear jargon, they suddenly find reason to believe that you are doing actual work. For this reason, Extreme Programming becomes a good candidate. When you practice Extreme Programming, you are inevitably paired with a colleague, sharing one computer. When two people work together at the same computer, they inevitably talk a lot more than if each of them were working solo, and therefore they will let people around them hear a good load of jargon.

There you have it, the side effect that contributes to the effectiveness of Extreme Programming.

Note: this post is entirely meant as a meek piece of I.T. workplace satire. It is not to be taken seriously. Dear Kent Beck, if you happen to be reading this, please don’t get mad.

Gentoo Is Not for Everybody

Gentoo Linux is not for everybody, seriously. And by that I don’t mean only the average computer user, but also the majority of application developers. Of course, this is just my opinion, which I will explain below. I do not in any way mean that Gentoo Linux is inferior to other distros either; it’s great for people who care about customisation and speed, but not for those who just want a stable, working development environment.

To many people, the biggest attraction of Gentoo Linux is that you are effectively rolling your own flavour of Linux when you install it because you pick, configure, and build everything, even the installation CD. This aspect is different from most other Linux distros for which prebuilt installation CD images and binary software packages/repositories are available. The primary reason for picking, configuring, and building everything from source is the performance gain that results when the compiler and linker are optimising code for a particular architecture, even more so if object code is statically linked.

The majority of application developers, however, do not care about the performance of the OS hosting the development environment. Therefore, the performance gain from compiling everything from source is largely irrelevant to the application developer.

Furthermore, the difference in performance between Gentoo and other distros shall be significantly reduced nowadays. Gentoo, when it was still known as Enoch, used a fork of the GCC that provided about 10% performance gain over the mainstream, official version of GCC. This difference in performance no longer exists since the fork has been merged back into the official version of GCC. If you don’t care about the performance difference that remains, you might as well download binary packages; but if you are using binary packages, what’s the point in using Gentoo?

The Portage package system that Gentoo uses is also not something that the average application developer would like. Many of the packages have too many build options, and sometimes these very same options are under documented. For instance, the last time I counted, the package for the Apache server has around 60 build options, which do not exactly correspond to the options used with the configure script of the official source package from httpd.apache.org. Documentation of the options is impossible to find; even the popular Gentoo-Portage website has no information (http://gentoo-portage.com/www-servers/apache/USE#ptabs). Some of these options conflict with each other; the only way to eliminate all of the conflicting options is by repeatedly trying to install the package, allowing the portage commandline interface, emerge, to report conflicts one at a time, so that you can eliminate one of the conflicting options one at a time.

The application developer has enough to keep his hands busy and his mind on the brink of insanity. He cannot afford to lose time with compiling the development environment.

Gentoo may be great for top notch performance in the live, production environment; but if anyone asks me to pick a distro for development environment, I would pick Ubuntu over Gentoo any day.

Mobile development sucks – The cast

I think that the people involved in my story can be divided into the following four parties:
  • My Boss, a GM
  • Me and my colleague, (junior) consultants by title, we are actually more like software testers in this project 
  • Our programmers for the various mobile phone platforms
  • Our client, a mobile network operator in a foreign country

Do you realise what’s wrong with this cast? I think I DO.

Where the hell is the project manager, the person who is supposed keep everybody in check? When I say “everybody,” I make no exception to the GM because the GM does not always remember that the underlings are all up to their necks with work. A project manager would be the person who makes a scene with the GM to refresh the GM’s memory. Underlings like me are too busy hacking away already, you know; they don’t have time to defend themselves against the waves of order from the GM.

And why is there no software architect? Who is supposed to guide the design and specification of our product? Who is supposed to keep all the different platform versions from careless differentiation? It doesn’t take a genius to understand that I can’t act as the software architect: I have neither the knowledge nor the experience related to mobile development.

Oh, did I mention that there is no infrastructure support? The developers didn’t even have a version control server that they can commit their work into because there is nobody to set up such an environment and the developers certainly have neither the permission nor the time for such.

:~ Kal$ ping *

I know. I have not blogged for a few months now. The same goes for most of my friends. It seems that being away from school does drain the very soul out of us. Everybody must be busy with something and has lost interest in blogging. Let me list some of the things you can be busy with:

  1. Aimless browsing of Facebook
  2. Ignoring the hundreds of Facebook requests that you get daily
  3. Spamming your friends with Facebook requests
  4. Playing a Facebook game
  5. Reading Digg, Reddit, Engadget, or Slashdot
  6. Starting/engaging in a flame war on one of the aforementioned sites
  7. boys/girls at the office
  8. your boss’ hot secretary
  9. your boss/your boss’ hot daughter
  10. game consoles (and games) for which you finally have the money
  11. overtime work
  12. cooking (because you are living on your own now)
  13. laundry (same reason as above)
Pretty much everyone has been slacking off as far as blogging goes, except Jawaad, Skrud, Spiro, and Kevin.
Hall of Shame:
  • Nadia – has not blogged at all for almost 2 whole years now;
  • Terry – lost his domain name to some domain squatter.
Eric is clever, updating his MSN Space with sets after sets of pictures instead of writing.
I know all of you must have made new friends. I have, too, even all the way in Japan. But gee, try to give sign of life from time to time and not forget about TSG and everybody else from Concordia.

Mobile development sucks – Prologue

It’s been six months since my first full-time job has begun. The first month was spent in learning two products, of which I never got involved in any project. My boss finally decided to let me pick up a product development project that has been going on since before I joined the company. I must say that it has been a great learning opportunity so far and I want to share what little wisdom I have found.

Let’s start with some basic information:

  • The product is a software application for smartphones;
  • We are trying to support 5 smartphone platforms. I’m not talking about phone models – that would have been trivial. I’m talking about something more in line with a situation like Mac + Windows + Linux + Amiga.
  • As much as possible, the application is written in native code on all supported platforms (as opposed to Java, which would have been easier to port);

If you think that this is starting to stink, it gets better:

  • There is no vision document; the only available material that’s close to a vision document is a bunch of PowerPoint presentations;
  • There is no software requirements specification, not even an informal one;
  • There is no software architecture document, again, not even an informal one;
  • There is no software design document at all, again;
  • There is an overly simple blackbox testing checklist;

You probably saw this coming by now:

  • The programmers had started coding the damn thing and submitting test builds already.
If you think that this point above puts the final nail in the coffin, I am sorry that I must disappoint you. Read on.
  • There are, not one, but two versions of the product, with two different product activation models. Not only do you activate the two versions in different ways, but the sets of features that are disabled due to lack of activation are also different;
  • The two versions were being developed concurrently.
I suppose my dear reader has a good picture of the stage in mind now. I’ll let you sleep on it for a bit so you can imagine what can go wrong as the story unfolds itself.


Kalunite.net and all sub-sites may be down for a few days from January 24th onwards while I migrate everything to a dedicated server in Hong Kong. The new server shall shift the access speed in the favour of visitors from the Asia-Pacific region, possibly at the expense of visitors from North America. Check back in a week or two and hopefully everything will be up and running smoothly again by then.

Hacker Reborn