Setting Environment Variables Locally for Django Development

While reading the book “Two Scoops of Django“, I learned many useful things about Django. One of the really useful tips in chapter 5 of the book is the isolation of secrets (e.g. API keys and database passwords) from the source code repository, through the use of environment variables. The concept is great, but I think there is a better way for setting the environment variables on your local development machine.

The book says to put the environment variables in .bashrc, .profile, or .bash_profile, or to put them in the bin/activate script of the virtualenv. This approach is not ideal, for the following reasons:

  1. Using .bashrc and friends means that your API keys and database passwords are always loaded whenever you use the shell, even when you don’t mean to work on your Django project;
  2. In the long run, it contaminates your shell environment with useless environment variables as you work on more and more projects;
  3. At the same time, you will likely be forced to prefix the environment variable names with your project names, to prevent conflicts, making the names longer than necessary;
  4. You have to manually clean up your .bashrc (or its equivalent) after your project’s end-of-life.
  5. You have to use different syntaxes as you migrate between Linux/Unix/Mac and Windows

Even if you go with bin/activate, you are still exposing your API keys and database passwords to the shell, and you have to clean them up in the deactivate function. It’s troublesome and mistake-prone. This is actually a consequence of how virtualenv is doing it wrong.

So here is my approach, which I believe is an improvement:

  1. In the site-packages directory of your virtualenv (something like “lib/python2.7/site-packages/”), define a new mysecrets.py module. The content of this module is like this:
    # `secrets` **must** be a simple dictionary.
    # i.e. no logic to programmatically generate keys or
    # values.
    secrets = {
        'postgresql_username': 'bob',
        'postgresql_password': '132!^a8!#)',
        # etc
    }
  2. Then, in the settings/local.py module of your Django project, import this mysecrets.py module and use the secrets function:
    from .base import *
    import mysecrets
    
    SECRETS = mysecrets.secrets
    
    # The rest of the usual content in local.py…

Now the SECRETS dictionary is even accessible through the django.conf.settings object.

This approach deals with the problems above while still achieving the effect of isolating secrets from the source code repository:

  • It is shell-agnostic because you are using Python to declare environment variables, and you certainly already know Python since you are working on a Django project;
  • You don’t leak your secrets through your shell;
  • You don’t contaminate your shell environment with useless variables;
  • You don’t have to prepend your variable names with your project name;
  • You don’t have to manually edit .bashrc (or its equivalent) in order to clean up, after your project’s end-of-life. You would probably delete your project’s virtualenv anyway.
  • You don’t have to edit the deactivate function in bin/activate to clean up the environment.

Update

@pydanny is concerned that this approach may degenerate into the local_settings.py anti-pattern. While his concern is understandable, I perceive it as a matter of good sense and self-discipline. Like I said in this thread in the Hong Kong Python User Group, anybody who has read @pydanny and @pyaudrey‘s Two Scoops of Django should understand the intention of isolating secrets while keeping all the real logic under version control. The module is purposely named mysecrets.py; anybody who adds anything other than secrets (e.g. API keys and database passwords) into this module simply fails English and should not be programming in any programming language at all. Like the old saying goes: we are all consenting adults.

An even safer approach would involve writing Windows INI-style config files and using Python’s ConfigParser to parse them, as suggested by Jimmy Wong. While this avoids the degeneration into the local_settings.py anti-pattern, it also entails knowing and using an extra syntax (INI-style config file syntax), however trivial it is. It’s a trade-off that I am hesitant to take. I’m lazy this way.

Comments are most welcome.

Update 2

I don’t remember why I originally wrote `secrets` as a function. In retrospec, it should be just a dictionary. It would look more declarative and be less prone to degenerate into the local_settings.py anti-pattern. I’ve updated the code snippets above.

PHP Kiddies

Yesterday, my friend Jawaad showed me an interesting piece of ugly PHP code. I reiterated the fact that PHP sucks and that it remains trolling the web application development landscape only because a bunch of incompetent, self-proclaimed web development gurus are feeding it. I thought I should provide some evidence to support my claim, so here I present you a tip of the iceberg:

 

Notice how many groups are found in the above search, and notice how I have already filtered the results to the “Internet & Technology” category. I wonder how many more groups I can find in the “Computers & Hardware” and “Cyberculture” categories. A few pages later…

 

Now notice how I’m on page 6 of the search results. I have tried to look into the following pages of search results, just to make sure that I do not exaggerate. However, I got tired by the time I clicked to page 25. Now, get creative, interpolate, and extrapolate. If you don’t believe me, do the search yourself. I’m not making this shit up.

These would be the same people who don’t understand one of the simple rules in the etiquette of using support forums: “Search before you post a question!” In other words, these are just a bunch of wanna-be-web-developer kiddies.

Things I would like to see in “Office 14”

1. Unicode VBA

Seriously, Microsoft, what part of “internationalisation” and “Unicode” did you not understand? Unlike your majority of US-based coders, we Asians often need to work with multiple languages. Just Japanese+Chinese+English is enough to completely break the programming experience in Office 2007. Yes, it was 2006, but that’s no excuse to stick with code-page stupidity when the Apple and Linux people had moved almost anything-GUI to Unicode already.

Nobody wants to switch regional settings and reboot the computer just to set a couple of fucking button captions in VBA, m’kay?

2. One, Unified Set of Widgets

Just who decided to have Form Controls and ActiveX Controls? Tell you what, chances are that, if a geek like me can’t be bothered to learn about two different sets of widgets, most users can’t be bothered either.

3. A Real-Programmer-Style VBA Reference

A bunch of disconnected explanations about Ranges and shit is not what I call a “reference.” Just because you call it “VBA” and “macro” programming doesn’t give you an excuse to disregard types. You may document the VBA object model as though type-safety does not matter, but the VBA runtime would still give me a big, obvious “Your types don’t match. You are stupid. And I won’t tell you what type you gave me either. Now fuck off and get lost.” when I don’t give it the type it expects because, hey, you guys never told me what exact type that method returned.

Why Does Extreme Programming Work?

Here is a piece of cynical thought: Extreme Programming works not only because of its inherent traits, but also because of some interesting side effect.

Let’s start with two simple questions: how many times have you been interrupted in your work today; be it by your boss, your superior officer, or one of your colleagues? What were you doing when you experienced the interruptions?

Those of you who, most of the time, work solo in your cubicle probably experience the most interruptions. And you were probably staring at the monitor, trying to make sense out of some important piece of code or technical document.

The reason you get interrupted is because other people do not believe that you are doing actual work, regardless of how legitimate your endeavour was. Think about it: when you are staring at the monitor, most likely the font is too small for a bystander to read with ease. If he cannot read with ease, he won’t bother with reading and will not be convinced that you are busy. Therefore he assumes that it’s ok to bother you.

Now, have you ever noticed how, sometimes, your boss walks towards you, only to realise that you are on the phone with somebody business-related, so she walks away within 5 seconds without bothering you? The reason your boss walks away is because she can hear all kinds of jargon in your conversation, so they assume (rightly) that you are onto something important.

I have come to hypothesise that one gets the fewest useless, unscheduled interruptions if he picks a working methodology that involves lots of talking, thereby letting uninvolved parties hear lots of jargon. When people hear jargon, they suddenly find reason to believe that you are doing actual work. For this reason, Extreme Programming becomes a good candidate. When you practice Extreme Programming, you are inevitably paired with a colleague, sharing one computer. When two people work together at the same computer, they inevitably talk a lot more than if each of them were working solo, and therefore they will let people around them hear a good load of jargon.

There you have it, the side effect that contributes to the effectiveness of Extreme Programming.

Note: this post is entirely meant as a meek piece of I.T. workplace satire. It is not to be taken seriously. Dear Kent Beck, if you happen to be reading this, please don’t get mad.

Gentoo Is Not for Everybody

Gentoo Linux is not for everybody, seriously. And by that I don’t mean only the average computer user, but also the majority of application developers. Of course, this is just my opinion, which I will explain below. I do not in any way mean that Gentoo Linux is inferior to other distros either; it’s great for people who care about customisation and speed, but not for those who just want a stable, working development environment.

To many people, the biggest attraction of Gentoo Linux is that you are effectively rolling your own flavour of Linux when you install it because you pick, configure, and build everything, even the installation CD. This aspect is different from most other Linux distros for which prebuilt installation CD images and binary software packages/repositories are available. The primary reason for picking, configuring, and building everything from source is the performance gain that results when the compiler and linker are optimising code for a particular architecture, even more so if object code is statically linked.

The majority of application developers, however, do not care about the performance of the OS hosting the development environment. Therefore, the performance gain from compiling everything from source is largely irrelevant to the application developer.

Furthermore, the difference in performance between Gentoo and other distros shall be significantly reduced nowadays. Gentoo, when it was still known as Enoch, used a fork of the GCC that provided about 10% performance gain over the mainstream, official version of GCC. This difference in performance no longer exists since the fork has been merged back into the official version of GCC. If you don’t care about the performance difference that remains, you might as well download binary packages; but if you are using binary packages, what’s the point in using Gentoo?

The Portage package system that Gentoo uses is also not something that the average application developer would like. Many of the packages have too many build options, and sometimes these very same options are under documented. For instance, the last time I counted, the package for the Apache server has around 60 build options, which do not exactly correspond to the options used with the configure script of the official source package from httpd.apache.org. Documentation of the options is impossible to find; even the popular Gentoo-Portage website has no information (http://gentoo-portage.com/www-servers/apache/USE#ptabs). Some of these options conflict with each other; the only way to eliminate all of the conflicting options is by repeatedly trying to install the package, allowing the portage commandline interface, emerge, to report conflicts one at a time, so that you can eliminate one of the conflicting options one at a time.

The application developer has enough to keep his hands busy and his mind on the brink of insanity. He cannot afford to lose time with compiling the development environment.

Gentoo may be great for top notch performance in the live, production environment; but if anyone asks me to pick a distro for development environment, I would pick Ubuntu over Gentoo any day.

Mobile development sucks – The cast

I think that the people involved in my story can be divided into the following four parties:
  • My Boss, a GM
  • Me and my colleague, (junior) consultants by title, we are actually more like software testers in this project 
  • Our programmers for the various mobile phone platforms
  • Our client, a mobile network operator in a foreign country

Do you realise what’s wrong with this cast? I think I DO.

Where the hell is the project manager, the person who is supposed keep everybody in check? When I say “everybody,” I make no exception to the GM because the GM does not always remember that the underlings are all up to their necks with work. A project manager would be the person who makes a scene with the GM to refresh the GM’s memory. Underlings like me are too busy hacking away already, you know; they don’t have time to defend themselves against the waves of order from the GM.

And why is there no software architect? Who is supposed to guide the design and specification of our product? Who is supposed to keep all the different platform versions from careless differentiation? It doesn’t take a genius to understand that I can’t act as the software architect: I have neither the knowledge nor the experience related to mobile development.

Oh, did I mention that there is no infrastructure support? The developers didn’t even have a version control server that they can commit their work into because there is nobody to set up such an environment and the developers certainly have neither the permission nor the time for such.

Mobile development sucks – Prologue

It’s been six months since my first full-time job has begun. The first month was spent in learning two products, of which I never got involved in any project. My boss finally decided to let me pick up a product development project that has been going on since before I joined the company. I must say that it has been a great learning opportunity so far and I want to share what little wisdom I have found.

Let’s start with some basic information:

  • The product is a software application for smartphones;
  • We are trying to support 5 smartphone platforms. I’m not talking about phone models – that would have been trivial. I’m talking about something more in line with a situation like Mac + Windows + Linux + Amiga.
  • As much as possible, the application is written in native code on all supported platforms (as opposed to Java, which would have been easier to port);

If you think that this is starting to stink, it gets better:

  • There is no vision document; the only available material that’s close to a vision document is a bunch of PowerPoint presentations;
  • There is no software requirements specification, not even an informal one;
  • There is no software architecture document, again, not even an informal one;
  • There is no software design document at all, again;
  • There is an overly simple blackbox testing checklist;

You probably saw this coming by now:

  • The programmers had started coding the damn thing and submitting test builds already.
If you think that this point above puts the final nail in the coffin, I am sorry that I must disappoint you. Read on.
  • There are, not one, but two versions of the product, with two different product activation models. Not only do you activate the two versions in different ways, but the sets of features that are disabled due to lack of activation are also different;
  • The two versions were being developed concurrently.
I suppose my dear reader has a good picture of the stage in mind now. I’ll let you sleep on it for a bit so you can imagine what can go wrong as the story unfolds itself.

One thing about Mac OS X that annoys me.

Keyboard input methods – this is one thing that has been bugging me since my switch to Mac OS X.

Actually it is more like 3 problems.

The first one is the way I have to cycle through input methods using Cmd+Space and Alt+Cmd+Space. I know 4 languages (Chinese, English, French, and Japanese) and as a result use 7 keyboard layouts/input methods daily (English, French, Simplified Chinese Pinyin, Traditional Chinese Pinyin, Cantonese Pinyin, Hiragana, Katakana). Cycling through the keyboard layouts and input methods is becoming tedious. I still prefer the Windows way, where I can assign a keyboard combination (e.g. Alt+Shift+1) to each keyboard layout and input method.

The second problem is the lack of official support for non-West-European input in the X11 environment. All attempts to build, install, and use UIM or SCIM have been unsuccessful.

Finally, the Traditional Chinese Pinyin input method seems to be a bit quirky. Sometimes it tends to feed out the wrong character. For instance, typing “zhang1” gives me “章” among the candidate characters, but if I select that character, the input method sometimes feeds the “杖” character instead.

Apple, please tackle these problems in the near future. I believe it will do you good in expanding your market in East Asia.

Date and Time in PHP 5 and MySQL 5

Time for a little technical post.

Date and time manipulation are one of the most common components in web applications. This is true for a few reasons:

  1. Web applications can have users coming from anywhere around the globe, under many different timezones.
  2. Chances are that you receive date and time information when the user does something. For instance, a user sends an e-mail and there is a timestamp in the message header.
  3. Chances are that you want to do something about the timestamp. For instance, sorting e-mails according to the timestamp before showing them to the user.

Unfortunately, the PHP date and time manipulation functions are anything but properly documented by the PHP documentation team. This causes a lot of grief and frustration among novice PHP programmers. Here, I will talk about how to handle dates and times in a PHP 5 + MySQL 5 setup.

Always store dates and times in the UTC timezone.

This is a good idea for 3 reasons:

  1. The UTC time reference is the most precise and least debatable there is on Earth because it is based on atomic time and takes the Earth’s rotation speed into account for compensation. Furthermore, it has never been, is not, and will never be subject to the daylight saving rule.
  2. The UTC is already the preferred reference timezone among application developers. The Unix Epoch, for instance, starts on midnight of January 1st, 1970, UTC time. Therefore, storing dates and times in UTC will allow easier interoperability with other applications right off the bat.
    1. A consequence of the above is that robust conversion routines to and from UTC is likely to be available in any programming language you pick (including PHP).
  3. You never know when you may need to move or deploy your web application to a host on a different timezone, therefore you don’t want to rely on the server’s timezone for storing and converting dates and times.

Dealing with date and time in PHP 5

Let’s face it, the php.net documentation sucks. One of the issues I see in it is that it fails to adapt to the OOP paradigm. It’s a shame that OOP has been introduced to PHP 4 and 5, and yet the documentation is still largely function-oriented. PHP 5 comes with a class library, but most people whom I have talked with do not know how to use it because the classes are not well documented. As such, most PHP 5 projects are still largely procedural. My goal here is to bring you a little bit of enlightenment by explaining how to use those classes.

  1. First, you need to create an object to represent the timezone. This step is not strictly necessary, but is recommended because in a globally connected world, a time value by itself is meaningless. The DateTime constructor can optionally take it as an argument, which is the way I recommend you to use it.
    $timezonename = 'America/Montreal'; 
    $mytimezone = new DateTimeZone($timezonename);

    Note that $timezonename can be any timezone name supported in the Olson timezone database (a.k.a. zoneinfo) or any of the few extra ones. You can see the full list in Appendix I of the php.net documentation.

  2. Only then do you create the DateTime object.
    $datetimestring = '2007-03-13 23:10:00';
    $mydatetime =
      new DateTime('2007-03-13 23:10:00', $mytimezone);

    The exact acceptable format of $datetimestring is the same as the GNU Date Input Formats, in case you have not found out from the php.net documentation already.

  3. Now, let’s try outputting it as a string. The bad news is that you should not do echo $mydatetime; The good news is that you can decide what string representation to take by using the DateTime::format() method.
    $mydatetimeformat = 'Y-m-d H:i:s e'; 
    /**
     * 'Y' for year, 'm' for month, 'd' for date,
     * 'H' for hours, 'i' for minutes, 's' for seconds,
     * and 'e' for the timezone name
     */
    echo $mydatetime->format($mydatetimeformat), 
      $mydatetime->format('I')?' DST':'';
  4. Perhaps you would like to see the date and time in another timezone? No problem, PHP has that covered.
    $mytimezone2 = new DateTimeZone('Asia/Tokyo'); 
    $mydatetime->setTimezone($mytimezone2); 
    echo $mydatetime->format($mydatetimeformat), 
      $mydatetime->format('I')?' DST':''; 

Word of caution

The constructor method of DateTime is supposed to return false on failure, so you may be under the impression that you can rely on it to validate date and time strings. For instance:

$montrealtimezone = new DateTimeZone('America/Montreal'); 
$bogustime =
  new DateTime('2007-03-11 02:30:00', $montrealtimezone);

The above snippet is supposed to result in $bogustime holding the value false because ’02:30:00′ is not a valid time on that date due to Daylight Saving Time entering into effect. Unfortunately, the constructor does not catch the mistake and will return you a DateTime object representing March 11th, 2007, two-thirty in the morning, Montreal Daylight Saving Time. It is a bug that I recently discovered in PHP 5.2.0 and submitted to the php bug database. It does not seem to be fixed as of this writing, where the latest stable released version of PHP is 5.2.1. You can check the status of the bug here: http://bugs.php.net/?id=40340.

Edit:
A comment by Eric has prompted me to clarify that the above bug is NOT caused by an outdated zoneinfo database. You can test it by trying to instantiate a similarly bogus date of a past year, while having the most up-to-date zoneinfo database installed.