Setting Environment Variables Locally for Django Development

While reading the book “Two Scoops of Django“, I learned many useful things about Django. One of the really useful tips in chapter 5 of the book is the isolation of secrets (e.g. API keys and database passwords) from the source code repository, through the use of environment variables. The concept is great, but I think there is a better way for setting the environment variables on your local development machine.

The book says to put the environment variables in .bashrc, .profile, or .bash_profile, or to put them in the bin/activate script of the virtualenv. This approach is not ideal, for the following reasons:

  1. Using .bashrc and friends means that your API keys and database passwords are always loaded whenever you use the shell, even when you don’t mean to work on your Django project;
  2. In the long run, it contaminates your shell environment with useless environment variables as you work on more and more projects;
  3. At the same time, you will likely be forced to prefix the environment variable names with your project names, to prevent conflicts, making the names longer than necessary;
  4. You have to manually clean up your .bashrc (or its equivalent) after your project’s end-of-life.
  5. You have to use different syntaxes as you migrate between Linux/Unix/Mac and Windows

Even if you go with bin/activate, you are still exposing your API keys and database passwords to the shell, and you have to clean them up in the deactivate function. It’s troublesome and mistake-prone. This is actually a consequence of how virtualenv is doing it wrong.

So here is my approach, which I believe is an improvement:

  1. In the site-packages directory of your virtualenv (something like “lib/python2.7/site-packages/”), define a new mysecrets.py module. The content of this module is like this:
    # `secrets` **must** be a simple dictionary.
    # i.e. no logic to programmatically generate keys or
    # values.
    secrets = {
        'postgresql_username': 'bob',
        'postgresql_password': '132!^a8!#)',
        # etc
    }
  2. Then, in the settings/local.py module of your Django project, import this mysecrets.py module and use the secrets function:
    from .base import *
    import mysecrets
    
    SECRETS = mysecrets.secrets
    
    # The rest of the usual content in local.py…

Now the SECRETS dictionary is even accessible through the django.conf.settings object.

This approach deals with the problems above while still achieving the effect of isolating secrets from the source code repository:

  • It is shell-agnostic because you are using Python to declare environment variables, and you certainly already know Python since you are working on a Django project;
  • You don’t leak your secrets through your shell;
  • You don’t contaminate your shell environment with useless variables;
  • You don’t have to prepend your variable names with your project name;
  • You don’t have to manually edit .bashrc (or its equivalent) in order to clean up, after your project’s end-of-life. You would probably delete your project’s virtualenv anyway.
  • You don’t have to edit the deactivate function in bin/activate to clean up the environment.

Update

@pydanny is concerned that this approach may degenerate into the local_settings.py anti-pattern. While his concern is understandable, I perceive it as a matter of good sense and self-discipline. Like I said in this thread in the Hong Kong Python User Group, anybody who has read @pydanny and @pyaudrey‘s Two Scoops of Django should understand the intention of isolating secrets while keeping all the real logic under version control. The module is purposely named mysecrets.py; anybody who adds anything other than secrets (e.g. API keys and database passwords) into this module simply fails English and should not be programming in any programming language at all. Like the old saying goes: we are all consenting adults.

An even safer approach would involve writing Windows INI-style config files and using Python’s ConfigParser to parse them, as suggested by Jimmy Wong. While this avoids the degeneration into the local_settings.py anti-pattern, it also entails knowing and using an extra syntax (INI-style config file syntax), however trivial it is. It’s a trade-off that I am hesitant to take. I’m lazy this way.

Comments are most welcome.

Update 2

I don’t remember why I originally wrote `secrets` as a function. In retrospec, it should be just a dictionary. It would look more declarative and be less prone to degenerate into the local_settings.py anti-pattern. I’ve updated the code snippets above.

PHP Kiddies

Yesterday, my friend Jawaad showed me an interesting piece of ugly PHP code. I reiterated the fact that PHP sucks and that it remains trolling the web application development landscape only because a bunch of incompetent, self-proclaimed web development gurus are feeding it. I thought I should provide some evidence to support my claim, so here I present you a tip of the iceberg:

 

Notice how many groups are found in the above search, and notice how I have already filtered the results to the “Internet & Technology” category. I wonder how many more groups I can find in the “Computers & Hardware” and “Cyberculture” categories. A few pages later…

 

Now notice how I’m on page 6 of the search results. I have tried to look into the following pages of search results, just to make sure that I do not exaggerate. However, I got tired by the time I clicked to page 25. Now, get creative, interpolate, and extrapolate. If you don’t believe me, do the search yourself. I’m not making this shit up.

These would be the same people who don’t understand one of the simple rules in the etiquette of using support forums: “Search before you post a question!” In other words, these are just a bunch of wanna-be-web-developer kiddies.

Date and Time in PHP 5 and MySQL 5

Time for a little technical post.

Date and time manipulation are one of the most common components in web applications. This is true for a few reasons:

  1. Web applications can have users coming from anywhere around the globe, under many different timezones.
  2. Chances are that you receive date and time information when the user does something. For instance, a user sends an e-mail and there is a timestamp in the message header.
  3. Chances are that you want to do something about the timestamp. For instance, sorting e-mails according to the timestamp before showing them to the user.

Unfortunately, the PHP date and time manipulation functions are anything but properly documented by the PHP documentation team. This causes a lot of grief and frustration among novice PHP programmers. Here, I will talk about how to handle dates and times in a PHP 5 + MySQL 5 setup.

Always store dates and times in the UTC timezone.

This is a good idea for 3 reasons:

  1. The UTC time reference is the most precise and least debatable there is on Earth because it is based on atomic time and takes the Earth’s rotation speed into account for compensation. Furthermore, it has never been, is not, and will never be subject to the daylight saving rule.
  2. The UTC is already the preferred reference timezone among application developers. The Unix Epoch, for instance, starts on midnight of January 1st, 1970, UTC time. Therefore, storing dates and times in UTC will allow easier interoperability with other applications right off the bat.
    1. A consequence of the above is that robust conversion routines to and from UTC is likely to be available in any programming language you pick (including PHP).
  3. You never know when you may need to move or deploy your web application to a host on a different timezone, therefore you don’t want to rely on the server’s timezone for storing and converting dates and times.

Dealing with date and time in PHP 5

Let’s face it, the php.net documentation sucks. One of the issues I see in it is that it fails to adapt to the OOP paradigm. It’s a shame that OOP has been introduced to PHP 4 and 5, and yet the documentation is still largely function-oriented. PHP 5 comes with a class library, but most people whom I have talked with do not know how to use it because the classes are not well documented. As such, most PHP 5 projects are still largely procedural. My goal here is to bring you a little bit of enlightenment by explaining how to use those classes.

  1. First, you need to create an object to represent the timezone. This step is not strictly necessary, but is recommended because in a globally connected world, a time value by itself is meaningless. The DateTime constructor can optionally take it as an argument, which is the way I recommend you to use it.
    $timezonename = 'America/Montreal'; 
    $mytimezone = new DateTimeZone($timezonename);

    Note that $timezonename can be any timezone name supported in the Olson timezone database (a.k.a. zoneinfo) or any of the few extra ones. You can see the full list in Appendix I of the php.net documentation.

  2. Only then do you create the DateTime object.
    $datetimestring = '2007-03-13 23:10:00';
    $mydatetime =
      new DateTime('2007-03-13 23:10:00', $mytimezone);

    The exact acceptable format of $datetimestring is the same as the GNU Date Input Formats, in case you have not found out from the php.net documentation already.

  3. Now, let’s try outputting it as a string. The bad news is that you should not do echo $mydatetime; The good news is that you can decide what string representation to take by using the DateTime::format() method.
    $mydatetimeformat = 'Y-m-d H:i:s e'; 
    /**
     * 'Y' for year, 'm' for month, 'd' for date,
     * 'H' for hours, 'i' for minutes, 's' for seconds,
     * and 'e' for the timezone name
     */
    echo $mydatetime->format($mydatetimeformat), 
      $mydatetime->format('I')?' DST':'';
  4. Perhaps you would like to see the date and time in another timezone? No problem, PHP has that covered.
    $mytimezone2 = new DateTimeZone('Asia/Tokyo'); 
    $mydatetime->setTimezone($mytimezone2); 
    echo $mydatetime->format($mydatetimeformat), 
      $mydatetime->format('I')?' DST':''; 

Word of caution

The constructor method of DateTime is supposed to return false on failure, so you may be under the impression that you can rely on it to validate date and time strings. For instance:

$montrealtimezone = new DateTimeZone('America/Montreal'); 
$bogustime =
  new DateTime('2007-03-11 02:30:00', $montrealtimezone);

The above snippet is supposed to result in $bogustime holding the value false because ’02:30:00′ is not a valid time on that date due to Daylight Saving Time entering into effect. Unfortunately, the constructor does not catch the mistake and will return you a DateTime object representing March 11th, 2007, two-thirty in the morning, Montreal Daylight Saving Time. It is a bug that I recently discovered in PHP 5.2.0 and submitted to the php bug database. It does not seem to be fixed as of this writing, where the latest stable released version of PHP is 5.2.1. You can check the status of the bug here: http://bugs.php.net/?id=40340.

Edit:
A comment by Eric has prompted me to clarify that the above bug is NOT caused by an outdated zoneinfo database. You can test it by trying to instantiate a similarly bogus date of a past year, while having the most up-to-date zoneinfo database installed.

How to properly embed Macromedia Flash objects

If any of you has been visiting this blog through Mozilla with the Tidy-based HTML validator extension, you may notice that the page now validates perfectly with no error and no warning at all, instead of having 1 warning like before. The problem had to do with the Flash movie from YouTube. The HTML code that YouTube supplied contained the <embed /> tag, which is not recognised by the W3C‘s HTML and XHTML specifications. Being a standard-compliant freak, I have been asking around for weeks about how to get rid of the warning; and then, just an hour ago, I read the HTML 4.01 specifications again and found >this<.

Before – the original code from YouTube:


<object width="425" height="350">
    <param name="movie"
        value="http://www.youtube.com/v/iIIoc8R26iM" />
    <embed src="http://www.youtube.com/v/iIIoc8R26iM"
        type="application/x-shockwave-flash"
        width="425" height="350" />
</object>

And after – the new code I figured out:


<object width="425" height="350"
    data="http://www.youtube.com/v/iIIoc8R26iM"
    type="application/x-shockwave-flash">
    Yellow Fever
</object>

Notice, however, that Microsoft Internet Explorer does not understand the second, correct version. It is a long-known fact that Internet Explorer’s HTML rendering engine is broken in many different places, with respect to the W3C specifications. Therefore, JavaScript needs to be used to determine which version to use:


<script type="text/javascript">
IEString = '<object width="425" height="350">'+
  '<param name="movie" '+
  'value="http://www.youtube.com/v/iIIoc8R26iM" />'+
  '<embed src="http://www.youtube.com/v/iIIoc8R26iM" '+
  'type="application/x-shockwave-flash" '+
  'width="425" height="350" />'+
'</object>';
MozString = '<object width="425" height="350" '+
  'data="http://www.youtube.com/v/iIIoc8R26iM" '+
  'type="application/x-shockwave-flash">'+
  'Yellow Fever'+
'</object>';

// The "browser" string is assigned by a nice
// browser detection script from quirksmode.org
if(browser == "Internet Explorer")
  document.write(IEString);
else
  document.write(MozString);
</script>

I hope this little piece of information helps those of you who want to use Flash on your webpages while being standard-compliant. I admit that the script does not take even other browsers like Opera, Safari, and Konqueror into account. I have not used those browsers long enough to know their behaviours off the back of my head. You are welcome to leave comments to help me improve the script. Notice also that, due to the default text processing options of WordPress, just pasting the script section inside a post usually does not work. You need to employ one of the solutions described >here<.