Setting Environment Variables Locally for Django Development

While reading the book “Two Scoops of Django“, I learned many useful things about Django. One of the really useful tips in chapter 5 of the book is the isolation of secrets (e.g. API keys and database passwords) from the source code repository, through the use of environment variables. The concept is great, but I think there is a better way for setting the environment variables on your local development machine.

The book says to put the environment variables in .bashrc, .profile, or .bash_profile, or to put them in the bin/activate script of the virtualenv. This approach is not ideal, for the following reasons:

  1. Using .bashrc and friends means that your API keys and database passwords are always loaded whenever you use the shell, even when you don’t mean to work on your Django project;
  2. In the long run, it contaminates your shell environment with useless environment variables as you work on more and more projects;
  3. At the same time, you will likely be forced to prefix the environment variable names with your project names, to prevent conflicts, making the names longer than necessary;
  4. You have to manually clean up your .bashrc (or its equivalent) after your project’s end-of-life.
  5. You have to use different syntaxes as you migrate between Linux/Unix/Mac and Windows

Even if you go with bin/activate, you are still exposing your API keys and database passwords to the shell, and you have to clean them up in the deactivate function. It’s troublesome and mistake-prone. This is actually a consequence of how virtualenv is doing it wrong.

So here is my approach, which I believe is an improvement:

  1. In the site-packages directory of your virtualenv (something like “lib/python2.7/site-packages/”), define a new mysecrets.py module. The content of this module is like this:
    # `secrets` **must** be a simple dictionary.
    # i.e. no logic to programmatically generate keys or
    # values.
    secrets = {
        'postgresql_username': 'bob',
        'postgresql_password': '132!^a8!#)',
        # etc
    }
  2. Then, in the settings/local.py module of your Django project, import this mysecrets.py module and use the secrets function:
    from .base import *
    import mysecrets
    
    SECRETS = mysecrets.secrets
    
    # The rest of the usual content in local.py…

Now the SECRETS dictionary is even accessible through the django.conf.settings object.

This approach deals with the problems above while still achieving the effect of isolating secrets from the source code repository:

  • It is shell-agnostic because you are using Python to declare environment variables, and you certainly already know Python since you are working on a Django project;
  • You don’t leak your secrets through your shell;
  • You don’t contaminate your shell environment with useless variables;
  • You don’t have to prepend your variable names with your project name;
  • You don’t have to manually edit .bashrc (or its equivalent) in order to clean up, after your project’s end-of-life. You would probably delete your project’s virtualenv anyway.
  • You don’t have to edit the deactivate function in bin/activate to clean up the environment.

Update

@pydanny is concerned that this approach may degenerate into the local_settings.py anti-pattern. While his concern is understandable, I perceive it as a matter of good sense and self-discipline. Like I said in this thread in the Hong Kong Python User Group, anybody who has read @pydanny and @pyaudrey‘s Two Scoops of Django should understand the intention of isolating secrets while keeping all the real logic under version control. The module is purposely named mysecrets.py; anybody who adds anything other than secrets (e.g. API keys and database passwords) into this module simply fails English and should not be programming in any programming language at all. Like the old saying goes: we are all consenting adults.

An even safer approach would involve writing Windows INI-style config files and using Python’s ConfigParser to parse them, as suggested by Jimmy Wong. While this avoids the degeneration into the local_settings.py anti-pattern, it also entails knowing and using an extra syntax (INI-style config file syntax), however trivial it is. It’s a trade-off that I am hesitant to take. I’m lazy this way.

Comments are most welcome.

Update 2

I don’t remember why I originally wrote `secrets` as a function. In retrospec, it should be just a dictionary. It would look more declarative and be less prone to degenerate into the local_settings.py anti-pattern. I’ve updated the code snippets above.

PHP Kiddies

Yesterday, my friend Jawaad showed me an interesting piece of ugly PHP code. I reiterated the fact that PHP sucks and that it remains trolling the web application development landscape only because a bunch of incompetent, self-proclaimed web development gurus are feeding it. I thought I should provide some evidence to support my claim, so here I present you a tip of the iceberg:

 

Notice how many groups are found in the above search, and notice how I have already filtered the results to the “Internet & Technology” category. I wonder how many more groups I can find in the “Computers & Hardware” and “Cyberculture” categories. A few pages later…

 

Now notice how I’m on page 6 of the search results. I have tried to look into the following pages of search results, just to make sure that I do not exaggerate. However, I got tired by the time I clicked to page 25. Now, get creative, interpolate, and extrapolate. If you don’t believe me, do the search yourself. I’m not making this shit up.

These would be the same people who don’t understand one of the simple rules in the etiquette of using support forums: “Search before you post a question!” In other words, these are just a bunch of wanna-be-web-developer kiddies.

Things I would like to see in “Office 14″

1. Unicode VBA

Seriously, Microsoft, what part of “internationalisation” and “Unicode” did you not understand? Unlike your majority of US-based coders, we Asians often need to work with multiple languages. Just Japanese+Chinese+English is enough to completely break the programming experience in Office 2007. Yes, it was 2006, but that’s no excuse to stick with code-page stupidity when the Apple and Linux people had moved almost anything-GUI to Unicode already.

Nobody wants to switch regional settings and reboot the computer just to set a couple of fucking button captions in VBA, m’kay?

2. One, Unified Set of Widgets

Just who decided to have Form Controls and ActiveX Controls? Tell you what, chances are that, if a geek like me can’t be bothered to learn about two different sets of widgets, most users can’t be bothered either.

3. A Real-Programmer-Style VBA Reference

A bunch of disconnected explanations about Ranges and shit is not what I call a “reference.” Just because you call it “VBA” and “macro” programming doesn’t give you an excuse to disregard types. You may document the VBA object model as though type-safety does not matter, but the VBA runtime would still give me a big, obvious “Your types don’t match. You are stupid. And I won’t tell you what type you gave me either. Now fuck off and get lost.” when I don’t give it the type it expects because, hey, you guys never told me what exact type that method returned.

Why Does Extreme Programming Work?

Here is a piece of cynical thought: Extreme Programming works not only because of its inherent traits, but also because of some interesting side effect.

Let’s start with two simple questions: how many times have you been interrupted in your work today; be it by your boss, your superior officer, or one of your colleagues? What were you doing when you experienced the interruptions?

Those of you who, most of the time, work solo in your cubicle probably experience the most interruptions. And you were probably staring at the monitor, trying to make sense out of some important piece of code or technical document.

The reason you get interrupted is because other people do not believe that you are doing actual work, regardless of how legitimate your endeavour was. Think about it: when you are staring at the monitor, most likely the font is too small for a bystander to read with ease. If he cannot read with ease, he won’t bother with reading and will not be convinced that you are busy. Therefore he assumes that it’s ok to bother you.

Now, have you ever noticed how, sometimes, your boss walks towards you, only to realise that you are on the phone with somebody business-related, so she walks away within 5 seconds without bothering you? The reason your boss walks away is because she can hear all kinds of jargon in your conversation, so they assume (rightly) that you are onto something important.

I have come to hypothesise that one gets the fewest useless, unscheduled interruptions if he picks a working methodology that involves lots of talking, thereby letting uninvolved parties hear lots of jargon. When people hear jargon, they suddenly find reason to believe that you are doing actual work. For this reason, Extreme Programming becomes a good candidate. When you practice Extreme Programming, you are inevitably paired with a colleague, sharing one computer. When two people work together at the same computer, they inevitably talk a lot more than if each of them were working solo, and therefore they will let people around them hear a good load of jargon.

There you have it, the side effect that contributes to the effectiveness of Extreme Programming.

Note: this post is entirely meant as a meek piece of I.T. workplace satire. It is not to be taken seriously. Dear Kent Beck, if you happen to be reading this, please don’t get mad.

Gentoo Is Not for Everybody

Gentoo Linux is not for everybody, seriously. And by that I don’t mean only the average computer user, but also the majority of application developers. Of course, this is just my opinion, which I will explain below. I do not in any way mean that Gentoo Linux is inferior to other distros either; it’s great for people who care about customisation and speed, but not for those who just want a stable, working development environment.

To many people, the biggest attraction of Gentoo Linux is that you are effectively rolling your own flavour of Linux when you install it because you pick, configure, and build everything, even the installation CD. This aspect is different from most other Linux distros for which prebuilt installation CD images and binary software packages/repositories are available. The primary reason for picking, configuring, and building everything from source is the performance gain that results when the compiler and linker are optimising code for a particular architecture, even more so if object code is statically linked.

The majority of application developers, however, do not care about the performance of the OS hosting the development environment. Therefore, the performance gain from compiling everything from source is largely irrelevant to the application developer.

Furthermore, the difference in performance between Gentoo and other distros shall be significantly reduced nowadays. Gentoo, when it was still known as Enoch, used a fork of the GCC that provided about 10% performance gain over the mainstream, official version of GCC. This difference in performance no longer exists since the fork has been merged back into the official version of GCC. If you don’t care about the performance difference that remains, you might as well download binary packages; but if you are using binary packages, what’s the point in using Gentoo?

The Portage package system that Gentoo uses is also not something that the average application developer would like. Many of the packages have too many build options, and sometimes these very same options are under documented. For instance, the last time I counted, the package for the Apache server has around 60 build options, which do not exactly correspond to the options used with the configure script of the official source package from httpd.apache.org. Documentation of the options is impossible to find; even the popular Gentoo-Portage website has no information (http://gentoo-portage.com/www-servers/apache/USE#ptabs). Some of these options conflict with each other; the only way to eliminate all of the conflicting options is by repeatedly trying to install the package, allowing the portage commandline interface, emerge, to report conflicts one at a time, so that you can eliminate one of the conflicting options one at a time.

The application developer has enough to keep his hands busy and his mind on the brink of insanity. He cannot afford to lose time with compiling the development environment.

Gentoo may be great for top notch performance in the live, production environment; but if anyone asks me to pick a distro for development environment, I would pick Ubuntu over Gentoo any day.

Mobile development sucks – The cast

I think that the people involved in my story can be divided into the following four parties:
  • My Boss, a GM
  • Me and my colleague, (junior) consultants by title, we are actually more like software testers in this project 
  • Our programmers for the various mobile phone platforms
  • Our client, a mobile network operator in a foreign country

Do you realise what’s wrong with this cast? I think I DO.

Where the hell is the project manager, the person who is supposed keep everybody in check? When I say “everybody,” I make no exception to the GM because the GM does not always remember that the underlings are all up to their necks with work. A project manager would be the person who makes a scene with the GM to refresh the GM’s memory. Underlings like me are too busy hacking away already, you know; they don’t have time to defend themselves against the waves of order from the GM.

And why is there no software architect? Who is supposed to guide the design and specification of our product? Who is supposed to keep all the different platform versions from careless differentiation? It doesn’t take a genius to understand that I can’t act as the software architect: I have neither the knowledge nor the experience related to mobile development.

Oh, did I mention that there is no infrastructure support? The developers didn’t even have a version control server that they can commit their work into because there is nobody to set up such an environment and the developers certainly have neither the permission nor the time for such.

:~ Kal$ ping *

I know. I have not blogged for a few months now. The same goes for most of my friends. It seems that being away from school does drain the very soul out of us. Everybody must be busy with something and has lost interest in blogging. Let me list some of the things you can be busy with:

  1. Aimless browsing of Facebook
  2. Ignoring the hundreds of Facebook requests that you get daily
  3. Spamming your friends with Facebook requests
  4. Playing a Facebook game
  5. Reading Digg, Reddit, Engadget, or Slashdot
  6. Starting/engaging in a flame war on one of the aforementioned sites
  7. boys/girls at the office
  8. your boss’ hot secretary
  9. your boss/your boss’ hot daughter
  10. game consoles (and games) for which you finally have the money
  11. overtime work
  12. cooking (because you are living on your own now)
  13. laundry (same reason as above)
Pretty much everyone has been slacking off as far as blogging goes, except Jawaad, Skrud, Spiro, and Kevin.
Hall of Shame:
  • Nadia – has not blogged at all for almost 2 whole years now;
  • Terry – lost his domain name to some domain squatter.
Eric is clever, updating his MSN Space with sets after sets of pictures instead of writing.
I know all of you must have made new friends. I have, too, even all the way in Japan. But gee, try to give sign of life from time to time and not forget about TSG and everybody else from Concordia.

Mobile development sucks – Prologue

It’s been six months since my first full-time job has begun. The first month was spent in learning two products, of which I never got involved in any project. My boss finally decided to let me pick up a product development project that has been going on since before I joined the company. I must say that it has been a great learning opportunity so far and I want to share what little wisdom I have found.

Let’s start with some basic information:

  • The product is a software application for smartphones;
  • We are trying to support 5 smartphone platforms. I’m not talking about phone models – that would have been trivial. I’m talking about something more in line with a situation like Mac + Windows + Linux + Amiga.
  • As much as possible, the application is written in native code on all supported platforms (as opposed to Java, which would have been easier to port);

If you think that this is starting to stink, it gets better:

  • There is no vision document; the only available material that’s close to a vision document is a bunch of PowerPoint presentations;
  • There is no software requirements specification, not even an informal one;
  • There is no software architecture document, again, not even an informal one;
  • There is no software design document at all, again;
  • There is an overly simple blackbox testing checklist;

You probably saw this coming by now:

  • The programmers had started coding the damn thing and submitting test builds already.
If you think that this point above puts the final nail in the coffin, I am sorry that I must disappoint you. Read on.
  • There are, not one, but two versions of the product, with two different product activation models. Not only do you activate the two versions in different ways, but the sets of features that are disabled due to lack of activation are also different;
  • The two versions were being developed concurrently.
I suppose my dear reader has a good picture of the stage in mind now. I’ll let you sleep on it for a bit so you can imagine what can go wrong as the story unfolds itself.

Downtime

Kalunite.net and all sub-sites may be down for a few days from January 24th onwards while I migrate everything to a dedicated server in Hong Kong. The new server shall shift the access speed in the favour of visitors from the Asia-Pacific region, possibly at the expense of visitors from North America. Check back in a week or two and hopefully everything will be up and running smoothly again by then.

Short咗

星期日行街行到腳仔軟。先係朝早同屋企人响荃灣兆和街嘅合發茶餐廳喫早餐,跟住我同家姐疙咗去如心廣場睇吓落成咗係乜樣。點知原來咩都冇,得間酒店開咗。隔離荃新天地又係未裝修完。結果唯有走去荃灣廣場行Jusco。不過一入廣場門口,就俾Wanko雞乸咁大隻字嘅「二折」牌嗍咗埋去。就咁樣就開咗市,家姐即刻搵到聖誕禮物俾阿媽同係Montreal嘅死黨同學。跟住再上幾層,喺間DVD/唱碟鋪搵埋老竇嘅禮物。Jusco嗰度反而乜都冇買就走咗。

第一回合完,就哽係食嘢補充能量。上次喺綠楊食越南河,發然仲有好多檔可以試。所以今日又番去綠楊,試另一檔唔知”Euro”乜鬼嘅(家姐溫馨提示:「係EuroGoGo呀。」)。啲意粉都OK,係嗰兩支奇異菓汁同橙汁甜咗少少。

補給完,我地就先分道揚鑣:家姐去銅鑼灣會合企鵝姐,開始佢嘅第二回合;我就去展開我嘅旺角探險。點解去旺角?因為我隻踎屎死咗。讀者一定諗:『踎屎死咗關旺角咩事呀?』。咁我就話你知啦。話說PCCW寄咗封信嚟話多謝我申請寬頻,所以送無綫踎屎,要去旺角挪。不過我點會淨係為咗隻踎屎去旺角丫。前世未迫過咩。其實我主要仲想搵OS X Leopard Family Pack,買番隻模型鉗,同撞吓仲搵唔搵到MG版F91。模形鉗喺烟廠街嗰邊好易搵。有個店員同我講有田宮兩款一貴一平,仲話貴啲嘅唔會咁易剪崩。我話「唔係啊嘛?剪膠咋啵。」所以最後去咗Modern買把最平嘅(50蚊)。隻F91我差啲搵唔到,得番一間有一千零一隻。原來因為唔好賣,好多鋪都冇再入貨。Leopard搵到我傻。出街前我就明知唔熟旺角,去中原地圖睇過信和中心同旺角電腦中心大概係咩位置。點知…去到旺角兜到我傻,嗰度啲遊客資訊牌又退晒色,都睇唔到啲街線,淨係睇到街名。Sino同MKCC又攪笑到冇牌嘅。攪到我問咗幾次啊sir先搵到MKCC。最後我見MKCC兩間有賣Apple嘅都斷Leopard嘅貨,知道冇機,就連Sino都放棄搵就去咗銅鑼灣join家姐同企鵝姐。去到銅鑼灣先發現佢地係搵我去破財擋災。家姐買咗對平平地嘅太陽眼鏡唔駛一個鐘就買咗兩件幾形嘅黑色casual恤衫自己著同兩條呔送比Montreal嘅朋友;咁就900蚊。跟住去睇吓有咩形戒指補番老竇唔見咗嗰隻。不過最後冇買,淨係睇咗款,因為我地唔肯定老竇手指咩嘥屎,刻咗名嘅話又冇得換;要等我地噠老竇其它戒指嚟度吓先。企鵝姐見我地攪咁耐,就游咗去第二檔睇衫,仲執到筍嘢。

之後就去咗SOGO睇玩具。原來我真係大鄉里,依家先知Sylvanian Family系列咁誇張嘅。酒店,水車,菓汁檔,你諗到就有。一傾之下,先知道企鵝姐玩得仲pro。佢嫌Sylvanian Family本身嗰幾款屋太細,所以用Hello Kitty嘅一款大屋代替。真係orz。我仲好奇想睇吓SOGO有咩Gundam貨,一睇之下先知佢食水深;隻F91我喺旺角買先168蚊,喺SOGO要259蚊。

行到攰,又到食嘢時間。去咗檔佈置得幾有越南feel嘅餐廳。叫咗牛肉河,燒豬肉檬,炸春卷,仲有招牌炸雞翼。每人70蚊左右埋單。

跟住家姐順便係藥房買牙刷,然後就分手番屋企。

番到屋企,由於眞係太攰,發生咗啲好hard plastic嘅事…

我好心急咁打開合F91睇說明書,見到呢幅嘢:

當時腦入邊嘅反應係『死嘞。唔記得買較剪添。』

Hacker Reborn