The Grok Point

Whenever I have a significant programming task to complete I usually begin the process not by thinking about all the data structures I want, or how the output will look, or what will the biggest concerns for efficiency etc.

What I am usually doing at the start is trying move as fast as I can towards the “Grok Point”. This is the point in the process where you have gained a full understanding of what you are working with. Specifications,  Example Code, Documentation, Tests, etc all of these are tools on the way to this moment as are throwaway script, staring at code,  bike rides, and caffeine.

Before the Grok Point – the code is still doing things you don’t quite follow,  you can’t find the right channel of data in the new undocumented file format,  or you are unable to get even one triangle to draw on the display. After the Grok Point – that data schema you are forced to use may still be lame but you are now its master.

Reaching this point doesn’t mean you are close to done, mostly it just means you can really get started. Generally it means you now finally have a good boundary of the problem. There may still be a huge wildfire out there but you have fire breaks all around it and you know its not going to get out. The Grok Point should be where your ability to plan or schedule begins to have some meaning. Anything can happen, the fire does sometimes jump over the river, but we have at least accounted for all the known unknowns and should be able to really think about how you are going to finish.

I feel that we often don’t give the moment the full credit it is due. We move from start to end tweaking, pushing, testing, etc until we are done. When you have spent some time working on something and reach the Grok Point I think its a good time to stop and reflect. Make notes of what you need to so you can be sure you won’t forget but then maybe its time to step away from the keyboard for a bit. Enjoy it, it feels good.

This is also a great time to think about your goals again for this task.  You are smarter now than when you began. You have just inflected from “WTF is this doing?” to “A-ha!”. At this point anything is possible. Once you start coding again you will be making choices, some of which may be hard to undo.

These moments of inspiration are sacred, use them wisely.

Doing what you aren’t supposed to do with Django tests

So I wanted to run my django test scripts against my development database. The django test system really wants to create a database from scratch but I wanted to see whether I could run against my local scratch copy since its already got a lot of data in it and I am too lazy to make fixtures etc.

Some googling produced alot of very helpful “You are doing it wrong!” comments which is fine as far as that goes but I am not good at taking advice.

Digging a little deeper I found you can override the DjangoTestSuiteRunner. Now we are getting somewhere.

put this in

TEST_RUNNER = 'myapp.test_extras.NoTestDbDatabaseTestRunner'

And then define the NoTestDbDatabaseTestRunner:

from django.test.simple import DjangoTestSuiteRunner
class NoTestDbDatabaseTestRunner(DjangoTestSuiteRunner):
    def setup_databases(self, **kwargs):
    def teardown_databases(self, old_config, **kwargs):

I’m feeling pretty good about this. I try a simple “test” which just counts the number of items in a particular table.

Only what I get is a completely wiped out local db. Nice.

The trick is that my TestCase was using

from django.test import TestCase

which has its own ideas about what to do with the database and it was wiping it out. Switching to:



from django.test import SimpleTestCase

if you are using django trunk. Looks like it fixes the problem, but it doesn’t make me feel any better.

Maybe I should just listen more.

No Virgina, there is no such thing as Write Once Run Anywhere

I think the first time I heard “Write Once Run Anywhere” was the uscd-p system pascal.

From Wikipedia:

UCSD p-System achieved machine independence by defining a virtual machine,

UCSD p-System began around 1974 as the idea of UCSD’s Kenneth Bowles[1], who believed that the number of new computing platforms coming out at the time would make it difficult for new programming languages to gain acceptance.

Ok I was 10 in 1974 and probably wasn’t paying attention right at the start but its safe to say the idea has been around a long time. But read the article and weep about the fact that 37 years later we are still chasing (or hiding from) the same paper tiger.

Being cross platform is easy until you try to actually touch something about the platform. JNI anyone?

Its not just about companies trying to embrace and extend. Its just a hard problem and its everywhere.
Here’s a fun little PySide snippet adapted from some code which bit me yesterday.

from PySide.QtCore import QSettings
settings = QSettings('foo','bar')
if a:
    print "Windows"
    print "Mac"

So this will print Windows on Win7 and Mac on Snow Leopard. The reason is apparent if you add this line:
print a, type(a)

On Win7 you get:
false <type 'unicode'>
On the Mac:
False <type 'bool'>

Ok so maybe you think the Mac is correct and the Win7 version of PySide has a bug. Doesn’t matter. The worst part is you can “fix” this by changing:
settings.setValue('key',int(False))  # or a var which is <type 'bool'>; with a value of False

And it all works. The windows registry (in PySide’s world at least) seems to understand Ints but converts bools to strings. Yay!

Of course we can all think of other examples. The point is just to remember that there is no cross platform development environment and there is unlikely ever to be one.

For a while people wanted to think that the Web was the answer. You know make some cool web app and if you obey the standards it will work for everyone without any special hacks. Ha. Good one.

With current trends in Virtual Machines, I think Write Once Run Anywhere might happen because we will just ship the App as a complete Virtual Machine and the OS will just be something like VMWare with a switcher. Then the switcher will get more and more complicated and there will be competing companies making them. Each will add their own features to differentiate themselves until…

Making a nice little Netbook

Step 1: Get an Acer Aspire One
I got mine at Costco, last time I was there they were on sale/instant rebate. Model D255E

Step 2: Add Ubuntu Natty Narwal
I went with splitting the disk in 1/2 because its pretty big and If I want to punish myself I can still run windows 7.

Step 3: Upgrade the Ram
I had a few things which didn’t work for me with 1GB (under windows) so I bumped it to 2G for < $25.00. This part worked for me: Kingston ValueRAM 2GB 1333MHz PC3-1066 DDR3 SO-DIMM Notebook Memory (KVR1333D3S9/2GETR) I had tried an different one from crucial that did not work. The upgrade is not too hard. The trickiest part is prying out the keyboard. You can push the little tabs back with a tiny flat screwdriver and then you pry out the keyboard. I recommend a credit card to help. Then you remove 4 screws marked "DOOR" and you can then push out the door at the bottom to access the Ram. There are vids/howtos on Youtube. Step 4: Add a wireless mouse and keyboard Step 5: Connect it to a big monitor Less cool factor than an Ipad but cheaper and no real limits on what kind of software it can run.

The other reason why Fixed Rate Contracts suck

You have a prospective client and you are discussing the project. Often the client would like you to quote a fixed fee for the work, so maybe you think thats ok and come up with some number by using a fancy estimation tool or tossing chicken bones. It really doesn’t much matter, something is likely to go wrong anyways.

The most obvious problem is the one everyone talks about. The client wants more work for the same price or changes his mind and wants you to just do the changes for free. Sure you could go down the route of detailed specs and change requests etc. but that’s means you are spending your time pushing paper instead of making code work, which is hopefully why you are in this game in the first place. So often we absorb a little bit (which hopefully we accounted for in the quote) and it only becomes a real problem when you have a particularly difficult client. There’s alot more to say about this and I don’t pretend to be an expert on the situation or how to resolve it.

I’d like to talk about the other side of this problem.

You agree to do a job for a fixed fee. While doing the work, you realize that there is much more you could do to make the product better than was originally envisioned by you or the client. In a good situation you can talk it over with the client and they will to pay more for this additional value. However, often the client’s budget is limited and they are unable to pay for this work that you know “ought to be” done. I run into this relatively often my desire to build something great runs up against my desire to not work for free. Usually the two desires compromise but that never leads to a feeling of satisfaction.

The simple answer is to just avoid doing fixed fee work. Is there a better answer?

Django Logging Setup

Post 1.3, Django has some nice integration with the python logging module. It can dictConf to specify the formatters, handlers, etc. This is all pretty well documented here.

What wasn’t immediately obvious (to me anyway) was that you can use any of the handlers in django logging and pass the __init__ params to them in this config. So for example if you want to have a log which rotates everyday and keep a week of these files, then you can simply define:

'handlers': {
        # Other handlers here
            'level' : 'DEBUG',
            'formatter' : 'verbose', # from the django doc example
            'class' : 'logging.handlers.TimedRotatingFileHandler',
            'filename' :   MY_LOG_FILENAME, # full path works
            'when' : 'midnight',
            'interval' : 1,
            'backupCount' : 7,

and then use this handler for a logger:

'loggers': {
         # other loggers
         'my_logger': {
            'handlers': ['rotating_file'],
            'level': 'DEBUG',

You can use this logger in your app by grabbing the logger:

logger = logging.getLogger('my_logger')

And log away!

logger.warn("Here be Dragons!")

PySide + PyInstaller = Joy

I have a small project to make a pretty simple windows GUI executable. In the past I have always used wxpython + py2exe for this kind of thing but decided to try something new.

After some poking around I settled on PySide which is the bindings for QT from Nokia. Its nice. Seems a little cleaner but actually pretty similar to wxPython in many ways.

At first I tried to use py2exe to make a single exe binary but I could never get this to work properly. The exe would always die somewhere during load and after spending too much time trying to sort that out I decided to use cx_freeze to much the same result. I also tried going back to Python 2.5 since that seems to have fewer MS DLL issues and PySide is available for that as well. Still no love.

Finally I tried Py-installer and it all pretty much worked right out of the box. Just go through the install step by step and you should be fine. Py-installer integrates with Upx and my final exe came in at around 6MB.

If you are stuck with having to deploy an app on windows, you don’t need anything more that python, pyside and pyinstaller. As a bonus the app will also run on your real computer.

Removing old iPhone/iPad Backups FTW

A few days ago I tweeted this:

getting tired of having more free space on my phone than on my laptop

Which got me thinking. How much space is being used up on my laptop just for backing up my phone? Some investigation yielding the following obvious result. Alot.

But what’s worse is that there are backups for old iOS devices I don’t even use anymore, like my old 3G iPhone and the iPhone4 that Apple so nicely replaced when it had headphone issues. Anyway after culling all that crap out, I now have plenty of Gigs free on my old Macbook Pro, possibly putting the day when I need to upgrade even further into the future.

Email only Signup

One of the things that bugs me about the web is having to create logins everywhere. There are lots of attempts to deal with this problem (OpenID, Facebook Connect, 1Password) etc. but they all seem to have problem or require me to do things like:

  • Use only sites that implement them
  • Copy and paste stuff on my phone
  • Spend Money
  • Trust Mark Zuckerberg

Many of these websites are just some new thing I want to try out.  I’m not talking about sites which require high security such as bank accounts or email (if you use the same password for you email account and other random web accounts, stop reading and change it right now!).

I am talking about this weeks random new social/mashup/mobile/fu thing your sister just txt’ed you and said “Try this out, Its Awesome!”. These sites are all trying to get as many people to sign up as possible so the make the barrier to entry a low as possible and signup typically requires only a email and a password. I’m starting to think that is one thing to many. I don’t think most of these sites need to ask me for a password, they just need my email address.

How do I know this?
Since almost of these sites let me reset my password via email then my email address (and the ability to read it) is evidentially good enough authentication.

Ok yeah, I know what you are thinking, great more spam in my email, more steps to login etc. But the thing is that most of these sites also leave you logged in for a long time (forever?) and often I never go back to them after a week anyway.

In this scheme login (and in fact signup) looks like this:

  • Enter your email address.
  • Click login.
  • Receive an email with a magic one time use key.
  • Click the key and you are logged in.

Note: that anyone who tries to log in as you will cause you to get an email notifying you of the attempt…

You can also add a captcha etc in the email for the first time this login is used.

Sites can always still allow the user to update the account with passwords/2-factor/security questions etc later, but why bother unless it becomes necessary?

Sure its not the best solution for every case but nothing is.

I have played around with this idea on a demo site for myself and it seems to be workable. What am I missing?

A simple implementation using django is on github here

Buying Time

For the last several years I have been working essentially as a contractor/consultant rather than a full time employee. These engagements are of varying lengths and generally center around specific projects. This arrangement suits me well as I prefer to think of my job as working on a Project rather than for a Company and I like the flexibility for my own personal time that contracting allows me.

In these situations the inevitable question which arises is something like “What is your rate?” by which its generally meant “How much do you charge per hour?”. The problem is that I have no interest in charging by the hour for several reasons. First and foremost is that I find the idea of tracking my time at this level to be too much trouble. I don’t like bookkeeping in the first place and even if I did its too hard to decide what counts and what does not. I do some of my best work while riding my bike. This is actually one of the reasons I love bike commuting. If I drive to work, I will listen to NPR and arrive at the office frustrated about some stupid political issue or maybe just the traffic. When I ride in, I spend a lot of that time thinking about what I am going to do when I get to work, how to solve a problem etc., and can’t wait to get to the keyboard. Do I get to charge for that time? How about that hour I spent trying to sleep while running over ideas for an algorithm in my head?

Essentially the smallest unit of time I am willing to sell is one day. For things which are less than one day of work for me I have two special rates, free and no thanks.  I will spend a few hours helping you for free if I want to for whatever reason but I am not interested in a couple hours of paid work unless the rate was equal to my daily rate in the first place. During that day (or those weeks depending) I will work on the client’s project as If I am a full time employee. That means giving at least full days work M-F and sometime a little extra on nights and weekends that I don’t bill extra for when I feel like its warranted. The details of how my time is spent is my problem, and the client needs to judge whether I am worth it by the work produced.

As a contractor I expect no paid time off. If I take a week off to go to Central America, I don’t bill for that. If I am sick and can’t do any work, I don’t bill for that either. Its my job to make sure that my work is all done by an appropriate deadlines but If I can do that and take friday off, I will. If I take the day off, the client does not pay and I try not to think about them. Its easier to stop thinking about them if there is a beach.

Does charging by the hour really work for anyone in software development?