Django CRUD and Angular.js

In the post on Simple Django Class based CRUD views the focus was on setting up simple views with ModelForms to build all the CRUD views for a Model. I have been playing around a bit with Angular.js and wanted to see if I could link the two up.

A word of caution. I really don’t have any idea what I am doing in Angular.js land so take this all with a grain of salt.

Angular seems like a kind of retro-future solution to the javascript application problem. Backbone.js seems to want to get away from the DOM as anything other than presentation. In Backbone the models exist in a pure JS land and the views render them out into the DOM as needed when things change by subscribing to events on the model. Angular in contrast, embraces the DOM more, turning <input> elements into models and providing updates via some dependency injection magic.

Angular also lets you essentially extend html by letting you add your own element and attribute types using directives. But we are getting ahead of ourselves.

Every Angular tutorial shows some example like this:

 <input type="text" ng-model="MyModel"/>
{{ MyModel }}

In this case angular will dynamically update the page by reprocessing the stuff inside {{}} each time the user types something new into the model. Neat.

Another thing Angular can do is provide validation of fields. This works well with html5 <input> types like number and email. If the input element has an ng-model attribute, angular will add some css classes ng-valid or ng-invalid in response to user input.

Which got me thinking. Can I link up Django ModelForms on the server side with all this Angular stuff on the client? The idea is adding some functionality into the Django forms so that they render in way that angular will take advantage of.

If you build the forms by hand in Django you can basically set the widget attributes to whatever you like, but if you want this to happen more automatically when using ModelForms you are probably going to want to check out formfield_callback. In each model form you can add a new formfield_callback function that will determine how django converts the fields in the model to fields in the form. Unfortunately, it doesn’t seem like you can inherit this function from your own ModelForm base class. You have to add it into each class definition. Cry.

class MyForm(forms.ModelForm):
    formfield_callback = angular_formfield_callback
    class Meta:
        model = MyModel

Where angular_formfield_callback is the function we are going to write.

Lets think first about the <input> types. By default django doesn’t seem to do much here. Even if your model has a field of type PositiveIntegerField, the element django gives by default is just type=’text’. So lets change this by defining our own widget for positive integers.
Here we set the input type and also force the min attribute to be zero since this is a PositiveNumber field:

class PositiveIntWidget(TextInput):
    input_type = 'number'
    def __init__(self, *args, **kwargs):
        if not 'attrs' in kwargs:
            kwargs['attrs'] = dict(min=0)
        else:
            kwargs['attrs'].setdefault('min', 0)
        super(PositiveIntWidget, self).__init__(*args, **kwargs)

Then we would define our form field callback something like:
def angular_formfield_callback(f, **kwargs):
    if isinstance(f, PositiveIntegerField):
        field = f.formfield(widget = PositiveIntWidget(),**kwargs)
    else:
        # We don't have a special one just let django pick for us
        field = f.formfield(**kwargs)
    if field: # guard against the case where there is no field...
        if 'ng-model' not in field.widget.attrs:
            # set up a model for this so ng can do stuff with it
            # the dj_ prefix helps avoid name collision
            field.widget.attrs['ng-model'] = 'dj_%s' % f.name

Now when we render the form from django in our django template using {{form}} it will be decorated with the ng-model tag with its value set to the field name of the django model field.

But there is still a problem. Angular really wants the initial values of the fields to be stored in the $scope object. When you use the technique above by itself and you are trying to have it work for updating existing data, the values which django sets as the value attritbute in the <input> element don’t seem to get copied into Angular’s scope.

I tried a few ways around this but ended up writing my own directive for this. The directive just copies the value attribute into the scope using the element’s model name.

var app = angular.module('dj_app', []);
app.directive('djInit', function() {
            return function(scope, element) {
                initial_value = element.val();
                if(element[0].type == 'number')
                {
                    initial_value = parseInt(initial_value);
                }
                scope[element[0].attributes['ng-model'].value] = 
                      initial_value;
            }

That junk with type==’number’ makes it so the scope value is actually a numeric instead of string value.

Now we just need to add the dj-init attribute to all the field objects in the form field callback at the same spot where we set the model. For fun we can also copy over some info from Validators on the original model field and we get:

def angular_formfield_callback(f, **kwargs):
    """
    Applies our own assignment of widgets when using model forms
    """
    if isinstance(f, PositiveIntegerField):
        field = f.formfield(widget = PositiveIntWidget(),**kwargs)
    else:
        # We don't have a special one just let django pick for us
        field = f.formfield(**kwargs)

    if field:  # guard against missing field
        for v in f.validators: # Copy any model validation into the widget
            if isinstance(v, MaxValueValidator):
                field.widget.attrs['max'] = v.limit_value
            elif isinstance(v, MinValueValidator):
                field.widget.attrs['min'] = v.limit_value

        if 'ng-model' not in field.widget.attrs:
            # set up a model for this so ng can do stuff with it
            field.widget.attrs['ng-model'] = 'dj_%s' % f.name
            # set the model name as an attribute of the 
            # init directive in case we want to know it later
            field.widget.attrs['dj_init'] = 
                           field.__class__.__name__.lower()

    return field

Its easy enough to play the same trick with other data types like email or url.

Things to watch out for

Both django and Angular.js use {{ for macro expansion. {% verbatim %} {% endverbatim %} is your friend.

An Angular directive called fooBar is referenced in the markup as foo-bar.

Simple Django Class based CRUD views

Class based CRUD views in django have been around for a while, but I haven’t been using them because every time I tried I ended up with more code that seemed no better than the old way. I just started a small project from scratch and decided to give it a go again.

The basic idea is pretty simple. There are default views like DetailView, ListView, UpdateView etc. you can build your views from. Then in urls.py you just say MyDerivedView.as_view() for the view function.

So before it might have been:

url (
    regex = '^detail/(?P<pk>\d+)/$',
    view = 'scrud.view.detail_view',
    name = 'scrud_detail'
),

Now its

url (
    regex = '^detail/(?P<pk>\d+)/$',
    view = ScrudDetailView.as_view(),
    name = 'scrud_detail'
),

The trick comes in how we define these views. The defaults are fine if you aren’t doing anything fancy. There are lots of nice parameters to the base classes you can modify as shown in the docs.

urlpatterns = patterns('',
    (r'^publishers/$', ListView.as_view(
        model=Publisher,
        context_object_name="publisher_list",
    )),
)

Personally, I don’t like gumming up my urls.py with that stuff so I go ahead and make a views.py and define new classes there.

Either way lets try to see how this works for a simple model.

class Scrud(models.Model):
    foo = models.IntegerField(default=0)
    bar = models.CharField(max_length=20)
    owned_by = models.ForeignKey(User, blank=True)
    last_change = models.DateTimeField(auto_now=True)
 
    def get_absolute_url(self):
        return reverse('scrud_detail', args=[self.pk])
 
    def __unicode__(self):
        return u"Foo:%d Bar:%s  Owner:%s Date:%s" %
        (self.foo, self.bar, self.owned_by, self.last_change)

The get_absoulte_url() method is going to be what some of the views use by default.

To make things easier lets just create a Mixin class we can use for all the views we want.

class ScrudMixin(object):
    model = Scrud
    def get_success_url(self):
        return reverse('scrud_list')
    def get_queryset(self):
        return Scrud.objects.filter(owned_by=self.request.user)

This is nice because it will limit the queryset to objects owned by the user as well as set up the defaults needed by each of the generic CRUD (and List) views.

We can force login by changing ScrudMixin(object) to ScrudMixin(LoginRequiredMixin) and add something like:

from django.contrib.auth.decorators import login_required
from django.utils.decorators import method_decorator
 
class LoginRequiredMixin(object):
    @method_decorator(login_required)
    def dispatch(self, *args, **kwargs):
        return super(LoginRequiredMixin, self).dispatch(*args, **kwargs)

Now we can define views like:

class ScrudListView(ScrudMixin, ListView):
    pass
class ScrudDetailView(ScrudMixin, DetailView):
    pass
class ScrudCreateView(ScrudMixin, CreateView):
    pass
class ScrudDeleteView(ScrudMixin, DeleteView):
    pass
class ScrudUpdateView(ScrudMixin, UpdateView):
    pass

and use them in urls.py like:

urlpatterns = patterns('',
       url (
           regex = '^list/$',
           view =  ScrudListView.as_view(),
           name = 'scrud_list'
       ),
 
       url (
           regex = '^detail/(?P\d+)/$',
           view =  ScrudDetailView.as_view(),
           name = 'scrud_detail'
       ),
       url (
           regex = '^create/$',
           view =  ScrudCreateView.as_view(),
           name = 'scrud_create'
       ),
 
       url (
           regex = '^delete/(?P\d+)/$',
           view =  ScrudDeleteView.as_view(),
           name = 'scrud_delete'
       ),
       url (
           regex = '^update/(?P\d+)/$',
           view =  ScrudUpdateView.as_view(),
           name = 'scrud_update'
       ),
)

Oh yeah. We are going to need some templates. You can override the names if you like but you’ll need something like:
scrud_list.html

{% for object in object_list %}
  <a href="{{ object.get_absolute_url }}">{{ object }}</a>
{% endfor %}

scrud_detail.html

{{ object }}

scrud_form.html

<form action="." method="POST">{{ form }}{% csrf_token %}
   <input type="submit" value="Go" />
</form>

scrud_confirm_delete.html

<h1>Delete This?</h1>
{{ object }}
<form action="." method="post">{% csrf_token %}
    <input type="submit" value="Delete" />
</form>

Now there should be a fully working crud system for your object. Read some hackernews to celebrate.

When you come back you will realize that you probably want something a little different. Like for example. Maybe you want the owned_by field to be set automatically based on the user. You don’t want it in the default form. No problem you can override the form with your own custom model form.

class ScrudForm(forms.ModelForm):
    class Meta:
        model = Scrud
        exclude = ('owned_by')

The top of your ScrudMixin is now looking like:

class ScrudMixin(LoginRequiredMixin):
    model = Scrud
    form_class = ScrudForm

Which takes care of not letting the user change the owner but doesn’t actually set it. For that you need to add one last thing your ScrudMixin class, a form_valid override. This changes what happens when the form is valid so that the field can be set correctly.

from django.views.generic.edit import ModelFormMixin
def form_valid(self, form):
    # save but don't commit the model form
    self.object = form.save(commit=False)
    # set the owner to be the current user
    self.object.owned_by = self.request.user
    #
    # Here you can make any other adjustments to the model
    #
    self.object.save()
    # ok now call the base class and we are done.
    return super(ModelFormMixin, self).form_valid(form)

I hope this at least helps some people get started using class based CRUD views. Better yet, I hope someone will tell me an even better way.

Flattening the Backbone.js Learning Curve

I just spent some time getting into Backbone.js finally.  The learning curve was a little steeper than I expected, mostly because I think I expected Backbone.js to be something that it is not.

From the docs:

Backbone.js gives structure to web applications by providing models with key-value binding and custom events, collections with a rich API of enumerable functions,views with declarative event handling, and connects it all to your existing API over a RESTful JSON interface.

Well not exactly. Backbone.js doesn’t give structure. It gives you the tools to create structure.  A minor nitpick maybe, but it makes a difference when you are trying to understand how something works and you can’t find the parts you expect to be there.

The docs are well written, the examples numerous, and there are also several blog posts which try to help you get started. Oh, and there is the totally awesome annotated source.

Still there were a few things I wish I had understood before I dove into a tutorial.  This post is my attempt to try to help boost someone up the learning curve a bit. The idea is not to get into many details or app design philosophy.  Not being an MVC ideologue myself, I care little about how things ought to be.  I am just trying to get to the essence of what the parts actually are. I’m also skipping some things on purpose, as I said the docs are well written.

Models/Collections

Key Idea 1: Wrapper for a hash of properties and that can trigger events when the properties change

Key Idea 2: Includes code to load and store from a Rest API

Models and Collections are the data management objects in Backbone.js.  The Model is really a wrapper for state for an object. Each model object has a field “.attributes” which is a map of the data managed by this model.  Since you change the data using .set methods of the model, this allows the model to trigger events when the data changes:

A Collection is a collection/list of models. Duh.

Both Collections and Models are able to fetch themselves from a server if you have a nice API.  Just set .url for the object foo and call foo.fetch().  An ajax call will be made to the server and the data will be loaded or added (you can chose with an option). You can also save a model to the server. And you can do lots more, including overriding virtually all of this.

Views

Key Idea: Wrapper for a DOM object which renders it based on events and other state

Views are what tie into the DOM. Each view attaches to a DOM object. You can set this to one in your document or the View will create a new one ( a <div> by default) when you create the View object.  It’s your job to make sure the element is in the page in the right place in the hierarchy. The View always keeps a reference to the DOM element in .el or .$el if you prefer the JQuery wrapped version.

A view generally has an associated model and listens for changes in that model,  re-rendering the DOM objects when a change occurs. The view subscribes to events on the model and calls the views render method to update the DOM objects based on the new data. Generally this is done via a client side template mapping the Model’s attributes into HTML.

Routes

Key Idea: Custom events or state which work with browser history

Routes allow you to define new URLs within a single page application that can trigger events in your app. You create a router object which includes a bunch of defined url routes which are pretty similar to the urls.py in django actually. The routes/urls have parameters which are passed to the callback function when that state is reached.

Calling router.navigate() will set a new state which allow the user to move around using forward and back in the browser. This is done via #tags which you give as names to the routes.

Don’t forget to call Backbone.history.start() before using Routes.

 


 


Advice for Tech Handicapped Computer Users

After several years of helping non tech savvy neighbors with various problems, I wanted to make this list as a standard advice that anyone should follow. This is not intended to be a list of best practices, rather a list of simple important practices. If you are smart enough to figure out a better solution go for it. I don’t personally follow much of  this advice, I just wish my neighbors did.

DO: Use a router when you connect to the internet
DSL or Cable, It doesn’t matter. What does matter is that you don’t connect a computer directly to your DSL or cable modem. If you don’t know what this means then you are at risk for having violated this one. Using a router is the first line of defense against involuntary participation in Anonymous.  Some providers include a router with the modem but in most cases if you only have 1 little box next to your computer and you don’t have wifi, then you need another one. It doesn’t really matter which one you get so just get one from linksys.

Seriously, if you don’t understand the above have someone help you.

DO: Use Gmail
Don’t use email from your internet company. This means if your email ends with @comcast, @centurylink, @bellsouth etc. then its time to switch to using webmail. There is no best solution here, but they are all better than using @comcast so just sign up for www.gmail.com

DO: Use good passwords
You should probably use 1Password or a similar product or service but you probably won’t so just remember the most important password is the one for your email. DONT USE THIS FOR ANYTHING BUT YOUR EMAIL. Once someone can read your email they can pretty much get all your other passwords if they are clever, so every place you use your email password is a potential place where someone can get all your stuff. Its also a good idea to have another separate password you use only for banking.

Tip:
Have a personal affirmation statement and use the first character of each word as you password. If you are Stuart Smiley “I’m Good Enough, I’m Smart Enough, and Doggone It, People Like Me!” your password is Igeise&dgiplm!. That way you can start the day with a little positive thinking.

DO: Turn on Auto Updates and Accept Updates
Always have your operating system set to auto update or notify you and always install all the updates. Same goes for software you installed on your machine.

DO: Buy a Mac
If you have money to spend (like $1500) get the 13in Macbook Pro. Select Apple Care then then Click Add to cart. There isn’t any need to anguish over this.

DO: Okay if you insist or can’t afford a Mac By a Windows PC
If you are trying to save money buy a ASUS laptop for around $400-600$. If you want to spend $1000 on a pc, buy a Mac instead.

DONT: Obsess about Antivirus Software
If you want to purchase something go ahead I won’t stop you. Not doing stupid things, see Below is more important.

DO: Use Google Chrome as your browser
If you don’t understand what that means. Than its ok, keep using the Blue E for be “the internet”.

DONT: Use your PC to look at Porn, or Pirate things
Look I don’t care what you do with your time, and I don’t care about the moral implications of either of those things but this is the kind of stuff that will end up getting your computer infected with viruses etc, and I don’t want to have to come figure out what happened and stumble on your search history.

DONT: Accept any “toolbars”, “spyware cleaners”, “accelerators”, etc.

DONT: Click on any links in an email.

DO: Backup what you care about
If your email is all you care about then gmail will do it for you. If you have other stuff sign up for www.backblaze.com and backup over the internet while you sleep (if your pc is on). Local backups are likely to fall victim to crime, fire, or laziness

DONT: Forward chain emails
No matter how clever, funny, important you think they are.

DONT: Be paranoid about using your computer one the wifi at the coffee shop
Despite what you have read (possibly in a chain email) its not that unsafe. If you are really concerned or have special needs, get a VPN or use your own wireless connection.

DO: Get a no spill coffee (or other beverage) cup
The Contigo is a nice one. Keyboard spills are a pain and they happen to everyone.

DO: Reboot the pc every few days
If you leave your pc on its good to restart it every few days to clean out any crap and make sure you haven’t installed or done something to mess it up. When you get that funky message on startup you might remember what you did if it has only been a day or two since the last reboot.

 

Learning Spanish

For the past few years I have been trying to learn Spanish. My main motivation is that I enjoy traveling in Latin American and speaking the language opens up many opportunities. I have tried several different methods, cds, books, rosetta, immersion school in Costa Rica, personal tutors etc. They all have their benefits but in many ways my current method is my favorite when all the time and cost is factored in.

What I am doing now is paying for a personal tutor who lives in Guatemala to talk with me over Skype twice a week. The cost is fair and I get the added benefit of learning about the culture there which is exactly the vocabulary and context I am interested in. Things like local politics etc. My favorite thing I learned had to do with the most recent presidential elections in Guatemala. The president is elected for 6 years and cannot be re-elected, and neither can they be followed by a close family member. Last year, the president and first lady got divorced so that she could run for president but in the final weeks the Supreme Court ruled her ineligible, although I think the divorce was still valid.

I also learned about the difference between Elote and Maiz.

There are several services you can go through to find a tutor, I ended up just working directly with mine. If you are interested in trying it out, let me know in the comments and I will hook you up.

Coder’s Block

I have tried before to write about what makes programming hard. I’m not really happy with that post or the followup. I still believe that Complexity in all forms is the biggest problem thought its not the only one. Sometimes designing an algorithm can be really challenging even for something which feels pretty simple. Same goes for things like performance optimization. It can be pretty difficult to properly analyze a system to determine what parts are slowing it down even if the system (or at least the part of if you have any control over) is pretty small.

But even beyond all that, much of the time it seems that the really hard thing about building Software isn’t writing the code, it’s deciding what code to write.  Its one reason we see so many projects which amount to “X now written in Y”. The objective and scope are clear and easily understood. Lets fire up vim and have at it!

Currently I’m building a large software product, hoping to turn it into money in the future. I don’t have specific tasks or deadlines besides the ones I set for myself and I sometimes fall victim to Coder’s block. Writer’s block isn’t an inability to type – it is a failure to be able to decide what to type next. Coder’s Block is similar.  When I am stuck (aka. hitting Command-R on news.ycombinator.com) it is rarely because I am faced with something I don’t know how to do. It is way more likely that I just can’t decide which of three different methods I could use to do it.

One way I deal with this is I try to keep two different Todo lists. One is the hard decisions I still have to make about how the software should work (Ex. what is the best representation for a huge list of genomic variants). The other is just a list of things I can knock out whenever I am stuck on the first list (Ex. Fix the report CSS styles so they don’t look like crap).  When I was a manager, I tried to encourage this habit among the coders who worked for me, but It mostly came out like I was trying to make them double up on work. Um sorry.

Another technique is to just code your way out of it. This works pretty well for small to medium size blockages. The other day I was stuck trying to decide what kind of reporting system I wanted to build. I wasn’t getting anywhere in my head so I just starting using python to think with.  I think I have written it about 3 different ways in the past week but now I am pretty happy with the result.  It certainly is easier to play around with options and keep transforming your code using new technologies than It was trying to do this with a C++ app using Boost and other crap.

Sometimes it is hard to get up the desire to use either approach. That’s when it’s time to step away from the keyboard and go for a bike ride.  Sure you could sit there and refresh hacker news, but it will be there when you get back.

Sshuttle

Thanks to @elasticdog I now have sshuttle all set up on my laptop. He has an excellent guide here. Personally I just went with the brew install option and its working just fine. Seems much easier to maintain than setting up a real VPN on the server side.

There is really no server side config at all. Its all just magic.

Accessing the Child Class In Django Multi-Table Inheritance

Django has 3 different ways to deal with model inheritance:

The differences are described pretty well in the linked documentation so I am not going to recapitulate them here. I just want to talk about a specific pattern I have been playing with for Multi-table inheritance.

In Multi-table inheritance the base class has its own separate table. This makes it possible to query all the objects from all the subclasses at once.

So if we have:

class Animal(models.Model):
    def speak(self):
        print "Generic Animal Sound"
 
class Dog(Animal):
    def speak(self):
        print "Woof!"
 
class Cat(Animal):
    def speak(self):
        print "Meow!"

The Animals.objects.all() will find all the Cats and Dogs in the DB. The trouble is they will all be of type Animal and so if you run:

Dog.objects.create()
Dog.objects.create()
Cat.objects.create()
 
for animal in Animal.objects.all():
    animal.speak()

you get:

Generic Animal Sound
Generic Animal Sound
Generic Animal Sound

There is a way to access the child class specific information. animal.dog.speak() will produce Woof! if animal is actually a Dog. It will however toss an exception if it is a Cat. Worse is that testing this out generates some useless Db queries which becomes even more impractical as the number of subclasses increases. One approach is to store the name of the child class with the base class so you only have the one extra query needed to grab the data for the child object. If we create an Abstract base class like:

class KnowsChild(models.Model):
    # Make a place to store the class name of the child
    _my_subclass = models.CharField(max_length=200) 
 
    class Meta:
        abstract = True
 
    def as_child(self):
        return getattr(self, self._my_subclass)
 
    def save(self, *args, **kwargs):
        # save what kind we are.
        self._my_subclass = self.__class__.__name__.lower() 
        super(KnowsChild, self).save(*args, **kwargs)

And derive Animal from it.

class Animal(KnowsChild):
    def speak(self):
        print "Generic Animal Sound"

Then we can run:
Dog.objects.create()
Dog.objects.create()
Cat.objects.create()
for animal in Animal.objects.all():
    animal.as_child().speak()

and get:
Woof!
Woof!
Meow!

I am still honestly not sure if I like this or not but its working out well so far for a reporting system I am making where ReportItems can be one of several different types but sometimes I just want to work with the elements in the base class. A lot depends on how often you need to know the specifics of the child class.

Oversampling Canvas for better quality

The html5 canvas is a very fast and simple immediate mode graphics system, but unfortunately the image quality (especially text rendering) is often lacking. A quick and easy way to address this is to oversample the canvas.  This works by making the internal size of the canvas a multiple of the size you want it to appear on the screen. Doubling the size is generally good enough for most purposes.

The following image (made with chrome) shows the difference. The text on top was rendered with a standard canvas and the bottom text was rendered with an oversampled canvas.

 

The code for this is pretty straightforward.

In the document you can set up a canvas with double the resolution “canvas_big”  you need and then use a css style to resize it to the same visual size as “canvas_small”.

<!DOCTYPE html>
<html>
<head>
  <!-- A style to scale down the oversampled canvas -->
  <style> #canvas_big { width: 300px; } </style>
</head>
<body>
  <!-- This is a standard html5 canvas -->
  <canvas id="canvas_small" width=300 height=50></canvas>
  <br />
  <!-- This canvas is oversampled. -->
  <!-- Its internal size is 600 but it is set to display at 300 -->
  <canvas id="canvas_big" width=600 height=100></canvas>
</body>

In the javascript you can use scale on the oversampled canvas so that the same coordinates you were using before will still work on the larger canvas.

<script type="text/javascript" language="javascript">
  // Get the context for the standard canvas
  context = document.getElementById('canvas_small').getContext('2d')
 
  // Draw some text
  context.font = "22pt Arial"
  context.fillText("This text looks bad", 20, 25)
 
  // Grab the context for the oversampled canvas
  context = document.getElementById('canvas_big').getContext('2d')
  // Scale the transform so the coordinates are the same as before
  context.scale(2,2)
 
  // USe the same drawing commands as before
  context.font = "22pt Arial"
  context.fillText("This text looks better", 20, 25)
</script>
</html>