Schinckel.net

From xkcd - A webcomic of romance, sarcasm, math, and language - By Randall Munroe

Ah, this combines my two great loves… python and Calvin and Hobbes.

For a Uni project, I have to write the same algorithm different ways, and in different programming paradigms. I also need to collect data on the execution of said programs. Since some algorithms may run in large time on large data sets, I need to stop execution after 30 minutes of run time. And, since there needs to be between 50 and 64 runs of each data size, I wanted to automate the process.

Using the $ time `other-command` is the most obvious way to time execution of any command line application on any decent operating system. However, the time that comes with OS X is somewhat limited - from the man page:

NAME

time – time command execution

SYNOPSIS

time [-lp] utility

It only has two options, and they aren’t that useful. However, it is possible to use this command to time execution.

Because it outputs it’s timings to stderr, then you need to do some tricky python to capture it and save it to a file. I used sys.popen3(), and then read from the stderr stream. Which worked okay, but there isn’t really a way to stop execution after a certain time frame. You can try to use TimeoutFunctionException - which I can’t remember where I got it from, but it’s cool. It doesn’t work in this case, since the sys.popenX calls run in a sub-process, and thus continue to run after the exception is raised. Fail.

About this time in my thought process, I came across a better time. GNU time allows you to select the output format (excellent, no need to parse the output quite so much!), and to output to a file instead of stderr/stdout (even better). It also allows to append to a file instead of overwriting, and some other cool stuff.

But it doesn’t solve the issue of commands running too long. This was a killer, since the process continues to run, but worse than this, new processes are started. So the machine clogs to a virtual halt.

Then I came across ulimit. This handy tool can limit the resources a user or process is able to use.

$ ulimit -t X; time -f %U -a -o {DATAFILE} {COMMAND} {ARGS}

This command will limit a command to X seconds (or slightly less, since the time command itself uses some time. It will then execute COMMAND, with arguments ARGS, and time the execution, appending the run time only, on a seperate line, to the file DATAFILE.

Note that this is no excuse for not using code profiling. I have already run profile.run() on my code to work out where the slowdowns are in the python versions, and then optimised them. This is more like the last phase, actual comparisons.

Java almost handles arrays well.

Almost.

Maybe I’m spoilt by python, but having datatypes that are effectively a hybrid between lists and arrays is excellent. You get both of the advantages - being able to iterate easily, and access by index (attributes of arrays), and having dynamic sizes and non-sparse lists (the only decent attributes of lists).

In fact, the text I am reading now has a three-and-a-half page code fragment called “Partially-filled lists”, which is about 200 lines of code, which implements what I describe. Except the upper limit of the size, which must be determined at compile-time. And it requires a new class if you want it to be for anything other than doubles, or whatever you have written it for.

The other thing which was bugging me was the looping of arrays. In python you can do cool stuff easily iterate over elements of an array. Recent versions of Java can also do this.

Python:

1
2

    for element in theList:  
        print element

Java:

1
2

    for (element: theList)  
        System.out.println(element);

It gets pretty close. I think I still like the simplicity of the python notation - brackets only where they are really required to indicate function/method calls, and for expression ordering. Having a required bracket around if test-expressions and the like just makes me think if, switch and so on are functions. Which they can’t possibly be, since Java doesn’t have functions, only objects and methods.

And don’t get me started on braces…

For my day job, I am developing a SOAP server in python. I have been using ZSI as a framework, and it is very good. It will, with mod_python, allow you to build a complete application in python (and even without mod_python you can have it as a standalone process). One of the touted features is easy return of lists and dictionaries, without having to declare ComplexTypes classes.

However, it doesn’t quite work. And the not-working-bit is really odd.

If you return a dict, such as the following:

1

  return {"uid":23,"gid":993,"cid":333}

Then ZSI creates a SOAP response like:

1
2
3

    <uid id="1234" xsi:type="xsd:int">23</uid>  
    <gid id="5678" xsi:type="xsd:int">993</gid>  
    <cid id="0987" xsi:type="xsd:int">333</cid>

But, if you return a dictionary with values that happen to be the same, as I did with my boilerplate code:

1
2
3

    return {"uid":"xsd__string",
            "gid":"xsd__string",
            "cid":"xsd__string"}

Then it fails. The second and any other instance of any dict key where the value has already been used by another key is empty, and the wrong type:

1
2
3

    <uid id="1234" xsi:type="xsd:string">xsd__string</uid>  
    <gid href="#5678"></gid>  
    <cid href="#0987"></cid>

This can be overcome with liberal use of classes (or subclasses, since most of the time I am returning dicts or lists). It is a bit of a pain in the arse, though. I’ve filed a bug report. And stopped using ZSI. If I found this bug this fast, then I don’t want to know how many more there are. It’s just easier to convert the XML to python objects and back again, and package it up to look like a SOAP request. Which is kind of what SOAP does anyway.

I never thought I’d manage to find a job coding in python for a living. But that is what my full-time job is, right now, anyway.

I can’t really talk about the work I’m doing, since it is a commercial enterprise, but I can talk about what sort of things I am coding. I have spent the last couple of weeks rebuilding a server in python that uses SOAP to communicate with the outside world (well, a client application, anyway) over SSL connections. Python is used so it is easily extensible, without having to recompile. Eventually it will be dynamic, where new modules can be added to a database, and depending on the userid of a request, a different function will be called. It’s really quite exciting.

I’ve come across a couple of new software programs - one of which is NX (nomachine) Client, which is a remote tunnel for X windowing. I can remote in via this to work from home, as well as ssh or sftp. Which is fairly cool. Speedier than VNC, since I think the local X-Windowing system is responsible for some of the drawing tasks. Feels about the same over ADSL as ARD does over WLAN.

In my “free” time I’ve been doing a couple of other things, both programming tasks. The first is a web application for an art gallery to create HTML and PDF invitations and newsletters. Originally I planned to use a web app so that it could eventually be rolled out as a blog-alike - in fact originally it was just going to be a WordPress installation with some minor modifications. It turned out to be easier to rewrite it from scratch. I have learned from this process that PHP is crap: it’s never clear about the way to do stuff, and many functions have weird names. count_chars, for instance, doesn’t really count the characters, unless you decide that things like #@! aren’t characters. In which case you want strlen. Which had me tricked for some time, since I stopped looking once I had found count_chars. Python and len(anything) is much better.

Speaking of python (again), I’ve also been working on a Regular Expression helper - similar to the one that comes with Komodo IDE. I started (and pretty much finished, in a matter of hours, to the extent it solved my first use problem) this after having to load up Komodo just to get a visual representation of which bits of a text block were being matched by a regex. Still some kinks to work out - I need to figure out how to put stuff into an outline view, so I can see more than just matches, but match groups. Then it will be all good.

In the process of my work job, I downloaded SOAP Client, a freeware tool for testing SOAP packets. It was all good until I tried sending HTTPS requests, which it fails unfathomably on (cannot connect to endpoint…). I emailed the author, and he then promptly released the source code. I’ve snaffled that from Google Code, and I’ll try to hack through it a bit to implement SSL connections. Not sure how to go about it at this stage - dunno if it is with WebKit or something else I need to do. I also plan to add in the ability to edit the SOAP request manually before it is sent off to the server.

I start Uni in a couple of weeks - I’m doing an introductory Java course in intensive mode, which I expect to be fairly easy. I’m really only doing it so I can do the meatier sounding subjects, like Programming Language Concepts and Systems Programming. I really think I’m going to enjoy this course. I will be interested when I come to the Internet Computing subject, since I’ve been doing a fair bit of that in various forms over the past few years. Be interesting to see what the academics think it means.

Well, that’s been my life over the time period since coming back from the beach for a 10 day holiday over New Year. Apart from squeezing in a few games of Touch Football here and there, I’ve pretty much been chained to my laptop.

And loving it.

Using python, it’s easy to get data from IMDb into your iTunes database.

You’ll need a few tools to get this working. appscript, py-IMDb and CocoaDialog. Install each of these, and for the latter, note where you store it. I’m not going to go through the code, as it is fairly well commented. If you can’t get it to work, leave a comment and I’ll see if I can help.

IMDB2iTunes.py

Note that you will need to save the file above, and rename it so it has a .py extension.

I’ve consolidated all of my media onto my new Mac Mini, but didn’t really think some things through when I first did it. For instance, I put all of my video data in before transferring my music across. I’ve still kept a copy of my music on the laptop, for what it’s worth, but because I just copied files, then I lost all of my rating data.

I looked at a couple of solutions for transferring the ratings, the most promising was a semi-manual method of creating smart playlists, one for each rating.

That was too old-tech for me, so I came up with a solution that uses Remote Apple Events. Now, to make the code easier, I’m using python and appscript, so make sure you have both of those installed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

    #! /usr/bin/python  
    
    import appscript  
    
    # Set up the two iTunes libraries.  
    local = appscript.app('itunes')  
    
    # You'll need to replace jens.local with your remote machine's name  
    remote = appscript.app(url='eppc://jens.local/itunes')  
    
    local.lib = local.playlists()[0].tracks()  
    remote.lib = remote.playlists()[0].tracks()  
    
    # Create a dictionary with all local track names/artists/albums  
    library = {}  
    
    for each in local.lib:  
        key = each.name()+":"+each.artist()+":"+each.album()  
        library[key] = each  
    
    # Iterate over every remote track.  
    # If it is in the local library, take the local rating and  
    #    apply that to the remote track.  
    for each in remote.lib:  
        key = each.name()+":"+each.artist()+":"+each.album()  
        print key,  
        if library.has_key(key):  
            each.rating.set(library[key].rating())  
            print "rated."  
        else:  
            print "doesn't exist in local library"  

That’s it. I’ve used this to transfer all of my ratings from local to remote iTunes. Granted, there is no check to see if I’ve got all of the local tracks on the remote machine - but that is mainly because I don’t have my Podcast library on the remote machine, and I hope (know?) I have all of the music, and that’s the stuff I care about.

Here’s a screenshot from my remote library.

I’m going back to Uni next year, to complete a post-grad degree in Computer Science. This is a course that people without a background in Computing can enrol in, so I’ll be able to get a bit of credit towards some units, but not too much, since when I did part of a Computer Systems Engineering degree, Java wasn’t the language taught. Now, at all three Universities in Adelaide, it almost seems to be the only language being taught.

Just for my own personal gain, I’m also teaching myself Objective-C. This is the language used by Apple and third-party coders for most of the OS X application software. All of their APIs are available under Objective C, and Java, whilst still somewhat supported, doesn’t appear to be as high on Apple’s agenda as it is on the local Universities.

I’ve borrowed books in the past on Cocoa/ObjC, but never really made much progress. This time, having a laptop means I can actually do my learning on that, wherever I happen to be. In this coffee shop, for instance.

I’ve also found some handouts and example program “tests” at Stanford University, and have started to work through those. The first program, a simple conversion application for temperatures, was fairly simple, and I wrote that. I even managed to complete the extra credit sections, or at least two of them.

Psuedo-evidence (in that you can’t actually see this working):

(Note, the temperatures change to red when above a certain value, and to blue when below a certain value. Within the ’normal’ range, they are black).

This wasn’t a programmatically onerous task, more about making sure you can connect up various components and have them affect one another.

Instead of doing the next task, I’m pretty confident I’ll be able to write the application I gave to my Year 12 students, a 9-letter word puzzle solver. It will be interesting to see if I can effectively re-implement the python version I’ve already written:

The only reason I really want to learn Obj-C is that it enables me to use a GUI design tool, rather than having to create the interface using coordinates. And packaging up python applications isn’t exactly fun, either.

And, after the 9-letter puzzle solver, I might reimplement a sudoku solver I wrote. Not sure if I have the python source code for that one. And I never wrote a GUI for it, either.

Or Python3.0, as it is also known, has been released, or rather, the first alpha of it has.

It builds okay on OS X, although I’ll need to download some stuff to get it to build bsddb and readline, among other things. Since I use these two modules, I’ll get them at least.

There are lots of changes in python3.0, including integer division returning floats, and input() being what it should be (instead of having to use raw_input()). print has become a function, and a whole lot of other things have changed. In fact, programs are likely to not run under 3.0 without a rewrite.

I’ll keep 2.5 around for a while longer, methinks. Hell, I’ve still got 2.3 on here, just incase.

Okay, I think I’ve found an appscript bug. If you attempt to connect to a networked machine, such as doing: »> from appscript import * »> it = app(url=“eppc://user@machine.local/iTunes”) »> it.activate() Which works fine if you have a password, fails if there is no password. And the error reported is: RuntimeError: Can’t get terminology for application (aem.Application(url=‘eppc://user@machine.local/iTunes’)): CommandError -128: userCanceledErr Just because I haven’t put in a password, doesn’t mean I have cancelled the operation. I’m still working on a fix to get it to work.