ecto: Auto abbr/acronym

There are a couple of instances of scripts out there that automatically apply abbr and acronym tags to pages, but I wanted to be able to do the same in ecto. This is also the first time I wrote a plugin script for ecto, and I wanted to do it in python. Please note that this here script is untested until I get onto my Mac and test the hell out of it. The script works, with the caveat listed in TODO.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    #! /usr/bin/env python
    
    'A script for ecto that adds abbr and acronym tags to the text'
        
    TODO = '''
    Fix it so that acronyms without a space either side (for example,
    that finish a sentence) work.
        
    Lookup on the internet for a list of acronyms/abbreviations?
    '''
        
    acronyms={'WYSIWYG':'What You See Is What You Get',
              'DOM':'Document Object Model'}
    abbrs={'XHTML':'eXtensible HyperText Markup Language',
           'NSLU2':'[Linksys] Network Storage Link (USB) 2.0'}
        
    # Add more values to your hearts content…
    
    # get input data - depends on implementation.  For ecto:
    import sys
    data = open(sys.argv[1]).read()
        
    # replace only the first instance of each acronym/abbreviation
    for each in acronyms:
        data.replace(' '+each+' ', '<acronym title="'+acronyms[each]+'">'+each+'</acronym>',1)
    for each in abbrs:
        data.replace(' '+each+' ', '<abbr title="'+abbrs[each]+'">'+each+'</abbr>',1)
        
    #return data to ecto
    open(sys.argv[1],'w').write(data)

Note: Comments turned off: too much Spam on this entry.

iTunes Shared Library Checker

I now have the code to check the library XML file and see if there are missing tracks (ie, the files are not where they are expected to be). This code is quite slow, but simple to follow. (It took around 2 min to run). I also have the code that gets a list of the files in the library directory. It must be simple to combine this information, and work out which ones are:

  1. Files that are ’new’: they only exist on disk, not in the library. Chances are they were added by another user.
  2. Files that have become detached: there is a file and a library location, but they don’t quite match up. This is probably because the users have ‘Keep Library Arranged’ turned on, and one of them has made a change to a track name, artist or album; or made a change to the compilation flag.

The trick will be having the list of files, and removing items from the list that have been located in the library. This will leave list that just need to be sorted into alpha and beta above.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
    import os
    import urllib
        
    library = os.path.expanduser('~')+'/Music/iTunes/iTunes Music Library.xml'
    startpath = '/Volumes/Media/Music'
    def greppy(library)
        data = open(library).readlines()
        tracks = {}
        this_track = 0
        for line in data:
            if line.count('<key>Track ID'):
                this_track = line.split('integer>')[1][:-2]
            elif line.count('<key>Location</key>'):
                tracks[this_track] = urllib.url2pathname(
                         line.split('string>')[1][16:-2]).replace(
                         '&#38;','&')
        
    findstr = "find "+startpath+"-type f -not -name .aacgained -not -name ._* -not -name .DS_Store | sort"
    treedata = os.popen(findstr).readlines()
        
    data = greppy(library)
        
    missing = {}
    surplus = treedata[:]
        
    for i in data:
        try:
            surplus.remove(urllib.urlopen(data[i]).url[7:]+'\n')
        except IOError:
            missing[i] = data[i]
        except ValueError:
            pass

This leaves two data structures of interest: missing, a dictionary with the ‘missing tracks’ from iTunes, and surplus, a list with files that do not have an associated iTunes library entry. Note: I’ve turned off comments, as this post seems to get a lot of comment spam.

Extracting Data from XML

Python does have tools for grepping XML files, but I’ve never been able to get them to work to my liking. I’ve generally just stripped out the data I need. And I will continue to do so, as it’s probably much faster than filtering through all of the crud I don’t need.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
    library = os.path.expanduser('~')+'/Music/iTunes/iTunes Music Library.xml'
    data = open(library).readlines()
        
    tracks = {}
    this_track = 0
    for line in data:
        if line.count('<key>Track ID'):
            this_track = line.split('integer>')[1][:-2]
        elif line.count('<key>Location</key>'):
            tracks[this_track] = urllib.url2pathname(line.split('string>')[1][16:-2]).replace('&#38;','&')

The above code will search through the library file, and grab info on each track: just the database ID, and the location (which is a URI, encoded to remove spaces and dodgy characters. The info is then put into a dictionary, where the key is the database ID, and the value is the location. Note that there is a replace() at the end of the last line - for some reason python’s urllib.url2pathname() function doesn’t replace & characters - I guess that’s because these aren’t really intended to be in a filename. Also, on my NSLU2 the extended characters are replaced by underscores, but I’m going to update to samba 3 (at the risk of mucking up the entire library…) to see if this fixes that issue. Anyway, after coding this, I had a bit of a think, and came up with the following method of doing the same (ensure it’s all on one line):

1
2
3
    grep Location ~/Music/iTunes/iTunes\ Music\ Library.xml |
      awk 'sub("<key>Location</key%gt;<string>file://localhost","",$1)' |
      sed 'sx</string>xx'

The python version uses between 5-8 seconds of CPU time, the grep version around 1.5, but does not associate the database ID’s with the locations, which I need. It also looks to be much easier to do the changing of characters (%20, for instance, into a space) that I need to do so I can check to see if files exist. Actually, using urllib.urlopen(), I can use the escaped/quoted version to see if the file exists, but it might be slow.

Find Missing Tracks

Continuing on from my last programming related post, I’ve been giving some thought to the opposite problem: when tracks are in the iTunes Library, but the file they refer to is not there. iTunes marks these with an exclaimation mark in front of the track, but only if it realises they are in fact missing. Often, just clicking on a track does not mean iTunes knows that track is missing. A Get Info on such a track prompts the user for a choice between removing the track from the library, or locating it. It might be possible to use the following AppleScript paradigm to get a list of the tracks that are missing:

1
2
3
4
5
6
7
8
9
tell application "iTunes"
    set theTracks to every track in playlist 1
    set missing to {}
    repeat with theTrack in theracks
        if location of theTrack is [unknown] then
            set missing to missing + theTrack
        end if
    end repeat
end tell

Of course, that would be dead slow, as it relies on a couple of AppleScript events for each track. (Also, it may not work, but some sort of try - else - end try clause might work.) A better solution might be to sort through the iTunes Music Library.xml file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
    import os
        
    data = open("iTunes Music Library.xml").readlines()
    missing = []
        
    for line in data:
        if "<location>" in line:
            #extract location
            location = line.split("/Volumes")[1].split("</location>")[0]
            #check if location is not a file
            if not os.path.isfile(location):
                missing.append(location)

Assuming that the mount point is the same as the last time the XML file was written (and iTunes seems to be pretty clever about working out things like /Volumes/Media vs. /Volumes/Media-1, then your list missing should contain a list of files that are in the library, but not where iTunes expects them to be. It might be possible to then match these two lists (from this script), and see which ones are likely to be ‘modified tracks’, where one user has changed the track name, artist or album. It may even be worthwhile having a list of all track locations, and seeing if there are any duplicates, that will solve the problem of duplicates int he library that are the same file. I’d like to have python parse the XML file, and create data structures reflecting the XML data - but skip everything but the database ID and filename, as these are the things that we really need - but I haven’t had much luck with this type of process in the past. I jsut had a thought on how to do it, though.

Find tracks not in iTunes library

I’ve written a python script that walks through a path tree, checking to see if each file in the tree is a track in the current user’s iTunes Music Library.xml file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
    import os, re
    
    startpath = "/Volumes/Media/Music/"
    prefix = "file://localhost"
    library = os.path.expanduser("~")+"/Music/iTunes/iTunes Music Library.xml"
    
    def eachpath(arg, path, tracks):
        for track in tracks:
            if os.path.isfile(os.path.abspath(path)+'/'+track):
                trackpath = os.path.join(os.path.abspath(path),track)
                grepstr = prefix+trackpath.replace(" ","%20")
                if grepstr not in data:
                    arg.append(grepstr)
    
    data = open(library).read()
    missing = []
    os.path.walk(startpath, eachpath, missing)
    
    print missing

It’s not flawless - on my machine it eats up over 11 meg of memory, and takes ages to run, but as a proof of concept, it works okay. The memory it uses is mostly because is stores the whole iTunes library file in memory, so that’s 9 meg on my system already. The main loop is doing a string1 not in string2, which is probably not optimal, but it was easy to code, for now. I’m still waiting to see how long it takes to do my whole library, but I’m getting bored with waiting. Edit: to reduce the time taken, I used the following code in the final if clause in the function:

1
2
3
4
5
6
try:
    if not re.search(grepstr, data):
        arg.append(grepstr)
except:
    if grepstr not in data:
        arg.append(grepstr)

The re one is much faster, but fails in some cases: the second one, while slower, is a fallback. There are also some other issues, at this stage I have not cared that much about escaped characters, which iTunes uses when storing the information. But, I came up with a quicker method than python’s os.path.walk(). Using the find command is much quicker:

1
    find /Volumes/Media/Music -type f -not -name .aacgained -not -name ._* -not -name .DS_Store

takes between 12-36 seconds for my 5700+ library stored on my NSLU2. If I telnet into the NSLU2 and run the equivalent command:

1
    find ~media/Music -type f -not -name .aacgained -not -name ._* -not -name .DS_Store

it takes on average less than one second to complete. So, that’s more than an order of magnitude, even if the network traffic is low. Oh, and it compares very favourably with the python version, which takes at least one minute to run.

Sudoku Solver: Sudokudo

The Advertiser, great newspaper as it is (that was sarcasm) has been plugging Sudoku puzzles of late. For the uninitiated, a sudoku puzzle is a 9x9 grid, that is broken up into 9 smaller 3x3 squares. Each row, column, and 3x3 grid contains all of the digits 1-9, and (obviously) each row, column and small grid contains each digit only once. The trick is to work out which ones go where. It’s all about logic, and working out which digits will not fit. So, I thought I would write a small python program to do it. I was prompted to do this by a puzzle I was working on on the way home, and I thought to myself “this puzzle appears to not have enough information for me to continue”. So, I started marking 9 marks in each box, showing which values could fit and not fit into each box. This was time consuming, but should be easy enough to do programmatically. Once you have this down, it’s simple to see if there are any boxes where only one value can go, or any rows, columns or squares where there is only once slot a particular digit can fit. And my program did just this. (I actually made an object-oriented one, that uses classes for cells, and puzzles, and methods for checking rows, columns, squares, etc). But, I seem to be missing one logical leap. The first puzzle I tried got solved with a few iterations of my program, but the other two I brought home get stuck. And one of these was one I solved earlier today, so I cannot recall how I was able to make the leap of logic my program cannot.

txt2pml.py

I found a heap of Palm TEXt REAd documents, but I don’t like the way they appear as (Doc) in PalmReader. So I wrote a script to convert them to PML, the format used by DropBook to make PNRd PPrs documents.

Basically, I found that on OS X the translation of txt2pdbdoc -d (decode back into text) wasn’t so good; a heap of characters needed to be changed.

I did have them all listed here, but if I edit them, ecto fucks up the encoding!

I also changed the === to an 80% horizontal line.

The [[[ - ]]] blocks were indented, and a footer line [* <Text>] is indented also.

I assumed the only use of a / was for italics, and _ for underlining.

I also assume the first non-empty, non === line is the Title, and the Author line starts with By. I use this info to create a ‘Title Page’.

The tricky bit was getting the Chapter Heading sorted, I needed to break the text into a list of strings to do this, and scan through. This slows the script down a lot, but it still works okay. I might profile it a bit and see where the slowdown is.

Anyway, here’s the latest version I’ve uploaded: txt2pml.py

I plan to make a version to process Project Gutenberg texts, but that’s on the back burner.

Sharing iTunes Songs

I finally got around to downloading the 10.3.4 OS X update, SharePoints, and Xbox Media Centre (XBMC). My aim for some time has been to share my iTunes library to my Xbox, which is connected up to the TV and Stereo in the lounge.

The MacOS X update provided (apparently) updated Samba sharing, but I still couldn’t get XMP (Xbox Media Player - the previous incarnation of XBMC) to connect to the SMB share I had set up, with all of my MP3s on it. SharePoints fixed that - all I had to do was create a publicly available share.

Setting up XBMC was a breeze, and it even has a cool interface! UnleashX is (almost) banished from my system, and XBMC is my main Dash.

Then I discovered that XCMC can play AAC files! So, I started (and have not yet finished) converting all of my MP3s to AAC. And not just converting, but re-importing, at a bitrate of 128 (small file size, but apparently equivalent to 256 in MP3). See this hint for some tips (and my comments).

I also wanted to re-jig how I had set up sharing my iTunes songs between users - I had a couple of links to the relevant files in each user’s ~/Music/iTunes, and all files were located in ~shared/Music (which all of admin can RW, and all of the world can R). I needed to have seperate Library files, so that we can have our own ratings (I hate Abba, and she isn’t that big a fan of Paul Kelly, for instance). So, I knocked up a script that scanned the ~shared/Music folder and compared any files found to the iTunes LIbrary. If they weren’t in it, it added them.

Here we go:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
set _tracks to “”
tell application “iTunes”
    set sel to tracks of playlist 1
    –set sel to get a reference to selection
    repeat with t in sel
        set _tracks to _tracks & “
” & (location of t as string)
    end repeat
end tell

tell application “Finder”
    set _files to “”
    set _library to “Macintosh HD:Users:Shared:Music”
    set _artists to folders of folder _library
    repeat with _artist in _artists
        set _albums to folders of _artist
        repeat with _album in _albums
            set temp to files of _album
            repeat with _file in temp
                set _files to _files & “
” & _file
            end repeat
        end repeat
    end repeat
end tell
set _missing to “”
repeat with para in paragraphs of _files
    set para to para as string
    if para is not in _tracks then
        set _missing to _missing & “
” & para
    end if
end repeat

tell application “iTunes”
    repeat with para in paragraphs of _missing
        set para to para as string
        if para is not “” then
            set _file to para
            add _file
        end if
    end repeat
end tell

Which worked, but was a bit slow. So I came up with the following system instead:

Create a file called ~/.last_check

Enter the following into a script, and then run it. (I’ve had to modify some lines to get it to look good - particularly the fp=os.popen… line).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#! /usr/bin/env python 
# Check for songs newer than ~/.last_check, and add them to iTunes

import os
import sys

fp = os.popen(
"find ~shared/Music -name *.m?? -newer ~/.last_check")

data = fp.readlines()

if len(data) == 0:
    print "No New Songs."
    sys.exit(0)
    
for line in data:
    filename = "Macintosh HD"+line[:-1]
    filename = filename.replace("/",":")
    print "Adding", filename
    os.system('''osascript < <END
tell application "iTunes"
add "'''+filename+'''"
end tell
END
''')

Should be pretty quick - I only tested it with a few files, but seems to work okay. iTunes is even smart enough to not re-add files that are already there (I think!), so it won’t add duplicates!

My library lives in ~shared/Music, yours may vary!

You will, however, need to use something like Super Remove Dead Tracks if you update one library with a new encoder!

I accept no responsibility if it screws up your iTunes Library file!

PyMail - a futile exercise?

I spent a few hours last night playing around with scripting Mail.app - what I really want to be able to do is read my email from a shell interface (like pine, or something), but still use the emails I’ve stored in my Mail.app mailboxes.

Option 1: Use pine.

I first tried this by creating sym-links from the various mailboxes found in ~/Library/Mail/Mailboxes/*.mbox/mbox, and POP-``_accountname@server_``/*.mbox/mbox into ~/mail/ (the default location of pine mailboxes). Mail.app stores a seemingly normal unix/pine mbox file, but there are some other files there as well: Info.plist content_index table_of_contents and a folder for each attachment, with a decoded version of the attached file. However, creating a symlink from the Finder, or a softlink (ln -s) doesn’t work - pine won’t recognise the files as being there. Using a hard link (ln -i for safety) works, but has the undesired side effect of resetting the mark (Unread/Read/Flagged - the blue dot that notes a new message) on all messages in the mailbox to Unread. Not acceptable.

Cleverly, I thought, what about making the original file read-write, and the ~/mail/ hard links (which are a bit like another version of the file that the system keeps synchronised) read-only. However, changing the flags on one reference to a file changes it on the other. Does it do the same for a change of ownership? Yes, interesting. [Fixes up a problem that was occurring with a shared itunes library because of this. Sometimes sym-links are better then hard links!] So, it looks like Option 1 is no good.

Option 2: use OSA scripting to get mailbox and message details from Mail.app

First thing I noticed was that this is dog slow. Maybe because I can’t get the osa module working in python, so I have to create a string, and os.popen() it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
def get_mailboxes():
    cmd_str = """osascript &lt;&lt;END
tell application "Mail"
get mailboxes
end tell
END
"""
    fp = os.popen(cmd_str)
    data = fp.read()[8:-1].split(", ")
    return data

Even so, I persevered, eventually creating a program that is able to get a list of mailboxes, and then a list of messages within a mailbox, and then all of the data from a message. Since python has it’s own email module, it is easier to just do an email.message_from_string() on a string of text, rather than ask Mail.app for the headers. However, as soon as you start having attachments, it’s quicker to get the headers from Mail. And waiting 30 sec just to look in my inbox is too long. Still, better than getting my GF to switch users. Then, I discovered something worse than this. A switched-out user is unable to run any AppleScripts - the same reason that you can’t telnet/ssh in and run an Application (open -a) without sudo-ing - cannot access the Display Server.

Option 3: directly read the mailboxes from python, but don’t make any changes.

This looks like it might be the only way to go - can be done with a switched-out (or not even logged-in) user. Of course, it was 3am by the time I came to this decision, so It’ll have to wait for another day. In the meantime, checkout pysh, a fully python replacement for bash/sh.