Schinckel.net

I had thought that renaming a track from one user of a shared iTunes library (where the files are shared, but each has her own library file, with ratings/play counts, etc) would cause iTunes to not be able to find it for the other users. But, I renamed “Tom’s Dinner” to “Tom’s Diner” (Suzanne Vega), and switched over to the other user, thinking I’d have to do the old Find Track trick, but iTunes was smart enough to realise that the track it needed had changed name…and updated the library automatically.

Livin’ Thing • Greatest Hits • Electric Light Orchestra

I have one feature I’d like to see in iTunes - the ability to mark tracks as ‘identical’, and have them belonging to two (or more) albums, but sharing the rating, play count, and even disk space. That would be cool. Especially since I have lots of tracks in this situation.

I now have the code to check the library XML file and see if there are missing tracks (ie, the files are not where they are expected to be). This code is quite slow, but simple to follow. (It took around 2 min to run). I also have the code that gets a list of the files in the library directory. It must be simple to combine this information, and work out which ones are:

1. Files that are ’new’: they only exist on disk, not in the library. Chances are they were added by another user.
2. Files that have become detached: there is a file and a library location, but they don’t quite match up. This is probably because the users have ‘Keep Library Arranged’ turned on, and one of them has made a change to a track name, artist or album; or made a change to the compilation flag.

The trick will be having the list of files, and removing items from the list that have been located in the library. This will leave list that just need to be sorted into alpha and beta above.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

    import os
    import urllib
        
    library = os.path.expanduser('~')+'/Music/iTunes/iTunes Music Library.xml'
    startpath = '/Volumes/Media/Music'
    def greppy(library)
        data = open(library).readlines()
        tracks = {}
        this_track = 0
        for line in data:
            if line.count('<key>Track ID'):
                this_track = line.split('integer>')[1][:-2]
            elif line.count('<key>Location</key>'):
                tracks[this_track] = urllib.url2pathname(
                         line.split('string>')[1][16:-2]).replace(
                         '&#38;','&')
        
    findstr = "find "+startpath+"-type f -not -name .aacgained -not -name ._* -not -name .DS_Store | sort"
    treedata = os.popen(findstr).readlines()
        
    data = greppy(library)
        
    missing = {}
    surplus = treedata[:]
        
    for i in data:
        try:
            surplus.remove(urllib.urlopen(data[i]).url[7:]+'\n')
        except IOError:
            missing[i] = data[i]
        except ValueError:
            pass

This leaves two data structures of interest: missing, a dictionary with the ‘missing tracks’ from iTunes, and surplus, a list with files that do not have an associated iTunes library entry. Note: I’ve turned off comments, as this post seems to get a lot of comment spam.

Continuing on from my last programming related post, I’ve been giving some thought to the opposite problem: when tracks are in the iTunes Library, but the file they refer to is not there. iTunes marks these with an exclaimation mark in front of the track, but only if it realises they are in fact missing. Often, just clicking on a track does not mean iTunes knows that track is missing. A Get Info on such a track prompts the user for a choice between removing the track from the library, or locating it. It might be possible to use the following AppleScript paradigm to get a list of the tracks that are missing:

1
2
3
4
5
6
7
8
9

tell application "iTunes"
    set theTracks to every track in playlist 1
    set missing to {}
    repeat with theTrack in theracks
        if location of theTrack is [unknown] then
            set missing to missing + theTrack
        end if
    end repeat
end tell

Of course, that would be dead slow, as it relies on a couple of AppleScript events for each track. (Also, it may not work, but some sort of try - else - end try clause might work.) A better solution might be to sort through the iTunes Music Library.xml file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

    import os
        
    data = open("iTunes Music Library.xml").readlines()
    missing = []
        
    for line in data:
        if "<location>" in line:
            #extract location
            location = line.split("/Volumes")[1].split("</location>")[0]
            #check if location is not a file
            if not os.path.isfile(location):
                missing.append(location)

Assuming that the mount point is the same as the last time the XML file was written (and iTunes seems to be pretty clever about working out things like /Volumes/Media vs. /Volumes/Media-1, then your list missing should contain a list of files that are in the library, but not where iTunes expects them to be. It might be possible to then match these two lists (from this script), and see which ones are likely to be ‘modified tracks’, where one user has changed the track name, artist or album. It may even be worthwhile having a list of all track locations, and seeing if there are any duplicates, that will solve the problem of duplicates int he library that are the same file. I’d like to have python parse the XML file, and create data structures reflecting the XML data - but skip everything but the database ID and filename, as these are the things that we really need - but I haven’t had much luck with this type of process in the past. I jsut had a thought on how to do it, though.

I’ve written a python script that walks through a path tree, checking to see if each file in the tree is a track in the current user’s iTunes Music Library.xml file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

    import os, re
    
    startpath = "/Volumes/Media/Music/"
    prefix = "file://localhost"
    library = os.path.expanduser("~")+"/Music/iTunes/iTunes Music Library.xml"
    
    def eachpath(arg, path, tracks):
        for track in tracks:
            if os.path.isfile(os.path.abspath(path)+'/'+track):
                trackpath = os.path.join(os.path.abspath(path),track)
                grepstr = prefix+trackpath.replace(" ","%20")
                if grepstr not in data:
                    arg.append(grepstr)
    
    data = open(library).read()
    missing = []
    os.path.walk(startpath, eachpath, missing)
    
    print missing

It’s not flawless - on my machine it eats up over 11 meg of memory, and takes ages to run, but as a proof of concept, it works okay. The memory it uses is mostly because is stores the whole iTunes library file in memory, so that’s 9 meg on my system already. The main loop is doing a string1 not in string2, which is probably not optimal, but it was easy to code, for now. I’m still waiting to see how long it takes to do my whole library, but I’m getting bored with waiting. Edit: to reduce the time taken, I used the following code in the final if clause in the function:

1
2
3
4
5
6

try:
    if not re.search(grepstr, data):
        arg.append(grepstr)
except:
    if grepstr not in data:
        arg.append(grepstr)

The re one is much faster, but fails in some cases: the second one, while slower, is a fallback. There are also some other issues, at this stage I have not cared that much about escaped characters, which iTunes uses when storing the information. But, I came up with a quicker method than python’s os.path.walk(). Using the find command is much quicker:

1

    find /Volumes/Media/Music -type f -not -name .aacgained -not -name ._* -not -name .DS_Store

takes between 12-36 seconds for my 5700+ library stored on my NSLU2. If I telnet into the NSLU2 and run the equivalent command:

1

    find ~media/Music -type f -not -name .aacgained -not -name ._* -not -name .DS_Store

it takes on average less than one second to complete. So, that’s more than an order of magnitude, even if the network traffic is low. Oh, and it compares very favourably with the python version, which takes at least one minute to run.

Jaq and I share two computers, an iMac G4, and a Dell PC. I also bought a Linksys NSLU2 and a large USB Hard Drive, so that all of our music and videos can be stored on a server, and accessed from either computer (or the Xbox) without having to make sure the iMac was on. (That was the main computer, and the one we fight to get onto). Of course, the NSLU2 helped remove clutter from the iMac’s hard drive, not to mention freeing up a heap of space. Anyway, because we have a rather large music collection, it’s meaningless and wasteful to have copies of music stored in two places - I set up an SMB share on the NSLU2, wrote a small AppleScript to mount this on bootup, and pointed iTunes towards this location. This has the feature of storing one copy of all of our music, in the one location. There are some drawbacks, however:

1. If I import music, it doesn’t appear in Jaq’s library by default. Similarly, if she imports, I don’t see it. Every now and then you need to drag the Music folder onto iTunes, and wait for it to update the library. Both of us need to do this, incase both of us have imported music.
2. If one of us edits a track’s artist, title or album, iTunes for the other user sometimes cannot find the track. If you then re-import you wind up with two copies of the track, one of them (the one with the rating, and playcount) is a ‘dead track’.
3. Sometimes a re-import causes a track to appear twice in the library. Sometimes it creates a second copy of the file in the directory.
4. There is a lot of music in our library that one or other of us doesn’t really like that much. For instance, I listen to a lot of Classical music, but don’t like Red Hot Chilli Peppers. Even if you remove a track from your library, it gets re-added when you do a re-import.

Of course, there are some great benefits, too:

1. Each person gets to have their own rating and playcount for each track. Initially we had a shared iTunes library file (both of us had read/write access to it), which worked well when only one user could run iTunes at a time, but fails dismally when multi-user is taken into account.
2. Our iTunes library only takes up half of the space.
3. I can modify the tags belonging to a track, and it gets propogated to her library.

I’m fairly confident the benefits outweigh the costs, but I’m still keen to come up with a better solution. Here are some ideas I have had to resolve some of the issues. iTunes stores a copy of it’s library in an XML file - it should be a trivial task to scan this and get some information that might be useful. For instance, compare the location field of each track to the directory structure, and work out if there are tracks that need to be added, or the path they have needs to be updated.

This requires a couple of things:

• A decent XML parser (in some cases just a simple grep will do the trick - for instance seeing if a filename exists in the XML file).
• An interface (AppleScript) to iTunes, to tell it to add/remove/re-locate files.

I have been using Jaq’s iPod the last couple of days - mainly because when I am waiting at the Adelaide Railway Station, and at Lynton Station, I cannot get good enough radio reception on my phone’s radio. So, I copied all of my Classical music onto the iPod, and I’ve been listening away. I’d like to be able to rate tracks on the iPod, but I don’t really want to, for one big reason. I would only be able to rate them at 0-5 stars, not using the 0-100% ratings I use from iTunesRater. I wonder if it’s possible to hack the iPod firmware so that rating occurs by a finer gradient?

I have got (at last count) 5715 tracks in my iTunes library. I have rated 1988 of these, or around 35%. Show Duplicate Songs tells me I have 409 duplicates. (Maths tells me I have at least one triple!). I have 225 unrated duplicates, so I have rated nearly half of the duplicate songs. Whilst some of the duplicates are live versions, and I may not like one or other version quite as much, it’s a fair bet the ratings would be similar, if for instance the tracks were the same, but were from best-of albums, compliations and the like. So, it might be nice to have a program that gets all of the tracks that are duplicates, but at least one of them has a rating (and at least one of them doesn’t), and apply the rating from the rated one to those that don’t have ratings. Of course, you could have it that if there were three songs, and two of them were rated, that the third gets the average rating of the other two. I may write an AppleScript that does this. Rag Doll • Songs About Jane • Maroon 5

Using the program Meow, it’s possible to see what notifications different programs give to the system - things like changing the menu-bar, that the menu was activated, and in iTunes, that the song has changed. Yes, you heard right - there will be no need to poll the system every n seconds to see if the song is the same! Or if the song is paused! Now I just need to find out how to discover these notifications, and act upon them.

Since I now carry my Zire everywhere, I’d like to be able to keep a database of my iTunes Library, and if I get around to it, my physical books as well. Perhaps a reading list too…