Carl Hutzler’s Blog

A few weeks ago, I borrowed a robotic DVD/CD duplication machine (made by Primera Technologies) from one of my clients. As of the first of the year, Primera released some software that would allow the machine to be used for CD ripping (instead of burning). I thought this was great and I could finally rip the rest of my 4000 CDs into iTunes. Over a period of 2 weeks or so, I ripped everything at a fairly high bit rate. Great! I was almost done….well, not quite.

So over the years, my wife and I have purchased music online, ripped books on tape, and accumulated a fairly large library of music that was in addition to our tangible CD collection. So now that I had the CD’s ripped, I needed to combine the libraries somehow. I knew that some of the music in our current iTunes library had been ripped from our CD collection originally so I could not just combine them straight away. So what I had was a big mess of stuff which if I just consolidated them in iTunes, would result in many duplicate songs.

What I did was a multi-step process which I think could be streamlined if I had some more time and/or was a bit stronger in Unix. But here it is if it helps someone with the same issue. One note: I thought about trying to use iTunes and the library file structure for this, but I could not find a way to do it due to the issue I had on my hands. I was trying to stay away from duplicate songs which iTunes does a poor job of managing unless you are actively ripping a CD where itunes does warn you. Too bad they don’t have this for moving files around.

Let’s call the two music libraries ripped and current (the just ripped CD collection and the current iTunes library we had created over the years). And the actual MP3 files are located in directories (by iTunes) for these collections as follows:

./ripped/iTunes Music/Artist Name/Album Name/musicfiles.mp3
./current/iTunes Music/Artist Name/Album Name/musicfiles.mp3

1) First step was to simply copy all Artist Name folders from ripped to current. This would mostly combine the music files in one easy step. When the Mac OS Finder warns that some duplicates have been detected, just say to skip all of them. So if the Artist Name did not exist in the current library, we would copy the Artist and all albums and songs over to the current library. Great!

2) But now the ripped side is left with the artists where we actually had that artist on the current side. And to make matters more complicated, we might have had some, none, or all of the albums for that artist on the current side, compared to the ripped side.

I started moving Albums from ripped to current manually at first to see how slow the process was, but also to see what the criteria for moving the data over would be. There were over 700 of them so doing something manually was out of the question.

I searched around for a tool and while there is a lot of software out there, nothing seemed to do what I wanted. So I decided to use unix file system tools and it turned out to be fairly easy. First was to diff the two iTunes Music directories against each other using the following command:

diff -qr -X exclude "/Volumes/Music/ripped/iTunes Music" "/Volumes/Music/current/iTunes Music" > diffoutput.txt

The command will look at any differences between the directories while excluding regular expressions in my exclude file (aptly named “exclude”). The exclude file had *.mp3 and *.m4a in it because I did not care about difference in files…I only cared about differences in directories! As long as the Artist Name/Album Name path existed on the ripped side and not on the current side, I wanted to know. I assumed the actual music files inside Album Name should be unique if the artist and album directories were unique.

The output from the diff is cat’d into a file called diffoutput.txt.

3) Next I opened the diff output in excel. If my unix skills were better, I would not have needed this, but hey, it worked. In excel, I sorted the output so I could find just the directories that I had on the ripped side which were not on the current side. I did not care about the other direction.

Note: I found a few .DS_Store and .m4p files in the mix which I should have included in the exclude file for the diff command. Next time :-)

In excel I massaged the data into unix copy command statements. For example, if the diff output line said this:

Only in /Volumes/Music/ripped/iTunes Music/Neil Young: Rust Never Sleeps

I used excel concatenate and text manipulation formulas to massage it into this:

cp -R "/Volumes/Music/ripped/iTunes Music/Neil Young/Rust Never Sleeps/" "/Volumes/Music/current/iTunes Music/Neil Young/Rust Never Sleeps/"

I would imagine AWK/SED and other unix utilities could have made quick work of this step, but I am not a unix god yet. But once I had these copy lines, I was almost there.

4) Take all the copy lines and put them into a shell script and run it. Viola, I now had all the music files combined physically in the same directory structure. The last step is to drag/drop the entire iTunes Music folder back onto iTunes. iTunes then will “re-import” all the music and organize it. And since the music is already organized and much of it is already in iTunes (the current stuff is), the process is fairly quick as iTunes just imports the new stuff.

That’s it.