Faulbaer's Schlafmulde :: english :: hacking :: aperture-library spring-cleaning for the brave
2010.02.05, 21:09

aperture-library spring-cleaning for the brave

first of all, let me warn you: make no mistake - this is dangerous business! if you don't know what you are doing, you can and most certainly will lose some or all of your photos! read everything top to bottom before doing anything else. read everything repeatedly until you know what's going on and what I'm doing here. don't come whining to me if you lost any data - you've been warned!

now that that's off my chest let's dive right into it after a short introduction, shall we?

when apple announces aperture some years ago it seemed to be the perfect solution to all the problems I had been experiencing with iphoto. like every other apple product it wasn't perfect but it did the job much better than most competitors' products - at least after the upgrade to it's second version. unsurprisingly it was not the solution to all the problems but at least some of them had been addressed properly. what were the remaining problems then?

- it's still slow - especially with large libraries

- backups are taking ages and they break easily

- you can lose photos and projects from aperture crashing

- decentralized and distributed storage hasn't been addressed properly

there were several other small and large issues I'm not going to fix here.

most of the problems come down to one issue that can be cured: huge libraries beyond several 100 gb.
with huge libraries, backups need to take longer, going through the previews needs to take longer, aperture will consume more memory and crashes probably lead to larger chunks of data to be lost.

the solution to most of my problems was to shrink my aperture library. I could have deleted several thousand photos but as you can imagine I didn't want to delete my work. what I did was to divide my one large aperture library into many smaller libraries. for all the years until 2006 there was one new library. for the years after 2006 I had a library for each year. for 2009 I'm pondering to split the year in halves or even quarters because this library alone takes up about 550 gb. if I had reoccurring events I might have divided the library not only by date but also by topic.

the problem with the solution

the usual aperture way of doing what I was going to do can be described as the following:

a) open the source library

b) right-click the project and select "export project" from the pop-up menu

c) chose an export path and tell it to move the master-files into the project-file

d) wait and hope aperture doesn't crash

e) right-click the project and select "delete project" from the pop-up menu

f) click a bunch of silly questions away

g) close aperture to make sure the changes have been written to the library

h) wait and hope aperture doesn't crash

i) open the destination library (if it doesn't exist yet, open aperture while pressing the shift-key to make a new library)

j) import the just exported project

k) wait and hope aperture doesn't crash

l) close aperture to make sure the changes have been written to the library

m) go back to a) until all the projects are where you want them to be, shave, go outside and take a look at what the world is like in 2050 ... or whenever you've finished exporting projects one-by-one from aperture.

ok - you could have exported those projects one-by-one but then could have imported them as a folder which could have saved you time - but if you played save, the above list is pretty much what you had to do.

the hacker approach

needless to say that I wasn't going to do this the hard way but the risky hackery way. there had to be a way to get that work done much faster without repeating so many small steps and without answering the same silly questions over and over again ... and there was! It was just a bad idea ;)

as you might know apple uses so called packages for many things that need to be stored. applications, documents, sparse-disk-bundle disk-images and also the aperture libraries are stored in those packages. these packages usually are just a folder with an extension in it's name as well as a bunch of files (usually xml) and folders inside. you can take a look into a package by right-clicking it in the finder and selecting "show package contents" from the pop-up menu. if you were command-line savvy you could just navigate into the package in your preferred shell with the usual cd command.

a few words about the aperture library.aplibrary

the aperture library package isn't too exciting either. it contains some .plist and other xml-files, the main library as well as some folders with the photos organized in project-packages which again contain some xml-files, the photos, their previews and versions, some meta-data and albums as well as smart albums. albums and smart-albums also lie in those previously mentioned folders and either the main library or the .plist files link to them. I decided for saving time and agains saving rogue album data since I only had just about ten albums that were not within project package boundaries but instead in a folder somewhere else. I couldn't find an easy way to import such rogue albums and smart-albums but honestly I didn't care.

how big was it and how long did it take?

my original aperture library was just short of being 800 gb big and it contained about 754000 files. this includes the meta-data, xml files, the photos, previews and versions in different formats like jpeg, raw, psd and the like. to copy this library took me a day, partially because I hadn't invested into fast storage. to back new images up into the aperture vault never took any less than five minutes because aperture needed to check the vaults integrity first - also it had to seek through and write into huge xml main library files almost a gigabyte in size. with this many files and such a large xml library to parse through, performance had to be low.

two ways to shrink it, two days to get the job done

my first approach was to just move all the projects and folders out of the package and delete their references from the database/library by erasing the folders within aperture. I soon found out that this was taking ages and I had to repeat too many steps too often. this was the right approach in only one case - in the case there were less projects to be removed than there were folders to be imported. in the case of photographs imported from iphoto containing all the collected digital photographs I had taken between 1997 and 2006 there were hundreds of folders containing hundreds of projects containing between one and some hundred photos ... to sort through them and consolidate them into less projects with albums referring to them seemed to be far too much work. so what I did was to remove everything past 2006 first on the filesystem-level and then from within the library. all in all that included only six folders: 2007, 2008, 2009, 2010, misc and airsoft. I still had to right-click each one of them, answer some questions and wait for aperture to clean up the database, hoping it wouldn't crash. after each folder I quit aperture and fired it up again. luckily it didn't crash within this procedures - not even once! to finalize the library, I saved it into a new aperture vault of the same name.

my second approach was to open aperture while holding the option-key and by that starting aperture without a library. it then would ask me to select an existing library or instead start with a new one. I chose the latter, added some folders like 2007, 1st quarter, 2nd quarter and so on and afterwards imported the projects I had moved out the aperture library.aplibrary package into their corresponding folders. after repeating this for each of the mentioned folders I quit aperture and launched it again. aperture crashed five times which still makes me angry because after a crash I had to remove the just imported folders, quit aperture, relaunch it and import the same folders again, just to be sure the library was consistent - I've had bad experiences with corrupt aperture libraries in the past. I lost about 70 photos, some of them really nice ones because of those crashes. finally I saved the new library into an aperture vault of the same name again.

it took me two days to complete the task but now I feel much safer and the best thing is that aperture itself feels much faster and much more responsive.

why I did it - the aftermath

now that I know everything worked and still works I can say why dividing up my aperture library into smaller chunks in an ordered way was a good if not great idea.

for one, it feels faster which is not a big surprise. aperture needs to address less memory to parse through a smaller library. it needs less time to load and write into the library. there is less to be backed up if anything changes and there is less to be restored if anything goes wrong.

also the libraries are smaller and therefore much more flexible to be worked with. I could easily archive those smaller libraries to several old hard-drives that I could easily store away off-site. before any of those drives had to be at least of the size of 1 gb and it took me ages to get one backup stored away. time in which I couldn't work with aperture which proved to be annoying several times.

today it's also much easier for me to just fire up the aperture vault backup-procedure because I know it's not going to take too long in most cases. this can prove to be a life-saver in the future because I won't 'forget' about backing up crucially needed data.

it's not all good

there are some downsides I need to mention before you dive into it - it's not just all good. as it is with all the other database-oriented programs from apple, the ilife system doesn't expect you to divide your databases into many smaller ones - I mean ... well ... it's obvious apple doesn't expect you to outgrow their products faster than they make faster hardware available if not affordable to you - but in some cases that's just what happens if you use your computer professionally or . this happened to my itunes library in the past, to my iphoto libraries and now to my aperture library. in the case of itunes apple just got me faster hardware so I could reconsolidate my itnues library back into one. in case of iphoto i switched to aperture which solved the performance issue at least for some years. now in aperture I reached the borders of what I felt bearable and again I divided a library for good - let's see if apple comes up with either a better library/database format for aperture 3 or a better mac pro I can buy that can handle the load.

what are the downsides of a divided aperture library?

not so many to be honest but they are pretty annoying nevertheless.

- itunes get's all confused - it will only put photos from your current library onto your ipods, iphones and apple-tv. this is a biggie for me so I'm considering to export things I always want on my iphone into the iphoto library since I'm not working in iphoto anyway it's more or less static. I'd then fill my media-devices from this iphoto library.

- needless to say that the above is true for all the applications sharing media via the ilife hub. that includes iwork, ilife, ecto and many more. it's really a pity that apple offers to divide libraries but doesn't care for keeping track of those. there should be a way to link to those decentralized media-stores.

- it's annoying to seek through several databases by quitting and relaunching aperture. but that again is apples fault - there is not really one good reason why aperture can't handle several open libraries parallely like mail.app handles several mail-accounts and almost any app handles several open files. I sincerely hope apple is going to address this issue in it's future aperture versions.

- keeping track of several libraries and versioning several libraries could prove to be a hassle in the future. I can imagine deltas between several versions of the same library backed up to different drives in different locations but there are ways to deal with this and it's nothing really new either - it just needs for me to keep it in mind, doesn't it? ;)

a final word

back everything up, quit and relaunch aperture as often as you can. aperture is a beast with memory leaks and it's obese and heavy and stupid. I don't know any better product for the job it does and that's the only reason why I use it. keep in mind that aperture is only there to lose your photos and make your life a misery. so again: make backups before you do what I proposed - even better for you is if you don't do it at all! you are probably better off if you do it the painful slow way or not at all - maybe apple is going to supply us with much faster hardware really soon - you better wait until your new machine has arrived and keep your aperture library.aplibrary intact as it is!

also don't forget that you probably lose some albums and smart-albums not stored in any particular project but a folder or anywhere else.

don't do it and if you do it, make backups (that's plural for backup which is going to be too few - make more than one, make several backups!).

Faulbaer (you've been warned - not once, not twice but three times!)

 
add a comment