A few years ago, when Flickr was new, we made kind of a silly decision to only store our images on Flickr. While this did make a transition between computers easier and freed up a little bit of drive space, we decided that we'd like to pull those pictures back onto our own system.

In the past, I've tried several Flickr Downloadr (missing 'e' intended as a pun) programs, and everything choked or did strange things. Last night, right after I crawled into bed, I decided that I knew a better way.

I crawled out of bed, did a bit of Googling, and found an excellent Python Flickr library, and someone that wrote a python script to backup Flickr pictures to Amazon's S3. In 15 minutes or so, I had a solution that would page through our public pictures, check to see if they were already downloaded, and store them in year and month folders.

While running the script, I noticed that the obvious slow part of the process was downloading the images. I wanted a way to download them in parallel, and found my solution in an excellent Python thread pool solution.

I added the following method to their code, that will tell me the number of jobs pending in the thread pool:

I had to rewrite my code a little to put the download code into a method, and then I let 'er rip. Downloading now happened 3 at a time. It took about 2 hours to download our 5GB collection of approx 3,300 pictures, and the job was done!

I don't bother with authenticated requests (all of our pictures are public), and I don't do any error checking. It is a one off script, after all. :)

I love python. I really do.