I'm working on a project that will require a list of all sets including all of the parts they contain. I'm aware of at least three sites which have this information (Peeron, BrickLink, and Rebrickable), but I haven't been able to find a way to download all of this data easily. The closest thing I have been able to find so far is the Brickset API, but Brickset only has a limited amount of set metadata (year, theme, part count, etc) and does not seem to have a full listing of parts contained in each set.
asked Dec 29, 2012 at 13:43 41.5k 11 11 gold badges 125 125 silver badges 248 248 bronze badgesI wasn't able to find anything like this, so I contacted the admins of the sites that I mentioned in the question. Rebrickable was kind enough to send me a dump of their set inventories under a Creative Commons 3.0 BY-SA license, which basically means that you can do whatever you want with the data as long as Rebrickable is acknowledged as the source and you share any updates that you make.
I've created an archive which contains a spreadsheet for each set that was in the dump that I received. Hopefully this is useful to someone else.
The raw data that I used to create these spreadsheets is now available directly from Rebrickable and is updated monthly there. I also maintain software to build an SQLite database from these dumps.