I’d love to know if anyone’s aware of a bulk metadata export feature or repository. I would like to have a copy of the metadata and .torrent files of all items.
I guess one way is to use the CLI but this relies on knowing which item you want and I don’t know if there’s a way to get a list of all items.
I believe downloading via BitTorrent and seeding back is a win-win: it bolsters the Archive’s resilience while easing server strain. I’ll be seeding the items I download.
Edit: If you want to enumerate all item names in the entire archive.org repository, take a look at https://archive.org/developers/changes.html. This will do that for you!
You must log in or register to comment.
I use the CLI tool, even right now waiting to finish some downloads. The CLI tool can actually give you a list of all items with
ia list {ID}(replace {ID} with the actual id of the stuff you want to download). But you don’t even need to list the items, because you can download with a glob (in example*.torrentlike your shell has. Or if you have the ID anyway, you can specify the filenames too with{ID}_archive.torrentHere is an example how to do this with my own upload https://archive.org/details/gamefaqs_txt where the id becomes
gamefaqs_txtia download gamefaqs_txt --glob *.torrentor use a variable to set id and download all files that start with the id, which should be all the meta data
id=gamefaqs_txt ; ia download "${id}" --glob "${id}_*"Thank you for the tips. I am actually interested in enumerating metadata for all the “items” as defined by the API page ever uploaded. For example, one item = one ID:
Archive.org is made up of “items”. An item is a logical “thing” that we represent on one web page on archive.org. An item can be considered as a group of files that deserve their own metadata.
You did cause me to look at the API docs again, though, and I think I found something that does enumerate all item names, and as a bonus, it will keep you updated when changes are made: https://archive.org/developers/changes.html
We’ll see how much progress I can make. It might take a while to get through all the millions of them.
Isn’t “item” and “id” basically the same thing? Because every item has a unique id. So in my example
gamefaqs_txtwould be the item and id.Yes, I think so. I’ll definitely use the example for downloading some of the files (.torrent, metadata file) once I have some items. But first I need to find all the items ever uploaded.
I just found out you can get all metadata with
ia metadata ID > metadata.json(replace ID with gamefaqs_txt in example). So from there you could extract any information too, if you know how to handle json. (Edit: Just load the metadata.json in your browser to see a better formatted list.)I like to pipe my json to
python -m json.toolfor quick formatting in the terminal.Take a look at jq, it’s a really nice tool for handling json in the terminal, also gron for searching json
