Sara Farmer's List of Dataset Resources

Quoting Sara Farmer :

> Hi Terra,
>
> I've been thinking about this for a while, as part of the Cicada crisis
> data system (an extension of the data work that the SBTF did with ACAPS,
> scraping data out of various development websites). I also started
> collecting dataset links from the crisismapping list, and writing some
> code to harvest links from it using the Google api, then thought that if
> ever there was a task that could be crowdsourced, this is it... if any
> groups are interested in doing that...
>
> I also put up a CKAN node to play with
> (http://ec2-54-228-69-142.eu-west-1.compute.amazonaws.com) - e.g. use
> its API to add in all the datasets, links etc that I've found over the
> past few years (several other places to look are listed on
> http://opencrisis.net/datascience under "development data" and if I ever
> get the Google search API to stop thinking I'm a spammer, I also have
> code to search them all for dataset filetypes, e.g. CSVs etc) and link
> back to a friendlier interface - but as usual, I got busy on other
> things (there's this white paper on sensors... ), so there's nothing in
> there at the moment, and I'm still thinking hard about how to best
> organise datasets in it so they're easy for a non-expert to find... We
> have issues like multiple copies (or worse, almost-copies) of the same
> data from different sources, updated files etc. that we solved for ACAPS
> by creating one file for each combination of indicator, source and year:
> we could do the same again, but need to think about how that maps to
> CKAN, which clusters datasets by group, with each dataset containing
> resources (e.g. single links or files) and links to apps, ideas, posts,
> papers etc. Datasets have tags that we could use (e.g. for country,
> topic, indicator name etc) and resources have additional fields that we
> could use too (e.g. for source and year). CKAN groups are usually the
> data curators (e.g. SBTF etc) with groups having editing rights across
> the datasets belonging to those groups: with so many people here
> belonging to multiple groups, that might not be the best use of CKAN
> groups here, but can deal with that separately.
>
> I'm aware that I rambled on there. Anyways, you're welcome to come play
> with me on the CKAN node (and encourage me to add in all those links to
> datasets etc) - if you create an account on it, please tell me so I can
> give you explicit editing permissions - datahub.io has been overrun with
> spam recently, and I want to make sure this node doesn't have the same
> problem.
>
> Thanks,
>
>
> Sj.