Re: st: Re: Web APIs and JSON file format reading

[Andrew Dyck <andrew@andrewdyck.com>]

Thanks Eric,

In my brief understanding of json, a generic -injson- program may not
be possible but there may be a way to create some function(s) that can
be readily adapted for use on different sites. My approach will be to
start with WDI/Twitter data as examples and see how it goes. At the
least this will be a good chance to dive into some stata methods I may
not know well.

I would love to see some examples of programs you've worked on to
parse json data to see your strategy. For now my approach is to read
the data from an api to text file, then parse into the dta structure
using some string functions.

Cheers,
Andrew



On Sun, Dec 19, 2010 at 7:11 PM, Eric Booth <ebooth@ppri.tamu.edu> wrote:
> <>
>
>
> Hi Andrew:
>
> I've written a couple of do/ado files to copy data from argument-based APIs (mostly Google APIs) to a txt file, then -infile-/-intext-(from SSC)/-insheet-  them into Stata, and then  parse them using a combination of string_functions (though nothing in Mata), but I gave up on the JSON parsing program I was writing because I found Google Refine and it's  ".parseJSON" class made quick work of getting the data I needed.
>
> See this example:
> http://code.google.com/p/google-refine/wiki/FetchingURLsFromWebServices
> (yahoo uses JSON too:  http://developer.yahoo.com/javascript/json.html)
>
> My opinion is that it'd be useful to have a JSON (or feed-XML) parsing program for Stata, but as I've started a couple of these types of programs, my limited understanding of  JSON had me kludging together a site-specific program that would likely fail for other websites'/APIs with different query structures.
>
> To answer your questions:  I don't think it's been done, but I think it's possible (or at least it is via the [-copy-, -infile-, parse] approach I've taken  -- I have no idea about using Mata for something like this).  If you do get something written, I'd have an interest in using it -- I like to have all my workflow in one program rather than having to go to something external (like Google Refine) that breaks up the workflow.   The Worldbank API page explains their query structure very clearly, so it might be a good place to start : http://data.worldbank.org/node/11
>
> A final tip:  you can now get most of Google's Data APIs in JSON instead of XML by using the 'alt=json' parameter , see this link for examples: http://code.google.com/apis/gdata/samples.html
>
>
> - Eric
>
> __
> Eric A. Booth
> Public Policy Research Institute
> Texas A&M University
> ebooth@ppri.tamu.edu
> Office: +979.845.6754
>
>
>
> On Dec 19, 2010, at 6:40 PM, Andrew Dyck wrote:
>
>> Hi Stata users,
>>
>> I'm interested in using Stata to access data from web APIs such as
>> data.worldbank.org or Twiter. I use 64-bit Stata MP for Windows and
>> Linux.
>>
>> I'm aware of the -net-, -copy- and -webuse- commands, however, my
>> searches so far haven't uncovered any tools for reading in data
>> directly from an API. At this time it seems that this may not be
>> possible in Stata without using an external python or perl script.
>> However, if you have any experience in using JSON with Stata I'd be
>> very happy to hear from you.
>>
>> I have begun research into writing a native Stata module (using mata)
>> for reading JSON data from a web API but I thought I'd tap the
>> Statalist knowledge bank first to:
>>
>> 1) stop me early if it's just not possible or already done;
>> 2) suggest any strategies for how to go about this; and
>> 3) see if there is any interest in such a feature or if users are
>> already using some other methods that would work better than something
>> native to Stata.
>>
>> Any help, advice or links to more info would be greatly appreciated.
>>
>> Thanks,
>> Andrew
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/statalist/faq
>> *   http://www.ats.ucla.edu/stat/stata/
>
>
>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/