st: re: r(601) "server says file temporarily redirected to [URL]"

[Christopher Baum <baum@bc.edu>]

<>
Sergiy says

I would like to download a file from the web with Stata's -copy-  
command. The server that provides the data uses redirection. After  
accessing a given URL1, the server issues a special code to the  
browser (I presume it is 307) and the new location (URL2). This  
happens automatically in the browser and no special actions from the  
user are required. A typical use would be to provide a link in a form  
of "www.foobar.com/recent_event" and redirect to the page that  
describes the most recent event, which obviously changes with time, so  
at one point redirection will be to "www.foobar.com/events/alpha.htm"  
and in another to "www.foobar.com/events/beta.htm" (URL is made up).  
The following happens in Stata:
. copy [URL1] [LocalFile]
file [URL1] not found
server says file temporarily redirected to [URL2]
1. The file as it is mentioned in URL2 exists, but the data provider  
insists to always access URL1 and follow the redirection route  
received in response to access the data (otherwise I might download an  
outdated version). However I don't see a way to programmatically  
access the value of URL2 (which is correctly displayed in the error  
message). Is there anything that can be done to it, except logging to  
a text file and then parsing it for the new URL?
2. Stata seems to recode the URL2 to lowercase, which on case- 
sensitive servers completely eliminates a possibility to download the  
file (my case), even if I parse the log and extract URL2 from Stata's  
output. Is there a way to overcome this?
3. Perhaps there is a way to tell Stata to follow redirection  
instructions received from the server?

It sounds like it would just be a whole lot simpler to use a shell  
command from within Stata:
!curl http://www.foobar.com/recent_event > localfile.name
or
!wget www.foobar.com/recent_event
whatever is most convenient for your operating system.
Stata has a habit, when acting as a web browser, of not being fond of  
redirection. That is why the -net- commands cannot refer to a  
different server than that which you have -net from-. That is a  
sensible security provision.  Stata implements certain elements of the  
HTTP protocol just as your web browser does, but I don't think  
StataCorp makes any claim that it handles everything defined in that  
protocol. The kind of thing you're running into here might be a bit  
farther than Stata will go. Why not just rely on a utility such as  
wget or curl to do the job?

Kit Baum, Boston College Economics and DIW Berlin
http://ideas.repec.org/e/pba1.html
An Introduction to Modern Econometrics Using Stata:
http://www.stata-press.com/books/imeus.html

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/