Name
DownloadFile -- download file via HTTP, FTP or other protocol (V5.0)
Synopsis
data$, count = DownloadFile(url$[, options, func, userdata])
Function
This command allows you to conveniently download a file from a network server. By default, HTTP and FTP servers are supported but Hollywood plugins may provide support for additional protocols. You have to pass the URL of the file in the url$ argument. DownloadFile() will then download the file and return it as a string. Storing binary data inside strings is possible because Hollywood strings are not limited to printable characters. Instead, they can also contain control characters and the NULL character. The second return value indicates the size of the downloaded file in bytes.

Alternatively, you can also set the File tag in the optional table argument named options to a filename. In that case, the downloaded file won't be returned as a string but it will be saved as the file specified in the File tag. This is recommended for bigger files because strings are obviously stored in memory so downloading a large file to a string is generally not a good idea because it will require lots of memory.

The URL passed in url$ must begin with a protocol prefix like http:// or ftp://, and it must not contain any escaped characters. Escaping will be done by DownloadFile() so make sure that you pass only unescaped URLs, e.g. passing "http://www.mysite.net/cool%20file.html" will not work. You must specify an URL without escaped characters, so the correct version would be: "http://www.mysite.net/cool file.html". If you want to pass a URL that has already been escaped, you have to set the Encoded tag to True (see below). In that case, DownloadFile() won't do any further escaping on your URL.

DownloadFile() also supports authentification for the HTTP and FTP protocols. In that case, username and password have to be passed after the protocol identifier in the form username:password, followed by an @ character and the server. Here is an example for user "joe" and password "secret": http://joe:secret@www.test.net/private/files.lha. Note that HTTP authentification support was not available before Hollywood 6.0. When downloading from an FTP server, "anonymous" is used as the default username and "anonymous@anonymous.org" as the default password. If you want to use a different login account, you have to pass the username/password combination in the URL, for example: ftp://joe:secret@ftp.test.net/pub/files.lha.

The URL you pass to this function can also contain a port number. If you want to specify a port number, you have to put it behind the host name and separate it using a colon. Here is an example for using port 1234: http://www.test.com:1234/test/image.jpg. If no port is specified, DownloadFile() will use port 80 for HTTP servers and port 21 for FTP servers.

The second argument can be used to specify further options for the download operation. It is a table that recognizes the following tags:

File:
If you specify a filename in this table tag, DownloadFile() will stream the downloaded data directly to this file instead of returning it as a string. This is useful for very large files on the one hand, but it is also useful for other files because it saves you the hassle of having to save the string data manually to a file and set it to Nil afterwards. Therefore, if you are going to save the downloaded string to a file anyway, it is more efficient to use this tag. If you decide not to use this tag, please also read below for some important information when downloading files to strings.

TransferMode:
This tag is only supported when downloading files from an FTP source. In that case, you can use this tag to specify whether DownloadFile() should transfer the file in ASCII or in binary mode. For ASCII mode, specify #FTPASCII here. For binary mode, use #FTPBINARY. The default transfer mode is #FTPBINARY.

Proxy:
This tag is only supported when downloading files from an HTTP source. In that case, you can specify a server here that should act as a proxy server for the connection.

Fail404:
This tag specifies whether or not DownloadFile() should fail with a "file not found" error when you pass a URL that points to a non-existent file. Normally, when you request a non-existent file, HTTP servers will generate a special HTML page with a "404 - file not found" error, and send that to you instead. So you will always be getting a file even if you are requesting a non-existent file. If you do not want this behaviour, set this tag to True. In that case, DownloadFile() will fail when requesting an invalid file and you will not get any 404 error page. By default, this tag is set to False which means that an error page is generated. This tag is only supported for downloading files from an HTTP source.

SilentFail:
If you set this tag to True, DownloadFile() will never throw an error but simply exit silently and return an error message in the first return value, and -1 in the second return value to indicate that an error has happened. If it is set to False, DownloadFile() will throw a system error for all errors that occur. Defaults to False.

Redirect:
Specifies whether or not the web server is allowed to redirect you to a new URL. This defaults to True which means that redirection is allowed. This tag is only supported when downloading files from an HTTP source.

Post:
This tag is only supported when working with a HTTP server. If this tag is specified, DownloadFile() will send a POST request to the HTTP server instead of a GET request. A POST request has the advantage that you can attach additional data to your request. Thus, it is often used for submitting the contents of web forms, or for uploading files via HTTP. The data that shall be attached to the POST request must be specified in this tag as a string. You can set the type of the data by using the PostType tag (see below).

PostType:
This tag is only handled when the Post tag was also specified. If that is the case, PostType specifies the type of data inside the Post tag. The type must be passed as a MIME content type string. This tag defaults to "application/x-www-form-urlencoded" which is the MIME type used for submitting the contents of web forms to Perl (CGI) scripts.

UserAgent:
This tag allows you to change the user agent that DownloadFile() sends to the target server. This is useful with servers that refuse to cooperate with unknown user agents. By default, DownloadFile() will send "Hollywood" in the user agent field of HTTP requests. This tag is only supported for downloading files from an HTTP source. (V5.2)

CustomHeaders:
This tag allows you to specify a string of custom headers that should be sent to the HTTP server when making the request. This can be useful for some fine-tuned adjustments for some servers. Keep in mind that the individual header elements have to be terminated by a carriage return and a line feed. This tag is only supported when using the HTTP protocol. (V6.0)

Encoded:
Set this tag to True if the URL you passed to this function has already been correctly escaped. If this tag is set to True, DownloadFile() won't escape any characters. Instead, it expects you to pass a URL that has already been correctly escaped so that it can be directly used for server requests without any additional escaping. (V6.1)

Protocol:
This tag can be used to specify the Internet protocol that should be used when opening the connection. This can be one of the following special constants:

#IPV4:
Use Internet Protocol version 4 (IPv4).
#IPV6:
Use Internet Protocol version 6 (IPv6). Note that #IPV6 is currently unsupported on AmigaOS and compatible systems.
#IPAUTO:
Let the host operating system determine the Internet protocol to use.

This tag defaults to the default protocol type set using SetNetworkProtocol(). By default, this is #IPV4 due to historical and portability reasons. See SetNetworkProtocol for details. (V8.0)

Adapter:
This tag allows you to specify one or more network adapters that should be asked to establish the specified connection. This must be set to a string containing the name(s) of one or more adapter(s). Defaults to the adapter set using SetDefaultAdapter(). See Loaders and adapters for details. (V8.0)

SSL:
Set this tag to True to request a connection through TLS/SSL encryption. Note that setting this tag when using Hollywood's inbuilt network adapter doesn't have any effect because Hollywood's inbuilt network adapter doesn't support TLS/SSL connections. However, there might be a network adapter provided by a plugin that supports TLS/SSL and if you set this tag to True Hollywood will forward your wish to have a TLS/SSL connection to the network adapter provided by the plugin. Do note, though, that you normally don't have to set this tag in case the URL's scheme already indicates an SSL connection by using a prefix such as "https://" or "ftps://". (V8.0)

Async:
If this is set to True, DownloadFile() will operate in asynchronous mode. This means that it will return immediately, passing an asynchronous operation handle to you. You can then use this asynchronous operation handle to finish the operation by repeatedly calling ContinueAsyncOperation() until it returns True. This is very useful in case your script needs to do something else while the operation is in progress, e.g. displaying a status animation or something similar. By putting DownloadFile() into asynchronous mode, it is easily possible for your script to do something else while the operation is being processed. See ContinueAsyncOperation for details. Defaults to False. (V9.0)

Verbose:
This tag can be set to True to request detailed log information about the connection and the protocol interaction with the server. This is currently only used by Hollywood plugins so if you use Hollywood's internal network adapter, setting this tag to True has no effect. Plugins, however, may choose to provide extended connection information when this tag has been set to True. Defaults to False. (V9.0)

FileAdapter:
This tag is only used if the File tag is set as well. In that case, FileAdapter allows you to specify one or more file adapters that should be asked if they want to save the specified file. If you use this tag, you must set it to a string containing the name(s) of one or more adapter(s). Defaults to default. See Loaders and adapters for details. (V10.0)

UserTags:
This tag can be used to specify additional data that should be passed to file and network adapters. If you use this tag, you must set it to a table of key-value pairs that contain the additional data that should be passed to plugins. See User tags for details. (V10.0)

The optional parameter func can be used to pass a callback function which will be called from time to time by DownloadFile() so you can update a progress bar for example. The callback function you specify here will be called with a single argument: A table that contains more information. Here is an overview of the table fields that will be initialized before DownloadFile() runs your callback function:

Action:
#DOWNLOADFILE_STATUS

Count:
Contains the number of bytes that have already been downloaded.

Total:
Contains the size of the file being downloaded.

UserData:
Contains the value you passed in the userdata argument.

The callback function of type #DOWNLOADFILE_STATUS should normally return False. If it returns True, the download operation will be aborted.

Finally, there is a fourth optional argument called userdata. The value you specify here is passed to your callback function whenever it is called. This is useful if you want to avoid working with global variables. Using the userdata argument you can easily pass data to your callback function. You can specify a value of any type in userdata. Numbers, strings, tables, and even functions can be passed as user data.

If you are downloading to a string, you can use the StringToFile() shortcut function to convert the string returned by DownloadFile() to a file. Alternatively, you could use the DefineVirtualFileFromString() function to create a virtual file from a string source. This can be useful, for example, when you download an image file that you want to load into Hollywood using LoadBrush(). By using DefineVirtualFileFromString() you can load this file directly into Hollywood without having to save it to a temporary file first.

Important note: Make sure that you set the string returned by this function to Nil when you no longer need it. By setting the string to Nil, you signal to the Hollywood garbage collector that you no longer need this string and that its memory can be freed. This is especially important for large files. If you download a large file, save it to disk and do not set its string to Nil, you will waste a lot of memory. So make sure to be careful with strings returned by DownloadFile().

Inputs
url$
URL to file that shall be downloaded
options
optional: a table containing further options for this download
func
optional: a callback function that shall be called from time to time
userdata
optional: user defined data that should be passed to callback function
Results
data$
the data that was read from the network buffer or an empty string if the File tag was specified in the options table
count
number of bytes successfully transmitted
Example
DownloadFile("http://www.airsoftsoftwair.de/images/products/" ..
             "hollywood/47_shot1.jpg", {File = "47_shot1.jpg"})
The code above downloads the specified file and saves it as "47_shot1.jpg" to the current directory.


DownloadFile("http://www.<your server>.com/cgi-bin/formmailer.cgi",
             {Post = "sender=Hollywood&mail=me@hollywood-mal.de" ..
             "&message=Hello from Hollywood!"}))
The code above shows to invoke a CGI script using DownloadFile(). The data specified in the Post tag will be passed to the HTTP server using the POST method.


DownloadFile("http://www.hollywood-mal.com/index.html", {
             File = "index.html",
             CustomHeaders = "Accept-Encoding: gzip, deflate\r\n"})
The code above downloads the specified file and sends a custom header to tell the server that it can also send the file as a gzip or flate compressed file.


@REQUIRE "hurl"
...
DownloadFile("https://www.hollywood-mal.com/index.html", {
             File = "index.html", Adapter = "hurl"})
The code above downloads a file using the HTTPS protocol. Since Hollywood doesn't support SSL/TLS by default, this code uses the hURL plugin for the operation because hURL supports SSL/TLS. hURL is activated by passing hurl in the Adapter tag.

Show TOC