tornado.httpclient
— Asynchronous HTTP client¶
Blocking and non-blocking HTTP client interfaces.
This module defines a common interface shared by two implementations,
simple_httpclient
and curl_httpclient
. Applications may either
instantiate their chosen implementation class directly or use the
AsyncHTTPClient
class from this module, which selects an implementation
that can be overridden with the AsyncHTTPClient.configure
method.
The default implementation is simple_httpclient
, and this is expected
to be suitable for most users’ needs. However, some applications may wish
to switch to curl_httpclient
for reasons such as the following:
curl_httpclient
has some features not found insimple_httpclient
, including support for HTTP proxies and the ability to use a specified network interface.curl_httpclient
is more likely to be compatible with sites that are not-quite-compliant with the HTTP spec, or sites that use little-exercised features of HTTP.curl_httpclient
is faster.curl_httpclient
was the default prior to Tornado 2.0.
Note that if you are using curl_httpclient
, it is highly recommended that
you use a recent version of libcurl
and pycurl
. Currently the minimum
supported version is 7.18.2, and the recommended version is 7.21.1 or newer.
HTTP client interfaces¶
-
class
tornado.httpclient.
HTTPClient
(async_client_class=None, **kwargs)[source]¶ A blocking HTTP client.
This interface is provided for convenience and testing; most applications that are running an IOLoop will want to use
AsyncHTTPClient
instead. Typical usage looks like this:http_client = httpclient.HTTPClient() try: response = http_client.fetch("http://www.google.com/") print response.body except httpclient.HTTPError as e: print "Error:", e http_client.close()
-
fetch
(request, **kwargs)[source]¶ Executes a request, returning an
HTTPResponse
.The request may be either a string URL or an
HTTPRequest
object. If it is a string, we construct anHTTPRequest
using any additional kwargs:HTTPRequest(request, **kwargs)
If an error occurs during the fetch, we raise an
HTTPError
.
-
-
class
tornado.httpclient.
AsyncHTTPClient
[source]¶ An non-blocking HTTP client.
Example usage:
def handle_request(response): if response.error: print "Error:", response.error else: print response.body http_client = AsyncHTTPClient() http_client.fetch("http://www.google.com/", handle_request)
The constructor for this class is magic in several respects: It actually creates an instance of an implementation-specific subclass, and instances are reused as a kind of pseudo-singleton (one per
IOLoop
). The keyword argumentforce_instance=True
can be used to suppress this singleton behavior. Constructor arguments other thanio_loop
andforce_instance
are deprecated. The implementation subclass as well as arguments to its constructor can be set with the static methodconfigure()
-
close
()[source]¶ Destroys this HTTP client, freeing any file descriptors used. Not needed in normal use, but may be helpful in unittests that create and destroy http clients. No other methods may be called on the
AsyncHTTPClient
afterclose()
.
-
fetch
(request, callback=None, **kwargs)[source]¶ Executes a request, asynchronously returning an
HTTPResponse
.The request may be either a string URL or an
HTTPRequest
object. If it is a string, we construct anHTTPRequest
using any additional kwargs:HTTPRequest(request, **kwargs)
This method returns a
Future
whose result is anHTTPResponse
. TheFuture
wil raise anHTTPError
if the request returned a non-200 response code.If a
callback
is given, it will be invoked with theHTTPResponse
. In the callback interface,HTTPError
is not automatically raised. Instead, you must check the response’serror
attribute or call itsrethrow
method.
-
classmethod
configure
(impl, **kwargs)[source]¶ Configures the
AsyncHTTPClient
subclass to use.AsyncHTTPClient()
actually creates an instance of a subclass. This method may be called with either a class object or the fully-qualified name of such a class (orNone
to use the default,SimpleAsyncHTTPClient
)If additional keyword arguments are given, they will be passed to the constructor of each subclass instance created. The keyword argument
max_clients
determines the maximum number of simultaneousfetch()
operations that can execute in parallel on eachIOLoop
. Additional arguments may be supported depending on the implementation class in use.Example:
AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")
-
Request objects¶
-
class
tornado.httpclient.
HTTPRequest
(url, method='GET', headers=None, body=None, auth_username=None, auth_password=None, auth_mode=None, connect_timeout=None, request_timeout=None, if_modified_since=None, follow_redirects=None, max_redirects=None, user_agent=None, use_gzip=None, network_interface=None, streaming_callback=None, header_callback=None, prepare_curl_callback=None, proxy_host=None, proxy_port=None, proxy_username=None, proxy_password=None, allow_nonstandard_methods=None, validate_cert=None, ca_certs=None, allow_ipv6=None, client_key=None, client_cert=None)[source]¶ HTTP client request object.
All parameters except
url
are optional.Parameters: - url (string) – URL to fetch
- method (string) – HTTP method, e.g. “GET” or “POST”
- headers (
HTTPHeaders
ordict
) – Additional HTTP headers to pass on the request - body – HTTP body to pass on the request
- auth_username (string) – Username for HTTP authentication
- auth_password (string) – Password for HTTP authentication
- auth_mode (string) – Authentication mode; default is “basic”.
Allowed values are implementation-defined;
curl_httpclient
supports “basic” and “digest”;simple_httpclient
only supports “basic” - connect_timeout (float) – Timeout for initial connection in seconds
- request_timeout (float) – Timeout for entire request in seconds
- if_modified_since (
datetime
orfloat
) – Timestamp forIf-Modified-Since
header - follow_redirects (bool) – Should redirects be followed automatically or return the 3xx response?
- max_redirects (int) – Limit for
follow_redirects
- user_agent (string) – String to send as
User-Agent
header - use_gzip (bool) – Request gzip encoding from the server
- network_interface (string) – Network interface to use for request
- streaming_callback (callable) – If set,
streaming_callback
will be run with each chunk of data as it is received, andHTTPResponse.body
andHTTPResponse.buffer
will be empty in the final response. - header_callback (callable) – If set,
header_callback
will be run with each header line as it is received (including the first line, e.g.HTTP/1.0 200 OK\r\n
, and a final line containing only\r\n
. All lines include the trailing newline characters).HTTPResponse.headers
will be empty in the final response. This is most useful in conjunction withstreaming_callback
, because it’s the only way to get access to header data while the request is in progress. - prepare_curl_callback (callable) – If set, will be called with
a
pycurl.Curl
object to allow the application to make additionalsetopt
calls. - proxy_host (string) – HTTP proxy hostname. To use proxies,
proxy_host
andproxy_port
must be set;proxy_username
andproxy_pass
are optional. Proxies are currently only supported withcurl_httpclient
. - proxy_port (int) – HTTP proxy port
- proxy_username (string) – HTTP proxy username
- proxy_password (string) – HTTP proxy password
- allow_nonstandard_methods (bool) – Allow unknown values for
method
argument? - validate_cert (bool) – For HTTPS requests, validate the server’s certificate?
- ca_certs (string) – filename of CA certificates in PEM format,
or None to use defaults. Note that in
curl_httpclient
, if any request uses a customca_certs
file, they all must (they don’t have to all use the sameca_certs
, but it’s not possible to mix requests withca_certs
and requests that use the defaults. - allow_ipv6 (bool) – Use IPv6 when available? Default is false in
simple_httpclient
and true incurl_httpclient
- client_key (string) – Filename for client SSL key, if any
- client_cert (string) – Filename for client SSL certificate, if any
New in version 3.1: The
auth_mode
argument.
Response objects¶
-
class
tornado.httpclient.
HTTPResponse
(request, code, headers=None, buffer=None, effective_url=None, error=None, request_time=None, time_info=None, reason=None)[source]¶ HTTP Response object.
Attributes:
- request: HTTPRequest object
- code: numeric HTTP status code, e.g. 200 or 404
- reason: human-readable reason phrase describing the status code (with curl_httpclient, this is a default value rather than the server’s actual response)
- headers:
tornado.httputil.HTTPHeaders
object - buffer:
cStringIO
object for response body - body: response body as string (created on demand from
self.buffer
) - error: Exception object, if any
- request_time: seconds from request start to finish
- time_info: dictionary of diagnostic timing information from the request.
Available data are subject to change, but currently uses timings
available from http://curl.haxx.se/libcurl/c/curl_easy_getinfo.html,
plus
queue
, which is the delay (if any) introduced by waiting for a slot underAsyncHTTPClient
‘smax_clients
setting.
Exceptions¶
-
exception
tornado.httpclient.
HTTPError
(code, message=None, response=None)[source]¶ Exception thrown for an unsuccessful HTTP request.
Attributes:
code
- HTTP error integer error code, e.g. 404. Error code 599 is used when no HTTP response was received, e.g. for a timeout.response
-HTTPResponse
object, if any.
Note that if
follow_redirects
is False, redirects become HTTPErrors, and you can look aterror.response.headers['Location']
to see the destination of the redirect.
Command-line interface¶
This module provides a simple command-line interface to fetch a url using Tornado’s HTTP client. Example usage:
# Fetch the url and print its body
python -m tornado.httpclient http://www.google.com
# Just print the headers
python -m tornado.httpclient --print_headers --print_body=false http://www.google.com