cURL is a versatile, multi-platform command-line tool and system library for networking, which supports a wide variety of network protocols and options. Although powerful, it can also be fickle to get working properly in certain cases. Today I will discuss some quirks that I have encountered while using cURL, in hopes of saving others the time it took for me to resolve them.
Issue: Following a successfully-posted HTTP request by the client, the remote host resets the connection before it can send back a successful response.
This was a major issue I encountered while attempting to fetch some XML data via SOAP request from an NTLM authenticated server. This request looked something like the following in cURL:
curl -X POST -d @soap.xml -H 'Content-type: text/xml;charset="utf-8"' -H 'Accept: text/xml' -H 'Cache-Control: no-cache' -H 'Pragma: no-cache' -H 'SOAPAction: "http://tempuri.org/ProductService/find"' -H 'Content-length: 623' -H 'Connection: Keep-Alive' --anyauth -u myUserName:myPassword http://someurl.org/
The server would successfully process the SOAP message on the backend, but right at the moment when it should have sent back the XML response, it would instead terminate the connection with a TCP RST packet.
My cURL client would then exit with the following error message:
curl: (56) Recv failure: Connection reset by peer
Such a message is fairly ambiguous as to the exact source of the problem, which can often lie somewhere in the network or firewall configuration. In my case, however, the issue lied within cURL itself:
/* We got no data, we attempted to re-use a connection and yet we want a "body". This might happen if the connection was left alive when we were done using it before, but that was closed when we wanted to read from it again. Bad luck. Retry the same request on a fresh connect! */ infof(conn->data, "Connection died, retrying a fresh connect\n"); *url = strdup(conn->data->change.url); if(!*url) return CURLE_OUT_OF_MEMORY; conn->bits.close = TRUE; /* close this connection */ conn->bits.retry = TRUE; /* mark this as a connection we're about to retry. Marking it this way should prevent i.e HTTP transfers to return error just because nothing has been transferred! */
Here cURL tried to reuse the NTLM connection, but the server took too long to respond and so cURL assumed that the connection was closed. To workaround this issue, I found that the solution is to use chunked transfer encoding, like so:
curl -X POST -d @soap.xml -H 'Content-type: text/xml;charset="utf-8"' -H 'Accept: text/xml' -H 'Cache-Control: no-cache' -H 'Pragma: no-cache' -H 'SOAPAction: "http://tempuri.org/ProductService/find"' -H 'Transfer-Encoding: chunked' --anyauth -u myUserName:myPassword http://someurl.org/
Issue: A request which uses authentication fails with “401 Unauthorized”, even though the credentials are valid.
Another problem I ran into with cURL was when trying to use NTLM authentication in an HTTP request:
curl -X POST -d @soap.xml -H 'Content-type: text/xml;charset="utf-8"' -H 'Accept: text/xml' -H 'Cache-Control: no-cache' -H 'Pragma: no-cache' -H 'SOAPAction: "http://tempuri.org/ProductService/find"' -H 'Transfer-Encoding: chunked' -H 'Connection: Keep-Alive' --anyauth -u myUserName:myPassword http://someurl.org/
Although this request worked without issue on my MacBook, it failed when I tried it from my server. I would receive continuous “401 Unauthorized” responses from the host, until the connection was closed.
To fix this, I had to explicitly tell cURL to use NTLM for its authentication method, like so:
curl -X POST -d @soap.xml -H 'Content-type: text/xml;charset="utf-8"' -H 'Accept: text/xml' -H 'Cache-Control: no-cache' -H 'Pragma: no-cache' -H 'SOAPAction: "http://tempuri.org/ProductService/find"' -H 'Transfer-Encoding: chunked' -H 'Connection: Keep-Alive' --ntlm -u myUserName:myPassword http://someurl.org/
A note on different versions of cURL:
I have had issues with versions of cURL < 7.30.0, such as the fixes described above not working at all.
Thus, if you are using cURL as a library that is integrated with a scripting language such as PHP, be sure to note which version of the library your scripting environment uses. PHP in particular has some cURL-related bugs that should be kept in mind when troubleshooting a script.
You can check the version of cURL that PHP is using via the command:
php --info | grep cURL
This should give you output which looks something like this:
cURL support => enabled
cURL Information => 7.30.0
This information can also be found in the output of PHP’s phpinfo() function.
Installing PHP with a working version of cURL on OS X Mountian Lion
Mac OS X 10.8 Mountain Lion Server includes PHP 5.3, which utilizes libcurl v7.21. As mentioned in the previous section, this version of cURL is problematic. I will provide brief instructions on how to install the Macports PHP 5.4 with libcurl 7.36. The Macports PHP is installed side-by-side with OS X server’s default PHP.
1.) If Macports is not installed, run the installer, available from http://www.macports.org/install.php
2.) Once Macports has been installed, open a Terminal window and run this command to install PHP 5.4, plus its necessary libraries:
sudo port install php54 php54-apache2handler php54-openssl php54-curl php54-pear pear-HTTP_Request2
3.) Next, setup Apache to use the newly-installed PHP library, copy over the new PHP extension (mod_php54.so), backup the original PHP extension (libphp5.so), and rename the new extension to the old extension’s name:
sudo /opt/local/apache2/bin/apxs -a -e -n php5 mod_php54.so
sudo cp mod_php54.so /usr/libexec/apache2
sudo mv libphp5.so libphp5.so.old
sudo mv mod_php54.so libphp5.so
4.) Lastly, PHP needs to be configured with a timezone in order to function correctly. The PHP configuration file must be created and then modified accordingly:
- Copy /opt/local/etc/php54/php.ini-production to /opt/local/etc/php54/php.ini
- Edit the php.ini file and set the “date.timezone” value to your desired timezone. Refer to PHP’s List of Supported Timezones for more details.
Since many applications and libraries – PHP or otherwise – are built on top of cURL, it is useful to know at least the basics of how this software functions. When troubleshooting or debugging a program, a seemingly higher-level network error could turn out to be an issue with the underlying cURL library, and so it is often helpful to test out specific network requests first on the command-line in order to eliminate other confounding factors.
I hope this post proved useful and informative, and if you have any other cURL caveats and their solutions that you want to discuss, go ahead and share your knowledge in the comments section below.