Curl (acronym for "Client for url") is a powerful command line tool and library for transferring files between two hosts. It supports a variety of protocols, including HTTP, SMTP, FTP, and POP3.
A few things we can do with the curl command include:
- download files
- upload files
- testing services and APIs
- setting up custom headers in HTTP requests
- debug network connections
The curl command can be used directly from the command line. This command can be invoked manually or from scripts written in bash, python or any other language. Curl is also available as an api in most programming languages like php, python etc.
If you want to use the curl api in your programming language of choice, then you should look up the api documentation of that particular language. In this article we shall just take a quick look at how to use the curl command from the terminal in linux.
It should be noted that command line curl is also available for Windows 10/11. Windows 10/11 have curl pre-installed provided directly by microsoft.
When i run the command with the --version on Windows 10 flag it shows details like version and supported protocols.
C:\Users\Silver>curl --version curl 7.83.1 (Windows) libcurl/7.83.1 Schannel Release-Date: 2022-05-13 Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp Features: AsynchDNS HSTS IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI UnixSockets C:\Users\Silver>
If you want the latest version of curl you can download it from their official website: https://curl.se/windows/
Alternatively you can also get curl by installing the cygwin platform.
On my ubuntu system the number of supported protocols is more compared to windows. On linux protocols like smtp, ldap, gopher along with their ssl versions are also available.
$ curl --version curl 7.85.0 (x86_64-pc-linux-gnu) libcurl/7.85.0 OpenSSL/3.0.5 zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.3 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.49.0 librtmp/2.3 Release-Date: 2022-08-31 Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd $
Examples and use cases of curl command
Now lets see how to use the curl command. The commands have been specifically tested on linux, but should work pretty much the same way on Windows as well.
1. Install curl if it's not already installed.
In most of the latest Linux OS versions, the curl command comes out of the box, but we may have to install it in some cases.
For RHEL / CentOS
$ sudo yum install curl
For Debian/Ubuntu based distros
$ sudo apt install curl
2. Check the installed version of curl.
This also helps to verify the installation and make sure that curl is working properly.
$ curl --version curl 7.68.0 (x86_64-pc-linux-gnu) ...
Basic syntax of the curl command
$ curl [options] [URL]
3. Perform HTTP requests with curl command.
By executing the below command, you should get the raw index.html or index.php (in most cases) of the subject URL.
$ curl https://example.com
<title>Example Domain</title>
. . .
. . .
. . .
4. Follow redirects.
curl command will not follow redirects by default. Hence, we need to specifically mention that we want the curl command to follow redirects.
Below is an example highlighting when we use curl with redirect and when we use curl without redirect.
$ curl google.com <TITLE>301 Moved</TITLE> <H1>301 Moved</H1> The document has moved <A HREF="http://www.google.com/">here</A>.
curl -L https://google.com
<title>Google</title>(function(){window.google={kEI:'YSosZK3kMreG0PEP6MS06AU',kEXPI:'0,18167,1341242,6058,207,4804,2316,383,246,5,1129120,1197746,303216,77529,16114,28684,22431,1361,12320,17579,4998,13228,3847,36218,2226,2872,2891,4139,8221,50059,10631,2614,13142,3,346,230,1014,1,16916,2652,4,1528,2304,42127,13658,21223,5785,2572,4094,7596,1,11943,27099,2,3110,2,16737,23024,5679,1021,31121,4568,6256,23422,1251,5835,14968,4332,7484,445,2,2,1,23827,10960,7381,2,15968,872,19634,7,1922,9779,21391,14763,2523,3782,2007,18191,17624,2513,14,82,9800,10406,1622,1749,29,12,4965,14891,6375,2106,991,3030,427,5684,141.
.
.
[output shortened]
Follow redirects
You can also use the curl command with -i -L both options, which will output the full HTTP response, including the redirects.
curl -i -L https://google.com
5. Download a file with curl
Download by specifying the download file name with curl the -o option.
$ curl -o ~/myfile https://speed.hetzner.de/100MB.bin % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 5 100M 5 5439k 0 0 298k 0 0:05:42 0:00:18 0:05:24 387k $ ls -ltr total 102656 -rw-r--r-- 1 ubuntu ubuntu 104857600 Feb 21 10:14 myfile
If you prefer to keep the original name as it is for the file you download, you can use -O option. However, this could replace the files you already have, hence caution is advised.
$ curl -O https://speed.hetzner.de/100MB.bin % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 2 100M 2 2511k 0 0 50378 0 0:34:41 0:00:51 0:33:50 470k
When downloading a file with curl, it shows a download meter with additional details. If you prefer a progress bar instead of the download meter, you can use -# as an option. Also, you could use -silent to disable everything.
6. Download / Upload files using FTP, SFTP, SCP protocols.
In most cases (also the best practice) the FTP, SFTP, SCP protocols will be protected by passwords. Hence, we will have to provide authentication credentials when using the curl command.
$ curl -u demo:password ftp://test.rebex.net 10-19-20 03:19PM pub 12-17-21 11:58AM 405 readme.txt
Download files with SSH Key using SFTP protocol
$ curl -u username: --key ~/path-to-private-key -O sftp://sftp.example.com/folder/filename
Resume paused or broken downloads
The -C option can be used to resume the downloads which have been stopped due to many reasons.
$ curl -C - -O ftp://ftp.example.com/folder/filename
Rate limiting
Rate limiting comes in handy when you don't want to use all the bandwidth you have for the curl download or upload. You can specify how much data rate you would like to allow for the download.
$ curl --limit-rate 1000K -O ftp://ftp.example.com/folder/filename
This limits the download speed to 1000K per second.
The given speed is measured in bytes/second, unless a suffix is appended. Appending 'k' or 'K' will count the number as kilobytes, 'm' or 'M' makes it megabytes, while 'g' or 'G' makes it gigabytes. Examples: 100K, 2m and 3G.
Upload files to FTP/SFTP server
the -T option is used when uploading a file to a destination server.
$ curl -u {username}:{password} -T {filename} {FTP_Location}
If you want to append an already existing FTP file, you can use the -a or –append option.
7. How to use the curl command through a proxy server.
A proxy server is used when someone wants to route the traffic through another endpoint instead of directly communicating with the destination server. There can be few reasons to use a proxy server in an organization.
- Accessing blocked content
- Enhancing security
- Improving performance
In short, it's useful to use a proxy server with curl command to access restricted content, for enhanced security and to improve performance.
$ curl -x [proxy_name]:[port] [URL...]
Additionally you can use the below syntax to authenticate to a proxy server where it's required.
$ curl -u [user]:[password] -x [proxy_name]:[port] [URL...]
8. How to change the user agent
In some cases, the remote web server may refuse HTTP requests from the curl command. To avoid this, we can simulate a browser and create an HTTP request as if it were coming from a browser. We may accomplish this by using the -A option in conjunction with the curl command.
curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3" https://example.com/scrape-data
9. How to send a cookie with curl
Cookies are a block of data that stores session information, user preferences, etc. However, they can also be used to authenticate a user to a server. When making HTTP requests, we can add '-b' or '--cookie' option.
You can use the below syntax to send a cookie with a curl command.
curl -b "cookie_name=cookie_value" https://example.com
curl -L -b "oraclelicense=a" -O http://download.oracle.com/otn-pub/java/jdk/10.0.2+13/19aef61b38124481863b1413dce1855f/jdk-10.0.2_linux-x64_bin.rpm
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 545 100 545 0 0 587 0 --:--:-- --:--:-- --:--:-- 0
100 7073 100 7073 0 0 2322 0 0:00:03 0:00:03 --:--:-- 8841
10. How to use curl to test web services, including verifying HTTP response codes, testing timeouts.
There are situations where the server can't reach HTTP destinations or is unable to download packages through package managers such as yum and apt. In a situation like this, the most common practice is to do a ping test to an outside IP address and then to do a ping test to an outside DNS to make sure the traffic is reaching both IPs and DNS as well.
But there are special cases, where even when you can reach outside via IP & DNS, you might still not be able to browse the internet. The most common problem for this is the firewall restrictions on ports 80 or 443, where the firewalls of the organization restrict web browsing for the source server/computer.
To troubleshoot this, we can use the telnet command. But most of the time, the telnet command is not installed by default. Since you don't have the internet at the moment, you'll find it difficult to install it as a fresh package as well. curl command comes in handy in a situation like this because it can check the connectivity with HTTP HTTPS ports.
$ curl -v https://www.google.com
With this method, you can analyze the response and see if there are any issues with the network connectivity using curl.
In a situation where you have hosted a web server such as NGINX or Apache in one of the servers in the local network, and the server doesn't have a GUI to check the HTTP response from a browser window, you can simply use the curl command mentioning the localhost as the destination server. You will be able to get an HTTP response with headers.
9. How to use curl with SSL and TLS connections.
curl is a robust tool. It can handle many secure protocols, including HTTPS which protects the data that you transfer with SSL/TSL encryption. From this section, you can get to know the steps to use curl with SSL and TLS connection. This will also guide you on how to add certificates, private keys, and CA certificates with the curl command.
Specify SSL/TLS Version
By default, curl is using the most recent version of the SSL/TLS present on the server. In a situation where you don't want to use the latest version of SSL/TLS on the server, you can specify the version you need using --tlsv1.0, --tlsv1.1, or --tlsv1.2 options. As an example, if you would like to use TLS 1.2, you may follow the below command to achieve the same.
$ curl --tlsv1.2 https://example.com
SSL/TLS certificate verification
curl is verifying the SSL/TLS certificate provided by the server to validate if it's a valid certificate and if it's issued by a trusted certificate authority (CA), by default. Sometimes you might want to deactivate this manually when you don't have a valid SSL/TLS signed by a trusted CA. This can happen if someone installed a Self Signed Certificate to their web servers. But this method is not recommended because it can expose your connection to a man-in-the-middle attack.
To deactivate the SSL/TLS verification, you may use --insecure or -k options
$ curl -k https://example.com
This will help you connect to the server, bypassing the SSL/TLS verification. But keep in mind that you must use this within trusted environments only.
If you have the certificate bundle of private CA, you can use it with the curl command. This will allow you to access servers with web servers which don't have a SSL/TLS certificate signed by a global CA with curl command.
$ curl --cacert /etc/ssl/certs/ca-bundle.crt https://example.com
In some configurations, the server is requesting a client-side SSL/TLS as well for authentication. In this case, you need to provide the client-side certificate and the private key in order to make a successful connection with the server. Assuming your client-side certificate file is client.crt and the private key is client.key, you can use the below command to authenticate to this type of a server.
$ curl --cert /path/to/certs/client.crt --key /path/to/certs/client.key https://example.com
If your SSL/TLS certificate is encrypted with a passphrase, you can use --pass option to provide the passphrase with the curl command.
$ curl --cert /path/to/certs/client.crt --key /path/to/certs/client.key --pass password https://example.com
If you have a SSL/TLS certificate with a PKCS#12 file, then you can use this with the curl command as follows.
$ curl --cert-type P12 --cert /path/to/certs/client.p12 https://example.com
Conclusion
On linux there are actually many tools for performing http requests from the command line and curl is one of them. Tools like wget can also be used for this task, but those are available only as stand-alone applications.
Curl on the other hand provides programming apis which makes it a widely used tool.
Curl can be used to automate content transfer between hosts through scripting. This is useful in scenarios when you need to automatically upload/download files across servers.
Let us know how you are using curl on your linux machines and servers.
