curl

File transfer tool that works from the command line using URL rules

Supplementary instructions

curl command is a file transfer tool that uses URL rules to work under the command line. It supports file upload and download, so it is a comprehensive transmission tool. However, according to tradition, curl is called a download tool. As a powerful tool, curl supports many protocols including HTTP, HTTPS, ftp, etc. It also supports features such as POST, cookies, authentication, downloading partial files from specified offsets, user agent strings, speed limits, file sizes, progress bars, etc. . To automate web page processing and data retrieval, curl can help.

grammar

curl(options)(parameters)

Options

-a --append # When uploading a file, append to the target file
-A --user-agent # Set the user agent to send to the server
-anyauth # can use "any" authentication method
-b --cookie # cookie string or file reading position
--basic # Use HTTP basic authentication
-B --use-ascii # Use ASCII/text transfer
-c --cookie-jar #Write cookies to this file after the operation is completed
-C --continue-at # Resume upload from breakpoint
-d --data # HTTP POST method to transmit data
--data-ascii # Post data in ascii format
--data-binary # Post data in binary format
--negotiate # Use HTTP authentication
--digest # Use digital authentication
--disable-eprt # Disable the use of EPRT or LPRT
--disable-epsv # Disable the use of EPSV
-D --dump-header # Write header information to the file
--egd-file # Set the EGD socket path for random data (SSL)
--tcp-nodelay # Use TCP\_NODELAY option
-e --referer # Source URL
-E --cert # Client certificate file and password (SSL)
--cert-type # Certificate file type (DER/PEM/ENG) (SSL)
--key #Private key file name (SSL)
--key-type # Private key file type (DER/PEM/ENG) (SSL)
--pass # Private key password (SSL)
--engine # Encryption engine to use (SSL). "--engine list" for list
--cacert # CA certificate (SSL)
--capath # CA directory (made using c\_rehash) to verify peer against (SSL)
--ciphers # SSL passwords
--compressed #Requires the returned format to be compressed (using deflate or gzip)
--connect-timeout # Set the maximum request time
--create-dirs # Create the directory hierarchy of the local directory
--crlf # Uploading converts LF into CRLF
-f --fail # Do not display http errors when connection fails
--ftp-create-dirs # If the remote directory does not exist, create the remote directory
--ftp-method \[multicwd/nocwd/singlecwd] # Control the use of CWD
--ftp-pasv # Use PASV/EPSV instead of port
--ftp-skip-pasv-ip # When using PASV, ignore this IP address
--ftp-ssl # Try to use SSL/TLS for ftp data transmission
--ftp-ssl-reqd # Require SSL/TLS for ftp data transmission
-F --form # Simulate http form submission data
--form-string # Simulate http form submission data
-g --globoff # Disable URL sequences and ranges using {} and \[]
-G --get # Send data in get mode
-H --header # Custom header information passed to the server
--ignore-content-length # The length of the ignored HTTP header information
-i --include #Include protocol header information when outputting
-I --head # Only display request header information
-j --junk-session-cookies # Read the file and ignore session cookies
--interface # Use the specified network interface/address
--krb4 # Use krb4 with the specified security level
-k --insecure # Allow SSL sites without certificates
-K --config #Read the specified configuration file
-l --list-only # List the file names in the ftp directory
--limit-rate # Set transfer speed
--local-port # Force the use of local port number
-m --max-time # Set the maximum transmission time
--max-redirs # Set the maximum number of directories to read
--max-filesize # Set the maximum total number of downloaded files
-M --manual # Display full manual
-n --netrc # Read username and password from netrc file
--netrc-optional # Use .netrc or URL to override -n
--ntlm # Use HTTP NTLM authentication
-N --no-buffer # Disable buffered output
-o --output #Write output to this file
-O --remote-name #Write output to this file, retaining the file name of the remote file
-p --proxytunnel # Use HTTP proxy
--proxy-anyauth # Select any proxy authentication method
--proxy-basic # Use basic authentication on the proxy
--proxy-digest # Use digital authentication on proxy
--proxy-ntlm # Use ntlm authentication on proxy
-P --ftp-port # Use port address instead of PASV
-q # As first argument, close .curlrc
-Q --quote # Before file transfer, send command to server
-r --range # Retrieve byte range from HTTP/1.1 or FTP server
--range-file # Read (SSL) random files
-R --remote-time # When generating files locally, keep the remote file time
--retry # Number of retries when there is a problem with transmission
--retry-delay # When there is a problem with the transmission, set the retry interval
--retry-max-time # Set the maximum retry time when there is a transmission problem
-s --silent # Silent mode. does not output anything
-S --show-error # Show errors
--socks4 # Use socks4 to proxy the given host and port
--socks5 # Use socks5 to proxy the given host and port
--stderr #
-t --telnet-option # Telnet option settings
--trace # Debug the specified file
--trace-ascii # Like --trace but no hex output
--trace-time # Add timestamp when tracing/detailed output
-T --upload-file # Upload file
--url <url> # The URL to use
-u --user # Set the server user and password
-U --proxy-user # Set proxy username and password
-w --write-out \[format] # What output is completed?
-x --proxy # Use HTTP proxy on given port
-X --request #Specify what command
-y --speed-time # The time required to give up the speed limit, the default is 30
-Y --speed-limit # Stop transmission speed limit, speed time

Example

Download Document

The curl command can be used to perform operations such as downloading, sending various HTTP requests, and specifying HTTP headers. If the system does not have curl, you can use yum install curl to install it, or you can download and install it. Curl outputs the downloaded file to stdout and the progress information to stderr. Use the --silent option to not display the progress information.

curl URL --silent

This command outputs the downloaded file to the terminal, and all downloaded data is written to stdout.

Use option -O to write the downloaded data to a file. The absolute address of the file must be used:

curl http://example.com/text.iso --silent -O

The option -o writes the download data to a file with the specified name and uses --progress to display a progress bar:

curl http://example.com/test.iso -o filename.iso --progress
######################################## 100.0%

Do not output error and progress information

The -s parameter will not output error and progress information.

curl -s https://www.example.com
# Once an error occurs in the above command, no error message will be displayed. If no errors occur, the operation results will be displayed normally.

If you want curl to not produce any output, you can use the following command.

curl -s -o /dev/null https://example.com

http

Curl can continue downloading from a specific file offset. It can download part of a file by specifying an offset:

curl URL/File -C offset

#The offset is an integer in bytes. If you want curl to automatically infer the correct resumption location, use -C -:
curl -C -URL

Use curl to set the reference page string

The reference page is a string located in the HTTP header. It is used to indicate the page from which the user arrived at the current page. If the user clicks on a link in web page A, the user will jump to web page B. The header of web page B The reference page string in this section contains the URL of web page A.

Use the --referer option to specify a referrer string:

curl --referer http://www.example.com http://example.com

Set user agent string with curl

Some websites will prompt that they can only be accessed using the IE browser. This is because these websites are set to check the user agent. You can use curl to set the user agent to IE, so that you can access it. Use the --user-agent or -A option:

curl URL --user-agent "Mozilla/5.0"
curl URL -A "Mozilla/5.0"

Other HTTP header information can also be sent using curl. Use -H "header information" to pass multiple header information, for example:

curl -H "Host:example.com" -H "accept-language:zh-cn" URL

curl bandwidth control and download quota

Use --limit-rate to limit curl’s download speed:

curl URL --limit-rate 50k

Use k (kilobytes) and m (megabytes) in the command to specify the download speed limit.

Use --max-filesize to specify the maximum file size that can be downloaded:

curl URL --max-filesize bytes

If the file size exceeds the limit, the command returns a non-zero exit code, or 0 if the command is normal.

curl --limit-rate 200k https://example.com
# The above command limits bandwidth to 200K bytes per second.

Use curl for authentication

Use the curl option -u to complete HTTP or FTP authentication. You can specify a password, or you can enter the password in subsequent operations without specifying a password:

curl -u user:pwd http://example.com
curl -u user http://example.com

Only print response header information

Pass -I or -head to print out only the HTTP header information:

[root@localhost text]# curl -I http://example.com
HTTP/1.1 200 OK
Content-Encoding: gzip
Accept-Ranges: bytes
Age: 275552
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Mon, 24 Apr 2023 14:39:36 GMT
Etag: "3147526947+gzip"
Expires: Mon, 01 May 2023 14:39:36 GMT
Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT
Server: ECS (sec/96EE)
X-Cache: HIT
Content-Length: 648

GET request

curl "http://www.example.com" # If the URL here points to a file or image, it can be downloaded directly to the local
curl -i "http://www.example.com" # Display all information
curl -l "http://www.example.com" # Display page content
curl -v "http://www.example.com" # Display the entire process of get request analysis

POST request

$ curl -d "param1=value1&param2=value2" "http://www.example.com/login"

$ curl -d'login=emma&password=123' -X POST https://example.com/login
# or
$ curl -d 'login=emma' -d 'password=123' -X POST https://example.com/login

The --data-urlencode parameter is equivalent to -d, sending the data body of the POST request. The difference is that the sent data will be automatically URL encoded.

curl --data-urlencode 'comment=hello world' https://example.com/login
# In the above code, there is a space between hello world and the data sent, which needs to be URL encoded.

Send text in local file

curl -d '@data.txt' https://example.com/upload
# Read the contents of the data.txt file and send it to the server as a data body.

POST request in JSON format

curl -l -H "Content-type: application/json" -X POST -d '{"phone":"13888888888","password":"test"}' http://example.com/apis/users. json

Send Cookie to server

Use the --cookie "COKKIES" option to specify cookies. Use semicolons to separate multiple cookies:

curl http://example.com --cookie "user=root;pass=123456"

To save cookies as a file, use the --cookie-jar option:

curl URL --cookie-jar cookie_file

The -b parameter is used to send Cookie to the server.

curl -b 'foo=bar' https://example.com
# The above command will generate a header Cookie: foo=bar, and send a Cookie named foo and value bar to the server.

curl -b 'foo1=bar' -b 'foo2=baz' https://example.com
# The above command sends two cookies.

```shell
curl -b cookies.txt https://www.example.com
# The above command reads the local file cookies.txt, which contains the cookies set by the server (see the -c parameter), and sends it to the server.

Cookie writes to a file

curl -c cookies.txt https://www.example.com
# The above command writes the cookies set by the server's HTTP response into the text file cookies.txt.

Source of request

The -e parameter is used to set the HTTP header Referer, indicating the source of the request.

curl -e 'https://example.com?q=example' https://www.example.com
# The above command sets the Referer header to https://example.com?q=example.

The -H parameter can achieve the same effect by directly adding the header Referer.

curl -H 'Referer: https://example.com?q=example' https://www.example.com

Upload binary file

The -F parameter is used to upload binary files to the server.

curl -F 'file=@photo.png' https://example.com/profile
# The above command will add the header Content-Type: multipart/form-data to the HTTP request, and then upload the file photo.png as the file field.

The -F parameter can specify the MIME type.

curl -F 'file=@photo.png;type=image/png' https://example.com/profile
# The above command specifies the MIME type as image/png, otherwise curl will set the MIME type to application/octet-stream.

The -F parameter can also specify a file name.

curl -F 'file=@photo.png;filename=me.png' https://example.com/profile
# In the above command, the original file name is photo.png, but the file name received by the server is me.png.

Set request header

The -H parameter adds headers to the HTTP request.

curl -H 'Accept-Language: en-US' https://example.com
# The above command adds the HTTP header Accept-Language: en-US.

curl -H 'Accept-Language: en-US' -H 'Secret-Message: xyzzy' https://example.com
# The above command adds two HTTP headers.

curl -d '{"login": "emma", "pass": "123"}' -H 'Content-Type: application/json' https://example.com/login
# The above command adds the HTTP request header Content-Type: application/json, and then uses the -d parameter to send JSON data.

Skip SSL detection

curl -k https://www.example.com
# The above command will not check whether the server's SSL certificate is correct.

Request to follow server's redirect

The -L parameter causes HTTP requests to follow server redirects. curl does not follow redirects by default.

curl -L -d 'tweet=hi' https://api.example.com/tweet

It is worth noting that this redirection method does not apply to redirections in the returned HTML, such as redirects that are not recognized by curl (this part is generated by curl -v -L <url>)

* Connected to example.com (*.*.*.*) port 80 (#0)
>GET/HTTP/1.1
> Host: example.com
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 200 OK
....
< Content-Type: text/html
<
<html>
<meta http-equiv="refresh" content="0;url=http://www.example.com/">
</html>

Debug Parameters

The -v parameter outputs the entire process of communication for debugging.

curl -v https://www.example.com
# The --trace parameter can also be used for debugging and will also output raw binary data.

curl --trace - https://www.example.com

Get the local external network IP

curl ipecho.net/plain

Use curl to test website loading speed

The command has a little-known option, -w, which prints the statistics of this request to the standard output after the request ends.

First, we define the formatting string that controls printing behavior. Create a new text file fmt.txt and fill in the following content:

\n
Response Time for: %{url_effective}\n\n
DNS Lookup Time:\t\t%{time_namelookup}s\n
Redirection Time:\t\t%{time_redirect}s\n
Connection Time:\t\t%{time_connect}s\n
App Connection Time:\t\t%{time_appconnect}s\n
Pre-transfer Time:\t\t%{time_pretransfer}s\n
Start-transfer Time:\t\t%{time_starttransfer}s\n\n
Total Time:\t\t\t%{time_total}s\n

Curl provides many substitution variables, which can be used in the format string in the form of %{var}. The complete list of variables can be viewed in curl’s manpage. Let’s briefly introduce the variables we use:

• url_effective: the final URL after performing address redirection;
• time_namelookup: The time it takes from the beginning of the request to the completion of name resolution, in seconds, the same below;
• time_redirect: The time it takes to perform all redirections;
• time_connect: The time it takes from the beginning of the request to the establishment of the TCP connection;
• time_appconnect: The time it takes from the beginning of the request to the completion of the SSL/SSH handshake;
• time_pretransfer: The time it takes from the request to the server to prepare to transfer the file, including the transfer negotiation time;
• time_starttransfer: The time it takes from the start of the request to the time the server is ready to transfer the first byte;
• time_total: Complete time taken.

Then execute the request, specifying the file in which the formatted string is saved via @filename:

curl -L -s -w @fmt.txt -o /dev/null http://www.example.com

Output:

Response Time for: http://www.google.co.jp/?gfe_rd=cr&dcr=0&ei=cjIaWpTkHeiQ8QfnxYzoBA

DNS Lookup Time: 0.000038s
Redirection Time: 0.207271s
Connection Time: 0.000039s
App Connection Time: 0.000039s
Pre-transfer Time: 0.000067s
Start-transfer Time: 0.260115s

Total Time: 0.467691s

Request to return compressed status

$ curl --compressed -o- -L https://yarnpkg.com/install.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    54  100    54    0     0     42      0  0:00:01  0:00:01 --:--:--    42
100  2341  100  2341    0     0   1202      0  0:00:01  0:00:01 --:--:--  9289
Installing Yarn!
> Downloading tarball...

[1/2]: https://yarnpkg.com/latest.tar.gz --> /var/folders/j7/3xly5sk567s65ny5dnr__3b80000gn/T/yarn.tar.gz.XXXXXXXXXX.9hJsBsrA
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    57  100    57    0     0     72      0 --:--:-- --:--:-- --:--:--    72
100    93  100    93    0     0     63      0  0:00:01  0:00:01 --:--:--    63
100   643  100   643    0     0    248      0  0:00:02  0:00:02 --:--:--   707
100 1215k  100 1215k    0     0   153k      0  0:00:07  0:00:07 --:--:--  305k

[2/2]: https://yarnpkg.com/latest.tar.gz.asc --> /var/folders/j7/3xly5sk567s65ny5dnr__3b80000gn/T/yarn.tar.gz.XXXXXXXXXX.9hJsBsrA.asc
100    61  100    61    0     0    356      0 --:--:-- --:--:-- --:--:--   356
100    97  100    97    0     0    325      0 --:--:-- --:--:-- --:--:--   325
100   647  100   647    0     0   1283      0 --:--:-- --:--:-- --:--:--  1283
100   832  100   832    0     0   1107      0 --:--:-- --:--:-- --:--:--  812k