Back to Transfer Sh

transfer.sh

README.md

1.6.124.2 KB
Original Source

transfer.sh

Easy and fast file sharing from the command-line. This code contains the server with everything you need to create your own instance.

Transfer.sh currently supports the s3 (Amazon S3), gdrive (Google Drive), storj (Storj) providers, and local file system (local).


Disclaimer

@stefanbenten happens to be a maintainer of this repository and the person who host a well known public installation of the software in the repo.

The two are anyway unrelated, and the repo is not the place to direct requests and issues for any of the pubblic installation.

No third-party public installation of the software in the repo will be advertised or mentioned in the repo itself, for security reasons.

The official position of me, @aspacca, as maintainer of the repo, is that if you want to use the software you should host your own installation.


Usage

This section outlines how to use transfer.sh

Upload

bash
$ curl -v --upload-file ./hello.txt https://transfer.sh/hello.txt

Encrypt & Upload

bash
$ gpg --armor --symmetric --output - /tmp/hello.txt | curl --upload-file - https://transfer.sh/test.txt

Download & Decrypt

bash
$ curl https://transfer.sh/1lDau/test.txt | gpg --decrypt --output /tmp/hello.txt

Upload to Virustotal

bash
$ curl -X PUT --upload-file nhgbhhj https://transfer.sh/test.txt/virustotal

Deleting

bash
$ curl -X DELETE <X-Url-Delete Response Header URL>

Request Headers

This section explains how to handle request headers with curl:

Max-Downloads

bash
$ curl --upload-file ./hello.txt https://transfer.sh/hello.txt -H "Max-Downloads: 1" # Limit the number of downloads

Max-Days

bash
$ curl --upload-file ./hello.txt https://transfer.sh/hello.txt -H "Max-Days: 1" # Set the number of days before deletion

X-Encrypt-Password

Beware, use this feature only on your self-hosted server: trusting a third-party service for server side encryption is at your own risk

bash
$ curl --upload-file ./hello.txt https://your-transfersh-instance.tld/hello.txt -H "X-Encrypt-Password: test" # Encrypt the content server side with AES256 using "test" as password

X-Decrypt-Password

Beware, use this feature only on your self-hosted server: trusting a third-party service for server side encryption is at your own risk

bash
$ curl https://your-transfersh-instance.tld/BAYh0/hello.txt -H "X-Decrypt-Password: test" # Decrypt the content server side with AES256 using "test" as password

Response Headers

This section explains how to handle response headers:

X-Url-Delete

The URL used to request the deletion of a file and returned as a response header:

bash
curl -sD - --upload-file ./hello.txt https://transfer.sh/hello.txt | grep -i -E 'transfer\.sh|x-url-delete'
x-url-delete: https://transfer.sh/hello.txt/BAYh0/hello.txt/PDw0NHPcqU
https://transfer.sh/hello.txt/BAYh0/hello.txt

Examples

See good usage examples on examples.md

Create direct download link:

https://transfer.sh/1lDau/test.txt --> https://transfer.sh/get/1lDau/test.txt

Inline file:

https://transfer.sh/1lDau/test.txt --> https://transfer.sh/inline/1lDau/test.txt


Usage

ParameterDescriptionValueEnv
listenerport to use for http (:80)LISTENER
profile-listenerport to use for profiler (:6060)PROFILE_LISTENER
force-httpsredirect to httpsfalseFORCE_HTTPS
tls-listenerport to use for https (:443)TLS_LISTENER
tls-listener-onlyflag to enable tls listener onlyTLS_LISTENER_ONLY
tls-cert-filepath to tls certificateTLS_CERT_FILE
tls-private-keypath to tls private keyTLS_PRIVATE_KEY
http-auth-useruser for basic http auth on uploadHTTP_AUTH_USER
http-auth-passpass for basic http auth on uploadHTTP_AUTH_PASS
http-auth-htpasswdhtpasswd file path for basic http auth on uploadHTTP_AUTH_HTPASSWD
http-auth-ip-whitelistcomma separated list of allowed ips to upload without auth challengeHTTP_AUTH_IP_WHITELIST
virustotal-keyVirusTotal API keyVIRUSTOTAL_KEY
ip-whitelistcomma separated list of ips allowed to connect to the serviceIP_WHITELIST
ip-blacklistcomma separated list of ips not allowed to connect to the serviceIP_BLACKLIST
temp-pathpath to temp foldersystem tempTEMP_PATH
web-pathpath to static web files (for development or custom front end)WEB_PATH
proxy-pathpath prefix when service is run behind a proxy (a / prefix will be trimmed)PROXY_PATH
proxy-portport of the proxy when the service is run behind a proxyPROXY_PORT
email-contactemail contact for the front endEMAIL_CONTACT
ga-keygoogle analytics key for the front endGA_KEY
providerwhich storage provider to use(s3, storj, gdrive or local)
uservoice-keyuser voice key for the front endUSERVOICE_KEY
aws-access-keyaws access keyAWS_ACCESS_KEY
aws-secret-keyaws access keyAWS_SECRET_KEY
bucketaws bucketBUCKET
s3-endpointCustom S3 endpoint.S3_ENDPOINT
s3-regionregion of the s3 bucketeu-west-1S3_REGION
s3-no-multipartdisables s3 multipart uploadfalseS3_NO_MULTIPART
s3-path-styleForces path style URLs, required for Minio.falseS3_PATH_STYLE
storj-accessAccess for the projectSTORJ_ACCESS
storj-bucketBucket to use within the projectSTORJ_BUCKET
basedirpath storage for local/gdrive providerBASEDIR
gdrive-client-json-filepathpath to oauth client json config for gdrive providerGDRIVE_CLIENT_JSON_FILEPATH
gdrive-local-config-pathpath to store local transfer.sh config cache for gdrive providerGDRIVE_LOCAL_CONFIG_PATH
gdrive-chunk-sizechunk size for gdrive upload in megabytes, must be lower than available memory (8 MB)GDRIVE_CHUNK_SIZE
lets-encrypt-hostshosts to use for lets encrypt certificates (comma separated)HOSTS
logpath to log fileLOG
cors-domainscomma separated list of domains for CORS, setting it enable CORSCORS_DOMAINS
clamav-hosthost for clamav featureCLAMAV_HOST
perform-clamav-prescanprescan every upload using clamav (clamav-host must be local clamd unix socket)PERFORM_CLAMAV_PRESCAN
rate-limitrequest per minuteRATE_LIMIT
max-upload-sizemax upload size in kilobytesMAX_UPLOAD_SIZE
purge-daysnumber of days after the uploads are purged automaticallyPURGE_DAYS
purge-intervalinterval (hours) to run automatic purge for (excluding S3 and Storj)PURGE_INTERVAL
random-token-lengthlength of random token for upload path (double the size for delete path)6RANDOM_TOKEN_LENGTH

If you want to use TLS using lets encrypt certificates, set lets-encrypt-hosts to your domain, set tls-listener to :443 and enable force-https.

If you want to use TLS using your own certificates, set tls-listener to :443, force-https, tls-cert-file and tls-private-key.


Development

Switched to GO111MODULE

bash
go run main.go --provider=local --listener :8080 --temp-path=/tmp/ --basedir=/tmp/

Build

bash
$ git clone [email protected]:dutchcoders/transfer.sh.git
$ cd transfer.sh
$ go build -o transfersh main.go

Docker

For easy deployment, we've created an official Docker container. There are two variants, differing only by which user runs the process.

The default one will run as root:

[!WARNING] It is discouraged to use latest tag for WatchTower or similar tools. The latest tag can reference unreleased developer, test builds, and patch releases for older versions. Use an actual version tag until transfer.sh supports major or minor version tags.

bash
docker run --publish 8080:8080 dutchcoders/transfer.sh:latest --provider local --basedir /tmp/

No root

The -noroot tags indicate image builds that run with least priviledge to reduce the attack surface might an application get compromised.

[!NOTE] Using -noroot is recommended

The one tagged with the suffix -noroot will use 5000 as both UID and GID:

bash
docker run --publish 8080:8080 dutchcoders/transfer.sh:latest-noroot --provider local --basedir /tmp/

[!NOTE] Development history details at:

Tags

NameUsage
latestLatest CI build, can be nightly, at commit, at tag, etc.
latest-norootLatest CI build, can be nightly, at commit, at tag, etc. using [no root]
nightlyScheduled CI build every midnight UTC
nightly-norootScheduled CI build every midnight UTC using [no root]
edgeLatest CI build after every commit on main
edge-norootLatest CI build after every commit on main using [no root]
vx.y.zCI build after tagging a release
vx.y.z-norootCI build after tagging a release using [no root]

Building the Container

You can also build the container yourself. This allows you to choose which UID/GID will be used, e.g. when using NFS mounts:

bash
# Build arguments:
# * RUNAS: If empty, the container will run as root.
#          Set this to anything to enable UID/GID selection.
# * PUID:  UID of the process. Needs RUNAS != "". Defaults to 5000.
# * PGID:  GID of the process. Needs RUNAS != "". Defaults to 5000.

docker build -t transfer.sh-noroot --build-arg RUNAS=doesntmatter --build-arg PUID=1337 --build-arg PGID=1338 .

S3 Usage

For the usage with a AWS S3 Bucket, you just need to specify the following options:

  • provider --provider s3
  • aws-access-key (either via flag or environment variable AWS_ACCESS_KEY)
  • aws-secret-key (either via flag or environment variable AWS_SECRET_KEY)
  • bucket (either via flag or environment variable BUCKET)
  • s3-region (either via flag or environment variable S3_REGION)

If you specify the s3-region, you don't need to set the endpoint URL since the correct endpoint will used automatically.

Custom S3 providers

To use a custom non-AWS S3 provider, you need to specify the endpoint as defined from your cloud provider.


Storj Network Provider

To use the Storj Network as a storage provider you need to specify the following flags:

  • provider --provider storj
  • storj-access (either via flag or environment variable STORJ_ACCESS)
  • storj-bucket (either via flag or environment variable STORJ_BUCKET)

Creating Bucket and Scope

You need to create an access grant (or copy it from the uplink configuration) and a bucket in preparation.

To get started, log in to your account and go to the Access Grant Menu and start the Wizard on the upper right.

Enter your access grant name of choice, hit Next and restrict it as necessary/preferred. Afterwards continue either in CLI or within the Browser. Next, you'll be asked for a Passphrase used as Encryption Key. Make sure to save it in a safe place. Without it, you will lose the ability to decrypt your files!

Afterwards, you can copy the access grant and then start the startup of the transfer.sh endpoint. It is recommended to provide both the access grant and the bucket name as ENV Variables for enhanced security.

Example:

export STORJ_BUCKET=<BUCKET NAME>
export STORJ_ACCESS=<ACCESS GRANT>
transfer.sh --provider storj

Google Drive Usage

For the usage with Google drive, you need to specify the following options:

  • provider
  • gdrive-client-json-filepath
  • gdrive-local-config-path
  • basedir

Creating Gdrive Client Json

You need to create an OAuth Client id from console.cloud.google.com, download the file, and place it into a safe directory.

Usage example

go run main.go --provider gdrive --basedir /tmp/ --gdrive-client-json-filepath /[credential_dir] --gdrive-local-config-path [directory_to_save_config]


Shell functions

Bash, ash and zsh (multiple files uploaded as zip archive)

Add this to .bashrc or .zshrc or its equivalent
bash
transfer() (if [ $# -eq 0 ]; then printf "No arguments specified.\nUsage:\n transfer <file|directory>\n ... | transfer <file_name>\n">&2; return 1; fi; file_name=$(basename "$1"); if [ -t 0 ]; then file="$1"; if [ ! -e "$file" ]; then echo "$file: No such file or directory">&2; return 1; fi; if [ -d "$file" ]; then cd "$file" || return 1; file_name="$file_name.zip"; set -- zip -r -q - .; else set -- cat "$file"; fi; else set -- cat; fi; url=$("$@" | curl --silent --show-error --progress-bar --upload-file "-" "https://transfer.sh/$file_name"); echo "$url"; )

Now you can use transfer function

$ transfer hello.txt

Bash and zsh (with delete url, delete token output and prompt before uploading)

Add this to .bashrc or .zshrc or its equivalent
<details><summary>Expand</summary><p>
bash
transfer()
{
    local file
    declare -a file_array
    file_array=("${@}")

    if [[ "${file_array[@]}" == "" || "${1}" == "--help" || "${1}" == "-h" ]]
    then
        echo "${0} - Upload arbitrary files to \"transfer.sh\"."
        echo ""
        echo "Usage: ${0} [options] [<file>]..."
        echo ""
        echo "OPTIONS:"
        echo "  -h, --help"
        echo "      show this message"
        echo ""
        echo "EXAMPLES:"
        echo "  Upload a single file from the current working directory:"
        echo "      ${0} \"image.img\""
        echo ""
        echo "  Upload multiple files from the current working directory:"
        echo "      ${0} \"image.img\" \"image2.img\""
        echo ""
        echo "  Upload a file from a different directory:"
        echo "      ${0} \"/tmp/some_file\""
        echo ""
        echo "  Upload all files from the current working directory. Be aware of the webserver's rate limiting!:"
        echo "      ${0} *"
        echo ""
        echo "  Upload a single file from the current working directory and filter out the delete token and download link:"
        echo "      ${0} \"image.img\" | awk --field-separator=\": \" '/Delete token:/ { print \$2 } /Download link:/ { print \$2 }'"
        echo ""
        echo "  Show help text from \"transfer.sh\":"
        echo "      curl --request GET \"https://transfer.sh\""
        return 0
    else
        for file in "${file_array[@]}"
        do
            if [[ ! -f "${file}" ]]
            then
                echo -e "\e[01;31m'${file}' could not be found or is not a file.\e[0m" >&2
                return 1
            fi
        done
        unset file
    fi

    local upload_files
    local curl_output
    local awk_output

    du -c -k -L "${file_array[@]}" >&2
    # be compatible with "bash"
    if [[ "${ZSH_NAME}" == "zsh" ]]
    then
        read $'upload_files?\e[01;31mDo you really want to upload the above files ('"${#file_array[@]}"$') to "transfer.sh"? (Y/n): \e[0m'
    elif [[ "${BASH}" == *"bash"* ]]
    then
        read -p $'\e[01;31mDo you really want to upload the above files ('"${#file_array[@]}"$') to "transfer.sh"? (Y/n): \e[0m' upload_files
    fi

    case "${upload_files:-y}" in
        "y"|"Y")
            # for the sake of the progress bar, execute "curl" for each file.
            # the parameters "--include" and "--form" will suppress the progress bar.
            for file in "${file_array[@]}"
            do
                # show delete link and filter out the delete token from the response header after upload.
                # it is important to save "curl's" "stdout" via a subshell to a variable or redirect it to another command,
                # which just redirects to "stdout" in order to have a sane output afterwards.
                # the progress bar is redirected to "stderr" and is only displayed,
                # if "stdout" is redirected to something; e.g. ">/dev/null", "tee /dev/null" or "| <some_command>".
                # the response header is redirected to "stdout", so redirecting "stdout" to "/dev/null" does not make any sense.
                # redirecting "curl's" "stderr" to "stdout" ("2>&1") will suppress the progress bar.
                curl_output=$(curl --request PUT --progress-bar --dump-header - --upload-file "${file}" "https://transfer.sh/")
                awk_output=$(awk \
                    'gsub("\r", "", $0) && tolower($1) ~ /x-url-delete/ \
                    {
                        delete_link=$2;
                        print "Delete command: curl --request DELETE " "\""delete_link"\"";

                        gsub(".*/", "", delete_link);
                        delete_token=delete_link;
                        print "Delete token: " delete_token;
                    }

                    END{
                        print "Download link: " $0;
                    }' <<< "${curl_output}")

                # return the results via "stdout", "awk" does not do this for some reason.
                echo -e "${awk_output}\n"

                # avoid rate limiting as much as possible; nginx: too many requests.
                if (( ${#file_array[@]} > 4 ))
                then
                    sleep 5
                fi
            done
            ;;

        "n"|"N")
            return 1
            ;;

        *)
            echo -e "\e[01;31mWrong input: '${upload_files}'.\e[0m" >&2
            return 1
    esac
}
</p></details>

Sample output

bash
$ ls -lh
total 20M
-rw-r--r-- 1 <some_username> <some_username> 10M Apr  4 21:08 image.img
-rw-r--r-- 1 <some_username> <some_username> 10M Apr  4 21:08 image2.img
$ transfer image*
10240K  image2.img
10240K  image.img
20480K  total
Do you really want to upload the above files (2) to "transfer.sh"? (Y/n):
######################################################################################################################################################################################################################################## 100.0%
Delete command: curl --request DELETE "https://transfer.sh/wJw9pz/image2.img/mSctGx7pYCId"
Delete token: mSctGx7pYCId
Download link: https://transfer.sh/wJw9pz/image2.img

######################################################################################################################################################################################################################################## 100.0%
Delete command: curl --request DELETE "https://transfer.sh/ljJc5I/image.img/nw7qaoiKUwCU"
Delete token: nw7qaoiKUwCU
Download link: https://transfer.sh/ljJc5I/image.img

$ transfer "image.img" | awk --field-separator=": " '/Delete token:/ { print $2 } /Download link:/ { print $2 }'
10240K  image.img
10240K  total
Do you really want to upload the above files (1) to "transfer.sh"? (Y/n):
######################################################################################################################################################################################################################################## 100.0%
tauN5dE3fWJe
https://transfer.sh/MYkuqn/image.img

Contributions

Contributions are welcome.


Creators

Remco Verhoef

Uvis Grinfelds


Maintainers

  • Andrea Spacca
  • Stefan Benten

Code and documentation copyright 2011-2018 Remco Verhoef. Code and documentation copyright 2018-2020 Andrea Spacca. Code and documentation copyright 2020- Andrea Spacca and Stefan Benten.

Code released under the MIT license.