© Venti Views

Deploy your static site with curl

Long time I somehow struggled what would be the best way to deploy my static site. Hosting provider offer either FTP or a REST API to push your files onto. Depending on 3rd-party tools was not really an option.

I use Bunny.net for hosting my page. They offer either FTP or a REST API. For whatever reason I started with FTP and much later settled for their API.

I wanted these requirements to be met:

• No 3rd-party tools: I do not want to depend on tools, that are unsupported. Further I do not want to be unsure if some licencing change, or a business model needs to generate revenue
• FOSS: No closed-source software, no payment model, no forced attribution
• Dependency-free: No installation of any prerequisites, certain packages like python 3.12 or similar
• Parallelization: Uploading multiple files at once should bring a big time-saver

In the beginning I broke all of these points. So this blog post is a journey.

Usage of FTP

The first problem occured when I looked for a proper command-line FTP client. While there is ftp for Linux I had the feeling this was not a good option. I could not properly pass authentication and also was not able to select a complete folder to be uploaded to my target machine.

There is also lftp which seems to work pretty well. The last release is from November 2024 - with over 200 open issues. Seems kind of not being supported anymore.

In hindsight, I think it might have worked with one of these clients, but, for whatever reasons, I settled with duck.sh. Besides that this programm goes against my previously mentioned requirements it worked pretty well - if you consider pulling a 200MB+ dependency every time you change a spelling mistake as working well.

After some time I got upset about how long the pipeline runs, I thought about getting rid of the chonker of software. So I thought about writing my own Python script that can upload files.

A custom python script

After I convinced myself that running a deployment where half of the time is pulling in dependencies is not a good idea, I ditched the duck.sh/ftp approach and came up with a Python script. I thought I was on the save side now, although I felt it was a bit bloated - but hey, better than pulling in a 200MB dependency. Well, … now I had to install python-3.13 into the image. That also takes time.

You can argue that using a Docker image where Python is already included can be used. But I also needed NodeJS to be installed, which then does not solve the installation of one or the other.

And again, I was at a point, where installing dependencies takes more time than the original deployment process. Further I was just unsatisfied with the script while drove me to my final decision: just use what is already there. curl is your friend.

curl is your friend

Digging a bit deeper into curl and its parameters I figured out, that is ships with some pretty nice features like parallel request and arguments that makes life uploading files so easy.

curl has a config file. What?! And instead of using the file you can use arguments likes this.

1
2
3
4
5
6
7

curl --parallel --parallel-max 8 --config - << 'EOF'
header = "AccessKey: token123"
upload-file = "./local/dir/photo.jpg"
url = "https://example.com"
upload-file = "./another/dir/text1.txt"
url = "https://example.com"
EOF

So I built a little script to collect all files I want to upload, and then pipe it to curl, and voila, you have you uploader.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

FILES="$(find "./website-files" -type f)"

CONFIG="$(
  while IFS= read -r file; do
    printf 'upload-file = "%s"\nurl = "%s"\n' \
      "$file" "$BASE/${file#"$SOURCE_DIR"/}"
  done <<<"$FILES"
)"

printf '%s\n' "$CONFIG" | curl --config -

There are some more checks included in the script so have a look at the full version on Codeberg.

This script and the usage of a combination of shell scripting and curl gives me

• parallel uploads
• a sort of uploading a whole folder while just collecting all files in bash
• FOSS tools, no other (closed-source) software or tools needed
• highly adjustable to my needs

No overhead, full control. Beautiful.