Zsync HTTP-based File Transfer Utility Transfers Large Files Efficiently

Zsync is an opensource file transfer utility built on top of rsync algorithm. This helps to download partial/differential files over the HTTP protocol. The utility allows downloading only new parts of a file from a centralized location,  where the older version of the file is already within your computer.

While rsync is for syncing data from one computer to another,  zsync allows file distribution, where the file hosted in a server using any web server can be distributed to many and downloaded seamlessly.

How it works

The command-line utility will do all the differential calculations in the client, instead of doing it in the server as in rsync. Server metadata will be created only once and stored as part of the control file. And rest of the operations and decision making will be handled by the client-side application. This will reduce the huge processing needed on the server-side, even when thousands of clients are trying to fetch the file.

Zsync WorkFlow Diagram
Zsync WorkFlow

Server

The server will calculate the checksum and create the control file in advance. This contains checksums,  block size of the file, length and file permissions. Since the rest of the comparisons happens at the client-side, there aren’t any checksum calculations happening on the server, and each time the client requests for data.

Client

The Zsync client is written in “C”, and capable of pulling the metadata from the server first. This will compare with the current files checksum data (generated by rsync) and requests the remaining data from the server.

How to Create .zsync files

There are multiple ways to create and transfer files using Zsync –

  • Transfer Compressed Files  (gzip, tar, etc)

By running the command “zsyncmake” on the target file, the utility will automatically look into the compressed file and produce a .zsync file for the content. This is one the major differentiation with “rsync”, which doesn’t support compressed files. If you want to share compressed individual files such as rpm, deb, etc. may not help to utilize the capabilities of zsync. At the same time, it will be useful for ISO files, which contain multiple types of files and for each release some changes may happen.

  • Compress and Transfer

Zsyncmake is capable of compressing the file in gzip format before sharing the files. And this compression approach will the best, better than using any standard compression tool, since this may offer better transfer efficiency.


The above command will create a gzipped version of the file, a .zsync file, and a URL by assuming that these two files will be served from the same document root of a  web server.

Zsync Example – Debian/Ubuntu ISO Downloads

One of the best use cases of zsync is the distribution of Ubuntu/Debian iso images as .zsync downloads. You can choose Ubuntu iso .zsync from the below link, and the tool will download only the new changes in the file, no need to download the whole ISO image.

Download using zsync


If you are downloading this file for the first time,  you will be able to see the messages as below and starts the download –

References

Support CNX Software - Donate via PayPal or become a Patron on Patreon

6
Leave a Reply

avatar
4 Comment threads
2 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
6 Comment authors
OlleIgor_khzoobabJean-Luc Aufranc (CNXSoft)Brian Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Alex
Guest
Alex

I don’t understand why this article was posted here. This belongs on HowToForge. Also, zsync does not support HTTPS, and is therefore inappropriate outside of a local area network, where “large file transfers” are rarely an issue. I would prefer if you stop posting such useless articles. This kind of content is irrelevant and will push away your regular readers.

Jean-Luc Aufranc (CNXSoft)
Admin

It might be possible to leverage zsync for distributing firmware files.

Igor_kh
Guest
Igor_kh

Indeed. It is older than dirt https://news.ycombinator.com/item?id=15488235. It is barely maintained also.

Brian
Guest
Brian

Interesting. Would probably struggle with compressed data (the zsync homepage confirms this) as the file alignment would change wildly then, it creates an argument for storing large individual files uncompressed if using this method.

zoobab
Guest

I guess we can go faster with UDP based protocols, such as QUIC.

TCP goes bonkers when you have packet loss.

Olle
Guest

I have been working on a similar tool for some time, which typical use would be for updating embedded systems with an a/b partition setup. Allowing for https. Still actively developing and adding features. If interested source can be found at https://github.com/oll3/bita