Android Backups – Nextcloud vs. Resilio Sync

Issue: I want to backup specific, existing folders of multiple Android devices. The backup has to be done locally inside the LAN and encryption at rest is a must-have.

tl;dr: Resilio Sync hosted on a Raspberry Pi / Odroid / NUC does the job perfectly with their app. Nextcloud is far slower and the FolderSync app, which I used to transfer only specific folders, is unreliable.

Requirements

For months I’ve been using a selfhosted Nextcloud on a Raspberry Pi 3 and an Odroid N2. On the Raspberry I’ve used a bare metal installation of Nextcloud, the Odroid ran a Docker image of Nextcloud. Although it was far from perfect, the usage of Nextcloud fulfilled all features I required:

• On premise hosting with ARM support
• file encryption at rest
• multi-user support
• an Android app to only backup specific, already existing folders

As Nextcloud is a widespread solution for sharing files and backups, I figured my use case should be very easy to implement. But that thought turned out to be false. The issue here lay with the combination of two features I required: encryption at rest and the backup of specific, already existing folders. In my research I found 2 types of Android apps:

• Apps that feature encryption of files and folders, but do not work with Nextcloud and are proprietary solutions. Encryption can only the used with newly created folders and / or on premise hosting was not possible.
• Apps that work with Nextcloud and can make use of the server encryption Nextcloud features but never allow to select only specific subfolders.

FolderSync, Webdav and Nextcloud

Having both features at the same time was not possible in most apps, except one: FolderSync. This app featured exactly every aspect I was looking for in a backup solution. While most apps, like the official Nextcloud app at that time, didn’t allow to synchronize selected folders with encryption, FolderSync featured this by using the Webdav protocol, which made it a very versatile app. I created folders with server encryption in Nextcloud and added a folder pair in the FolderSync app to sync specific folders.

This was a working solution, but as I witnessed after a few days, not a very good one. It worked okay-ish for a only a couple of small files, but the transfer rate was most of the time below 1 MB/s. When synchronizing a folder with ~5000 files the app ran for hours, even when most of the files have already been synchronized and only a single new files was added. When working with larger amount of files, the time it took to check what already was added, and if there was a delta, was much longer than the actual upload time. Moreover when running the synchronization, the app feeled sluggish and at times it took about 10 seconds to display what it was doing – it looked as if there was no synchronization happening. So to see the actual state of the process, after opening the app or passing the lock screen of the phone, I had to wait a couple of seconds for FolderSync to display what it was doing – and even if it was doing anything. I wasn’t happy with this solution but there wasn’t a better solution possible – or so I thought.

It works until it doesn’t

One day FolderSync refused to synchronize data. The issue appeared after an update of the app. For some reason the app didn’t allow unencrypted connections anymore (I never bothered setting up TLS for Nextcloud, as I was only using it in my LAN) and an error message appeared:

CLEARTEXT communication not permitted by network security policy

FolderSync, 2020, colorized

The fix should be quick, I thought to myself, when I found an example on the official Github repository which featured an nginx reverse proxy with some custom foo – just something to add to my docker-compose.yaml and I’m good to go. I have setup a test environment on a Digitalocean droplet, reproduced the error I was getting on premise and fixed it by adding the containers from the example. However finally setting it up on the Odroid I noticed this way was not as easy as I thought. The images from the example where only compiled for x86 architectures, but the Odroid was running on ARM. So I would have to create a custom image or set up a reverse proxy on my own. At that point I remembered Resilio Sync, an app which looked promising but the official documentation and website didn’t make it clear if it was appropriate for my usecase – I wanted to have a look at it for quite some time and now the timing seemed right.

Resilio Sync

Resilio Sync works with the Bittorrent P2P protocol and is even compatible to the Bittorrent Sync app, although I used the official Resilio Sync app. There’s a docker image by resilio for x86 architectures. When using ARM I found a very solid custom image by linuxserver. I’m currently running the official image on my Intel NUC and the ARM image on the Odroid N2.

The official app of Resilio Sync features the possibility to add folders with certain permissions. More on that later on. The android app does not allow to add an encrypted folder, but it’s possible in the Resilio Sync server (the one started by the docker image). As the concept of adding and synchronizing folders in Resilio is somewhat confusing in the beginning I looked into hard drive encryption with Linux and chose to proceed with this approach. A good read to set hard drive encryption up with luks can be found here. The result left me with an encrypted partition on which I stored the data by using a mount inside the docker-compose.yaml. For this to work the encrypted partition has to be available, which it will be not after a reboot for example. My most recent post deals with the concept and checking if the mount is available.

Performance Tests

As I researched this while still having the Raspberry Pi 3 B+ as a homeserver instead of the Intel NUC, the benchmarks were created on the Raspberry Pi. I have however not done extensive benchmarking and performance testing as it was apparent that Resilio is far superior in that regard. I chose to sync a 130 MB folder from my phone which contained about 45 files.

With Nextcloud and the FolderSync app it took about 7:24 min to sync the entire folder once. That’s about 300 kbps, which is slow in every regard. I suppose it’s due to the amount of files – larger files can be synced with about 500 kbps to 1 MB/s. A greater impact is due to the use of server encryption in Nextcloud, which results in Nextcloud encrypting every file before storing it to the hard drive. I cannot present valid benchmarks for that, as I’ve tested the impact of this a long time ago, but I think it was about twice the speed with server encryption turned off. On a side note: I also did some benchmarks some months ago with Nextcloud hosted on a Digitalocean $5 droplet. The amount of data was 210 MB / 65 files. The Raspberry Pi achieved 13:32 min (264 kbps) while the Digitalocean droplet achieved 5:31 min (649 kbps).

So while Nextcloud / FolderSync ran with 300 kbps and synchronized the folder in 7:24 min, Resilio synchronized the same folder in 1:10 min. That’s a really big difference. Another great improvement was the reliability of the Android app. Not only is it much faster in the synchronization, it actually worked through the entire, infamous WhatsApp folder in just a few minutes and did not crash or freeze like the FolderSync app. When testing the folder, which got 1:10 min on the Raspberry Pi on the Intel Nuc, the results where even more astonishing: 0:35 min – and the real I/O of that was only 20 seconds when looking at the stats Netdata displays.

So how does it work?

In a single word: confusing. It actually took some time for me to understand how Resilio is supposed to be used. You create a folder on a device and then allow peers (other devices to sync with this folder) to access this folder. It’s possible to have a one-way or a bidirectional synchronization as well as a backup possibility. If a new folder is added by using the + Button and the info-Icon Button, which is right behind the name of the newly created folder, is pressed, you will get the properties to customize the permissions and a sync target.

The third entry called Share (Verteilen in this screenshot) displays the mode of the permission for the link you will send out to create a pairing of the folder. By using the gear-icon at the end you can choose between read and read/write. When clicked on the copy symbol the link gets copied and can be entered on another peer, e.g. the Resilio Sync server side.

Read permissions means that the peers of this folder (e.g. the Resilio sync server spawned by the Docker image) can only read data – this is a one-way sync. If a new file gets created in the Android folder, it will get synced – if it’s deleted, it will get deleted. However if a file of the sync folder gets deleted on the storage of the Resilio Sync server, it will not get deleted on the Android device. If this is a requirement then the permission has to be read/write – a bidirectional sync. It’s possible to add multiple peers (sync targets) and share the folder with different permissions.

To achieve some kind of real backup feature a specific type of folder can be created by the + Button. This backup folder acts like a one-way sync, however if a file on the source device (the Android device) gets deleted, it will not get deleted on the side of the peer. Some kind of mixed operation can be realized when using an archive. This feature is on by default (at least it was for me) and can be toggled by accessing the preferences of the folder (Voreinstellungen in the screenshot above). This also can be set on the preferences of the folder on the server side (see screenshot beneath).

If the archive toggle is activated, the resilio sync peer will delete files, but move them in some kind of archive folder from which they can be restored. The feature is limited to the device on which it’s activated. So if you want to have an archive on the Resilio Sync server side, you will have to activate it on the preferences there, but it’s also possibile to have this on the side of e.g. the Android app, when a bidirectional sync is established.

Relay, Tracker, LAN

A very important finding was the impact of the relay-server and tracker-server toggles. Resilio (as in the company, the creator of this software) runs servers to give users the chance to sync with no direct access to the peer. Resilio hosts relay-servers, which act as a command & control server. If no direct connection can be established between two peers, each peer connects to the Resilio relay server and data is routed through this server. Resilio emphasizes that the entire data is transport-layer encrypted, but it’s in any case a feature one should be aware of.

To discover peers the tracker-server is contacted. On this issue I did not do a lot of research, but i believe this to be a server hosted by Resilio as well. It’s job is to get the IP address of peers to sync with. To have a LAN-only setup, with no data transmitted to resilio, I disabled the relay- and the tracker-server and chose to search the LAN. In my case it was necessary for me to add the host I want to sync with in the predefined hosts section, as without this setting no connection could be made. In my case I added the local IP address of the NUC which runs the Resilio Sync Docker image and the port used.

So to wrap this up: Resilio seems to be a far superior solution for the requirements and usecases I outlined in the beginning. Nextcloud and FolderSync are more instable and surprisingly slower. However it may be important to have disk encryption enabled to achieve encryption-at-rest if no other solution can be found with the usage of encrypted folders in Resilio.