Today I would like to once again talk about making backup in the cloud. I will not discuss whether it is good or bad, but I want to share examples of solutions implementations for this most cloud-based backup – from finished software to own mistakes.

3-2-1 let’s go

It is believed that the history of the 3-2-1 backup rule begins with Peter Krogh, who outlined it in his book “Digital Asset Management for Photographers”. I briefly will recall this principle:

There must be at least 3 copies of the data.

•     At least 2 copies must be on physical media of different types. For example, one copy is working data on a disk array, the second copy is data on magnetic tape.
•     At least one backup must be stored outside the office.

Personally, I most often use slightly different rules in creating backups.

Firstly, I take backup copies as the initial data, and secondly, it is not always convenient and budgetary to store them on various types of media – especially for small and medium-sized businesses. My usual backup storage strategy is this:

•     Online backups. Their main goal is to ensure the fastest recovery in case of a small failure. Depending on the infrastructure, these backups can even be stored on the server being backed up – only on a separate disk.
•     Archive backups. They are always stored at least on another server and with history (most often – 6 daily backups, 4 weekly and 4 quarterly).
•     Remote backups. Backups are necessarily stored in another place – on a server in a remote data center or in the cloud. A good option is to synchronize the archive backup directory with the remote storage, if possible.

With operational and archive backups, usually everything is quite simple, except that you should adhere to certain recommendations.

But with remote backups there are many questions. In particular, you must choose where to store these copies and how to put them there. First, I will give some examples of “where”.

Choose a cozy cloud

One option would be to simply rent a dedicated server or install your server in a data center on a collocation.

Indeed, the “cloud” that you built yourself gives more control over what is happening, and the choice of a solution for storage and direct backup is up to the system administrator. 

On the other hand, control also means responsibility – it will be necessary to monitor the status of the server in case of hardware and software problems, while the lack of clouds in the form of Internet dependence and the issue of trust for outsiders has not been canceled.

Another option would be to use specialized services that were created just for storing backups. The most famous example are Amazon Glacier services. They are legends about the technology used – from tape cassettes to blue ray-disks and robotic arms. But officially this is an inexpensive HDD.

Unlike a rented server, the solution is already starting to smell like a bloody enterprise with many “nines of reliability” after the decimal point. True, like many web services at Amazon, it has a complex costing formula. To simplify it, downloading data to the service is free, storage is quite inexpensive ($ 1 for 1 TB per month), but you will have to pay to get the data.

Classic storage services like Amazon S3 and Yandex Object Storage can also be used for backups, of course, but the price tag in this case will be less pretty – ~ $ 10 \ month for 1 TB for Yandex. Also, we cannot fail to mention all-inclusive solutions from manufacturers of backup systems. For example, Acronis Cloud Storage as an addition to Acronis products literally for $ 299 per year will give 250 GB on its servers.

The third option would be to use cloud storage, which is not very designed to store backups of the company, but more focused on ordinary users. Here are just a few of them that are heard:

•     Dropbox
•     OneDrive
•     Google Drive
•     Mega
•     Yandex.Disk
•     Cloud Mail.Ru

I will not compare cloud platforms now, I will give it to the mercy of numerous materials on the network. Personally, I settled on Yandex.Disk for my needs, because it is one of the few who can use WebDAV, APIs and snapshots (history) of files on disk on free plans. Well, of course, I have accumulated a certain amount of free gigabytes on it.

Of course, when choosing, you should pay attention not only to the free number and cost of gigabytes, but also to the license agreement. Separately, it is worth noting the points on which the cloud provider does not bear any responsibility, can delete all files at any time and there will be nothing for it. But almost all of these services have software that allows you to upload files to the service, which brings us to the next point in today’s story.

How to upload data to a cozy cloud

Personally, I don’t really like to use the software provided by the services (unless, of course, we are talking about a specialized service like Acronis): it’s not always possible to set the synchronization schedule, and the story is still alive when Yandex.Disk arranged for Barmin’s patch when updating operating system. Fortunately, there are special software supporting various providers. As usual, I will give a few examples of mostly free and near-free solutions.

Handy Backup. It is issued on the first page of Google for the query “backup to the cloud.” There are paid versions of various functionalities, separate plugins (for example, for Exchange and 1C). There is even a cloud – HBDrive. But most importantly, there is still a free version that can only backup to the cloud – Handy Backup Free for Cloud. Unfortunately, as part of the testing, I was not able to get it to work stably with Yandex.Disk – the periodically scheduled task did not work. It is difficult to want something from a free solution, but I refused to use this software.

CloudBerry Backup. The product is good for everyone, there are even solutions for recovering individual Exchange objects, there is support for many different providers. The use was stopped by the lack of a free version and support of the usual Yandex.Disk, only S3 compatible Yandex Object Storage.

Duplicati 2. Already a completely free product, even for commercial use. There are all popular platforms from Windows to GNU \ Linux, you can work both through the web interface and through the command line, there is also encryption of backups out of the box.

Unfortunately, Yandex.Disk is not supported “out of the box” – only in WebDAV mode. In this mode, the solution from Yandex does not work perfectly – there are problems with large files. But there is one on the valid destination list that solves this problem. 

Rclone Perhaps this is my undisputed leader among other software. A command line utility for many platforms, on the official website, downloads are also available for rare operating systems like Plan9 and Solaris. The list of supported cloud providers is also impressive – it even supports Cephs and OwnCloud. Yandex.Disk is listed. Until recently, the configuration was carried out only through the interactive console menu, but relatively recently it became possible to launch the web interface and configure through it.

The disadvantages include the lack of any built-in schedulers. The utility works exclusively as a transport to / from the clouds, but does not require installation. Including because of this, I use it in conjunction with Yandex.Disk to transfer information from one remote server to another – it turned out that large files are faster to upload to the cloud and download from the cloud than to organize direct file sharing. Yes, and upload backups is a pleasure. For example, to copy only fresh files to the cloud, you can use the command:

rclone copy --max-age 24h --no-traverse D:\backups yandex:backups

Where yandex is the name of the config created in advance, and backups is the folder with backups.

In general, as already a complete backup solution, rclone can be used with Duplicati, choosing rclone as the type of storage. Then Duplicati will create backups using vss (snapshots) according to the scheduler, and the first will be responsible for downloading the backups to the cloud we need. Of course, you can use any other solution like Cobian or even do vss snapshots with the diskshadow command, archive and upload to the cloud using rclone. 

Create your script

Of course, if the cloud provider provides WebDAV access, downloading data will be easy. Example for cmd and Yandex.Disk:

net use Z: "https://webdav.yandex.ru/backup/" /User:[email protected] password rem копируем файлы любым способом net use Z: /delete

But not all providers are good at WebDAV, and there are questions about speed and stability. Therefore, you can use the API, if, of course, the provider provides such access. Let’s analyze an example with the same Yandex.

Yandex uses OAuth for authorization, so for our script we need to get a special token. First you need to create an application in the “Create an Application” section of the site.

You must not forget to give access to the application on Yandex.Disk.

After receiving the application ID, follow the link

https://oauth.yandex.ru/authorize?response_type=token&client_id=12345678&display=popup

Where 12345678 is the received ID. After granting the application access, we will receive the coveted OAuth token, which can already be used in scripts. Here, for example, uploading a file to Yandex.Disk using PowerShell:

$filepath = "D:\backup.zip"
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization" ,'OAuth НашOauthТокен')
$headers.Add("Content-Type","application/json")


$UploadUrl= (Invoke-RestMethod -method GET -URI ("https://cloud-api.yandex.net:443/v1/disk/resources/upload?path=backup.zip") -Headers $headers).href


Invoke-WebRequest -uri $UploadUrl -Method Put -Infile $filepath -ContentType 'application/zip'

It is proposed to organize file rotation, download control and other “body kits” on their own, since the Yandex API is well documented. But personally, I prefer not to reinvent the wheel, but to use rclone.

Well, when backing up to the cloud, I strongly recommend encrypting the archives,  not to be in the inconvenient situation.