When I started working for my employer , they used an on-site server and file storage with overnight backup to external drives. All been limited with the internet speed (which in England is quite crazy that business parks rarely got access to an inexpensive fibre connection, where at home you can have cheap Full Fibre), but when things change (or rather the company need to invest in change), we moved it into the cloud.
We chose Microsoft 365 as our provider.
Once migration completed and people used to use the new way of accessing their files (which haven’t have been different with the integration of the OneDrive app and Windows 10, apart from links to files), I needed to find a method, how to have a backup of all files in case of emergency (or accidental removal of important files by an employee).
Despite that Microsoft provides an option to restore files for a short period (typically 30 days), sometimes it may be too short and before you notice that files are missing it will be too late. It’s better to have a backup (even an old version of the file) than don’t have it at all.
Without going into costly online backup solutions I decided to build my backup system and I made it.
For this purpose, I used Raspberry Pi with Raspberry Pi OS (formerly RaspbianOS). Initially, I used Pi 3, but moved to Pi Zero W, as Pi 3 was wasting itself for this purpose. For storage, I used an externally connected hard drive (2.5" 1 TB hard drive in USB 3.0 case) connected via micro USB to USB converter. As a power supply, I used 2A @ 5V iPad charger with a USB to micro USB cable.
Quick initial setup, a couple of commands and the backup system was ready to go. I have been using it successfully since mid-2019 and now I would like to share it with you.
You will need:
- Raspberry Pi with 8GB (min.) micro SD Card (for the operating system);
- USB Connected external Hard Drive or Memory Stick (for external storage, capacity depend on your needs);
- A charger that will provide enough power (especially with an external hard drive);
- Raspberry Pi OS (previously known as Raspbian OS);
- Keyboard and mouse for initial setup along with Screen connected via HDMI cable.
Firstly we head to Raspberry OS website and downloading the desired operating system.
We will need a system with a desktop environment, as despite that backup and further management can be done through the terminal, we need a web browser to get an authentication token.
From the above desktop download Raspberry Pi OS with desktop.
We will need additional software called balenaEtcher to put a downloaded image into our SD card.
We can use dedicated Raspberry Pi Imager to put a system into our SD card, but I prefer the other way.
Once our system image is downloaded, we connect our SD card via card reader to our operating system.
From the balenaEtcher window select Flash from image and point to the downloaded Raspberry OS file (compressed, no need to extract).
Select your card by clicking Select target and confirm all by clicking Flash! button.
Once the job is done we need to put our card into Raspberry Pi and boot it up.
In the initial stage I am connecting my Raspberry Pi to the network using Ethernet Cable. In the case of Pi Zero W, we need to wait until the desktop is ready and connect to our WiFi network.
Once the desktop is up and running we got the option to update our system via GUI, but we can skip that. Also, at the initial stage, we can connect our Pi to a WiFi network to get rid of one cable.
Before we start going into configuration it’s worth setting up storage.
Let’s take our hard drive and connect it to our Raspberry Pi.
If we will get a
Low voltage warningwarning and constant lightning symbol, we may need to use a much more powerful charger. Normally, this warning is shown when we connect a new device and booting system.
To set up our hard drive, we need to have it formatted and ready for use.
My preferred tool for that is GParted that we can install from the terminal.
sudo apt install gparted
Once installed, we can access it from Pi Menu > System Tools > GParted.
From GParted select our hard drive from the top-right drop-down menu. Typically will it be
If there is no partition on a hard drive, we will be creating one.
If there are any, let’s remove them and start from the beginning.
I prefer to create partition
exfat as this will allow you to connect it quickly to your Windows computer and work swiftly on saved files. Surprisingly
exfat working very well in Linux despite being a Microsoft creation but GParted is not able to create it natively.
To add it, install through the terminal:
sudo apt install exfat-utils exfat-fuse
In GParted create unformatted partition (this will create
/dev/sda1) and from terminal format it to
sudo mkfs.exfat /dev/sda1
When you disconnect the hard drive and connect it back again you will see that it will be automatically mounted and ready to use.
Having a view on our desktop on a connected screen via HDMI we will be connecting to our device via the terminal.
We can use PuTTY for that if we are connecting from Windows or just through the command in the macOS terminal.
IP address we can read by hovering our mouse on network manager in the right-top corner of the Raspberry OS desktop.
Before we do that, we need to enable it.
We need to run the terminal on our desktop and go into
In the Configuration Tool, we head to 3 Interface Options > P2 SSH and on the question: Would you like the SSH server to be enabled? answering Yes.
We can finish at that stage and try to connect as above.
Default password will be raspberry. We can change it by typing
Once we are connected, we can move further.
For the backup purpose, we will be using a package called rclone .
To install it we are using the following command:
sudo apt install rclone
Then we can go and set up a new config. We are doing this on the Raspbian OS desktop initially.
From the menu, we are creating a new remote connection (n). Let’s name it o365onedrive (for OneDrive) and later once again o365sharepoint for (SharePoint Document Library).
In the next step, we need to decide what service needs to be used.
For Microsoft, you could think to select (17) Microsoft OneDrive. This is the one when you need to backup the User OneDrive folder. It will also require an authentication token (not a password), this is why we will need to do this with a graphic environment on Raspberry OS.
For SharePoint storage (Document Library) we need to select (22) Webdav. For this purpose, we will need an app password (specially generated password, not the account password).
First thing first.
Let’s select Storage> 17 or type onedrive (numbers can be different in different versions of rclone).
We don’t need to put anything in
client_secret. When asked Edit advanced config? answering n for No.
When asked Use auto config? answer Yes.
This will open Chromium Web Browser where we need to log in with our Microsoft Account.
When everything goes well, we will have a simple page with the word Success!.
At this stage, we can close the web browser and go back to the terminal.
From the final option, we need to select what type of account it is. We are choosing 1 for OneDrive Personal or Business.
You may see SharePoint here as well, but about that later.
In the last step we need to confirm which drive to use (there shall be one) but selecting 0 (zero).
One last step:
Found drive 'root' of type 'business', URL: https://org-my.sharepoint.com/personal/you/Documents Is that okay?
We just confirm with y for Yes and once again y for Yes to confirm that all is fine and configuration will be saved.
In such a way, we got a configuration for OneDrive done. It’s time for SharePoint.
See the official guide of setting up OneDrive with Rclone .
However, before we go to the next step, let’s see if our configuration is working.
By executing command
rclone lsd o365onedrive: we shall bring a list of folders on our OneDrive.
rclone ls o365onedrive: will bring a list of files (long).
To create a backup of the whole SharePoint document library we need to use an admin account that has access to all files, even restricted to some users. Sometimes the user account may not be sufficient.
As you saw in the previous point, it was possible to add in the same method another clone configuration for SharePoint. However, this will be restricted to files, that the user (myself) has access to.
If we have a Super Admin account that by default has access to it, we need to use a different approach.
If copying what the user is allowed to, we can set up everything exactly like in point above, but instead of OneDrive Personal or Business we select Sharepoint site.
For this purpose, we will use Webdav.
See official guide of setting up Webdav with Rclone .
Once again lets do
- New (n) remote
- name> o365sharepoint
- Storage> 22 (or type webdav)
Now we need to type an URL for our SharePoint site that we want to get access to and where we store our Document Library. Mine was something like that
- vendor> select 3 (or type sharepoint)
- user> type email address of Super Admin
- Password> select y and type your user password.
Because the password is stored hashed but in plain text, I highly recommend not using the User Password but instead generate App Password . Also, if we got implemented two-step authentication, the user password will not work here and why an app-specific password is required.
By executing the command
rclone lsd o365sharepoint: we shall see the list of folders on our SharePoint site.
Typically, we will be interested in the folder “Shared Documents” where we are storing files that are used in the OneDrive app across company users.
Now it’s time to execute commands that will start the backup process.
We don’t need to do that on the Raspberry OS desktop. We can make them via SSH.
Because when we execute a command we need to wait until it finishes (we cannot close the terminal window). I highly recommend installing screen (
sudo apt install screen) and execute the command inside it.
When we close the terminal window, the command still will be working in the background, and we can always come back to it to check progress by reattaching the screen (
Before we start, we also need to know where is our hard drive mounted by using the command
Let’s got to our mount point (for example
cd /media/pi/F05C-FB86) and let’s create two folders. One for OneDrive backup and one for SharePoint.
mkdir OneDrive mkdir SharePoint
Now we are ready.
screen and execute the following command.
rclone copy --progress o365onedrive: /media/pi/F05C-FB86/OneDrive --log-file=/media/pi/F05C-FB86/OneDrive.log --log-level ERROR
This command will start copying our files from OneDrive into our hard drive.
As you see, the command will show not only
--progress but also will record in
--log-file any errors that faced during accessing files.
After successful command, it’s always worth checking the logs.
The following command will copy our “Shared Documents” folder on our SharePoint site.
rclone copy --progress --ignore-size --ignore-checksum --update o365sharepoint:"Shared Documents" /media/pi/F05C-FB86/SharePoint --log-file=/media/pi/F05C-FB86/SharePoint --log-level ERROR
Apart from the above
--log-file there is option to
--ignore-checksum to fix some annoyances with rclone and SharePoint via Webdav.
In such a way we got a great backup system of our files in Microsoft 365 Cloud.
Excluding the cost of setting this up, raspberry pi and external hard drives (and electricity) there are no subscriptions to pay, and we go full control over our files.
I have been using it for years now and I use it at least twice over that time to restore some crucial files.
Additionally, we can add local encryption, to protect our drives, but that’s a different story.