Microsoft 365 (SharePoint & OneDrive) backup for free with Raspberry Pi and Rclone

Contents

When I started working for my employer, they used an on-site server and file storage with overnight backup to external drives. All been limited by the internet speed (which in England is quite crazy that business parks rarely got access to an inexpensive fibre connection, whereas at home you can have cheap Full Fibre), but when things change (or rather the company need to invest in change), we moved it into the cloud.

We chose Microsoft 365 as our provider.

Once the migration was completed and people were used to using the new way of accessing their files (which haven’t been different with the integration of the OneDrive app and Windows 10, apart from links to files), I needed to find a method, how to have a backup of all files in case of emergency (or accidental removal of important files by an employee).

Despite that Microsoft provides an option to restore files for a short period (typically 30 days), sometimes it may be too short and before you notice that files are missing it will be too late. It’s better to have a backup (even an old version of the file) than don’t have it at all.

Without going into costly online backup solutions I decided to build my backup system and I made it.

For this purpose, I used Raspberry Pi with Raspberry Pi OS (formerly RaspbianOS). Initially, I used Pi 3, but moved to Pi Zero W (currently Pi Zero 2 W), as Pi 3 was wasting itself for this purpose. For storage, I used an externally connected hard drive (2.5" 1 TB hard drive in USB 3.0 case) connected via micro USB to USB converter. As a power supply, I used 2A @ 5V iPad charger with a USB to micro USB cable.

Quick initial setup, a couple of commands and the backup system was ready to go. I have been using it successfully since mid-2019 and now I would like to share it with you.


You will need:

  • Raspberry Pi with 8GB (min.) micro SD Card (for the operating system);
  • USB Connected external Hard Drive or Memory Stick (for external storage, capacity depends on your needs);
  • A charger that will provide enough power (especially with an external hard drive);
  • Raspberry Pi OS (previously known as Raspbian OS);
  • Keyboard and mouse for initial setup along with Screen connected via HDMI cable.

As keeping Pi connected to screen via HDMI is not always viable, we can configure a Remote Desktop that utilise Microsoft RDP protocol. Read section Remote Desktop.


Raspberry OS

Firstly we head to the Raspberry OS website and download the desired operating system.

We will need a system with a desktop environment, as despite that backup and further management can be done through the terminal, we need a web browser to get an authentication token.

From the above desktop download ‌Raspberry Pi OS with desktop.

We will need additional software called balenaEtcher to put a downloaded image into our SD card.

We can use dedicated Raspberry Pi Imager to put a system into our SD card, but I prefer the other way.

Once our system image is downloaded, we connect our SD card via card reader to our operating system.

balenaEtcher

From the balenaEtcher window select Flash from the image and point to the downloaded Raspberry OS file (compressed, no need to extract).

Select your card by clicking Select target and confirm all by clicking Flash! button.

Once the job is done we need to put our card into Raspberry Pi and boot it up.

In the initial stage I am connecting my Raspberry Pi to the network using Ethernet Cable. In the case of Pi Zero 2 W, we need to wait until the desktop is ready and connect to our WiFi network.

Once the desktop is up and running we got the option to update our system via GUI, but we can skip that. Also, at the initial stage, we can connect our Pi to a WiFi network to get rid of one cable.

Storage

Before we start going into configuration it’s worth setting up storage.

Let’s take our hard drive and connect it to our Raspberry Pi.

If we will get a Low voltage warning warning and constant lightning symbol, we may need to use a much more powerful charger. Normally, this warning is shown when we connect a new device and booting system.

To set up our hard drive, we need to have it formatted and ready for use.

My preferred tool for that is GParted which we can install from the terminal.

sudo apt install gparted

Once installed, we can access it from Pi Menu > System Tools > GParted.

From GParted select our hard drive from the top-right drop-down menu. Typically will it be /dev/sda/.

If there is no partition on a hard drive, we will be creating one.

If there are any, let’s remove them and start from the beginning.

I prefer to create partition exfat as this will allow you to connect it quickly to your Windows computer and work swiftly on saved files. Surprisingly exfat works very well in Linux despite being a Microsoft creation but GParted is not able to create it natively.

To add it, install it through the terminal:

sudo apt install exfat-utils exfat-fuse

In GParted create an unformatted partition (this will create /dev/sda1) and from terminal format it to exfat.

GParted - New partition

GParted - partition unformatted

sudo mkfs.exfat /dev/sda1

When you disconnect the hard drive and connect it back again you will see that it will be automatically mounted and ready to use.

Remote connection

Having a view on our desktop on a connected screen via HDMI we will be connecting to our device via the terminal.

We can use PuTTY for that if we are connecting from Windows or just through the command in the macOS terminal.

ssh pi@192.168.1.216

IP address we can read by hovering our mouse on network manager in the right-top corner of the Raspberry OS desktop.

Before we do that, we need to enable it.

We need to run the terminal on our desktop and go into raspi-config.

sudo raspi-config

In the Configuration Tool, we head to 3 Interface Options > P2 SSH and on the question: Would you like the SSH server to be enabled? answering Yes.

We can finish at that stage and try to connect as above.

Default password will be raspberry. We can change it by typing passwd.

Once we are connected, we can move further.

Rclone

For the backup purpose, we will be using a package called rclone.

To install it we are using the following command:

sudo apt install rclone

Then we can go and set up a new config. We are doing this on the Raspbian OS desktop initially.

rclone config

rclone config

From the menu, we are creating a new remote connection (n). Let’s name it o365onedrive (for OneDrive) and later once again o365sharepoint for (SharePoint Document Library).

In the next step, we need to decide what service needs to be used.

For Microsoft, you will need to select Microsoft OneDrive. This is the one when you need to backup the User OneDrive folder. It will also require an authentication token (not a password), this is why we will need to do this with a graphic environment on Raspberry OS.

For SharePoint storage (Document Library) we need to select Webdav. For this purpose, we will need an app password (specially generated password, not the account password).

First thing first.


To make further steps much smoother, especially on devices like Pi Zero and using Remote Desktop I would recommend installing an alternative browser, instead using chromium, like midori.

sudo apt install midori

We can remove chromium from the system using the following command:

sudo apt remove --auto-remove --purge chromium-browser

OneDrive connection

rclone config - onedrive

Let’s select Storage> and type onedrive.

We don’t need to put anything in client_id and client_secret. When asked Edit advanced config? answering n for No.

When asked Use auto config? answer Yes.

This will open Chromium Web Browser where we need to log in with our Microsoft Account.

If that will not happens (especially when we using Remove Desktop rather than being connected to the screen using HDMI cable) we will be given a local address to paste into our web browser like:

http://127.0.0.1:53682/auth?state=XyZ-1fdsFSsdxSgdfsdFDS

When everything goes well, we will have a simple page with the word Success!.

rclone - OneDrive authentication - Success!

At this stage, we can close the web browser and go back to the terminal.

rclone - OneDrive Personal of Business

From the final option, we need to select what type of account it is. We are choosing 1 for OneDrive Personal or Business.

You may see SharePoint here as well, but about that later.

In the last step, we need to confirm which drive to use (there shall be one) but select 0 (zero).

One last step:

Found drive 'root' of type 'business', URL: https://org-my.sharepoint.com/personal/you/Documents
Is that okay?

We just confirm with y for Yes and once again y for Yes to confirm that all is fine and configuration will be saved.

In such a way, we got a configuration for OneDrive done. It’s time for SharePoint.

See the official guide of setting up OneDrive with Rclone.

However, before we go to the next step, let’s see if our configuration is working.

By executing the command rclone lsd o365onedrive: we shall bring a list of folders to our OneDrive.

Using the command rclone ls o365onedrive: will bring a list of files (long).

SharePoint connection

To create a backup of the whole SharePoint document library we need to use an admin account that has access to all files, even restricted to some users. Sometimes the user account may not be sufficient.

As you saw in the previous point, it was possible to add in the same method another clone configuration for SharePoint. However, this will be restricted to files, that the user (myself) has access to.

If we have a Super Admin account that by default has access to it, we need to use a different approach.

If copying what the user is allowed to, we can set up everything exactly like in point above, but instead of OneDrive Personal or Business we select Sharepoint site.

For this purpose, we will use Webdav.

See official guide of setting up Webdav with Rclone.

Once again let’s do rclone config

  • New (n) remote
  • name> o365sharepoint
  • Storage> and type webdav

Now we need to type an URL for our SharePoint site that we want to get access to and where we store our Document Library. Mine was something like that

https://company123.sharepoint.com/sites/Company123Site
  • vendor> type sharepoint
  • user> type email address of Super Admin
  • Password> select y and type your user password.

Because the password is stored hashed but in plain text, I highly recommend not using the User Password but instead generate App Password. Also, if we got implemented two-step authentication, the user password will not work here and why an app-specific password is required.

By executing the command rclone lsd o365sharepoint: we shall see the list of folders on our SharePoint site.

Typically, we will be interested in the folder “Shared Documents” where we are storing files that are used in the OneDrive app across company users.

Pre-backup steps

Now it’s time to execute commands that will start the backup process.

We don’t need to do that on the Raspberry OS desktop. We can make them via SSH.

Because when we execute a command we need to wait until it finishes (we cannot close the terminal window). I highly recommend installing the screen (sudo apt install screen`) and executing the command inside it.

When we close the terminal window, the command still will be working in the background, and we can always come back to it to check progress by reattaching the screen (screen -R).

Before we start, we also need to know where is our hard drive mounted by using the command df /dev/sda1.

Let’s go to our mount point (for example cd /media/pi/F05C-FB86) and let’s create two folders. One for OneDrive backup and one for SharePoint.

mkdir OneDrive
mkdir SharePoint

Now we are ready.

Backup

Run the screen command and execute the following:

rclone copy --progress o365onedrive: /media/pi/F05C-FB86/OneDrive --log-file=/media/pi/F05C-FB86/OneDrive.log --log-level ERROR

This command will start copying our files from OneDrive into our hard drive.

As you see, the command will show not only --progress but also will record in --log-file any errors that are faced during accessing files.

After successful command, it’s always worth checking the logs.

The following command will copy our “Shared Documents” folder on our SharePoint site.

rclone copy --progress --ignore-size --ignore-checksum --update o365sharepoint:"Shared Documents" /media/pi/F05C-FB86/SharePoint --log-file=/media/pi/F05C-FB86/SharePoint --log-level ERROR

Apart from the above --progress and --log-file there is an option to --ignore-sizeand–ignore-checksum` to fix some annoyances with rclone and SharePoint via Webdav.


In such a way we got a great backup system for our files in Microsoft 365 Cloud.

Excluding the cost of setting this up, raspberry pi and external hard drives (and electricity) there are no subscriptions to pay, and we go full control over our files.

I have been using it for years now and I use it at least twice over that time to restore some crucial files.

Additionally, we can add local encryption, to protect our drives, but that’s a different story.


Remote Desktop

As mentioned at the beginning, when our device is deployed and we experience any issues, sometimes is not viable (or possible) to connect our Pi to an external screen. In that case, we may want to connect remotely using Microsoft RDP (Remote Desktop Protocol).

To do that, from the terminal, we need to install the following package:

sudo apt install xrdp

When that’s done we can use the build-in into Windows operating system Remote Desktop app.

For macOS, we need to install the app from the AppStore

To connect, we need to know our Pi IP address (hostname -I), and when requested, use our pi user and password.

And that’s all.

Regards

Comments
Categories