A personal tale of recovering from a catastrophic situation.
What Happened? #
I went to bed Sunday night after watching a movie with my family. The in-laws were visiting and my son had gone to sleep a while before.
All was well. I had ripped a movie and decided to set up my transcoding container on my server to run overnight while utilization was low. This was nothing new.
Monday morning rolled around. The grind was about to begin. I opened up my laptop to check if my transcodes were done. 503 Service Unavailable.
I navigated to another core service. 503 Service Unavailable.
I browsed to my server’s home page. No response.
SSH for a remote connection. Not responding.
Reboot. No luck.
Plug in keyboard and display then reboot. Nothing.
The Server #
My home server is the core, the home if you will, of my data. It serves as a data repository and media streaming device for my whole family. I offer multiple services on it like Immich (photo backup), Nextcloud (documents), and Jellyfin (media streaming).
Before this incident, my server had been up for nearly 6 months. That last outage was an update. Prior to that, it was up for almost an entire year without a hiccup.
Unraid1 is the Operating System (OS) of choice for me. It was chosen to handle one particular case that TrueNAS2 and other ZFS3 style NAS offerings do not: Disaster Recovery of individual drives. In the event of a total systems failure, where the array is irrecoverable, I can pull out each individual hard drive to access its data because they are plain old XFS4 volumes. In ZFS you cannot do this; you have to boot the array as a unit and access it together because they are joined permanently and the drives mean nothing on their own.
An important note about Unraid’s design decisions. While they have changed their tune on this, historically they only supported booting the OS off of an external flash drive. This meant you did not need to use up one of your valuable and possibly limited SATA ports for an OS boot drive, but it comes at the risk of USB device failure. That failure rate is often higher than internal drives.
What Next? #
I decided to start investigating full recovery of the system. It was early on a Monday and no one was awake. I had not yet finished my first cup of coffee and I really did not want to be doing a full rebuild of the system. The good news is that Unraid offers a free online backup of your boot drive configuration.
It costs them next to nothing because the config folder in your boot drive is merely 1-2MB. I went to their Unraid Connect Portal and logged in with my account. My server displayed as offline as expected. I tried to navigate to managing my server but it would not go.
Perplexed as to why I could not click on my server in the sidebar list, I realized they moved the boot drive backups to a new option in the same list separate from the servers. “That’s convenient,” I thought to myself. I clicked on the new boot drive backups. Blank screen. No backups found.
Surely, I thought, the team renamed the location in the last couple of months, somehow my account just got missed in the migration, and they have my backups in a separate location. So I sent Unraid Support a ticket requesting some assistance finding my backups to resolve my critical outage.
Interlude #
At this point I had to put everything down and go work my real, paying job. Let me take this moment to express my absolute gratitude and displeasure with hosting a home server.
I love having my data at home, away from the privacy breaching internet around us. From Google training AI on your documents and emails5 to trackers hoarding your data for advertising and government sharing6, the internet is becoming a dark place to wander alone. Keeping your data in cloud hosted offerings is a concern for those who find it unethical or immoral to read and use someone’s personal emails, data, and habits to create profiles and make a business out of.
The power of a home server comes in to save the day. It provides the ability to have Network Attached Storage (NAS)7 and run some services that go beyond simple local file shares to more powerful photo processing, local AI, security systems that run in-home, and more.
Should everyone host their own home server? In short, no. With great power comes great responsibility. Data can be turned into knowledge and knowledge is power. But the protection and maintenance of that data can become critical.
For example, this post is about me having an outage for the better part of a day. Luckily, no one needed the photos or documents on the server for that day. But if someone did need that data for something like a doctor’s appointment or banking issues, they would have to wait till I recover the server or the data.
While I dislike many of the practices of big tech, they excel at service availability. In decades of use, I’ve almost never seen Gmail, Google Drive, or Search go down for a full day. Even with giants like Microsoft, Meta, or Dropbox, outages are rare and usually brief.
The teams that provide these services are paid a lot of money to ensure highest uptime (availability) and reliability (recovery from outages, failure/high availability, multi-region deployments). They can have a Gmail server running on both coasts of America so that if one gets hit with a bomb, Lord forbid, the other would continue to operate and users on both coasts could use the one remaining instance. I do not have that in my basement.
Stress and Faith #
While at work, I thought it best to avoid panicking or getting anxious over the issue. Yes, we store valuable data on the server and the inability to recover that data would be awful. But ultimately, I have faith in a God who cares.
Luke 12:22-34 tells us about how God cares for the smallest details of our lives. This does not mean we are exempt from trouble, but rather that we are cared for in a greater sense, both here and in eternity.
In this case, I prayed about the situation and asked for clarity on what to do. While I heard no audible voice, the idea dropped into my head to look into recovering the boot drive manually. More on that in a minute.
If you are stuck in a hard place or you have something going on that makes you anxious, remember who you are and who God is. If you do not know him, I recommend you read Luke 12:22-348 along with some other passages like John 3:16 and honestly any of the four gospels in their entirety. God is perfect love and he wants to know you personally and have you know him personally.
Recovery #
The bad news is that the Unraid Support team had nothing to offer to me. We shared a few back and forth posts to the support thread, but all were fruitless to my recovery effort.
After a long day at work, I returned home and found the inspiration, alluded to earlier, to check a very specific page of the Unraid documentation. The Unraid documentation includes a little section about recovering a non-bootable drive. This is where they explain that the boot device in Unraid contains the necessary files to boot Unraid itself as well as load your configurations. It is also where they tell you that all of the things that are unique to your server, that is your settings, array configuration, shares, and more, are all stored in a single directory called config.
With that knowledge, I plugged the USB boot drive into my laptop and copied the config folder off of the drive. I reformatted the drive and ran it through Unraid’s configuration tool to make it the latest OS version available.
The last steps were upon me. It was time to put the config folder back onto the drive, shove it in the server, and pray for a clean boot. With that, everything worked. It was as simple as could be. The server booted. No data lost. All docker containers and VMs were exactly where I left them and their respective configs too.
Backup, Backup, Backup! #
Praise God it was an easy solution! But this poked a new hole in my entire data recovery plan. You see, in the worst case scenario of not being able to recover the server, I have a backup. In fact, I follow the 3-2-1 backup strategy9 for all of my important data.
3 copies of the data, in 2 different media, and 1 is offsite. That is great for the critical data, but it does not account for the server OS itself. Now that I know about the plain config directory on that boot drive, I plan to include that in its own backup.
Another issue this exposed is that while I can restore from my backups, the backup tool is running on the server itself. The data is on a separate drive that can easily be plugged into another device, but it is formatted by the backup tool Kopia10. I need Kopia running in order to view and restore the backups. My next upgrade will be to properly run the Kopia instance on a separate device and to monitor its health separately from the Unraid server.
What does all of this mean? The pathway to recovering from a disaster scenario includes having a plan. But remember, the best plans can fall apart quickly. Even when you think you have covered all of the possibilities, something else will get through your defenses. A little bit of faith can go a long way in troubling times, whether it’s for saving your data or keeping you calm during the process.