Why is backing up so difficult, and why is simply copying files not enough?

Because important data can be lost due to malfunctions or user errors, it's crucial to back up your data regularly. However, simply copying files to another location isn't always enough for data backup. Software engineer Aleksandar Filipovsky explains the increasing challenges that arise as you set up backup systems, drawing on his own experiences.

Backups aren't simple
https://filipovski.net/2026/09/16/backups-arent-simple.html

Filipovsky quotes a saying passed down among system administrators: 'There are two kinds of people: those who have experienced catastrophic data loss, and those who will experience it.' This means that 'data loss happens more often than you think, and many people are not adequately prepared.'

Filipovsky himself had a close call with losing family photos. He had been collecting family photos on an external hard drive to free up space on his home PC, but his father formatted the hard drive to use as storage for his TV set-top box . As a result, the file index was deleted, and the hard drive appeared to be empty.

However, Filipovsky points out that the loss of the photo data wasn't solely the father's fault, but also due to the fact that the photos were kept in one place without being backed up, and that the set-top box didn't display a clear warning that formatting would result in data loss. He further questioned the assumption that someone unfamiliar with technology would understand what formatting meant.

Fortunately, the photos on the HDD were recovered, but Filipovsky says, 'The lesson I learned from this experience is not to keep important data in just one place.' HDDs can fail or be stolen, and even when stored unused, the magnetic state that holds the data can change. SSDs can also lose their charge, which can corrupt data. Therefore, the first thing you need to do is 'make a copy of your files in another location.'

However, having a copy doesn't solve everything. The connected drive could be encrypted by ransomware, and users might accidentally delete files or run scripts that overwrite everything with zeros. A system that simply reflects changes from the original drive would also reflect the results of accidental operations in the copy.

Therefore, what's important isn't just whether or not a copy is being made, but a 'mechanism to revert to a previous state.' Filipovsky argues that mirroring, such as RAID 1, which writes the same data to multiple drives, alone cannot fulfill this purpose, and that a 'snapshot' that saves the state at a specific point in time is necessary.

One of the challenges in saving snapshots is deciding 'how often to save them.' The goal of determining how far back in time data should be recoverable is called 'RPO (Recovery Point Objective),' and Filippovsky explains that 'the required RPO level varies, from less than 30 seconds for critical financial institutions to more than 24 hours for smaller businesses.' For example, when backing up a collection of family photos, one might decide to 'accept the loss of up to 6 days and 23 hours of data.'

However, once you decide on a saving frequency, storage capacity becomes an issue. Even if you create a snapshot every day, if you don't delete the old ones, you'll end up with 7 snapshots a week and 365 a year. This would require a tremendous amount of storage space just for the backup data, so you need to implement a system to delete older snapshots sequentially.

For example, one could consider an operation where 'only the most recent 14 days' worth of data is kept, and the oldest data is deleted each time a new one is created.' However, with this method, if data corruption goes unnoticed for more than two weeks, there's a possibility that a backup from before the corruption won't exist. On the other hand, keeping a full year's worth of backups is difficult in terms of storage capacity, and such detailed records of older periods aren't always necessary.

Therefore, Filipovsky introduces a method of 'keeping recent backups frequently, and saving older ones with wider intervals.' For example, an effective approach is to keep 14 days' worth of daily backups, 7 weeks' worth of weekly backups, and 12 months' worth of monthly backups. This method of saving snapshots in three generations—daily, weekly, and monthly—is called 'GFS (Grandfather-Father-Son).'

Furthermore, it reduces the duplication of saved content. Most of the files being backed up remain unchanged, while only a few are frequently updated. Therefore, by saving only the changed files while referencing already saved data, you can save storage space.

One method is ' hard linking ,' which references the same file entity from multiple locations. Because each snapshot references the same entity, even if you delete an old snapshot, the data referenced by other snapshots remains. Deduplication not only saves storage space but also reduces the amount of data transferred when transferring data to another machine. This is especially important when using cloud services as the storage destination, as the amount of data transferred directly impacts operational costs.

Filipovsky explains that this system can be built by combining the file transfer tool ' rsync ' with ' cron ' to perform tasks periodically. This backup method features incremental storage, which shares files that haven't changed since the last backup via hard links, and GFS-style version control. It also allows for adding destination machines. Furthermore, it can retain supplementary information such as file access permissions and ownership.

However, applying this mechanism to a home server environment presents a different problem. For example, if you run 10 Docker containers, the Docker containers may create files owned by the administrator (root), while the backup process is performed with the privileges of a regular user. In such cases, the files may not be readable, leading to backup failure.

Furthermore, databases can also present problems with simple file copies. To speed up processing, databases sometimes temporarily hold data in memory and then write it to disk in batches. Depending on the timing of the copy, an inconsistent state may be saved, potentially leading to restoration failure. Therefore, it becomes necessary to create a 'dump' of the database contents for backup purposes and to ensure that you have the necessary permissions to read the data storage area in Docker.

Attention must also be paid to the devices and locations where data is stored. If a particular HDD product experiences frequent failures, storing data on different types of storage media is a good countermeasure. Furthermore, storing copies in remote locations such as the cloud or a family member's home reduces the risk of losing everything due to power failures, floods, or fires. This approach of 'preparing three copies of data, storing them on two different types of media, and keeping one of them in a separate location' is called ' 3-2-1 backup .'

What is the '3-2-1 backup rule' that prevents the loss of important data in unforeseen circumstances? - GIGAZINE

However, if you choose object storage like Amazon S3 as a remote storage location, you cannot use the same mechanism as described above. Simply uploading files as they are will not transfer information such as access rights or ownership, and sending a large number of small files will result in high upload request costs.

Therefore, combining multiple files into a tar archive can reduce the number of uploads while preserving associated information. However, putting everything into one huge archive negates the advantages of using hard links to save only the changes. Filipovsky has also suggested splitting the files into chunks of around 50MB, but implementing this in a way that ensures security is not easy.

Filipovsky states, 'At this stage, the burden of continuing to develop it yourself is not worthwhile.' Instead, he recommends backup tools such as ' Borg backup ' and ' Restic .' These tools also support encryption, deduplication of files into chunks, and checksums to detect data corruption. Filipovsky says, 'It took a lot of trial and error by open-source developers to make these complex processes easy for users to handle.'

Furthermore, creating backups is meaningless unless you can actually test whether they can be restored. Therefore, in addition to setting up a backup system, it is also necessary to verify the restoration process, such as by performing a restore every six months.

Backups must preserve a past state in case data is accidentally deleted or corrupted, while minimizing storage space and costs, and ensuring data integrity and the security of the storage location. Furthermore, when you include the task of verifying whether the data can actually be restored, many challenges arise that go beyond simple copying. Filipovsky stated, 'Backups are difficult not only because you have to make copies of files, but also because you have to maintain a state where they can be correctly restored when needed.'