Introduction pgBackRest is an open source tool that allows you to perform automated backup and restore operations for a PostgreSQL server. It allows you to take both full and incremental file system backups of your data. With pgBackRest, you can set up multiple repositories for your backups, both local and remote. You can further customize the configuration to take backups at specific times of the day so that regular operations are not affected, and backup retention and rotation can be customized as needed. ...

Introduction The HDFS (Hadoop Distributed File System) and PostgreSQL databases are both powerful tools for data storage, queries and analyses. Each have their own unique strengths that make them well suited for specific tasks. The HDFS, being distributed across several computing nodes, is robust and amenable to storing massive datasets, provided your computing infrastructure has the prerequisite width (the number of nodes in your cluster) and the depth(the available memory on each individual node). The HDFS is optimized for batch processing of massive datasets, making it suitable for big data applications like data warehousing, log processing, and large-scale data analytics. In fact, Spark, the HDFS’ natural companion, has it’s own machine learning library MLlib, making large scale data analytics very much possible. ...

What is Mamba? Mamba is intended to be a drop-in replacement for /reimplementation of conda (written in C++). Mamba has been something that I have implemented into all of my pipelines, since it trivializes the package management process. I do almost all of my work within the context of containers/virtual environments, and mamba makes my work life so much easier. Previously, I used to use Conda as my package manager of choice, relying on it to cut down the amount of time I would need to build an environment using pip as an installer. Once I discovered Mamba, however, I never went back to Conda. ...

Introduction Docker is a container service that we have discussed previously in my blog posts. The default storage location for Docker is at /var/lib/docker. Speaking from experience, as images and containers are built over a period of time, especially if there are multiple users using the Docker service, the root filesystem can run into issues where the size of the Docker storage directory can cause potential out-of-space crises, and significantly deteriorate overall system performance. ...

Hive Metastore Hive, a powerful data warehousing system built on top of Hadoop, relies on a component known as the metastore to efficiently manage metadata about the data stored within it. This metadata is crucial for organizing, querying, and processing data in Hive. In this blog post, we’ll explore the role of the Hive Metastore and the significance of selecting the right relational database management system (RDBMS) for it. Metadata Management The Hive Metastore serves as a central repository for storing and managing metadata related to datasets within Hive. This metadata includes essential information such as: ...