PostgreSQL

Introduction PostgreSQL (and other RDBMS) have entity types such as functions, views, materialized views, triggers, and policies. These entity types do a lot of useful work, and a well designed database will use them heavily. These entity types are not typically considered part of the SQLAlchemy ORM, and by extension, the data model itself. By default, Alembic has no functionality to detect the creation of new entity types when it autogenerates migration files. The default Alembic ORM also has no functionality to define these entity types with classes. ...

Introduction As the scope of a data model (and by extension, its downstream APIs) changes, it will need to be updated and expanded to account for this new scope. When changes are made to a data model, especially to a model that is split across development and production environments, schema drift becomes a constant problem that looms in the background. Typical coding workflows are managed by version control as an industry standard. Database schemas, however, are typically not submitted to a version control system. As different team members collaborate on a data model split across different environments, the possibility of schema drift gradually grows in an environment where the data model is not tracked in a central repository. ...

Introduction Data Modeling Data modeling is the process of creating a representation of data that defines the way it is structured and used in complex systems. When it comes to relational databases, designing a normalized data model is essential for efficient database operation, and to make your data consistent and clean. With a fully normalized data model, data in your relational database becomes: Understandable A good data model can decompose the complexity of real-world systems and the data they generate into a relational model that is easy to read and comprehend, and standardizes the relationship of one data domain with every other data domain. ...

Introduction pgBackRest is an open source tool that allows you to perform automated backup and restore operations for a PostgreSQL server. It allows you to take both full and incremental file system backups of your data. With pgBackRest, you can set up multiple repositories for your backups, both local and remote. You can further customize the configuration to take backups at specific times of the day so that regular operations are not affected, and backup retention and rotation can be customized as needed. ...

Introduction The HDFS (Hadoop Distributed File System) and PostgreSQL databases are both powerful tools for data storage, queries and analyses. Each have their own unique strengths that make them well suited for specific tasks. The HDFS, being distributed across several computing nodes, is robust and amenable to storing massive datasets, provided your computing infrastructure has the prerequisite width (the number of nodes in your cluster) and the depth(the available memory on each individual node). The HDFS is optimized for batch processing of massive datasets, making it suitable for big data applications like data warehousing, log processing, and large-scale data analytics. In fact, Spark, the HDFS’ natural companion, has it’s own machine learning library MLlib, making large scale data analytics very much possible. ...