Our Blog

Using pgTAP to automate database testing

By Josh Tolley
March 16, 2022

Photo from PxHere

Recently I started learning to tune pianos. There are many techniques and variations, but the traditional method, and the one apparently most accepted by ardent piano tuning purists, involves tuning one note to a reference, tuning several other notes in relation to the first, and testing the results by listening closely to different combinations of notes.

The tuner adjusts each new note in relation to several previously tuned notes. Physics being what it is, no piano can play all its tones perfectly, and one of the tricks of it all is adjusting each note to minimize audible imperfections. The tuner achieves this with an exacting series of musical intervals tested against each other.

Databases need tests too

One of our customers needed to add security policies to their PostgreSQL database, to limit data visibility for certain new users. This can quickly become complicated and ticklish, ensuring that the rules work properly for the affected users while leaving other users unmolested.

This struck me as an excellent opportunity to create some unit tests, not that there’s any short supply of good opportunities to add unit tests! This is not just because it helps …

Automating reading the screen and interacting with GUI programs on X Window System

By Constante “Tino” Gonzalez
March 10, 2022

A while back, Google Earth made some changes to the layer select menu in the sidebar, which broke a program that toggles the 3D imagery on VisionPort systems. These run the X Window System (also known as X11, or just X) on Ubuntu Linux.

In looking for a workaround, I found that shell scripts can fully interact with GUI apps using the xdotool, xwd, and convert commands. This script would send a series of keystrokes to open the sidebar, navigate the layers menu, and toggle the box for the 3D buildings layer.

Changing the series of keystrokes to match the new number of layers should have fixed the issue, but there was more to this script. The next part of the script would take a screenshot, crop the checkbox, and compare it to saved files of other cropped boxes. Fixing this part of the script required correcting the positions of the captures and replacing the reference files with ones that pictured the updated Google Earth checkbox states.

Here I will explain how the script works and how we changed it so that it no longer needs these reference files and ultimately runs faster.

Overview of how the script works

xwd takes a screenshot of a window on the screen. convert transforms the …

Database Design: Using Documents

By Emre Hasegeli
March 9, 2022

Using documents in relational databases is increasingly popular. This technique can be practical and efficient when used in fitting circumstances.

Example

Let’s start with an example. Imagine we are scraping web sites for external URLs and store them in a table. We’ll have the web sites table to store the scrape timestamp and another table to store all of the references.

CREATE TABLE web_sites (
    web_site_domain text NOT NULL,
    last_scraped_at timestamptz,

    PRIMARY KEY (web_site_domain)
);

CREATE TABLE refs (
    web_site_domain text NOT NULL,
    ref_location text NOT NULL,
    link_url text NOT NULL,

    PRIMARY KEY (web_site_domain, ref_location, link_url),
    FOREIGN KEY (web_site_domain) REFERENCES web_sites
);

We do not need to bother adding an id to the web_sites table, because we assume there won’t be too many of them. The domain is small and more practical to use as an identifier. If you are curious about advantages of using natural keys, see my previous article.

Normalized Tables

There may be many thousands of unique URLs for a single web site and other web sites may refer to the same URLs. To try to minimize the storage, we can keep the …

Using a YubiKey as authentication for an encrypted disk

By Zed Jensen
March 7, 2022

Image by Silas Köhler on Unsplash

Recently I built a small desktop computer to run applications that were a bit much for my laptop to handle, intending to bring it with me when I work outside my apartment. However, there was an immediate issue with this plan. Because this computer was intended for use with sensitive information/​source code, I needed to encrypt the disk, which meant that I’d need to enter a passphrase before I could boot it up.

I didn’t really want to haul a keyboard and monitor around with me, so I came up with an alternative solution: using a YubiKey as my method of authentication. This allowed me to avoid the need to type a password without giving up security. In this post I’ll show you how you can do the same.

Preparation

First off, you need a YubiKey, if you don’t have one already. I ended up getting the YubiKey 5C NFC.

While I waited for my YubiKey to arrive, I installed Ubuntu 20.04 with full-disk encryption (using the default option of LUKS, or Linux Unified Key Setup) on the computer. I set a passphrase like normal—the process I describe in this post allows access with either this passphrase or the YubiKey.

Next, there were two packages …

Optimizing media delivery with Cloudinary

By Juan Pablo Ventoso
March 1, 2022

I remember how we needed to deal with different image formats and sizes years ago: From using the WordPress-style approach of automatically saving different resolutions on the server when uploading a picture, to using a PHP script to resize or crop images on the fly and return the result as a response to the frontend. Of course, many of those approaches were expensive, and not fully optimized for different browsers or device sizes.

With those experiences in mind, it was a nice surprise for me to discover Cloudinary when working on a new project a couple of months ago. It’s basically a cloud service that saves and delivers media content with a lot of transformations and management options for us to use. There is a free version with a usage limit: Up to 25K transformations or 25 GB of storage/​bandwidth, which should be enough for most non-enterprise websites. The cheapest paid service is $99 per month.

Here’s a list of the image features we used on that project. I know they offer many other things that can be used as well, but I think this is a good start for anyone who hasn’t used this service yet:

Resizing and cropping

When you make a request for an image, you …

String Processing in SQL, the Hard Way

By Josh Tolley
February 28, 2022

Photo by Kristy Lee

When I was a kid, my family visited a prehistoric cliff-dweller settlement, reconstructed and preserved as a museum exhibit. The homes were built in caves in high sandstone cliffs, and though I followed a well-marked path to visit them, the original inhabitants climbed in and out via holes cut into the rock face. Apparently to deter unwanted visitors, the builders carefully spaced the foot holds so that climbers needed to start their ascent on the correct foot. Someone who didn’t know the secret of the path, or who carelessly forgot, might get halfway up the cliff and discover he or she couldn’t stretch to the next foot hold and had to turn back.

The other day I found myself confronted with several comma-separated lists of strings, and a problem: I needed the longest possible substring from each, starting from the beginning, ending in a comma, and shorter than a certain length, and I needed to do it in the database. “Easy peasy,” you may say to yourself, if you have a little SQL programming experience and a penchant for cheesy clichés, and you’d be right except for one thing: I let my mind wander a little too far down into the …

Migrating a Node.js app database from MongoDB to PostgreSQL

By Phineas Jensen
February 25, 2022

Recently I worked on a web app designed for tracking, editing, and reporting on archaeological survey data. When I joined the project, it used React on the front-end, Node.js on the back-end, GraphQL as the API interface, and MongoDB as the database. For the most part, this architecture and choice of technology worked well, and I was quickly able to make meaningful contributions to the app both in the user interface and in the backend. However, after a while, things started to get difficult and we began to consider why we were using MongoDB, and what might be better.

MongoDB vs. PostgreSQL

First, why was MongoDB difficult to use in our case? The biggest difficulty was using their query language for certain aspects of our database. Part of the database involved representing artifacts found at a site using a specific type hierarchy defined by the state government. Representing that hierarchy within Mongo was difficult, and querying it even more so. It was hard to find how to query a recursive structure using the MongoDB query language.

As I worked on this, scouring the documentation and Q&A websites to find ways to query this data efficiently, I began to realize that it was a …

Data Reporting with EpiTrax

By Shannon Sandall
February 16, 2022

Early COVID-19 form-building in Kansas

In March of 2020, many states struggled with disease surveillance systems that were not ready for the COVID-19 pandemic. States were forced to wait lengthy periods for their vendors or systems administrators to stand up customized forms for COVID-19 investigations.

State of Kansas informatics staff were able to configure their form within a matter of hours. When the first COVID-19 records came rolling in, the state was ready to go. The Kansas Department of Health and Environment (KDHE) was using EpiTrax, a National Electronic Disease Surveillance System (NEDSS) compliant application created by the state of Utah.

EpiTrax contains a powerful form-building utility that was easily customizable, allowing jurisdictional users to create forms for investigation and reporting purposes. Users could configure questions and value sets based on configurable lists, cutting down on free-text entries and ensuring data quality. Forms could be attached automatically or manually and assigned to specified agencies and conditions.

Screenshot of the form builder

December 2021 updates

Flash forward to December 2021. State of Utah developers rolled out an update …