The Pragmatic Programmer book, 20th anniversary edition
By Jon Jensen
October 16, 2020
The Pragmatic Programmer is a now-classic book about software development, first published in 1999. It is old enough that it predates the famous Agile Manifesto of 2001. The authors of the book, Andy and Dave, were two of the 14 authors of that manifesto.
For its 20th anniversary in 2019, Dave and Andy created a new edition. They updated things that had become dated, such as mentions of programming languages, tools, operating systems, websites, etc. That was, I imagine, the easy part. They went on to extensively revise the entire text and incorporate new lessons learned in the past two decades.
A classic
This book is part of our company’s “work philosophy canon” that I ask every software developer at End Point to read, so for that reason and others I wanted to be familiar with the new edition and make sure it is still something I want to recommend so broadly.
The book is also required reading for university courses at Cornell (CS 3110: Data Structures and Functional Programming) and the University of Washington (CSE 331: Software Design and Implementation), probably among others!
I first read this book 19 years ago and really enjoyed it. I found that it clearly expressed many …
books programming development
Python concurrency: asyncio for threading users
By Matt Vollrath
October 5, 2020
You’ve probably heard this classic software engineering mantra:
Concurrency is hard.
The undeniable fact is that an entire category of software bugs, known for being elusive and frustrating to reproduce, is gated behind the introduction of concurrency to a project. Race conditions, mutual exclusion, deadlock, and starvation, to name a few.
Most programming languages with concurrency features ship with some or all of the classical concurrency primitives: threads, locks, events, semaphores, mutexes, thread-safe queues, and so on. While practically any concurrency problem can be solved with this toolkit, let me share a relevant life mantra:
Just because you can doesn’t mean you should.
In the case of Python, you have access to a standard library alternative to threading, which factors out many of the trickier parts of concurrent programming: asyncio. Many existing applications of Python threads can be replaced by asyncio coroutines, potentially eliminating many of the difficulties of concurrency.
Understanding the differences between asyncio and threading can help you make informed choices about which to apply and when, so let’s take a closer look.
The …
python performance
Using CTEs to do a binary search of large tables with non-indexed correlated data
By David Christensen
October 2, 2020
Photo by Colin Knowles, used under CC BY-SA 2.0.
Query optimization can take different forms depending on the data represented and the required needs. In a recent case, we had a large table that we had to query for some non-indexed criteria. This table was on an appliance that we were unable to modify, so we had to find a way to query efficiently without indexes that would have made it easier.
The straightforward approach for this query was something along these lines:
SELECT accounts.*
FROM accounts
JOIN logs
ON accounts.id = logs.account_id
WHERE
logs.created_at BETWEEN $1 - 'interval 1 minute' AND $1 + 'interval 1 minute' AND
logs.field1 = $2 AND
logs.field2 = $3
FETCH FIRST ROW ONLY
Unfortunately, none of the fields involved in this query were indexed, nor could they be, due to our access level on this database system. This lack of indexes means that our query against those fields would end up doing a sequential scan of the whole table which made things unacceptably slow. This specific table held time-series data with ~ 100k records per 1 minute period over a period of several weeks, which meant we were dealing with a lot of data.
While we could …
postgres database sql
An introduction to automated testing for web applications with Symfony
By Kevin Campusano
September 22, 2020
Testing is an immense topic in software engineering. A lot has been written and a lot of experience has been collected on it by the greater software development community. There are many different tests, techniques, approaches, philosophies, strategies.
With such a big topic, it would be futile to try touching on every aspect of it in this article. Instead, I’ll try to take a pragmatic approach and discuss a testing strategy I’ve found success with in the past as well how much testing is necessary before I feel comfortable putting code into production. This article could also serve as a sort of introduction to automated testing using the Symfony framework as a vehicle to explore various types of testing without diving too deep into edge cases or framework specifics, and instead leaning more into the concepts and design decisions that go into writing them. Still, we’ll make sure to have a running and competent test suite by the end.
So we’re going to talk about automated testing, which in its own right is a very important part of the larger discipline of software testing. It’s also a topic that, as a developer (and as such, responsible for implementing this type of tests), I’m …
testing symfony php
Liquid Galaxy at the Nano Museum in Seoul
By Dave Jenkins
September 17, 2020
We’re excited to share the news of another great project End Point has launched via our partner in South Korea! The Nano Museum in Seoul has added a brand new 21-screen Liquid Galaxy as part of their exhibits. This huge video wall is interactive and includes pre-programmed flights around the world, deep dives into Google Street View at select locations, and the ability to fly the screens with a 6-axis joystick and touchscreen.
This project presented some technical challenges for our hardware team: the 21-screen layout is 3× our normal 7-screen layout (but all very doable). For this configuration, we deployed an “LGOne” server stack which has a head node server for the core applications, media storage, and overall management. It also has a large display node server with multiple Nvidia video cards to power the displays. For this large array of screens, we are able to ‘bridge’ the video cards together (not unlike a RAID array for video cards) to produce multiple hi-resolution video outputs. These video outputs then go to the screens, where they are tiled by the displays’ own built-in capabilities.
We wrote these specific configurations in our build lab in Tennessee, then shipped …
visionport clients
COVID-19 Support for the Kansas Department of Health and Environment
By Steve Yoman
September 14, 2020
Kansas’s existing EpiTrax system
End Point has worked on Kansas’s disease surveillance systems since 2011. In 2018 we migrated them from their legacy TriSano application to the open source EpiTrax surveillance system created by Utah’s Department of Health. The new EpiTrax system had been in full production for about eight months when COVID-19 cases started to grow in the United States.
COVID-19: Help needed
In March 2020, the Director of Surveillance Systems at the Kansas Department of Health and Environment (KDHE) asked us at End Point to create a web-based portal where labs, hospitals, and ad-hoc testing locations could enter COVID-19 test data. While systems existed for gathering data from labs and hospitals, they needed a way to quickly gather data from the many new and atypical sites collecting COVID-19 test information.
Our approach
Since the portal was intended for people who were unfamiliar with the existing EpiTrax application, we were able to create a new design that was simple and direct, unconstrained by other applications. It required a self-registration function so users could access the system quickly and without administrative overhead, and users needed to …
casepointer epitrax clients case-study rails vue emsa
Introduction to BorgBackup
By Kannan Ponnusamy
September 10, 2020
What is Borg?
BorgBackup (Borg for short) is a ‘deduplicating’ backup program that eliminates duplicate or redundant information. It optionally supports compression and authenticated encryption.
The main objective of Borg is to provide an efficient and secure way to backup data. The deduplication technique utilized to produce the backup process is very quick and effective.
Step 1: Install the Borg backups
On Ubuntu/Debian:
apt install borgbackup
On RHEL/CentOS/Fedora:
dnf install borgbackup
Step 2: Initialize Local Borg repository
Firstly, the system that is going to be backed up needs a new designated backup directory. Name the parent directory ‘backup’ and then create a child directory called ‘borgdemo’, which serves as the repository.
mkdir -p /mnt/backup
borg init --encryption=repokey /mnt/backup/borgdemo
Step 3: Let’s create the first backup (archive)
In Borg terms, each backup instance will be called an archive. The following demonstrates how to backup the ‘photos’ directory and designate the archive as ‘archive_1’.
borg create --stats --progress /mnt/backup/borgdemo::archive_1 /home/kannan/photos
Note: the archive label for each backup run needs to be …
sysadmin storage backups
Rclone: upload to the cloud from your command line and much more
By Ardyn Majere
September 9, 2020
The Swiss army knife of storage
Cloud storage providers like Google Drive are great solutions for storing files. You can upload your data and not worry about maintaining a separate system to host it, or all the security hassles that can bring. However, very few major cloud storage providers offer a command line interface or any other official way to upload without using their web interface or closed-source binary tools, if they even offer that.
This obviously makes uploading files from servers difficult, but not impossible if you know the right tools.
About a year ago Jon Jensen penned a blog post about gdrive, a Google Drive command-line tool. However, due to changes with Google’s Drive security, that tool no longer works. This led me to look for a replacement.
Our use case
Recently I had to put some large files in to long term storage on Google Drive, since we needed the local space back. We wanted to retain the data, but didn’t foresee needing to access it for some time, if ever. Google Drive was a good solution for us, but the problem became how to get it there.
The files were too big, and some of them were not stored sparsely—empty space was tacked on to the disk images. We …
sysadmin cloud storage