Our Blog

EMSA: Electronic Messaging Staging Area

By Elizabeth Garrett Christensen
October 30, 2020

Photo by Lee Roberts, CC BY-SA 2.0

End Point has been involved with developing disease surveillance software for over a decade. One component of that work is EMSA, short for “Electronic Messaging Staging Area”.

The EMSA system we’re currently using was developed by a consortium of states led by Utah, Kansas, and Nevada. It is a PHP web application and can be hosted in the cloud or on-premises as you would any other PHP application.

EMSA:

• Is an ingest system for HL7 or CSV-formatted lab reports from doctor offices, labs, and other official reporting groups.
• Allows exporting XML output to your disease surveillance system.
• Is a web console for managing messages.
• Allows you to see messages that are malformed.
• Has configurable rules for disease types and message routing; some messages may go directly into a surveillance system while others may be stored for later.
• Has configurable “automated workflows” to validate message information and perform tasks.
• Can be connected to a BI (Business Intelligence) tool for reporting, or the database can be queried directly.
• Works with integration packages like Rhapsody to bring in data from other systems.

End Point has also made custom …

Detecting gaps in time-series data in PostgreSQL

By David Christensen
October 26, 2020

Photo by Phoebe Baker

A client has a number of data feeds that are supposed to update at regular intervals. Like most things in the universe (thanks, entropy!) this does not always end up working out the way we want. We recently discovered that some of the data had not loaded as expected, and were brought in to assess the extent of the issue.

There were 2 main feeds that had issues, with different types of time data. The first had date-based data with batches and the second had full-range timestamps. We will examine each type individually, since they have similar — but not the same — characteristics.

Date-based, with batches

In the first data feed there was a table which tracked which files had been loaded by file date and batch number (there was a morning/​evening batch designated by a batch field with either A or B). Since the files that had not loaded successfully did not have entries in this table we could find that one or both of the day’s batch files would be missing, but since the table tracked those which did get loaded, we needed to turn this list into something useful.

Any time I find myself considering sequences of data, I remember my friend generate_series(). While …

ROS 2 Migration

By Matt Vollrath
October 23, 2020

Migrating from ROS 1 to ROS 2 is much more than a “flip of a switch”. The internals of ROS are fundamentally changed, client libraries and their interfaces have been rewritten. The benefits of these changes are significant for users embedding ROS in a product.

Some of these benefits include:

• First-class support for real-time and embedded platforms such as microcontrollers
• Higher reliability under less-than-ideal network conditions
• Distributed, cooperative robotics systems without a single “master”
• More prescribed patterns for building systems with ROS

Because official ROS 1 support ends in 2025, the clock is ticking for all ROS 1 users to migrate their projects to ROS 2. While the amount of work required to port a project scales with its size, codebases of all sizes should have a migration plan.

By carefully approaching your ROS 2 migration, disruption to ongoing development and maintenance of the project can be minimized while maintaining the mission critical reliability required in the robotics domain.

Dependencies

If your project depends on any of the numerous free open source ROS packages, they will need to be ported to ROS 2. Much progress has been made here, but it is not …

The Pragmatic Programmer book, 20th anniversary edition

By Jon Jensen
October 16, 2020

The Pragmatic Programmer is a now-classic book about software development, first published in 1999. It is old enough that it predates the famous Agile Manifesto of 2001. The authors of the book, Andy and Dave, were two of the 14 authors of that manifesto.

For its 20th anniversary in 2019, Dave and Andy created a new edition. They updated things that had become dated, such as mentions of programming languages, tools, operating systems, websites, etc. That was, I imagine, the easy part. They went on to extensively revise the entire text and incorporate new lessons learned in the past two decades.

A classic

This book is part of our company’s “work philosophy canon” that I ask every software developer at End Point to read, so for that reason and others I wanted to be familiar with the new edition and make sure it is still something I want to recommend so broadly.

The book is also required reading for university courses at Cornell (CS 3110: Data Structures and Functional Programming) and the University of Washington (CSE 331: Software Design and Implementation), probably among others!

I first read this book 19 years ago and really enjoyed it. I found that it clearly expressed many …

Python concurrency: asyncio for threading users

By Matt Vollrath
October 5, 2020

Photo by Adrian Schwarz

You’ve probably heard this classic software engineering mantra:

Concurrency is hard.

The undeniable fact is that an entire category of software bugs, known for being elusive and frustrating to reproduce, is gated behind the introduction of concurrency to a project. Race conditions, mutual exclusion, deadlock, and starvation, to name a few.

Most programming languages with concurrency features ship with some or all of the classical concurrency primitives: threads, locks, events, semaphores, mutexes, thread-safe queues, and so on. While practically any concurrency problem can be solved with this toolkit, let me share a relevant life mantra:

Just because you can doesn’t mean you should.

In the case of Python, you have access to a standard library alternative to threading, which factors out many of the trickier parts of concurrent programming: asyncio. Many existing applications of Python threads can be replaced by asyncio coroutines, potentially eliminating many of the difficulties of concurrency.

Understanding the differences between asyncio and threading can help you make informed choices about which to apply and when, so let’s take a closer look.

The …

Using CTEs to do a binary search of large tables with non-indexed correlated data

By David Christensen
October 2, 2020

Photo by Colin Knowles, used under CC BY-SA 2.0.

Query optimization can take different forms depending on the data represented and the required needs. In a recent case, we had a large table that we had to query for some non-indexed criteria. This table was on an appliance that we were unable to modify, so we had to find a way to query efficiently without indexes that would have made it easier.

The straightforward approach for this query was something along these lines:

SELECT accounts.*
FROM accounts
JOIN logs
    ON accounts.id = logs.account_id
WHERE
    logs.created_at BETWEEN $1 - 'interval 1 minute' AND $1 + 'interval 1 minute' AND
    logs.field1 = $2 AND
    logs.field2 = $3
FETCH FIRST ROW ONLY

Unfortunately, none of the fields involved in this query were indexed, nor could they be, due to our access level on this database system. This lack of indexes means that our query against those fields would end up doing a sequential scan of the whole table which made things unacceptably slow. This specific table held time-series data with ~ 100k records per 1 minute period over a period of several weeks, which meant we were dealing with a lot of data.

While we could …

An introduction to automated testing for web applications with Symfony

By Kevin Campusano
September 22, 2020

Testing is an immense topic in software engineering. A lot has been written and a lot of experience has been collected on it by the greater software development community. There are many different tests, techniques, approaches, philosophies, strategies.

With such a big topic, it would be futile to try touching on every aspect of it in this article. Instead, I’ll try to take a pragmatic approach and discuss a testing strategy I’ve found success with in the past as well how much testing is necessary before I feel comfortable putting code into production. This article could also serve as a sort of introduction to automated testing using the Symfony framework as a vehicle to explore various types of testing without diving too deep into edge cases or framework specifics, and instead leaning more into the concepts and design decisions that go into writing them. Still, we’ll make sure to have a running and competent test suite by the end.

So we’re going to talk about automated testing, which in its own right is a very important part of the larger discipline of software testing. It’s also a topic that, as a developer (and as such, responsible for implementing this type of tests), I’m …

Liquid Galaxy at the Nano Museum in Seoul

By Dave Jenkins
September 17, 2020

We’re excited to share the news of another great project End Point has launched via our partner in South Korea! The Nano Museum in Seoul has added a brand new 21-screen Liquid Galaxy as part of their exhibits. This huge video wall is interactive and includes pre-programmed flights around the world, deep dives into Google Street View at select locations, and the ability to fly the screens with a 6-axis joystick and touchscreen.

This project presented some technical challenges for our hardware team: the 21-screen layout is 3× our normal 7-screen layout (but all very doable). For this configuration, we deployed an “LGOne” server stack which has a head node server for the core applications, media storage, and overall management. It also has a large display node server with multiple Nvidia video cards to power the displays. For this large array of screens, we are able to ‘bridge’ the video cards together (not unlike a RAID array for video cards) to produce multiple hi-resolution video outputs. These video outputs then go to the screens, where they are tiled by the displays’ own built-in capabilities.

We wrote these specific configurations in our build lab in Tennessee, then shipped …