Our Blog

Creating a Messaging App Using Spring for Apache Kafka, Part 3

By Kürşat Kutlu Aydemir
May 21, 2020

Photo by Pascal Debrunner on Unsplash

This article is part of a series. The GitHub repository with code examples can be found here.

In this article we’ll create the persistence and cache models and repositories. We’re also going to create our PostgreSQL database and the basic schema that we’re going to map to the persistence model.

Persistence

Database

We are going to keep the persistence model as simple as possible so we can focus on the overall functionality. Let’s first create our PostgreSQL database and schema. Here is the list of tables that we’re going to create:

• users: will hold the users who are registered to use this messaging service.
• access_token: will hold the unique authentication tokens per session. We’re not going to implement an authentication and authorization server specifically in this series but rather will generate a simple token and store it in this table.
• contacts: will hold relationships of existing users.
• messages: will hold messages sent to users.

Let’s create our tables:

CREATE TABLE kafkamessaging.users (
    user_id BIGSERIAL PRIMARY KEY,
    fname VARCHAR(32) NOT NULL,
    lname VARCHAR(32) NOT NULL,
    mobile VARCHAR(32) NOT NULL,
    created_at …

Shopify Admin API: Importing Products in Bulk

By Patrick Lewis
May 4, 2020

Photo by Chris Young, used under CC BY-SA 2.0, cropped from original.

I recently worked on an interesting project for a store owner who was facing a daunting task: he had an inventory of hundreds of thousands of Magic: The Gathering (MTG) cards that he wanted to sell online through his Shopify store. The logistics of tracking down artwork and current market pricing for each card made it impossible to do manually.

My solution was to create a custom Rails application that retrieves inventory data from a combination of APIs and then automatically creates products for each card in Shopify. The resulting project turned what would have been a months- or years-long task into a bulk upload that only took a few hours to complete and allowed the store owner to immediately start selling his inventory online. The online store launch turned out to be even more important than initially expected due to current closures of physical stores.

Application Requirements

The main requirements for the Rails application were:

• Retrieving product data for MTG cards by merging results from a combination of sources/APIs
• Mapping card attributes and metadata into the format expected by the Shopify Admin API …

Creating a Messaging App Using Spring for Apache Kafka, Part 2

By Kürşat Kutlu Aydemir
April 29, 2020

This article is part of a series. The GitHub repository with code examples can be found here.

In this part I’ll walk through Kafka’s servers and processes, the basics of spring-kafka producers and consumers, persistence, and caching configurations.

Kafka Servers

Kafka uses Apache ZooKeeper as the distributed coordination server. You can download the Apache Kafka with ZooKeeper bundle here.

When you download and untar the Kafka bundle Kafka’s console scripts can be found in the bin directory. To enable Kafka connectivity and prepare the Kafka configuration let’s start the Kafka servers and see how to create Kafka topics and test console producers and consumers.

ZooKeeper

To start ZooKeeper with the default properties run the following command:

bin/zookeeper-server-start.sh config/zookeeper.properties

Kafka Server

A single Kafka server with the default properties can be started with following command:

bin/kafka-server-start.sh config/server.properties

Kafka Topics

Creating Kafka Topics

Let’s create a test Kafka topic:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic myTestTopic

List Topics

To list all previously created Kafka …

Convenient Reporting with Jasper

By Árpád Lajos
April 28, 2020

Business Intelligence (BI) reporting is a huge problem space in custom software. There’s a wide range of business needs for looking at past and predictive behavior. Building a reporting tool can be a very cost effective way to get this data, especially compared to writing individual queries or manually generating reports.

I’ve been working with Jasper in the Java project space and wanted to write about some research I’ve collected on the topic.

JasperReports takes .jrxml files as input and outputs a .jasper report. Possible output targets include:

• Screen
• Printer
• PDF
• HTML
• Excel files
• RTF
• ODT
• CSV
• XML

Jasper history

• June 2001: Teodor Danciu began working on JasperReports.
• September 2001: Jasper was registered on SourceForge.
• November 2001: JasperReports 0.1.5 was released.
• 2004: Panscopic teamed up with Teodor Danciu, acquired ownership of the product and changed its name to Jaspersoft.
• 2005: JasperReports 1.0 was released.
• 2007: Brian Gentile became CEO of the company.
• 2014: TIBCO acquired Jaspersoft for ~$185 million.

Best reporting tools

Let’s compare some popular reporting tools:

• JasperReports is a free and open source Java-based reporting tool, which supports lots of …

Migrating large PostgreSQL databases

By Árpád Lajos
April 21, 2020

Photo by Harshil Gudka on Unsplash

The challenge

One of our clients has a large and important health-related application. It’s built on an end-of-life Ruby on Rails-based open source framework, heavily customized over the years. They wanted to upgrade to a newer, supported, Java-based open source application a partner organization had developed as a replacement. Both organizations used the old system previously. To do that we would need to migrate all their existing PostgreSQL data from the old system to the new one, retaining important customizations while adapting to the new database schema.

Although there were many similarities between the old system and the new, the differences were significant enough to require careful study of the database schemas and the migration scripts designed to move the data:

• There were schema-level differences between our old database and the partner organization’s old database.
• Even where the two old databases were similar there were differences on the data level, such as different standards, different values in table records, different representation, etc.
• We had different content, so if a script was working well for their data, it was not …

Creating a Messaging App Using Spring for Apache Kafka, Part 1

By Kürşat Kutlu Aydemir
April 8, 2020

Photo by Click and Learn Photography at Unsplash

This article is part of a series. The GitHub repository with code examples can be found here.

Spring is a popular Java application framework. Apache Kafka is a fault-tolerant, fast, and horizontally scalable distributed stream-message broker. Spring for Apache Kafka applies the overall concepts of Spring to Java applications based on Kafka.

Since Kafka can establish a fast and fault-tolerant stream data pipeline it can be used as an orchestrator. In this article I’ll explain how to create a spring-kafka project, add dependencies and use Kafka to create a messaging app.

Initialize Spring project

Spring projects can be built from scratch using Spring Initializr. I like to keep the default options. Most Spring projects use Maven. I set the group id as com.endpoint and the artifact as SpringKafkaMessaging which makes the base package name com.endpoint.SpringKafkaMessaging.

When we are done with the initial project setup we press the “GENERATE” button to download an empty Spring Boot project in a zip file. You can then use your favorite IDE to open and start developing your project. I prefer Eclipse for Java projects. Here’s what it …

Installing Ubuntu 18.04 to a different partition from an existing Ubuntu installation

By Bharathi Ponnusamy
April 6, 2020

Photo by Patryk Grądys on Unsplash

Our Liquid Galaxy systems are running on Ubuntu 14.04 LTS (Trusty). We decided to upgrade them to Ubuntu 18.04 LTS (Bionic) since Ubuntu 14.04 LTS reached its end of life on April 30, 2019.

Upgrading from Ubuntu 14.04 LTS

The recommended way to upgrade from Ubuntu 14.04 LTS is to first upgrade to 16.04 LTS, then to 18.04 LTS, which will continue to receive support until April 2023:

14.04 LTS → 16.04 LTS → 18.04 LTS

Ubuntu has LTS → LTS upgrades, allowing you to skip intermediate non-LTS releases, but we can’t skip intermediate LTS releases; we have to go via 16.04, unless we want to do a fresh install of 18.04 LTS.

For a little more longevity, we decided to do a fresh install of Ubuntu 18.04 LTS. Not only is this release supported into 2023 but it will offer a direct upgrade route to Ubuntu 20.04 LTS when it’s released in April 2020.

Installing Clean Ubuntu 18.04 LTS from Ubuntu 14.04 LTS

Install debootstrap

The debootstrap utility installs a very minimal Debian system. Debootstrap will install a Debian-based OS into a sub-directory. You don’t need an installation CD for this. However, you need to have access to the corresponding Linux …

Magento 2: Creating a custom module

By Juan Pablo Ventoso
April 1, 2020

Photo by Babatunde Olajide, cropped from original

A Magento module is a set of classes and routines that will depend on and interact with other Magento classes in order to add a specific feature to a Magento application. While a theme is orientated towards the front-end and user experience, a module is orientated towards backend logic and application flow.

We will need to create a custom module if we want to add or change the existing logic at a level where Magento doesn’t provide a setting or option for it. For example, if our business has a specific feature or set of features or requirements that are not common to the market, a module can fill that gap for us.

Creating a basic Magento 2 module

Creating a simple module in Magento 2 is not that hard. We will need to accomplish the following tasks:

• Create a new directory for the module
• Create a registration.php script
• Create a etc/module.xml information file
• Install the new module

Creating a new directory for the module

Where should the new directory for our module be placed? We have two options to choose from:

• app/code/{vendor}/
• vendor/{vendor}/

If your module is intended for a specific website you’re working on, you can use …