Our Blog

Install Tested Packages on Production Server

By Selvakumar Arumugam
September 7, 2015

One of our customers has us to do scheduled monthly OS updates following a specific rollout process. First week of the month, we will update the test server and wait for a week to confirm that everything looks as expected in the application; then next week we apply the very same updates to the production servers.

Since not long ago we used to use aptitude to perform system updates. While doing the update on the test server, we also executed aptitude on production servers to “freeze” the same packages and version to be updated on following week. That helped to ensure that only tested packages would have been updated on the production servers afterward.

Since using aptitude in that way wasn’t particularly efficient, we decided to use directly apt-get to stick with our standard server update process. We still wanted to keep our test-production synced updated process cause software updates released between the test and the production server update are untested in the customer specific environment. Thus we needed to find a method to filter out the unneeded packages for the production server update.

In order to do so we have developed a shell script that automates the process and maintain the package version in test and production in sync during OS updates. I’ll explain the processes involved used on both the test and the production servers.

Test Server Update Process:

• Update the repository index
• Get the list of installed packages and version before update
• Complete the dist-upgrade
• Get the list of installed packages and version after update
• Compare the packages before and after the update and generate a diff output which has the information about the packages installed, updated and removed

Example:

root@selva_test:~# ./auto-tested-server-update.sh -t test

    ##########################################################
    ############ Auto Tested Server Update Script ############
    ##########################################################

Packages index list and server update will be completed here...

Files were generated in default directory. after_* and before_* contains the installed packages list before and after server update. diff_* contains the changes of the packages in the system.

root@selva_test:~# ls tmp/
after_update_selva_test.2015-09-02.txt  before_update_selva_test.2015-09-02.txt  diff_server_update_selva_test.2015-09-02.txt

# head -5 tmp/before_update_selva_test.2015-09-02.txt 
accountsservice=0.6.15-2ubuntu9.6
acl=2.2.51-5ubuntu1
acpi-support=0.140.1
acpid=1:2.0.10-1ubuntu3
activity-log-manager-common=0.9.4-0ubuntu3.2

# head -5 tmp/after_update_selva_test.2015-09-02.txt 
accountsservice=0.6.15-2ubuntu9.6
acl=2.2.51-5ubuntu1
acpi-support=0.140.1
acpid=1:2.0.10-1ubuntu3
activity-log-manager-common=0.9.4-0ubuntu3.2

# cat tmp/diff_server_update_selva_test.2015-09-02.txt 
             > git=1:1.7.9.5-1ubuntu0.1
             > git-man=1:1.7.9.5-1ubuntu0.1
metacity=1:2.34.1-1ubuntu11          <
mysql-client=5.43-0ubuntu0.12.04.1         | mysql-client=5.5.43-0ubuntu0.12.04.1
ubuntu-desktop=1.267.1           <
unity-2d=5.14.0-0ubuntu1          <
             > wget=1.13.4-2ubuntu1

Production Server Update Process:

• Get the diff output file (which has information about packages installed, updated and removed packages) from the test server
• Collect the packages which will be changed on server update
• Process the packages to categorize under install, upgrade and remove
• Filter the packages from test server installed, upgraded and removed packages
• Perform the install/upgrade/remove actions based on the filtered tested packages

Example:

root@selva_prod:~# ./auto-tested-server-update.sh -t prod -S selva_test -U root

    ##########################################################
    ############ Auto Tested Server Update Script ############
    ##########################################################

*** Production Server Update Process ***

      remote_user: root
      remote_server: selva_prod
      remote_path: /root/tmp
    
1./root/tmp/diff_server_update_selva_test.2015-09-02.txt
Enter a number to choose file: 1
INFO: You have selected the file: diff_server_update_selva_test.2015-09-02.txt
INFO: scp root@selva_test:/root/tmp/diff_server_update_selva_test.2015-09-02.txt /root/tmp
diff_server_update_selva_test.2015-09-02.txt                       100%  763     0.8KB/s   00:00  
  
INFO: Fetching the list of packages need to be changed...!

*********************** Installation *************************
INFO: No packages to install
**************************************************************

************************* Upgrade ****************************
apt-get install mysql-client=5.5.43-0ubuntu0.12.04.1
Reading package lists... Done
Building dependency tree
etc...
**************************************************************

************************** Removal ***************************
apt-get remove zenity=3.4.0-0ubuntu4 zenity-common=3.4.0-0ubuntu4
Reading package lists... Done
Building dependency tree
etc...
**************************************************************

As showed in the sample output above, the script will then run through different phases in which it will install/upgrade/remove all the needed packages in the target production system.

In this way we ensure that all packages tested in the test environment will be replayed on the production server to minimize the chances of introducing bugs and maximize application stability during production system updates.

The Git repository below contains the script and a few additional information to use it.

https://github.com/selvait90/auto-tested-server-update

automation linux sysadmin ubuntu update