Month: March 2023

Continuous integration (CI – Continuous Integration) and continuous delivery (CD – Continuous Delivery) are one of the typical DevOps practices that allow developers to deploy software changes reliably and quickly. The key difference from manual development lies precisely in the automation of testing and code assembly.

For a long time, due to tools lack in the Qlik ecosystem for fully extracting application source code and its corresponding automatic assembly from source codes, the practice of CI / CD approach implementation was not available for developing and effectively managing projects based on Qlik technologies.

For Qlik developers QOps utility provides opportunities that were previously available to all other developers using a git repository for storing source codes and configured runners for automated tests and deployment.

Let's take a look at the main aspects of the concept of CI / CD as a DevOps practice

Continuous Integration. Developer regularly makes changes in the applications development process. And these changes are uploaded to the repository. Automated testing and verification are provided through special tools such as events, that initiate the integration as a process consisting of stages and steps set. The execution of each step is accompanied by logging, that reflects all changes.

Continuous delivery automates deployment in any environment (production, test environment, development environment). Automated testing and deployment process provide developer with opportunity to focus on improving the software.

The advantage of CI/CD is to reduce the time spent on developing a software product, minimizing and identifying errors and defects in the early stages of code creation, reducing the time to correct errors, and reducing feedback cycles.

For example, there is a site that displays sales analytics for a certain company. The architecture of this solution includes a backend (can be represented by a database with a specific ETL process for data processing) and a frontend (in the case of using Qlik technology, the frontend is provided by an installed web server). During development and updates, individual developers make changes to one or both parts. Then changes are merged into the repository at the source code level, go through the entire pipeline and, ultimately, the changes are reflected immediately in the entire product.

CI/CD cycle stages:

1. Creation (writing a certain part of the code and subsequent testing; after successful testing, code parts are combined into one whole and transferred to the working development branch);
2. Assembly (version control implementation using Git, automatic assembly taking into account changes and received code testing);
3. Testing (code verification implementation by a team of testers);
4. Release (code transfer to release, creation of a working build to update the current version of the product);
5. Deployment (broadcasting the update on the developers' servers to update the software to the latest version);
6. Support (monitoring user reviews);
7. Planning (creating a list of changes for future versions).

Using QOps in the Development Life Cycle of a Qlik Application

Automation of integration and deployment processes is performed by the installed GitHub/GitLab runner, that is integrated with a repository. Bitbucket automation tools can also integrate with the Jira tracking system. It simplifies the task management process, makes it possible to see in which repository branch the desired update is located and follow its further progress. To receive commands from a runner, QOps must be installed on the same server as the runner. This makes it possible to include QOps teams in the integration and deployment pipeline of Qlik applications.

Benefits of using QOps

• no need to manually transfer code from one development environment to another;
• possibility of dividing the project into more tasks that can be performed in parallel;
• absence of routine operations in the deployment process, that helps to save time.

An example of using QOps in the integration and deployment pipeline of a complex Qlik application

The following pipeline is integrated to automate the processes of supporting and updating complex Qlik application of one of company's clients using GitHub Actions.

Qlik application has a 4-layer structure (transformers – model – dashboard – extractors) and is made in QlikView architecture.

The pipeline is built in such a way that the stages of its execution are determined by the name of Git repository branch. All changes are tracked there.

Below are the active steps that will be executed if a branch named UAT-* is used. The task of this approach is to prepare the necessary files in a separate folder for the trial deployment of a new task when creating a new branch.

After confirming task completion current development is merged with the main repository branch and execution of all process stages is initialized. At the same time, the pipeline uses the logic of selecting only those application files that have been changed. These files are transferred to a given environment, for example, to production.

Development process of the pipeline greatly simplifies runner functionality in the form of using matrix operations on a list of same type applications. This approach is convenient to process transformers that have a similar structure and purpose.

This is how the result of executing successive pipeline steps within one stage looks like. Some steps depending on the conditions set can also be skipped, ignored, or stopped if errors are found.

In case of a runtime error, the pipeline will signal it as follows.

For a more detailed study of each execution step, console output is available at each stage. This makes it convenient to track and resolve errors that occur.

So, CI/CD introduction into Qlik application development and maintenance process has reduced the time spent on converging the results of parallel development and greatly simplified the process of preparing application files for further deployment.

CI/CD pipelines help to minimize potential risks in the process of integrating code changes into the repository, isolate the impact of possible errors and simplify fixing process. The main CI/CD goal is to speed up development process and value delivery to the end user. However, there are always ways and tools to make the process even more efficient. The matrix approach is one such option.

The basic pipeline structure involves the execution of tasks simultaneously at a certain stage. Tasks at the next stage can be completed if the previous ones are completed. This process continues at all stages. Different tasks in a pipeline take different times to complete. So, team members must wait to make their changes to the project. This significantly slows down the workflow and reduces productivity. Also, the presence of the same pipelines and creation scripts can lead to pipelines blocking. To optimize resources and increase productivity, it makes sense to create tasks clones and run them in parallel.

Previously, it was necessary to manually define tasks for their parallel execution. With the advent of parallel matrix jobs, it became possible to create jobs at runtime based on specified variables.

The matrix strategy uses variables when defining a job and allows to provide automatic creation of multiple job executions. This strategy can be used in the process of testing code in different languages and/or different operating systems. The matrix can be created with different job configurations specifying one or more variables. By defining variables (one or more), task will be applied for each variable combination.

It is also worth considering one feature. So, organizations often use mono-repositories for better project management. However, pipeline performance is degraded when there are many projects in the repository and one pipeline definition is used to run different automated processes for different components. Using parent and child pipelines makes pipelines more efficient. This approach minimizes the chance of merge conflicts and allows editing of pipeline parts if necessary.

On the one hand, pipelines optimization reduces the time developers spend on maintenance. On the other hand, it frees up time and space for new ideas, creativity and increased productivity. For example, using a matrix, it is possible to break down large pipelines into manageable parts for more efficient maintenance and maximization of tasks amount that run in parallel. The order in which jobs are created dictates the order of the variables in the matrix. The first variable is the first job in the run.

The complex Qlik application architecture consists of several layers (transformers, model, dashboard and extractors). And when using QOps, the matrix strategy is the best suited for managing applications of the same type within the same layer. These include applications in the layer of transformers and extractors.

GitHub, GitLab and Jenkins allow to build pipelines based on a matrix strategy that iterates over the available tasks according to cartesian multiplication. This was done to expand CI/CD capabilities, namely in testing on different platforms or, for example, with different frameworks versions.

The screenshot below contains an example pipeline source file for GitHub, where transformer overloading is implemented using a matrix strategy. The keyword for this is matrix in strategy block. The required list of applications is specified as a list on line 403. In this case, the substitution of the iterated application will be performed each time matrix.app is specified.

This is how used applications list in GitHub graphical interface looks like when executing the pipeline. At the same time, scaling the number ofprocessed applications is easily performed. To do this, it is enough just to change their list and not make changes to the executable part of the code.

Qlik is one of the leading BI system providers. Its main products are QlikView and Qlik Sense (data analysis and visualization tools that provide the ability to access and explore data from different sources). Qlik users can independently develop visualization objects (reports, data models, analytical programs, etc.) without IT specialists participation. Qlik solutions are relevant for different use cases, for example:

• sales analysis (market analysis, marketing campaign management, demand analysis, product ratings, managers, suppliers, buyers, etc.);
• financial analysis (liquidity indicator, budgeting, income, net profit, margin, income taxes, expenses, etc.);
• inventory analysis (purchase volume, purchase prices and discounts, their relationship with sales, income and profitability, inventory value, «frozen» goods, excess stock, calculation of the optimal order, analysis of purchase prices and suppliers, etc.);
• analysis of logistics processes (cost of delivery and shipment, analysis of cargo, routes, carriers, etc.);
• marketing analysis (marketing campaigns cost, conversion, customer segmentation, customer life cycle assessment, sales funnel, etc.).

However, all this is preceded by an extensive process of Qlik application development with individual needs and business requests. QOps console tool developed by DataLabs is designed to solve a number of tasks in the development, version control and subsequent deployment of Qlik applications.

Development

The main QOps advantage for developers is the ability to use previously developed components in one project and implement them in another. Developers often work on several projects in parallel, and the need to transfer ready-made features from one project to another is relevant. The traditional approach involves some complexity, steps and time costs to implement the transfer. So, developer needs to go to the previous project that may be on a different server, find the appropriate reference points where there may be changes (in the script, charts, variables, etc.). QOps allows to solve this problem very quickly. Using this tool, it becomes possible to save a component developed in one project as a separate commit and, by selecting it directly in the repository, see all the necessary changes. And then it is possible to transfer it to the required project. That is, QOps is an auxiliary tool that allows to extract source codes from Qlik applications.

Also, QOps allows to reduce the routine manual work of developers during deployment and troubleshooting. For example, several developers work on one application, and each of them develops its own part. Then, to create one whole project, developers need to agree on how changes will be made from each developer. Another problem is that 2 separate finished parts may not work together. QOps automates this process with GitLab, GitHub, or Bitbucket repositories where it merges, checks for conflicts, and resolves if necessary. Thus, developer doesn’t need to keep all the changes in his mind to implement a certain task or functionality. It is important to note that a developer is an ordinary person who sometimes makes mistakes. Any manual process, in particular deployment, increases the risk of making mistakes. QOps for its part helps prevent this from happening.

Another benefit of QOps is the ability to move a project from one environment to another. Today a lot of organizations are moving to the cloud that allows users to work from anywhere. However, there are companies that require developer to work from a specific isolated device. QOps allows to extract source code from one environment and work in another. There is a question of compliance with safety requirements. Since QOps only extracts a source code and not the data itself, there is no security breach.

Deployment

Using QOps, it’s possible to simplify deployment process and even deploy applications without detailed knowledge of its development specifics. This can be done by a team leader, or a project manager, or a DevOps. User can also use placeholders in the source code that will be automatically substituted when migrating source codes between environments or clients. This ensures correct automatic deployment without any unusual moments or manual changes. So, this reduces time spent on deployment and opens up the possibility of finding new ideas and/or creating a new product.

Version Control

The main task of the version control system is to record changes in a file or a set of files in a certain period. This allows users to return to a previously defined version at any time, return selected files or the entire project to a previous state, track changes and its author, and correct errors if necessary. The version control system allows to divide the source code into branches (separate parts of product development). During the development process they can merge (Development →Master), separate into a separate product that provides complete freedom of action for each developer and each version. It is important to note that a control is based on specific version and only commits changes, not the entire version. QOps allows to get code from developed Qlik applications, put it in a repository in any branch, accumulate changes and explore them (track changes, errors, author of changes, etc.). QOps is an intermediate tool between Qlik and modern Git-based version control systems.

The QOps installation procedure is quite simple, however, it involves some requirements:

• version of QlikView 12+;
• QlikView license;
• version of PowerShell 5.1+;
• .NET Framework 4.7.2+;
• Windows Server 2016, Windows Server 2012 R2 or higher.

An example of merge of collaborative Qlik Sense app development

Let's create a new app in Qlik Sense. As a data model we use a synthetic set of values specified in a table form using the INLINE operator.

We'll create a draft of the future sheet design in visualization part

The first developer is working on the left side of the sheet where he designed the bar chart and the table below it.

At the same time the second developer is designing the right side of the sheet, where the data is visualized using a donut pie chart and a scatter plot.

Below is fully designed application sheet after merging and subsequent assembly with results of both developers work.