Another interesting feature of enigma.js are interceptors. Interceptors are similar to the mixins functions but they are executed on lower level - request/response level.

Request interceptors are invoked right before the request is send to the Engine and response interceptors are invoked right after the response from the Engine is received and before the data is send to the calling function.

You can think of the interceptors as "enigma.js middleware".

There are couple of functions available for request and response:

• Request
  • onFulfiled - invoked just before the request is going to be send to the Engine so you can alter the request is needed
• Response
  • onFulfiled - invoked when the response from the engine is ok (no errors)
  • onRejected - invoked when the Engine returns an error so you can handle the errors on enigma.js level

The code below shows how to construct request interceptor:

const enigma = require('enigma.js');
const schema = require('enigma.js/schemas/12.20.0.json');
const WebSocket = require('ws');

const session = enigma.create({
  schema,
  url: 'ws://localhost:9076/app/engineData',
  createSocket: (url) => new WebSocket(url),
  requestInterceptors: [{
    onFulfilled: function myHandler(sessionReference, request) {
      // do somthing with the request data here
      // OR return the original request 
      return request
    },
  }],
});

Example

A simple example of response interceptor can be - change the return data format of a getLayout() method which gets the current selections data.

This interceptor will "listen" for getLayout requests and if the response is for a specific object type it will "flatten" the returned data and return it in the following format:

[
    ...
    {"field": "name-of-the-field", "value": "selected-value-1"},
    {"field": "name-of-the-field", "value": "selected-value-2"},
    {"field": "some-other-field", "value": "selected-value-1"}
    ...
]

• First lets prepare the session object and define our interceptor there:

const session = enigma.create({
    schema,
    url: 'ws://localhost:9076/app/engineData',
    createSocket: (url) => new WebSocket(url),
    responseInterceptors: [{
        onFulfilled: function toggleDelta(session, request, response) {
          // check if the request method is getLayout and 
          // the response is for object with type current-selections-modified
            if (
                request.method === 'GetLayout' && 
                response.qInfo.qType == 'current-selections-modified'
            ) {
                // if the above is valid then "flatten" the response
                return response.qSelectionObject.qSelections.map((s) => {
                    return s.qSelectedFieldSelectionInfo.map((f) => {
                        return {
                            field: s.qField,
                            value: f.qName
                        }
                    })
                }).flat()
            }

            // for every other response return the original response 
            return response;
        },
    }],
});

• Let's select some values first:

    let field = await qDoc.getField('Product Sub Group Desc')
    let selectValues = await field.selectValues([
        { qText: 'Ice Cream' },
        { qText: 'Juice' },
        { qText: 'Chips' }
    ])

• Once we have the interceptor code ready and some selections are made then we can prepare the properties of the object that will hold the selections:

    let selectionObjectProps = {
        "qInfo": {
            "qId": "",
            "qType": "current-selections-modified"
        },
        "qSelectionObjectDef": {}
    }

Check out the type of our object - current-selections-modified. Our interceptor will change the data output only if the response is for this object type. This way we will not "disrupt" the data for the other object types.

• once we have the properties set, we continue our enigma calls as usual: create session object and get its layout:

let sessionObj = await qDoc.createSessionObject(selectionObjectProps);
let sessionObjLayout = await sessionObj.getLayout(); // out interceptor will be invoked here
console.log(sessionObjLayout)

The result of the last row will print the following "flatten" array:

[
  { field: 'Product Sub Group Desc', value: 'Juice' },
  { field: 'Product Sub Group Desc', value: 'Ice Cream' },
  { field: 'Product Sub Group Desc', value: 'Chips' }
]

It's probably not the most practical example but it will give you an idea what enigma.js interceptors are. There are couple of more examples in enigma.js repository itself:

qlik-oss/enigma.js

JavaScript library for consuming Qlik’s Associative Engine. - qlik-oss/enigma.js

qlik-ossGitHub

Conclusion

The same interceptor can be written as mixin as well and instead intercepting the response just call the mixin which will return the same result. But if you want low level / fine grain control over the request/response traffic then interceptors are the way.

Code

Check out the repository below for the full example code

countnazgul/enigma-js-interceptors

Small script to demonstrate the usage of enigma.js interceptors - countnazgul/enigma-js-interceptors

countnazgulGitHub

Hope you liked it!

Stefan

In the previous post we saw what are enigma.js mixins. I've started an repo some time ago (and npm package) where I've been adding mixins that I've found useful (or mixins that I've found boring to write all the time). And want to introduce you to this list.

At the moment the repo contains only mixins that are relevant to application/document.

(to distinct the mixins methods from the default enigma.js ones all mixin methods, from this package, are having prefix m)

The mixins are "split" into groups:

• Selections

  • mSelectionsAll - returns the "native" reponse from Qlik ragarding the current selections

  • mSelectionsFields - an array of only the field names, having selections in it

  • mSelectionsSimple - without any parameters returns a flat array with all selections

    [
      {field: "Field 1": value: 'Value 1'},
      {field: "Field 1": value: 'Value 2'},
      {field: "Field 2": value: 'Value 10'},
      ...
    ]
    

    if groupByField parameter (optional and default is false) is passed then the result will be grouped by a field name:

    [
      {field: "Field 1": values: [...]},
      {field: "Field 2": values: [...]},
      ...
    ]
    

  • mSelectInField - provide an field name and array of values to be selected

• Tables and fields
  • mGetTablesAndFields - returns an array with {table: "data-table-name", field: "field-name"} values
  • mGetTables - an array with all data tables
  • mGetFields - an array with all fields
  • mCreateSessionListbox - creates a listbox session object for the provided field
• Variables
  • mVariableGetAll - array with all variables
  • mVariableUpdateById - updates the definition of an existing variable by providing the variable ID
  • mVariableUpdateByName - updates the definition of an existing variable by providing the variable name
  • mVariableCreate - creates a new variable from the provided name, comment (optional) and definition
• Extensions
  • mExtensionObjectsAll - returns an array with all extension objects in the current app. Returned fields: appId, appName, objId, objType, extName, extVersion, extVisible, extIsBundle, extIsLibrary, qProps
• Unbuild
  • mUnbuild - extracts all parts of an Qlik Sense app (apart from the data itself) into JSON object
• Build
  • mBuild - the opposite of un-building - provide a json object with the app components and this mixin will update (or create) all the objects in the target app

If you want to use the build and unbuild commands outside JavaScript then I highly recommend using Corectl tool

qlik-oss/corectl

corectl is a command line tool to perform reloads, fetch metadata and evaluate expressions in Qlik Core apps. - qlik-oss/corectl

qlik-ossGitHub

Installation

The installation is very simple:

npm install --save enigma-mixins

Usage

(make sure that enigma.js is already installed)

import enigma from "enigma.js";
import schema from "enigma.js/schemas/12.20.0.json";
import enigmaMixins from "enigma-mixins";

// Create enigma session as usual but include the mixin property
const session = enigma.create({
  schema,
  mixins: enigmaMixins, // this is where the mixinx are added
  url: "ws://localhost:4848/app/engineData",
  createSocket: (url) => new WebSocket(url),
});

Once the session is established and the document is open, then you'll have all the mixins available:

Contribution

Feel free to open an PR with your mixins :)

Repository and npm

countnazgul/enigma-mixin

Set of Enigma mixin. Contribute to countnazgul/enigma-mixin development by creating an account on GitHub.

countnazgulGitHub

enigma-mixin

Set of Qlik Sense enigma.js mixin

npm

Hope you liked it!

Stefan

You know it already but enigma.js is the JavaScript wrapper around the Qlik's Engine API.

If you are building a mashup/web app or some sort of automation that require Engine communication you'll have to use Engine API and here enigma.js is your friend.

Just to mention that JavaScript is not the only "flavor" of Enigma. You can find enigma for Go and .NET:

qlik-oss/enigma-go

Go library for consuming Qlik’s Associative Engine. - qlik-oss/enigma-go

qlik-ossGitHub

q2g/enigma.net

.NET library for consuming Qlik’s Associative Engine. - q2g/enigma.net

q2gGitHub

But in this post I want to introduce you to one very cool (but not very popular) functionality that Enigma provides - mixins.

Mixins allow us to extend, with additional methods, or override existing methods. Mixins are bound to at least one type (type being - Doc, GenericObject, GenericBookmark etc.). The official documentation can be found in the official enigma.js repo.

Let's write some code (basics)

Essentially each mixin is a JS object with few properties - init, types and extends and/or overrides.

A very simple mixin will be:

const docMixin = {
    types: ['Doc'],
    init(args) {
        console.log(`My mixin is being initialized`);        
    },
    extend: {
        myMixin() {
            console.log('My mixin was called!');
        },

    }
}

From the snipped above, we can see:

• this snipped is available for docs (apps)
• when the mixin is initialised a message will be printed in the console
• when its called the mixin will only print a message

To use the mixin we have to add this object (or objects) to the Enigma session:

const session = enigma.create({
    schema,
    mixins: [docMixin],
    url: 'ws://localhost:9076/app/engineData',
    createSocket: (url) => new WebSocket(url),
});

Once we have the session established then we can open an app and use our mixin:

let global = await session.open()
let doc = await global.openDoc('/data/Consumer Sales.qvf')
doc.myMixin()

In our case the mixin is pretty general (just printing a message) and we might want to use it for other types and not just in the apps context. To add it to more types just edit the type array:

...
types: ['Doc', 'GenericObject', 'GenericBookmark']
...

A bit more advanced example

Getting the layout of an object is a two step process:

let qObject = await doc.getObject('some-object-id-here');
let qLayout = await qObject.getLayout();

What about if we write an mixin that is doing these two steps for us so we can have a single command to get the layout? An example mixin can look like this:

const docMixin = {
    types: ['Doc'],
    init(args) { },
    extend: {
        getLayoutMixin(objectId) {
            let qObject = await doc.getObject(objectId);
            let qLayout = await qObject.getLayout();

            return [ qObject, qLayout ]
        }
    }
}

const session = enigma.create({
    schema,
    mixins: [docMixin],
    url: 'ws://localhost:9076/app/engineData',
    createSocket: (url) => new WebSocket(url),
});

let global = await session.open()
let doc = await global.openDoc('/data/Consumer Sales.qvf')

let [myObject, myObjectLayout] = await doc.getLayoutMixin('some-object-id');

Conclusion

Personally I found the mixins to be very cool and handy feature. But also I found them very underused (including myself here as well).

I've been setting my own collection of useful mixins in my free time and the next post will be about this collection but you can have a look in the repo if you want:

countnazgul/enigma-mixin

Set of Enigma mixin. Contribute to countnazgul/enigma-mixin development by creating an account on GitHub.

countnazgulGitHub

Hope you liked it!

Stefan

qlbuilder

In general qlbuilder is a tool that communicates with Qlik apps (using the Engine API) and set the script. The script is located on the local machine and can be "stitched" from multiple files.

Why?

Everyone have their own preferred IDE/text editor and we can write own Qlik Sense load scripts there and use include/must_include inside the app to "bring" the script inside the app.

But it will be nice if we can write the scrips locally and upload them directly to the app as it is.

The basic idea is that we will write each Qlik Sense script tab in a separate .qvs file and qlbuilder will combine them (when needed) in a single file and set it's content to the dedicated Qlik Sense app (on dedicated QS environment).

qlbuilder can perform few more actions, apart from combining and setting the script: check for syntax errors, reload app, "watching" for script changes and check for syntax errors etc.

With this approach I can write my script in Visual Studio Code (in combination with Qlik for Visual Studio Code
extension) and use all VS Code shortcuts, themes etc.

Installation

qlbuilder is a Node JS module and can be installed as a global package

npm install -g qlbuilder

Folder structure

The folder structure used from qlbuilder looks like this:

project-name
│   config.yml
│───dist
│      LoadScript.qvs
└───src
    │   0--Main.qvs
    │   1--Config.qvs
    │   2--Mapping.qvs
    │   ...

src is the main folder. In this folder are located all qvs files. Each file will result in a separate script tab, when uploaded. The .qvs files are combined in alphabetical order (for now. please have a look at the Road map section at the end of this post)

Config files

There are two configuration files that are used from qlbuilder:

config.yml

This file is mandatory and it's created during the project/folder creation. This file is located in the root folder and defines the Qlik Sense environments and apps ID that are relevant to this script.

The file is in yaml format and looks like this:

- name: dev
  host: 192.168.0.100 # IP/FQDN of QS engine (central node) 
  secure: false # default value is true 
  appId: 12345678-1234-1234-1234-12345678901
  authentication:
    type: winform

The above config defines:

• QS environment called dev
• the environment's host is 192.168.0.100
• the environment is access through http
• the appId (that this script belongs to) is 12345678-1234-1234-1234-12345678901
• the authentication for this environment is Windows/Forms - winform

(for more information on authentication types and other possible settings please have a look at qlbuilder GitHub repo)

The config contains the "public" environment information and can be commit to git repo.

.qlbuilder.yml

The non-public info (authentication information) is held in another configuration file .qlbuilder.yml (there is option to use environment variables as well)

This file is not created through the install/create process and should be located in current user home folder. For example: C:\Users\My-User-Name\.qlbuilder.yml. Using the above example, with winform authentication and environment named dev, the config will looks like:

dev:
  QLIK_USER: DOMAIN\username
  QLIK_PASSWORD: my-secret-password

This information will be used by qlbuilder during the authentication process to generate the sessionId

Commands

qlbuilder is executed from command line and accept few parameters:

• create [project name] - create the initial project folder structure, sample script and config file. If folder with the same name already exists, and the developer agree to re-create, then this command will re-create only the qlbuilder specific files and folders and will not modify any other folders or files

• build - combines all qvs files from src folder into a single load script. The result script is created in dist folder (ideally this file should not be edited)

• checkscript [env name] - check the script for syntax errors. The check is made against temp (session) Qlik app

• getscript [env name] - gets the script from the app id (specified in config.yml) and splits it into multiple qvs files (in src folder). This operation deletes all files in src folder as first step!

• reload [env name] - reloads the QS app and output the reload progress. (the reload is performed on the QS Engine itself)

• setscript [env name] -

  • builds the script from the files in src folder
  • checks the script for syntax errors (if there are errors the process is stopped)
  • opens the QS app and replaces the script with the build one
  • saves the app

• watch [env name] - now this is a funny one :) ... when started in watch mode qlbuilder will check for syntax error on each file save (any qvs file in src folder). During watch mode few sub-commands that can be used:

  • s or set - sets the script to the app and saves it
  • r or rl - reloads the app (if successful the app is saved)
  • c or cls - clears the console
  • x - exit qlbuilder
  • ? - outputs these commands again

Version Control

Having the script locally allows us to put the project folder under version control and keep history of the script changes (anyone thinking about git hooks as well? 😉)

Road map

More or less qlbuilder includes all the features I had in mind initially. But there are few things that can be added:

• include / must_include - (optional config) detect if these statements are present in the script. If found - get the content of the included scripts and replace the statement with the actual script. This can "save" apps from failing because of external script being edited after the app is published
• different way to organize script files? - at the moment the script files are combined based on alphabetical order. It might be a bit annoying when adding new file in the middle of the script - all files that follow need to be renamed

For more information please have a read at the readme at the qlbuilder GitHub repo

Fell free to open issues there if you experiencing any problems or have requests/suggestions

Any suggestions are very welcome!

Hope you liked it!
Stefan

Deploy Qlik extension/mashup with Qlik-CLI

Creating Qlik extensions/mashups is usually quite fun. But it will be better if you have a way to develop locally and deploy to Qlik.

Usually I'm using qExt or sense-go but in this specific case I didn't had the chance to use either (enterprise restrictions)

Luckily for me Qlik-CLI was available. And Qlik-CLI is doing a great job dealing with Qlik Repository API.

I've decided to write a small PowerShell script that can be used to import the extension from my local PC to Qlik.

The script will perform the following actions when started:

• generate wbfolder.wbl file - this is not mandatory but without this file the extension files will not be available for edit in dev-hub
• compress the src folder and place the archive into the build folder
• connect to Qlik
• check if extension with the same name already exists and if there is one - delete it (assume this is an older version)
• import the archive from build folder

After the script is done the new extension/mashup code will be available.

Folder structure

The script assume the following folder structure (need to exists before run)

root
│
└──+ build
│    └─── extension-name.zip (auto-generated and uploaded)
│   
└──+ src
│    │───+ static (optional)
│    │     │─── image.png
│    │     └─── ...
│    │      
│    │─── extension-name.css
│    │─── extension-name.js
│    │─── extension-name.qext
│    │─── wbfolder.wbl
│    └─── ...
│
└─── upload.ps1

Usage

• copy the code from below and create upload.ps1 file in your root folder (the file name can be anything).
• edit the initial variables ($extensionName and $qEnv) to match your extension name and QS DNS
• every time when need to upload the extension/mashup just start the upload.ps1

Summary

Probably not the most elegant way to achieve this but might help if you need quick and "dirty" way to make (or edit exiting one) extension/mashup and use your text editor/IDE of choice.

And since this approach is using PowerShell you can extend it to your needs ;)

PowerShell Script

$extensionName = "mashup-extension-name"
$qEnv = "computerName"

function generateWBL {
    # remove the existing wbfolder.wbl (if exists)
    Remove-Item .\src\wbfolder.wbl -ErrorAction Ignore

    # list all files under the src folder and add them to the wbl file
    $existingFiles = Get-ChildItem -Path .\src\ -File |
        foreach {$_.name} |
        Out-File -FilePath .\src\wbfolder.wbl
}

function compressFolder{
    # archive the content of the src folder
    Compress-Archive -Path .\src\* -DestinationPath .\build\$extensionName.zip -Force
    Write-Host ZIP file created
}

function connectQlik {
    ## connect to Qlik using Qlik-CLI
    Connect-Qlik -computerName "$qEnv" -TrustAllCerts
    Write-Host Connected to Qlik
}

function removeOldExtVersion {
    # Query Qlik for extension with the same name
    $ext = Get-QlikExtension -Filter "name eq '$extensionName'" | Measure-Object

    # if there is such extension - delete it
    if ( $ext.Count -gt 0) {
        Remove-QlikExtension -ename "$extensionName"
        Write-Host Older version found and removed
    }
}

function importExtension {
    # Import the archive for the build folder
    Import-QlikExtension -ExtensionPath .\build\$extensionName.zip
    Write-Host New version was imported
}

function main {
  generateWBL
  compressFolder
  connectQlik
  removeOldExtVersion
  importExtension
  
  Write-Host DONE
}

main

With multi-stage Qlik environment (DEV, TEST, PROD etc) the load scripts are usually including some kind of a check - "In which environment am I?".

Based on the result different/additional actions might be performed. For example:

• if the environment is not PROD then mask/scramble the client names
• use different data connection names (providers) on each environment
• ...

It will be nice if there is something more centralised that can provide this information (and not only) to any app on the cluster. Something like a global meta-data "storage" that can be changed/updated (by the administrators)

In general my approach is to use a dedicated Custom Property and each value to be in format key:pair. Values can be accessed for reading using a (dedicated) REST data connection in the beginning of each script. If we keep the keys the same then we can write one script and use it on every environment.

Custom Property

We will create new custom property (in my case I've named it ENVIRONMENT) and all the global values will be added to this CP.

In QMC:

• create new custom property
• add as many CP values as you want as key:value pairs (I've picket : as a separator but this can be anything). For example:
  • name:DEV
  • central repository ip:192.168.1.10
  • central repository host:qlik.central.repo

Data Connection

We will use the Repository Service REST API to get the ENVIRONMENT custom property values. We need to create a new data connection that will be used to extract the data.

During the installation QS creates few monitor_apps_REST_* data connection which are used by the documents in the Monitoring apps stream (like License Monitor or Operations Monitor). We can use one of these connectors, as a base and create new one that will return the values only for our custom property.

• open any application and create new Data Connection (at this point there is no need the connection to be a specific type. We are creating just a placeholder)
• In QMC -> Data Connection -> monitor_apps_REST_task -> copy the Connection string
• go back and edit the placeholder connection (in my case Environment)
• replace the connection string with the one from monitor_apps_REST_task and apply the change
• go back to the application and edit the Environment connection
• in the URL replace /task/ with /custompropertydefinition/
• in the Query parameters section add new one:
  • Name: filter
  • Value: name eq 'ENVIRONMENT'
• Save

At this point you can preview the data from the connection and if everything is ok you should see something like this:

Usage

Once all is set we can create a small QS script that gets the raw values and "convert" them to a more readable QS format in form of a table with two columns: ENV_NAME and ENV_VALUE

LIB CONNECT TO 'Environment';

RestConnectorMasterTable:
SQL SELECT 
  "__KEY_root",
  (SELECT 
    "@Value",
    "__FK_choiceValues"
   FROM "choiceValues" FK "__FK_choiceValues" ArrayValueAlias "@Value")
FROM JSON (wrap on) "root" PK "__KEY_root";

ENVIRONMENT:
Load
  subfield([@Value], ':', 1) as ENV_NAME,
  subfield([@Value], ':', 2) as ENV_VALUE
;
Load
  [@Value],
  [__FK_choiceValues] as [__KEY_root]
RESIDENT 
  RestConnectorMasterTable
WHERE 
  NOT IsNull([__FK_choiceValues]);

DROP TABLE RestConnectorMasterTable;

This script can be placed in the beginning of each app OR even better - host the script somewhere and each app (on each environment) can Must_Include it.

Upon reload only one table is returned:

Conclusion

Having a centralised place for a cluster-wide variables is a nice to have. I've seen multiple approaches on how to deal with this: specific document prefixes for each env, flat files in each data folder etc. But the problem there is that there is always enough space for someone to make a mistake (overwrite the flat file when moving data is my favorite)

Another plus for custom property and data connection is that these objects are subject of security rules. Which means that we can grant everyone read access over them but very limited audience can edit them which (might 🤞) lead to less errors and maintenance.

Bonus

If you have/use Qlik-CLI then the PowerShell script below will do the "dirty" setup for you. The script will:

• create new custom property with some pre-defined values
• create new REST data connection using the monitor_apps_REST_task as a template
• edit the new data connection
  • change the URL
  • add the additional query parameter

# QS Central Repo Address
$computerName = "qlik.central.repo" # or localhost if started on the server itself

# Properties for the new data connection
# ideally the user/pass should be the ones
# under which the QS service is running
$dataConnProps = @{
  name = "Environment"
  userName = "domain\username"
  password = "password"
}

# Properties for the new custom property
# Can populate some values when creating it
# "Objects = App" is just to create the CP (cant create it without it from Qlik-CLI)
$customPropProps = @{
  name = "ENVIRONMENT"
  values = @("name:DEV", "central repository ip:192.168.1.10")
  objects = @("App")
}


function createNewDataConnection {
    # Get the existing monitor_apps_REST_task connection details
    $connector = Get-QlikDataConnection -filter "name eq 'monitor_apps_REST_task'"

    # Replace the endoint to point to return custom prop details instead of a task
    # Add one more query parameter which will return data only for the Environment custom prop
    $newConnectionString = $connector.connectionString.
        Replace("/qrs/task/full", "/qrs/custompropertydefinition/full").
        Replace("queryParameters=xrfkey%20000000000000000", "queryParameters=xrfkey%20000000000000000%1filter%2name eq '$($customPropProps.name)'")

    # Create the connection
    New-QlikDataConnection -name $dataConnProps.name -connectionstring $newConnectionString -type "QvRestConnector.exe" -username $dataConnProps.userName
}

function createNewCustomProperty {

    # Create the Environment custom property
    New-QlikCustomProperty -name $($customPropProps.name) -choiceValues $($customPropProps.values) -objectTypes $($customPropProps.objects)
}

function connectToQlik {
    Connect-Qlik -computerName $computerName -TrustAllCerts
}

function main {
    connectToQlik
    createNewCustomProperty
    createNewDataConnection
}

main

Stefan

I like how flexible Qlik Sense is. With its massive API if we want we can re-build the GUI (and not only) from scratch and just use the background services (like the Engine or Repository).

I have some (small) issues with how the current Hub looks and behaves and decided to build something simple that fits my needs.

At the end I've wanted something simple that will provide me with quick search capabilities, few shortcuts and the options to create, delete and duplicate files.

And below is the final result

The code is available in GitHub

Just a search box, list with all the apps (to which I have access) and create new app button.

Searching

For the search easyautocomplete plugin is used.

The search works as any other search box. The search is performed on the QS apps titles

Create new app

Pressing the top right green button will show the create new app modal

Here the name of the new app is specified and which will be the initial screen after the app is created. As developer (and not only) "Data editor" or "Data manager" are the main ones.

Options

Agains each app few shortcuts are present

• to which stream the app belongs
• open the "App Overview"
• open the "Data Manager"
• open the "Data Editor"
• open the "Data Model Viewer"
• Duplicate app (not available in Desktop mode)
• Delete app

Hope you liked it!
Stefan