Requirements

Butler depends on various pieces of software. Some are required, others are optional, as described below.

Running natively

If you opt to run Butler nateively on Node.js installed on a computer, the following software is needed:

What Comment
Sense Enterprise Butler is developed with Sense Enterprise in mind. While many Butler features might also work with Sense Desktop, you are on your own there.
MQTT broker MQTT is used for both in- and out-bound pub-sub messaging. Butler assumes a working MQTT broker is available, the IP of which is defined in the Butler config file. Mosquitto is a nice open source broker. It requires very little hardware to run, even the smallest (free!) Amazon EC2 instance is enough, if you want a dedicated MQTT server. If you don't care about the pubsub features of Butler, you don't need a MQTT broker. In this case you can disable the MQTT features in the config YAML file.
node.js Butler is written in Node - which is thus a firm requirement.

Running on Docker

If you have access or can set up a Docker runtime environment, this is by far the preferred solution for running Butler.
Installation is less error prone compared to installing Butler natively, you get all the benefits from the Docker ecosystem (monitoring of running containers etc), and upgrades to future Butler versions become trivial.

What Comment
Sense Enterprise Same as above.
MQTT broker Same as above.
Docker A Docker runtime environment on any supported platform. This means you can run Butler on any platform where Docker is available, including Linux, Mac OS, Windows and most cloud providers.

Windows or OSX

While Sense is a Windows only system, Butler should be able to run on any OS where node.js is available.
Butler has been succesfully used - both during development and production use - on Windows, Linux (Debian and Ubuntu tested) and OSX.

Installation

Below are the general steps needed to install Butler. Please note that you might need to adapt these to your particular system configuration.

Install on native Node.js

  • Install node.js
    Butler has been developed and tested using the 64 bit version of node.js 8.11.4.

  • Decide where to install Butler
    It is usually a good starting point to run Butler on the Sense server. If there are more than one server in the Sense cluster, Butler can be placed on the reload server (as the /createDir endpoint then can be used to create folders in which QVD and other files can be stored). That said, it is quite possible to run Butler on any server, as long as there is network connectivity to the Sense server(s).

  • Download Butler
    Download the repository zip file or clone the Butler repository using your git tool of choice. Both options achieve the same thing, i.e. a directory such as d:\node\butler, which is then Butler's root directory.

  • Install node dependencies
    From a Windows command prompt (assuming Butler was installed to d:\node\butler):

    d:
    cd \node\butler\src
    npm install
    

    This will download and install all node.js modules used by Butler.

  • MQTT message broker

Many of Butler's features use MQTT for sending and receiving messages.
MQTT is a standardised messaging protocol, and it should be possible to use any broker following the MQTT standard.
Butler has been developed and tested using Mosquitto running on OSX and Debian Linux - both work flawlessly.

  • Documentation dependencies

If you plan to modify or extend Butler's documentation, you will need to install MkDocs.
MkDocs is used to create the pages you are reading right now.

Install on Docker

The following steps should give some guidance on how to get Butler running on Docker.

proton:~ goran$ mkdir /Users/goran/butler
proton:~ goran$ cd /Users/goran/butler
proton:butler goran$ mkdir -p config/certificate
proton:butler goran$
proton:butler goran$ wget https://raw.githubusercontent.com/ptarmiganlabs/butler/master/src/docker-compose.yml
--2018-09-26 16:27:14--  https://raw.githubusercontent.com/ptarmiganlabs/butler/master/src/docker-compose.yml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.84.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.84.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 565 [text/plain]
Saving to: 'docker-compose.yml.1'

docker-compose.yml.1                                       100%[========================================================================================================================================>]     565  --.-KB/s    in 0s

2018-09-26 16:27:15 (6.34 MB/s) - 'docker-compose.yml.1' saved [565/565]

proton:butler goran$ cat docker-compose.yml
# docker-compose.yml
version: '2.2'
services:
  verisure-mqtt:
    image: ptarmiganlabs/butler:latest
    container_name: butler
    restart: always
    ports:
      - "8180:8080"     # REST API available on port 8180 to services outside the container
      - "9997:9997"     # UDP port for session connection events
      - "9998:9998"     # UDP port for task failure events
    volumes:
      # Make config file accessible outside of container
      - "./config:/nodeapp/config"
    environment:
      - "NODE_ENV=production"
    logging:
      driver: json-file
proton:butler goran$

At this point you should

  1. copy the certificates that you have exported from the Qlik Sense QMC, and
  2. copy the template config file from the GitHub repository to the ./config directory, modify it as needed based on your system(s), and rename it to production.yaml.

When done, you should see something like this:

proton:butler goran$ pwd
/Users/goran/butler
proton:butler goran$ ls -la
total 8
drwxr-xr-x   4 goran  staff   128 Sep 26 16:36 .
drwxr-xr-x+ 59 goran  staff  1888 Sep 26 16:24 ..
drwxr-xr-x   4 goran  staff   128 Sep 26 16:36 config
-rw-r--r--   1 goran  staff   565 Sep 26 16:25 docker-compose.yml
proton:butler goran$
proton:butler goran$ ls -la config/
total 8
drwxr-xr-x  4 goran  staff   128 Sep 26 16:36 .
drwxr-xr-x  4 goran  staff   128 Sep 26 16:36 ..
drwxr-xr-x  6 goran  staff   192 Sep 26 16:36 certificate
-rw-r--r--  1 goran  staff  1861 Sep 26 16:36 production.yaml
proton:butler goran$
proton:butler goran$ ls -la config/certificate/
total 32
drwxr-xr-x  6 goran  staff   192 Sep 26 16:36 .
drwxr-xr-x  4 goran  staff   128 Sep 26 16:36 ..
-rw-r--r--@ 1 goran  staff  1166 Sep 26 16:36 client.pem
-rw-r--r--@ 1 goran  staff  1702 Sep 26 16:36 client_key.pem
-rw-r--r--@ 1 goran  staff  1192 Sep 26 16:36 root.pem
proton:butler goran$

At this point everything is ready and you can start the Butler container using docker-compose:

proton:butler goran$ docker-compose up
Creating butler ... done
Attaching to butler
butler           | 2018-09-26T14:38:57.895Z - debug: Server for UDP server: localhost
butler           | 2018-09-26T14:38:57.912Z - info: REST server listening on http://[::]:8080
butler           | 2018-09-26T14:38:57.918Z - info: UDP server listening on 127.0.0.1:9997
butler           | 2018-09-26T14:38:57.930Z - info: UDP server listening on 127.0.0.1:9998
butler           | 2018-09-26T14:38:58.124Z - info: Connected to MQTT server 192.168.1.51:1884, with client ID mqttjs_215c09dc

If you don't have an MQTT server, you won't see the "...Connected to MQTT server" line.

Let's make sure things are working by opening a new terminal window and from there requesting a list of all apps on the server:

proton:~ goran$
proton:~ goran$ curl "http://localhost:8180/v2/senseListApps"
[{"id":"492a1bca-1c41-4a01-9104-543a2334c465","name":"2018 sales targets"},
{"id":"5b243cb2-8d00-44c9-b865-08b00a0af18b","name":"App 1"},
...
...
{"id":"181d101f-986c-49c5-a457-d351058c05b4","name":"Template app 1 DEV"}]
proton:~ goran$

Nice, looking good.

In the terminal where you ran docker-compose, you will see a new line saying that a app list was retrieved:

butler           | 2018-09-26T14:40:32.740Z - debug: Server for UDP server: localhost
butler           | 2018-09-26T14:40:32.746Z - info: REST server listening on http://[::]:8080
butler           | 2018-09-26T14:40:32.748Z - info: UDP server listening on 127.0.0.1:9997
butler           | 2018-09-26T14:40:32.750Z - info: UDP server listening on 127.0.0.1:9998
butler           | 2018-09-26T14:43:05.381Z - info: Getting list of all apps

Security considerations

  • You should make sure to configure the firewall of the server where Buter is running, so it only accepts connections from the desired clients/IP addresses.

A reasonable first approach would be to configure the firewall to only allow calls from localhost. That way calls to Butler can only be made from the server where Butler itself is running.

  • As of right now the MQTT connections are not secured by certificates or passwords.

For use within a controlled network that might be fine, but nonetheless something to keep in mind. Adding certificate based authentication (which MQTT supports) would solve this.

  • Butler uses various node.js modules from npm. If concerned about security, you should review these dependencies and decide whether there are issues in them or not.

Same thing with Butler itself - while efforts have been made to make Butler secure, you need to decide for yourself whether the level of security is enough for your use case.

Butler is continuously checked for security vulnerabilities by using Snyk, with status badges shown in the readme files.

Configuration

Butler uses configuration files in YAML format. The config files are stored in the src\config folder. Prior to v2.1 JSON config files were used, but as YAML is much more human readable than JSON, a switch was made. JSON config files can still be used, but YAML is the default starting with v2.1.

Butler comes with a default config file called default_template.yaml. Make a copy of it, then rename the copy to default.yaml or production.yaml.
Update it as needed (see below for details).

Trying to run Butler with the default config file (the one included in the files download from GitHub) will not work - you need to adapt it to your server environment.

Note: Butler uses the node-config module to handle config files. As per node-config's documentation, to switch to using the production.yaml config file, at a command prompt type (for Windows)

set NODE_ENV=production

before starting Butler with node butler.js.
If developing on OSX or Linux, instead use

export NODE_ENV=production

You may want to set that variable during server boot, to ensure Butler starts properly when the server is rebooted.

Config file syntax

The default_template.yaml config file looks like this:

---
Butler:
  slackConfig:
    webhookURL: <fill in your web hook URL from Slack>
    loginNotificationChannel: sense-user-activity
    taskFailureChannel: sense-task-failure
  mqttConfig:
    brokerHost: <FQDN or IP of MQTT server>
    brokerPort: 1883
    taskFailureTopic: qliksense/task_failure
    taskFailureServerStatusTopic: qliksense/butler/task_failure_server
    sessionStartTopic: qliksense/session/start
    sessionStopTopic: qliksense/session/stop
    connectionOpenTopic: qliksense/connection/open
    connectionCloseTopic: qliksense/connection/close
    sessionServerStatusTopic: qliksense/butler/session_server
    activeUserCountTopic: qliksense/users/active/count
    activeUsersTopic: qliksense/users/active/usernames
  udpServerConfig:
    serverIP: <FQDN or IP of server where Butler is running>
    portSessionConnectionEvents: 9997
    portTaskFailure: 9998
  restServerConfig:
    serverPort: 8080
  configEngine:
    engineVersion: 12.20.0        # Qlik Associative Engine version to use with Enigma.js
    server: <FQDN or IP of Sense server where Sense Engine is running>
    serverPort: '<Port to connect to, usually 4747>'
    isSecure: 'true'
    headers:
      X-Qlik-User: UserDirectory=Internal;UserId=sa_repository
    cert: <Path to cert file>
    key: <Path to key file>
    rejectUnauthorized: 'false'
  configQRS:
    authentication: certificates
    host: <FQDN or IP of Sense server where QRS is running>
    useSSL: true
    port: 4242
    headerKey: X-Qlik-User
    headerValue: UserDirectory=Internal; UserId=sa_repository
    cert: <Path to client.pem>
    key: <Path to client_key.pem>
    ca: <Path to root.pem>
  qvdPath: <Path to folder under which QVDs are stored>
  gitHub:
    host: api.github.com
    pathPrefix: ''

Comments:

  • Currently Butler assumes that a MQTT broker is present, and that status messages should be sent to Slack. Butler will fail with error messages if it cannot connect to a MQTT server, or if the Slack Webhook URL is not properly set.

Future versions may make MQTT, Slack and other similar channels optional, using the config file.

  • The default location cert/key files are found in (assuming a standard install of Qlik Sense) C:\ProgramData\Qlik\Sense\Repository\Exported Certificates.Local Certificates

The files to use are client.pem and client_key.pem. The config file can point straight to these files.
Or export a new client cert/keys from the QMC and use these with Butler.

log4net extender config files

Butler includes a couple of xml files that when deployed to the Sense server will create real-time UDP messages for certain events (tasks failing, user sessions starting/ending etc).
These xml files should be deployed as follows:

  • Task failure events
  • XML file found in log4net_task-failed/LocalLogConfig.xml. This file includes settings to both send an email, and a UDP message when a task fails.
  • XML file should be deployed to the server where reloads are done, in the C:\ProgramData\Qlik\Sense\Scheduler directory
  • User audit events
  • XML file found in log4net_user-audit-event/LocalLogConfig.xml
  • XML file should be deployed on the server where the proxy is running, in the C:\ProgramData\Qlik\Sense\Proxy directory
  • If there multiple proxies on different servers, it might make sense deploying the xml file on all of them, to capture all user audit events.

Customisation

As Butler offers a rather diverse set of features, everyone might not need all features. There is no single config file in which individual features can be turned on/off, but given the structure of the Butler source code it is relatively easy to disable speciifc features, or add new ones.

For example, to disable a particular REST API endpoint, you could just remove the registration of that endpoint from within the src/butler.js file.