Helm Charts for Apache Drill

!NOTE This project was originally forked from Agirish/drill-helm-charts, but was moved to this repo for automated build and discoverability.

Overview

This repository contains a collection of files that can be used to deploy Apache Drill on Kubernetes using Helm Charts. Supports single-node and cluster modes.

What are Helm and Charts?

Helm is a package manager for Kubernetes. Charts are a packaging format in Helm that can simplify deploying Kubernetes applications such as Drill Clusters.

Pre-requisites

• A Kubernetes Cluster (this project is tested on a K3s cluster)
• Helm version 3 or greater
• Kubectl version 1.16.0 or greater

Usage

Helm must be installed to use the charts. Please refer to Helm's documentation to get started.

Once Helm has been set up correctly, add the repo as follows:

helm repo add wearefrank https://wearefrank.github.io/charts

If you had already added this repo earlier, run helm repo update to retrieve the latest versions of the packages. You can then run helm search repo wearefrank to see the charts.

To install the Drill chart:

helm install drill wearefrank/drill

To uninstall the chart:

helm delete drill

Values

Helm Charts use values.yaml for providing default values to 'variables' used in the chart templates. These values may be overridden either by editing the values.yaml file or during helm install. For example, such as the namespace, number of drillbits and more to the template files

Please refer to the values.yaml file for details on default values for Drill Helm Charts.

Access Drill Web UI

There is a service that can be used, but this one will jump from pod, which isn't very friendly. Depending on ingress class you can make this sticky with annotations. You could also change the

Chart Structure

Drill Helm charts are organized as a collection of files inside the drill directory. As Drill depends on Zookeeper for cluster co-ordination, a zookeeper chart added as dependency in the chart definition. The Zookeeper chart is maintained by Bitnami.

drill/   
  Chart.yaml    # A YAML file with information about the chart
  Chart.lock    # A YAML file containing information about the fetched dependencies
  values.yaml   # The default configuration values for this chart
  charts/       # A directory containing the ZK charts
  templates/    # A directory of templates, when combined with values, will generate valid Kubernetes manifest files
  docs/         # A directory containing files for the documentation

Templates

Helm Charts contain templates which are used to generate Kubernetes manifest files. These are YAML-formatted resource descriptions that Kubernetes can understand. These templates contain 'variables', values for which are picked up from the values.yaml file.

Drill Helm Charts contain the following templates:

Autoscaling Drill Clusters

The size of the Drill cluster (number of Drill Pod replicas / number of drill-bits) can not only be manually scaled up or down, but can also be autoscaled to simplify cluster management. When enabled, with a higher CPU utilization, more drill-bits are added automatically and as the cluster load goes down, so do the number of drill-bits in the Drill Cluster. The drill-bits deemed excessive gracefully shut down, by going into quiescent mode to permit running queries to complete.

!IMPORTANT For the graceful shutdown to succeed, a sigfile is made in the $DRILL_HOME folder. This requires running as root ( uid 0). If the application is run as drilluser the stop commando will be used.

Enable autoscaling by editing the autoscale section in drill/values.yaml file.

Parameters

Common parameters

Drill image parameters

Drill deployment parameters

Traffic Exposure Parameters

Other Parameters

Drill configuration

Configuring Drill can be done with override files or in the web ui, although some properties can only be set in the override file. When using the web ui, ZooKeeper will be used to store the values. Make sure that the storage of ZooKeeper is persistent if you intend to configure this way.

This is an example where the web ui and authentication for local (plain) users is enabled.

drill.exec: {
http.enabled: true,
impersonation: {
enabled: true,
max_chained_user_hops: 3
},
security: {
auth.mechanisms: ["PLAIN"]
},
security.user.auth: {
enabled: true,
packages += "org.apache.drill.exec.rpc.user.security",
impl: "pam4j",
pam_profiles: [ "sudo", "login" ]
}
}

For more options refer to the Apache Drill documentation.

Persistence

Persistence is used for logging and for JDBC drivers. These can be configured separately.

Configuration for Drill will be saved in ZooKeeper. Make sure that ZooKeeper is persistent if you want to keep changes in the Web UI.

Notable changes

1.2.7

The notation for .Values.service has been changed. This makes it possible to configure web and user services separately.

1.2.6

.Values.replicaCount has been changed from 3 to 1. This is to default to a less complex install. Having three replicas introduces some complexity regarding, authentication, logging and how queries will be executed. Until features have been added to simple this, the user needs to take these things into account.

The notation for .Values.persistence has changed so storage for logs and data can be configured secretly. The values for persistent logging are now located at .Values.persistence.dataLogDir.