--- title: Configure Cluster ---
In this section we'll be configuring all the components that make up a Heroic cluster. These should have been installed in the previous step.
Heroic (by default) uses the heroic
keyspace, which can be
configured using the Heroic shell.
tools/heroic-shell -P cassandra -X cassandra.seeds=<seeds> -X datastax.configure
...
heroic> configure
If you want to use Google Cloud Bigtable to store metrics, you can configure it with the following command.
tools/heroic-shell -P bigtable \
-X bigtable.project=<project> \
-X bigtable.instance=<instance> \
-X bigtable.credentials=default \
-X bigtable.configure
...
heroic> configure
Elasticsearch is also configured using the Heroic shell.
tools/heroic-shell -P elasticsearch-suggest -P elasticsearch-metadata -X elasticsearch.seeds=<seeds>
...
heroic> configure
Heroic suggest also requires dynamic scripting to be enabled, you do this by adding the following to your elasticsearch.yml
script:
disable_dynamic: false
Enabling dynamic scripting for a public Elasticsearch instance is a security risk. Make sure that your server is on a private network. See issue #10091
The following configuration assumes that you've managed to either download or build the heroic project.
We will look into setting up three different kinds of heroic nodes, each with a different purpose.
All the different node types run the same code, the only difference is how they are configured.
All the nodes use the same type of configuration, but they are configured to do different things. A single node can have one or more role, the exact setup depends on how you wish to scale the system.
API nodes are nodes designated to receive user traffic over HTTP. The following is the minimal required configuration in order to set one up.
# heroic.yaml
port: 8080
cluster:
protocols:
- type: grpc
discovery:
type: static
nodes:
- "grpc://localhost"
Data nodes are designated to process requests for the API nodes. They are responsible for fetching and aggregating metrics from a backend.
Data nodes must have the QUERY
node capability, as seen in
cluster.capabilities
.
The following configuration shows how to setup a data nodes that fetches metrics from Cassandra, and uses Elasticsearch for metadata and suggestions.
# heroic.yaml
port: 8080
cluster:
protocols:
- type: grpc
discovery:
type: static
nodes:
- "grpc://localhost"
# This showcases two different metric backends, choose which one you want.
metrics:
backends:
- type: datastax
seeds:
- localhost
- type: bigtable
instance: heroic
credentials:
type: default
metadata:
backends:
- type: elasticsearch
connection:
seeds:
- localhost
suggest:
backends:
- type: elasticsearch
connection:
seeds:
- localhost
For more details in how the service can be configured, see the Configuration Section.
A consumer node is responsible for reading data from kafka, and writing it into the configured backends.
The below configuration is just an extension to the data node configuration above, by adding consumers you are instructing the service to act as a consumer.
# heroic.yaml
port: 8080
# Data Node configuration...
consumers:
- type: kafka
schema: com.spotify.heroic.consumer.schemas.Spotify100
topics:
- "metrics"
config:
group.id: heroic-consumer
zookeeper.connect: localhost
auto.offset.reset: smallest
auto.commit.enable: true
$ java -cp heroic.jar com.spotify.heroic.HeroicService heroic.yaml
Test that the node is running properly. Healthy instances should produce a output like the following.
$ curl http://localhost:8080/status
{ "ok": true, "consumers": { "ok": true, "available": 0, "ready": 0, "errors": 0 }, "backends": { "ok": true, "available": 0, "ready": 0 }, "metadataBackends": { "ok": true, "available": 0, "ready": 0 }, "cluster": { "ok": true, "onlineNodes": 1, "offlineNodes": 0 } }