/
Pack Hosting Panel

Grafana cloud

How to use grafana cloud on the Hipex platform

Grafana cloud is a very easy way to get more extensive dashboards for your server stats and log aggregation. Because Prometheus is already being used for our monitoring, you can query the Prometheus exporters on your server and send that data to Grafana cloud. In this article we will explain how to do so.

We will be using Docker to run Prometheus and Promtail. If you are not familiar with Docker, please read the docker basics.

Configure Prometheus

The Prometheus endpoints exported by Hipex are:

  • Node exporter (port 9100)
  • MySQL exporter (port 9104)
  • Grok exporter (port 9144)
  • Nginx VTS exporter (port 9913)
  • Postfix exporter (port 9154)

Create a ~/prometheus/docker-compose.yml with the Prometheus configuration:

version: "3.2"

services:
    prometheus:
        image: prom/prometheus:v2.27.0
        restart: "always"
        volumes:
        - './etc:/etc/prometheus'
        command:
        - '--web.enable-lifecycle'
        - '--config.file=/etc/prometheus/prometheus.yml'
        ports:
        - "19090:9090"

And create the ~/prometheus/etc/prometheus.yml configuration file:

global:
    scrape_interval: 60s
    scrape_timeout: 10s

scrape_configs:
# Scrape server endpoints
-   job_name: 'node'
    static_configs:
    -   targets: ['172.17.0.1:9100']
    relabel_configs:
    -   source_labels: [__address__]
        regex: '.*'
        target_label: instance
        replacement: '<HOSTNAME>'
-   job_name: 'mysql'
    static_configs:
    -   targets: ['172.17.0.1:9104']
    relabel_configs:
    -   source_labels: [__address__]
        regex: '.*'
        target_label: instance
        replacement: '<HOSTNAME>'
-   job_name: 'grok'
    static_configs:
    -   targets: ['172.17.0.1:9144']
    relabel_configs:
    -   source_labels: [__address__]
        regex: '.*'
        target_label: instance
        replacement: '<HOSTNAME>'
-   job_name: 'nginx'
    static_configs:
    -   targets: ['172.17.0.1:9913']
    relabel_configs:
    -   source_labels: [__address__]
        regex: '.*'
        target_label: instance
        replacement: '<HOSTNAME>'
-   job_name: 'postfix'
    static_configs:
    -   targets: ['172.17.0.1:9154']
    relabel_configs:
    -   source_labels: [__address__]
        regex: '.*'
        target_label: instance
        replacement: '<HOSTNAME>'

remote_write:
-   url: <REMOTE_WRITE_URL>
    basic_auth:
        username: <REMOTE_WRITE_USERNAME>
        password: <REMOTE_WRITE_PASSWORD>
    write_relabel_configs:
    -   source_labels: [__name__]
        action: drop
        regex: (mysql_info_schema_table_size|node_interrupts_total|mysql_info_schema_table_rows|mysql_info_schema_table_version|node_systemd_unit_state)
        replacement: $1

For a detailed explanation of the configurations, please check out the prometheus manual.

Within the Prometheus configuration file replace the <HOSTNAME>, <REMOTE_WRITE_URL>, <REMOTE_WRITE_USERNAME> and <REMOTE_WRITE_PASSWORD> tags. The tag <HOSTNAME> will just be the server name. All the other tags needs to be copied from your GrafanaLabs dashboard. In there you need to open the Prometheus details to view al these variables.

Grafana cloud is not very expensive when used correctly, but if you start sending every single metric we export, the costs will quickly grow. In our example, we already excluded some metrics with the write_relabel_configs. Please check out the following two articles to analyse the usage and add metrics you'd like to exclude to the regex field.

Configure Promtail

Promtail is used to read your log files and send them to Grafana cloud.

Add the promtail service to the ~/prometheus/docker-compose.yml file.

version: "3.2"

services:
    prometheus:
        ...

    promtail:
        image: grafana/promtail:2.2.1
        command:
        "-config.file=/etc/promtail/promtail.yml"
        volumes:
        - "./data:/tmp/positions"
        - "./etc:/etc/promtail"
        - "../domains/<domain>/var/log:/var/log"

You will need to create the ~/prometheus/etc/promtail.yml configuration file:

server:
    http_listen_port: 0
    grpc_listen_port: 0

positions:
    filename: /tmp/positions/positions.yaml

client:
    url: <GRAFANA_PUSH_URL>

scrape_configs:
-   job_name: system
    static_configs:
    -   targets:
        - localhost
        labels:
            job: nginx_access
            __path__: /var/log/*.nginx.access.log
    -   targets:
        - localhost
        labels:
            job: nginx_error
            __path__: /var/nginx/*.nginx.error.log

For a detailed explanation of the configurations, please check out the promtail manual.

Create the ~/prometheus/data folder and replace the <GRAFANA_PUSH_URL> with the URL found in grafana cloud.

Alerting

In Grafana cloud you can configure an alert manager to configure alerts on for example disk usage. A great resource for common configurations is https://monitoring.mixins.dev/.

Dashboards

Dashboards that work with the default exporters: