Installing Fluentbit

The following installation procedure helps you install Fluentbit.

Helm based installation

  1. The helm chart is available for the plugin.

To test, use the following command.

"https://artifacts-observability.sixthsense.rakuten.com/api/v4/projects/33/packages/generic/sixthsense-fluentbit-agent/1.0.0/fluentbit_helm.tgz"
  1. Add ss repository configuration.
kubectl create secret docker-registry regcred --docker-server=gitlab-registry.34.120.183.186.sslip.io --docker-username=<token-name> --docker-password=<token>
  1. Clean install the charts
helm uninstall ssfluentd
helm install -f overrides.yaml ssfluentd https://artifacts-observability.sixthsense.rakuten.com/api/v4/projects/33/packages/generic/sixthsense-fluentbit-agent/1.0.0/fluentbit_helm.tgz
  1. Parameters that can be overwritten in the following file.

overrides.yaml

image:
repository: asia-south1-docker.pkg.dev/sixthsense-pre-prod/sixthsense-fluentd-gcp/ssbit
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: 337bd8eb
configMap:
match:
apiKey: <<to get got from dashboard settings page>>
baseUri: https://http-collector-observability.sixthsense.rakuten.com/log/v1/log
serviceName: <<name of the agent - user defined>>
source:
path: /var/log/pods/*/*/*.log
posDb: /var/log/fluentd-json.log.db
excludePath: /var/log/pods/json*
service:
logLevel: debug
namespace: <<logging namespace>>
  1. Set your own parameters.

You can overwrite the values in the configuration file by writing your own file.

helm uninstall ssfluentd
helm install ssfluentd https://artifacts-observability.sixthsense.rakuten.com/api/v4/projects/33/packages/generic/sixthsense-fluentbit-agent/1.0.0/fluentbit_helm.tgz -f overrides.yaml

Installation in Linux environment

For more information about getting started with Flutentbit, see Getting started with Flutentbit.

Installing the plugin

Download the plugin file using the following command.

curl -vvv https://atrifactory.url.rakuten.com

Configure the plugin using the following files.

plugins.conf

[PLUGINS]
Path /fluent-bit/bin/out_ssfb.so

parsers.conf

[PARSER]
Name json
Format json

fluentbit.conf

[SERVICE]
Flush 1
Daemon off
Log_Level INFO
storage.checksum off
storage.path /var/log/flb-storage/
storage.backlog.mem_limit 20M
storage.metrics On
Plugins_File plugins.conf
parsers_file parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
[INPUT]
Name tail
Path {{ .Values.configMap.source.path }}
Tag raw.*
DB {{ .Values.configMap.source.posDb }}
Exclude_Path {{ .Values.configMap.source.excludePath }}
Path_Key log_name
Parser json
storage.type filesystem
Refresh_Interval 1
Rotate_Wait 15
Inotify_Watcher false
[FILTER]
Name record_modifier
Match raw.*
Record logType raw
[FILTER]
Name record_modifier
Match json.*
Record logType json
[OUTPUT]
name sixthsense
match *
tls true
apiKey {{ .Values.configMap.match.apiKey }}
logType raw
Endpoint {{ .Values.configMap.match.baseUri }}
serviceName {{ .Values.configMap.match.serviceName }}
Streaming true
  • {{ .Values.configMap.source.path }}= to be replaced with the path to be monitored.
note

Wild cards are also supported.

  • {{ .Values.configMap.source.posDb }} to be replaced with location of a database.
  • {{ .Values.configMap.source.excludePath }} to be replaced with paths that need to be excluded.

Sample configuration

[INPUT]
Name tail
Path /var/log/message*
Tag raw.*
DB /var/log/logs.db
Exclude_Path /var/log/system
Path_Key log_name
Parser json
storage.type filesystem
Refresh_Interval 1
Rotate_Wait 15
Inotify_Watcher false

  • {{ .Values.configMap.match.apiKey }} You can get the value from the Settings page in the SixthSense portal. For more information, see Accessing your access token.

  • {{ .Values.configMap.match.baseUri }} The value for the base URI

note

For GCP: https://http-collector-observability.sixthsense.rakuten.com/log/v1/log=

  • {{ .Values.configMap.match.serviceName }} The name that is shown in the UI of the dashboard.

Sample configuration

[OUTPUT]
name sixthsense
match *
tls true
apiKey eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0ZWFtSWQiOiIxOGZhMDJmMS1iNjc1LTQ1MmQtYjllYy0yN2ZhYzdlZWM5MzUiLCJiaWxsaW5nX2lkIjoiYjVjNjVkMDgtMTVhYi00NDkwLTliZDMtM2FiZDc3ODZkN2YxIiwiaWF0IjoxNjIwMzY2NDgyLCJhdWQiOiJvYXAiLCJpc3MiOiJzaXh0aC1zZW5zLWF1dGgifQ.5U4Tlh8Abf25Wa-SJaf4KQHnNz8C-S35hG7ZG5PIHo8
logType raw
Endpoint https://http-collector-observability.sixthsense.rakuten.com/log/v1/log
serviceName demo_service
Streaming true