Rakuten recommends installing td-agent version 4. However, you can also install the community Fluentd agent. For more information about installing Fluentd, see Overview.
Downloading and installing the SixthSense Fluentd plugin
Download the latest SixthSense plugin (.gem file) from gitlab artifactory using the https://artifacts-observability.sixthsense.rakuten.com/ URL. Credentials will be provided by the onboarding team.
The gem file package is available at https://artifacts-observability.sixthsense.rakuten.com/sixthsense/sixthsense-log-monitoring/-/packages/4103.
Download the fluent-plugin-sixthsense-1.2.1.gem file and configure the agent.
Installing SixthSense agent
If you installed fluentd from source or from gem, then plugins can be installed using the following gem command in the folder where the .tar or .zip files are available.
If you used Windows while installing fluentd, use the following command.
Adding SixthSense Agent plugin to tdagent
If you used the td-agent packaging while installing fluentd from DEB, RPM, .msi and so on use the following command in the folder where the .tar or .zip files are available.
Fluentd configuration file
In the above, example, there are three sections as follows:
In the example we are using tail plugin. The tail plugin is configured to read all the logs in the /var/log/containers/ directory. The position of the log file till which parsing is completed, is stored in the pos_file.
In the mandatory parse section, the multiline plugin is used for parsing stack traces which can span multiple lines. If this is not required, you can disable parsing using the none plugin
In the filter section, all logs tagged with kubernetes are processed. The record_transformer plugin is used for enriching the data with log_name, and hostname.
In the match section, SixthSense plugin is used. Here there are 3 fields:
api_key - The api key needs to be copied from the settings page in the SixthSense dashboard. base_uri - The SixthSense endpoint url. (currently pointing to satellite server directly) service_name - The service_name is the name given for the agent which will be displayed in the SixthSense dashboard with this name.
The services are now listed and displayed in the Log Monitoring dashboard.