6.4.6.2 X-Pack on Logstash

请参考官方文档完成部署 ， 部署步骤如下

Run bin/logstash-plugin install from the Logstash installation directory.

bin/logstash-plugin install x-pack

The plugin install scripts require direct internet access to download and install X-Pack. If your server doesn’t have internet access, specify the location of the X-Pack zip file that you downloaded to a temporary directory.

bin/logstash-plugin install file:///path/to/file/x-pack-6.2.4.zip

Configuring Security in Logstash

Logstash needs to be able to manage index templates, create indices, and write and delete documents in the indices it creates.

1. Use the the Management > Roles UI in Kibana or the role API to create a logstash_writer role. For cluster privileges, add manage_index_templates and monitor. For indices privileges, add write, create, delete, and create_index.

   If you plan to use index lifecycle management, also add manage_ilm for cluster and manage and manage_ilm for indices.

   POST _xpack/security/role/logstash_writer
   {
     "cluster": ["manage_index_templates", "monitor", "manage_ilm"], 
     "indices": [
       {
         "names": [ "logstash-*" ], 
         "privileges": ["write","create","delete","create_index","manage","manage_ilm"]  
       }
     ]
   }

   1. The cluster needs the manage_ilm privilege if index lifecycle management is enabled.

   2. If you use a custom Logstash index pattern, specify your custom pattern instead of the default logstash-* pattern.

   3. If index lifecycle management is enabled, the role requires the manage and manage_ilm privileges to load index lifecycle policies, create rollover aliases, and create and manage rollover indices.

2. Create a logstash_internal user and assign it the logstash_writer role. You can create users from the Management > Users UI in Kibana or through the user API:

   POST _xpack/security/user/logstash_internal
   {
     "password" : "x-pack-test-password",
     "roles" : [ "logstash_writer"],
     "full_name" : "Internal Logstash User"
   }

3. Configure Logstash to authenticate as the logstash_internal user you just created. You configure credentials separately for each of the Elasticsearch plugins in your Logstash .conf file. For example:

   input {
     elasticsearch {
       ...
       user => logstash_internal
       password => x-pack-test-password
     }
   }
   filter {
     elasticsearch {
       ...
       user => logstash_internal
       password => x-pack-test-password
     }
   }
   output {
     elasticsearch {
       ...
       user => logstash_internal
       password => x-pack-test-password
     }
   }

Monitoring for Logstash

关键参数说明

• xpack.monitoring.enabled

  设置为true或false来打开或者关闭monitoing, 默认为关闭

• xpack.monitoring.elasticsearch.url

  接受logstash发送的metries的elasticsearch地址, ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"]

• xpack.monitoring.elasticsearch.username and xpack.monitoring.elasticsearch.password

  elasticsearch开启了认证功能需要提供logstash_system的用户名和密码，

• xpack.monitoring.elasticsearch.sniffing

  将嗅探设置为true，以便发现elasticsearch集群的其他节点，默认值为false。

• xpack.monitoring.collection.interval

  控制在logstash端收集和发布metries的时间， 默认为10s

开启Monitoring

编辑logstash.yml, 添加如下内容

xpack.monitoring.enabled: True
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: lBJPQyPl0NBwGE6Cq8d0
xpack.monitoring.elasticsearch.url:
  - "http://es-welog02cn-p004.pek3.example.net:9200"
  - "http://es-welog02cn-p005.pek4.example.net:9200"
xpack.monitoring.elasticsearch.sniffing: true
xpack.monitoring.collection.interval: 10s

Centralized pipeline management

开启X-Pack Management功能后，启动logstash的时候就不用再配置logstash.conf文件了，启动的时候也不用再使用-f指定这个文件进行启动了， 开启这个功能

关键参数说明

• xpack.management.enabled

设置为true表示为Logstash开启X-Pack 集中式配置管理。

• xpack.management.logstash.poll_interval

Logstash实例轮询来自Elasticsearch的管道更改的频率。默认值为5s。

• xpack.management.pipeline.id

指定以逗号分隔的管道标识列表，以便为集中式管道生产管理注册。更改此设置后，您需要重新启动Logstash来使更改生效。 需要登录kibana --> management --> logstash --> pipelines创建pipeline

• xpack.management.elasticsearch.url

   存储Logstash管道配置和元数据的Elasticsearch示例。可以是和`outputs`中的相同的实例，也可以是不同的。默认是 `http://localhost:9200`.

• xpack.management.elasticsearch.usernameand xpack.management.elasticsearch.password

   如果你的Elasticsearch集群使用基本认证进行保护，这些设置提供用户名和密码，Logstash实例使用这些用户名和密码对访问配置数据进行身份验证。你在这里指定的用户名和密码必须具有`logstash_admin`角色，它提供对于`.logstash-*`的索引的认证。

开启pipe集中管理

编辑logstash.yml添加如下内容

xpack.management.enabled: True
xpack.management.pipeline.id:  
  - "prod_output_es"
  - "prod_output_kafka_plain"
  - "prod_output_kafka_json"
  - "prod_output_k8s_es"
  - "dev_output_es"
xpack.management.elasticsearch.username: logstash_admin_user
xpack.management.elasticsearch.password: MzZqbT44FU2oVJxatz4b
xpack.management.elasticsearch.url:
  - "http://es-welog02cn-p004.pek3.example.net:9200"
  - "http://es-welog02cn-p005.pek4.example.net:9200"
xpack.management.logstash.poll_interval: 5s