JournalIt mainly includes system log, application log, and security log. System operations and developers can understand the software and hardware information of the server by log, check the errors in the configuration process and the cause of the error. Regular analysis logs can understand server load, performance security, and take timely measures to correct errors.Mistake.
Under the architecture of micro service, useCentralized log management，Collect the log collection on all the servers.
After centralizing the management log, logStatistics and retrievalIt has become a more troublesome matter. In general, we can use grep, awk, WC and other Linux commands to achieve retrieval and statistics, but for higher requirements.Query, sort, and statisticsSuch requirements are still hard to avoid because of the large number of machines and other requirements.
A complete centralized log system should include the following main features:
- Collection – the ability to collect log data from a variety of sources
- Transmission – can transmit log data to the central system steadily.
- Storage – how to store log data
- Analysis – can support UI analysis
- Notice－Able to provide error reporting, monitoring mechanism
Open source real time log analysis ELKThe platform can perfectly solve the above problems.ELKBy ElasticSearch, Logstash and KiabanaThe three open source tools consist of:
（ELKA new member beates has been added to the system, so the renamed ELK Stack member has joined the Beats tool, so it has been renamed Elastic Stack: Elastic Stack == （ELK Stack + Beats））
ElasticsearchIt is an open source distributed search engine. Its characteristics are:Distributed，Zero configuration，Automatic discovery, index automatic fragmentation, index replica mechanism, restful style interface, multiple data sources, automatic search load, etc.
LogstashIt’s a completely open source tool. He can do your journal.Collect, filter, and store itFor future use (for example, search).
Kibana It is also an open source and free tool, and Kibana can provide friendly log analysis for Logstash and ElasticSearch. Web Interface helps you to aggregate, analyze, and search important data logs.。
- Beats，It is a lightweight log collector, in fact, there are 6 members of the Beats family, and the early ELK architecture uses Logstash to collect and parse logs, but Logstash has a high consumption of memory, CPU, IO and other resources. Compared to Logstash, BeatThe CPU and memory of the s system are negligible.
At present, Beats contains six tools:
- Packetbeat： Network data (collection of network traffic data)
- Metricbeat： Index (collection system, process and file system level CPU and memory usage data)
- Filebeat： Log files (collection of file data)
- Winlogbeat： windowsEvent log (collecting Windows event log data)
- Auditbeat：Audit data (collection of audit logs)
- Heartbeat：Run time monitoring (collection of data at runtime)