How better debug your ECS containers LOG


How better debug your ECS containers LOG


Tired of looking log in  AWS cloudwatch?


business auto GIF

  Try to use Graylog + Elasticsearch =😄
Simple steps*, install the Graylog in a small instance, configure the input, configure the driver for docker log, receive the logs.
*The security configuration like ports, network, users and password isn't in this post.


How To Install Graylog 3.0 on Ubuntu 18.04 / Ubuntu 16.04

Graylog is an open-source log management tool that helps you to collect, index and analyze any machine logs centrally. This guide focuses on installing Graylog on Ubuntu 18.04 / Ubuntu 16.04.

Components

MongoDB – Acts as a database, stores the configurations and meta information.
Elasticsearch – It stores the log messages and offers a searching facility. It is recommended to allocate more memory and use SAS or SAN disks for Elasticsearch nodes. Here, where all your searching happens.
Graylog Server – Log Parser. It collects the logs from various inputs and provides output to a built-in web interface for managing the logs.

Prerequisites

As you know, Elasticsearch is a java based application. Install either OpenJDK or Oracle JDK on your machine to proceed further.
Here, I will go for OpenJDK 8.
sudo apt update
sudo apt install -y apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen curl dirmngr
Verify the Java version.
java -version
Output:
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.18.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

Install Elasticsearch

Elasticsearch is one of the main components which requires Graylog to run, acts as a search server, offers a real-time distributed search and analytics with the RESTful web interface.
Elasticsearch stores all the logs sent by the Graylog server and displays the messages whenever user request over the built-in web interface.
Download and install the GPG signing key.
Set up Elasticsearch repository by running below command.
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
Update the repository cache and install Elasticsearch package.
sudo apt update
sudo apt install -y elasticsearch
Make Elasticsearch service to start automatically on the system startup.
sudo systemctl enable elasticsearch
The only important thing is to set a cluster name as graylog. Edit the configuration file of Elasticsearch and update it accordingly.
sudo nano /etc/elasticsearch/elasticsearch.yml
Set the cluster name shown like below.
cluster.name: graylog
Start the Elasticsearch service to read the new configurations.
sudo systemctl restart elasticsearch
Wait at least a minute to let the Elasticsearch get fully restarted. Elastisearch should be now listening on 9200 for the processing HTTP request. Use a CURL to check the response.
Output:
Ensure that cluster name shows as graylog.
{
  "name" : "bgVbYrc",
  "cluster_name" : "graylog",
  "cluster_uuid" : "-wECQlwnSZWftd_XdWSz-g",
  "version" : {
    "number" : "6.6.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "1fd8f69",
    "build_date" : "2019-02-13T17:10:04.160291Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
Optional: Test the health of Elasticsearch cluster.
Output:
Make sure the output yields the cluster status as green.
{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Install MongoDB

Download and install the latest version of MongoDB from the official website. Import the public key on the terminal to begin.
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
Add mongodb repository by creating the /etc/apt/sources.list.d/mongodb-org.list file using the following command.
### Ubuntu 18.04 ###
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
### Ubuntu 16.04 ###
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
Install MongoDB using the following command.
sudo apt update
sudo apt install -y mongodb-org
Start the MongoDB and enable it on the system start-up.
sudo systemctl start mongod
sudo systemctl enable mongod

Install Graylog

Graylog Server accepts and processes the log messages and then displays it for the requests that come from the graylog web interface.
Download and Install graylog 3.x repository.
wget https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.deb
sudo dpkg -i graylog-3.0-repository_latest.deb
Update the repository cache.
sudo apt update
Install the Graylog server using the following command.
sudo apt install -y graylog-server

Configure Graylog

You must set a secret to secure the user passwords. Use the pwgen command to the same.
pwgen -N 1 -s 96
Output:
fGoTI07CooB6xNy5sdPVSKSuq6QSu2QyWf6G9z3haolgwbERTQ9ZbfbF6hxRYbJMMAlEZX7CXHxJLBkNyfM0420u8aFuZy9M
Edit the server.conf file to begin the graylog configuration.
sudo nano /etc/graylog/server/server.conf
Place the secret like below.
password_secret = fGoTI07CooB6xNy5sdPVSKSuq6QSu2QyWf6G9z3haolgwbERTQ9ZbfbF6hxRYbJMMAlEZX7CXHxJLBkNyfM0420u8aFuZy9M
Next is to set a hash (sha256) password for the root user (not to be confused with the system user, the root user of graylog is admin).
You will need this password to login into the Graylog web interface. Admin’s password can’t be changed using the web interface. So, you must edit this variable to set.
Replace yourpassword with the choice of yours.
echo -n yourpassword | sha256sum
Output:
e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951
Edit the server.conf file again.
sudo nano /etc/graylog/server/server.conf
Place the hash password.
root_password_sha2 = e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951
You can set up email address admin user.
root_email = "itzgeek.web@gmail.com"
Set timezone of root (admin) user.
root_timezone = UTC
Set only one master node by defining the below variable. If you add a second Graylog node, set this to false to make the second node as a slave as the Master node does some periodic tasks that slave nodes won’t perform.
is_master = true
Set the number of log messages to keep per index; it is recommended to have several smaller indices instead of larger ones.
elasticsearch_max_docs_per_index = 20000000
The following parameter defines to have a total number of indices if this number is reached old index will be deleted.
elasticsearch_max_number_of_indices = 20
Shards setting rely on the number of nodes in the particular Elasticsearch cluster. If you have only one node, set it as 1.
elasticsearch_shards = 1
This setting defines the number of replicas for your indices. If you have only one node in the Elasticsearch cluster, then set it as 0.
elasticsearch_replicas = 0

Setup Graylog web interface

From the version 2.x, the web interface is being served directly by the Graylog server. Configure the Graylog web interface by editing the server.conf file.
sudo nano /etc/graylog/server/server.conf
Update the below entry with your system ip address from which you will access the Graylog web interface.
http_bind_address = 0.0.0.0:9000
Restart Graylog service.
sudo systemctl restart graylog-server
Make Graylog server to start automatically on system startup.
sudo systemctl enable graylog-server
You can check out the server startup logs, and it will be useful for you to troubleshoot Graylog in case of an issue.
sudo tail -f /var/log/graylog-server/server.log
On the successful start of graylog-server, you should get the following message in the log file.


 AWS ECS
Update the ECS-INIT-SCRIPT


# ECS Configuration
 - echo ECS_CLUSTER=${ecsname} >> /etc/ecs/ecs.config;echo ECS_AVAILABLE_LOGGING_DRIVERS=[\"json-file\",\"awslogs\",\"gelf\"] >> /etc/ecs/ecs.config


Access Graylog

http://yourip-ordomain:9000

Create a input



Open graylog and create a input, in menu  System> Inputs.

New input, GELF UDP



·        bind_address:
0.0.0.0
·        decompress_size_limit:
8388608
·        number_worker_threads:
2
·        override_source:
·        port:
3333
·        recv_buffer_size:
262144


Update Task Definition AWS

Create a new version,
Inside Amazon ECS
Go to Taks Definitions > Create new revision, get the last.





In the log configuration 
 
 
ADD the values
gelf-address
udp://mygraylogserver.com:3333



Other configuration in EC2 instances

Inside of the instance EC2, if is a old instance need to update the ECS configuration,
Update the configuration file, add in the logging drivers gelf
Nano /etc/ecs/ecs.config
ECS_CLUSTER=WordPress-DEV
ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","awslogs","gelf"]

“update and restart” the services in the AWS ECS instances.
Sudo yum update
sudo yum update -y ecs-init

sudo service docker restart && sudo start ecs




Thanks to passing here 😉
working rocket science GIF by US National Archives

Popular posts from this blog

Remote Server returned '550 5.2.3 RESOLVER.RST.RecipSizeLimit; message too large for this recipient'

Office 365 or Exchange online problem Remote Server returned '550 5.2.3 RESOLVER.RST.RecipSizeLimit; message too large for this recipient' Connect in your office365/Exchangeonlime with PowerShell Confirm the problem,   Get-Mailbox | fl mailboxplan,maxsendsize,maxreceivesize If you have   Get-Mailbox user@mailbox.com | fl mailboxplan,maxsendsize,maxreceivesize MailboxPlan    : ExchangeOnline-1ab122338-4f1a-bac1-c28ff8a1234 MaxSendSize    : 0 B (0 bytes) MaxReceiveSize : 0 B (0 bytes) Execute the command  Set-Mailbox user@mailbox.com -MaxReceiveSi ze 55 MB -MaxSend Size 55 MB  or   Set-Mailbox user@mailbox.com -MaxReceiveSize 150MB -MaxSendSize 150MB To apply for all users run this.  Get-Mailbox | Set-Mailbox -MaxReceiveSize 55MB -MaxSendSize 55MB
Read more

Haproxy error inconsistencies between private key and certificate loaded from PEM file

Error in haproxy with lets encrypt error msg bind *:443' : inconsistencies between private key and certificate loaded from PEM file '/etc/letsencrypt/live/ Need to create a new file cat cert.pem privkey.pem > haproxy_cert.pem Add in haproxy frontend www         bind *:80         bind *:443  ssl crt /etc/letsencrypt/live/mydomain.com/haproxy_cert.pem and make a test haproxy -c -V -f /etc/haproxy/haproxy.cfg
Read more

Porque a centralização de arquivos?

Image
Porque a centralização de arquivos?   Muitas empresas gastam boa parte de sua estrutura armazenando a mesma coisa, de forma desorganizada.  Para entender o problema vou tentar explicar de maneira simplificada, como visualizo o problema de gestão sobre os arquivos e a falta de cultura. E como resolver.
Read more