How to Setup ILM in Kibana to Automate Log Management

Automating Log Management with ILM Policies and Index Templates in Kibana

As a DevOps engineer managing millions of logs per month, implementing an efficient log management strategy is crucial for maintaining optimal system performance and resource utilization. In this blog post, we'll explore how to set up Index Lifecycle Management (ILM) policies and index templates in Kibana to automatically delete old logs after a specified period, while ensuring data security by modifying index patterns.

Understanding Index Lifecycle Management

Index Lifecycle Management (ILM) is a powerful feature in Elasticsearch that automates the management of indices throughout their lifecycle. It allows you to define policies that govern the behavior of indices at different stages, such as hot, warm, cold, and delete. With ILM, you can specify rules for actions like data compression, allocation, and deletion based on criteria like age, size, or performance requirements.

Setting Up ILM Policies

To set up ILM policies for deleting old logs, follow these steps:

1. Create an ILM Policy for 30-Day Log Retention

PUT _ilm/policy/delete_after_30_days
{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

This policy specifies that indices should be deleted after 30 days (min_age: "30d").

2. Create an ILM Policy for 90-Day Log Retention

PUT _ilm/policy/delete_after_90_days
{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

This policy specifies that indices should be deleted after 90 days (min_age: "90d").

If you want to include additional phases such as hot, warm, and cold, you can modify the ILM policy to define the desired behavior for each phase. Here's an example of an ILM policy with hot, warm, and cold phases:

PUT _ilm/policy/hot_warm_cold_delete
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "shrink": {
            "number_of_shards": 1
          },
          "allocate": {
            "require": {
              "data": "warm"
            }
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

In this policy:

• The hot phase specifies that indices should be rolled over (new indices created) after 7 days (max_age: "7d").
• The warm phase starts after 7 days (min_age: "7d"). In this phase, the indices are force-merged (forcemerge), shrunk to a single shard (shrink), and allocated to data nodes with the "warm" attribute (allocate).
• The cold phase starts after 30 days (min_age: "30d"). In this phase, the indices are frozen to reduce their footprint and optimize for colder data access patterns.
• The delete phase starts after 90 days (min_age: "90d"), and the indices are deleted.

By including these additional phases, you can optimize your Elasticsearch cluster's performance and storage utilization based on the access patterns and priorities of your log data.

Setting Up Index Templates

To associate the created ILM policies with specific indices, you need to create index templates. These templates define the index patterns and the corresponding ILM policy to be applied.

1. Create an Index Template for 30-Day Log Retention

PUT _index_template/delete_after_30_days_template
{
  "index_patterns": [
    "your-web-logs-pattern-*",
    "your-php-logs-pattern-*",
    "your-pattern-1-*",
    "your-pattern-2-*",
    "your-pattern-3-*",
    "your-pattern-4-*"
  ],
  "template": {
    "settings": {
      "index.lifecycle.name": "delete_after_30_days"
    }
  }
}

This index template applies the delete_after_30_days ILM policy to the specified index patterns. Note that we've modified the index patterns for security reasons, replacing them with generic names like your-web-logs-pattern-* and your-php-logs-pattern-*.

2. Create an Index Template for 90-Day Log Retention

PUT _index_template/delete_after_90_days_template
{
  "index_patterns": [
    "analytics_fr_cobra_realtime-*",
    "analytics_br_cobra_realtime-*"
  ],
  "template": {
    "settings": {
      "index.lifecycle.name": "delete_after_90_days"
    }
  }
}

This index template applies the delete_after_90_days ILM policy to the specified index patterns.

Applying ILM Policies to Existing Indices

If you have existing indices that need to be managed by the ILM policies, you can apply the policies to them using the following commands:

PUT /your-web-logs-pattern-*,your-php-logs-pattern-*,your-pattern-1-*,your-pattern-2-*,your-pattern-3-*,your-pattern-4-*,analytics_fr_cobra_realtime-*,analytics_br_cobra_realtime-*/_settings
{
  "settings": {
    "index": {
      "lifecycle": {
        "name": "delete_after_30_days"
      }
    }
  }
}

PUT /analytics_fr_cobra_realtime-*,analytics_br_cobra_realtime-*/_settings
{
  "settings": {
    "index": {
      "lifecycle": {
        "name": "delete_after_90_days"
      }
    }
  }
}

These commands update the settings of the specified indices to apply the delete_after_30_days and delete_after_90_days ILM policies, respectively.

By implementing ILM policies and index templates, you can automate the deletion of old logs, ensuring efficient storage management and optimal performance of your Elasticsearch cluster. Additionally, by modifying the index patterns for security reasons, you can protect sensitive information from unauthorized access.

Remember to periodically review and adjust your log retention policies based on your organization's requirements, storage capacity, and security considerations.