How to manually purge data from Graylog 2.1
I have a Graylog 2.1 server that has been running for some time. I hadn't paid attention to my retention rate recently and came in this morning to find Graylog partially crashed because the disk was out of space. Nearly 100% of the disk space is currently being taken up by Elasticsearch Shards. The web interface for Graylog is not currently usable in the state it's in. I tried some of the standard Ubuntu tricks for freeing up disk space like apt-get autoremove and clean, but wasn't able to get enough to get the web interface functional.
The problem is all of the documentation I can currently find for changing the retention rate and cycling the shards, is via the web interface. The only config options no longer appear present in the Graylog config file.
Does anyone know of a manual, CLI, way of purging data from the Elasticsearch Shards in Graylog 2.1?
Answer
First aid: check which indices are present:
curl http://localhost:9200/_cat/indices
Then delete the oldest indices (you should not delete all)
curl -XDELETE http://localhost:9200/graylog_1
curl -XDELETE http://localhost:9200/graylog_2
curl -XDELETE http://localhost:9200/graylog_3
Fix: You can then reduce the parameter elasticsearch_max_number_of_indices in /etc/graylog/server/server.conf to a value that fits your disk.