Collect website request stats into InfluxDB
Closed, ResolvedPublic

Description

The Grafana dashboards should have statistics on HTTP requests to our different websites. Apart from the usefulness of such graphs on their own, we could for example correlate peaks in CPU use to which particular website had a peak in requests.

We can feed the complete access logs to InfluxDB and then do aggregate queries on that. I experimented with this a few weeks ago.

  • Create separate InfluxDB retention policy (might need separate database instead?) so that we drop the per-request data sooner. The current default retention is several months, which is bad for storage and bad for our privacy policy. I guess we want 14 days.
  • Enable the Telegraf 'logparser' input plugin to parse access logs.
  • Add continuous queries to aggregate the data into another retention policy. This will let us keep longer-term statistics on request count per $time (minute I guess?), per website, per status code. but not individual requests.
  • Add graphs to Grafana dashboards.
  • If possible, backfill the database with existing logs.
nalvarez created this task.May 2 2020, 5:49 AM
nalvarez triaged this task as Low priority.
Restricted Application added a subscriber: sysadmin. · View Herald TranscriptMay 2 2020, 5:49 AM
bcooksley closed this task as Resolved.Jun 10 2020, 8:33 PM