Adguard Logs to Elasticsearch

with open(r'C:\Users\Edward\Desktop\querylog2.json', 'r') as file:
    for line in file:
        data = json.loads(line)
        print(data['T'])
        cleaned_timestamp = re.sub('(\d{6})(\d{0,3})(\+\d{2})(:)(\d{2})', r'\1\3\5', data['T'])
        parsed_timestamp = datetime.datetime.strptime(cleaned_timestamp, "%Y-%m-%dT%H:%M:%S.%f%z")

        print(parsed_timestamp.astimezone(datetime.timezone.utc).strftime('%Y-%m-%d %H:%M:%S'))
        if data['IP'].startswith('172'):
            continue
        data_to_send = {
            "timestamp": parsed_timestamp.isoformat(),
            "ip": data['IP'],
            "query": data['QH'],
        }
        print(data_to_send)
        es.index(index="adguard_query_log", document=data_to_send)

ELK DNS Heat Map

Scripted Fields

LocalDateTime.ofInstant(Instant.ofEpochMilli(doc['timestamp'].value.millis), ZoneId.of('Australia/Sydney')).getHour()
LocalDateTime.ofInstant(Instant.ofEpochMilli(doc['timestamp'].value.millis), ZoneId.of('Australia/Sydney')).getDayOfWeek().getDisplayName(TextStyle.FULL, Locale.getDefault())

Using Painless in Kibana scripted fields

ElasticTanya Bragin

importing