Tool for query logs

[uNitrous]

Prerequisites

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
  • Already Collapsed query log for time series data #1164

Problem Description

i want to analyze every month query log to check unwanted domains and boost my own personal blocklist. hard to analyze the querylog.json because it has hundreds duplicate domain

1. Can you recommend me any free tool or editor (atom, notepad etc) can use to remove duplicates? eg: test.net (50) so i need to cleanup duplicate querylog rows
2. If not, can adguardteam provide any script?

[uNitrous]

I saw this possible on Sublime Text. it's paid software
if you recommend solution, plz test first with querylog

I found that projects. i'm not sure yet
https://github.com/abame/ferret (8yrs ago)
https://github.com/justinnbt/SnaPy

[ameshkov]

Well, Sublime is available for free.

The more advanced solution would be to use ELK, but it'd require a lot of technical expertise.

[lolorc]

quick and ugly PoF ruby script to feed ES with querylog.json https://gist.github.com/lolorc/bcda84446f6ee0fc1815893769386a88

[cameronj86]

Late to the party, but the pandas library in python does the trick. Ideally you'd use jupyter notebook or a related extension in your IDE of choice. There are plenty of tutorial on YT to get your install/set-up going fwiw.

import pandas as pd
df= pd.read_json('querylog.json', lines=True)
df.head()