Visual GPU Log Analytics Part I: CPU Baseline in Python Pandas#
Graphistry is great – Graphistry and RAPIDS/BlazingDB is better!
This tutorial series visually analyzes Zeek/Bro network connection logs using different compute engines:
Part I: CPU Baseline in Python Pandas
Part III: GPU SQL - deprecated as Dask-SQL replaced BlazingSQL in the RAPIDS ecosystem
Part I Contents:
Time using CPU-based Python Pandas and Graphistry for a full ETL & visual analysis flow:
Load data
Analyze data
Visualize data
[ ]:
#!pip install graphistry -q
import pandas as pd
import graphistry
graphistry.__version__
# To specify Graphistry account & server, use:
# graphistry.register(api=3, username='...', password='...', protocol='https', server='hub.graphistry.com')
# For more options, see https://github.com/graphistry/pygraphistry#configure
1. Load data#
[ ]:
%%time
# download data
#!if [ ! -f conn.log ]; then \
# curl https://www.secrepo.com/maccdc2012/conn.log.gz | gzip -d > conn.log; \
#fi
[ ]:
#!head -n 3 conn.log
[ ]:
# OPTIONAL: For slow or limited devices, work on a subset:
LIMIT = 1200000
[ ]:
%%time
df = pd.read_csv("./conn.log", sep="\t", header=None,
names=["time", "uid", "id.orig_h", "id.orig_p", "id.resp_h", "id.resp_p", "proto", "service",
"duration", "orig_bytes", "resp_bytes", "conn_state", "local_orig", "missed_bytes",
"history", "orig_pkts", "orig_ip_bytes", "resp_pkts", "resp_ip_bytes", "tunnel_parents"],
na_values=['-'], index_col=False, nrows=LIMIT)
[ ]:
df.sample(3)
2. Analyze Data#
Summarize network activities between every communicating src/dst IP, split by connection state
[ ]:
df_summary = df\
.assign(
sum_bytes=df.apply(lambda row: row['orig_bytes'] + row['resp_bytes'], axis=1))\
.groupby(['id.orig_h', 'id.resp_h', 'conn_state'])\
.agg({
'time': ['min', 'max', 'size'],
'id.resp_p': ['nunique'],
'uid': ['nunique'],
'duration': ['min', 'max', 'mean'],
'orig_bytes': ['min', 'max', 'sum', 'mean'],
'resp_bytes': ['min', 'max', 'sum', 'mean'],
'sum_bytes': ['min', 'max', 'sum', 'mean']
}).reset_index()
[ ]:
df_summary.columns = [' '.join(col).strip() for col in df_summary.columns.values]
df_summary = df_summary\
.rename(columns={'time size': 'count'})\
.assign(
conn_state_uid=df_summary.apply(lambda row: row['id.orig_h'] + '_' + row['id.resp_h'] + '_' + row['conn_state'], axis=1))
[ ]:
print ('# rows', len(df_summary))
df_summary.sample(3)
3. Visualize data#
Nodes:
IPs
Bigger when more sessions (split by connection state) involving them
Edges:
src_ip -> dest_ip, split by connection state
[ ]:
hg = graphistry.hypergraph(
df_summary,
['id.orig_h', 'id.resp_h'],
direct=True,
opts={
'CATEGORIES': {
'ip': ['id.orig_h', 'id.resp_h']
}
})
[ ]:
hg['graph'].plot()
Next Steps#
Part I: CPU Baseline in Python Pandas
Part III: GPU SQL - deprecated as Dask-SQL replaced BlazingSQL in the RAPIDS ecosystem
[ ]:
[ ]: