Visual GPU Log Analytics Part I: CPU Baseline in Python Pandas

Visual GPU Log Analytics Part I: CPU Baseline in Python Pandas#

Graphistry is great – Graphistry and RAPIDS/BlazingDB is better!

This tutorial series visually analyzes Zeek/Bro network connection logs using different compute engines:

Part I Contents:

Time using CPU-based Python Pandas and Graphistry for a full ETL & visual analysis flow:

  1. Load data

  2. Analyze data

  3. Visualize data

[ ]:
#!pip install graphistry -q


import pandas as pd

import graphistry
graphistry.__version__

# To specify Graphistry account & server, use:
# graphistry.register(api=3, username='...', password='...', protocol='https', server='hub.graphistry.com')

# For more options, see https://github.com/graphistry/pygraphistry#configure

1. Load data#

[ ]:
%%time
# download data
#!if [ ! -f conn.log ]; then \
#    curl https://www.secrepo.com/maccdc2012/conn.log.gz | gzip -d > conn.log; \
#fi
[ ]:
#!head -n 3 conn.log
[ ]:
# OPTIONAL: For slow or limited devices, work on a subset:
LIMIT = 1200000
[ ]:
%%time
df = pd.read_csv("./conn.log", sep="\t", header=None,
                 names=["time", "uid", "id.orig_h", "id.orig_p", "id.resp_h", "id.resp_p", "proto", "service",
                        "duration", "orig_bytes", "resp_bytes", "conn_state", "local_orig", "missed_bytes",
                        "history", "orig_pkts", "orig_ip_bytes", "resp_pkts", "resp_ip_bytes", "tunnel_parents"],
                 na_values=['-'], index_col=False, nrows=LIMIT)
[ ]:
df.sample(3)

2. Analyze Data#

Summarize network activities between every communicating src/dst IP, split by connection state

[ ]:
df_summary = df\
.assign(
    sum_bytes=df.apply(lambda row: row['orig_bytes'] + row['resp_bytes'], axis=1))\
.groupby(['id.orig_h', 'id.resp_h', 'conn_state'])\
.agg({
    'time': ['min', 'max', 'size'],
    'id.resp_p':  ['nunique'],
    'uid': ['nunique'],
    'duration':   ['min', 'max', 'mean'],
    'orig_bytes': ['min', 'max', 'sum', 'mean'],
    'resp_bytes': ['min', 'max', 'sum', 'mean'],
    'sum_bytes':  ['min', 'max', 'sum', 'mean']
}).reset_index()

[ ]:
df_summary.columns = [' '.join(col).strip() for col in df_summary.columns.values]
df_summary = df_summary\
.rename(columns={'time size': 'count'})\
.assign(
    conn_state_uid=df_summary.apply(lambda row: row['id.orig_h'] + '_' + row['id.resp_h'] + '_' + row['conn_state'], axis=1))
[ ]:
print ('# rows', len(df_summary))
df_summary.sample(3)

3. Visualize data#

  • Nodes:

    • IPs

    • Bigger when more sessions (split by connection state) involving them

  • Edges:

    • src_ip -> dest_ip, split by connection state

[ ]:

hg = graphistry.hypergraph( df_summary, ['id.orig_h', 'id.resp_h'], direct=True, opts={ 'CATEGORIES': { 'ip': ['id.orig_h', 'id.resp_h'] } })
[ ]:

hg['graph'].plot()

Next Steps#

[ ]:

[ ]: