GFQL CPU, GPU Benchmark#

This notebook examines GFQL progerty graph query performance on 1-8 hop queries using CPU + GPU modes on various real-world 100K - 100M edge graphs. The data comes from a variety of popular social networks. The single-threaded CPU mode benefits from GFQL’s novel dataframe engine, and the GPU mode further adds single-GPU acceleration. Both the chain() and hop() methods are examined.

The benchmark does not examine bigger-than-memory and distributed scenarios. The provided results here are from running on a free Google Colab T4 runtime, with a 2.2GHz Intel CPU (12 GB CPU RAM) and T4 Nvidia GPU (16 GB GPU RAM).

Data#

From SNAP

Network

Nodes

Edges

Facebook

4,039

88,234

Twitter

81,306

2,420,766

GPlus

107,614

30,494,866

Orkut

3,072,441

117,185,082

Results#

Definitions:

  • GTEPS: Giga (billion) edges traversed per second

  • T edges / $: Estimated trillion edges traversed for 1$ USD based on observed GTEPS and a 3yr AWS reservation (as of 12/2023)

Tasks:

  1. chain() - includes complex pre/post processing

Task: g.chain([n({'id': some_id}), e_forward(hops=some_n)])

Dataset

Max GPU Speedup

CPU GTEPS

GPU GTEPS

T CPU edges / $ (t3.l)

T GPU edges / $ (g4dn.xl)

Facebook

1.1X

0.66

0.61

65.7

10.4

Twitter

17.4X

0.17

2.81

16.7

48.1

GPlus

43.8X

0.09

2.87

8.5

49.2

Orkut

N/A

N/A

12.15

N/A

208.3

AVG

20.7X

0.30

4.61

30.3

79.0

MAX

43.8X

0.66

12.15

65.7

208.3

  1. hop() - core property search primitive similar to BFS

Task: g.hop(nodes=[some_id], direction='forward', hops=some_n)

Dataset

Max GPU Speedup

CPU GTEPS

GPU GTEPS

T CPU edges / $ (t3.l)

T GPU edges / $ (g4dn.xl)

Facebook

3X

0.47

1.47

47.0

25.2

Twitter

42X

0.50

10.51

50.2

180.2

GPlus

21X

0.26

4.11

26.2

70.4

Orkut

N/A

N/A

41.50

N/A

711.4

AVG

22X

0.41

14.4

41.1

246.8

MAX

42X

0.50

41.50

50.2

711.4

Optional: GPU setup - Google Colab#

[1]:
# Report GPU used when GPU benchmarking
# ! nvidia-smi
Tue Dec 26 00:50:30 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   54C    P8              10W /  70W |      0MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
[8]:
# if in google colab
#!git clone https://github.com/rapidsai/rapidsai-csp-utils.git
#!python rapidsai-csp-utils/colab/pip-install.py
[3]:
import cudf
cudf.__version__
[3]:
'23.12.01'

1. Install & configure#

[2]:
#! pip install graphistry[igraph]
  Preparing metadata (setup.py) ... done

Imports#

[3]:
import pandas as pd

import graphistry

from graphistry import (

    # graph operators
    n, e_undirected, e_forward, e_reverse,

    # attribute predicates
    is_in, ge, startswith, contains, match as match_re
)
graphistry.__version__
[3]:
'0.32.0+12.g72e778c'
[6]:
import cudf
[7]:
#work around google colab shell encoding bugs

import locale
locale.getpreferredencoding = lambda: "UTF-8"

2. Perf benchmarks#

Facebook: 88K edges#

[10]:
df = pd.read_csv('https://raw.githubusercontent.com/graphistry/pygraphistry/master/demos/data/facebook_combined.txt', sep=' ', names=['s', 'd'])
print(df.shape)
df.head(5)
(88234, 2)
[10]:
s d
0 0 1
1 0 2
2 0 3
3 0 4
4 0 5
[11]:
fg = graphistry.edges(df, 's', 'd').materialize_nodes()
print(fg._nodes.shape, fg._edges.shape)
fg._nodes.head(5)
(4039, 1) (88234, 2)
[11]:
id
0 0
1 1
2 2
3 3
4 4
[12]:
%%time
for i in range(100):
  fg2 = fg.chain([n({'id': 0}), e_forward(hops=2)])
CPU times: user 13.6 s, sys: 2.08 s, total: 15.7 s
Wall time: 18 s
[ ]:
%%time
fg_gdf = fg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
for i in range(100):
  fg2 = fg_gdf.chain([n({'id': 0}), e_forward(hops=2)])
print(fg._nodes.shape, fg._edges.shape)
print(fg2._nodes.shape, fg2._edges.shape)
del fg_gdf
del fg2
(4039, 1) (88234, 2)
(1519, 1) (4060, 2)
CPU times: user 11.8 s, sys: 28.1 ms, total: 11.8 s
Wall time: 11.9 s
[ ]:
%%time
for i in range(50):
  fg2 = fg.chain([n({'id': 0}), e_forward(hops=5)])
print(fg._nodes.shape, fg._edges.shape)
print(fg2._nodes.shape, fg2._edges.shape)
(4039, 1) (88234, 2)
(3829, 1) (86074, 2)
CPU times: user 15.4 s, sys: 808 ms, total: 16.2 s
Wall time: 16.2 s
[ ]:
%%time
fg_gdf = fg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
for i in range(50):
  fg2 = fg_gdf.chain([n({'id': 0}), e_forward(hops=5)])
print(fg._nodes.shape, fg._edges.shape)
print(fg2._nodes.shape, fg2._edges.shape)
del fg_gdf
del fg2
(4039, 1) (88234, 2)
(3829, 1) (86074, 2)
CPU times: user 9.82 s, sys: 133 ms, total: 9.95 s
Wall time: 10.1 s
[ ]:
%%time
for i in range(100):
  fg2 = fg.chain([e_forward(source_node_match={'id': 0}, hops=5)])
CPU times: user 11.8 s, sys: 377 ms, total: 12.1 s
Wall time: 13.1 s
[ ]:
%%time
fg_gdf = fg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
for i in range(100):
  fg2 = fg_gdf.chain([e_forward(source_node_match={'id': 0}, hops=5)])
print(fg._nodes.shape, fg._edges.shape)
print(fg2._nodes.shape, fg2._edges.shape)
del fg_gdf
del fg2


(4039, 1) (88234, 2)
(348, 1) (347, 2)
CPU times: user 14.1 s, sys: 48.5 ms, total: 14.2 s
Wall time: 14.2 s
[17]:
%%time
start_nodes = pd.DataFrame({fg._node: [0]})
for i in range(100):
  fg2 = fg.hop(
      nodes=start_nodes,
      direction='forward',
      hops=2)
print(fg2._nodes.shape, fg2._edges.shape)
(1519, 1) (4060, 2)
CPU times: user 4.5 s, sys: 1.35 s, total: 5.85 s
Wall time: 6.09 s
[18]:
%%time
start_nodes = cudf.DataFrame({fg._node: [0]})
fg_gdf = fg.nodes(cudf.from_pandas(fg._nodes)).edges(cudf.from_pandas(fg._edges))
for i in range(100):
  fg2 = fg_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=2)
print(fg2._nodes.shape, fg2._edges.shape)
del start_nodes
del fg_gdf
del fg2
(1519, 1) (4060, 2)
CPU times: user 2.58 s, sys: 6.75 ms, total: 2.59 s
Wall time: 2.58 s
[19]:
%%time
start_nodes = pd.DataFrame({fg._node: [0]})
for i in range(100):
  fg2 = fg.hop(
      nodes=start_nodes,
      direction='forward',
      hops=5)
print(fg2._nodes.shape, fg2._edges.shape)
(3829, 1) (86074, 2)
CPU times: user 13.2 s, sys: 2 s, total: 15.2 s
Wall time: 18.3 s
[20]:
%%time
start_nodes = cudf.DataFrame({fg._node: [0]})
fg_gdf = fg.nodes(cudf.from_pandas(fg._nodes)).edges(cudf.from_pandas(fg._edges))
for i in range(100):
  fg2 = fg_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=5)
print(fg2._nodes.shape, fg2._edges.shape)
del start_nodes
del fg_gdf
del fg2
(3829, 1) (86074, 2)
CPU times: user 5.72 s, sys: 159 ms, total: 5.88 s
Wall time: 5.86 s

Twitter#

  • edges: 2420766

  • nodes: 81306

[21]:
#! wget 'https://snap.stanford.edu/data/twitter_combined.txt.gz'
--2023-12-25 21:58:27--  https://snap.stanford.edu/data/twitter_combined.txt.gz
Resolving snap.stanford.edu (snap.stanford.edu)... 171.64.75.80
Connecting to snap.stanford.edu (snap.stanford.edu)|171.64.75.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10621918 (10M) [application/x-gzip]
Saving to: ‘twitter_combined.txt.gz’

twitter_combined.tx 100%[===================>]  10.13M  3.00MB/s    in 4.0s

2023-12-25 21:58:32 (2.52 MB/s) - ‘twitter_combined.txt.gz’ saved [10621918/10621918]

[22]:
#! gunzip twitter_combined.txt.gz
[24]:
#! head -n 5 twitter_combined.txt
214328887 34428380
17116707 28465635
380580781 18996905
221036078 153460275
107830991 17868918
[25]:
%%time
te_df = pd.read_csv('twitter_combined.txt', sep=' ', names=['s', 'd'])
te_df.shape
CPU times: user 474 ms, sys: 61.9 ms, total: 536 ms
Wall time: 534 ms
[25]:
(2420766, 2)
[26]:
import graphistry
[27]:
%%time
g = graphistry.edges(te_df, 's', 'd').materialize_nodes()
g._nodes.shape
CPU times: user 86.4 ms, sys: 106 ms, total: 193 ms
Wall time: 191 ms
[27]:
(81306, 1)
[29]:
%%time
for i in range(10):
  g2 = g.chain([n({'id': 17116707}), e_forward(hops=1)])
g2._nodes.shape, g2._edges.shape
CPU times: user 11.8 s, sys: 8.4 s, total: 20.2 s
Wall time: 23 s
[29]:
((140, 1), (615, 2))
[30]:
%%time
g_gdf = g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
for i in range(10):
  out = g_gdf.chain([n({'id': 17116707}), e_forward(hops=1)])._nodes
print(out.shape)
del g_gdf
del out
(140, 1)
CPU times: user 1.33 s, sys: 46.6 ms, total: 1.38 s
Wall time: 1.63 s
[31]:
%%time
for i in range(10):
  out = g.chain([n({'id': 17116707}), e_forward(hops=2)])
print(out._nodes.shape, out._edges.shape)
(2345, 1) (68536, 2)
CPU times: user 13.3 s, sys: 8.05 s, total: 21.4 s
Wall time: 21.6 s
[36]:
%%time
g_gdf = g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
for i in range(10):
  out = g_gdf.chain([n({'id': 17116707}), e_forward(hops=2)])._nodes
print(out.shape)
del g_gdf
del out
(2345, 1)
CPU times: user 1.67 s, sys: 55.8 ms, total: 1.72 s
Wall time: 1.75 s
[37]:
%%time
for i in range(10):
  out = g.chain([n({'id': 17116707}), e_forward(hops=8)])
print(out._nodes.shape, out._edges.shape)
(81304, 1) (2417796, 2)
CPU times: user 1min 56s, sys: 17.1 s, total: 2min 13s
Wall time: 2min 22s
[38]:
%%time
g_gdf = g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
for i in range(10):
  out = g_gdf.chain([n({'id': 17116707}), e_forward(hops=8)])._nodes
print(out.shape)
del g_gdf
del out
(81304, 1)
CPU times: user 5.3 s, sys: 1.48 s, total: 6.78 s
Wall time: 7.89 s
[39]:
%%time
start_nodes = pd.DataFrame({g._node: [17116707]})
for i in range(10):
  g2 = g.hop(
      nodes=start_nodes,
      direction='forward',
      hops=1)
print(g2._nodes.shape, g2._edges.shape)
(0, 1) (0, 2)
CPU times: user 2.58 s, sys: 1.59 s, total: 4.17 s
Wall time: 6.02 s
[44]:
%%time
start_nodes = cudf.DataFrame({g._node: [17116707]})
g_gdf = g.nodes(cudf.from_pandas(g._nodes)).edges(cudf.from_pandas(g._edges))
for i in range(10):
  g2 = g_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=5)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del g_gdf
del g2
(61827, 1) (1473599, 2)
CPU times: user 822 ms, sys: 179 ms, total: 1 s
Wall time: 997 ms
[40]:
%%time
start_nodes = pd.DataFrame({g._node: [17116707]})
for i in range(10):
  g2 = g.hop(
      nodes=start_nodes,
      direction='forward',
      hops=2)
print(g2._nodes.shape, g2._edges.shape)
(2345, 1) (68536, 2)
CPU times: user 8.93 s, sys: 5.92 s, total: 14.9 s
Wall time: 15.8 s
[41]:
%%time
start_nodes = cudf.DataFrame({g._node: [17116707]})
g_gdf = g.nodes(cudf.from_pandas(g._nodes)).edges(cudf.from_pandas(g._edges))
for i in range(10):
  g2 = g_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=2)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del g_gdf
del g2
(2345, 1) (68536, 2)
CPU times: user 374 ms, sys: 6.92 ms, total: 381 ms
Wall time: 379 ms
[42]:
%%time
start_nodes = pd.DataFrame({g._node: [17116707]})
for i in range(10):
  g2 = g.hop(
      nodes=start_nodes,
      direction='forward',
      hops=8)
print(g2._nodes.shape, g2._edges.shape)
(81304, 1) (2417796, 2)
CPU times: user 38.8 s, sys: 8.7 s, total: 47.5 s
Wall time: 48.2 s
[43]:
%%time
start_nodes = cudf.DataFrame({g._node: [17116707]})
g_gdf = g.nodes(cudf.from_pandas(g._nodes)).edges(cudf.from_pandas(g._edges))
for i in range(10):
  g2 = g_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=8)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del g_gdf
del g2
(81304, 1) (2417796, 2)
CPU times: user 1.8 s, sys: 506 ms, total: 2.3 s
Wall time: 2.3 s

GPlus#

  • edges: 30494866

  • nodes: 107614

[4]:
#! wget https://snap.stanford.edu/data/gplus_combined.txt.gz
--2023-12-26 18:36:29--  https://snap.stanford.edu/data/gplus_combined.txt.gz
Resolving snap.stanford.edu (snap.stanford.edu)... 171.64.75.80
Connecting to snap.stanford.edu (snap.stanford.edu)|171.64.75.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 398930514 (380M) [application/x-gzip]
Saving to: ‘gplus_combined.txt.gz’

gplus_combined.txt. 100%[===================>] 380.45M  34.7MB/s    in 9.6s

2023-12-26 18:36:39 (39.7 MB/s) - ‘gplus_combined.txt.gz’ saved [398930514/398930514]

[5]:
#! gunzip gplus_combined.txt.gz
[6]:
%%time
ge_df = pd.read_csv('gplus_combined.txt', sep=' ', names=['s', 'd'])
print(ge_df.shape)
ge_df.head(5)
(30494866, 2)
CPU times: user 16 s, sys: 1.45 s, total: 17.5 s
Wall time: 22.5 s
[6]:
s d
0 116374117927631468606 101765416973555767821
1 112188647432305746617 107727150903234299458
2 116719211656774388392 100432456209427807893
3 117421021456205115327 101096322838605097368
4 116407635616074189669 113556266482860931616
[7]:
%%time
gg = graphistry.edges(ge_df, 's', 'd').materialize_nodes()
gg = graphistry.edges(ge_df, 's', 'd').nodes(gg._nodes, 'id')
print(gg._edges.shape, gg._nodes.shape)
gg._nodes.head(5)
(30494866, 2) (107614, 1)
CPU times: user 4.49 s, sys: 1.25 s, total: 5.74 s
Wall time: 5.97 s
[7]:
id
0 116374117927631468606
1 112188647432305746617
2 116719211656774388392
3 117421021456205115327
4 116407635616074189669
[49]:
%%time
gg.chain([ n({'id': '116374117927631468606'})])._nodes
CPU times: user 534 ms, sys: 598 ms, total: 1.13 s
Wall time: 1.65 s
[49]:
id
0 116374117927631468606
[ ]:
%%time
out = gg.chain([ n({'id': '116374117927631468606'}), e_forward(hops=1)])._nodes
out.shape
CPU times: user 27.5 s, sys: 11.1 s, total: 38.5 s
Wall time: 39.5 s
(1473, 1)
[ ]:
%%time
gg_gdf = gg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
out = gg_gdf.chain([ n({'id': '116374117927631468606'}), e_forward(hops=1)])
print(out._nodes.shape, out._edges.shape)
del gg_gdf
del out
(1473, 1) (13375, 2)
CPU times: user 4.57 s, sys: 2.11 s, total: 6.68 s
Wall time: 7.63 s
[ ]:
%%time
out = gg.chain([ n({'id': '116374117927631468606'}), e_forward(hops=2)])._nodes
out.shape
CPU times: user 45.8 s, sys: 17 s, total: 1min 2s
Wall time: 1min 5s
(44073, 1)
[ ]:
%%time
gg_gdf = gg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
out = gg_gdf.chain([ n({'id': '116374117927631468606'}), e_forward(hops=2)])
print(out._nodes.shape, out._edges.shape)
del gg_gdf
del out
(44073, 1) (2069325, 2)
CPU times: user 4.97 s, sys: 2.36 s, total: 7.34 s
Wall time: 10.6 s
[ ]:
%%time
out = gg.chain([ n({'id': '116374117927631468606'}), e_forward(hops=3)])._nodes
out.shape
CPU times: user 3min 45s, sys: 1min 5s, total: 4min 50s
Wall time: 4min 52s
(102414, 1)
[ ]:
%%time
gg_gdf = gg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
out = gg_gdf.chain([ n({'id': '116374117927631468606'}), e_forward(hops=3)])
print(out._nodes.shape, out._edges.shape)
del gg_gdf
del out
(102414, 1) (24851333, 2)
CPU times: user 6.95 s, sys: 2.63 s, total: 9.57 s
Wall time: 9.84 s
[8]:
%%time
out = gg.chain([ n({'id': '116374117927631468606'}), e_forward(hops=4)])
print(out._nodes.shape, out._edges.shape)
(105479, 1) (30450354, 2)
CPU times: user 4min 36s, sys: 1min 25s, total: 6min 2s
Wall time: 6min 4s
[ ]:
%%time
gg_gdf = gg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
out = gg_gdf.chain([ n({'id': '116374117927631468606'}), e_forward(hops=4)])
print(out._nodes.shape, out._edges.shape)
del gg_gdf
del out
(105479, 1) (30450354, 2)
CPU times: user 7.44 s, sys: 2.45 s, total: 9.88 s
Wall time: 9.9 s
[9]:
%%time
out = gg.chain([ n({'id': '116374117927631468606'}), e_forward(hops=5)])
print(out._nodes.shape, out._edges.shape)
(105604, 1) (30468335, 2)
CPU times: user 5min 36s, sys: 1min 39s, total: 7min 16s
Wall time: 7min 15s
[ ]:
%%time
gg_gdf = gg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
out = gg_gdf.chain([ n({'id': '116374117927631468606'}), e_forward(hops=5)])
print(out._nodes.shape, out._edges.shape)
del gg_gdf
del out
(105604, 1) (30468335, 2)
CPU times: user 8.82 s, sys: 2.71 s, total: 11.5 s
Wall time: 11.9 s
[50]:
%%time
start_nodes = pd.DataFrame({gg._node: ['116374117927631468606']})
for i in range(1):
  g2 = gg.hop(
      nodes=start_nodes,
      direction='forward',
      hops=1)
print(g2._nodes.shape, g2._edges.shape)
(1473, 1) (13375, 2)
CPU times: user 19.9 s, sys: 9.36 s, total: 29.2 s
Wall time: 41.8 s
[52]:
%%time
start_nodes = cudf.DataFrame({gg._node: ['116374117927631468606']})
gg_gdf = gg.nodes(cudf.from_pandas(gg._nodes)).edges(cudf.from_pandas(gg._edges))
for i in range(1):
  g2 = gg_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=1)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del gg_gdf
del g2
(1473, 1) (13375, 2)
CPU times: user 3.71 s, sys: 2.09 s, total: 5.8 s
Wall time: 6.05 s
[53]:
%%time
start_nodes = pd.DataFrame({gg._node: ['116374117927631468606']})
for i in range(1):
  g2 = gg.hop(
      nodes=start_nodes,
      direction='forward',
      hops=2)
print(g2._nodes.shape, g2._edges.shape)
(44073, 1) (2069325, 2)
CPU times: user 27.8 s, sys: 13.2 s, total: 41 s
Wall time: 43.9 s
[54]:
%%time
start_nodes = cudf.DataFrame({gg._node: ['116374117927631468606']})
gg_gdf = gg.nodes(cudf.from_pandas(gg._nodes)).edges(cudf.from_pandas(gg._edges))
for i in range(1):
  g2 = gg_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=2)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del gg_gdf
del g2
(44073, 1) (2069325, 2)
CPU times: user 4.26 s, sys: 2.37 s, total: 6.63 s
Wall time: 7.91 s
[55]:
%%time
start_nodes = pd.DataFrame({gg._node: ['116374117927631468606']})
for i in range(1):
  g2 = gg.hop(
      nodes=start_nodes,
      direction='forward',
      hops=3)
print(g2._nodes.shape, g2._edges.shape)
(102414, 1) (24851333, 2)
CPU times: user 1min 3s, sys: 22.7 s, total: 1min 26s
Wall time: 1min 35s
[56]:
%%time
start_nodes = cudf.DataFrame({gg._node: ['116374117927631468606']})
gg_gdf = gg.nodes(cudf.from_pandas(gg._nodes)).edges(cudf.from_pandas(gg._edges))
for i in range(1):
  g2 = gg_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=3)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del gg_gdf
del g2
(102414, 1) (24851333, 2)
CPU times: user 3.96 s, sys: 2.11 s, total: 6.07 s
Wall time: 6.05 s
[57]:
%%time
start_nodes = pd.DataFrame({gg._node: ['116374117927631468606']})
for i in range(1):
  g2 = gg.hop(
      nodes=start_nodes,
      direction='forward',
      hops=4)
print(g2._nodes.shape, g2._edges.shape)
(105479, 1) (30450354, 2)
CPU times: user 1min 34s, sys: 30.6 s, total: 2min 5s
Wall time: 2min 5s
[58]:
%%time
start_nodes = cudf.DataFrame({gg._node: ['116374117927631468606']})
gg_gdf = gg.nodes(cudf.from_pandas(gg._nodes)).edges(cudf.from_pandas(gg._edges))
for i in range(1):
  g2 = gg_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=4)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del gg_gdf
del g2
(105479, 1) (30450354, 2)
CPU times: user 5.25 s, sys: 2.41 s, total: 7.67 s
Wall time: 7.69 s
[59]:
%%time
start_nodes = pd.DataFrame({gg._node: ['116374117927631468606']})
for i in range(1):
  g2 = gg.hop(
      nodes=start_nodes,
      direction='forward',
      hops=5)
print(g2._nodes.shape, g2._edges.shape)
(105604, 1) (30468335, 2)
CPU times: user 2min 16s, sys: 39.1 s, total: 2min 55s
Wall time: 2min 58s
[60]:
%%time
start_nodes = cudf.DataFrame({gg._node: ['116374117927631468606']})
gg_gdf = gg.nodes(cudf.from_pandas(gg._nodes)).edges(cudf.from_pandas(gg._edges))
for i in range(1):
  g2 = gg_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=5)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del gg_gdf
del g2
(105604, 1) (30468335, 2)
CPU times: user 5.79 s, sys: 2.51 s, total: 8.3 s
Wall time: 8.29 s

Orkut#

  • 117M edges

  • 3M nodes

[8]:
#! wget https://snap.stanford.edu/data/bigdata/communities/com-orkut.ungraph.txt.gz
--2023-12-26 00:55:52--  https://snap.stanford.edu/data/bigdata/communities/com-orkut.ungraph.txt.gz
Resolving snap.stanford.edu (snap.stanford.edu)... 171.64.75.80
Connecting to snap.stanford.edu (snap.stanford.edu)|171.64.75.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 447251958 (427M) [application/x-gzip]
Saving to: ‘com-orkut.ungraph.txt.gz’

com-orkut.ungraph.t 100%[===================>] 426.53M  45.1MB/s    in 9.7s

2023-12-26 00:56:02 (44.0 MB/s) - ‘com-orkut.ungraph.txt.gz’ saved [447251958/447251958]

[9]:
#! gunzip com-orkut.ungraph.txt.gz
[10]:
#! head -n 7 com-orkut.ungraph.txt
# Undirected graph: ../../data/output/orkut.txt
# Orkut
# Nodes: 3072441 Edges: 117185083
# FromNodeId    ToNodeId
1       2
1       3
1       4
[11]:
import pandas as pd

import graphistry

from graphistry import (

    # graph operators
    n, e_undirected, e_forward, e_reverse,

    # attribute predicates
    is_in, ge, startswith, contains, match as match_re
)

import cudf

#work around google colab shell encoding bugs
import locale
locale.getpreferredencoding = lambda: "UTF-8"

cudf.__version__, graphistry.__version__
[11]:
('23.12.01', '0.32.0+12.g72e778c')
[12]:
#! nvidia-smi
Tue Dec 26 00:56:27 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   47C    P0              27W /  70W |    103MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
[13]:
%%time
co_df = cudf.read_csv('com-orkut.ungraph.txt', sep='\t', names=['s', 'd'], skiprows=5).to_pandas()
print(co_df.shape)
print(co_df.head(5))
print(co_df.dtypes)
#del co_df
(117185082, 2)
   s  d
0  1  3
1  1  4
2  1  5
3  1  6
4  1  7
s    int64
d    int64
dtype: object
CPU times: user 2.56 s, sys: 4.2 s, total: 6.76 s
Wall time: 6.76 s
[14]:
%%time
co_g = graphistry.edges(cudf.DataFrame(co_df), 's', 'd').materialize_nodes(engine='cudf')
co_g = co_g.nodes(lambda g: g._nodes.to_pandas()).edges(lambda g: g._edges.to_pandas())
print(co_g._nodes.shape, co_g._edges.shape)
co_g._nodes.head(5)
(3072441, 1) (117185082, 2)
CPU times: user 1.96 s, sys: 2.95 s, total: 4.91 s
Wall time: 4.92 s
[14]:
id
0 1
1 2
2 3
3 4
4 5
[15]:
#! nvidia-smi
Tue Dec 26 00:56:39 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   49C    P0              27W /  70W |   2819MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
[ ]:
%%time
# crashes
if False:
  out = co_g.chain([ n({'id': 1}), e_forward(hops=1)])._nodes
  print(out.shape)
  del out
CPU times: user 4 µs, sys: 1 µs, total: 5 µs
Wall time: 47.7 µs
[ ]:
%%time
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
  out = co_gdf.chain([ n({'id': 1}), e_forward(hops=1)])
#! nvidia-smi
print(out._nodes.shape, out._edges.shape)
del co_gdf
del out
Mon Dec 25 06:23:46 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   63C    P0              30W /  70W |   1925MiB / 15360MiB |     35%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Mon Dec 25 06:23:49 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   66C    P0              72W /  70W |   2845MiB / 15360MiB |     84%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(12, 1) (11, 2)
CPU times: user 4.42 s, sys: 131 ms, total: 4.55 s
Wall time: 4.42 s
[ ]:
%%time
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
  out = co_gdf.chain([ n({'id': 1}), e_forward(hops=2)])
#! nvidia-smi
print(out._nodes.shape, out._edges.shape)
del co_gdf
del out
Mon Dec 25 06:24:52 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   61C    P0              29W /  70W |   1925MiB / 15360MiB |     22%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Mon Dec 25 06:24:58 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   64C    P0              71W /  70W |   2845MiB / 15360MiB |     57%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(391, 1) (461, 2)
CPU times: user 5.34 s, sys: 132 ms, total: 5.47 s
Wall time: 6.13 s
[ ]:
%%time
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
  out = co_gdf.chain([ n({'id': 1}), e_forward(hops=3)])
#! nvidia-smi
print(out._nodes.shape, out._edges.shape)
del co_gdf
del out
Mon Dec 25 06:25:25 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   61C    P0              29W /  70W |   1925MiB / 15360MiB |     31%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Mon Dec 25 06:25:31 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   65C    P0              71W /  70W |   2849MiB / 15360MiB |     58%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(21767, 1) (28480, 2)
CPU times: user 6.25 s, sys: 100 ms, total: 6.35 s
Wall time: 6.37 s
[ ]:
%%time
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
  out = co_gdf.chain([ n({'id': 1}), e_forward(hops=4)])
#! nvidia-smi
print(out._nodes.shape, out._edges.shape)
del co_gdf
del out
Mon Dec 25 06:26:04 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   61C    P0              29W /  70W |   1927MiB / 15360MiB |     36%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Mon Dec 25 06:26:13 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   65C    P0              71W /  70W |   2931MiB / 15360MiB |     90%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(718640, 1) (2210961, 2)
CPU times: user 9.01 s, sys: 1.03 s, total: 10 s
Wall time: 9.84 s
[ ]:
%%time
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
  out = co_gdf.chain([ n({'id': 1}), e_forward(hops=5)])
#! nvidia-smi
print(out._nodes.shape, out._edges.shape)
del co_gdf
del out
Mon Dec 25 06:27:18 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   60C    P0              29W /  70W |   1927MiB / 15360MiB |     28%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Mon Dec 25 06:27:57 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   72C    P0              43W /  70W |   4351MiB / 15360MiB |    100%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(3041556, 1) (47622917, 2)
CPU times: user 34.9 s, sys: 4.76 s, total: 39.6 s
Wall time: 39.2 s
[ ]:
%%time
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
out = co_gdf.chain([ n({'id': 1}), e_forward(hops=6)])._nodes
print(out.shape)
del co_gdf
del out
[ ]:
#!lscpu

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  2
  On-line CPU(s) list:   0,1
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) CPU @ 2.20GHz
    CPU family:          6
    Model:               79
    Thread(s) per core:  2
    Core(s) per socket:  1
    Socket(s):           1
    Stepping:            0
    BogoMIPS:            4399.99
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clf
                         lush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_
                         good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fm
                         a cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hyp
                         ervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsb
                         ase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsa
                         veopt arat md_clear arch_capabilities
Virtualization features:
  Hypervisor vendor:     KVM
  Virtualization type:   full
Caches (sum of all):
  L1d:                   32 KiB (1 instance)
  L1i:                   32 KiB (1 instance)
  L2:                    256 KiB (1 instance)
  L3:                    55 MiB (1 instance)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0,1
Vulnerabilities:
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Mitigation; PTE Inversion
  Mds:                   Vulnerable; SMT Host state unknown
  Meltdown:              Vulnerable
  Mmio stale data:       Vulnerable
  Retbleed:              Vulnerable
  Spec rstack overflow:  Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Vulnerable: __user pointer sanitization and usercopy barriers only; no swap
                         gs barriers
  Spectre v2:            Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
  Srbds:                 Not affected
  Tsx async abort:       Vulnerable
[ ]:
#!free -h

               total        used        free      shared  buff/cache   available
Mem:            12Gi       717Mi       8.0Gi       1.0Mi       3.9Gi        11Gi
Swap:             0B          0B          0B
[ ]:
%%time
start_nodes = pd.DataFrame({'id': [1]})
#! nvidia-smi
for i in range(1):
  g2 = co_g.hop(
      nodes=start_nodes,
      direction='forward',
      hops=1)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
#del start_nodes
#del co_gdf
#del g2
Tue Dec 26 01:01:43 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   64C    P0              30W /  70W |   2821MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
[16]:
%%time
start_nodes = cudf.DataFrame({'id': [1]})
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
  g2 = co_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=1)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del co_gdf
del g2
Tue Dec 26 00:56:45 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   49C    P0              28W /  70W |   1923MiB / 15360MiB |     37%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Tue Dec 26 00:56:47 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   52C    P0              70W /  70W |   2819MiB / 15360MiB |     79%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(12, 1) (11, 2)
CPU times: user 1.6 s, sys: 37.3 ms, total: 1.64 s
Wall time: 1.84 s
[17]:
%%time
start_nodes = cudf.DataFrame({'id': [1]})
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
  g2 = co_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=2)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del co_gdf
del g2
Tue Dec 26 00:56:47 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   51C    P0              35W /  70W |   1923MiB / 15360MiB |     59%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Tue Dec 26 00:56:49 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   53C    P0              59W /  70W |   2821MiB / 15360MiB |     86%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(391, 1) (461, 2)
CPU times: user 2.32 s, sys: 58.5 ms, total: 2.38 s
Wall time: 2.51 s
[18]:
%%time
start_nodes = cudf.DataFrame({'id': [1]})
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
  g2 = co_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=3)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del co_gdf
del g2
Tue Dec 26 00:56:50 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   52C    P0              36W /  70W |   1925MiB / 15360MiB |     55%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Tue Dec 26 00:56:53 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   54C    P0              75W /  70W |   2825MiB / 15360MiB |     74%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(21767, 1) (28480, 2)
CPU times: user 3.04 s, sys: 63.6 ms, total: 3.1 s
Wall time: 3.25 s
[19]:
%%time
start_nodes = cudf.DataFrame({'id': [1]})
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
  g2 = co_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=4)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del co_gdf
del g2
Tue Dec 26 00:56:53 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   54C    P0              36W /  70W |   1927MiB / 15360MiB |     54%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Tue Dec 26 00:56:58 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   56C    P0              38W /  70W |   2907MiB / 15360MiB |     89%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(718640, 1) (2210961, 2)
CPU times: user 4.58 s, sys: 309 ms, total: 4.89 s
Wall time: 5.02 s
[20]:
%%time
start_nodes = cudf.DataFrame({'id': [1]})
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
  g2 = co_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=5)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del co_gdf
del g2
Tue Dec 26 00:56:58 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   55C    P0              37W /  70W |   1925MiB / 15360MiB |     59%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Tue Dec 26 00:57:10 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   60C    P0              48W /  70W |   4325MiB / 15360MiB |     99%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(3041556, 1) (47622917, 2)
CPU times: user 10.8 s, sys: 1.29 s, total: 12.1 s
Wall time: 12 s
[21]:
%%time
start_nodes = cudf.DataFrame({'id': [1]})
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
  g2 = co_gdf.hop(
      nodes=start_nodes,
      direction='forward',
      hops=6)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del co_gdf
del g2
Tue Dec 26 00:57:10 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   59C    P0              38W /  70W |   1925MiB / 15360MiB |     44%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Tue Dec 26 00:57:38 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   68C    P0              55W /  70W |   6445MiB / 15360MiB |     95%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(3071927, 1) (117032738, 2)
CPU times: user 23.5 s, sys: 2.68 s, total: 26.2 s
Wall time: 28.2 s
[ ]: