GFQL CPU, GPU Benchmark#
This notebook examines GFQL progerty graph query performance on 1-8 hop queries using CPU + GPU modes on various real-world 100K - 100M edge graphs. The data comes from a variety of popular social networks. The single-threaded CPU mode benefits from GFQL’s novel dataframe engine, and the GPU mode further adds single-GPU acceleration. Both the chain()
and hop()
methods are examined.
The benchmark does not examine bigger-than-memory and distributed scenarios. The provided results here are from running on a free Google Colab T4 runtime, with a 2.2GHz Intel CPU (12 GB CPU RAM) and T4 Nvidia GPU (16 GB GPU RAM).
Data#
From SNAP
Network |
Nodes |
Edges |
---|---|---|
4,039 |
88,234 |
|
81,306 |
2,420,766 |
|
GPlus |
107,614 |
30,494,866 |
Orkut |
3,072,441 |
117,185,082 |
Results#
Definitions:
GTEPS: Giga (billion) edges traversed per second
T edges / $: Estimated trillion edges traversed for 1$ USD based on observed GTEPS and a 3yr AWS reservation (as of 12/2023)
Tasks:
chain()
- includes complex pre/post processing
Task: g.chain([n({'id': some_id}), e_forward(hops=some_n)])
Dataset |
Max GPU Speedup |
CPU GTEPS |
GPU GTEPS |
T CPU edges / $ (t3.l) |
T GPU edges / $ (g4dn.xl) |
---|---|---|---|---|---|
1.1X |
0.66 |
0.61 |
65.7 |
10.4 |
|
17.4X |
0.17 |
2.81 |
16.7 |
48.1 |
|
GPlus |
43.8X |
0.09 |
2.87 |
8.5 |
49.2 |
Orkut |
N/A |
N/A |
12.15 |
N/A |
208.3 |
AVG |
20.7X |
0.30 |
4.61 |
30.3 |
79.0 |
MAX |
43.8X |
0.66 |
12.15 |
65.7 |
208.3 |
hop()
- core property search primitive similar to BFS
Task: g.hop(nodes=[some_id], direction='forward', hops=some_n)
Dataset |
Max GPU Speedup |
CPU GTEPS |
GPU GTEPS |
T CPU edges / $ (t3.l) |
T GPU edges / $ (g4dn.xl) |
---|---|---|---|---|---|
3X |
0.47 |
1.47 |
47.0 |
25.2 |
|
42X |
0.50 |
10.51 |
50.2 |
180.2 |
|
GPlus |
21X |
0.26 |
4.11 |
26.2 |
70.4 |
Orkut |
N/A |
N/A |
41.50 |
N/A |
711.4 |
AVG |
22X |
0.41 |
14.4 |
41.1 |
246.8 |
MAX |
42X |
0.50 |
41.50 |
50.2 |
711.4 |
Optional: GPU setup - Google Colab#
[1]:
# Report GPU used when GPU benchmarking
# ! nvidia-smi
Tue Dec 26 00:50:30 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 54C P8 10W / 70W | 0MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
[8]:
# if in google colab
#!git clone https://github.com/rapidsai/rapidsai-csp-utils.git
#!python rapidsai-csp-utils/colab/pip-install.py
[3]:
import cudf
cudf.__version__
[3]:
'23.12.01'
1. Install & configure#
[2]:
#! pip install graphistry[igraph]
Preparing metadata (setup.py) ... done
Imports#
[3]:
import pandas as pd
import graphistry
from graphistry import (
# graph operators
n, e_undirected, e_forward, e_reverse,
# attribute predicates
is_in, ge, startswith, contains, match as match_re
)
graphistry.__version__
[3]:
'0.32.0+12.g72e778c'
[6]:
import cudf
[7]:
#work around google colab shell encoding bugs
import locale
locale.getpreferredencoding = lambda: "UTF-8"
2. Perf benchmarks#
Facebook: 88K edges#
[10]:
df = pd.read_csv('https://raw.githubusercontent.com/graphistry/pygraphistry/master/demos/data/facebook_combined.txt', sep=' ', names=['s', 'd'])
print(df.shape)
df.head(5)
(88234, 2)
[10]:
s | d | |
---|---|---|
0 | 0 | 1 |
1 | 0 | 2 |
2 | 0 | 3 |
3 | 0 | 4 |
4 | 0 | 5 |
[11]:
fg = graphistry.edges(df, 's', 'd').materialize_nodes()
print(fg._nodes.shape, fg._edges.shape)
fg._nodes.head(5)
(4039, 1) (88234, 2)
[11]:
id | |
---|---|
0 | 0 |
1 | 1 |
2 | 2 |
3 | 3 |
4 | 4 |
[12]:
%%time
for i in range(100):
fg2 = fg.chain([n({'id': 0}), e_forward(hops=2)])
CPU times: user 13.6 s, sys: 2.08 s, total: 15.7 s
Wall time: 18 s
[ ]:
%%time
fg_gdf = fg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
for i in range(100):
fg2 = fg_gdf.chain([n({'id': 0}), e_forward(hops=2)])
print(fg._nodes.shape, fg._edges.shape)
print(fg2._nodes.shape, fg2._edges.shape)
del fg_gdf
del fg2
(4039, 1) (88234, 2)
(1519, 1) (4060, 2)
CPU times: user 11.8 s, sys: 28.1 ms, total: 11.8 s
Wall time: 11.9 s
[ ]:
%%time
for i in range(50):
fg2 = fg.chain([n({'id': 0}), e_forward(hops=5)])
print(fg._nodes.shape, fg._edges.shape)
print(fg2._nodes.shape, fg2._edges.shape)
(4039, 1) (88234, 2)
(3829, 1) (86074, 2)
CPU times: user 15.4 s, sys: 808 ms, total: 16.2 s
Wall time: 16.2 s
[ ]:
%%time
fg_gdf = fg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
for i in range(50):
fg2 = fg_gdf.chain([n({'id': 0}), e_forward(hops=5)])
print(fg._nodes.shape, fg._edges.shape)
print(fg2._nodes.shape, fg2._edges.shape)
del fg_gdf
del fg2
(4039, 1) (88234, 2)
(3829, 1) (86074, 2)
CPU times: user 9.82 s, sys: 133 ms, total: 9.95 s
Wall time: 10.1 s
[ ]:
%%time
for i in range(100):
fg2 = fg.chain([e_forward(source_node_match={'id': 0}, hops=5)])
CPU times: user 11.8 s, sys: 377 ms, total: 12.1 s
Wall time: 13.1 s
[ ]:
%%time
fg_gdf = fg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
for i in range(100):
fg2 = fg_gdf.chain([e_forward(source_node_match={'id': 0}, hops=5)])
print(fg._nodes.shape, fg._edges.shape)
print(fg2._nodes.shape, fg2._edges.shape)
del fg_gdf
del fg2
(4039, 1) (88234, 2)
(348, 1) (347, 2)
CPU times: user 14.1 s, sys: 48.5 ms, total: 14.2 s
Wall time: 14.2 s
[17]:
%%time
start_nodes = pd.DataFrame({fg._node: [0]})
for i in range(100):
fg2 = fg.hop(
nodes=start_nodes,
direction='forward',
hops=2)
print(fg2._nodes.shape, fg2._edges.shape)
(1519, 1) (4060, 2)
CPU times: user 4.5 s, sys: 1.35 s, total: 5.85 s
Wall time: 6.09 s
[18]:
%%time
start_nodes = cudf.DataFrame({fg._node: [0]})
fg_gdf = fg.nodes(cudf.from_pandas(fg._nodes)).edges(cudf.from_pandas(fg._edges))
for i in range(100):
fg2 = fg_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=2)
print(fg2._nodes.shape, fg2._edges.shape)
del start_nodes
del fg_gdf
del fg2
(1519, 1) (4060, 2)
CPU times: user 2.58 s, sys: 6.75 ms, total: 2.59 s
Wall time: 2.58 s
[19]:
%%time
start_nodes = pd.DataFrame({fg._node: [0]})
for i in range(100):
fg2 = fg.hop(
nodes=start_nodes,
direction='forward',
hops=5)
print(fg2._nodes.shape, fg2._edges.shape)
(3829, 1) (86074, 2)
CPU times: user 13.2 s, sys: 2 s, total: 15.2 s
Wall time: 18.3 s
[20]:
%%time
start_nodes = cudf.DataFrame({fg._node: [0]})
fg_gdf = fg.nodes(cudf.from_pandas(fg._nodes)).edges(cudf.from_pandas(fg._edges))
for i in range(100):
fg2 = fg_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=5)
print(fg2._nodes.shape, fg2._edges.shape)
del start_nodes
del fg_gdf
del fg2
(3829, 1) (86074, 2)
CPU times: user 5.72 s, sys: 159 ms, total: 5.88 s
Wall time: 5.86 s
Twitter#
edges: 2420766
nodes: 81306
[21]:
#! wget 'https://snap.stanford.edu/data/twitter_combined.txt.gz'
--2023-12-25 21:58:27-- https://snap.stanford.edu/data/twitter_combined.txt.gz
Resolving snap.stanford.edu (snap.stanford.edu)... 171.64.75.80
Connecting to snap.stanford.edu (snap.stanford.edu)|171.64.75.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10621918 (10M) [application/x-gzip]
Saving to: ‘twitter_combined.txt.gz’
twitter_combined.tx 100%[===================>] 10.13M 3.00MB/s in 4.0s
2023-12-25 21:58:32 (2.52 MB/s) - ‘twitter_combined.txt.gz’ saved [10621918/10621918]
[22]:
#! gunzip twitter_combined.txt.gz
[24]:
#! head -n 5 twitter_combined.txt
214328887 34428380
17116707 28465635
380580781 18996905
221036078 153460275
107830991 17868918
[25]:
%%time
te_df = pd.read_csv('twitter_combined.txt', sep=' ', names=['s', 'd'])
te_df.shape
CPU times: user 474 ms, sys: 61.9 ms, total: 536 ms
Wall time: 534 ms
[25]:
(2420766, 2)
[26]:
import graphistry
[27]:
%%time
g = graphistry.edges(te_df, 's', 'd').materialize_nodes()
g._nodes.shape
CPU times: user 86.4 ms, sys: 106 ms, total: 193 ms
Wall time: 191 ms
[27]:
(81306, 1)
[29]:
%%time
for i in range(10):
g2 = g.chain([n({'id': 17116707}), e_forward(hops=1)])
g2._nodes.shape, g2._edges.shape
CPU times: user 11.8 s, sys: 8.4 s, total: 20.2 s
Wall time: 23 s
[29]:
((140, 1), (615, 2))
[30]:
%%time
g_gdf = g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
for i in range(10):
out = g_gdf.chain([n({'id': 17116707}), e_forward(hops=1)])._nodes
print(out.shape)
del g_gdf
del out
(140, 1)
CPU times: user 1.33 s, sys: 46.6 ms, total: 1.38 s
Wall time: 1.63 s
[31]:
%%time
for i in range(10):
out = g.chain([n({'id': 17116707}), e_forward(hops=2)])
print(out._nodes.shape, out._edges.shape)
(2345, 1) (68536, 2)
CPU times: user 13.3 s, sys: 8.05 s, total: 21.4 s
Wall time: 21.6 s
[36]:
%%time
g_gdf = g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
for i in range(10):
out = g_gdf.chain([n({'id': 17116707}), e_forward(hops=2)])._nodes
print(out.shape)
del g_gdf
del out
(2345, 1)
CPU times: user 1.67 s, sys: 55.8 ms, total: 1.72 s
Wall time: 1.75 s
[37]:
%%time
for i in range(10):
out = g.chain([n({'id': 17116707}), e_forward(hops=8)])
print(out._nodes.shape, out._edges.shape)
(81304, 1) (2417796, 2)
CPU times: user 1min 56s, sys: 17.1 s, total: 2min 13s
Wall time: 2min 22s
[38]:
%%time
g_gdf = g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
for i in range(10):
out = g_gdf.chain([n({'id': 17116707}), e_forward(hops=8)])._nodes
print(out.shape)
del g_gdf
del out
(81304, 1)
CPU times: user 5.3 s, sys: 1.48 s, total: 6.78 s
Wall time: 7.89 s
[39]:
%%time
start_nodes = pd.DataFrame({g._node: [17116707]})
for i in range(10):
g2 = g.hop(
nodes=start_nodes,
direction='forward',
hops=1)
print(g2._nodes.shape, g2._edges.shape)
(0, 1) (0, 2)
CPU times: user 2.58 s, sys: 1.59 s, total: 4.17 s
Wall time: 6.02 s
[44]:
%%time
start_nodes = cudf.DataFrame({g._node: [17116707]})
g_gdf = g.nodes(cudf.from_pandas(g._nodes)).edges(cudf.from_pandas(g._edges))
for i in range(10):
g2 = g_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=5)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del g_gdf
del g2
(61827, 1) (1473599, 2)
CPU times: user 822 ms, sys: 179 ms, total: 1 s
Wall time: 997 ms
[40]:
%%time
start_nodes = pd.DataFrame({g._node: [17116707]})
for i in range(10):
g2 = g.hop(
nodes=start_nodes,
direction='forward',
hops=2)
print(g2._nodes.shape, g2._edges.shape)
(2345, 1) (68536, 2)
CPU times: user 8.93 s, sys: 5.92 s, total: 14.9 s
Wall time: 15.8 s
[41]:
%%time
start_nodes = cudf.DataFrame({g._node: [17116707]})
g_gdf = g.nodes(cudf.from_pandas(g._nodes)).edges(cudf.from_pandas(g._edges))
for i in range(10):
g2 = g_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=2)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del g_gdf
del g2
(2345, 1) (68536, 2)
CPU times: user 374 ms, sys: 6.92 ms, total: 381 ms
Wall time: 379 ms
[42]:
%%time
start_nodes = pd.DataFrame({g._node: [17116707]})
for i in range(10):
g2 = g.hop(
nodes=start_nodes,
direction='forward',
hops=8)
print(g2._nodes.shape, g2._edges.shape)
(81304, 1) (2417796, 2)
CPU times: user 38.8 s, sys: 8.7 s, total: 47.5 s
Wall time: 48.2 s
[43]:
%%time
start_nodes = cudf.DataFrame({g._node: [17116707]})
g_gdf = g.nodes(cudf.from_pandas(g._nodes)).edges(cudf.from_pandas(g._edges))
for i in range(10):
g2 = g_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=8)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del g_gdf
del g2
(81304, 1) (2417796, 2)
CPU times: user 1.8 s, sys: 506 ms, total: 2.3 s
Wall time: 2.3 s
GPlus#
edges: 30494866
nodes: 107614
[4]:
#! wget https://snap.stanford.edu/data/gplus_combined.txt.gz
--2023-12-26 18:36:29-- https://snap.stanford.edu/data/gplus_combined.txt.gz
Resolving snap.stanford.edu (snap.stanford.edu)... 171.64.75.80
Connecting to snap.stanford.edu (snap.stanford.edu)|171.64.75.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 398930514 (380M) [application/x-gzip]
Saving to: ‘gplus_combined.txt.gz’
gplus_combined.txt. 100%[===================>] 380.45M 34.7MB/s in 9.6s
2023-12-26 18:36:39 (39.7 MB/s) - ‘gplus_combined.txt.gz’ saved [398930514/398930514]
[5]:
#! gunzip gplus_combined.txt.gz
[6]:
%%time
ge_df = pd.read_csv('gplus_combined.txt', sep=' ', names=['s', 'd'])
print(ge_df.shape)
ge_df.head(5)
(30494866, 2)
CPU times: user 16 s, sys: 1.45 s, total: 17.5 s
Wall time: 22.5 s
[6]:
s | d | |
---|---|---|
0 | 116374117927631468606 | 101765416973555767821 |
1 | 112188647432305746617 | 107727150903234299458 |
2 | 116719211656774388392 | 100432456209427807893 |
3 | 117421021456205115327 | 101096322838605097368 |
4 | 116407635616074189669 | 113556266482860931616 |
[7]:
%%time
gg = graphistry.edges(ge_df, 's', 'd').materialize_nodes()
gg = graphistry.edges(ge_df, 's', 'd').nodes(gg._nodes, 'id')
print(gg._edges.shape, gg._nodes.shape)
gg._nodes.head(5)
(30494866, 2) (107614, 1)
CPU times: user 4.49 s, sys: 1.25 s, total: 5.74 s
Wall time: 5.97 s
[7]:
id | |
---|---|
0 | 116374117927631468606 |
1 | 112188647432305746617 |
2 | 116719211656774388392 |
3 | 117421021456205115327 |
4 | 116407635616074189669 |
[49]:
%%time
gg.chain([ n({'id': '116374117927631468606'})])._nodes
CPU times: user 534 ms, sys: 598 ms, total: 1.13 s
Wall time: 1.65 s
[49]:
id | |
---|---|
0 | 116374117927631468606 |
[ ]:
%%time
out = gg.chain([ n({'id': '116374117927631468606'}), e_forward(hops=1)])._nodes
out.shape
CPU times: user 27.5 s, sys: 11.1 s, total: 38.5 s
Wall time: 39.5 s
(1473, 1)
[ ]:
%%time
gg_gdf = gg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
out = gg_gdf.chain([ n({'id': '116374117927631468606'}), e_forward(hops=1)])
print(out._nodes.shape, out._edges.shape)
del gg_gdf
del out
(1473, 1) (13375, 2)
CPU times: user 4.57 s, sys: 2.11 s, total: 6.68 s
Wall time: 7.63 s
[ ]:
%%time
out = gg.chain([ n({'id': '116374117927631468606'}), e_forward(hops=2)])._nodes
out.shape
CPU times: user 45.8 s, sys: 17 s, total: 1min 2s
Wall time: 1min 5s
(44073, 1)
[ ]:
%%time
gg_gdf = gg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
out = gg_gdf.chain([ n({'id': '116374117927631468606'}), e_forward(hops=2)])
print(out._nodes.shape, out._edges.shape)
del gg_gdf
del out
(44073, 1) (2069325, 2)
CPU times: user 4.97 s, sys: 2.36 s, total: 7.34 s
Wall time: 10.6 s
[ ]:
%%time
out = gg.chain([ n({'id': '116374117927631468606'}), e_forward(hops=3)])._nodes
out.shape
CPU times: user 3min 45s, sys: 1min 5s, total: 4min 50s
Wall time: 4min 52s
(102414, 1)
[ ]:
%%time
gg_gdf = gg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
out = gg_gdf.chain([ n({'id': '116374117927631468606'}), e_forward(hops=3)])
print(out._nodes.shape, out._edges.shape)
del gg_gdf
del out
(102414, 1) (24851333, 2)
CPU times: user 6.95 s, sys: 2.63 s, total: 9.57 s
Wall time: 9.84 s
[8]:
%%time
out = gg.chain([ n({'id': '116374117927631468606'}), e_forward(hops=4)])
print(out._nodes.shape, out._edges.shape)
(105479, 1) (30450354, 2)
CPU times: user 4min 36s, sys: 1min 25s, total: 6min 2s
Wall time: 6min 4s
[ ]:
%%time
gg_gdf = gg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
out = gg_gdf.chain([ n({'id': '116374117927631468606'}), e_forward(hops=4)])
print(out._nodes.shape, out._edges.shape)
del gg_gdf
del out
(105479, 1) (30450354, 2)
CPU times: user 7.44 s, sys: 2.45 s, total: 9.88 s
Wall time: 9.9 s
[9]:
%%time
out = gg.chain([ n({'id': '116374117927631468606'}), e_forward(hops=5)])
print(out._nodes.shape, out._edges.shape)
(105604, 1) (30468335, 2)
CPU times: user 5min 36s, sys: 1min 39s, total: 7min 16s
Wall time: 7min 15s
[ ]:
%%time
gg_gdf = gg.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
out = gg_gdf.chain([ n({'id': '116374117927631468606'}), e_forward(hops=5)])
print(out._nodes.shape, out._edges.shape)
del gg_gdf
del out
(105604, 1) (30468335, 2)
CPU times: user 8.82 s, sys: 2.71 s, total: 11.5 s
Wall time: 11.9 s
[50]:
%%time
start_nodes = pd.DataFrame({gg._node: ['116374117927631468606']})
for i in range(1):
g2 = gg.hop(
nodes=start_nodes,
direction='forward',
hops=1)
print(g2._nodes.shape, g2._edges.shape)
(1473, 1) (13375, 2)
CPU times: user 19.9 s, sys: 9.36 s, total: 29.2 s
Wall time: 41.8 s
[52]:
%%time
start_nodes = cudf.DataFrame({gg._node: ['116374117927631468606']})
gg_gdf = gg.nodes(cudf.from_pandas(gg._nodes)).edges(cudf.from_pandas(gg._edges))
for i in range(1):
g2 = gg_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=1)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del gg_gdf
del g2
(1473, 1) (13375, 2)
CPU times: user 3.71 s, sys: 2.09 s, total: 5.8 s
Wall time: 6.05 s
[53]:
%%time
start_nodes = pd.DataFrame({gg._node: ['116374117927631468606']})
for i in range(1):
g2 = gg.hop(
nodes=start_nodes,
direction='forward',
hops=2)
print(g2._nodes.shape, g2._edges.shape)
(44073, 1) (2069325, 2)
CPU times: user 27.8 s, sys: 13.2 s, total: 41 s
Wall time: 43.9 s
[54]:
%%time
start_nodes = cudf.DataFrame({gg._node: ['116374117927631468606']})
gg_gdf = gg.nodes(cudf.from_pandas(gg._nodes)).edges(cudf.from_pandas(gg._edges))
for i in range(1):
g2 = gg_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=2)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del gg_gdf
del g2
(44073, 1) (2069325, 2)
CPU times: user 4.26 s, sys: 2.37 s, total: 6.63 s
Wall time: 7.91 s
[55]:
%%time
start_nodes = pd.DataFrame({gg._node: ['116374117927631468606']})
for i in range(1):
g2 = gg.hop(
nodes=start_nodes,
direction='forward',
hops=3)
print(g2._nodes.shape, g2._edges.shape)
(102414, 1) (24851333, 2)
CPU times: user 1min 3s, sys: 22.7 s, total: 1min 26s
Wall time: 1min 35s
[56]:
%%time
start_nodes = cudf.DataFrame({gg._node: ['116374117927631468606']})
gg_gdf = gg.nodes(cudf.from_pandas(gg._nodes)).edges(cudf.from_pandas(gg._edges))
for i in range(1):
g2 = gg_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=3)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del gg_gdf
del g2
(102414, 1) (24851333, 2)
CPU times: user 3.96 s, sys: 2.11 s, total: 6.07 s
Wall time: 6.05 s
[57]:
%%time
start_nodes = pd.DataFrame({gg._node: ['116374117927631468606']})
for i in range(1):
g2 = gg.hop(
nodes=start_nodes,
direction='forward',
hops=4)
print(g2._nodes.shape, g2._edges.shape)
(105479, 1) (30450354, 2)
CPU times: user 1min 34s, sys: 30.6 s, total: 2min 5s
Wall time: 2min 5s
[58]:
%%time
start_nodes = cudf.DataFrame({gg._node: ['116374117927631468606']})
gg_gdf = gg.nodes(cudf.from_pandas(gg._nodes)).edges(cudf.from_pandas(gg._edges))
for i in range(1):
g2 = gg_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=4)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del gg_gdf
del g2
(105479, 1) (30450354, 2)
CPU times: user 5.25 s, sys: 2.41 s, total: 7.67 s
Wall time: 7.69 s
[59]:
%%time
start_nodes = pd.DataFrame({gg._node: ['116374117927631468606']})
for i in range(1):
g2 = gg.hop(
nodes=start_nodes,
direction='forward',
hops=5)
print(g2._nodes.shape, g2._edges.shape)
(105604, 1) (30468335, 2)
CPU times: user 2min 16s, sys: 39.1 s, total: 2min 55s
Wall time: 2min 58s
[60]:
%%time
start_nodes = cudf.DataFrame({gg._node: ['116374117927631468606']})
gg_gdf = gg.nodes(cudf.from_pandas(gg._nodes)).edges(cudf.from_pandas(gg._edges))
for i in range(1):
g2 = gg_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=5)
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del gg_gdf
del g2
(105604, 1) (30468335, 2)
CPU times: user 5.79 s, sys: 2.51 s, total: 8.3 s
Wall time: 8.29 s
Orkut#
117M edges
3M nodes
[8]:
#! wget https://snap.stanford.edu/data/bigdata/communities/com-orkut.ungraph.txt.gz
--2023-12-26 00:55:52-- https://snap.stanford.edu/data/bigdata/communities/com-orkut.ungraph.txt.gz
Resolving snap.stanford.edu (snap.stanford.edu)... 171.64.75.80
Connecting to snap.stanford.edu (snap.stanford.edu)|171.64.75.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 447251958 (427M) [application/x-gzip]
Saving to: ‘com-orkut.ungraph.txt.gz’
com-orkut.ungraph.t 100%[===================>] 426.53M 45.1MB/s in 9.7s
2023-12-26 00:56:02 (44.0 MB/s) - ‘com-orkut.ungraph.txt.gz’ saved [447251958/447251958]
[9]:
#! gunzip com-orkut.ungraph.txt.gz
[10]:
#! head -n 7 com-orkut.ungraph.txt
# Undirected graph: ../../data/output/orkut.txt
# Orkut
# Nodes: 3072441 Edges: 117185083
# FromNodeId ToNodeId
1 2
1 3
1 4
[11]:
import pandas as pd
import graphistry
from graphistry import (
# graph operators
n, e_undirected, e_forward, e_reverse,
# attribute predicates
is_in, ge, startswith, contains, match as match_re
)
import cudf
#work around google colab shell encoding bugs
import locale
locale.getpreferredencoding = lambda: "UTF-8"
cudf.__version__, graphistry.__version__
[11]:
('23.12.01', '0.32.0+12.g72e778c')
[12]:
#! nvidia-smi
Tue Dec 26 00:56:27 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 47C P0 27W / 70W | 103MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
[13]:
%%time
co_df = cudf.read_csv('com-orkut.ungraph.txt', sep='\t', names=['s', 'd'], skiprows=5).to_pandas()
print(co_df.shape)
print(co_df.head(5))
print(co_df.dtypes)
#del co_df
(117185082, 2)
s d
0 1 3
1 1 4
2 1 5
3 1 6
4 1 7
s int64
d int64
dtype: object
CPU times: user 2.56 s, sys: 4.2 s, total: 6.76 s
Wall time: 6.76 s
[14]:
%%time
co_g = graphistry.edges(cudf.DataFrame(co_df), 's', 'd').materialize_nodes(engine='cudf')
co_g = co_g.nodes(lambda g: g._nodes.to_pandas()).edges(lambda g: g._edges.to_pandas())
print(co_g._nodes.shape, co_g._edges.shape)
co_g._nodes.head(5)
(3072441, 1) (117185082, 2)
CPU times: user 1.96 s, sys: 2.95 s, total: 4.91 s
Wall time: 4.92 s
[14]:
id | |
---|---|
0 | 1 |
1 | 2 |
2 | 3 |
3 | 4 |
4 | 5 |
[15]:
#! nvidia-smi
Tue Dec 26 00:56:39 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 49C P0 27W / 70W | 2819MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
[ ]:
%%time
# crashes
if False:
out = co_g.chain([ n({'id': 1}), e_forward(hops=1)])._nodes
print(out.shape)
del out
CPU times: user 4 µs, sys: 1 µs, total: 5 µs
Wall time: 47.7 µs
[ ]:
%%time
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
out = co_gdf.chain([ n({'id': 1}), e_forward(hops=1)])
#! nvidia-smi
print(out._nodes.shape, out._edges.shape)
del co_gdf
del out
Mon Dec 25 06:23:46 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 63C P0 30W / 70W | 1925MiB / 15360MiB | 35% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Mon Dec 25 06:23:49 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 66C P0 72W / 70W | 2845MiB / 15360MiB | 84% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(12, 1) (11, 2)
CPU times: user 4.42 s, sys: 131 ms, total: 4.55 s
Wall time: 4.42 s
[ ]:
%%time
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
out = co_gdf.chain([ n({'id': 1}), e_forward(hops=2)])
#! nvidia-smi
print(out._nodes.shape, out._edges.shape)
del co_gdf
del out
Mon Dec 25 06:24:52 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 61C P0 29W / 70W | 1925MiB / 15360MiB | 22% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Mon Dec 25 06:24:58 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 64C P0 71W / 70W | 2845MiB / 15360MiB | 57% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(391, 1) (461, 2)
CPU times: user 5.34 s, sys: 132 ms, total: 5.47 s
Wall time: 6.13 s
[ ]:
%%time
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
out = co_gdf.chain([ n({'id': 1}), e_forward(hops=3)])
#! nvidia-smi
print(out._nodes.shape, out._edges.shape)
del co_gdf
del out
Mon Dec 25 06:25:25 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 61C P0 29W / 70W | 1925MiB / 15360MiB | 31% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Mon Dec 25 06:25:31 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 65C P0 71W / 70W | 2849MiB / 15360MiB | 58% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(21767, 1) (28480, 2)
CPU times: user 6.25 s, sys: 100 ms, total: 6.35 s
Wall time: 6.37 s
[ ]:
%%time
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
out = co_gdf.chain([ n({'id': 1}), e_forward(hops=4)])
#! nvidia-smi
print(out._nodes.shape, out._edges.shape)
del co_gdf
del out
Mon Dec 25 06:26:04 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 61C P0 29W / 70W | 1927MiB / 15360MiB | 36% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Mon Dec 25 06:26:13 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 65C P0 71W / 70W | 2931MiB / 15360MiB | 90% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(718640, 1) (2210961, 2)
CPU times: user 9.01 s, sys: 1.03 s, total: 10 s
Wall time: 9.84 s
[ ]:
%%time
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
out = co_gdf.chain([ n({'id': 1}), e_forward(hops=5)])
#! nvidia-smi
print(out._nodes.shape, out._edges.shape)
del co_gdf
del out
Mon Dec 25 06:27:18 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 60C P0 29W / 70W | 1927MiB / 15360MiB | 28% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Mon Dec 25 06:27:57 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 72C P0 43W / 70W | 4351MiB / 15360MiB | 100% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(3041556, 1) (47622917, 2)
CPU times: user 34.9 s, sys: 4.76 s, total: 39.6 s
Wall time: 39.2 s
[ ]:
%%time
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
out = co_gdf.chain([ n({'id': 1}), e_forward(hops=6)])._nodes
print(out.shape)
del co_gdf
del out
[ ]:
#!lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) CPU @ 2.20GHz
CPU family: 6
Model: 79
Thread(s) per core: 2
Core(s) per socket: 1
Socket(s): 1
Stepping: 0
BogoMIPS: 4399.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clf
lush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_
good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fm
a cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hyp
ervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsb
ase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsa
veopt arat md_clear arch_capabilities
Virtualization features:
Hypervisor vendor: KVM
Virtualization type: full
Caches (sum of all):
L1d: 32 KiB (1 instance)
L1i: 32 KiB (1 instance)
L2: 256 KiB (1 instance)
L3: 55 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0,1
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Mitigation; PTE Inversion
Mds: Vulnerable; SMT Host state unknown
Meltdown: Vulnerable
Mmio stale data: Vulnerable
Retbleed: Vulnerable
Spec rstack overflow: Not affected
Spec store bypass: Vulnerable
Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swap
gs barriers
Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
Srbds: Not affected
Tsx async abort: Vulnerable
[ ]:
#!free -h
total used free shared buff/cache available
Mem: 12Gi 717Mi 8.0Gi 1.0Mi 3.9Gi 11Gi
Swap: 0B 0B 0B
[ ]:
%%time
start_nodes = pd.DataFrame({'id': [1]})
#! nvidia-smi
for i in range(1):
g2 = co_g.hop(
nodes=start_nodes,
direction='forward',
hops=1)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
#del start_nodes
#del co_gdf
#del g2
Tue Dec 26 01:01:43 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 64C P0 30W / 70W | 2821MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
[16]:
%%time
start_nodes = cudf.DataFrame({'id': [1]})
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
g2 = co_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=1)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del co_gdf
del g2
Tue Dec 26 00:56:45 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 49C P0 28W / 70W | 1923MiB / 15360MiB | 37% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Tue Dec 26 00:56:47 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 52C P0 70W / 70W | 2819MiB / 15360MiB | 79% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(12, 1) (11, 2)
CPU times: user 1.6 s, sys: 37.3 ms, total: 1.64 s
Wall time: 1.84 s
[17]:
%%time
start_nodes = cudf.DataFrame({'id': [1]})
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
g2 = co_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=2)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del co_gdf
del g2
Tue Dec 26 00:56:47 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 51C P0 35W / 70W | 1923MiB / 15360MiB | 59% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Tue Dec 26 00:56:49 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 53C P0 59W / 70W | 2821MiB / 15360MiB | 86% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(391, 1) (461, 2)
CPU times: user 2.32 s, sys: 58.5 ms, total: 2.38 s
Wall time: 2.51 s
[18]:
%%time
start_nodes = cudf.DataFrame({'id': [1]})
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
g2 = co_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=3)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del co_gdf
del g2
Tue Dec 26 00:56:50 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 52C P0 36W / 70W | 1925MiB / 15360MiB | 55% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Tue Dec 26 00:56:53 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 54C P0 75W / 70W | 2825MiB / 15360MiB | 74% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(21767, 1) (28480, 2)
CPU times: user 3.04 s, sys: 63.6 ms, total: 3.1 s
Wall time: 3.25 s
[19]:
%%time
start_nodes = cudf.DataFrame({'id': [1]})
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
g2 = co_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=4)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del co_gdf
del g2
Tue Dec 26 00:56:53 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 54C P0 36W / 70W | 1927MiB / 15360MiB | 54% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Tue Dec 26 00:56:58 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 56C P0 38W / 70W | 2907MiB / 15360MiB | 89% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(718640, 1) (2210961, 2)
CPU times: user 4.58 s, sys: 309 ms, total: 4.89 s
Wall time: 5.02 s
[20]:
%%time
start_nodes = cudf.DataFrame({'id': [1]})
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
g2 = co_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=5)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del co_gdf
del g2
Tue Dec 26 00:56:58 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 55C P0 37W / 70W | 1925MiB / 15360MiB | 59% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Tue Dec 26 00:57:10 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 60C P0 48W / 70W | 4325MiB / 15360MiB | 99% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(3041556, 1) (47622917, 2)
CPU times: user 10.8 s, sys: 1.29 s, total: 12.1 s
Wall time: 12 s
[21]:
%%time
start_nodes = cudf.DataFrame({'id': [1]})
co_gdf = co_g.nodes(lambda g: cudf.DataFrame(g._nodes)).edges(lambda g: cudf.DataFrame(g._edges))
#! nvidia-smi
for i in range(10):
g2 = co_gdf.hop(
nodes=start_nodes,
direction='forward',
hops=6)
#! nvidia-smi
print(g2._nodes.shape, g2._edges.shape)
del start_nodes
del co_gdf
del g2
Tue Dec 26 00:57:10 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 59C P0 38W / 70W | 1925MiB / 15360MiB | 44% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Tue Dec 26 00:57:38 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 68C P0 55W / 70W | 6445MiB / 15360MiB | 95% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
(3071927, 1) (117032738, 2)
CPU times: user 23.5 s, sys: 2.68 s, total: 26.2 s
Wall time: 28.2 s
[ ]: