GFQL vs Apache Spark GraphFrames on One Node#

This page compares GFQL with Apache Spark GraphFrames on one machine. GFQL is Graphistry’s open-source graph query language: Cypher and Python chains that run in-process on dataframes, with no database or cluster. GraphFrames is Spark’s graph library, run here on local[*], a single-node JVM using all cores. The workload is four tasks on two SNAP graphs, LiveJournal and Orkut, with Friendster as the larger-than-memory size measured last.

GFQL queries billion-edge graphs on one machine. On Orkut (117M edges), GFQL filters the graph and expands one hop in under a second. On Friendster (1.8B edges, 65.6M nodes), GFQL filters in 3111.7 ms and expands one hop in 6123.4 ms. GraphFrames did not load Friendster at all.

GFQL wins some tasks and loses others. Both sides are shown here:

PageRank: GFQL on GPU wins. It is 17.5x faster than GraphFrames on LiveJournal and 12.0x faster on Orkut.
Filter and 1-hop: GFQL on CPU wins on both graphs.
2-hop: GraphFrames wins on both graphs.
PageRank without a GPU: GraphFrames wins on both graphs.

LiveJournal task times: GFQL and GraphFrames for filter, 1-hop, 2-hop, and PageRank, with the PageRank solver time shaded inside the GFQL bar

Orkut task times: GFQL and GraphFrames for filter, 1-hop, 2-hop, and PageRank, with the PageRank solver time shaded inside the GFQL bar

Friendster task times: GFQL CPU streaming filter, 1-hop, and 2-hop; PageRank and GraphFrames not measured

LiveJournal#

Task	GFQL polars (CPU)	GFQL polars-gpu (GPU)	GraphFrames (local[*])	GFQL CPU vs GraphFrames
filter (degree >= 42)	`24.5 ms`	`25.3 ms`	`80.4 ms`	`3.3x`
1-hop (50 seeds)	`224.8 ms`	`1180.6 ms`	`1460.2 ms`	`6.5x`
2-hop (50 seeds)	`6803.5 ms`	`7611.6 ms`	`3475.8 ms`	`0.5x` (GraphFrames wins)
PageRank (full graph)	`53417.1 ms`; solver `2870.9 ms (diagnostic)`	`933.4 ms`; solver `62.8 ms (diagnostic)`	`16311.2 ms`	GPU: `17.5x`; CPU: `0.3x` (GraphFrames wins)

Orkut#

Task	GFQL polars (CPU)	GFQL polars-gpu (GPU)	GraphFrames (local[*])	GFQL CPU vs GraphFrames
filter (degree >= 162)	`57.7 ms`	`66.6 ms`	`66.1 ms`	`1.1x`
1-hop (50 seeds)	`498.5 ms`	`4277.8 ms`	`3632.2 ms`	`7.3x`
2-hop (50 seeds)	`27473.2 ms`	`34039.7 ms`	`10885.8 ms`	`0.4x` (GraphFrames wins)
PageRank (full graph)	`170556.7 ms`; solver `7848.6 ms (diagnostic)`	`2897.5 ms`; solver `155.3 ms (diagnostic)`	`34860.7 ms`	GPU: `12.0x`; CPU: `0.2x` (GraphFrames wins)

Friendster#

On the Polars CPU streaming path; the GPU path stopped at the 1-hop and no other system ran (see Friendster (1.8B edges): the ceiling).

Task	GFQL polars (CPU)	Result
filter (degree >= 148, the 90th percentile)	`3111.7 ms`	6.6M nodes
1-hop (50 seeds)	`6123.4 ms`	166.6k nodes
2-hop (50 seeds)	`164890.2 ms`	15.9M nodes
PageRank	not attempted	see Friendster (1.8B edges): the ceiling

Result sizes agree across the systems that ran each task, as recorded in the run records:

Graph	filter	1-hop	2-hop	PageRank
LiveJournal	403.6k	119.9k	1.4M	4.0M
Orkut	308.7k	435.0k	2.0M	3.1M
Friendster (GFQL only)	6.6M	166.6k	15.9M	not attempted

Which engine to use#

Whole-graph PageRank: use GFQL on GPU (engine="polars-gpu", cuGraph). The solver is a small share of the GFQL time; the rest is the conversion of the edge frame and the join of scores back onto the nodes.
Filter and 1-hop: use GFQL on CPU (engine="polars"). It is faster than GraphFrames on both graphs, and the GPU streaming executor does not help at these result sizes.
2-hop from hub seeds: GraphFrames wins on both graphs today.
PageRank without a GPU: GFQL routes the CPU path through igraph, and loses to GraphFrames on both graphs. The igraph solver itself is 2870.9 ms (diagnostic) of the 53417.1 ms LiveJournal row; the rest is the conversion into igraph and the join-back, tracked in #2032. Use the CPU path for convenience, not for speed.
Larger than one node’s memory: see Friendster (1.8B edges): the ceiling.

The tasks#

filter: keep nodes with degree >= threshold. SNAP graphs have no attributes, so both systems compute degree during load.

# GFQL
from graphistry import n
from graphistry.compute.predicates.numeric import ge
g.gfql([n(filter_dict={'degree': ge(42)})], engine="polars")  # or "polars-gpu"

# GraphFrames
gf.degrees.filter("degree >= 42").count()

1-hop and 2-hop: undirected expansion from a fixed set of 50 high-degree seed nodes.

# GFQL
from graphistry import n, e_undirected
g.gfql([n(filter_dict={'id': is_in(seeds)}), e_undirected(hops=1), n()], engine="polars")

GraphFrames has no k-hop primitive. Its bfs finds shortest paths between predicates and find matches a fixed motif. The Spark side therefore expands with one iterated undirected edge join per hop and ends in .count().

PageRank: full graph, damping 0.85. GFQL GPU calls g.compute_cugraph('pagerank') on an eager cuDF copy of the graph; GFQL CPU calls g.compute_igraph('pagerank') on an eager pandas copy. GraphFrames calls gf.pageRank(resetProbability=0.15, maxIter=20). All return the full vertex set. The shaded part of a GFQL PageRank bar is the solver alone (cuGraph or igraph) on a graph object built outside the timer; the light part is the rest of the query, which is the conversion into that graph object and the join of scores back onto the nodes.

Friendster (1.8B edges): the ceiling#

Friendster has 1.8B edges and 65.6M nodes (SNAP). The eager harness that produced the earlier version of this page could not load it on the test node (about 119 GB unified memory): a pandas edge frame plus a second pass for degrees exceeds physical RAM, a direct cuDF read exceeds the unified pool, and a 90 GB Spark driver heap swaps.

The harness binds from pl.scan_parquet and collects through GFQL’s streaming paths (GFQL_POLARS_CPU_STREAMING=1 for the Polars streaming engine, GFQL_POLARS_GPU_EXECUTOR=streaming for the cudf-polars streaming executor), with a peak-memory record at every size. On Friendster the CPU streaming run loaded the graph (scan plus degree pass in about 20 seconds, 55.0 GiB resident), answered the degree filter and the 1-hop from 50 hub seeds (table above), and peaked at 103.6 GiB resident after the 1-hop; a second run answered the 2-hop, a 15.9M-node ball, in 164890.2 ms at 67.9 GiB resident. The streaming collect keeps the load out of memory, but the traversal still materializes the edges it touches, and that is where the GPU path stops: the cudf-polars streaming executor completed the degree filter at 103.8 GiB resident, then the watchdog ended the run during the 1-hop when host free memory fell to 17 GB against its 20 GB floor, so the GPU column has no Friendster cell. Whole-graph PageRank does not fit on either path: the GPU preflight refused it (an estimated 87 GB peak against an 80 GB budget), and the CPU path was not attempted: its Orkut row peaked at 29.9 GiB resident for 117M edges, and Friendster has fifteen times the edges. That is the single-server ceiling this page measured. GraphFrames on local[*] stays at the boundary it hit above.

Method and limits#

Scope: single node, in memory. local[*] is Spark’s single-node mode. A cluster amortizes scheduling and shuffle cost across machines and changes the trade-off at larger scale. Use a Spark cluster when the data already lives there or the graph exceeds one node’s memory.
Timing: median of 5 runs after 2 warmups per cell, each system loaded once and resident across iterations. Load is not timed.
Materialization: Spark is lazy, so every task ends in .count() or .vertices.count(). GFQL materializes with len(_nodes).
Comparability: a task is comparable only when every system reports the same result size; a cell that disagrees is published as a direct time with a disclosure. Cells marked diagnostic are never quoted as GFQL’s number.
PageRank convergence: GraphFrames runs a fixed maxIter=20; cuGraph runs to its default tolerance. Times compare wall-clock to a usable ranking.
Run records: one run at a time under a host lock, after two clean checks five minutes apart; a load monitor samples the host every second and a classifier invalidates the run if a process outside the benchmark ran during it. Invalidated attempts stay in the package under stale-attempts/; one Orkut GraphFrames run is valid by reclassification after the classifier learned that Spark’s own shutdown cleanup is the benchmark’s process (RECLASSIFIED.txt in that run’s directory).
Harness: the GFQL streaming harness and every run record live in pyg-bench; the GraphFrames baseline is benchmarks/gfql/bench_graphframes.py --systems graphframes in this repository, run from a host Spark with the GraphFrames assembly jar.

Provenance#

Every figure on this page is printed from docs/source/_data/gfql_benchmarks.json, which pyg-bench publishes. The documentation build and docs/test_bench_numbers.py reject missing, stale, or unpublished values.

Measurement

Measured:

2026-09-20 and 2026-09-04

Host:

dgx-spark (NVIDIA GB10), 20 CPU

Notes:

Single node (dgx-spark, NVIDIA GB10 unified memory, 20 CPU); GraphFrames on Spark local[*] over all cores with a 90g driver, PySpark 3.5.3; GFQL binds the graph from a lazy Polars scan and its filter and hop rows run under the Polars CPU streaming collect or the cudf-polars streaming executor; PageRank rows re-bind an eager cuDF (GPU) or pandas (CPU) copy outside the timer.
GFQL filter, hop and CPU PageRank rows were measured at the head of pull request #2024 (commit e951e9a2c), the fix for issue #2023 (a per-edge Python loop that made undirected multi-hop ~30x slower at 0.59.0); it had not landed on master when measured. GFQL GPU PageRank rows and the GraphFrames rows were measured at 0.59.0 (3fb216dd); the PageRank code is the same at both commits.
Polars CPU streaming versus eager collect (receipts fix2024/lj-polars and fix2024/orkut-polars, same commit and protocol): filter and 2-hop medians agree within 9% (LiveJournal 2-hop 6.5 s streaming vs 7.0 s eager; Orkut 27.3 s vs 27.3 s) and streaming is faster on 1-hop (LiveJournal 220 ms vs 336 ms; Orkut 514 ms vs 1,031 ms); the published rows are the streaming lane.
CPU PageRank runs the igraph solver: on LiveJournal the solver alone is 2.8 s of the 52.6 s row (the GraphFrames row is 16.3 s) and on Orkut 7.8 s of 170.9 s (GraphFrames 34.9 s); the rest is the frame-to-igraph conversion and the join-back of scores (pygraphistry issue #2032).
Friendster (1.8B edges) filter and 1-hop ran on the Polars CPU streaming lane; resident memory after the 1-hop was 106,095 MiB (103.6 GiB) of the 119 GiB host.
Friendster 2-hop ran on the same lane without the host address-space cap: 164.9 s, a 15,878,312-node ball, 69,526 MiB (67.9 GiB) resident. The first attempt under a 100 GB address-space cap (fix2024/friendster-polars-hop2.failed-attempt1) ended at load with a failed 7.2 GB allocation, exit 133.
Friendster on the cudf-polars streaming executor (fix2024/friendster-polars-gpu-streaming.killed-attempt1): the degree filter completed (4.0 s, a diagnostic from a killed lane, not a cell) with the process at 106,308 MiB resident; the watchdog killed the run during the 1-hop when host free memory fell to 17 GB against the 20 GB floor, exit 137. The GPU column has no Friendster cell.
Friendster GPU PageRank was refused by preflight (fix2024/friendster-polars-gpu-pagerank.refused-attempt1: estimated 87 GB peak against the 80 GB budget). CPU PageRank was not attempted: Orkut’s CPU PageRank peaked at 30,586 MiB resident at 117M edges and Friendster has 15x the edges. GraphFrames on local[*] did not run on Friendster (a 90 GB driver heap swapped in the earlier eager attempt). Friendster rows are direct times with no comparison.
Component of graphframes.lj.pagerank.gfql_polars: the PageRank solver alone on a graph object built outside the timer; the parent time adds the conversion and the join of scores back onto the nodes.
Component of graphframes.lj.pagerank.gfql_polars_gpu: the PageRank solver alone on a graph object built outside the timer; the parent time adds the conversion and the join of scores back onto the nodes.
Component of graphframes.orkut.pagerank.gfql_polars: the PageRank solver alone on a graph object built outside the timer; the parent time adds the conversion and the join of scores back onto the nodes.
Component of graphframes.orkut.pagerank.gfql_polars_gpu: the PageRank solver alone on a graph object built outside the timer; the parent time adds the conversion and the join of scores back onto the nodes.

GFQL vs Apache Spark GraphFrames on One Node

Contents