Choosing a GFQL Engine: pandas, Polars, cuDF, Polars-GPU#

GFQL runs the same query on four interchangeable execution engines. You pick the engine with one keyword — engine=, accepted uniformly by g.gfql() and g.hop() — and GFQL returns identical results on every one (differential parity is a release gate). Unsupported engine/query combinations are declined during validation, compilation, or planning before query execution whenever they can be known statically, so the safety contract is same answer or pre-execution error, not silent fallback. Pick the engine that fits your hardware and workload; nothing else changes.

Note

New to GFQL? This page assumes you already have a graph g and a query. If not, build one first — see 10 Minutes to GFQL (10 Minutes to GFQL).

The one-line speedup#

On real graphs, switching the default pandas engine to the columnar Polars engine is a one-keyword change — no GPU, same results:

import graphistry
g = graphistry.edges(df, 'src', 'dst')   # df: your edges dataframe (pandas / Polars / cuDF)
query = "MATCH (a)-[e]->(b) RETURN b"     # any GFQL / Cypher query

g.gfql(query)                    # engine='pandas' (default)
g.gfql(query, engine='polars')   # often much faster on query-heavy workloads, identical results

On the prrao87/graph-benchmark q1–q9 Cypher suite, Polars beats pandas on all nine queries at both graph sizes measured, and by more on the larger one. The per-query numbers are on the GFQL Performance: Vectorization and GPU Acceleration page.

Your existing pandas, Polars, or cuDF graph works as-is: the input frames are accepted and coerced once; the only change is the keyword. The catch: a few exotic Cypher features still require engine='pandas' (they decline during validation, compilation, or planning rather than silently bridge), and the GPU engines only pay off on larger work. On CPU, Polars wins the common graph-query shapes (traversal, WHERE/ORDER, aggregation) — see When not to use Polars below.

Note

Already a Polars user? The default now keeps you native. With the default engine='auto', a graph whose bound frames are all polars.DataFrame runs on the Polars engine and returns Polars frames. If the query uses a shape the Polars engine declines, GFQL falls back to pandas for that call — so auto is native when it can be, and pandas otherwise. Pass engine='polars' explicitly when you want a decline to raise instead of silently falling back:

import polars as pl, graphistry
g = graphistry.edges(edges_pl, 'src', 'dst').nodes(nodes_pl, 'id')  # polars frames
out = g.gfql(query)                    # auto -> native Polars (out._nodes is polars)
out = g.gfql(query, engine='polars')   # same, but a declined shape raises

Note

Result frames match the engine. With engine='polars' or 'polars-gpu' the output is Polars — result._nodes and result._edges are polars.DataFrame (and cudf.DataFrame for engine='cudf'). If downstream code is pandas-specific (.iloc, .loc, groupby().apply()), call result._nodes.to_pandas() to convert back.

The four engines#

Engine	Hardware	Frame type	Opt-in?	In one line
`pandas`	CPU	`pandas`	default	Universal default; best on small/interactive graphs.
`polars`	CPU	`polars`	explicit	Columnar + fused lazy plan; the CPU speed win, no GPU needed.
`cudf`	NVIDIA GPU	`cudf`	explicit	RAPIDS GPU, eager op-by-op; great for one very large materialization.
`polars-gpu`	NVIDIA GPU	`polars`	explicit	The Polars fused plan executed on GPU (cudf_polars); fastest on heavy multi-hop.

engine='auto' follows the input frames: Polars frames run on polars, cuDF frames on cudf, everything else on pandas. Two AUTO fast paths go further — all-Polars frames are tried on polars, and all-cuDF frames are tried on polars-gpu when a GPU collect probes usable — each falling back to pandas / cudf respectively if the query uses a shape that engine declines. Passing the engine explicitly turns those declines into errors instead of a fallback (see What auto does below).

How the engines compare#

Each engine has a shape it is built for:

Polars-CPU is the everyday win. It beats pandas on all nine queries of the q1–q9 Cypher suite at both sizes measured (GFQL Performance: Vectorization and GPU Acceleration), with no GPU, because it builds one fused lazy plan and collects once instead of materializing an intermediate per operation.
Polars-GPU runs that same fused plan on the GPU. It pays off once a step carries enough work to amortize a kernel launch: large frontiers, dense joins, full-graph aggregation.
cuDF executes eagerly, op by op. That suits one very large materialization where a single join dominates the query and the in-memory Polars-GPU executor comes under memory pressure.
pandas carries no plan overhead, so it stays the right default for trivially small operations and the widest-compatibility path.
Seeded / selective lookups are an indexing problem, not an engine race: the opt-in resident index turns the O(E) scan into an O(degree) gather on every engine, so the cost tracks the seeds rather than the graph — see [F5] below and Seeded Traversal Indexes (CSR Adjacency).

Coming from another graph tool#

GFQL is dataframe-native: pip install, then query the pandas, Polars, or cuDF frame you already have, in your own process. There is no server to stand up, no ETL to load, no projection step, no cluster to size. The query, the analytic, and the scoring stay in one pipeline over one set of frames.

The table says what you get when the query runs there instead of somewhere else. The measured comparison, with its lane and its provenance, is on the GFQL Performance: Vectorization and GPU Acceleration page.

Coming from	Written in Cypher today?	What GFQL gives you
Neo4j + GDS	Yes — GFQL accepts the same `MATCH ... RETURN` shapes.	Filter → PageRank → filter runs as one in-process call: no graph projection, no write-back, no round trip. The pipeline and its reproducer are in GFQL Cypher Benchmark: CPU/GPU DataFrames vs Neo4j.
Kuzu	Yes.	The measured q1–q9 board is against embedded Kuzu; see The q1–q9 board: GFQL vs an embedded graph database for the lane, the per-query numbers, and the losses. The GFQL side queries a frame that is already in memory — nothing to load, nothing to index first.
LadybugDB	Yes.	The same dataframe-native path: in-process, GPU-capable, no separate store. Polars streaming (`GFQL_POLARS_CPU_STREAMING=1`) and the cudf-polars streaming executor (`GFQL_POLARS_GPU_EXECUTOR=streaming`) spill query intermediates and results beyond RAM.
networkx	No — GFQL adds a declarative query language over the same graph.	Columnar CPU execution and a one-keyword move to GPU, on frames rather than Python objects.
igraph	No.	igraph is the CPU PageRank backend inside GFQL, so you keep it and gain the query layer, the Polars engines, and the GPU path.
Spark GraphFrames	No — GFQL is Cypher; GraphFrames is a DataFrame API.	Single-node execution with interactive latency and no cluster to provision or tune. The measured head-to-head, with its committed raw results, is in GFQL Graph Benchmark: DataFrame-Native vs Apache Spark GraphFrames.
PuppyGraph	Yes.	GPU and CPU graph analytics — PageRank, centrality, community — on the pulled subgraph, in the same pipeline as the query.

Route by shape: selective seeded lookups favor the GFQL resident index, scan and aggregate volume favors Polars, and bulk frontier expansion and full pipelines favor Polars or a GPU engine.

What is not benchmarked: motif, triangle, and other cyclic multi-way-join patterns. They run in GFQL, and we publish no performance claim about them.

Decision matrix#

Workload shape	Size (edges)	Hardware	Recommended engine	Notes
Filter / `WHERE` / aggregation	past small/interactive	CPU	`polars`	takes over past small graphs; gap grows with size [F1]
Bulk 1-hop frontier expansion	past small/interactive	CPU	`polars`	takes over past small graphs; gap grows with size [F1]
Heavy multi-hop (2-hop+)	large	GPU	`polars-gpu`	fastest until extreme materialization [F3]; GPU-or-error [F4]
Full-graph aggregation	very large	GPU	`polars-gpu` / `cudf`	GPU work-bound [F2]
One very large single materialization	huge output row count	GPU	`cudf`	Polars-GPU can hit memory pressure here [F3]
Trivially small op (bare equality filter)	any	CPU	`pandas`	boolean mask beats Polars plan overhead; immaterial [F1]
Selective / seeded traversal	any	CPU	`pandas`/`polars` + CSR index	O(degree), not an engine choice [F5]

[F1] Polars leads on CPU, and by more as the graph grows. On the q1–q9 Cypher suite it beats pandas on all nine queries at both sizes measured, and the pandas-to-Polars gap is wider on the larger graph for every query (GFQL Performance: Vectorization and GPU Acceleration). Pandas only edges out on a trivially small operation (a bare equality mask), where the absolute difference is immaterial. The real small-size floor is GPU-only — cuDF / Polars-GPU need enough work to amortize kernel launch ([F2]).

[F2] GPU is work-bound, not size-bound. A GPU wins when there is enough work to amortize its millisecond-scale kernel-launch floor: big frontiers, dense joins, full-graph aggregation. Tiny or seeded work finishes faster on CPU.

[F3] Polars-GPU memory pressure. On an extreme single materialization (a huge output row count from one join) raw cudf leads and polars-gpu slips as its in-memory GPU executor comes under memory pressure. Prefer cudf for that regime.

[F4] Polars-GPU is GPU-or-error. It never silently falls back to CPU and reports the result as a GPU run (see Honesty below).

[F5] Selective traversal is an indexing problem, not an engine choice. A seeded hop from a few nodes is fastest with the opt-in CSR adjacency index (g.gfql_index_all() / g.create_index(...), index_policy=), which turns the O(E) scan into an O(degree) gather — a complexity-class change, so the cost tracks the seeds’ neighborhood rather than the graph (index routing for the native seeded g.hop() currently engages on pandas, not yet the Polars hop path). It works on all four engines, but seeded work is so small that CPU wins — the gather is tiny work on pandas/Polars, below the GPU kernel-launch floor on cuDF — the clean inverse of bulk, where the GPU pulls ahead. So pick the index for selective traversal and a CPU engine to drive it. See Seeded Traversal Indexes (CSR Adjacency) for the full guide.

Switching engines#

The engine is a single keyword on g.gfql() (and g.hop()). The graph and the query never change — only engine= does, and the answer stays identical (or the compiler/planner declines the unsupported engine before execution rather than silently changing it).

import graphistry
g = graphistry.edges(df, 'src', 'dst')   # your existing graph (any frame type)
query = "MATCH (a)-[e]->(b) RETURN b"     # any GFQL / Cypher query

g.gfql(query)                       # engine='pandas' (default)
g.gfql(query, engine='polars')      # CPU columnar, no GPU, identical results
g.gfql(query, engine='cudf')        # NVIDIA GPU (RAPIDS)
g.gfql(query, engine='polars-gpu')  # same fused plan on GPU

Getting results back as pandas#

The result’s ._nodes / ._edges come back in the engine’s frame type: a polars.DataFrame for 'polars' / 'polars-gpu', a cudf.DataFrame for 'cudf'. When downstream code is pandas-only (matplotlib, scikit-learn, .iloc / groupby().apply()), convert once with .to_pandas():

out = g.gfql(query, engine='polars')       # or 'cudf' / 'polars-gpu'
nodes_pd = out._nodes.to_pandas()          # -> pandas for matplotlib / sklearn / ...
nodes_pd.plot.scatter(x='x', y='y')        # pandas-only downstream code, unchanged

Mixing engines#

The build frame type and the run engine are independent — GFQL coerces the input frames to the engine you ask for. A pandas graph runs on engine='polars', a Polars graph runs on engine='pandas', and so on. The only cost is a one-time convert of the input frames at the start of the call; the query then runs fully on the chosen engine. Note that engine='auto' (the default) follows the input frames — Polars frames run natively on polars, cuDF frames on cudf (or polars-gpu when that GPU path probes usable), everything else on pandas — falling back to pandas / cudf only for query shapes the native engine declines.

Tip

For selective, seeded traversal, build the CSR adjacency index once with g.gfql_index_all() (or index_policy=) — it works on all four engines and turns the O(E) scan into an O(degree) gather. An AUTO build on Polars frames now keeps them native, so g.gfql_index_all() and g.gfql_index_all(engine='polars') build the same index. See Seeded Traversal Indexes (CSR Adjacency).

Analytics under Polars (`umap` / `hypergraph` / `compute_cugraph` …)#

A GFQL call() that runs a whole-graph analytic — umap, hypergraph, compute_cugraph / compute_igraph, the *_layout ops, collapse — has no native Polars implementation (these wrap pandas / cuDF / GPU libraries and always will). Under engine='polars' / 'polars-gpu' GFQL runs them as a mode-gated, off-engine modality switch rather than declining outright:

``call_mode=’auto’`` (the default): the analytic runs off-engine — on pandas for polars, on cuDF (on device) for polars-gpu — and its result is coerced back to Polars losslessly (via Arrow). A one-time RuntimeWarning per analytic notes the off-engine run. polars-gpu is GPU-or-error: it bridges to cuDF and declines if the GPU/cuDF stack is missing (it never silently drops a GPU analytic to host pandas).
``call_mode=’strict’``: decline before running the analytic instead of bridging — for benchmark integrity (no hidden modality switch attributed to the Polars engine) or a hard memory ceiling.

Note

Memory on a very large graph. The bridge materializes a copy of the graph in the off-engine format — pandas (host) for polars, cuDF (device / unified memory) for polars-gpu. That transient copy is the same allocation you’d incur running the analytic on engine='cudf' directly, so GFQL does not add a per-call size cap (a row count is a poor memory proxy, and the real cap belongs at the RMM / container / deployment layer). For a graph large enough that the copy is a concern, either set call_mode='strict' (decline the bridge) or run the analytic under an RMM device-memory limit / container memory limit, exactly as you would for any cuDF workload.

This is deliberately narrower than traversal / filter / row ops (hop, WHERE, RETURN …), which stay parity-or-static-decline and are never bridged — a bridge there would hide a missing native impl and misreport pandas performance as Polars. Set the mode from Python or the environment (live, Python override > env > default):

from graphistry.compute.gfql.lazy import set_call_mode, CALL_MODES  # ('auto', 'strict')

set_call_mode('strict')   # decline off-engine analytics (pass None to reset to env/default)
# or: export GFQL_POLARS_CALL_MODE=strict

cuDF vs Polars-GPU#

Both run on an NVIDIA GPU, so which do you use?

cuDF is not deprecated. It remains a first-class, supported engine and is the right choice for one very large materialization (footnote F3).
They execute differently. cudf runs GFQL eagerly — each hop is a separate kernel launch with a materialized intermediate. polars-gpu runs the same fused lazy plan as the CPU Polars engine, collected once on the GPU. Fusing the plan is why polars-gpu leads on heavy multi-hop and why even CPU Polars often beats eager cuDF on bulk work.
Frame type. cudf operates on cudf.DataFrame; polars-gpu operates on polars.DataFrame (only the lazy .collect() runs on the GPU). Either way, a graph built from pandas frames is accepted and coerced for you — only the keyword changes.
Install. cudf and polars-gpu both need the RAPIDS GPU stack; polars-gpu additionally uses cudf_polars. polars (CPU) only needs pip install polars.

Larger-than-memory: streaming execution#

The default Polars engines run in-memory: fastest and most stable while the graph and its query intermediates fit in RAM (or device memory). When a query’s intermediates would blow past memory — a wide multi-hop frontier, a large join, a big aggregation — GFQL has two opt-in streaming modes that trade a little latency for a much larger working set:

Mode	Engine	What it does
`GFQL_POLARS_CPU_STREAMING=1`	`polars`	Collects the fused plan with Polars’ streaming engine — processes in batches and spills to disk, so intermediates can exceed RAM.
`GFQL_POLARS_GPU_EXECUTOR=streaming`	`polars-gpu`	Uses the cudf-polars streaming executor — the escape hatch for results larger than device memory (the default in-memory executor would OOM).

Both are off by default on purpose: they add overhead that mildly regresses small/interactive work, and for the in-memory regime the benchmarks measure, the default is faster and more stable. Results are parity-identical to the default — streaming changes how the plan runs, not what it returns.

Set them by environment variable:

# CPU: batched + disk-spill for larger-than-RAM intermediates
export GFQL_POLARS_CPU_STREAMING=1

# GPU: streaming executor for larger-than-device-memory results
export GFQL_POLARS_GPU_EXECUTOR=streaming

…or from Python at runtime — the setting is read live (per collect), and a Python override takes precedence over the environment variable:

from graphistry.compute.gfql.lazy import (
    set_cpu_streaming, set_gpu_executor, GPU_EXECUTORS,
)

set_cpu_streaming(True)          # CPU streaming collect (pass None to reset to env/default)
set_gpu_executor('streaming')    # one of GPU_EXECUTORS == ('in-memory', 'streaming')

Then use engine='polars' / engine='polars-gpu' exactly as before — no code change:

import graphistry            # env vars above must be set first
g = graphistry.edges(edges_df, 'src', 'dst')
result = g.gfql(query, engine='polars')       # streaming collect (CPU, disk-spill)
# result = g.gfql(query, engine='polars-gpu')  # streaming executor (GPU)

Note

What streaming does and does not cover today. These flags stream the query (collect), which helps when the input fits but the intermediates or result do not. They do not yet give out-of-core input: graphistry currently materializes edge/node frames at ingestion (a passed polars.LazyFrame is collected immediately), so the source graph must still fit in memory. True out-of-core-from-disk — building GFQL directly on a lazy pl.scan_parquet source so a graph larger than RAM never fully materializes — is work in progress; see the Friendster (~1.8B edges) discussion in the GraphFrames benchmark page.

When not to use Polars#

Three cases, stated so you can route around them:

Trivially small operations (a bare node-equality filter): pandas’ boolean mask beats Polars’ plan overhead, and in absolute terms it is immaterial. For traversal / WHERE / ORDER / aggregation, Polars leads on CPU (footnote F1). The real small-size caveat is GPU-only (cuDF / Polars-GPU need larger work — footnote F2).
A few exotic Cypher features are not yet native on Polars (e.g. cross-entity same-path WHERE, some temporal/entity-text forms). GFQL rejects those shapes during validation, compilation, or planning before query execution and points at engine='pandas' — it never silently bridges Polars to pandas, because that would misreport pandas performance as Polars (see Honesty).
One extreme materialization (a huge output row count): prefer cudf over polars-gpu (footnote F3).

Parity and honesty#

Identical results across engines. Differential parity — every engine’s output must match the pandas oracle — is a release gate, exercised across forward/reverse/undirected, 1-3 hop, filters, and aggregations.
No silent fallback for traversal / filter / row ops — parity-verified. For hop / WHERE / RETURN / aggregation, the Polars engine runs natively or the query is declined before execution during validation, compilation, or planning. For string GFQL / Cypher queries, known unsupported syntax and unsupported lowering shapes are rejected by the compiler/validator before execution starts; Python-built ASTs hit the same safety boundary in the local planner before the unsupported engine path runs. GFQL never quietly converts to pandas, so a traversal latency you measure is real work on the engine you asked for. polars-gpu is GPU-or-error: if any step of the plan cannot run on the GPU, the plan is rejected rather than silently running on CPU and labelling it a GPU result.
Whole-graph analytics are the one mode-gated exception. umap / hypergraph / compute_cugraph and friends have no Polars kernel; under call_mode='auto' (default) they run off-engine and warn once (see Analytics under Polars). This is not silent — it warns — and call_mode='strict' restores strict parity-or-pre-execution-decline for benchmark integrity, so a benchmarked run can guarantee no hidden modality switch.

Methodology#

Hosts, datasets, warm-median protocol, cross-engine result validation, provenance, and reproducers live with the numbers on the GFQL Performance: Vectorization and GPU Acceleration page. A figure that cannot be traced to a committed benchmark artifact is not published.

Install#

pip install graphistry          # base; pandas engine works out of the box
pip install graphistry polars   # adds the CPU 'polars' engine
# 'cudf' and 'polars-gpu' require the NVIDIA RAPIDS stack (GPU);
# 'polars-gpu' additionally uses cudf_polars.

Then change one keyword — your existing graph and query are unchanged:

import graphistry
g = graphistry.edges(df, 'src', 'dst')          # your existing pandas, Polars, or cuDF graph
g.gfql("MATCH (a)-[e]->(b) RETURN b", engine='polars')      # CPU columnar
g.gfql("MATCH (a)-[e]->(b) RETURN b", engine='polars-gpu')  # same plan on GPU

What auto does#

auto prefers the native engine for your frames and keeps a safety net. A few exotic Cypher features still require engine='pandas': the Polars engine declines them before execution rather than silently bridging. Under auto that decline is caught and the call is re-served on pandas (all-cuDF frames decline back to cudf), so a query that works today keeps working while everything the native engine does support stays native.

Pass the engine explicitly when you would rather know: engine='polars' / 'polars-gpu' raise NotImplementedError on a declined shape instead of falling back, which is what you want in a benchmark or a pipeline that must not silently change engines. engine='polars-gpu' is additionally GPU-or-error and never quietly runs on CPU.

Performance is rarely the downside — CPU Polars wins common graph queries past small/interactive sizes; only trivially small operations favor pandas, immaterially.

Note

Non-GFQL surfaces (layouts, plotting, featurization) still consume Polars frames as an input format and compute in pandas, so auto coerces there. The native-under-auto behavior described above is specific to GFQL query execution.

Choosing a GFQL Engine: pandas, Polars, cuDF, Polars-GPU

Contents

Choosing a GFQL Engine: pandas, Polars, cuDF, Polars-GPU#

The one-line speedup#

The four engines#

How the engines compare#

Coming from another graph tool#

Decision matrix#

Switching engines#

Getting results back as pandas#

Mixing engines#

Analytics under Polars (`umap` / `hypergraph` / `compute_cugraph` …)#

cuDF vs Polars-GPU#

Larger-than-memory: streaming execution#

When not to use Polars#

Parity and honesty#

Methodology#

Install#

What auto does#

See also#

Choosing a GFQL Engine: pandas, Polars, cuDF, Polars-GPU

Contents

Choosing a GFQL Engine: pandas, Polars, cuDF, Polars-GPU#

The one-line speedup#

The four engines#

How the engines compare#

Coming from another graph tool#

Decision matrix#

Switching engines#

Getting results back as pandas#

Mixing engines#

Analytics under Polars (umap / hypergraph / compute_cugraph …)#

cuDF vs Polars-GPU#

Larger-than-memory: streaming execution#

When not to use Polars#

Parity and honesty#

Methodology#

Install#

What auto does#

See also#

Analytics under Polars (`umap` / `hypergraph` / `compute_cugraph` …)#