Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion .github/workflows/build-and-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,20 @@ jobs:
echo "=== Neo4j logs ==="
docker logs neo4j-test 2>&1 | tail -30 || true

- name: Run Memgraph HA Cluster
env:
MEMGRAPH_ENTERPRISE_LICENSE: ${{ secrets.MEMGRAPH_ENTERPRISE_LICENSE }}
MEMGRAPH_ORGANIZATION_NAME: ${{ secrets.MEMGRAPH_ORGANIZATION_NAME }}
run: |
if [ -z "$MEMGRAPH_ENTERPRISE_LICENSE" ] || [ -z "$MEMGRAPH_ORGANIZATION_NAME" ]; then
echo "::warning::Memgraph enterprise license secrets are not set; skipping the HA cluster and the routing tests."
exit 0
fi

bash scripts/ha_cluster.sh start
echo "MEMGRAPH_HA_COORDINATOR_HOST=127.0.0.1" >> "$GITHUB_ENV"
echo "MEMGRAPH_HA_COORDINATOR_PORT=7703" >> "$GITHUB_ENV"

- name: Install More Packages (Python <= 3.12)
if: ${{ matrix.python-version == '3.10' || matrix.python-version == '3.11' || matrix.python-version == '3.12' }}
run: |
Expand Down Expand Up @@ -193,6 +207,10 @@ jobs:
# pyproject.toml from the repo root) and is covered by the source run.
pytest tests --ignore=tests/test_packaging.py -o addopts='' -vvv -m "not slow and not ubuntu and not docker and not tfgnn"

- name: Stop Memgraph HA Cluster
if: always()
run: bash scripts/ha_cluster.sh stop || echo "HA cluster does not exist"

- name: Use the Upload Artifact GitHub Action
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always()
Expand Down Expand Up @@ -256,7 +274,7 @@ jobs:
mkdir -p /var/lib/memgraph/data-windows-${{ matrix.python-version }}
chown -R memgraph:memgraph /var/lib/memgraph/data-windows-${{ matrix.python-version }}
pkill -f '/usr/lib/memgraph/memgraph' || true
runuser -l memgraph -c 'nohup /usr/lib/memgraph/memgraph --bolt-address 0.0.0.0 --bolt-port 7687 --data-directory="/var/lib/memgraph/data-windows-${{ matrix.python-version }}" --storage-properties-on-edges=true --storage-snapshot-interval-sec=0 --storage-wal-enabled=false --storage-snapshot-on-exit=false --telemetry-enabled=false --log-level=TRACE --also-log-to-stderr=true --log-file=/mnt/c/memgraph/memgraph-windows-${{ matrix.python-version }}.log >/mnt/c/memgraph/memgraph-windows-${{ matrix.python-version }}-stdout.log 2>&1 < /dev/null &'
runuser -l memgraph -c 'nohup /usr/lib/memgraph/memgraph --bolt-address 0.0.0.0 --bolt-port 7687 --data-directory="/var/lib/memgraph/data-windows-${{ matrix.python-version }}" --storage-properties-on-edges=true --storage-snapshot-interval-sec=0 --storage-wal-enabled=false --storage-snapshot-on-exit=false --telemetry-enabled=false --log-level=TRACE --also-log-to-stderr=true --log-file=/mnt/c/memgraph/memgraph-windows-${{ matrix.python-version }}.log >/mnt/c/memgraph/memgraph-windows-${{ matrix.python-version }}-stdout.log 2>&1 < /dev/null &'

- name: Wait for Memgraph to be ready (WSL)
shell: wsl-bash {0} # root shell
Expand Down
169 changes: 169 additions & 0 deletions docs/how-to-guides/high-availability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
# How to connect to a high-availability cluster

Through this guide, you will learn how to use GQLAlchemy to connect to a
[Memgraph high-availability (HA) cluster](https://memgraph.com/docs/clustering/high-availability)
with client-side routing: you point GQLAlchemy at a *coordinator* and it routes
each query to the right data instance — writes to the main, reads to a replica.

- [How to connect to a high-availability cluster](#how-to-connect-to-a-high-availability-cluster)
- [Route a client through a coordinator](#route-a-client-through-a-coordinator)
- [Choose an access mode](#choose-an-access-mode)
- [Run managed transactions](#run-managed-transactions)
- [Handle transient errors and tune retries](#handle-transient-errors-and-tune-retries)
- [Inspect the routing table](#inspect-the-routing-table)

>If you have any more questions, join our community and ping us on [Discord](https://discord.gg/memgraph).

!!! info
High availability is a [Memgraph Enterprise](https://memgraph.com/docs/database-management/enabling-memgraph-enterprise)
feature, so you need a running HA cluster (coordinators and data instances)
to use the features below. Client-side routing requires `pymgclient>=1.6.0`,
which GQLAlchemy depends on. If you're unsure how to run an HA cluster, check
out the Memgraph [high-availability documentation](https://memgraph.com/docs/clustering/high-availability).

## Route a client through a coordinator

Create a [`Memgraph`](../reference/gqlalchemy/vendors/memgraph.md) client with
`routing=True` and point its `host`/`port` at a cluster **coordinator** rather
than a specific data instance. GQLAlchemy fetches the cluster topology from the
coordinator and routes queries accordingly:

```python
from gqlalchemy import Memgraph

db = Memgraph(host="coordinator-1", port=7687, routing=True)

# Routed to the main.
db.execute("MERGE (:Person {name: 'Ada'})")

# Also routed to the main (the default access mode serves both reads and writes).
for result in db.execute_and_fetch("MATCH (n:Person) RETURN n.name AS name"):
print(result["name"])
```

With `routing=False` (the default) GQLAlchemy connects directly to the given
`host`/`port`, exactly as before.

## Choose an access mode

`access_mode` selects which instance a routed client connects to: `"WRITE"` (the
default) targets the main, `"READ"` targets a replica. You can pass the string
or the `mgclient` constant:

```python
from gqlalchemy import Memgraph

# A read-only client, routed to a replica.
reader = Memgraph(host="coordinator-1", port=7687, routing=True, access_mode="READ")
for result in reader.execute_and_fetch("MATCH (n:Person) RETURN count(n) AS count"):
print(result["count"])
```

```python
import mgclient
from gqlalchemy import Memgraph

writer = Memgraph(
host="coordinator-1",
port=7687,
routing=True,
access_mode=mgclient.ACCESS_MODE_WRITE,
)
```

!!! info
A routed client opens its connection to whichever instance was current when
it connected. To keep working through a failover — where the main changes —
use the managed transactions described below, which re-route and retry
automatically.

## Run managed transactions

For failover-safe operations, use [`execute_write()`](../reference/gqlalchemy/vendors/memgraph.md#execute_write)
and [`execute_read()`](../reference/gqlalchemy/vendors/memgraph.md#execute_read).
Each takes a `work` callable that receives a transaction object exposing the
usual `execute` / `execute_and_fetch`; the transaction is routed (writes to the
main, reads to a replica), committed for you, and **retried automatically** if
the cluster is briefly in flux (for example while a new main is being elected).
Whatever `work` returns is returned to you:

```python
from gqlalchemy import Memgraph

db = Memgraph(host="coordinator-1", port=7687, routing=True)

def add_person(tx):
tx.execute("MERGE (:Person {name: $name})", {"name": "Ada"})
rows = list(tx.execute_and_fetch("MATCH (n:Person) RETURN count(n) AS count"))
return rows[0]["count"]

count = db.execute_write(add_person)

def count_people(tx):
rows = list(tx.execute_and_fetch("MATCH (:Person) RETURN count(*) AS count"))
return rows[0]["count"]

total = db.execute_read(count_people)
```

!!! warning
Because a managed transaction may run **more than once** on retry, make the
`work` idempotent — use `MERGE` rather than `CREATE` so a re-run cannot
duplicate a write. Materialize any results you need (e.g. wrap
`execute_and_fetch` in `list(...)`) before returning them, since the
transaction is closed once `work` returns.

## Handle transient errors and tune retries

If a managed transaction cannot complete even after refreshing the routing table
and exhausting its retries, GQLAlchemy raises
[`GQLAlchemyTransientError`](../reference/gqlalchemy/exceptions.md) — a subclass
of `GQLAlchemyDatabaseError`, so existing error handling keeps working while
letting you single out retryable failures:

```python
from gqlalchemy import Memgraph, GQLAlchemyTransientError

db = Memgraph(host="coordinator-1", port=7687, routing=True)

try:
db.execute_write(add_person)
except GQLAlchemyTransientError:
# The cluster did not settle within the retry budget; surface or re-queue.
raise
```

The retry budget is configurable on the client. Backoff is capped exponential —
`retry_backoff`, `2 * retry_backoff`, … up to `retry_backoff_cap` seconds:

```python
db = Memgraph(
host="coordinator-1",
port=7687,
routing=True,
max_retries=8,
retry_backoff=1.0,
retry_backoff_cap=15.0,
)
```

## Inspect the routing table

To see the topology the coordinator reports, use
[`get_routing_table()`](../reference/gqlalchemy/vendors/memgraph.md#get_routing_table).
It returns a dict with `ttl`, `write`, `read` and `route` entries:

```python
from gqlalchemy import Memgraph

db = Memgraph(host="coordinator-1", port=7687, routing=True)

table = db.get_routing_table()
print("main(s):", table["write"])
print("replica(s):", table["read"])
print("coordinator(s):", table["route"])
```

>Hopefully, this guide has taught you how to connect to a Memgraph
>high-availability cluster using GQLAlchemy. If you have any more questions, join
>our community and ping us on [Discord](https://discord.gg/memgraph).
9 changes: 9 additions & 0 deletions docs/how-to-guides/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,15 @@ statistics for index optimization.

- [**Manage Memgraph database**](manage-database.md)

## Connect to a high-availability cluster

GQLAlchemy can connect to a Memgraph high-availability cluster through a
coordinator and route each query to the right data instance — writes to the
main, reads to a replica — including failover-safe managed transactions that
retry transient cluster conditions automatically.

- [**Connect to a high-availability cluster**](high-availability.md)

## Transform Python graphs into Memgraph graphs

GQLAlchemy holds transformations that can transform NetworkX, PyG and DGL graphs
Expand Down
32 changes: 32 additions & 0 deletions docs/reference/gqlalchemy/connection.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,38 @@ def is_active() -> bool

Returns True if connection is active and can be used.

Comment thread
mattkjames7 marked this conversation as resolved.
## \_RoutedTransaction Objects

```python
class _RoutedTransaction()
```

The object handed to an ``execute_read`` / ``execute_write`` work function.

Wraps the pymgclient cursor of a managed transaction and exposes the same
``execute`` / ``execute_and_fetch`` surface as ``class MemgraphConnection()``,
converting rows to gqlalchemy values. The surrounding transaction is managed
by the router (begun, committed and retried around the work), so the work
must not commit or roll back itself.

#### execute

```python
def execute(query: str, parameters: Dict[str, Any] = {}) -> None
```

Executes Cypher query without returning any results.

#### execute\_and\_fetch

```python
def execute_and_fetch(
query: str,
parameters: Dict[str, Any] = {}) -> Iterator[Dict[str, Any]]
```

Executes Cypher query and returns iterator of results.

## Neo4jConnection Objects

```python
Expand Down
8 changes: 8 additions & 0 deletions docs/reference/gqlalchemy/exceptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@ sidebar_label: exceptions
title: gqlalchemy.exceptions
---

## GQLAlchemyTransientError Objects

```python
class GQLAlchemyTransientError(GQLAlchemyDatabaseError)
```

A database error that is worth retrying.

#### connection\_handler

```python
Expand Down
32 changes: 32 additions & 0 deletions docs/reference/gqlalchemy/vendors/memgraph.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,38 @@ def new_connection() -> Connection

Comment thread
mattkjames7 marked this conversation as resolved.
Creates new Memgraph connection.

#### execute\_write

```python
def execute_write(work)
```

Runs ``work(tx)`` as a managed write against the main.

The router wraps the work in a transaction, commits it, and retries
transient failover conditions (with a routing refresh and capped
exponential backoff). Because the work may run more than once, make it
idempotent (e.g. ``MERGE`` rather than ``CREATE``). Requires ``routing=True``.

#### execute\_read

```python
def execute_read(work)
```

Runs ``work(tx)`` as a managed read against a replica.

Same retry semantics as :meth:`execute_write`. Requires ``routing=True``.

#### get\_routing\_table

```python
def get_routing_table()
```

Returns a snapshot of the cluster routing table as a dict with
``ttl``, ``write``, ``read`` and ``route`` entries. Requires ``routing=True``.

#### create\_stream

```python
Expand Down
7 changes: 6 additions & 1 deletion gqlalchemy/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,12 @@
wait_for_docker_container,
wait_for_port,
)
from gqlalchemy.exceptions import GQLAlchemyError, GQLAlchemyWarning # noqa F401
from gqlalchemy.exceptions import ( # noqa F401
GQLAlchemyError,
GQLAlchemyWarning,
GQLAlchemyDatabaseError,
GQLAlchemyTransientError,
)

from gqlalchemy.query_builders import ( # noqa F401
neo4j_query_builder,
Expand Down
Loading