Backend-to-backend migration

cvcdocdb.migration copies an entire graph — nodes, then edges, then vector indexes — from one GraphStore into another (e.g. NetworkXGraph → Neo4jGraph or the reverse), using only the common GraphStore interface. It doesn’t know anything about WeakNode, Individu/be_value_properties, or any other Python-level entity class, so it works between any two backend implementations, current or future.

Quick-start

from cvcdocdb import NetworkXGraph, Neo4jGraph
from cvcdocdb.migration import migrate

source = NetworkXGraph("archive.pkl")
target = Neo4jGraph("bolt://localhost:7687", "neo4j", "secret")
stats = migrate(source, target)
print(stats)  # MigrationStats(nodes_migrated=120, edges_migrated=340, ...)

Notes

Nodes are matched between source and target by (main_label, pk). Some backends (currently Neo4jGraph) don’t persist which properties form a node’s primary key — when that happens, every property on the node is used as its pk instead, so the node is still faithfully reproduced.
label_filter/property_filter accept the same MongoDB-style syntax as GraphDataset.
Reads/writes are batched (chunk_size), with a faster round-trip path when the source/target is a Neo4jGraph.
on_error="skip" allows a best-effort migration of a large, possibly messy graph instead of aborting on the first error.

See also cvcdocdb.graph_store for the interface this module relies on, and cvcdocdb.schema_gen to generate Python entity classes from either side’s schema after migrating.

Generic backend-to-backend graph migration.

Copies an entire graph — nodes, then edges, then vector indexes — from one GraphStore into another (e.g. NetworkXGraph → Neo4jGraph or the reverse), using only the common GraphStore interface. It doesn’t know anything about WeakNode, Individu/be_value_properties, or any other Python-level entity class — it works at the plain node/edge level, so it works for any two backend implementations, current or future.

Quick-start:

from cvcdocdb import NetworkXGraph, Neo4jGraph
from cvcdocdb.migration import migrate

source = NetworkXGraph("archive.pkl")
target = Neo4jGraph("bolt://localhost:7687", "neo4j", "secret")
stats = migrate(source, target)
print(stats)  # MigrationStats(nodes_migrated=120, edges_migrated=340, ...)

class cvcdocdb.migration.MigrationStats(nodes_migrated=0, nodes_skipped=0, edges_migrated=0, edges_skipped=0, indexes_migrated=0, indexes_skipped=0, errors=<factory>)[source]

Bases: object

Counters describing a completed (or in-progress) migration.

Parameters:

nodes_migrated (int)
nodes_skipped (int)
edges_migrated (int)
edges_skipped (int)
indexes_migrated (int)
indexes_skipped (int)
errors (List[str])

nodes_migrated: int = 0

nodes_skipped: int = 0

edges_migrated: int = 0

edges_skipped: int = 0

indexes_migrated: int = 0

indexes_skipped: int = 0

errors: List[str]

__init__(nodes_migrated=0, nodes_skipped=0, edges_migrated=0, edges_skipped=0, indexes_migrated=0, indexes_skipped=0, errors=<factory>)

Parameters:

nodes_migrated (int)
nodes_skipped (int)
edges_migrated (int)
edges_skipped (int)
indexes_migrated (int)
indexes_skipped (int)
errors (List[str])

Return type:

None

cvcdocdb.migration.migrate(source, target, label_filter=None, property_filter=None, update=True, replace=False, chunk_size=500, on_error='raise')[source]

Copy an entire graph from source into target, in three phases: nodes, then edges, then vector indexes.

Generic across backends — only uses the common GraphStore interface (get_node_ids, get_node_attrs, get_edges, get_edge_attrs, insertNode, insertRelation, list_vector_indexes, enable_vector_index), so it works between any two implementations in either direction. Nodes are read and written in chunks of chunk_size (batched over the network for a Neo4jGraph source/target instead of one round-trip per node) so migrating a graph with many nodes/edges doesn’t require holding the whole thing in memory or paying a per-row round-trip.

Nodes are matched between source and target by (main_label, pk). Some backends (currently Neo4jGraph) don’t persist which properties form a node’s primary key — reading one back from such a backend can’t tell “this is the pk” apart from “this is a regular attribute”. When that happens (get_node_attrs() reports pk=None), every property on that node is used as its pk instead, so the node is still faithfully reproduced on target; the only consequence is that a repeated migration run matches on the full property set instead of a smaller natural key.

Edges are cascade-delete-safe: their attributes (including _propagate) are copied as-is, so WeakNode-style cascade delete behaves the same on target even though migration itself never constructs a WeakNode.

Locking: the whole migration — all three phases — runs under a single write lock held on both source and target for the entire duration, not one lock per phase. It’s a write lock, not an exclusive lock: concurrent reads of either store (by other processes/instances/threads) are never blocked, only concurrent writes are, so the migration is never torn by another writer mutating either store mid-flight, while readers keep seeing a consistent view throughout (the state as of just before the migration started, until it completes and its own writes land). Backend-specific granularity:

NetworkXGraph: the existing cross-process file lock (see NetworkXGraph.batch()/_guarded_write()), held for the whole function. Other mutating calls on the same persistence_path (any process) block until migration finishes; reads never touch this lock and are unaffected.
Neo4jGraph: a single explicit transaction spanning the whole function (see Neo4jGraph.batch()). Neo4j has no literal whole-database lock, so this is the closest honest approximation — row/relationship locks are taken as the transaction touches them, escalating in practice to the whole migrated subgraph for its duration, but it is not a database-wide exclusive lock at the server level. For a very large graph this also means the transaction accumulates a correspondingly large amount of uncommitted work before the single commit at the end; that’s the direct cost of “one lock for the whole migration” instead of the previous per-phase commits.

Constraints/indexes: this project deliberately does not create Neo4j-side NODE KEY/uniqueness constraints (the same main_label can be used with different pk shapes by different callers — see the CHANGELOG), so there is no such constraint to migrate. The one portable “index” concept is NetworkXGraph’s vector (ANN) index: every index reported by source.list_vector_indexes() is recreated on target via enable_vector_index(). If target doesn’t support vector indexes (e.g. a Neo4jGraph target, which always raises NotImplementedError), that index is counted in indexes_skipped rather than failing the migration — the graph data itself has already been migrated successfully at that point.

Parameters:

source (GraphStore) – The graph to read from.
target (GraphStore) – The graph to write to.
label_filter (str | List[str] | None) – Keep only nodes whose main_label is in this set (a bare string is a singleton). None migrates every node. An edge is migrated only if both endpoints were kept.
property_filter (Dict[str, Any] | None) – MongoDB-style filter (same syntax as GraphDataset), applied on top of label_filter.
update (bool) – If True (default), MERGE + SET a node/edge that already exists on target instead of failing — makes re-running the migration idempotent.
replace (bool) – If True, delete-and-recreate an existing node/edge on target instead of merging. Takes precedence over update when both are set (mirrors insertNode).
chunk_size (int) – Rows read/written per round-trip when a batched path is available (currently: a Neo4jGraph source or target).
on_error (str) – "raise" (default) lets an exception from a single node/edge/index abort the whole migration. "skip" records it in MigrationStats.errors and continues with the rest — useful for a best-effort migration of a large, possibly messy graph.

Returns:

A MigrationStats with the counts of what was copied.

Return type:

MigrationStats