{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "open-in-colab-kaggle",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/CVC-DAG/cvcdocdb/blob/main/docs/tutorials/notebooks/demos/jena_demo.ipynb)\n",
    "[![Open in Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://www.kaggle.com/kernels/welcome?src=https://raw.githubusercontent.com/CVC-DAG/cvcdocdb/main/docs/tutorials/notebooks/demos/jena_demo.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48445813",
   "metadata": {},
   "source": [
    "# Propagation Properties — Full Apache Jena (Fuseki) Demo\n",
    "\n",
    "The workflow of the Neo4j demo (`propagation_demo.ipynb`) on **Apache Jena\n",
    "Fuseki**. `JenaGraph` stores the graph as RDF 1.2 and has the same API and\n",
    "propagation behaviour as every other backend; what changes is how you query it:\n",
    "\n",
    "- **SPARQL** strings run on the Fuseki server, against the RDF the graph is\n",
    "  stored as. Every write goes through the cvcdocdb API, so SPARQL Update is\n",
    "  refused.\n",
    "- **Cypher** strings and dict filters run on `JenaGraph`'s in-memory copy of the\n",
    "  graph, as on `NetworkXGraph`.\n",
    "\n",
    "**Steps:**\n",
    "1. Connect to Fuseki and load the Game of Thrones dataset\n",
    "2. Generate the YAML schema and the Python entity classes\n",
    "3. Initialize propagation properties\n",
    "4. Create a Document → Section → Page group with `create_group()`\n",
    "5. Query in SPARQL and Cypher, get subdocuments, and explore the graph in the viewer\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "Apache Jena Fuseki **6.2.0 or later** (RDF 1.2) with an updatable dataset, for\n",
    "example in Docker:\n",
    "\n",
    "```bash\n",
    "docker run -p 3030:3030 -e FUSEKI_BASE=/tmp/base -w /jena-fuseki --entrypoint sh \\\n",
    "  secoresearch/fuseki:6.2.0 -c \"mkdir -p /tmp/db /tmp/base && exec java -cp '*:/javalibs/*' \\\n",
    "  org.apache.jena.fuseki.main.cmds.FusekiServerCmd --tdb2 --loc /tmp/db --update /ds\"\n",
    "```\n",
    "\n",
    "The notebook reads `FUSEKI_URL` (default `http://localhost:3030/ds`) and, if the\n",
    "dataset needs them, `FUSEKI_USER` and `FUSEKI_PASSWORD`.\n",
    "\n",
    "## The RDF layout\n",
    "\n",
    "With the default namespace `urn:cvcdocdb:`, a node is `<urn:cvcdocdb:node/ID>`,\n",
    "`a label:Character` for each label, with one `prop:name \"…\"` triple per property.\n",
    "A relation is the triple `<node/U> rel:MEMBER_OF <node/V>`, and its properties\n",
    "hang from an RDF 1.2 reifier of that triple. Step 8 queries both.\n",
    "\n",
    "**Warning:** Step 1.5 deletes every triple in the `urn:cvcdocdb:` namespace\n",
    "(other data in the dataset is left alone)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "332201e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib.util\n",
    "\n",
    "# cvcdocdb with the [notebook] extra (anywidget) for the graph viewer at the end.\n",
    "if (importlib.util.find_spec(\"cvcdocdb\") is None\n",
    "        or importlib.util.find_spec(\"cvcdocdb.viewer\") is None\n",
    "        or importlib.util.find_spec(\"anywidget\") is None):\n",
    "    print(\"⚠️ Installing cvcdocdb[notebook]...\")\n",
    "    %pip install -q --upgrade \"cvcdocdb[notebook]\"\n",
    "    print(\"✅ Installation complete. Restart the kernel if an import below fails.\")\n",
    "else:\n",
    "    print(\"✅ cvcdocdb and anywidget are already installed.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d07f415",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import tempfile\n",
    "from pathlib import Path\n",
    "\n",
    "from cvcdocdb import JenaGraph\n",
    "from cvcdocdb.exemples import load_got_characters\n",
    "from cvcdocdb.schema_gen import generate_classes\n",
    "\n",
    "def section(title):\n",
    "    \"\"\"Print a section header.\"\"\"\n",
    "    print(f\"\\n{'=' * 60}\")\n",
    "    print(f\"  {title}\")\n",
    "    print(f\"{'=' * 60}\")\n",
    "\n",
    "\n",
    "def sub_section(title):\n",
    "    \"\"\"Print a subsection header.\"\"\"\n",
    "    print(f\"\\n--- {title} ---\")\n",
    "\n",
    "\n",
    "# Generated files (schema and entity classes) go to a fresh temporary directory.\n",
    "output_dir = Path(tempfile.mkdtemp(prefix=\"cvcdocdb_got_\"))\n",
    "\n",
    "\n",
    "PREFIXES = \"\"\"\n",
    "PREFIX rdf:   <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n",
    "PREFIX label: <urn:cvcdocdb:label/>\n",
    "PREFIX prop:  <urn:cvcdocdb:prop/>\n",
    "PREFIX rel:   <urn:cvcdocdb:rel/>\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "def sparql(query):\n",
    "    \"\"\"Run a SPARQL query on the Fuseki server (with the cvcdocdb prefixes).\"\"\"\n",
    "    return graph.query(PREFIXES + query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8445210f",
   "metadata": {},
   "source": [
    "## Step 1: Connect to Fuseki\n",
    "\n",
    "`JenaGraph` checks the Fuseki version on connection and refuses servers older\n",
    "than 6.2.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe4cdd59",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = os.environ.get(\"FUSEKI_URL\", \"http://localhost:3030/ds\")\n",
    "graph = JenaGraph(url, os.environ.get(\"FUSEKI_USER\"), os.environ.get(\"FUSEKI_PASSWORD\"))\n",
    "print(f\"Connected to {url} (Fuseki {graph.server_version})\")\n",
    "\n",
    "total = sparql(\"SELECT (COUNT(DISTINCT ?n) AS ?total) WHERE { ?n a ?label }\")[0][\"total\"]\n",
    "print(f\"Typed resources in the dataset: {total}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b913b05a",
   "metadata": {},
   "source": [
    "## Step 1.5: Clear existing data\n",
    "\n",
    "`clear()` deletes everything in the graph's namespace, in one SPARQL Update."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8eb819a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "graph.clear()\n",
    "print(f\"✅ Graph cleared. Nodes remaining: {len(graph.get_node_ids())}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832d8069",
   "metadata": {},
   "source": [
    "## Step 2: Load the Game of Thrones dataset\n",
    "\n",
    "Load character and house data from [thronesapi.com](https://thronesapi.com).\n",
    "Falls back to bundled data if the API is unavailable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18d9cf32",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats = load_got_characters(graph, limit=25)\n",
    "print(\"Loaded:\", json.dumps(stats, indent=2))\n",
    "\n",
    "sub_section(\"Nodes by label (SPARQL)\")\n",
    "rows = sparql(\"\"\"\n",
    "SELECT ?label (COUNT(?n) AS ?cnt) WHERE {\n",
    "  ?n a ?type .\n",
    "  FILTER(STRSTARTS(STR(?type), STR(label:)))\n",
    "  BIND(STRAFTER(STR(?type), STR(label:)) AS ?label)\n",
    "} GROUP BY ?label ORDER BY DESC(?cnt)\n",
    "\"\"\")\n",
    "for row in rows:\n",
    "    print(f\"  {row['label']}: {row['cnt']}\")\n",
    "\n",
    "sub_section(\"Relations by type (SPARQL)\")\n",
    "rows = sparql(\"\"\"\n",
    "SELECT ?type (COUNT(*) AS ?cnt) WHERE {\n",
    "  ?a ?p ?b .\n",
    "  FILTER(STRSTARTS(STR(?p), STR(rel:)))\n",
    "  BIND(STRAFTER(STR(?p), STR(rel:)) AS ?type)\n",
    "} GROUP BY ?type ORDER BY DESC(?cnt)\n",
    "\"\"\")\n",
    "for row in rows:\n",
    "    print(f\"  {row['type']}: {row['cnt']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6b59bc9",
   "metadata": {},
   "source": [
    "## Step 3: Generate the YAML schema\n",
    "\n",
    "Introspect the graph and generate a YAML schema description: labels,\n",
    "properties, relationships and counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c041a0a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "yaml_content = graph.schema_yaml(db_name=\"got\")\n",
    "yaml_path = output_dir / \"got_schema.yaml\"\n",
    "yaml_path.write_text(yaml_content)\n",
    "\n",
    "print(f\"Written: {yaml_path} ({len(yaml_content)} bytes)\")\n",
    "print(\"\\nSchema preview:\")\n",
    "for line in yaml_content.split(\"\\n\")[:20]:\n",
    "    print(f\"  {line}\")\n",
    "print(\"  ...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a615c6a1",
   "metadata": {},
   "source": [
    "## Step 4: Generate Python entity classes\n",
    "\n",
    "Generate `Node`, `WeakNode`, `Relation` and `WeakRelation` subclasses for each\n",
    "label and relationship type in the schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "611da42f",
   "metadata": {},
   "outputs": [],
   "source": [
    "code = generate_classes(yaml_path.read_text())\n",
    "py_path = output_dir / \"got_entities.py\"\n",
    "py_path.write_text(code)\n",
    "\n",
    "print(f\"Written: {py_path} ({len(code)} bytes)\")\n",
    "print(\"\\nGenerated code preview:\")\n",
    "for line in code.split(\"\\n\")[:25]:\n",
    "    print(f\"  {line}\")\n",
    "print(\"  ...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f34d0b67",
   "metadata": {},
   "source": [
    "## Step 5: Initialize propagation properties\n",
    "\n",
    "`init_propagation()` scans the graph and initialises the propagation properties:\n",
    "\n",
    "- `_propagate` on edges with cascade delete enabled;\n",
    "- `is_weak` on the nodes at the child end of those edges;\n",
    "- `parent_relation`: the edge type linking parent → child;\n",
    "- `_weak_init_done` on the nodes already initialised.\n",
    "\n",
    "The Game of Thrones data has no WeakNodes, so the counts only change after the\n",
    "group of Step 6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be59914a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_counts(title):\n",
    "    sub_section(title)\n",
    "    row = sparql(\"\"\"\n",
    "    SELECT (COUNT(DISTINCT ?n) AS ?total) (COUNT(DISTINCT ?w) AS ?weak) (COUNT(DISTINCT ?d) AS ?done) WHERE {\n",
    "      ?n a ?label .\n",
    "      OPTIONAL { ?n prop:is_weak true . BIND(?n AS ?w) }\n",
    "      OPTIONAL { ?n prop:_weak_init_done ?flag . BIND(?n AS ?d) }\n",
    "    }\"\"\")[0]\n",
    "    print(f\"  Total nodes:                {row['total']}\")\n",
    "    print(f\"  Nodes with is_weak:         {row['weak']}\")\n",
    "    print(f\"  Nodes with _weak_init_done: {row['done']}\")\n",
    "    # Edge properties live on the RDF 1.2 reifier of the relation triple.\n",
    "    edges = sparql(\"SELECT (COUNT(?r) AS ?c) WHERE { ?r rdf:reifies ?t ; prop:_propagate true }\")[0][\"c\"]\n",
    "    print(f\"  Edges with _propagate:      {edges}\")\n",
    "\n",
    "show_counts(\"Before init_propagation()\")\n",
    "print(\"\\nRunning init_propagation()...\", graph.init_propagation())\n",
    "show_counts(\"After init_propagation()\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ba98cfe",
   "metadata": {},
   "source": [
    "## Step 6: Transactional group creation\n",
    "\n",
    "`create_group()` creates a strong node with its WeakNodes and WeakRelations in a\n",
    "single transaction: if any step fails, nothing is written. We create a\n",
    "Document → Section → Page hierarchy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4f4c5cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from cvcdocdb.base import Node, WeakNode\n",
    "\n",
    "doc = Node(\n",
    "    pk={\"doc_id\": \"GOT-DEMO-001\"},\n",
    "    main_label=\"Document\",\n",
    "    title=\"Game of Thrones — Episode Guide\",\n",
    "    author=\"Demo Script\",\n",
    ")\n",
    "section1 = WeakNode(parent=doc, parent_relation=\"HAS_SECTION\", pk={\"section\": \"intro\"},\n",
    "                    main_label=\"Section\", title=\"Introduction\", order=1)\n",
    "section2 = WeakNode(parent=doc, parent_relation=\"HAS_SECTION\", pk={\"section\": \"characters\"},\n",
    "                    main_label=\"Section\", title=\"Main Characters\", order=2)\n",
    "page1 = WeakNode(parent=section1, parent_relation=\"HAS_PAGE\", pk={\"page\": 1}, main_label=\"Page\",\n",
    "                 content=\"Welcome to the Game of Thrones episode guide.\", order=1)\n",
    "page2 = WeakNode(parent=section2, parent_relation=\"HAS_PAGE\", pk={\"page\": 1}, main_label=\"Page\",\n",
    "                 content=\"Jon Snow, Daenerys Targaryen, Tyrion Lannister...\", order=1)\n",
    "\n",
    "doc_id = graph.create_group(strong_node=doc, weak_nodes=[section1, section2, page1, page2])\n",
    "print(f\"✅ Group created (strong node id: {doc_id}): Document, 2 Sections, 2 Pages\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be60bb0c",
   "metadata": {},
   "source": [
    "## Step 7: Verify the group\n",
    "\n",
    "The same question twice: in SPARQL on the server, and in Cypher on the in-memory\n",
    "copy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e66b4a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_section(\"Sections and Pages (SPARQL)\")\n",
    "rows = sparql(\"\"\"\n",
    "SELECT ?label ?text ?weak WHERE {\n",
    "  ?doc a label:Document ; prop:doc_id \"GOT-DEMO-001\" ; (rel:HAS_SECTION|rel:HAS_SECTION/rel:HAS_PAGE) ?n .\n",
    "  ?n a ?type ; prop:is_weak ?weak .\n",
    "  OPTIONAL { ?n prop:title ?title } OPTIONAL { ?n prop:content ?content }\n",
    "  BIND(STRAFTER(STR(?type), STR(label:)) AS ?label)\n",
    "  BIND(COALESCE(?title, ?content) AS ?text)\n",
    "} ORDER BY ?label ?text\n",
    "\"\"\")\n",
    "for row in rows:\n",
    "    print(f\"  {row['label']}: {row['text'][:45]!r} is_weak={row['weak']}\")\n",
    "\n",
    "sub_section(\"Sections and Pages (Cypher, in-memory copy)\")\n",
    "# The in-memory Cypher has no variable-length paths (-[*1..2]->): one pattern per level.\n",
    "for pattern in (\"-[:HAS_SECTION]->(n)\", \"-[:HAS_SECTION]->()-[:HAS_PAGE]->(n)\"):\n",
    "    rows = graph.query(\n",
    "        f\"MATCH (d:Document {{doc_id: 'GOT-DEMO-001'}}){pattern} \"\n",
    "        \"RETURN labels(n)[0] AS label, n.is_weak AS is_weak\"\n",
    "    )\n",
    "    for row in rows:\n",
    "        print(f\"  {row['label']}: is_weak={row['is_weak']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2285a1a8",
   "metadata": {},
   "source": [
    "## Step 8: Sample queries\n",
    "\n",
    "`GROUP_CONCAT` gathers each house's characters. The second query reads the\n",
    "`_propagate` flag from the reifier of each relation triple, with the RDF 1.2\n",
    "triple-term syntax `<<( s p o )>>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d55f0724",
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_section(\"Characters by House (SPARQL)\")\n",
    "rows = sparql(\"\"\"\n",
    "SELECT ?house (GROUP_CONCAT(?name; separator=\", \") AS ?characters) WHERE {\n",
    "  ?c a label:Character ; prop:name ?name ; rel:MEMBER_OF ?h .\n",
    "  ?h prop:name ?house .\n",
    "} GROUP BY ?house ORDER BY ?house\n",
    "\"\"\")\n",
    "for row in rows:\n",
    "    names = row[\"characters\"].split(\", \")\n",
    "    print(f\"  {row['house']}: {', '.join(names[:5])}\" + (f\" ... and {len(names) - 5} more\" if len(names) > 5 else \"\"))\n",
    "\n",
    "sub_section(\"Edges with _propagate (SPARQL, RDF 1.2)\")\n",
    "rows = sparql(\"\"\"\n",
    "SELECT ?from_label ?rel_type ?to_label (COUNT(*) AS ?count) WHERE {\n",
    "  ?r rdf:reifies <<( ?a ?p ?b )>> ; prop:_propagate true .\n",
    "  ?a <urn:cvcdocdb:meta/mainLabel> ?from_label .\n",
    "  ?b <urn:cvcdocdb:meta/mainLabel> ?to_label .\n",
    "  BIND(STRAFTER(STR(?p), STR(rel:)) AS ?rel_type)\n",
    "} GROUP BY ?from_label ?rel_type ?to_label ORDER BY DESC(?count)\n",
    "\"\"\")\n",
    "for row in rows:\n",
    "    print(f\"  {row['from_label']} -[{row['rel_type']}]-> {row['to_label']}: {row['count']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a93bf99",
   "metadata": {},
   "source": [
    "## Step 9: Get subdocuments\n",
    "\n",
    "`get_subdocuments()` returns every node reachable from a strong node through\n",
    "`_propagate` edges: the document's sections and pages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01b2076f",
   "metadata": {},
   "outputs": [],
   "source": [
    "strong_doc = Node(pk={\"doc_id\": \"GOT-DEMO-001\"}, main_label=\"Document\")\n",
    "subdocuments = graph.get_subdocuments(strong_doc)\n",
    "\n",
    "print(f\"Found {len(subdocuments)} subdocuments:\")\n",
    "for sd in subdocuments:\n",
    "    print(f\"  - {sd['label']}: {sd['pk']}\")  # a WeakNode's key includes its parents' keys"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73d25fec",
   "metadata": {},
   "source": [
    "## Step 10: Explore the graph\n",
    "\n",
    "The graph viewer works on every backend. Click a node to see its properties,\n",
    "double-click it to expand its neighbours (try the Document), and search or filter\n",
    "from the sidebar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8dd5905",
   "metadata": {},
   "outputs": [],
   "source": [
    "from cvcdocdb.viewer import show\n",
    "\n",
    "show(graph, load_all=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38b729d8",
   "metadata": {},
   "source": [
    "## Cleanup\n",
    "\n",
    "Closing the graph empties its in-memory copy, and the viewer with it: run this\n",
    "once you are done exploring. The data stays in Fuseki."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d0abe71",
   "metadata": {},
   "outputs": [],
   "source": [
    "# graph.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (cvcdocdb)",
   "language": "python",
   "name": "cvcdocdb"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}