Uplift pipeline example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c3193d4b-0fcb-4efd-ac0d-573117eb21be",
   "metadata": {},
   "source": [
    "# Prerequisites \n",
    "\n",
    "To run this pipeline, you need the `ogc-na` Python module to be installed in your environment (for example, by running `pip install ogc-na`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e17ef49-98ad-4f3a-a989-7401fe233dd8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from ogc.na import download, ingest_json, update_vocabs\n",
    "from ogc.na.domain_config import DomainConfiguration\n",
    "import json\n",
    "from rdflib import Graph"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1dee44d-44d9-4cad-82dd-368ab5ffb787",
   "metadata": {},
   "source": [
    "# Running the pipeline\n",
    "\n",
    "This section shows a step-by-step usage of the OGC Rainbow data download + semantic uplift + entailment + validation pipeline.\n",
    "work\n",
    "We will run all the steps in the current directory. Outputs (downloads, generated files, etc.) will be saved to the `work` directory.\n",
    "\n",
    "## Download Google Spreadsheet as CSV\n",
    "\n",
    "Google Spreadsheets can be downloaded by using a specially-crafted URL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73804259-802e-4cd7-b6a0-1c6d31442488",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "GS_ID = '1zOGLWpTr784nTzBO-S_Es_WUyRsK7650'\n",
    "GS_URL = f\"https://docs.google.com/spreadsheets/d/{GS_ID}/export?format=csv\"\n",
    "CSV_DEST = 'work/iso19156-3-examples.csv'\n",
    "download.download_file(GS_URL, CSV_DEST, object_diff=False)\n",
    "print(\"Downloaded\", GS_URL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4ba2d1e-0d42-4a41-bc21-e2e6df35ae9b",
   "metadata": {},
   "source": [
    "The `object_diff` key is used when working with JSON files, so in this case we need to disable it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb318df8-e863-4a89-9c85-69585c8e991c",
   "metadata": {},
   "source": [
    "## Convert CSV to JSON\n",
    "\n",
    "Once we have our [iso19156-3-examples.csv](work/iso19156-3-examples.csv) file, we need to convert it to JSON.\n",
    "\n",
    "`ingest_json`, the module [used to perform semantic uplifts](https://opengeospatial.github.io/ogc-na-tools/reference/ogc/na/ingest_json/) (turning plain JSON into JSON-LD and/or RDF/Turtle), has an [input filter that can work with CSV files](https://opengeospatial.github.io/ogc-na-tools/reference/ogc/na/input_filters/csv/). Normally, we would create a whole semantic uplift definition (following the steps in [this tutorial](https://opengeospatial.github.io/ogc-na-tools/tutorials/#how-to-create-a-json-ld-uplift-context-definition)), but in this case we just want a JSON version of our spreadsheet, so we can use an extremely simple uplift definition:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecaead9f-b179-4448-80aa-2cf71713e085",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "csv_to_json_def = '''\n",
    "input-filter:\n",
    "  csv:\n",
    "'''\n",
    "with open('work/csv_to_json.yml', 'w') as f:\n",
    "    f.write(csv_to_json_def)\n",
    "print('work/csv_to_json.yml created')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "747a9938-a61f-4df4-a6e9-30e20e249dd8",
   "metadata": {},
   "source": [
    "Then we can run the `ingest_json` module with [that definition](work/csv_to_json.yml) to obtain our JSON document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8e49c7f-94a3-4053-8e00-1bd7bfb5c359",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result = ingest_json.process_file(input_fn='work/iso19156-3-examples.csv',\n",
    "                                  jsonld_fn='work/iso19156-3-examples.csv.json',\n",
    "                                  context_fn='work/csv_to_json.yml')\n",
    "print('iso19156-3-examples.csv.json created')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a356f40-9fd8-4407-8db3-3485805073f9",
   "metadata": {},
   "source": [
    "As you can see, the newly-created [iso19156-3-examples.csv.json](work/iso19156-3-examples.csv.json) document contains an object with two keys:\n",
    "\n",
    "* `metadata`, which contains metadata about the input file, the filter that was used, etc.\n",
    "* `data`, with an array of objects representing the rows in the spreadsheet.\n",
    "\n",
    "## Semantic uplift\n",
    "\n",
    "Once our data is in JSON format, we can perform a semantic uplift on it, which is basically converting plain old JSON into JSON-LD and/or RDF in Turtle format.\n",
    "\n",
    "For this step, we will use [an already existing uplift definition](https://raw.githubusercontent.com/avillar/iso19157-3-sample/master/properties-uplift.yml):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "426b1526-9d83-452a-b4ca-d7d8c296a96a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "download.download_file('https://raw.githubusercontent.com/avillar/iso19157-3-sample/master/properties-uplift.yml',\n",
    "                       'work/properties-uplift.yml',\n",
    "                       object_diff=False)\n",
    "print('properties-uplift.yml downloaded')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79df3e71-79ef-41d3-a321-2fad6eb6512b",
   "metadata": {},
   "source": [
    "Let us review what [this uplift definition](work/properties-uplift.yml) does:\n",
    "\n",
    "1. It contains 4 `transform`s ([jq](https://stedolan.github.io/jq/) expressions to manipulate the input document):\n",
    "    1. We take the value of the `data` key and discard the rest.\n",
    "    2. We walk through the data tree and remove (set to `null`) all values that empty strings, strings made up of blank space characters only, or strings that are just a dash (`-`).\n",
    "    3. Since some headers contained colons, we remove all colons inside property keys in the object.\n",
    "    4. We add a `skos:ConceptScheme` with some metadata as the top-level object, and put all of the row objects inside its `concepts` property.\n",
    "2. After the `transform`s run, we add the `skos:Concept` type for all of the rows (remember that they are now inside the `concepts` property of the top-level object).\n",
    "3. Finally, we add the JSON-LD context at the root level of the document (indicated by using the `$` JSON path).\n",
    "\n",
    "We run it like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47cc6af7-0f99-495a-a691-948e10afa9bd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result = ingest_json.process_file(input_fn='work/iso19156-3-examples.csv.json',\n",
    "                                  jsonld_fn='work/iso19156-3-examples.csv.jsonld',\n",
    "                                  ttl_fn='work/iso19156-3-examples.csv.ttl',\n",
    "                                  context_fn='work/properties-uplift.yml')\n",
    "print('Semantic uplift done')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58d9c997-10d7-4d90-9d54-6c8ee346168e",
   "metadata": {},
   "source": [
    "The above command will generate two files:\n",
    "\n",
    "* [iso19156-3-examples.csv.jsonld](work/iso19156-3-examples.csv.jsonld), which is the uplifted JSON-LD version after running the `transform`s, setting the `types` and adding the `context`.\n",
    "* [iso19156-3-examples.csv.ttl](work/iso19156-3-examples.csv.ttl), its equivalent version in Turtle.\n",
    "\n",
    "You will note that the Turtle version does not have all of the properties that the JSON-LD document does; this is because, from the RDF side of things, any property that is not linked to an RDF predicate is simply ignored.\n",
    "\n",
    "## Entailment and validation\n",
    "\n",
    "At this point, we have an RDF graph version of our initial spreadsheet inside the [iso19156-3-examples.csv.ttl](work/iso19156-3-examples.csv.ttl) file. The next step involves performing entailment (inferring new data that we can add to our graph) and validation (verifying that our data is \"up to code\"). We can do this by leveraging a couple of technologies:\n",
    "\n",
    "* [SHACL](https://www.w3.org/TR/shacl/) allows us to write entailment rules and validation constraints for RDF data.\n",
    "* [The profiles vocabulary](https://www.w3.org/TR/dx-prof/) can be used to create profiles, and link those with entailment and validation resources.\n",
    "  * The OGC defines several profiles that can be used for validation and entailment of RDF resources.\n",
    "\n",
    "To associate our own Turtle files with the profiles, we first need to create a [DomainConfiguration](https://opengeospatial.github.io/ogc-na-tools/reference/ogc/na/domain_config/#ogc.na.domain_config.DomainConfiguration) (you can find a full example with comments [here](https://opengeospatial.github.io/ogc-na-tools/examples/#sample-domain-configuration)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c91cd9e9-82f6-49ae-a41f-8be5d12914bf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "domain_cfg_content = '''\n",
    "@prefix dcfg: <http://www.example.org/ogc/domain-cfg#> .\n",
    "@prefix dcat: <http://www.w3.org/ns/dcat#> .\n",
    "@prefix dct: <http://purl.org/dc/terms/> .\n",
    "@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n",
    "@prefix profiles: <http://www.opengis.net/def/metamodel/profiles/> .\n",
    "\n",
    "_:iso19157-3-sample a dcat:Catalog ;\n",
    "  dct:title \"ISO19157-3 Sample\" ;\n",
    "  dcat:dataset _:examples ;\n",
    "  dcfg:hasProfileSource \"sparql:http://defs-dev.opengis.net/fuseki/query\" ;\n",
    "  dcfg:ignoreProfileArtifactErrors true ;\n",
    ".\n",
    "\n",
    "_:examples a dcat:Dataset, dcfg:DomainConfiguration ;\n",
    "  dct:identifier \"examples\" ;\n",
    "  dct:description \"Entailment and validation for examples\" ;\n",
    "  dcfg:glob \"work/*.ttl\" ;\n",
    "  dct:conformsTo profiles:skos_shared, profiles:skos_conceptscheme, profiles:skos_conceptscheme_ogc, profiles:vocprez_ogc ;\n",
    ".\n",
    "'''\n",
    "domain_cfg = DomainConfiguration(Graph().parse(data=domain_cfg_content))\n",
    "profile_registry = domain_cfg.profile_registry\n",
    "print('Found profiles:')\n",
    "print('\\n'.join(str(profile_uri) for profile_uri in sorted(profile_registry.profiles)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0eeecdb9-bcd8-480f-bc74-c642d8214015",
   "metadata": {},
   "source": [
    "The `profile_registry` above is a [ProfileRegistry](https://opengeospatial.github.io/ogc-na-tools/reference/ogc/na/profile/#ogc.na.profile.ProfileRegistry), which reads the values for the catalog's `dcfg:hasProfileSource` (`sparql:http://defs-dev.opengis.net:8080/rdf4j-server/repositories/profiles`, a SPARQL endpoint, in our case), and obtains all the profile definitions, including its resources, artifacts and dependencies, from them.\n",
    "\n",
    "Apart from that, we define a `dcfg:DomainConfiguration` to run entailment and validation processes on our Turtle document (which will fall inside of the `dcfg:glob`'s scope) with resources from 4 profiles: `skos_shared`, `skos_conceptscheme`, `skos_conceptscheme_ogc` and `vocprez_ogc`.\n",
    "\n",
    "Since the previous uplift `result` has a `graph` property with the RDF, we can run entailments directly on it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7bbb3ed-e844-44eb-b81a-6c44fdef13c2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cfg_entry = domain_cfg.entries.find_entry_for_file('work/iso19156-3-examples.csv.ttl')\n",
    "entailed_graph, entail_artifacts = profile_registry.entail(result.graph, cfg_entry.conforms_to, inplace=False)\n",
    "entailed_graph.serialize('work/iso19156-3-examples.csv-entailed.ttl', format='ttl')\n",
    "print('Entailement done. Artifacts used:')\n",
    "print('-','\\n- '.join(entail_artifacts))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cb0ac61-9d86-4246-9ed0-b962168a91ca",
   "metadata": {},
   "source": [
    "The above code will find the profiles defined for the [iso19156-3-examples.csv.ttl](work/iso19156-3-examples.csv.ttl) file name, and then run the specific entailments for the profiles found in the configuration entry's `dct:conformsTo`. The entailment result (`entailed_graph`) is stored, and a list of all the found entailment artifacts is then written to the console.\n",
    "\n",
    "It is important to note that in our example we could have skipped creating the full `DomainConfiguration`, instantiating a `ProfileRegistry` directly with the profiles SPARQL endpoint instead, but as we will see later, it is much easier to work with the former in an automated CI/CD environment.\n",
    "\n",
    "Validation is done similarly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3dfba945-93c1-411d-8f90-422de2fcb4fc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "validation_result = profile_registry.validate(entailed_graph, cfg_entry.conforms_to, log_artifact_errors=True)\n",
    "print('Validation done')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cc7d991-0344-44e7-92eb-3330a260fd5c",
   "metadata": {},
   "source": [
    "The validation process output logging warnings when profiles and/or artifacts are missing, but if `log_artifact_errors` is `True`, it will take a *best-effort* approach.\n",
    "\n",
    "The `validation_result` will then contain a summary of all the validation errors found, both globally and per profile:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a78d6a9a-c0c1-460b-9222-e5d409bf8ff2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Global:\", validation_result.result)\n",
    "for profile_report in validation_result.reports:\n",
    "    print(f\"{profile_report.profile_uri}: {profile_report.report.result}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75843c0c-682c-41c6-969a-053229da83e0",
   "metadata": {},
   "source": [
    "We also get full plain text error reports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67521800-e8b1-4cfe-acd9-7d84b2d4e1f5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(validation_result.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2f5603b-16c4-4717-9d54-e34d4d23996b",
   "metadata": {
    "tags": []
   },
   "source": [
    "As well as an RDF representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d8dd741-d216-4134-886a-d1b01a0783cd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(validation_result.graph.serialize(format='ttl'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73a317ea-467a-4cc0-b732-395af595d413",
   "metadata": {},
   "source": [
    "In our example, we can see that the validation for `skos_shared` fails because the SHACL resource employed has errors, but `vocprez_ogc` detects several errors in our data.\n",
    "\n",
    "## Uploading the results\n",
    "\n",
    "Finally, if we want to upload our entailed graphs to a [SPARQL Graph Store Protocol](https://www.w3.org/TR/sparql11-http-rdf-update/)-compatible service, we can use `update_vocabs.load_vocab` to do so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50511dc7-ccea-404f-9bb0-1a27c087c1ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "if False: # disable upload in this notebook\n",
    "    update_vocabs.load_vocab(result.graph, 'http://example.com/graph-identifier', 'http://example.com/sparql/graph-store', ('username', 'password'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e890bc4-e6e9-495a-93a2-7c97f6f3f3d6",
   "metadata": {},
   "source": [
    "The RDF data will be uploading using the [HTTP PUT method mechanism](https://www.w3.org/TR/sparql11-http-rdf-update/#http-put), which will replace all data in the specified graph URI (`http://example.com/graph-identifier` above) with the contents of the provided graph."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf7d9758-951e-4523-9ac9-8316249557ef",
   "metadata": {},
   "source": [
    "# CI/CD environments\n",
    "\n",
    "All of the ogc-na modules can be run directly from the command line, which means that in a CI/CD environment all of them can be run as shell commands.\n",
    "\n",
    "Take the [iso19157-3-sample](https://github.com/avillar/iso19157-3-sample) GitHub repository as an example. It contains several files that should be quite familiar to us by now:\n",
    "\n",
    "* [csv2python.yml](https://github.com/avillar/iso19157-3-sample/blob/master/csv2python.yml) is the uplift definition to turn CSV files into JSON.\n",
    "* [properties-uplift.yml](https://github.com/avillar/iso19157-3-sample/blob/master/properties-uplift.yml) is the uplift definition that converts our JSON rows into JSON-LD/Turtle.\n",
    "* [.ogc/catalog.ttl](https://github.com/avillar/iso19157-3-sample/blob/master/.ogc/catalog.ttl) contains a `DomainConfiguration` like the one we created before, as well as an `UplifConfiguration` to map `*.csv.json` files to `properties-uplift.yml`.\n",
    "\n",
    "Apart from that, we have [.ogc/config.yml](https://github.com/avillar/iso19157-3-sample/blob/master/.ogc/config.yml) with the file download configuration (which we typed in manually above).\n",
    "\n",
    "The pipeline is then run using the [download-and-uplift.yaml](https://github.com/avillar/iso19157-3-sample/blob/master/.github/workflows/download-and-uplift.yaml) GitHub workflow configuration; the important bits in that file are the chain of `python` commands that appear after the environment is set up:\n",
    "\n",
    "```shell\n",
    "# Download file(s)\n",
    "# Inputs:\n",
    "#  - .ogc/config.yml (file download spec)\n",
    "# Outputs:\n",
    "#  - iso19156-3-examples.csv\n",
    "python -m ogc.na.download --spec .ogc/config.yml\n",
    "\n",
    "# Search for CSV files and convert them to JSON\n",
    "# Inputs:\n",
    "#  - iso19156-3-examples.csv (input file)\n",
    "#  - csv2python.yml (uplift definition)\n",
    "# Outputs:\n",
    "#  - iso19156-3-examples.csv.json\n",
    "find . -name '*.csv' | while read CSV_FILE; do python -m ogc.na.ingest_json \\\n",
    "--skip-on-missing-context --json-ld --context csv2python.yml \"${CSV_FILE}\" > \"${CSV_FILE}.json\"\n",
    "done\n",
    "\n",
    "# Do the properties-uplift.yml semantic uplift. The --use-git-status instructs the script to\n",
    "# take its input file names from whatever is modified/added in the current git working directory\n",
    "# Inputs:\n",
    "#  - .ogc/catalog.ttl (domain configurations)\n",
    "#  - iso19156-3-examples.csv.json (input file)\n",
    "#  - properties-uplift.yml (uplift definition found in catalog)\n",
    "# Outputs:\n",
    "#  - iso19156-3-examples.csv.jsonld (uplifted JSON-LD)\n",
    "#  - iso19156-3-examples.csv.ttl (uplifted Turtle)\n",
    "python -m ogc.na.ingest_json --batch --use-git-status --skip-on-missing-context \\\n",
    "--json-ld --ttl --work-dir . --domain-config .ogc/catalog.ttl\n",
    "\n",
    "# Run the entailment, validation and upload in a single step\n",
    "# Inputs:\n",
    "#  - .ogc/catalog.ttl (domain configurations)\n",
    "#  - iso19156-3-examples.csv.ttl (input file)\n",
    "# Outputs:\n",
    "#  - entailed/iso19156-3-examples.csv.jsonld (entailed expanded JSON-LD)\n",
    "#  - entailed/iso19156-3-examples.csv.ttl (entailed Turtle)\n",
    "#  - entailed/iso19156-3-examples.csv.rdf (entailed RDF/XML)\n",
    "#  - entailed/iso19156-3-examples.csv.txt (validation report)\n",
    "python -m ogc.na.update_vocabs -w . .ogc/catalog.ttl --use-git-status \\\n",
    "--base-uri https://raw.githubusercontent.com/${{github.repository}}/${{github.ref_name}} \\\n",
    "--update --graph-store http://defs-dev.opengis.net:8061/fuseki-hosted/data\n",
    "```\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}