Ingest HMDB - initial iteration

[RichardBruskiewich]

HMDB Phase 2 Ingest Survey

Ingest code PR: #325 based initially on the RENCI Automat ORION implementation of HMDB knowledge ingest.

Parallel Phase 2 ingest metadata and computing details for RTX-KG2 and for MolePro are provided below.

[RichardBruskiewich]

For convenience and preservation, I'm copying here Stephen Ramsey's feedback on RTX-KG2 Phase 2 ingest meta edge details.

[RichardBruskiewich]

843317 (biolink:SmallMolecule, biolink:related_to, biolink:Gene)
530057 (biolink:Pathway, biolink:has_participant, biolink:SmallMolecule)
70482 (biolink:SmallMolecule, biolink:located_in, biolink:CellularComponent)
54208 (biolink:SmallMolecule, biolink:located_in, biolink:AnatomicalEntity)
24633 (biolink:SmallMolecule, biolink:related_to, biolink:Disease)
2801 (biolink:SmallMolecule, biolink:same_as, biolink:SmallMolecule)
2050 (biolink:SmallMolecule, biolink:located_in, biolink:GrossAnatomicalStructure)
1426 (biolink:SmallMolecule, biolink:related_to, biolink:Protein)
492 (biolink:ChemicalEntity, biolink:same_as, biolink:SmallMolecule)
334 (biolink:SmallMolecule, biolink:located_in, biolink:Cell)
200 (biolink:ChemicalEntity, biolink:related_to, biolink:Gene)
190 (biolink:Pathway, biolink:has_participant, biolink:ChemicalEntity)
74 (biolink:MolecularMixture, biolink:related_to, biolink:Gene)
61 (biolink:ChemicalEntity, biolink:located_in, biolink:AnatomicalEntity)
37 (biolink:ChemicalEntity, biolink:located_in, biolink:CellularComponent)
36 (biolink:SmallMolecule, biolink:same_as, biolink:ChemicalEntity)
33 (biolink:MolecularMixture, biolink:same_as, biolink:SmallMolecule)
29 (biolink:MolecularMixture, biolink:located_in, biolink:CellularComponent)
27 (biolink:MolecularMixture, biolink:located_in, biolink:AnatomicalEntity)
18 (biolink:SmallMolecule, biolink:located_in, biolink:Publication)
14 (biolink:SmallMolecule, biolink:same_as, biolink:MolecularMixture)
14 (biolink:ChemicalEntity, biolink:related_to, biolink:Disease)
6 (biolink:ChemicalEntity, biolink:same_as, biolink:ChemicalEntity)
5 (biolink:MolecularMixture, biolink:same_as, biolink:MolecularMixture)
5 (biolink:MolecularMixture, biolink:related_to, biolink:Disease)
5 (biolink:ChemicalEntity, biolink:same_as, biolink:MolecularMixture)
5 (biolink:ChemicalEntity, biolink:related_to, biolink:Protein)
3 (biolink:ChemicalEntity, biolink:located_in, biolink:Cell)
2 (biolink:SmallMolecule, biolink:related_to, biolink:NamedThing)
2 (biolink:Pathway, biolink:has_participant, biolink:MolecularMixture)
2 (biolink:NamedThing, biolink:same_as, biolink:SmallMolecule)
2 (biolink:NamedThing, biolink:located_in, biolink:CellularComponent)
2 (biolink:ChemicalEntity, biolink:located_in, biolink:GrossAnatomicalStructure)
1 (biolink:NamedThing, biolink:located_in, biolink:AnatomicalEntity)
1 (biolink:MolecularMixture, biolink:related_to, biolink:Protein)
1 (biolink:MolecularMixture, biolink:located_in, biolink:Cell)

[RichardBruskiewich]

Stephen got this using:

cat kg2.10.3-conflated-edges.jsonl | grep infores:hmdb > hmdb-edges.jsonl

and then running this script:

./extract_hmdb_metatriples.py hmdb-edges.jsonl > hmdb-metatriples.txt

and then analyzing the metatriples like this:

sort hmdb-metatriples.txt | uniq -c | sort -k1 -nr > hmdb-metatriples-unique-counted.txt

He needed a python script that will process a JSON-lines file of edges like this: and for each edge, look up (using the SRI Node Normalizer, which is documented here, using the subject and object nodes to obtain those nodes' Biolink category information (specifically, the most specific category in the Biolink hierarchy). Then print out, for each row in the JSON lines file, the meta-triple: (subject_node_category, predicate, object_node_category)

python3.13 -m venv venv
source venv/bin/activate
pip install bmt

with the script:

#!/usr/bin/env python3

"""
Read a JSON-lines edge file and print:
(subject_category, predicate, object_category)

For each CURIE in subject/object, query the SRI Node Normalizer and pick the
most specific Biolink category using the Biolink hierarchy.

Usage:
    python edge_meta_triples.py input_edges.jsonl

Optional:
    python edge_meta_triples.py input_edges.jsonl --batch-size 200
"""

from __future__ import annotations

import argparse
import json
import sys
import time
from collections import defaultdict
from typing import Any, Dict, Iterable, List, Optional

import requests

try:
    # pip install biolink-model-toolkit
    from bmt import Toolkit
except ImportError:
    Toolkit = None  # type: ignore


NODE_NORMALIZER_URL = "https://nodenormalization-sri.renci.org/get_normalized_nodes"
REQUEST_TIMEOUT_SECONDS = 60
DEFAULT_BATCH_SIZE = 200
RETRY_COUNT = 3
RETRY_SLEEP_SECONDS = 2


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser()
    parser.add_argument("input_jsonl", help="Path to input JSON-lines file of edges")
    parser.add_argument(
        "--batch-size",
        type=int,
        default=DEFAULT_BATCH_SIZE,
        help=f"Number of CURIEs to send per Node Normalizer request (default: {DEFAULT_BATCH_SIZE})",
    )
    parser.add_argument(
        "--fallback-category",
        default="biolink:NamedThing",
        help="Category to use if no category can be resolved (default: biolink:NamedThing)",
    )
    return parser.parse_args()


def batched(items: List[str], batch_size: int) -> Iterable[List[str]]:
    for i in range(0, len(items), batch_size):
        yield items[i : i + batch_size]


def post_with_retries(
    session: requests.Session,
    url: str,
    payload: Dict[str, Any],
    retries: int = RETRY_COUNT,
) -> Dict[str, Any]:
    last_exc: Optional[Exception] = None
    for attempt in range(1, retries + 1):
        try:
            response = session.post(url, json=payload, timeout=REQUEST_TIMEOUT_SECONDS)
            response.raise_for_status()
            return response.json()
        except Exception as exc:
            last_exc = exc
            if attempt < retries:
                time.sleep(RETRY_SLEEP_SECONDS * attempt)
    raise RuntimeError(f"Request failed after {retries} attempts: {last_exc}") from last_exc


def get_toolkit() -> Optional[Any]:
    if Toolkit is None:
        return None
    try:
        return Toolkit()
    except Exception:
        return None


def normalize_category_name(category: str) -> str:
    """
    Accept either 'biolink:Gene' or 'Gene' and return 'Gene' for hierarchy calls.
    """
    if category.startswith("biolink:"):
        return category.split(":", 1)[1]
    return category


def re_prefix_category(category: str) -> str:
    if category.startswith("biolink:"):
        return category
    return f"biolink:{category}"


def choose_most_specific_category(
    categories: List[str],
    toolkit: Optional[Any],
    fallback_category: str,
) -> str:
    """
    Pick the most specific category from a list of Biolink categories.

    Strategy:
      1. If toolkit is available, compare categories by ancestry.
      2. Otherwise, fall back to the first category in the returned list.
    """
    if not categories:
        return fallback_category

    # Normalize categories for consistency.
    categories = [re_prefix_category(cat) for cat in categories]

    if toolkit is None or len(categories) == 1:
        return categories[0]

    # Convert to local names for BMT, e.g. biolink:Gene -> Gene
    local_names = [normalize_category_name(cat) for cat in categories]

    # A category is "more specific" if it is a descendant of another category in the set.
    more_general_than = defaultdict(set)  # cat -> set of categories it is more general than

    for i, cat_i in enumerate(local_names):
        for j, cat_j in enumerate(local_names):
            if i == j:
                continue
            try:
                ancestors_of_j = set(toolkit.get_ancestors(cat_j, formatted=False) or [])
            except Exception:
                ancestors_of_j = set()
            if cat_i in ancestors_of_j:
                more_general_than[cat_i].add(cat_j)

    # Most specific categories are those that are not ancestors of any other category in the set.
    most_specific = [c for c in local_names if len(more_general_than[c]) == 0]

    if not most_specific:
        return categories[0]

    # If multiple remain, pick the first one that appeared in the original list.
    chosen_local = most_specific[0]
    return re_prefix_category(chosen_local)


def fetch_categories_for_curies(
    curies: List[str],
    session: requests.Session,
    batch_size: int,
    toolkit: Optional[Any],
    fallback_category: str,
) -> Dict[str, str]:
    """
    Return a mapping CURIE -> chosen most-specific category.
    """
    result: Dict[str, str] = {}

    for curie_batch in batched(curies, batch_size):
        payload = {
            "curies": curie_batch,
            "conflate": True,
            "description": False,
        }
        data = post_with_retries(session, NODE_NORMALIZER_URL, payload)

        for curie in curie_batch:
            node_info = data.get(curie)
            if not node_info:
                result[curie] = fallback_category
                continue

            categories = node_info.get("type") or node_info.get("categories") or []
            if not isinstance(categories, list):
                categories = []

            result[curie] = choose_most_specific_category(
                categories=categories,
                toolkit=toolkit,
                fallback_category=fallback_category,
            )

    return result


def main() -> int:
    args = parse_args()
    toolkit = get_toolkit()

    rows: List[Dict[str, Any]] = []
    curies_needed: List[str] = []

    with open(args.input_jsonl, "r", encoding="utf-8") as infile:
        for line_number, line in enumerate(infile, start=1):
            line = line.strip()
            if not line:
                continue

            try:
                row = json.loads(line)
            except json.JSONDecodeError as exc:
                print(
                    f"Skipping line {line_number}: invalid JSON: {exc}",
                    file=sys.stderr,
                )
                continue

            subject = row.get("subject")
            predicate = row.get("predicate")
            obj = row.get("object")

            if not subject or not predicate or not obj:
                print(
                    f"Skipping line {line_number}: missing one of subject/predicate/object",
                    file=sys.stderr,
                )
                continue

            rows.append(row)
            curies_needed.extend([subject, obj])

    unique_curies = sorted(set(curies_needed))

    with requests.Session() as session:
        category_map = fetch_categories_for_curies(
            curies=unique_curies,
            session=session,
            batch_size=args.batch_size,
            toolkit=toolkit,
            fallback_category=args.fallback_category,
        )

    for row in rows:
        subject = row["subject"]
        predicate = row["predicate"]
        obj = row["object"]

        subject_category = category_map.get(subject, args.fallback_category)
        object_category = category_map.get(obj, args.fallback_category)

        print(f"({subject_category}, {predicate}, {object_category})")

    return 0


if __name__ == "__main__":
    raise SystemExit(main())

In order to understand how the RTX-KG2 build system can ultimately produce those metatriples from the HMDB ingest, you will need this excerpt from the file RTX-KG2/maps/predicate-remap.yaml:

HMDB:at_cellular_location:
  operation: keep
  core_predicate: biolink:located_in
HMDB:at_tissue:
  operation: keep
  core_predicate: biolink:located_in
HMDB:disease:
  operation: keep
  core_predicate: biolink:related_to
HMDB:has_protein_association:
  operation: keep
  core_predicate: biolink:related_to
HMDB:in_biospecimen:
  operation: keep
  core_predicate: biolink:located_in
HMDB:in_pathway:
  operation: invert
  core_predicate: biolink:has_participant

That section of the YAML file governs how original HMDBL source relations are "remapped" (to Biolink predicates) by this script:

RTX-KG2/process/filter_kg_and_remap_predicates.py

[RichardBruskiewich]

The MolePro wrapper for HMDB data (coded by @vdancik) has the following few information resources:

• (MolePro) Attribute Map

Python

• HMDB knowledge publication "transformer" script

SQL

• HMDB SQL database data model
• HMDB database loading code

Java code

• MoleProAPI Java Server