Examples

This page contains a few examples of how you can use HyFI-Fetcher package.

Installation

You can install HyFI-Fetcher using pip:

pip install hyfi-fetcher

Requirement already satisfied: hyfi-fetcher in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (0.2.2)
Requirement already satisfied: click<9.0.0,>=8.1.3 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi-fetcher) (8.1.7)
Requirement already satisfied: hyfi<2.0.0,>=1.36.3 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi-fetcher) (1.37.0)
Requirement already satisfied: langchain<0.4.0,>=0.1.16 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi-fetcher) (0.3.20)
Requirement already satisfied: openai<2.0.0,>=1.21.2 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi-fetcher) (1.65.2)
Requirement already satisfied: playwright<2.0.0,>=1.43.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi-fetcher) (1.50.0)
Requirement already satisfied: chardet<=5.2.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (5.2.0)
Requirement already satisfied: colorama<0.5.0,>=0.4.3 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (0.4.6)
Requirement already satisfied: datasets<=2.21.0,>=2.0.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2.21.0)
Requirement already satisfied: gdown<=5.2.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (5.2.0)
Requirement already satisfied: hydra-colorlog<2.0.0,>=1.2.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (1.2.0)
Requirement already satisfied: hydra-core<2.0.0,>=1.3.2 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (1.3.2)
Requirement already satisfied: joblib<=1.4.2,>=1.2.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (1.4.2)
Requirement already satisfied: matplotlib<=3.9.2 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (3.9.2)
Requirement already satisfied: numpy<=2.0.2,>=1.26.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2.0.2)
Requirement already satisfied: pandas<=2.2.2,>=1.5.3 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2.2.2)
Requirement already satisfied: pathspec>=0.9.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (0.12.1)
Requirement already satisfied: pyarrow<=17.0.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (17.0.0)
Requirement already satisfied: pydantic<3.0.0,>=2.3.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2.10.6)
Requirement already satisfied: pydantic-settings<3.0.0,>=2.0.1 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2.8.1)
Requirement already satisfied: python-dotenv<2.0.0,>=1.0.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (1.0.1)
Requirement already satisfied: scipy<=1.13.1,>1.10.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (1.13.1)
Requirement already satisfied: tqdm<5.0.0,>=4.64.1 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (4.67.1)

Requirement already satisfied: langchain-core<1.0.0,>=0.3.41 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from langchain<0.4.0,>=0.1.16->hyfi-fetcher) (0.3.41)
Requirement already satisfied: langchain-text-splitters<1.0.0,>=0.3.6 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from langchain<0.4.0,>=0.1.16->hyfi-fetcher) (0.3.6)
Requirement already satisfied: langsmith<0.4,>=0.1.17 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from langchain<0.4.0,>=0.1.16->hyfi-fetcher) (0.3.11)
Requirement already satisfied: SQLAlchemy<3,>=1.4 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from langchain<0.4.0,>=0.1.16->hyfi-fetcher) (2.0.38)
Requirement already satisfied: requests<3,>=2 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from langchain<0.4.0,>=0.1.16->hyfi-fetcher) (2.32.3)
Requirement already satisfied: PyYAML>=5.3 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from langchain<0.4.0,>=0.1.16->hyfi-fetcher) (6.0.2)
Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from langchain<0.4.0,>=0.1.16->hyfi-fetcher) (4.0.3)
Requirement already satisfied: anyio<5,>=3.5.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from openai<2.0.0,>=1.21.2->hyfi-fetcher) (4.8.0)
Requirement already satisfied: distro<2,>=1.7.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from openai<2.0.0,>=1.21.2->hyfi-fetcher) (1.9.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from openai<2.0.0,>=1.21.2->hyfi-fetcher) (0.28.1)
Requirement already satisfied: jiter<1,>=0.4.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from openai<2.0.0,>=1.21.2->hyfi-fetcher) (0.8.2)
Requirement already satisfied: sniffio in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from openai<2.0.0,>=1.21.2->hyfi-fetcher) (1.3.1)
Requirement already satisfied: typing-extensions<5,>=4.11 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from openai<2.0.0,>=1.21.2->hyfi-fetcher) (4.12.2)
Requirement already satisfied: pyee<13,>=12 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from playwright<2.0.0,>=1.43.0->hyfi-fetcher) (12.1.1)
Requirement already satisfied: greenlet<4.0.0,>=3.1.1 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from playwright<2.0.0,>=1.43.0->hyfi-fetcher) (3.1.1)
Requirement already satisfied: exceptiongroup>=1.0.2 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from anyio<5,>=3.5.0->openai<2.0.0,>=1.21.2->hyfi-fetcher) (1.2.2)
Requirement already satisfied: idna>=2.8 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from anyio<5,>=3.5.0->openai<2.0.0,>=1.21.2->hyfi-fetcher) (3.10)

Requirement already satisfied: filelock in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (3.17.0)
Requirement already satisfied: dill<0.3.9,>=0.3.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (0.3.8)
Requirement already satisfied: xxhash in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (3.5.0)
Requirement already satisfied: multiprocess in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (0.70.16)
Requirement already satisfied: fsspec<=2024.6.1,>=2023.1.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from fsspec[http]<=2024.6.1,>=2023.1.0->datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2024.6.1)
Requirement already satisfied: aiohttp in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (3.11.13)
Requirement already satisfied: huggingface-hub>=0.21.2 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (0.29.1)
Requirement already satisfied: packaging in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (24.2)
Requirement already satisfied: beautifulsoup4 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from gdown<=5.2.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (4.13.3)
Requirement already satisfied: certifi in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from httpx<1,>=0.23.0->openai<2.0.0,>=1.21.2->hyfi-fetcher) (2025.1.31)
Requirement already satisfied: httpcore==1.* in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from httpx<1,>=0.23.0->openai<2.0.0,>=1.21.2->hyfi-fetcher) (1.0.7)
Requirement already satisfied: h11<0.15,>=0.13 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai<2.0.0,>=1.21.2->hyfi-fetcher) (0.14.0)

Requirement already satisfied: colorlog in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hydra-colorlog<2.0.0,>=1.2.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (6.9.0)
Requirement already satisfied: omegaconf<2.4,>=2.2 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hydra-core<2.0.0,>=1.3.2->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2.3.0)
Requirement already satisfied: antlr4-python3-runtime==4.9.* in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from hydra-core<2.0.0,>=1.3.2->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (4.9.3)
Requirement already satisfied: tenacity!=8.4.0,<10.0.0,>=8.1.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from langchain-core<1.0.0,>=0.3.41->langchain<0.4.0,>=0.1.16->hyfi-fetcher) (9.0.0)
Requirement already satisfied: jsonpatch<2.0,>=1.33 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from langchain-core<1.0.0,>=0.3.41->langchain<0.4.0,>=0.1.16->hyfi-fetcher) (1.33)
Requirement already satisfied: orjson<4.0.0,>=3.9.14 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from langsmith<0.4,>=0.1.17->langchain<0.4.0,>=0.1.16->hyfi-fetcher) (3.10.15)
Requirement already satisfied: requests-toolbelt<2.0.0,>=1.0.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from langsmith<0.4,>=0.1.17->langchain<0.4.0,>=0.1.16->hyfi-fetcher) (1.0.0)
Requirement already satisfied: zstandard<0.24.0,>=0.23.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from langsmith<0.4,>=0.1.17->langchain<0.4.0,>=0.1.16->hyfi-fetcher) (0.23.0)
Requirement already satisfied: contourpy>=1.0.1 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from matplotlib<=3.9.2->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (1.3.0)
Requirement already satisfied: cycler>=0.10 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from matplotlib<=3.9.2->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from matplotlib<=3.9.2->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (4.56.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from matplotlib<=3.9.2->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (1.4.7)
Requirement already satisfied: pillow>=8 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from matplotlib<=3.9.2->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (11.1.0)
Requirement already satisfied: pyparsing>=2.3.1 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from matplotlib<=3.9.2->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (3.2.1)
Requirement already satisfied: python-dateutil>=2.7 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from matplotlib<=3.9.2->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2.9.0.post0)
Requirement already satisfied: importlib-resources>=3.2.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from matplotlib<=3.9.2->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (6.5.2)

Requirement already satisfied: pytz>=2020.1 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from pandas<=2.2.2,>=1.5.3->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2025.1)
Requirement already satisfied: tzdata>=2022.7 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from pandas<=2.2.2,>=1.5.3->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2025.1)
Requirement already satisfied: annotated-types>=0.6.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from pydantic<3.0.0,>=2.3.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (0.7.0)
Requirement already satisfied: pydantic-core==2.27.2 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from pydantic<3.0.0,>=2.3.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2.27.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from requests<3,>=2->langchain<0.4.0,>=0.1.16->hyfi-fetcher) (3.4.1)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from requests<3,>=2->langchain<0.4.0,>=0.1.16->hyfi-fetcher) (2.3.0)

Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from aiohttp->datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2.4.8)
Requirement already satisfied: aiosignal>=1.1.2 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from aiohttp->datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (1.3.2)
Requirement already satisfied: attrs>=17.3.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from aiohttp->datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (25.1.0)
Requirement already satisfied: frozenlist>=1.1.1 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from aiohttp->datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (1.5.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from aiohttp->datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (6.1.0)
Requirement already satisfied: propcache>=0.2.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from aiohttp->datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (0.3.0)
Requirement already satisfied: yarl<2.0,>=1.17.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from aiohttp->datasets<=2.21.0,>=2.0.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (1.18.3)

Requirement already satisfied: zipp>=3.1.0 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from importlib-resources>=3.2.0->matplotlib<=3.9.2->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (3.21.0)
Requirement already satisfied: jsonpointer>=1.9 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from jsonpatch<2.0,>=1.33->langchain-core<1.0.0,>=0.3.41->langchain<0.4.0,>=0.1.16->hyfi-fetcher) (3.0.0)
Requirement already satisfied: six>=1.5 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib<=3.9.2->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (1.17.0)

Requirement already satisfied: soupsieve>1.2 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from beautifulsoup4->gdown<=5.2.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (2.6)
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages (from requests[socks]->gdown<=5.2.0->hyfi<2.0.0,>=1.36.3->hyfi-fetcher) (1.7.1)

Note: you may need to restart the kernel to use updated packages.

Usage

from hyfetcher import get_version

print(get_version())