lilfetch-mcp

jphdevsf/lilfetch-mcp

3.2

If you are the rightful owner of lilfetch-mcp and would like to certify it and/or have it hosted online, please leave a comment on the right or send an email to dayong@mcphub.com.

lilFetch is an MCP server designed to scrape webpages using crawl4ai and Playwright, providing structured Markdown outputs.

Tools
1
Resources
0
Prompts
0

lilFetch

An MCP (Model Context Protocol) server that scrapes webpages using crawl4ai and Playwright for more robust scraping. Tested on CoPilot in VSCode but may work for others.

Features

  • Enables html and/or text scraping of one or more urls directly in your chat prompt. Use the scraped response in followup queries for market research summarization, context for new file creation, etc.
  • Leverages Playwright and a headless instance of Chromium to load JS heavy sites and web apps where basic #fetch and curl commands fall short.
  • Strong focus on minimal commands and configuration to install and get scraping.

Prerequisites

Before installing, ensure:

  • Node.js 14+: Download from nodejs.org or via Homebrew (brew install node on macOS).
  • Python 3.10+: Auto-detected during setup. Install from python.org or Homebrew (brew install python@3.12 on macOS). If using pyenv, set a 3.10+ version active (pyenv global 3.12.0).
  • First Run Time: Setup downloads ~200MB (Playwright browsers) and takes 1-2 minutes.

Installation

Install globally for use across workspaces, or restrict to local installation if you just want to test in this repo or enhance it further.

Option 1: Global Install

For running npx lilfetch from any directory (portable CLI).

  1. Clone the Repo

    git clone https://github.com/jphdevsf/lilfetch-mcp.git lilfetch-mcp
cd lilfetch-mcp

  2. Install Globally

    npm run global-install
    • Sets up Python venv in ~/.lilfetch-venv (user-wide).

  3. Configure in Any VS Code Workspace (add to .vscode/mcp.json or global MCP settings):

    {
  "servers": {
    "lilFetch": {
      "type": "stdio",
      "command": "npx",
      "args": ["lilfetch"]
    }
  }
}

  4. Test It

    • In new terminal window, run npx lilfetch to start MCP server.
    • In VS Code, prompt with something like...
    Use lilFetch to scrape top news headlines from www.cnn.com and write to a markdown file in root of my repo.

Option 2: Local Install

For testing/extending in the repo.

  1. Clone the Repo

    git clone https://github.com/jphdevsf/lilfetch-mcp.git lilfetch-mcp
cd lilfetch-mcp

  2. Install Locally

    npm install
    • Sets up ./node_modules/lilfetch/ and .bin/lilfetch.
    • Python venv in repo .venv (local to this project).

  3. MCP.json Workspace Configuration Navigate to .vscode/mcp.json (create if missing) and add:

    {
  "servers": {
    "lilFetch": {
      "type": "stdio",
      "command": "node",
      "args": ["bin/lilfetch.js"]
    }
  }
}

    Note: Ensure bin/lilfetch.js is executable: Run chmod +x bin/lilfetch.js in the terminal.

  4. Test It

    • In new terminal window, navigate to this repo and run npm run dev or ./node_modules/.bin/lilfetch.
    • In VS Code, prompt with something like...
    Use lilFetch to scrape top news headlines from www.cnn.com and write to a markdown file in root of my repo.

Uninstallation

To fully remove a global installation (including the npm package and Python virtual environment):

Global Uninstall

From the repo directory (or anywhere):

npm run global-uninstall
  • This runs npm uninstall -g lilfetch to remove the global npm package and binary.
  • Followed by rm -rf ~/.lilfetch-venv to delete the user-wide Python venv (including installed deps and Playwright browsers).
  • Warning: The rm -rf command is irreversible. It only affects the .lilfetch-venv directory in your home folder. Back up if needed (unlikely).

Manual Uninstall (Alternative)

  1. Remove npm package: 
    npm uninstall -g lilfetch
  2. Remove Python venv: 
    rm -rf ~/.lilfetch-venv
    • On Windows: rmdir /s /q %USERPROFILE%\.lilfetch-venv

Local Uninstall

For local installs (e.g., after npm install):

npm uninstall
rm -rf .venv
  • This removes the local Node modules and repo-specific Python venv.

Verification

  • npm list -g --depth=0 (no lilfetch).
  • which lilfetch (empty).
  • ls ~/.lilfetch-venv (no such file).

For local: rm -rf node_modules .venv and verify no ./.bin/lilfetch.

Development

  • Edit mcp_server.py for Python logic.
  • Update bin/lilfetch.js for wrapper changes.
  • Bump version in package.json, then npm run pack.
  • For global testing: npm install -g . then npx lilfetch.

Troubleshooting

  • Permission Errors (Global Install): See Prerequisites for user-owned NPM setup. Avoid sudo—use the config steps.
  • Python Not Found/Version Error: Ensure Python 3.8+ is in PATH. For pyenv: pyenv install 3.12.0 && pyenv global 3.12.0, then re-run install. Check: python3 --version.
  • Venv/Deps Fail: For local: Delete .venv and re-run npm install. For global: Delete ~/.lilfetch-venv and re-run npm install -g .. Manual fix (local): python3 -m venv .venv && .venv/bin/pip install -r requirements.txt && .venv/bin/python -m playwright install. Manual fix (global): python3 -m venv ~/.lilfetch-venv && ~/.lilfetch-venv/bin/pip install -r requirements.txt && ~/.lilfetch-venv/bin/python -m playwright install.
  • Playwright Browsers Missing: Run python -m playwright install in the venv (or manually as logged).
  • MCP Not Detected in VS Code: Restart VS Code after config; ensure workspace is open correctly.
  • Uninstall:
    • Global: npm uninstall -g lilfetch + rm -rf ~/.lilfetch-venv.
    • Local: rm -rf node_modules package-lock.json .venv.

૮₍ • ᴥ • ₎ა

License: MIT