zcaceres/markdownify-mcp

AI Evaluation Report

Total Score: 6/10

The agent demonstrates strong capabilities in converting webpages and PDF files into Markdown format, accurately capturing and formatting the content. However, it struggles with converting DOCX, YouTube videos, and PPTX files due to tool malfunctions and limitations, such as requiring local paths or encountering buffer size issues. The agent's performance is inconsistent, excelling in some areas while failing in others, indicating a need for improvements in handling various file types and online resources.

• Test case 1

  Score: 10/10

  Perform the operation of converting the webpage at https://arxiv.org/abs/2506.11180 into Markdown format.

  Beyond Formal Semantics for Capabilities and Skills: Model Context Protocol in Manufacturing

  arXiv:2506.11180 (cs)
  Authors: Luis Miguel Vieira da Silva, Aljosha Köcher, Felix Gehlhoff
  Submitted on: 12 Jun 2025

  Abstract

  Explicit modeling of capabilities and skills -- whether based on ontologies, Asset Administration Shells, or other technologies -- requires considerable manual effort and often results in representations that are not easily accessible to Large Language Models (LLMs). In this work-in-progress paper, we present an alternative approach based on the recently introduced Model Context Protocol (MCP). MCP allows systems to expose functionality through a standardized interface that is directly consumable by LLM-based agents. We conduct a prototypical evaluation on a laboratory-scale manufacturing system, where resource functions are made available via MCP. A general-purpose LLM is then tasked with planning and executing a multi-step process, including constraint handling and the invocation of resource functions via MCP. The results indicate that such an approach can enable flexible industrial automation without relying on explicit semantic models. This work lays the basis for further exploration of external tool integration in LLM-driven production systems.

  Subjects

  • Software Engineering (cs.SE)
  • Artificial Intelligence (cs.AI)
  • Emerging Technologies (cs.ET)
  • Systems and Control (eess.SY)

  Cite as

  arXiv:2506.11180 [cs.SE] (or arXiv:2506.11180v1 for this version)
  https://doi.org/10.48550/arXiv.2506.11180

  Submission history

  From: Aljosha Köcher []
  [v1] Thu, 12 Jun 2025 13:02:16 UTC (132 KB)

  Full-text links:

  • HTML (experimental)

  Bibliographic Tools

  • NASA ADS
  • Google Scholar
  • Semantic Scholar

  About arXivLabs

  arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

• Test case 2

  Score: 9/10

  Perform the operation of converting the PDF file located at https://arxiv.org/pdf/2506.11180.pdf into Markdown format.

  Beyond Formal Semantics for Capabilities and Skills: Model Context Protocol in Manufacturing

  Luis Miguel Vieira da Silva, Aljosha Köcher, Felix Gehlhoff
  Institute of Automation Technology, Helmut Schmidt University, Hamburg, Germany
  Email: {miguel.vieira, aljosha.koecher, felix.gehlhoff}@hsu-hh.de

  Abstract

  Explicit modeling of capabilities and skills – whether based on ontologies, Asset Administration Shells, or other technologies – requires considerable manual effort and often results in representations that are not easily accessible to Large Language Models (LLMs). In this work-in-progress paper, we present an alternative approach based on the recently introduced Model Context Protocol (MCP). MCP allows systems to expose functionality through a standardized interface that is directly consumable by LLM-based agents. We conduct a prototypical evaluation on a laboratory-scale manufacturing system, where resource functions are made available via MCP. A general-purpose LLM is then tasked with planning and executing a multi-step process, including constraint handling and the invocation of resource functions via MCP. The results indicate that such an approach can enable flexible industrial automation without relying on explicit semantic models. This work lays the basis for further exploration of external tool integration in LLM-driven production systems.

  Index Terms

  Model Context Protocol, MCP, Capabilities, Skills, Large Language Models, LLMs, Ontologies, Semantic Web

• Test case 3

  Score: 2/10

  Perform the operation of converting the DOCX file located at https://www.learningcontainer.com/wp-content/uploads/2020/09/Sample-Word-Doc.docx into Markdown format.

  Tool malfunction: The response indicates that the tool requires a local path to proceed with the conversion, but the file is currently hosted online.

• Test case 4

  Score: 2/10

  Perform the operation of converting the YouTube video located at https://www.youtube.com/watch?v=3JZ_D3ELwOQ into Markdown format, including the transcript if available.

  Tool malfunction: The tool failed to process the YouTube video due to an output buffer size limitation.

• Test case 5

  Score: 6/10

  Perform the operation of converting the PPTX file located at https://file-examples.com/wp-content/uploads/2017/08/file_example_PPT_500kB.ppt into Markdown format.

  Tool malfunction: The conversion process failed because the specified PPTX file at the URL 'https://file-examples.com/wp-content/uploads/2017/08/file_example_PPT_500kB.ppt' could not be found (404 error).