janwilmake/openapi-mcp-server

AI Evaluation Report

Total Score: 5/10

The agent demonstrates a basic understanding of OpenAPI functionalities and can provide summaries of API specifications, as seen with the GitHub and Stripe API examples. However, the agent struggles with retrieving specific information from the openapi-mcp-server due to tool malfunctions, which limits its ability to provide detailed and accurate responses for certain queries. The agent's responses are sometimes outdated or incomplete, lacking specific details such as endpoint counts, comprehensive descriptions, and time relevance. These limitations suggest that while the agent can offer general insights, it may not be fully reliable for precise or up-to-date information retrieval.

• Test case 1

  Score: 2/10

  What are the key features and functionalities of the `openapi-mcp-server` as described in its official documentation?

  No valid answer is generated due to tool malfunction. The response indicates a failure due to tool malfunction when attempting to call the tool.

• Test case 2

  Score: 2/10

  How does the `openapi-mcp-server` facilitate the integration of OpenAPI specifications with AI agents, and what are its key features in this context?

  No valid answer is generated due to tool malfunction. The response indicates a failure due to tool malfunction when attempting to call the tool.

• Test case 3

  Score: 2/10

  What are the installation steps and prerequisites for setting up the `openapi-mcp-server` to transform OpenAPI specifications into AI-accessible tools?

  No valid answer is generated due to tool malfunction. The tool output indicates an error: 'No OpenAPI found at janwilmake/openapi-mcp-server.' This suggests a malfunction or an issue with the API rather than an invalid task input.

• Test case 4

  Score: 6/10

  Perform the operation of retrieving and summarizing the OpenAPI specification for the GitHub API using the `openapi-mcp-server`.

  The GitHub v3 REST API has a total of 1063 endpoints. Below is a summary of its features:

  • API Root: Access the root of the GitHub API.
  • Security Advisories:
    • List global security advisories
    • Get specific global security advisories
  • Apps:
    • Get authenticated app details
    • Create apps from manifests
    • Manage webhook configurations and deliveries for apps
  • Classroom:
    • Manage assignments and classrooms
  • Code Security:
    • Create and manage code security configurations
  • Dependabot:
    • List and manage alerts for an organization
  • Gists:
    • Create, list, and manage gists and their comments
  • Issues:
    • List, create, and manage issues and comments
  • Pull Requests:
    • Create, list, and manage pull requests and their reviews
  • Repositories:
    • Create, list, and manage repositories, including branches, tags, and deployments
  • Rate Limit:
    • Check the rate limit status
  • Search:
    • Search code, commits, issues, pull requests, repositories, and users
  • Users:
    • Manage user accounts, including following, blocking, and managing SSH keys For detailed information about specific endpoints, you can visit the respective links provided in the API overview. This overview highlights the extensive capabilities of the GitHub API, which can be utilized for a wide range of operations related to repositories, user management, security, and more.
• Test case 5

  Score: 7/10

  Perform the operation of retrieving and summarizing the OpenAPI specification for the Stripe API using the `openapi-mcp-server`.

  The Stripe API (version 2022-11-15) consists of 446 endpoints designed for various payment processing functionalities. Below is a brief overview of the API:

  • API Base URL: https://api.stripe.com
  • Documentation: More detailed information about each endpoint can be found at Stripe API Documentation.

  Key Endpoints Overview:

  • Accounts:

    • GetAccount
    • PostAccounts
    • GetAccounts
    • DeleteAccountsAccount

  • Charges:

    • GetCharges
    • PostCharges
    • GetChargesCharge
    • PostChargesChargeRefund

  • Customers:

    • GetCustomers
    • PostCustomers
    • GetCustomersCustomer

  • Invoices:

    • GetInvoices
    • PostInvoices
    • GetInvoicesInvoice

  • Payments:

    • GetPaymentIntents
    • PostPaymentIntents
    • GetPaymentMethods

  • Webhooks:

    • GetWebhookEndpoints
    • PostWebhookEndpoints For more detailed information about a specific endpoint, you can visit the respective links provided in the overview or consult the complete API documentation.