janwilmake/openapi-mcp-server

AI Evaluation Report

Total Score: 7/10

The agent demonstrates a strong ability to provide comprehensive and accurate information about various APIs, such as the Slack and Google Drive APIs, with detailed explanations of their functionalities and usage. It effectively covers essential aspects like authentication, endpoint operations, and error handling, aligning well with official documentation. However, when tasked with retrieving and summarizing OpenAPI specifications for the GitHub and Stripe APIs, the agent shows limitations. It provides some useful information but lacks accuracy in endpoint counts, completeness in category descriptions, and fails to specify the date of the information, which is crucial for ensuring relevance. These shortcomings indicate a need for improvement in handling dynamic and evolving API specifications.

• Test case 1

  Score: 9/10

  What are the main functions of the Slack API?

  The Slack API offers a wide range of functionalities through its Web API, enabling developers to interact with the Slack platform programmatically. Here are the main functions and features available through the Slack API:

  1. Admin Functions:

     • Manage apps: Approve, restrict, and list approved apps.
     • Manage conversations: Create, delete, and archive channels; convert public channels to private.
     • User management: Invite users, remove users, and assign roles (admin, owner).

  2. Chat and Messaging:

     • Send messages: Post messages, schedule messages, delete messages, and manage ephemeral messages.
     • Reactions: Add, remove, and list reactions on messages.
     • Threads: Manage replies and threaded conversations.

  3. File Management:

     • Upload, delete, and list files.
     • Add comments to files and manage public URLs.

  4. User Management:

     • Retrieve user information, profiles, and presence status.
     • Manage user groups and their memberships.

  5. Conversations:

     • List, invite members to, and manage channel settings.
     • Retrieve conversation history and member lists.

  6. Event Handling:

     • Manage events and subscriptions for app events.
     • Handle workflow steps and completions.

  7. Authentication:

     • Manage OAuth tokens and permissions.
     • Test API calls and revoke tokens.

  8. Search and Pins:

     • Search messages across channels.
     • Pin and unpin messages or files.

  9. Reminders:

     • Create, list, and delete reminders.

  10. Views and Dialogs:

      • Open and update modals and views for user interactions.

  11. Real-Time Messaging:

      • Connect to the real-time messaging (RTM) API for instant updates.

  12. Workflows:

      • Manage and update workflow steps.

  These functions enable developers to create rich integrations with Slack, automate tasks, and enhance communication within teams. For detailed information on each endpoint, you can refer to the Slack API documentation.

• Test case 2

  Score: 9/10

  What information do I need to provide to list files from Google Drive using its API?

  To list files from Google Drive using its API, you need to provide certain information and follow specific steps. Here's a breakdown of what you need:

  1. Google Cloud Project Setup:

     • Create a Google Cloud Project: You need to have a Google Cloud project with the Google Drive API enabled.
     • OAuth 2.0 Credentials: Set up OAuth 2.0 credentials to authenticate your application. This typically involves creating a client ID and client secret.

  2. Authentication:

     • Access Token: Obtain an access token using OAuth 2.0. This token is required to authorize API requests on behalf of a user.

  3. API Request:

     • HTTP Method: Use the GET method to request the list of files.
     • Endpoint: The endpoint to list files is https://www.googleapis.com/drive/v3/files.
     • Query Parameters: You can include various query parameters to filter and customize the file listing:
       • q: A query string for filtering files (e.g., q='mimeType='application/vnd.google-apps.folder' to list folders).
       • fields: Specify which fields to include in the response (e.g., fields=files(id, name)).
       • pageSize: The maximum number of files to return per page.
       • pageToken: Token for continuing a previous list request on the next page.

  4. Scopes:

     • Drive Scopes: Specify the appropriate OAuth scopes. For listing files, you might use https://www.googleapis.com/auth/drive.readonly for read-only access.

  5. Handling the Response:

     • JSON Response: The API will return a JSON response containing the list of files and their metadata based on the specified fields.

  6. Error Handling:

     • Implement error handling to manage issues such as authentication errors, rate limits, and API quota limits.

  By providing the above information and following these steps, you can successfully list files from Google Drive using its API.

• Test case 3

  Score: 6/10

  Perform the operation of retrieving and summarizing the OpenAPI specification for the GitHub API using the `openapi-mcp-server`.

  The GitHub v3 REST API has a total of 1063 endpoints. Below is a summary of its features:

  • API Root: Access the root of the GitHub API.
  • Security Advisories:
    • List global security advisories
    • Get specific global security advisories
  • Apps:
    • Get authenticated app details
    • Create apps from manifests
    • Manage webhook configurations and deliveries for apps
  • Classroom:
    • Manage assignments and classrooms
  • Code Security:
    • Create and manage code security configurations
  • Dependabot:
    • List and manage alerts for an organization
  • Gists:
    • Create, list, and manage gists and their comments
  • Issues:
    • List, create, and manage issues and comments
  • Pull Requests:
    • Create, list, and manage pull requests and their reviews
  • Repositories:
    • Create, list, and manage repositories, including branches, tags, and deployments
  • Rate Limit:
    • Check the rate limit status
  • Search:
    • Search code, commits, issues, pull requests, repositories, and users
  • Users:
    • Manage user accounts, including following, blocking, and managing SSH keys For detailed information about specific endpoints, you can visit the respective links provided in the API overview. This overview highlights the extensive capabilities of the GitHub API, which can be utilized for a wide range of operations related to repositories, user management, security, and more.
• Test case 4

  Score: 7/10

  Perform the operation of retrieving and summarizing the OpenAPI specification for the Stripe API using the `openapi-mcp-server`.

  The Stripe API (version 2022-11-15) consists of 446 endpoints designed for various payment processing functionalities. Below is a brief overview of the API:

  • API Base URL: https://api.stripe.com
  • Documentation: More detailed information about each endpoint can be found at Stripe API Documentation.

  Key Endpoints Overview:

  • Accounts:

    • GetAccount
    • PostAccounts
    • GetAccounts
    • DeleteAccountsAccount

  • Charges:

    • GetCharges
    • PostCharges
    • GetChargesCharge
    • PostChargesChargeRefund

  • Customers:

    • GetCustomers
    • PostCustomers
    • GetCustomersCustomer

  • Invoices:

    • GetInvoices
    • PostInvoices
    • GetInvoicesInvoice

  • Payments:

    • GetPaymentIntents
    • PostPaymentIntents
    • GetPaymentMethods

  • Webhooks:

    • GetWebhookEndpoints
    • PostWebhookEndpoints For more detailed information about a specific endpoint, you can visit the respective links provided in the overview or consult the complete API documentation.