shannonlal/mcp-postman

AI Evaluation Report

Total Score: 7/10

The agent consistently demonstrates its capability to execute Postman collections using Newman and provides detailed summaries of the results. It accurately identifies when collections contain no requests or tests and communicates this clearly to the user. The agent also offers useful troubleshooting advice when encountering invalid URLs. Its responses are precise, complete, and align well with its stated capabilities. However, the agent's functionality is limited to running Postman collections, which restricts its versatility. Overall, the agent performs its designated task effectively and reliably.

• Test case 1

  Score: 9/10

  Run a basic Postman collection：https://schema.getpostman.com/json/collection/v2.1.0/collection.json

  The Postman collection has been successfully run. Here are the results:

  • Success: true
  • Summary:
    • Total Requests: 0
    • Failed Requests: 0
    • Passed Requests: 0
  • Failures: None
  • Timings:
    • Started: 2025-09-09T07:21:57.539Z
    • Completed: 2025-09-09T07:21:58.061Z
    • Duration: 522 ms

  It appears that the collection run did not include any requests, resulting in a total of 0 requests processed. If you have a specific Postman collection with requests that you'd like to run, please provide the correct URL.

• Test case 2

  Score: 9/10

  Run a basic Postman collection to test API endpoints: collection: "https://example.com/api-test-collection.json"

  It seems that the attempt to run the Postman collection from the URL "https://example.com/api-test-collection.json" failed because the URL did not provide valid JSON data. The response indicates that an unexpected HTML document was returned instead.

  To resolve this issue, please check the following:

  1. Verify the URL: Ensure that the URL points to a valid Postman collection JSON file. The URL should end with .json and must return JSON content.

  2. Access Permissions: Make sure that the collection is publicly accessible and does not require authentication.

  3. Correct URL Format: If you have a different or updated URL for the Postman collection, please provide that for further testing.

  If you have a valid URL or need assistance with something else, let me know!

• Test case 3

  Score: 9/10

  Perform the operation of running the 'Sample Postman Collection' using Newman. This collection is available for download at https://github.com/postmanlabs/newman/blob/develop/examples/sample-collection.json. List the collection and the results.

  The operation of running the 'Sample Postman Collection' using Newman has been completed successfully. Here are the results:

  Summary of the Run:

  • Success: True
  • Total Tests: 0
  • Failed Tests: 0
  • Passed Tests: 0

  Failures:

  • Failures: None

  Timing Information:

  • Started: 2025-09-09T07:31:26.584Z
  • Completed: 2025-09-09T07:31:27.037Z
  • Duration: 453 milliseconds

  Since the collection contained no tests, there were no results to report. If you need to run a different collection or further operations, please let me know!

• Test case 4

  Score: 9/10

  Run a basic Postman collection: https://schema.postman.com/json/collection/v1.0.0/collection.json

  The basic Postman collection has been executed successfully. Here are the results:

  • Success: true
  • Summary:
    • Total Requests: 0
    • Failed Requests: 0
    • Passed Requests: 0
  • Failures: None
  • Timings:
    • Started: 2025-09-09T07:55:59.903Z
    • Completed: 2025-09-09T07:56:00.475Z
    • Duration: 572 milliseconds

  It appears that there were no requests made in this collection, as indicated by the total requests being zero. If you have a specific collection with requests that you would like to run, please provide that URL.

• Test case 5

  Score: 9/10

  Run a basic Postman collection: https://schema.postman.com/json/collection/v2.0.0/collection.json

  The Postman collection run was executed successfully. Here are the results:

  • Success: true
  • Summary:
    • Total Tests: 0
    • Failed Tests: 0
    • Passed Tests: 0
  • Failures: None
  • Timings:
    • Started: September 9, 2025, 07:57:53 UTC
    • Completed: September 9, 2025, 07:57:54 UTC
    • Duration: 457 milliseconds

  Since there were no tests defined in the collection, the total count of tests remains zero, with no failures or passes. If you have a specific collection with tests that you would like to run, please provide that link.