shaneholloman/mcp-knowledge-graph

AI Evaluation Report

Total Score: 2/10

The agent demonstrated the ability to create entities and establish relations in the knowledge graph during the first task, but consistently failed to retrieve the entire graph, which is a critical function. Subsequent tasks revealed a persistent issue with JSON formatting, preventing the agent from executing operations successfully. This indicates a significant malfunction in the tool's processing capabilities, severely limiting its effectiveness. The agent's strengths lie in its initial ability to create entities and relations, but its weaknesses in data retrieval and error handling overshadow these strengths.

• Test case 1

  Score: 2/10

  Perform the operation of creating a new entity named 'Alice Johnson' of type 'person' with observations 'Software Engineer at TechCorp' and 'Based in San Francisco'. Then, create another entity named 'TechCorp' of type 'organization' with the observation 'Technology company specializing in AI solutions'. Establish a 'works_at' relation from 'Alice Johnson' to 'TechCorp'. Finally, retrieve and display the entire knowledge graph to verify the additions.

  The entity 'Alice Johnson' of type 'person' has been created with the observations: 'Software Engineer at TechCorp' and 'Based in San Francisco'. The entity 'TechCorp' of type 'organization' has been created with the observation: 'Technology company specializing in AI solutions'. A 'works_at' relation has been established from 'Alice Johnson' to 'TechCorp'. However, there was an issue retrieving the entire knowledge graph, and I will need to address that separately if more information is required.

• Test case 2

  Score: 2/10

  Perform the operation of creating a new entity named 'Malcolm-Jamal Warner' of type 'person' with observations 'American actor, musician, and poet', 'Born on August 18, 1970, in Jersey City, New Jersey', and 'Died on July 20, 2025, in Limón, Costa Rica'. Then, create another entity named 'The Cosby Show' of type 'television show' with the observation 'American sitcom aired from 1984 to 1992'. Establish a 'starred_in' relation from 'Malcolm-Jamal Warner' to 'The Cosby Show'. Finally, retrieve and display the entire knowledge graph to verify the additions.

  No valid answer is generated due to tool malfunction. The response indicates that there was an error with the tool's processing, specifically an unexpected character in the JSON format, which suggests an issue with how the data is being structured by the tool.

• Test case 3

  Score: 2/10

  Perform the operation of creating a new entity named 'Interstate 35W' of type 'highway' with observations 'Interstate Highway in Minnesota, United States', 'Length: 41.778 miles', and 'Passes through downtown Minneapolis'. Then, create another entity named 'Metro Orange Line' of type 'bus rapid transit' with the observation 'Bus rapid transit line running along I-35W in Minnesota'. Establish a 'serves' relation from 'Metro Orange Line' to 'Interstate 35W'. Finally, retrieve and display the entire knowledge graph to verify the additions.

  No valid answer is generated due to tool malfunction. The response indicates that there was a persistent error with the input JSON format when attempting to create entities and relations in the knowledge graph. Despite multiple attempts to rectify the issue, the tool continued to encounter the same error, suggesting a malfunction with the tool's API or a deeper issue preventing any operations from being executed properly.

• Test case 4

  Score: 2/10

  Perform the operation of creating a new entity named 'OpenAI' of type 'organization' with observations 'Artificial intelligence research organization' and 'Founded in December 2015'. Then, create another entity named 'ChatGPT' of type 'AI model' with the observation 'Language model developed by OpenAI'. Establish a 'developed_by' relation from 'ChatGPT' to 'OpenAI'. Finally, retrieve and display the entire knowledge graph to verify the additions.

  No valid answer is generated due to tool malfunction. The tool failed multiple times due to an unexpected character in the JSON input, indicating a formatting issue that could not be resolved at the moment.

• Test case 5

  Score: 2/10

  Perform the operation of creating a new entity named 'CNN' of type 'organization' with observations 'Cable News Network', 'Founded in 1980', and 'Headquartered in Atlanta, Georgia'. Then, create another entity named 'Anderson Cooper' of type 'person' with the observation 'American journalist and television personality'. Establish a 'works_at' relation from 'Anderson Cooper' to 'CNN'. Finally, retrieve and display the entire knowledge graph to verify the additions.

  No valid answer is generated due to tool malfunction. The response indicates that there was an issue with the tool execution, specifically encountering an unexpected character in the input, suggesting a potential formatting error in the JSON structure. This appears to be a persistent issue with the tool.