Testing / evaluation framework

TLX Presentation MCP behavior review

Use this portal to run structured test cycles, record what works, document failures clearly, and give Brendon a practical change list with reproduction steps and evidence.

1. Evaluation charter

Goal

Confirm expected behavior

Run several cycles through the MCP to verify tool discovery, resource access, deck validation, rendering, and generated artifacts behave consistently.

Output

Actionable feedback

Each issue should include what happened, expected behavior, reproduction prompt/payload, severity, owner, and evidence links.

Audience

Katie + Brendon

Katie captures evaluation findings. Brendon can use the issue log and test cases to reproduce, prioritize, and patch the MCP.

Decision criteria

Dimension

Pass

Watch

Fail

Connection + discovery

Tools and resources register after restart and respond inside expected timeout.

Cold start is slow but succeeds within configured timeout.

DNS, auth, method, or timeout failure blocks use.

Instruction following

Deck output follows requested format, brand, and story intent.

Needs minor prompt refinement or layout choice correction.

Ignores requested format, wrong artifact, wrong tool, or wrong data source.

Validation quality

Validation catches schema/layout/content problems before render.

Validation catches structural issues but misses readability concerns.

Validation says valid but output is unusable or malformed.

Artifact quality

PPTX/HTML/PNG files open, are readable, and reflect the same slide content.

One format has minor rendering issues while another is usable.

Missing files, broken files, clipped slides, or inconsistent artifacts.

Developer handoff

Issue has repro steps, input payload, expected vs actual, and evidence.

Issue is understandable but missing one evidence item.

Feedback is subjective or not reproducible.

2. Test cycle runner

Run order: baseline connection → simple resource read → validation → deck compose → artifact inspection → regression retest.

3. Feedback capture

Cycle

Area

Status

Severity

Owner

Artifact / link

What was tested? Expected behavior Actual behavior / evidence Recommended change

Feedback log

4. Developer handoff

Minimum repro packet

Every fail/watch item should include:

Exact MCP tool used
Prompt or JSON payload
Expected output
Actual output/error
Generated file path or screenshot
Severity and recommended fix

Severity definitions

P0: blocks connection, tool use, or artifact generation.
P1: output is generated but wrong, misleading, clipped, or not usable for customer-facing work.
P2: improvement, polish, wording, speed, or developer-experience issue.

Known baseline from current setup

Check	Result	Evidence
Correct endpoint	Pass	https://ca-tlx-presentations-dev.livelytree-0c387508.westus3.azurecontainerapps.io/mcp
Tool discovery	Pass	8 tools discovered: catalog, compose, guidelines, design system, gallery, SBR guidelines, validate, sbr_calculate.
Resource discovery	Pass	4 resources discovered: gallery, design-system, presentation-design, sbr-guidelines.
GET /mcp	Watch	Returns Method Not Allowed; acceptable if MCP requires POST, but document for users who test URL in browser.

Share note for Brendon

Purpose: This portal tracks repeated evaluation cycles for the TLX Presentation MCP. Use the exported JSON/CSV and the feedback log to reproduce issues and confirm fixes. The framework is intentionally static so it can be shared as a link or edited directly in source control.

5. Cycle archive

Cycle	Date	Scope	Pass	Watch	Fail	Decision

Operating rule: Run the same P0/P1 test cases after every MCP change before adding new exploratory prompts.