Files
ai-agent-harness/sources/stacks/ai-llm.md

1.3 KiB

AI and LLM Workloads

  • Treat prompts, schemas, model routing, retrieval, pre-processing, post-processing, thresholds, and evaluations as source-controlled behavior.
  • Explain the behavioral impact before changing a provider, model, embedding, prompt, threshold, chunking, or ranking strategy.
  • Do not call paid model, embedding, transcription, vision, or other metered APIs without separate explicit approval.
  • Do not send proprietary, personal, customer, survey, receipt, transcript, or other sensitive data to an external model or service without exact authorization for that data and destination.
  • Prefer mocks and sanitized deterministic fixtures for development and tests.
  • Validate structured model output before business use and handle refusals, malformed output, timeouts, rate limits, and provider failures explicitly.
  • Keep API keys and provider credentials in the repository's approved secret mechanism, never in source, prompts, fixtures, logs, or committed configuration.
  • Redact sensitive prompt, response, token, trace, and provider-error content from logs and reports.
  • Separate deterministic business rules from probabilistic model behavior so critical rules remain testable.
  • Compare representative evaluation results before and after behavioral changes, and report material quality, cost, latency, privacy, and reproducibility tradeoffs.