creating-online-evaluations

creating-online-evaluations is an LLM evaluation & observability skill for AI agents, published by PostHog in the ai-plugin repository on GitHub. Skillspace rates creating-online-evaluations D (64/100) and it passed the safety scan; the rating is based on static analysis only.

Author continuously-running online evaluations in PostHog AI observability, grounded in a real failure mode you've identified. Use when the user wants an evaluation that automatically scores new `$ai_generation` events going forward — "create an eval to catch X", "continuously check that responses do Y", "turn this failure into an eval". Covers choosing the eval type (hog / llm_judge / sentiment), gating on the team's provider key before an llm_judge eval, scoping which events fire via conditions (property filters + rollout sampling), creating it disabled, verifying scope, and enabling. Finding and ranking the failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage evaluations that already exist, use exploring-llm-evaluations.

Grade

D

Quality

64 / 100

Safety

Passed

GitHub stars

59

Unproven: rated by static analysis only, not yet compared head-to-head.

Our two independent classifiers differ on this skill's category, so treat the placement as provisional.

This rating describes the version of the repository we crawled; it may have changed since, so go by its current files.

Skillspace indexes and rates this skill; it does not host or endorse it. Run third-party skills at your own risk.