Knowledge Retrieval Benchmark · v0.6.2 · Open & Reproducible

The eval for structured
knowledge retrieval

CKG vs RAG vs GraphRAG — measured across 64 domains and 11,031 queries on dependency chains, multi-hop paths, and aggregate lookups. Every result is reproducible from the public dataset.

64
Domains
11,031
Queries
3
Systems
3.8×
Best F1 gap
11×
Token gap
▤ Leaderboard
◈ By Task Type
⊞ By Domain
+ How to Submit
Overall results — click any column to sort
# System Macro F1 ↓ F1 bar Tokens / query Cost / 1K queries 5-hop F1 Queries run
CKG F1 improves with hop depth (0.37 → 0.77 from hop 0 to hop 5). RAG stays flat at ~0.13 regardless of depth — retrieval has no mechanism for traversing a chain. This is the architectural gap the benchmark was designed to surface.
Reproduce these results
git clone https://github.com/Yarmoluk/ckg-benchmark
pip install anthropic huggingface_hub
export ANTHROPIC_API_KEY=sk-...
python evaluation/krb_eval.py --system ckg --domain calculus
python evaluation/krb_eval.py --system ckg --all
F1 by query task type

CKG's advantage concentrates in structured tasks — dependency chains, path traversal, and aggregates. Entity lookup (T1) is the hardest for all systems. This breakdown shows where each architecture fits.

T4 (Aggregate) is the sharpest gap: CKG 0.964 vs GraphRAG 0.053. Aggregate queries require holding a complete subgraph in context — retrieval-based approaches cannot assemble the complete graph from chunks.
CKG F1 by domain — all 45 evaluated domains

CKG leads in every domain. The gap is narrowest in open-ended domains (systems thinking, theory of knowledge) and widest in domains with dense dependency structures (infographics, glp1, fft-benchmarking).

Domain CKG F1 RAG F1 Δ gap CKG tokens RAG tokens

Add your system to the leaderboard

Any retrieval system can be evaluated — RAG pipelines, embedding models, graph systems, hybrid approaches, or fine-tuned models. The harness downloads everything from HuggingFace. No proprietary data required.

① Run the eval harness
pip install anthropic huggingface_hub
git clone https://github.com/Yarmoluk/ckg-benchmark
cd ckg-benchmark

from evaluation.krb_eval import Retriever, run_eval

class MyRetriever(Retriever):
    def retrieve(self, domain, query_text, meta):
        return my_rag_function(domain, query_text)

# Writes krb_submission_my-rag-v1_<timestamp>.json
run_eval(MyRetriever(), system_name="my-rag-v1")
② Check your output

The harness writes krb_submission_*.json with your macro F1, per-task breakdown, token usage, and cost. Open it and copy the full JSON.

③ Submit via GitHub issue

Click below — the issue is pre-filled with the submission template. Paste your krb_submission_*.json into the Results section. Results are reviewed and added within 48 hours.

Submit results → Benchmark repo HF Dataset

Enterprise: Private evaluations available — run KRB against your proprietary knowledge base and receive a confidential report. Contact Graphify.md →