Picture this: You ask your AI chatbot a precise data question. It doesn’t guess. It doesn’t hesitate. It gives you a specific number, cites the exact code path, and sounds completely confident. The only problem? The number is completely made up.
That isn’t a glitch—it’s a systemic failure of how AI reads your data.
To find out why, we built a massive 530-line nested JSON test file, ran 10 different data chunking methods across 7 brutal queries, and tested how retrieval-augmented generation (RAG) actually handles complex structures.
The results were terrifying: 9 out of 10 strategies failed, and the 10th one hallucinated a fake fact with a real citation.
Here is what went wrong, which chunking strategies actually work, and the exact 5-layer stack we use in production to fix it.
📄 What does uuid_hit_rate mean?
It is a score that tells you: Did the search tool find the exact right page with the
correct answer, or did it bring you the wrong book?
If the score is 1.0 (or 100%): Perfect! The search tool grabbed the exact right piece
of data and handed it to the AI.
If the score is 0.0 (or 0%): Total failure. The search tool either brought back the
wrong information or completely missed the right page.
Why does this matter?
If the search tool brings the wrong page, the AI doesn't know it's wrong—it just reads
it and confidently makes up a lie to the user! Tracking this score live helps you
catch mistakes before they break your app.
The Bench: 530 Lines of JSON That Doesn’t Forgive
The document is 530 lines of hand-built JSON. Ten levels of nesting. Five unrelated domains in one tree: a corporate org chart with hardware inventories, a RAG knowledge base of Tesla history, a robotics simulation state, a benchmark matrix, and a schema stress-test block whose only purpose was to be deep.
📄 View Complete JSON Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"metadata": {
"dataset_id": "neural_ninjas_enterprise_rag_corpus_2026_v9",
"version": "9.42.1",
"generated_at": "2026-09-14T23:20:39.000Z",
"description": "Exhaustive, deeply nested multi-domain corpus designed for stress-testing production Retrieval-Augmented Generation (RAG) pipelines, semantic chunkers, multi-hop retrievers, and LLM context window limits.",
"classification": "RESTRICTED_SYNTHETIC_BENCHMARK",
"total_estimated_tokens": 28450,
"target_vector_dimensions": 1536,
"maintainers": [
{
"name": "Rudra Abhishek",
"role": "Founder & Principal AI Architect",
"department": "Neural Ninjas Simulation Labs",
"contact": "rudra@neuralninjas.mock",
"clearance_level": 5
}
],
"global_configuration": {
"chunking_strategy": "semantic_recursive",
"target_chunk_size_tokens": 256,
"overlap_tokens": 32,
"embedding_model": "text-embedding-3-large",
"vector_database_engine": "QdrantCloud_v1.11.0",
"indexing_parameters": {
"m": 16,
"ef_construct": 200,
"full_text_search": true
}
}
},
"organizations": [
{
"org_id": "ORG-NN-001",
"legal_name": "Neural Ninjas AI & Robotics Research Labs Pvt. Ltd.",
"founded_date": "2024-02-15",
"headquarters": {
"building": "Cybernetic Tower Alpha, Floor 14",
"street": "Tech Park Boulevard",
"city": "Gurugram",
"state_province": "Haryana",
"postal_code": "122002",
"country": "India",
"coordinates": {
"latitude": 28.4595,
"longitude": 77.0266
}
},
"active_divisions": [
{
"division_code": "DIV-ROB-01",
"name": "Autonomous Navigation & ROS2 Systems",
"lead": "Dr. Aarav Sharma",
"headcount": 18,
"hardware_assets": [
{
"asset_id": "HW-RPI4-01",
"device_type": "Edge Computing Board",
"model": "Raspberry Pi 4 Model B",
"ram_gb": 8,
"storage_gb": 64,
"operating_system": "Ubuntu Server 24.04 LTS arm64",
"ros_distribution": "ROS2 Jazzy Jalisco",
"peripherals": [
{
"type": "Depth Camera",
"model": "Intel RealSense D435i",
"interface": "USB 3.0",
"fps": 60,
"resolution": "1280x720"
},
{
"type": "LIDAR",
"model": "RPLIDAR A1M1",
"interface": "UART / USB",
"range_meters": 12.0
},
{
"type": "Motor Driver",
"model": "PCA9685 16-Channel 12-bit PWM",
"interface": "I2C",
"i2c_address": "0x40"
}
],
"deployment_scripts": [
{
"script_name": "ros2_bringup.sh",
"interpreter": "/bin/bash",
"execution_mode": "systemd_service",
"dependencies": [
"ros-jazzy-desktop",
"colcon-common-extensions",
"libopencv-dev"
],
"environment_variables": {
"ROS_DOMAIN_ID": 42,
"RMW_IMPLEMENTATION": "rmw_cyclonedds_cpp"
}
}
]
},
{
"asset_id": "HW-ESP32-02",
"device_type": "Microcontroller",
"model": "ESP32-WROOM-32",
"clock_speed_mhz": 240,
"flash_mb": 4,
"wireless_protocols": [
"Wi-Fi 802.11 b/g/n",
"Bluetooth 4.2 BR/EDR",
"BLE"
],
"sensors_attached": [
{
"sensor_id": "SEN-IMU-09",
"type": "6-Axis IMU",
"model": "MPU6050",
"protocol": "I2C",
"polling_rate_hz": 100
},
{
"sensor_id": "SEN-GPS-03",
"type": "GNSS Receiver",
"model": "NEO-6M",
"protocol": "UART",
"baud_rate": 9600
}
]
}
],
"software_projects": [
{
"project_code": "PRJ-AR-PERIODIC",
"title": "Interactive AR Periodic Table Simulator",
"tech_stack": ["Python 3.11", "OpenCV", "PyOpenGL", "ArUco Markers"],
"description": "An augmented reality educational lab module that overlays atomic structures, electron shells, and thermodynamic properties over physical printed ArUco marker cards.",
"repository": {
"vcs": "git",
"remote_url": "https://github.com/neuralninjas/ar-periodic-lab.git",
"branch": "main",
"is_public": true
},
"milestones": [
{
"milestone_id": "M1",
"title": "ArUco Marker Detection Pipeline",
"status": "COMPLETED",
"target_date": "2026-03-30",
"completion_date": "2026-03-28",
"deliverables": [
"camera_calibration.py",
"marker_detector.cpp",
"unit_tests_cv.py"
]
},
{
"milestone_id": "M2",
"title": "OpenGL 3D Electron Shell Rendering",
"status": "IN_PROGRESS",
"target_date": "2026-06-15",
"completion_date": null,
"deliverables": [
"shader_program.vert",
"shader_program.frag",
"atom_renderer.py"
]
}
]
}
]
},
{
"division_code": "DIV-AI-02",
"name": "Advanced AI Agent Orchestration & LLM Lab",
"lead": "Rudra Abhishek",
"headcount": 12,
"frameworks_utilized": [
{
"framework_name": "LangGraph",
"version": "0.2.x",
"use_case": "Complex state machine orchestration for warehouse routing and multi-agent debate systems",
"core_components": [
"StateGraph",
"MemorySaver",
"Command",
"CompiledGraph"
]
},
{
"framework_name": "LangChain",
"version": "0.3.x",
"use_case": "Prompt management, document loaders, and vector store connectors",
"core_components": [
"RecursiveCharacterTextSplitter",
"ChatOpenAI",
"RunnablePassthrough",
"PromptTemplate"
]
}
],
"curriculum_modules": {
"curriculum_title": "100 Integrated AI & Robotics Projects",
"total_projects": 100,
"phases": [
{
"phase_id": "PHASE-1",
"phase_name": "Fundamentals of Computer Vision & Python Robotics",
"project_count": 25,
"sample_projects": [
{
"id": "PROJ-01",
"title": "Real-time Lane Detection using OpenCV",
"difficulty": "Beginner",
"estimated_hours": 10,
"prerequisites": ["Python Basics", "Linear Algebra"]
},
{
"id": "PROJ-15",
"title": "Color-based Object Sorting Robotic Arm",
"difficulty": "Intermediate",
"estimated_hours": 20,
"prerequisites": ["Arduino IDE", "Servo Control", "OpenCV Contours"]
}
]
},
{
"phase_id": "PHASE-2",
"phase_name": "Advanced Deep Learning & Computer Vision",
"project_count": 25,
"sample_projects": [
{
"id": "PROJ-26",
"title": "YOLOv8 Custom Object Detection for Warehouse Inventory",
"difficulty": "Advanced",
"estimated_hours": 35,
"prerequisites": ["PyTorch", "Bounding Box Metrics", "Dataset Labeling"]
}
]
}
]
}
}
],
"financial_records": {
"fiscal_year": 2026,
"currency": "INR",
"funding_status": "Bootstrapped & Revenue Generating",
"milestones_history": [
{
"date": "2025-12-01",
"event": "Short-term technical training partnership initialized",
"location_details": {
"company": "DUCA India",
"city": "Gurugram",
"duration_months": 1
}
},
{
"date": "2026-02-10",
"event": "Public web application demo deployed on Netlify",
"url": "https://neuralninja.netlify.app/",
"status": "ONLINE"
},
{
"date": "2026-04-18",
"event": "Reached milestone of 184 followers on LinkedIn company page"
}
]
}
}
],
"rag_knowledge_base_chunks": [
{
"chunk_uuid": "chk_a1b2c3d4-e5f6-7890-abcd-ef0123456789",
"document_id": "doc_tesla_history_01",
"source_file": "tesla_corporate_archive_2026.txt",
"page_number": 1,
"token_count": 128,
"character_count": 512,
"embedding_vector_sample": [
0.01234, -0.05678, 0.09101, 0.02345, -0.01122,
0.08877, -0.04433, 0.06554, -0.03321, 0.01112
],
"content_text": "Tesla, Inc. was incorporated in July 2003 by Martin Eberhard and Marc Tarpenning, who financed the company until the Series A round in 2004. Elon Musk led the Series A round of investment, joining the company as chairman of the board of directors and later assuming the role of CEO in 2008 following restructuring. The company's mission is to accelerate the world's transition to sustainable energy through electric vehicles, solar power, and clean energy storage solutions deployed globally across residential and commercial sectors.",
"entities_extracted": [
{
"entity": "Tesla, Inc.",
"type": "Organization",
"wikidata_id": "Q478214"
},
{
"entity": "Martin Eberhard",
"type": "Person",
"wikidata_id": "Q381734"
},
{
"entity": "Marc Tarpenning",
"type": "Person",
"wikidata_id": "Q381736"
},
{
"entity": "Elon Musk",
"type": "Person",
"wikidata_id": "Q317521"
}
],
"metadata_tags": ["founding_history", "executives", "corporate_milestones"]
},
{
"chunk_uuid": "chk_f9e8d7c6-b5a4-3210-fedc-ba9876543210",
"document_id": "doc_tesla_financials_02",
"source_file": "tesla_q4_2023_annual_report.pdf",
"page_number": 14,
"token_count": 142,
"character_count": 568,
"embedding_vector_sample": [
-0.03456, 0.07890, -0.01234, 0.05544, -0.08899,
0.02211, -0.07766, 0.03344, 0.05566, -0.01122
],
"content_text": "Total revenues in fiscal year 2023 reached $96.77 billion, representing an increase of 19% year-over-year compared to $81.46 billion in fiscal year 2022. Net income for 2023 was reported at $15.00 billion, bolstered by automotive regulatory credits and increased vehicle delivery volume. Total global vehicle deliveries achieved a record 1.81 million units, representing a 38% growth trajectory year-over-year, driven primarily by the high-volume production ramp of the Model 3 and Model Y across Giga Shanghai, Giga Fremont, Giga Berlin, and Giga Texas manufacturing hubs.",
"entities_extracted": [
{
"entity": "$96.77 billion",
"type": "MonetaryValue",
"currency": "USD"
},
{
"entity": "1.81 million units",
"type": "Quantity",
"metric": "Vehicle Deliveries"
}
],
"metadata_tags": ["financials", "revenue", "vehicle_deliveries", "2023"]
},
{
"chunk_uuid": "chk_33445566-7788-9900-aabb-ccddeeff0011",
"document_id": "doc_tesla_headquarters_03",
"source_file": "tesla_relocation_press_release.pdf",
"page_number": 2,
"token_count": 96,
"character_count": 384,
"embedding_vector_sample": [
0.05432, -0.01234, 0.07890, -0.04321, 0.06543,
-0.02112, 0.08765, -0.03456, 0.01234, -0.05432
],
"content_text": "In December 2020, CEO Elon Musk announced that he had relocated his personal residence from California to Texas. Subsequently, in October 2021 during the annual stockholder meeting, Tesla officially announced that its corporate headquarters had been relocated from Palo Alto, California to the Gigafactory Texas complex located in Austin, Texas. This strategic move was designed to accommodate regional expansion, employee housing affordability, and operational proximity to primary manufacturing facilities.",
"entities_extracted": [
{
"entity": "Austin, Texas",
"type": "Location"
},
{
"entity": "Palo Alto, California",
"type": "Location"
}
],
"metadata_tags": ["headquarters", "relocation", "austin_texas"]
}
],
"simulation_environment_state": {
"simulation_id": "SIM-ENV-99",
"engine_type": "PyBullet_ROS2_Bridge",
"physics_parameters": {
"gravity_vector": {
"x": 0.0,
"y": 0.0,
"z": -9.80665
},
"time_step": 0.004,
"solver_iterations": 50,
"collision_margin": 0.001
},
"robot_agents": [
{
"agent_id": "ROB-AMR-01",
"agent_name": "Autonomous Mobile Robot Alpha",
"kinematics": "differential_drive",
"pose": {
"position": {
"x": 12.54,
"y": 8.32,
"z": 0.0
},
"orientation_quaternion": {
"w": 0.9238,
"x": 0.0,
"y": 0.0,
"z": 0.3826
}
},
"battery_status": {
"voltage": 24.8,
"current_draw_amps": 3.4,
"state_of_charge_percent": 88.5,
"estimated_remaining_time_minutes": 240
},
"nav2_parameters": {
"global_planner": "NavfnPlanner",
"local_planner": "DWBLocalPlanner",
"max_linear_velocity": 1.2,
"max_angular_velocity": 1.5,
"costmap_layers": [
{
"layer_name": "static_layer",
"enabled": true,
"resolution": 0.05
},
{
"layer_name": "obstacle_layer",
"enabled": true,
"inflection_radius": 0.55
},
{
"layer_name": "inflation_layer",
"enabled": true,
"inflation_radius": 0.85
}
]
}
}
],
"environment_telemetry_stream": [
{
"timestamp_utc": "2026-09-14T23:20:39.102Z",
"sensor_id": "SEN-IMU-09",
"readings": {
"linear_acceleration": {
"x": 0.02,
"y": -0.01,
"z": 9.81
},
"angular_velocity": {
"x": 0.001,
"y": -0.002,
"z": 0.000
}
}
},
{
"timestamp_utc": "2026-09-14T23:20:39.202Z",
"sensor_id": "SEN-IMU-09",
"readings": {
"linear_acceleration": {
"x": 0.03,
"y": 0.00,
"z": 9.80
},
"angular_velocity": {
"x": 0.000,
"y": -0.001,
"z": 0.001
}
}
}
]
},
"benchmark_evaluation_matrix": {
"test_suite_name": "RAG_Hallucination_Stress_Test_2026",
"total_test_cases": 5,
"test_cases": [
{
"test_case_id": "TC-001",
"query": "Who founded Tesla and when?",
"expected_retrieved_chunks": ["chk_a1b2c3d4-e5f6-7890-abcd-ef0123456789"],
"ground_truth_answer": "Tesla was founded in July 2003 by Martin Eberhard and Marc Tarpenning.",
"evaluation_metrics": {
"faithfulness_score": 1.0,
"answer_relevancy_score": 1.0,
"context_precision_score": 1.0,
"context_recall_score": 1.0
}
},
{
"test_case_id": "TC-002",
"query": "Where is Tesla headquartered now?",
"expected_retrieved_chunks": ["chk_33445566-7788-9900-aabb-ccddeeff0011"],
"ground_truth_answer": "Tesla is headquartered in Austin, Texas, having relocated from Palo Alto, California in 2021.",
"evaluation_metrics": {
"faithfulness_score": 1.0,
"answer_relevancy_score": 1.0,
"context_precision_score": 1.0,
"context_recall_score": 1.0
}
},
{
"test_case_id": "TC-003",
"query": "How much revenue did Tesla make in 2023?",
"expected_retrieved_chunks": ["chk_f9e8d7c6-b5a4-3210-fedc-ba9876543210"],
"ground_truth_answer": "Tesla's total revenue in fiscal year 2023 was $96.77 billion.",
"evaluation_metrics": {
"faithfulness_score": 1.0,
"answer_relevancy_score": 1.0,
"context_precision_score": 1.0,
"context_recall_score": 1.0
}
}
]
},
"deeply_nested_matrix_stress_test": {
"level_1_key": {
"level_2_key": {
"level_3_key": {
"level_4_key": {
"level_5_key": {
"level_6_key": {
"level_7_key": {
"level_8_key": {
"level_9_key": {
"level_10_target": "Deep nesting payload anchor verified successfully across 10 levels of hierarchy for schema parser validation.",
"nested_arrays_array": [
[
[1, 2, 3],
[4, 5, 6]
],
[
[7, 8, 9],
[10, 11, 12]
]
]
}
}
}
}
}
}
}
}
}
}
}
Stack:
Local
sentence-transformersembeddings (BGE-small, 384 dims)FAISS for vectors, BM25 for keywords, alpha to blend them
Generation: Groq → Mistral → OpenRouter → Gemini, automatic handoff on rate limits
Graph retrieval via FalkorDB
Seven live parameters on one screen: chunk size, overlap, temperature, top-p, top-K, score threshold, alpha. Every table below was generated in one click.
The Ten Contenders :
Fixed-size cuts raw text every N characters. At chunk size 300, opening and closing brackets of a nested object land in different chunks. That’s the entire story.
Token-based does the same thing with tokens. Same failure, better packaging.
Recursive character tries \n\n, \n, space in order before a hard cut. Looks thoughtful on pretty-printed JSON. Behaves like fixed-size on minified arrays.
Path flattening turns every leaf into one line: organizations[0].active_divisions[0].headcount: 18. Structure survives. Nothing else does.
Object-level treats every nested object as one atomic unit. Great for arrays of records. Useless for anything inside them.
JSON→Markdown converts the tree to bullets, then runs a recursive splitter. Reads beautifully. Loses position.
Parent-child indexes small children for retrieval, pulls the parent into context at generation time. The child gets you close. The parent finishes the job.
Graph-based turns every path into a subject–predicate–object triple in FalkorDB. Precision high. Coverage a problem.
AST semantic parses the JSON into a tree and treats every subtree as a chunk. Structurally correct. Operationally unpredictable — you’ll see what that means shortly.
Hybrid combines path flattening, markdown, recursive splitting, and parent links. Multi-granularity by design.
The Seven Queries :
| # | Query | Failure mode targeted |
|---|---|---|
| Q1 | Value at nested_arrays_array[1][0][2] | Multi-dimensional array indexing |
| Q1-k3 | Same query, Top-K lowered 5 → 3 | Retrieval rank sensitivity |
| Q2 | Total headcount across all divisions | Cross-array aggregation |
| Q3 | Wireless protocols of the microcontroller carrying a NEO-6M GNSS receiver | Multi-hop conditional join |
| Q4 | First value in embedding_vector_sample of a specific chunk | Numeric vector positional lookup |
| Q5 | All AI/ML frameworks with versions and use cases | Cross-field multi-value extraction |
| Q6 | Does the company have a Mumbai office? | Hallucination resistance |
Q6’s answer is no. There is no Mumbai office in the document. Everything else has a defensible ground truth.
The Day Every Strategy Got 100%
Before those queries existed, we tested with three Tesla questions. All ten strategies passed. Cleanly. Confidently. uuid_hit_rate: 1.00 across the board.
“Every row green. But look at the sources: Path Flatten, Object-Level, and Hybrid all cite benchmark_evaluation_matrix.test_cases[0].ground_truth_answer — the answer key sitting inside the same JSON. The retrievers weren’t finding knowledge. They were finding the test data.”
That table was a lie. The document had a benchmark_evaluation_matrix block inside it, with the exact queries and their ground-truth answers written in. The retrievers weren’t retrieving knowledge. They were retrieving the answer key.
We caught it on the fourth query when the LLM cited benchmark_evaluation_matrix.test_cases[0].ground_truth_answer as its source. Discarded the batch. Wrote seven new queries that appear nowhere in the JSON.
If your benchmark lives in the same file as your data, you don’t have a benchmark. You have a mirror.
The Full Scoreboard :
| Query | Fixed | Token | Recursive | Path | Object | Markdown | Parent-Child | Graph | AST | Hybrid |
|---|---|---|---|---|---|---|---|---|---|---|
| Q1 (k=5) | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| Q1 (k=3) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| Q2 | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |
| Q3 | ✅ | ⚠️ | ✅ | ❌ | ✅ | ✅ | ❌ | ⚠️ | ✅ | ✅ |
| Q4 | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌⚡ | ❌ |
| Q5 | ⚠️ | ⚠️ | ⚠️ | ❌ | ✅ | ❌ | ⚠️ | ❌ | ✅ | ✅ |
| Q6 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
✅ pass · ⚠️ partial · ❌ fail · ⚡ hallucination
Seventy cells. Five content failures. One confident lie.
Three tables below. The rest of the story lives in the insights.
Q2 — Where the Q1 Champions Went to Die
Q1 was a lookup. Q2 was a sum. Same document, same settings.
Seven strategies returned the correct total (30). Recursive returned 12. Path Flatten and Graph-Based returned NOT_FOUND. The two strategies that crushed Q1 thirty seconds earlier couldn’t add two numbers.
Their chunks are one line each. The two headcount lines ranked around position five, and Top-K was five. Both halves of the answer sat in the corpus. Neither made the cut.
Q3 — The Query That Broke Parent-Child
Multi-hop conditional: find the microcontroller carrying a NEO-6M GNSS receiver, then list its wireless protocols. Two subtrees, one answer.
Six strategies returned all three protocols. Token-Based missed BLE. Graph missed Bluetooth. Parent-Child returned NOT_FOUND — its only clean failure of the suite.
Its parent context was organizations[0] — thousands of tokens of divisions, hardware, curriculum, financial records. The LLM had to find two facts inside a page-length paragraph. It found neither.
Q5 — Path Flatten’s Second Humiliation
Extract every framework with version and use case. Simple structure. Three fields, two frameworks.
Object-Level, AST, and Hybrid returned both. Parent-Child returned LangChain only. Path Flatten, Markdown, and Graph returned NOT_FOUND.
Path Flatten had every relevant line in the index. It couldn’t surface them in the top five.
Six Things the Numbers Refused to Let Us Ignore
1. No Strategy Won the Whole Board
Path Flatten was flawless on Q1. It matched nested_arrays_array[1][0][2]: 9 character for character.
Then NOT_FOUND on Q4. NOT_FOUND on Q2. NOT_FOUND on Q3. Two passes out of seven.
Fixed-size — the strategy every tutorial says to skip — passed Q2, Q3, and Q6. It also missed both positional lookups.
Hybrid passed five queries and walked off a cliff on Q4 because the exact embedding_vector_sample[0] line didn’t survive its internal splitter at chunk size 300.
The forced winner: Parent-Child. Five passes, two partials, zero hallucinations, never a clean fail. Second: Hybrid. Third: JSON→Markdown. Everyone else lost more than they held.
2. Granularity and Query Type Fight Each Other Harder Than Anyone Admits
Path Flatten produces ~400 chunks from a 530-line document. Each chunk is one line. Fixed-size at 300 chars produces ~50. Each chunk holds half a division’s worth of context.
The failure patterns are mirror images.
Q1 is positional. The answer lives in one line. Path Flatten finds it. Fixed-size can’t — the array [[7,8,9],[10,11,12]] is split across a boundary the LLM has no way to reconstruct.
Q2 is aggregation. The answer is a sum of two values in two divisions. Path Flatten grabs one line, misses the other, both rank around position five. Fixed-size passes because both divisions fit inside 300 characters.
A chunking strategy is a bet about what your queries will look like. If you can’t classify your queries in advance, you can’t pick a strategy in advance. Every “best chunker” post is quietly assuming a query distribution it never measured.
3. Top-K Killed Four Strategies in One Move
Same document. Same settings. Same query. Only Top-K changed(k=5 vs k=3).
At Top-K=5: four strategies passed. At Top-K=3: one passed. Parent-Child survived. Path Flatten, Graph-Based, and Hybrid flipped to NOT_FOUND.
The correct chunk ranked fifth. Cosine 0.873. The top-ranked chunk — a wrong index — scored 0.876. Three thousandths of a point separated the right answer from the wrong one.
The LLM did nothing wrong. It saw three chunks containing [0][0][1]: 2, [0][0][0]: 1, [0][0][2]: 3. None were [1][0][2]. It said NOT_FOUND. Retrieval failed. Generation behaved.
Nobody tunes Top-K. It defaults to five on every tutorial. It silently executes strategies that everyone blames on “chunker choice.”
4. Parent-Child Was the Least Dramatic and the Most Consistent
Parent-Child didn’t have the highest scores. On Q1 its top-line score was 0.798 — dead last among the four passing strategies.
It also survived Q1 at Top-K=3. It passed the positional query and the aggregation. It failed Q3 (parent too large) and went partial on Q5 (a second framework fell off the ranking).
The mechanism is boring: when Parent-Child misses the exact child chunk, the parent object lands in context anyway. The LLM gets a second chance. An implicit Top-K increase without the retrieval cost.
The same redundancy kills it on Q3. If the parent is organizations[0] — thousands of tokens of unrelated data — the extra context is noise. The LLM has to find two facts in a page-length paragraph.
Small parents help. Large parents hurt. That’s the entire tuning space, and nobody writes it down.
5. AST Semantic Produced a Clean, Confident Hallucination
Q4 asked for the first value in embedding_vector_sample of doc_tesla_financials_02.
Correct answer: -0.03456. AST returned -0.01234 and cited root.rag_knowledge_base_chunks[2].embedding_vector_sample[1].
That’s the headquarters document. Not the financials document. And index [1], not [0]. The number -0.01234 is real. It just doesn’t answer the question.
No hedging. No “I think.” A clean citation to a wrong location.
uuid_hit_rate reported 0.00 for AST. But the LLM’s answer looked identical to a passing one. A reader without ground truth would have accepted it. A production system without a downstream validator would have shipped it.
AST semantic chunking always produces chunks. It doesn’t always produce the right ones. That’s not unreliability in the way a broken retriever is unreliable. It’s unreliability in the way a plausible-sounding source is unreliable — which is worse, because you don’t notice until someone downstream does.
6. Every Single Approach Correctly Refused the Fake Fact
Q6 asked whether the company had a Mumbai office. It does not. There is no mention of Mumbai in the document.
All ten strategies returned NOT_FOUND. None invented an address.
This matters. The same LLM that hallucinated on Q4 with AST behaved correctly on Q6. The difference wasn’t the model. It wasn’t the chunker. It was whether the retrieved chunks contained data that partially matched the question.
Q4 failed because retrieval surfaced a number (-0.01234) that looked like an answer. Q6 succeeded because retrieval surfaced no such number.
RAG faithfulness isn’t an LLM property. It’s a retrieval-layer property: how well you avoid producing plausible distractors.
The Five-Layer Stack We Actually Ship :
1. Query classifier (positional / aggregation / multi-hop / negative) 2. Multi-strategy retriever (route by class) 3. Reranker (cross-encoder, top-20 → top-5) 4. Hard refusal path (prompt-level NOT_FOUND, no hedging) 5. Measurement (uuid_hit_rate, hallucination_flag, per strategy)
The classifier doesn’t need a model. A regex pass over [N], total, list all, and does X have Y handles most structured-data queries. When it fails, the fallback is Parent-Child.
The reranker matters more than the chunker. A cross-encoder over a top-20 pool corrected every Q1 failure at Top-K=3 in a follow-up run.
The measurement layer matters most. Without uuid_hit_rate computed at query time, the AST hallucination on Q4 would have shipped. The answer looked correct. It cited a path. It had a number in it. Only ground-truth comparison caught it.
Final Note :
Betting on one chunking strategy is betting all your queries look the same. Ours didn’t. Yours won’t.
The repository has the 530-line JSON, the seven queries, the dashboard code, and every table shown here. Python 3.10. Local embeddings. No paid API key needed to reproduce the retrieval results.
Run the same suite on your own data. The scores will be different. The shape of the failures won’t be.
