happyz
/
llama.cpp
mirror of https://github.com/ggerganov/llama.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

docs: update llama-eval-discussion.md with session work summary 

Add summary of llama-server-simulator implementation work including
features, testing results, technical decisions, and refactoring.
This commit is contained in:
Georgi Gerganov 2026-01-31 15:49:43 +02:00
parent 23d4e21a81
commit c87af1d527
No known key found for this signature in database
GPG Key ID: 449E073F9DC10735
1 changed files with 36 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

36
examples/llama-eval/llama-eval-discussion.md
View File

@ -114,3 +114,39 @@ Questions:
## References
- PR #18892: https://github.com/ggml-org/llama.cpp/pull/18892
- Discussion #18195: https://github.com/ggml-org/llama.cpp/discussions/18195
## Session Work Summary
### llama-server-simulator Implementation
**Created:**
- `llama-server-simulator.py` - Standalone Python script simulating llama-server HTTP endpoint
- `test-simulator.sh` - Test script for verifying simulator functionality
- `llama-server-simulator-plan.md` - Implementation plan
- `simulator-summary.md` - Summary of implementation
**Features Implemented:**
1. HTTP Server - Flask-based `/v1/chat/completions` endpoint with OpenAI-compatible format
2. AIME Dataset Integration - Loads 90 questions from HuggingFace with automatic local caching
3. Intelligent Question Matching - Uses exact matching, LaTeX removal, and Levenshtein distance
4. Response Generation - Configurable success rate (0-1) for correct/wrong answer generation
5. Debug Logging - Helps troubleshoot matching issues
**Testing Results:**
- ✅ Correct answers returned when success rate allows
- ✅ Wrong answers returned when success rate doesn't allow
- ✅ No matching questions return errors
- ✅ Success rate verified (80% in 10 requests)
- ✅ HuggingFace dataset caching working correctly
**Key Technical Decisions:**
- Used Levenshtein distance for partial matching (threshold: 0.3)
- Automatic caching via HuggingFace datasets library
- Wrong answers generated by incrementing expected answer
- Debug output written to stderr for better visibility
**Refactoring:**
- Extracted repeating question string into TEST_QUESTION variable
- Created make_request() helper function to reduce code duplication
- Added proper error handling for error responses
- Fixed simulator stopping issue at script completion