Improving Documentation & Examples for llama-cpp-python – Looking for Volunteers

[JamePeng]

Hi everyone,

I'm JamePeng, the current maintainer of this fork (JamePeng/llama-cpp-python).

First of all, a big thank you to abetlen for creating the original llama-cpp-python project — it laid a solid foundation that many of us still rely on today.

The Current Situation

As the llama.cpp backend continues to evolve rapidly, our high-level Python bindings have kept pace with many exciting new features. However, the official documentation (especially in the /docs/ folder and ReadTheDocs) has become quite outdated. Many class usages, parameter explanations, and code examples no longer reflect the current API.

This has caused real friction for developers:

• New users often struggle with deprecated patterns.
• Advanced features are hard to discover.
• The main README.md has grown very large and is becoming difficult to search and navigate.

Recent Major Improvements in This Fork

In the past few months, we've added or significantly enhanced:

• Core refactoring: Overhauled generate() / eval() for better hybrid model support and new LlamaSampler chain API.
• Caching improvements and advanced sampling options.
• Dynamic LoRA Routing & Control Vectors (Multi-Tenant Serving)
• Strong multi-modal support via modern Chat Handlers:
  • Enhanced Qwen35ChatHandler and Qwen3.6 template (with preserve_thinking support)
  • Full Gemma-4 ChatHandler (vision + audio for E2B/E4B models, vision+text for others)
  • MTMD (Multi-Modal Transformer) style models through custom chat handlers
  • Better support for Qwen-VL, Gemma-3/4, GLMV, Step3-VL, and various other vision/audio models
• Embeddings & Reranking (GGUF): provides a high-performance, memory-efficient specialized class LlamaEmbedding for generating text embeddings and calculating reranking scores.
• Ongoing updates to keep the llama.cpp submodule current.

These powerful features deserve clearer, more up-to-date documentation and richer examples.

Personal Note & Call for Help

Maintaining this project is a one-person effort on my side. Between a busy day job, frequent migraines, and limited free time, it has become increasingly difficult to keep all files — especially the documentation — comprehensively updated and polished.

I’m seriously considering a complete overhaul of the docs/ section using an LLM Wiki approach: turning the documentation into a living, structured, LLM-maintained wiki that can stay current more easily.

Proposal: Let's Build Better Documentation Together

I’d love to open this up to the community and move toward:

• A well-structured, always-up-to-date wiki with clear class references
• Rich, copy-paste-ready examples (basic usage, chat, vision, function calling, embeddings, etc.)
• Dedicated pages for new features like multi-modal handlers, caching, and sampling
• Clear migration guides for outdated patterns
• Better organization so information is easy to find

Are you interested in helping?
Even small contributions would make a big difference:

• Writing or reviewing examples
• Helping design the wiki structure
• Testing examples with latest models (Qwen3.5, Gemma-4, TTS, etc.)
• Improving installation guides or troubleshooting
• Reviewing or editing wiki pages

If you’d like to contribute, please reply here and let me know:

• What area you’d be interested in helping with
• Any pain points you’ve experienced with the current documentation

Together we can make llama-cpp-python not only technically strong but also much more approachable and enjoyable to use.

Thank you in advance for any support or ideas!

Best regards,
JamePeng

[alcoftTAO]

Hi! I'd love to contribute to the multimodal documentation, since it's the part of the code I've looked more.

I haven't got any problems with the current documentation, however, I think it will be easier for most people to implement a generic multimodal chat handler; one for all models rather than one for each model/architecture.
I've been thinking about this recently and would like to implement this to the code.

I think this is possible, since all of the current chat handlers uses the same base class and just changes the chat template.
The chat template for each model could be gathered from the GGUF metadata.