* gguf-py: Bump sentencepiece version
There's a new version that's been out for a while that addresses the issues mentioned in https://github.com/ggml-org/llama.cpp/pull/14200. There's a long chain of reasons I would like this change, but the short version is that it allows people who use both `sentencepiece` and `gguf` to take advantage of these fixes. On conda-forge, currently, it locks the version (since there is no notion of optional dependencies).
Regardless, I don't think this should be too controversial.
* review feedback
- gguf-py : remove gguf-py/gguf/scripts/__init__.py because it's not needed
Implicit namespaces are supported since Python 3.3 (https://peps.python.org/pep-0420/),
and the entrypoints in pyproject.toml can directly refer to the main functions.
* style: format with nixfmt/rfc101-style
* build(nix): Package gguf-py
* build(nix): Refactor to new scope for gguf-py
* build(nix): Exclude gguf-py from devShells
* build(nix): Refactor gguf-py derivation to take in exact deps
* build(nix): Enable pytestCheckHook and pythonImportsCheck for gguf-py
* build(python): Package python scripts with pyproject.toml
* chore: Cleanup
* dev(nix): Break up python/C devShells
* build(python): Relax pytorch version constraint
Nix has an older version
* chore: Move cmake to nativeBuildInputs for devShell
* fmt: Reconcile formatting with rebase
* style: nix fmt
* cleanup: Remove unncessary __init__.py
* chore: Suggestions from review
- Filter out non-source files from llama-scripts flake derivation
- Clean up unused closure
- Remove scripts devShell
* revert: Bad changes
* dev: Simplify devShells, restore the -extra devShell
* build(nix): Add pyyaml for gguf-py
* chore: Remove some unused bindings
* dev: Add tiktoken to -extra devShells
Alex Trotta
compilade
Georgi Gerganov
Tushar