6 Commits

Author	SHA1	Message	Date
Jan Wassenberg	c8d92948f4	Move fields, io* and blob* from compression/ into io/ ... PiperOrigin-RevId: 755445712	2025-05-06 11:17:19 -07:00
Jan Wassenberg	8d0882b966	Huge refactor of weight handling and model loading. ... Weight handling: - new ModelStore2 supports both pre-2025 multi-file and single-file formats - simpler ForEachTensor with TensorArgs - tensors are constructed with their full suffixed name I/O: - support mmap and stride - Simplified SbsWriter, single insert(); add SbsReader Misc: - kMockTokenizer: allow creating with unavailable tokenizer - configs.h: Simpler enum validity checks via kSentinel - matmul.h: remove unused enable_bind (now in allocator.h) - tensor_info: single TensorInfoRegistry class, rename from tensor_index.h Frontends: - Replace Allocate/CreateGemma with ctor(LoaderArgs, MatMulEnv&) - Deduce model/weight type, remove --model and parsing - Replace most common.h includes with configs.h - Remove --compressed_weights, use --weights instead - Remove ModelInfo, replaced by ModelConfig. Backprop: - Reduce max loss, remove backward_scalar_test (timeout) - Update thresholds because new RandInit changes rng eval order and thus numerics PiperOrigin-RevId: 755317484	2025-05-06 04:44:21 -07:00
Jan Wassenberg	4e6aa36e9b	Minor cleanup: enable 0,0 Extents2D, add SerializedSpan typedef, include fixes ... PiperOrigin-RevId: 745068776	2025-04-08 03:35:55 -07:00
Jan Wassenberg	b18bd781f6	Windows build fixes: struct vs class, unused arg/var, avoid VLA, Deleter arg, casts ... PiperOrigin-RevId: 724340518	2025-02-07 07:38:55 -08:00
Ray Smith	9d40f0117e	Added ability to load/save a complete model file, including tokenizer. ... PiperOrigin-RevId: 707914366	2024-12-19 07:59:41 -08:00
Jan Wassenberg	52af531820	Serialization for class members for use with ModelConfig ... PiperOrigin-RevId: 689720027	2024-10-25 03:12:34 -07:00