Commit Graph
5 Commits
Author SHA1 Message Date
Simo LinandChang Su 5fbad308cd [router] add tokenizer chat template support (#9370)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-19 20:14:02 -07:00
Simo LinandChang Su 0b95a01a8f [router] add tiktokenizer and sequence in router (#9354)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-19 10:46:28 -07:00
Simo LinandChang Su 24247b4168 [router] add tokenizer metrics (#9307)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-18 09:25:51 -07:00
Simo LinandChang Su d08663eec1 [router] tokenizer factory, hf tokenizer, and stop sequence detector (#9293)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-17 22:38:38 -07:00
Simo Lin ff0cf51c8e [router] introducing tokenizer trait (#9287) 2025-08-17 16:30:01 -07:00