LMDeploy

LMDeploy, Star Count is a high performance inference framework for LLMs.