webllm python utility
This commit is contained in:
parent
767d6686ee
commit
ee66e9fdfa
1 changed files with 54 additions and 2 deletions
|
|
@ -20,8 +20,6 @@ No WebGPU → no in-browser LLM can run in this browser.
|
|||
Use a recent Chrome/Edge (≥ 113) or Safari 18+ (chrome://flags/#enable-unsafe-webgpu on Linux).
|
||||
```
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
- [Homepage du machine learning compiler (MLC)](https://llm.mlc.ai/docs/deploy/webllm.html)
|
||||
|
|
@ -30,4 +28,58 @@ Use a recent Chrome/Edge (≥ 113) or Safari 18+ (chrome://flags/#enable-unsafe-
|
|||
|
||||
---
|
||||
|
||||
- [Homepage Webllm](https://webllm.org/)
|
||||
- [Le projet WebLLM](https://github.com/mlc-ai/web-llm)
|
||||
|
||||
|
||||
## install webllm
|
||||
|
||||
|
||||
[Suivre l'install](https://github.com/mlc-ai/web-llm) dans le readme.
|
||||
|
||||
|
||||
## install the python mlc local chat utility
|
||||
|
||||
### install
|
||||
|
||||
L'install doit se faire avec des version très précises sinon il y a des difficultés.
|
||||
|
||||
- faire un environnement virtuel
|
||||
- installer de la manière suivante :
|
||||
|
||||
```
|
||||
sudo apt-get install git-lfs
|
||||
|
||||
pip install --pre -f https://mlc.ai/wheels \
|
||||
mlc-ai-cpu==0.20.0 \
|
||||
mlc-llm-cpu==0.20.0.dev0
|
||||
|
||||
|
||||
pip install psutil
|
||||
|
||||
# wget https://github.com/mlc-ai/package/releases/download/v0.9.dev0/mlc_llm_cpu-0.20.0.dev0-py3-none-manylinux_2_28_x86_64.whl
|
||||
# pip install ./mlc_llm_cpu-0.20.0.dev0-py3-none-manylinux_2_28_x86_64.whl
|
||||
|
||||
pip install apache-tvm-ffi==0.1.11
|
||||
# https://pypi.org/project/apache-tvm-ffi/#history
|
||||
```
|
||||
|
||||
### usage
|
||||
|
||||
```
|
||||
python -m mlc_llm chat HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC
|
||||
```
|
||||
|
||||
```
|
||||
You can use the following special commands:
|
||||
/help print the special commands
|
||||
/exit quit the cli
|
||||
/stats print out stats of last request (token/sec)
|
||||
/metrics print out full engine metrics
|
||||
/reset restart a fresh chat
|
||||
/set [overrides] override settings in the generation config. For example,
|
||||
`/set temperature=0.5;top_p=0.8;seed=23;max_tokens=100;stop=str1,str2`
|
||||
Note: Separate stop words in the `stop` option with commas (,).
|
||||
Multi-line input: Use escape+enter to start a new line.
|
||||
```
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue