update qat doc. #2366
update qat doc. #2366
Conversation
Signed-off-by: lkk <[email protected]>
There was a problem hiding this comment.
Pull request overview
This PR updates the QAT (Quantization-Aware Training) documentation and examples by removing an outdated quantization script and updating the README with corrected instructions.
Key Changes:
- Removed the standalone
quantize_autoround.pyscript - Updated README documentation to reference the centralized auto_round example instead
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| quantize_autoround.py | Removed outdated standalone quantization script |
| README.md | Updated Step 2 instructions with corrected command and reference to centralized auto_round example |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ##### Step 2: | ||
|
|
||
| Save the model directly to a get post training quantization model with using [auto-round](https://github.com/intel/auto-round). | ||
| Save the model directly to a get post training quantization model with following this example [auto_round |
There was a problem hiding this comment.
The phrase 'to a get post training' contains grammatical errors. It should be 'to get a post-training' or 'to get post-training'.
| Save the model directly to a get post training quantization model with following this example [auto_round | |
| Save the model directly to get a post-training quantization model by following this example [auto_round |
examples/pytorch/nlp/huggingface_models/language-modeling/quantization/llm_qat/README.md
Outdated
Show resolved
Hide resolved
Signed-off-by: lkk <[email protected]>
…tization/llm_qat/README.md Co-authored-by: Copilot <[email protected]>
for more information, see https://pre-commit.ci
|
|
||
| ``` | ||
| python quantize_autoround.py | ||
| CUDA_VISIBLE_DEVICES=0 python ../auto_round/quantize.py \ |
There was a problem hiding this comment.
|
|
||
| This section walks through an end-to-end example based on the provided code and examples in: | ||
|
|
||
| `examples/pytorch/nlp/huggingface_models/language-modeling/quantization/llm_qat/` |
There was a problem hiding this comment.
It would be better to add a link.
|
|
||
| `requirements.txt` includes (among others): | ||
|
|
||
| - `auto-round==0.8.0` |
| --model vllm \ | ||
| --model_args pretrained=./llama3.1-finetuned-qat,\ | ||
| tensor_parallel_size=1,data_parallel_size=1,\ | ||
| gpu_memory_utilization=0.3,max_model_len=32768,enforce_eager=True \ |
There was a problem hiding this comment.
gpu_memory_utilization is quite low,
enforce_eager cause poor perf.
| eval_size: int = 0 | ||
| ``` | ||
|
|
||
| 4. **QuantizationArguments** |
There was a problem hiding this comment.
Duplicate QuantizationArguments with the one at L144?
3 participants
Description
update qat example/api doc.