VLM: Model Tracing Guide

[kylesayrs]

Purpose

This guide explains the concepts of tracing as they relate to LLM Compressor and how to modify your model to support recipes which require using the Sequential Pipeline.

Through reading this guide, you will learn

1. Why tracing is required when compressing with recipes involving the Sequential Pipeline and modifiers such as GPTQModifier
2. How to determine if your model is traceable for your dataset
3. How to modify your model definition to be traceable

Prerequisites

• Explicit dataset tokenizer text kwarg #1031

Changes

• Add a model tracing guide src/llmcompressor/transformers/tracing/README.md with pictures
• Add a readme for the sequential pipeline which points to the Tracing Guide src/llmcompressor/pipelines/sequential/README.md
• Add a debug script to help users debug their models for traceability src/llmcompressor/transformers/tracing/debug.py
  • Add the llm-compressor.attempt_trace entrypoint for ease of use
• Swap the order of arguments in llava_example.py and and pixtral_example.py to match the order of arguments on the modifier

Testing

Use the llmcompressor.attempt_trace debug script

llmcompressor.attempt_trace \
    --model_id llava-hf/llava-1.5-7b-hf
    --model_class TraceableLlavaForConditionalGeneration
    --sequential-targets LlamaDecoderLayer
    --ignore "re:.*lm_head" "re:vision_tower.*" "re:multi_modal_projector.*"
    --multimodal_data

Stretch

It might be nice if this tracing debugger tool also printed the model graph to an svg

[horheynm]

We have click in the setup.py, might be worth using for cli

[kylesayrs]

I don't really see a good reason to https://click.palletsprojects.com/en/stable/why/#why-not-argparse, but thanks for the suggestion

[horheynm]

I read the whole thing.

I like how much time and thought you put into making this doc.

Right now, the audience needs to read until 3rd paragraph to know what the problem is and when to use the tracing -- encoder-decoder models using GPTQ, SparseGPT Modifiers. If we move those to the intro, it will be clearer for the audience to know if the doc is applicable to them or not.

Then a small paragraph introducing what 1, 2, and 3 will be helpful for --
1 shows the description of why it cannot use the previous methods and why the seq pipeline solves the problem, 2. is how to run using cli, and 3. is debugging/contribution.

This way I think the audience can have an easier time navigating to the appropriate section by reading less.

[kylesayrs]

Right now, the audience needs to read until 3rd paragraph to know what the problem is and when to use the tracing

As for when to use tracing, that's described in the second sentence

Through reading this guide, you will learn
1. Why tracing is required when compressing with recipes involving the Sequential Pipeline and modifiers such as GPTQModifier

As for what the problem is, that's described in the first section

## 1. Why is Tracing Required? ##

[kylesayrs]

Right now, the audience needs to read until 3rd paragraph to know what the problem is and when to use the tracing -- encoder-decoder models using GPTQ, SparseGPT Modifiers

That's incorrect, tracing is used for all model architectures, not just encoder-decoder models. As described in the second paragraph, tracing is used when compressing with recipes involving the Sequential Pipeline and modifiers such as GPTQModifier.

[kylesayrs]

Then a small paragraph introducing what 1, 2, and 3 will be helpful for
This way I think the audience can have an easier time navigating to the appropriate section by reading less.

I think the section titles + the list of things you will learn from reading each of the sections is enough context for a reader to go on. For example, if the reader doesn't care about the why, they can skip 1. If the reader doesn't care about what tracability is, they can skip 2. If the reader doesn't care about how to make a model traceable, they can skip 3.

[mgoin]

Great work, we should consider adding a readthedoc build like vLLM to render these out

[dsikka]

Great job.

A couple of nits:

1. I wouldnt refer to the SparseGPTModifier until we've actually started using data pipelines outside of the GPTQModifier
2. A helpful comment on what to focus on when looking at the images would be nice

[dsikka]

Because we're not yet using the data pipeline in the SparseGPTModifier, I would not include it in the ReadMe just yet

[dsikka]

What am I supposed to be taking away from this image?

[kylesayrs]

This image depicts the model graph referenced in the above paragraph. The image is a concrete example of what a model graph looks like and helps illustrated what the nodes and edges are within the graph.

[dsikka]

grammar their hessians be all be allocated ...

[dsikka]

docstring