[Question]: Poorer Performance when Using ComposableGraph

[tianqizhao-louis]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Hi all!

I have a question regarding using ComposableGraph, because I saw a poor performance when using it. Maybe it's because I'm using it wrong?

I loaded Paul Graham's essays What I Worked On, Fierce Nerds, and Crazy New Ideas into essay1_index, essay2_index, and essay3_index like this:

essay1_doc = SimpleDirectoryReader("data").load_data()

# bge-base embedding model
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-base-en-v1.5")

# ollama
Settings.llm = Ollama(model="llama3.1:8b", request_timeout=360.0)

essay1_index = VectorStoreIndex.from_documents(
    essay1_doc,
)

essay1_query = essay1_index.as_query_engine()
response = essay1_query.query("What did the author do growing up?")
print(response)

essay2_doc = SimpleDirectoryReader("data2").load_data()

essay2_index = VectorStoreIndex.from_documents(
    essay2_doc,
)

essay2_query = essay2_index.as_query_engine()
response = essay2_query.query("What are the good news and bad news mentioned by the author?")
print(response)

essay3_doc = SimpleDirectoryReader("data3").load_data()

essay3_index = VectorStoreIndex.from_documents(
    essay3_doc,
)

essay3_query = essay3_index.as_query_engine()
response = essay3_query.query("What is the opinion that the author is very afraid to express publicly?")
print(response)

The responses are closely related to the essays I fed. However, when I constructed a graph index:

graph_index = ComposableGraph.from_indices(
    SummaryIndex,
    [essay1_index, essay2_index, essay3_index],
    index_summaries=[
        "Paul Graham's Essay: What I Worked On",
        "Paul Graham's Essay: Fierce Nerds",
        "Paul Graham's Essay: Crazy New Ideas",
    ],
)

graph_query_engine = graph_index.as_query_engine()
response = graph_query_engine.query(
    "What are the good news and bad news mentioned by the author?"
)
print(response)

The responses are making no sense.

Any idea why? Would appreciate anyone who has been using ComposableGraph sharing their experiences!

Thanks ahead!

[dosubot]

Hey, @tianqizhao-louis! I'm here to help you with any bugs, questions, or contributions while you wait for a human maintainer. Let's take a look at the issue you're facing with the ComposableGraph.

When using ComposableGraph, there are a few considerations that might help improve the performance and relevance of the responses:

1. Index Summaries: Ensure that the summaries provided for each index are accurate and concise. These summaries are used by the ComposableGraph to understand the context of each index, which can affect the quality of the responses ^[1].

2. Recursive Querying: Check if the ComposableGraphQueryEngine is set up to handle recursive queries. This feature allows the engine to traverse through the graph and query sub-indices recursively, which can provide more comprehensive results ^[2].

3. Model Configuration: Verify that the LLM and embedding models are correctly configured and suitable for the type of content you are querying. Different models may have varying performance based on the data and query types.

4. Debugging and Logging: Utilize the debugging and logging features available in LlamaIndex to trace the query process and identify any potential issues. This can help in understanding how the queries are being processed and where they might be going wrong ^[3].

5. Query Specificity: Ensure that the queries are specific enough to guide the ComposableGraph in retrieving relevant information. Vague or overly broad queries might lead to less accurate responses.

By addressing these areas, you may be able to improve the performance and relevance of the responses when using ComposableGraph.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[tianqizhao-louis]

@dosu what's the best way of providing accurate and concise summary of indexes? are there existing ways to do that?

[dosubot]

To provide accurate and concise summaries for your indexes, consider the following approaches:

1. Manual Summarization: Read through the documents and manually create a summary that captures the main themes and key points. This ensures that the summary is tailored to the content and context of the documents.

2. Automated Summarization Tools: Use natural language processing tools or libraries that offer summarization capabilities. Tools like Hugging Face's Transformers library provide models that can generate summaries from text.

3. Key Points Extraction: Identify and list key points or topics covered in the documents. This can be done manually or with the help of text analysis tools that highlight important sentences or phrases.

4. Iterative Refinement: Start with a draft summary and refine it based on the responses you get from the ComposableGraph. Adjust the summary to better align with the information you want the graph to focus on.

5. Peer Review: If possible, have someone else review your summaries to ensure they are clear and accurately represent the content.

By using these methods, you can create summaries that effectively guide the ComposableGraph in understanding and retrieving relevant information from your indexes.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[tianqizhao-louis]

@dosu doesn't work though. tried all approaches mentioned above

[dosubot]

If the approaches for summarizing indexes aren't yielding the desired results, consider these additional strategies:

1. Contextual Summaries: Ensure that each summary not only captures the main points but also provides context about how the document relates to others in the graph. This can help the ComposableGraph understand the relationships between different indices.

2. Iterative Testing: Experiment with different levels of detail in your summaries. Sometimes, more detailed summaries can provide better guidance for the graph, while in other cases, more concise summaries might work better.

3. Feedback Loop: Use the responses from the ComposableGraph to iteratively refine your summaries. Analyze where the responses deviate from expectations and adjust the summaries to address these gaps.

4. Model Fine-Tuning: If possible, fine-tune your language model on a dataset similar to your documents. This can help the model better understand the nuances of your content, leading to more accurate summaries and responses.

5. Consultation with Experts: If the documents are specialized, consulting with subject matter experts to create summaries might provide insights that automated tools or general approaches miss.

6. Alternative Index Structures: Consider whether a different index structure or combination of indices might better suit your needs. Sometimes, restructuring how documents are indexed can lead to improved performance.

These strategies can help enhance the effectiveness of your summaries and improve the performance of the ComposableGraph.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}