Reusing parsed lxml trees #10

levic · 2023-02-06T07:45:04Z

While it's possible to use Prefix to narrow the scope of what you're parsing, that really only works if the shape of the data is predictable ahead of time.

It doesn't appear to be possible to avoid causing lxml to reparse the entire document when handling children.

An example:

data.html:

<html>
  <body>

    <div class="category">
      <h1>Fruit</h1>
      <p><div class="count">3</div>results</p>
      <div class="product">Apple</div>
      <div class="product">Pear</div>
      <div class="product">Orange</div>
    </div>

    <div class="category">
      <h1>Vegetables</h1>
      <p><div class="count">2</div>result</p>
      <div class="product">Potato</div>
      <div class="product">Pumpkin</div>
    </div>

  </body>
</html>

#!/usr/bin/env python3
from lxml.etree import tostring

from xextract import Element, String

html = open('data.html').read()

category_elements = Element(css='.category').parse(html)
for category_element in category_elements:
    # this does not work because XPathExtractor.get_root()
    # assumes body is a string
    #extractor = HtmlXPathExtractor(category_element)

    # this works because `BaseParser.parse()` has a special case for
    # `body` values that are already an `XPathExtractor`
    # but it is inefficient: we convert an lxml element into a string
    # and then re-parse it
    extractor = HtmlXPathExtractor(tostring(category_element))

    category_name = String(css='h1', quant=1).parse(extractor)
    product_count = int(String(css='.count', quant=1).parse(extractor))

    # note that `product_count` here was dynamically extracted from the html
    product_names = String(css='.product', quant=product_count).parse(extractor)

    print(f"Category '{category_name}' contains {product_count} products:")
    for product_name in product_names:
        print(f"  {product_name}")

The important part here is that we want to validate that the number of products matches the product count string. For a file this small it obviously doesn't make much a difference, but consider a very large file.

One workaround is to create a custom parser:

import lxml.etree
class ElementHtmlXPathExtractor(HtmlXPathExtractor):
    def _get_root(self, body):
        if isinstance(body, lxml.etree._Element):
            return body
        return super()._get_root()


# and then in our loop we can do:
for category_element in category_elements:
    ...
    extractor = ElementHtmlXPathExtractor(category_element)
    ...

It would be nice if the _get_root() special case check for _Element was integrated into XPathExtractor.

The text was updated successfully, but these errors were encountered:

levic mentioned this issue Feb 6, 2023

Allow Elements to be passed to parse_*() #11

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reusing parsed lxml trees #10

Reusing parsed lxml trees #10

levic commented Feb 6, 2023 •

edited

Loading

Reusing parsed lxml trees #10

Reusing parsed lxml trees #10

Comments

levic commented Feb 6, 2023 • edited Loading

levic commented Feb 6, 2023 •

edited

Loading