Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: When split the data record, the core data records python module still depends on the spark library #532

Open
csun-cpointe opened this issue Jan 15, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@csun-cpointe
Copy link
Contributor

csun-cpointe commented Jan 15, 2025

Description

After create a pyspark pipeline with the combined data record, split the data record, the project can't be built because the under the -shared module directory, the -core-python module still depends on the pyspark dependency for the pysaprk.sql Row import

Steps to Reproduce

  1. Navigate to the regression test script.
  2. Follow the steps for Getting Started, Instantiating the Regression Test Project, and Adding Pipelines, Record, and Dictionary Models
  3. After build complete and no more manual actions are required, follow the instruction to split data record
  4. After correct dependency on the pyspark pipeline pyproject.toml and pom.xm file, you will encounter below error with the build following:
[INFO] Executing Poetry command: poetry run pylint /Users/csun/bah/tests/test-511/test-511-shared/test-511-data-records-core-python/src --disable C,R,W
[INFO] ************* Module test_511_data_records_core_python.generated.record.py_person_base
[ERROR] src/test_511_data_records_core_python/generated/record/py_person_base.py:22:0: E0401: Unable to import 'pyspark.sql' (import-error)
[INFO] 

Expected Behavior

The build should pass without adding the pyspark dependency in the pyproject.toml file.

Actual Behavior

The build fails.

Additional Context

  • Log output
  • Screenshots (if applicable)
  • Solution Baseline Version
  • Environment details (local, cloud, Azure, AWS, etc.)
@csun-cpointe csun-cpointe added the bug Something isn't working label Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant