feat(RELEASE-1246): make publish-to-cgw task idempotent

[seanconroy2021]

Describe your changes

This update makes the publish-to-cgw task idempotent by checking if files already exist under the specified product name and version before processing. If files are present, they are skipped to avoid errors. Additionally, the pubtools-content-gateway command has been removed and we are now calling the CGW api directly.

Relevant Jira

CGW Docs

Checklist before requesting a review

• I have marked as draft or added do not merge label if there's a dependency PR
  • If you want reviews on your draft PR, you can add reviewers or add the release-service-maintainers handle if you are unsure who to tag
• [* ] My commit message includes Signed-off-by: My name <email>
• [ *] I have bumped the task/pipeline version string and updated changelog in the relevant README
• [ *] I read CONTRIBUTING.MD and commit formatting

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[seanconroy2021]

I still need to add my tests, but I'm nearly there. 😌

[johnbieren]

@pkhander @swickersh can you take a look at this change please?

[johnbieren]

It looks sane to me, but you will need to add test(s) for this change I think

[johnbieren]

s/if files/if files are/

[pkhander]

I would like to point out that there is another (copy) script that interacts with CGW for another pipeline - https://github.com/konflux-ci/release-service-utils/blob/main/developer-portal-wrapper/developer_portal_wrapper.py

Please keep that in mind with this work. Not ALL publishing to cgw will be idempotent when this is merged.

Ideally just the developer_portal_wrapper should be updated and used inside this task but I will let you decide the implementation details.

[mmalina]

I would like to point out that there is another (copy) script that interacts with CGW for another pipeline - https://github.com/konflux-ci/release-service-utils/blob/main/developer-portal-wrapper/developer_portal_wrapper.py

Please keep that in mind with this work. Not ALL publishing to cgw will be idempotent when this is merged.

Ideally just the developer_portal_wrapper should be updated and used inside this task but I will let you decide the implementation details.

Not having Python code directly in the task script would also help with testing - when Python is used directly, it's not possible to use our task testing framework and you can't do Python unit testing this way either. So I agree that switching to developer_portal_wrapper sounds like a good idea.

[seanconroy2021]

I would like to point out that there is another (copy) script that interacts with CGW for another pipeline - https://github.com/konflux-ci/release-service-utils/blob/main/developer-portal-wrapper/developer_portal_wrapper.py
Please keep that in mind with this work. Not ALL publishing to cgw will be idempotent when this is merged.
Ideally just the developer_portal_wrapper should be updated and used inside this task but I will let you decide the implementation details.

Not having Python code directly in the task script would also help with testing - when Python is used directly, it's not possible to use our task testing framework and you can't do Python unit testing this way either. So I agree that switching to developer_portal_wrapper sounds like a good idea.

Sweet, I will just move it over there, then ... and use pytest
I did find an interesting way to test in mock.sh by mocking each request, e.g. /products

[swickersh]

Sweet, I will just move it over there, then ... and use pytest

It wont be that simple. The two have deviated slightly.
For example, the matching_component logic, the short_url_prefix, argparse, etc.

developer_portal_wrapper.py was created as a modified version of the python section nested in this task to work with rhel_ai. We want to be careful not to break either workflow when combining these.

IMO, you're running into some scope creep here. You might ask to increase your jira story points and allow some capacity for testing. Maybe these should be separate commits?

Alternatively, you could make another developer_portal_wrapper.py (slight different name) to work with this workflow. Keep the existing one for rhel-ai to continue utilizing. Then in a future Jira we can make a unified script that works with both.

@mmalina @pkhander wdyt?