fix(idp-extraction-connector): remove double download in idp connector

[reiballa]

Description

I removed the double document download that happened because i used the asByteArray method. I reworked my solution to just use the input stream.

Related issues

closes #3868

Checklist

• PR has a milestone or the no milestone label.

[sbuettner]

Hey @reiballa, I took a quick look at it seems like we are reading the full stream to get the content length and then upload it which doesnt use a streaming approach as everything will be stored in-memory.

I did a quick research and the transfer manager which should support streaming uploads seems to be available now as of: aws/aws-sdk-java#474 (comment)

Developer docs: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/transfer-manager.html

[reiballa]

@sbuettner Are you suggesting we look into updating the library? Because i checked what we had now, and ommiting the contentLength wasnt possible:
This is from AsyncRequestBodyFromInputStreamConfiguration

private AsyncRequestBodyFromInputStreamConfiguration(DefaultBuilder builder) {
        this.inputStream = (InputStream)Validate.paramNotNull(builder.inputStream, "inputStream");
        this.contentLength = Validate.isNotNegativeOrNull(builder.contentLength, "contentLength");
        this.maxReadLimit = Validate.isPositiveOrNull(builder.maxReadLimit, "maxReadLimit");
        this.executor = (ExecutorService)Validate.paramNotNull(builder.executor, "executor");
}

[sbuettner]

@reiballa

Are you suggesting we look into updating the library?

Yeah, exactly. I think it provides what you are looking for to stream without needing to have the content length beforehand.