Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatic schema detection #14

Open
pwoods25443 opened this issue Oct 7, 2017 · 1 comment
Open

Automatic schema detection #14

pwoods25443 opened this issue Oct 7, 2017 · 1 comment

Comments

@pwoods25443
Copy link
Member

Detect the schema automatically for messages from the source table or query.

For the table, this is easy, just use the Bigquery API and get the schema. For a query, some possible things to try

  • Create a view with the query, get the schema of the view, then delete the view
  • Run the query and put the output into a temporary table. Use this table as the input, and delete the table when you're done
  • figure out how to get the schema out of io.gcp.bigquery.BigQuerySource. It is retrieved in the reader when the first row of data comes back, so if we can get it out of there and make it available to the later stages, that will be nice
@pwoods25443
Copy link
Member Author

This is currently working if you specify a table as the source, but not if you specify a query or a file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant