Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace Apache Arrow JS with Flechette? #364

Closed
jheer opened this issue Sep 13, 2024 · 1 comment · Fixed by #365
Closed

Replace Apache Arrow JS with Flechette? #364

jheer opened this issue Sep 13, 2024 · 1 comment · Fixed by #365

Comments

@jheer
Copy link
Member

jheer commented Sep 13, 2024

The flechette package provides faster and more flexible Arrow parsing than the JS reference implementation. It also provides a (more correct) implementation of the fast row object proxies used by our Arrow loader here. Should we switch out the apache-arrow tableFromIPC for the flechette version?

If so:

  • We can simply call flechette's version of tableFromIPC with the useProxy: true option to get fast row proxy objects.
  • We can add "duck typing" checks to pass existing tables (from either flechette or arrow-js) through to Vega, calling toArray() on them first to return Vega's desired array-of-objects format.
  • We could also consider adding support to pass-in flechette extraction options (like useBigInt, useMap, etc).

@domoritz any thoughts?

@jheer
Copy link
Member Author

jheer commented Sep 14, 2024

I've also added a PR for this: #365

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant