Investigate replacement for syntect (syntax highlighting)

[Keats]

Previous thread (#1787) was focused on tree-sitter but it might turn out to not be the best choice due to size (some syntaxes are 90MB+) and slowness to load (which is planned to be fixed eventually).

An alternative would be to write a textmate parser so we can re-use all of VSCode syntaxes/themes since it has a big momentum and should not be too hard to write. I have a branch with just the serde structs in https://github.com/getzola/giallo/tree/tm
We can use https://github.com/rust-onig/rust-onig for the regex library, like syntect is doing.

[si14]

Fwiw I had a look at about a dozen grammars, and all of them were using very simple regexes. I believe the regex crate should be able to handle them, maybe with a translation layer if there are any exceptions?

[Keats]

Most of the tm grammars are written with onig in mind, regex doesn't have look around or backtracking which is definitely used in some places

[si14]

Fair, I double checked and realised I didn't pay enough attention, there are plenty of ?<= and ?=. Sorry for the noise, I was wrong.

[nrdxp]

re: #1787 (comment)

Personally I haven't seen a compelling difference between tree-sitter and textmate for code snippets

This is probably more true with popular languages with well maintained grammars. A good counter-example would be the Nix language, which I haven't seen a decent highlighter for, except in the tree-sitter realm.

[Keavon]

Should this thread be given a Help Want label to attract contributors?

[Popolon]

Why not have the options between proposed highlighters, instead of only one ? Isn't some license problems with VScode syntax, due to MS politics, that would force people to create new ones, with possible legal issues?

This blog speak about pygments support in Zola, that is not the fast, but has a large support of languages and would avoid any legal issues in FLOSS usage: https://c.pgdm.ch/notes/zola-pygments/

There is also pygments for rust: https://github.com/Alignof/pygments-rs

[Keats]

Why not have the options between proposed highlighters, instead of only one ?

Too much work and inconsistencies.

Isn't some license problems with VScode syntax, due to MS politics, that would force people to create new ones, with possible legal issues

Can you expand on that? Most of the syntaxes i've seen are MIT licensed.

This blog speak about pygments support in Zola
There is also pygments for rust: https://github.com/Alignof/pygments-rs

If you look at that repo, you'll see there isn't a single Rust file. Probably just a fork of the original pygments codebase. I've actually started implementing pygments in Rust years ago for Zola but stopped because the output is nowhere near as nice as syntect/vscode/tree-sitter.

IMO the right play is to port https://github.com/microsoft/vscode-textmate to Rust just to benefit from up to date syntaxes and the super active community there. It's not a crazy amount of work except for the fact they seem to really not like to use comments to explain what's going on. Honestly if no one does it before me, I will likely do it but it's not going to be in any reasonable time frame x)