Simple string tokenizer
Note: python 3 and cmake is required
- Clone the repo
git clone https://github.com/berazo29/tokenizer.git
- Go inside the project
cd tokenizer
- Build the project with the python script
python build.py
Inside the folder build
there is the executable tokenizer
,
you can pass any string as argument.
Example:
./tokenizer 123+++=+
Output:
Number 1
Number 2
Number 3
increment: "++"
plus equals: "+="
addition: "+"
Distributed under the MIT License. See LICENSE.txt
for more information.