-=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- (c) WidthPadding Industries 1987 0|698|0 -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=-
SoCoder -> Snippet Home -> Misc


 
Cower
Created : 25 December 2009
 
Language : Delphi/Pascal

BlitzMax Source Tokenizer

Title just about does it

Bit of code I wrote today for tokenizing BlitzMax code. No parser yet, although you can sort of see where I thought I would put it (the only TODO I bothered to write). Probably going to have to write a TokenReader...

Anyhow, if for some horrifying reason you need to tokenize BlitzMax code, this should do the trick. You might want to comment out the extension bits (in the tuples, they're marked by "(x)", in the single tokens they're under the extensions comment).

This script requires Ruby 1.9 as a minimum due to the way the [] operator is implemented for Strings in Ruby 1.9. Ruby 1.8 will not work, I can guarantee this.


 

Comments