-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add setting to customise delimiters #26
Add setting to customise delimiters #26
Conversation
parser/scanner.go
Outdated
// Scan breaks a string into a sequence of Tokens. | ||
func Scan(data string, loc SourceLoc) (tokens []Token) { | ||
func Scan(data string, loc SourceLoc, delims []byte) (tokens []Token) { | ||
// Configure the token matcher to respect the delimeters passed to it |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
“delimeters” → “delimiters”
This looks like a reasonable way to do this. I added a benchmark to the project; it shows no performance penalty for this. Also using it on the Jekyll doc sources didn't show a measurable effect. [I also tried to remove the AppVeyor check from CI. Since master doesn't pass, and has never passed, this is just adding noise.]
|
Please ignore that last commit, I'm editing it to support full strings instead of bytes right now, I'll post benchmarks when I think I'm done |
I added support for delimiters of any length. I had to mess up the regex expression building a little bit as the
If you are worried about the complexity of this I think it might be prudent to provide a default case with a hardcoded regex, so if the defaults are being used (as they are in 99.9% of cases) the hardcoded expression is used. |
I am finished working on this PR unless you have any concerns that you'd like me to address. |
Great, looks good to go! |
I would like to be able to customise the delimiters I use in templating (this is intended as an analog of https://golang.org/pkg/text/template/#Template.Delims).
This PR has the following problems
tokenMatcher
regex into the function call will dramatically slow things down.I did it like this so I didn't have to change the existing code much.
I am happy to continue work on this, but I'm lodging a PR to see if you're willing to accept this sort of functionality, if I'm going about implementing in the right way, and to check if I haven't missed some other locations where the delimiters are hardcoded.