Splits a template source into tokens, with the rules of the Python Jinja2 lexer.

The lexer normalizes \r\n and \r to \n, and removes one trailing newline unless keep_trailing_newline is set. It applies the whitespace control of the tags: - strips all whitespace on that side, + keeps the whitespace that lstrip_blocks or trim_blocks would remove, trim_blocks removes the first newline after a block or comment tag, and lstrip_blocks removes the spaces and tabs before a block or comment tag at the start of a line. Comments do not make tokens, and the content of a raw block becomes text.

A token is {type, value, line}. The types are :data, :variable_begin, :variable_end, :block_begin, :block_end, :name, :string, :integer, :float, :operator and :eof.