# `Otera.Lexer`
[🔗](https://github.com/georgeguimaraes/otera/blob/v0.1.0/lib/otera/lexer.ex#L1)

Splits a template source into tokens, with the rules of the Python Jinja2 lexer.

The lexer normalizes `\r\n` and `\r` to `\n`, and removes one trailing newline unless `keep_trailing_newline` is set. It applies the whitespace control of the tags: `-` strips all whitespace on that side, `+` keeps the whitespace that `lstrip_blocks` or `trim_blocks` would remove, `trim_blocks` removes the first newline after a block or comment tag, and `lstrip_blocks` removes the spaces and tabs before a block or comment tag at the start of a line. Comments do not make tokens, and the content of a raw block becomes text.

A token is `{type, value, line}`. The types are `:data`, `:variable_begin`, `:variable_end`, `:block_begin`, `:block_end`, `:name`, `:string`, `:integer`, `:float`, `:operator` and `:eof`.

# `token`

```elixir
@type token() :: {atom(), term(), pos_integer()}
```

# `tokenize`

```elixir
@spec tokenize(String.t(), keyword()) :: [token()]
```

Returns the tokens of `source`. Raises `Otera.SyntaxError`.

Options: `:trim_blocks`, `:lstrip_blocks` and `:keep_trailing_newline`, all `false` by default.

---

*Consult [api-reference.md](api-reference.md) for complete listing*