fe::Lexer< K, S > Class Template Reference

The blueprint for a lexer with a buffer of K tokens to peek into the future (Lexer::ahead). More...

#include <fe/lexer.h>

Public Member Functions
	Lexer (std::istream &istream, const std::filesystem::path *path=nullptr)

Protected Member Functions
char32_t	ahead (size_t i=0) const
void	start ()
	Invoke before assembling the next token.
char32_t	next ()
	Get next char32_t in Lexer::istream_ and increase Lexer::loc_.

Protected Attributes
std::istream &	istream_
Ring< char32_t, K >	ahead_
Loc	loc_
	Location of the token we are currently constructing within Lexer::str_,.
Pos	peek_
	Position of ahead_::first;.
std::string	str_

Accept
Accept next character in Lexer::istream_, depending on some condition.
enum class	Append {   Off ,   On ,   Lower ,   Upper }
template<Append append = Append::On, class Pred>
bool	accept (Pred pred)
template<Append append = Append::On>
bool	accept (char32_t c)
template<Append append = Append::On>
bool	accept (char c)
template<Append append = Append::On>
bool	accept (char8_t c)

Detailed Description

template<size_t K, class S>
class fe::Lexer< K, S >

The blueprint for a lexer with a buffer of K tokens to peek into the future (Lexer::ahead).

You can "override" Lexer::next via CRTP (S is the child).

Definition at line 15 of file lexer.h.

Member Enumeration Documentation

Append

template<size_t K, class S>

enum class fe::Lexer::Append

strongprotected

What should happen to the accepted char? Normalize identifiers via Append::Lower or Append::Upper for case-insensitive languages like FORTRAN or SQL.

Enumerator
Off	Do not append accepted char to Lexer::str_.
On	Append accepted char as is to Lexer::str_.
Lower	Append accepted char via fe::utf8::tolower` to Lexer::str_.
Upper	Append accepted char via fe::utf8::toupper` to Lexer::str_.

Definition at line 64 of file lexer.h.

Constructor & Destructor Documentation

Lexer()

template<size_t K, class S>

fe::Lexer< K, S >::Lexer ( std::istream & istream, const std::filesystem::path * path = nullptr )

inline

Definition at line 21 of file lexer.h.

References istream_, and loc_.

Member Function Documentation

accept() [1/4]

template<size_t K, class S>

template<Append append = Append::On>

bool fe::Lexer< K, S >::accept ( char c )

inlineprotected

Definition at line 89 of file lexer.h.

References accept().

accept() [2/4]

template<size_t K, class S>

template<Append append = Append::On>

bool fe::Lexer< K, S >::accept ( char32_t c )

inlineprotected

Definition at line 88 of file lexer.h.

References accept().

accept() [3/4]

template<size_t K, class S>

template<Append append = Append::On>

bool fe::Lexer< K, S >::accept ( char8_t c )

inlineprotected

Definition at line 90 of file lexer.h.

References accept().

accept() [4/4]

template<size_t K, class S>

template<Append append = Append::On, class Pred>

bool fe::Lexer< K, S >::accept ( Pred pred )

inlineprotected

Returns

true if pred holds. In this case invoke Lexer::next() and append to Lexer::str_, if append.

Definition at line 74 of file lexer.h.

References ahead(), Lower, Off, str_, fe::utf8::tolower(), fe::utf8::toupper(), and Upper.

Referenced by accept(), accept(), and accept().

ahead()

template<size_t K, class S>

char32_t fe::Lexer< K, S >::ahead ( size_t i = 0 ) const

inlineprotected

Definition at line 32 of file lexer.h.

References ahead_.

Referenced by accept().

next()

template<size_t K, class S>

char32_t fe::Lexer< K, S >::next ( )

inlineprotected

Get next char32_t in Lexer::istream_ and increase Lexer::loc_.

Returns

utf8::Invalid on an invalid UTF-8 sequence.

Definition at line 42 of file lexer.h.

References ahead_, fe::utf8::BOM, fe::utf8::decode(), fe::utf8::EoF, istream_, loc_, and peek_.

start()

template<size_t K, class S>

void fe::Lexer< K, S >::start ( )

inlineprotected

Invoke before assembling the next token.

Definition at line 35 of file lexer.h.

References loc_, peek_, and str_.

Member Data Documentation

ahead_

template<size_t K, class S>

Ring<char32_t, K> fe::Lexer< K, S >::ahead_

protected

Definition at line 95 of file lexer.h.

Referenced by ahead(), and next().

istream_

template<size_t K, class S>

std::istream& fe::Lexer< K, S >::istream_

protected

Definition at line 94 of file lexer.h.

Referenced by Lexer(), and next().

loc_

template<size_t K, class S>

Loc fe::Lexer< K, S >::loc_

protected

Location of the token we are currently constructing within Lexer::str_,.

Definition at line 96 of file lexer.h.

Referenced by Lexer(), next(), and start().

peek_

template<size_t K, class S>

Pos fe::Lexer< K, S >::peek_

protected

Position of ahead_::first;.

Definition at line 97 of file lexer.h.

Referenced by next(), and start().

str_

template<size_t K, class S>

std::string fe::Lexer< K, S >::str_

protected

Definition at line 98 of file lexer.h.

Referenced by accept(), and start().

---

The documentation for this class was generated from the following file:

• fe/lexer.h