The lexer submodule

Warning

The functions and methods listed in this section are implementation details of CondConfigParser. Do not use them in your programs if you rely on API stability, as they are likely to change in incompatible ways without prior notice.

Lexer module of CondConfigParser.

This module defines a Token class, one subclass for every token type, a TokenType enum.Enum and a Lexer class.

class condconfigparser.lexer.TokenType(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: AutoNumber

Identifier objects for token types.

newline = 1

varAssignmentsStart = 2

varAssignmentsEnd = 3

predicateStart = 4

predicateEnd = 5

listStart = 6

listEnd = 7

openParen = 8

closeParen = 9

orOp = 10

andOp = 11

notOp = 12

equalsOp = 13

notEqualsOp = 14

inOp = 15

assignOp = 16

comma = 17

true = 18

false = 19

stringLiteral = 20

variable = 21

rawConfigLine = 22

class condconfigparser.lexer.Token(startline, startcol)

Bases: object

Class representing a token instance (lexeme).

Token.startline and Token.startcol are 1-based line and column numbers for the first character of the lexeme (i.e., where the token starts in the parsed stream). Assuming it fits on one line of the input stream (which is not necessarily the case for string literal tokens), the column number of its last character is given by:

Token.startcol + len(Token.string) - 1.

Instances of this class must be considered immutable. They can be used as dictionary keys and set elements, among others. Since these operations assume that the hash value of the object never changes, it is an error to modify such an object after an operation that relies on its hash value (cf. object.__hash__()).

type = None

Token type (TokenType instance)

string = None

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = None

String used for __str__(), in particular for the NewlineToken

pos()

formatStartPos()

class condconfigparser.lexer.NewlineToken(startline, startcol)

Bases: Token

type = 1

Token type (TokenType instance)

string = '\n'

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = '\\n'

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.VarAssignmentsStartToken(startline, startcol)

Bases: Token

type = 2

Token type (TokenType instance)

string = '{'

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = '{'

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.VarAssignmentsEndToken(startline, startcol)

Bases: Token

type = 3

Token type (TokenType instance)

string = '}'

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = '}'

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.PredicateStartToken(startline, startcol)

Bases: Token

type = 4

Token type (TokenType instance)

string = '['

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = '['

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.PredicateEndToken(startline, startcol)

Bases: Token

type = 5

Token type (TokenType instance)

string = ']'

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = ']'

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.ListStartToken(startline, startcol)

Bases: Token

type = 6

Token type (TokenType instance)

string = '['

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = '['

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.ListEndToken(startline, startcol)

Bases: Token

type = 7

Token type (TokenType instance)

string = ']'

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = ']'

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.OpenParenToken(startline, startcol)

Bases: Token

type = 8

Token type (TokenType instance)

string = '('

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = '('

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.CloseParenToken(startline, startcol)

Bases: Token

type = 9

Token type (TokenType instance)

string = ')'

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = ')'

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.OrToken(startline, startcol)

Bases: Token

type = 10

Token type (TokenType instance)

string = 'or'

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = 'or'

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.AndToken(startline, startcol)

Bases: Token

type = 11

Token type (TokenType instance)

string = 'and'

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = 'and'

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.NotToken(startline, startcol)

Bases: Token

type = 12

Token type (TokenType instance)

string = 'not'

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = 'not'

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.EqualsToken(startline, startcol)

Bases: Token

type = 13

Token type (TokenType instance)

string = '=='

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = '=='

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.NotEqualsToken(startline, startcol)

Bases: Token

type = 14

Token type (TokenType instance)

string = '!='

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = '!='

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.InToken(startline, startcol)

Bases: Token

type = 15

Token type (TokenType instance)

string = 'in'

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = 'in'

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.AssignToken(startline, startcol)

Bases: Token

type = 16

Token type (TokenType instance)

string = '='

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = '='

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.CommaToken(startline, startcol)

Bases: Token

type = 17

Token type (TokenType instance)

string = ','

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = ','

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.TrueToken(startline, startcol)

Bases: Token

type = 18

Token type (TokenType instance)

string = 'True'

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = 'True'

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.FalseToken(startline, startcol)

Bases: Token

type = 19

Token type (TokenType instance)

string = 'False'

The raw lexeme, i.e., the exact string read from the parsed file

stringRepr = 'False'

String used for __str__(), in particular for the NewlineToken

class condconfigparser.lexer.StringLiteralToken(startline, startcol, unprocessedString, value)

Bases: Token

type = 20

Token type (TokenType instance)

value

String obtained after escape sequences expansion in the string literal

class condconfigparser.lexer.VariableToken(startline, startcol, name)

Bases: Token

type = 21

Token type (TokenType instance)

class condconfigparser.lexer.RawConfigLineToken(startline, startcol, line)

Bases: Token

type = 22

Token type (TokenType instance)

condconfigparser.lexer.mayClose = {')': (<class 'condconfigparser.lexer.OpenParenToken'>,), ']': (<class 'condconfigparser.lexer.ListStartToken'>, <class 'condconfigparser.lexer.PredicateStartToken'>), '}': (<class 'condconfigparser.lexer.VarAssignmentsStartToken'>,)}

Which token(s) may be closed by a given closing delimiter

class condconfigparser.lexer.Lexer(stream)

Bases: object

WSandComments_cre = re.compile(' *(#.*)?')

keywordOrVariable_cre = re.compile('\\b ([a-zA-Z0-9_]+) \\b', re.VERBOSE)

equalsOp_cre = re.compile('==[^=!]')

notEqualsOp_cre = re.compile('!=[^=!]')

assign_cre = re.compile('=[^=!]')

backslashNewline_cre = re.compile('\\\\\\n')

readline()

peek()

skipWSandComments()

Skip spaces and comments (all on a single line).

skipWSNLandComments(delimStack=None)

Skip a possibly-multiline mix of spaces and comments.

By default, NewlineToken instances are collected as encountered and returned in the form of a list. However, if delimStack is a non-empty delimiter stack, some of the newline tokens are selectively omitted from the returned list.

scanStringLiteralToken()

checkMatchingDelimiters(delimStack, closing, closingName, opening, openingName)

scanBalancedTokens(delimStack)

Scan a balanced sequence of tokens.

Normally, the token sequence should start right after a VarAssignmentsStartToken ({) or ListStartToken ([), which should be found at the top of delimStack. All subsequent tokens will be scanned and yielded until a } or ] matching the top of delimStack is found. The method does not consume that closing delimiter, for symmetry with the handling of the opening delimiter.

scanEnclosedTokenGroup()

Scan a varAssignments or predicate.

scanRawConfig()

Scan a rawConfigLine.

tokenGenerator()

Generate all tokens from the input stream.