pypuppetdbquery documentation¶

Contents:

pypuppetdbquery¶

A port of Erik Dalén‘s PuppetDB Query language to Python. This module is designed to be paired with pypuppetdb but does not depend on it directly.

This module is a Python implementation of the query language also implemented in puppet-puppetdbquery (in Ruby) and node-puppetdbquery (in JavaScript/NodeJS).

Please see the pypuppetdbquery documentation courtesy of Read the Docs and Sphinx.

Installation¶

You can install this package from source or from PyPi.

$ pip install pypuppetdbquery

$ git clone https://github.com/bootc/pypuppetdbquery
$ python setup.py install

If you wish to hack on it clone the repository but after that run:

$ pip install -r requirements.txt
$ pip install -r requirements-dev.txt

This will install all the runtime requirements of pypuppetdbquery and the dependencies for the test suite and generation of documentation.

Usage Example¶

import pypuppetdb
import pypuppetdbquery

pdb = pypuppetdb.connect()

pdb_ast = pypuppetdbquery.parse(
    '(processorcount=4 or processorcount=8) and kernel=Linux')

for node in pdb.nodes(query=pdb_ast):
    print(node)

For further examples, see the Examples section of the pypuppetdbquery documentation.

License¶

This project is licensed under the Apache License Version 2.0.

API Documentation¶

All the user-facing components of this package are intended to be found in the pypuppetdbquery package documentation. The other sub-modules and the test suite are documented here for completeness.

pypuppetdbquery package¶

This Python package contains a port of Erik Dalén‘s PuppetDB Query language to Python.

All the user-facing components of this package are intended to be found in this module itself rather than any of the sub-modules.

pypuppetdbquery.parse(s, json=True, mode='nodes', lex_options=None, yacc_options=None)[source]¶

Parse a PuppetDBQuery-style query and transform it into a PuppetDB “AST” query.

This function is intented to be the primary entry point for this package. It wraps up all the various components of this package into an easy to consume format. The output of this function is designed to be passed directly into pypuppetdb.

For examples of the query syntax see puppet-puppetdbquery and node-puppetdbquery by Erik Dalén.

Parameters:	s (str) – The query to parse and transform mode (str) – The PuppetDB endpoint being queried json (bool) – Whether to JSON-encode the PuppetDB AST result lex_options (dict) – Options passed to `ply.lex.lex()` yacc_options (dict) – Options passed to `ply.yacc.yacc()`

pypuppetdbquery.query_fact_contents(pdb, s, facts=None, raw=False, lex_options=None, yacc_options=None)[source]¶

Helper to query PuppetDB for fact contents (i.e. within structured facts) on nodes matching a query string.

Adjusts the query to return only those strucutred fact keys requested in the function call.

The facts listed in the facts list are run through the query parser and treated as “identifier paths”. This means the same rules apply as for within the query language, e.g. foo.bar or foo.* or even foo.~"bar.*".

If raw is False (the default), the return value is a dict with node names as keys containing a dict of flattened fact paths to fact values. If True it returns raw query output: a list of dictionaries (see the PuppetDB fact-contents documentation).

Note

This function can only be used to search deeply within structured facts. It cannot return a whole structured fact, only individual elements within—but you can return all the elements within a structured fact if you want by using a regex match.

Parameters:

pdb (pypuppetdb.api.BaseAPI) – pypuppetdb connection to query from
s (str) – The query string (may be empty to query all nodes)
facts (Sequence) – List of fact paths to search for
raw (bool) – Whether to skip post-processing the facts into a dict structure grouped by node
lex_options (dict) – Options passed to ply.lex.lex()
yacc_options (dict) – Options passed to ply.yacc.yacc()

pypuppetdbquery.query_facts(pdb, s, facts=None, raw=False, lex_options=None, yacc_options=None)[source]¶

Helper to query PuppetDB for facts on nodes matching a query string.

Adjusts the query to return only those facts requested in the function call.

The fact names included in facts may be names or regular expressions. If the string starts and ends with /, it is considered a regular expression (e.g. /^lsb/).

If raw is False (the default), the return value is a dict with node names as keys containing a dict of fact names to fact values. If True it returns raw pypuppetdb.types.Fact objects as pypuppetdb.api.BaseAPI.nodes() does.

Note

This function can return only full facts, not elements of structured facts. For example, only the whole os fact may be returned but not the os.family key within the larger structured fact. If you need to do this, look at query_fact_contents().

Parameters:

pdb (pypuppetdb.api.BaseAPI) – pypuppetdb connection to query from
s (str) – The query string (may be empty to query all nodes)
facts (Sequence) – List of fact names to search for
raw (bool) – Whether to skip post-processing the facts into a dict structure
lex_options (dict) – Options passed to ply.lex.lex()
yacc_options (dict) – Options passed to ply.yacc.yacc()

pypuppetdbquery.ast module¶

Abstract Syntax Tree (AST) for the PuppetDBQuery language. These simple classes are used by the pypuppetdbquery.parser.Parser in order to represent the parsed syntax tree.

class pypuppetdbquery.ast.AndExpression(left, right)[source]¶: Bases: pypuppetdbquery.ast.BinaryExpression

class pypuppetdbquery.ast.BinaryExpression(left, right)[source]¶: Bases: pypuppetdbquery.ast.Node

class pypuppetdbquery.ast.BlockExpression(expression)[source]¶: Bases: pypuppetdbquery.ast.UnaryExpression

class pypuppetdbquery.ast.Comparison(operator, left, right)[source]¶: Bases: pypuppetdbquery.ast.Expression

class pypuppetdbquery.ast.Date(value)[source]¶: Bases: pypuppetdbquery.ast.Literal

class pypuppetdbquery.ast.Expression[source]¶: Bases: pypuppetdbquery.ast.Node

class pypuppetdbquery.ast.Identifier(name)[source]¶: Bases: pypuppetdbquery.ast.Node

class pypuppetdbquery.ast.IdentifierPath(components)[source]¶: Bases: pypuppetdbquery.ast.Node

class pypuppetdbquery.ast.Literal(value)[source]¶: Bases: pypuppetdbquery.ast.Node

class pypuppetdbquery.ast.Node[source]¶: Bases: object

class pypuppetdbquery.ast.NotExpression(expression)[source]¶: Bases: pypuppetdbquery.ast.UnaryExpression

class pypuppetdbquery.ast.OrExpression(left, right)[source]¶: Bases: pypuppetdbquery.ast.BinaryExpression

class pypuppetdbquery.ast.ParenthesizedExpression(expression)[source]¶: Bases: pypuppetdbquery.ast.UnaryExpression

class pypuppetdbquery.ast.Query(expression)[source]¶: Bases: pypuppetdbquery.ast.Node

class pypuppetdbquery.ast.RegexpIdentifier(name)[source]¶: Bases: pypuppetdbquery.ast.Identifier

class pypuppetdbquery.ast.RegexpNodeMatch(value)[source]¶: Bases: pypuppetdbquery.ast.Expression

class pypuppetdbquery.ast.Resource(res_type, title, exported, parameters=None)[source]¶: Bases: pypuppetdbquery.ast.Expression

class pypuppetdbquery.ast.Subquery(endpoint, expression)[source]¶: Bases: pypuppetdbquery.ast.Node

class pypuppetdbquery.ast.UnaryExpression(expression)[source]¶: Bases: pypuppetdbquery.ast.Node

pypuppetdbquery.evaluator module¶

class pypuppetdbquery.evaluator.Evaluator[source]¶

Bases: object

Converts a pypuppetdbquery.ast Abstract Syntax Tree into a PuppetDB native AST query.

DECAMEL_RE = re.compile('(?!^)([A-Z]+)')¶: Regular expression used when converting CamelCase class names to underscore_separated names.

evaluate(ast, mode='nodes')[source]¶

Process a parsed PuppetDBQuery AST and return a PuppetDB AST.

The resulting PuppetDB AST is a native Python list. It will need converting to JSON (using json.dumps()) before it can be used with PuppetDB.

Parameters:	ast (pypuppetdbquery.ast.Query) – Root of the AST to evaulate mode (str) – PuppetDB endpoint to target
Returns:	PuppetDB AST
Return type:	list

pypuppetdbquery.lexer module¶

exception pypuppetdbquery.lexer.LexException(message, position)[source]¶

Bases: Exception

Raised for errors encountered during lexing.

Such errors may include unknown tokens or unexpected EOF, for example. The position of the lexer when the error was encountered (the index into the input string) is stored in the position attribute.

class pypuppetdbquery.lexer.Lexer(**kwargs)[source]¶

Bases: object

Lexer for the PuppetDBQuery language.

This class uses ply.lex.lex() to implement the lexer (or tokenizer). It is used by pypuppetdbquery.parser.Parser in order to process queries.

The arguments to the constructor are passed directly to ply.lex.lex().

Note

Many of the docstrings in this class are used by ply.lex to build the lexer. These strings are not particularly useful for generating documentation from, so the built documentation for this class may not be very useful.

input(s)[source]¶

Reset and supply input to the lexer.

Tokens then need to be obtained using token() or the iterator interface provided by this class.

next()[source]¶

Implementation of iterator.next().

Return the next item from the container. If there are no further items, raise the StopIteration exception.

t_ASTERISK = '\\*'¶

t_AT = '@'¶

t_BOOLEAN(t)[source]¶: true|false

t_DOT = '\\.'¶

t_EQUALS = '='¶

t_EXPORTED = '@@'¶

t_FLOAT(t)[source]¶: -?d+.d+

t_GREATERTHAN = '>'¶

t_GREATERTHANEQ = '>='¶

t_HASH = '[#]'¶

t_LBRACE = '{'¶

t_LBRACK = '\\['¶

t_LESSTHAN = '<'¶

t_LESSTHANEQ = '<='¶

t_LPAREN = '\\('¶

t_MATCH = '~'¶

t_NOTEQUALS = '!='¶

t_NOTMATCH = '!~'¶

t_NUMBER(t)[source]¶: -?d+

t_RBRACE = '}'¶

t_RBRACK = '\\]'¶

t_RPAREN = '\\)'¶

t_STRING_bareword(t)[source]¶: [-w_:]+

t_STRING_double_quoted(t)[source]¶: “(\.|[^\”])*”

t_STRING_single_quoted(t)[source]¶: ‘(\.|[^\’])*’

t_error(t)[source]¶

t_ignore = ' \t\n\r\x0c\x0b'¶

t_keyword(t)[source]¶: not|and|or

token()[source]¶: Obtain one token from the input.

tokens = ('LPAREN', 'RPAREN', 'LBRACK', 'RBRACK', 'LBRACE', 'RBRACE', 'EQUALS', 'NOTEQUALS', 'MATCH', 'NOTMATCH', 'LESSTHANEQ', 'LESSTHAN', 'GREATERTHANEQ', 'GREATERTHAN', 'ASTERISK', 'HASH', 'DOT', 'NOT', 'AND', 'OR', 'BOOLEAN', 'NUMBER', 'STRING', 'FLOAT', 'EXPORTED', 'AT')¶: List of token names handled by the lexer.

pypuppetdbquery.parser module¶

exception pypuppetdbquery.parser.ParseException(message, position)[source]¶

Bases: Exception

Raised for errors encountered during parsing.

The position of the lexer when the error was encountered (the index into the input string) is stored in the position attribute.

class pypuppetdbquery.parser.Parser(lex_options=None, yacc_options=None)[source]¶

Bases: object

Parser for the PuppetDBQuery language.

This class uses ply.yacc.yacc() to implement the parser. In concert with pypuppetdbquery.lexer.Lexer, it produces an Abstract Syntax Tree (AST) as declared in pypuppetdbquery.ast.

Parameters:	lex_options (dict) – Passed as keyword arguments to `pypuppetdbquery.lexer.Lexer` yacc_options (dict) – Passed as keyword arguments to `ply.yacc.yacc()`

Note

Many of the docstrings in this class are used by ply.yacc to build the parser. These strings are not particularly useful for generating documentation from, so the built documentation for this class may not be very useful.

p_block_expr(p)[source]¶: block_expr : LBRACE expr RBRACE

p_boolean(p)[source]¶: boolean : BOOLEAN

p_comparison_expr(p)[source]¶: comparison_expr : identifier_path comparison_op literal

p_comparison_op(p)[source]¶

comparison_op : MATCH: NOTMATCH

EQUALS

NOTEQUALS

GREATERTHAN

GREATERTHANEQ

LESSTHAN

LESSTHANEQ

p_empty(p)[source]¶: empty :

p_error(p)[source]¶

p_expr(p)[source]¶

expr : resource_expr: comparison_expr

subquery

p_expr_and(p)[source]¶: expr : expr AND expr

p_expr_identifier_path(p)[source]¶: expr : identifier_path

p_expr_not(p)[source]¶: expr : NOT expr

p_expr_or(p)[source]¶: expr : expr OR expr

p_expr_parenthesized(p)[source]¶: expr : LPAREN expr RPAREN

p_float(p)[source]¶: float : FLOAT

p_identifier(p)[source]¶

identifier : string: integer

p_identifier_path(p)[source]¶: identifier_path : identifier

p_identifier_path_nested(p)[source]¶: identifier_path : identifier_path DOT identifier

p_identifier_regexp(p)[source]¶: identifier : MATCH string

p_identifier_wild(p)[source]¶: identifier : ASTERISK

p_integer(p)[source]¶: integer : NUMBER

p_literal(p)[source]¶

literal : boolean: string

integer

float

p_literal_date(p)[source]¶: literal : AT string

p_query(p)[source]¶

query : expr: empty

p_resource_expr(p)[source]¶: resource_expr : string LBRACK identifier RBRACK

p_resource_expr_exported(p)[source]¶: resource_expr : EXPORTED string LBRACK identifier RBRACK

p_resource_expr_exported_param(p)[source]¶: resource_expr : EXPORTED string LBRACK identifier RBRACK block_expr

p_resource_expr_param(p)[source]¶: resource_expr : string LBRACK identifier RBRACK block_expr

p_string(p)[source]¶: string : STRING

p_subquery_block(p)[source]¶: subquery : HASH string block_expr

p_subquery_comparison(p)[source]¶: subquery : HASH string DOT comparison_expr

parse(text, debug=0)[source]¶

Parse the input string and return an AST.

Parameters:	text (str) – The query to parse debug (bool) – Output detailed information during the parsing process
Returns:	An Abstract Syntax Tree
Return type:	pypuppetdbquery.ast.Query

precedence = (('left', 'OR'), ('left', 'AND'), ('left', 'EQUALS', 'MATCH', 'LESSTHAN', 'GREATERTHAN'), ('right', 'NOT'))¶: Precedence rules in lowest to highest order

start = 'query'¶: Non-terminal to use as the starting grammar symbol

Test Suite¶

class test_frontend.TestFrontendParse(methodName='runTest')[source]¶

Bases: unittest.case.TestCase

Test cases targetting pypuppetdbquery.parse().

test_empty_queries()[source]¶

test_simple_json()[source]¶

test_simple_raw()[source]¶

class test_frontend.TestFrontendQueryFactContents(methodName='runTest')[source]¶

Bases: unittest.case.TestCase

Test cases targetting pypuppetdbquery.query_fact_contents().

test_raw_output()[source]¶

test_with_query_and_facts_list()[source]¶

test_without_either()[source]¶

test_without_query()[source]¶

class test_frontend.TestFrontendQueryFacts(methodName='runTest')[source]¶

Bases: unittest.case.TestCase

Test cases targetting pypuppetdbquery.query_facts().

test_query_facts_in_raw_mode()[source]¶

test_query_facts_with_facts_list_only()[source]¶

test_query_facts_with_query_and_facts_list()[source]¶

test_query_facts_with_query_and_facts_list_regex()[source]¶

test_query_facts_without_query_or_facts()[source]¶

class test_integration.TestIntegration(methodName='runTest')[source]¶

Bases: unittest.case.TestCase

Test cases for the integrated combination of pypuppetdbquery.lexer.Lexer, pypuppetdbquery.parser.Parser, and pypuppetdbquery.evaluator.Evaluator.

setUp()[source]¶

test_boolean_values()[source]¶

test_capitalize_class_names()[source]¶

test_dates_in_queries()[source]¶

test_does_not_wrap_subquery_with_mode_none()[source]¶

test_double_quoted_strings()[source]¶

test_empty_queries()[source]¶

test_equals_operator()[source]¶

test_escape_non_match_parts_on_structured_facts_with_match_op()[source]¶

test_float_values()[source]¶

test_greater_than_eq_operator()[source]¶

test_greater_than_operator()[source]¶

test_less_than_eq_operator()[source]¶

test_less_than_operator()[source]¶

test_match_operator()[source]¶

test_negate_expressions()[source]¶

test_node_subqueries()[source]¶

test_node_subqueries_with_block_of_conditions()[source]¶

test_node_subqueries_with_fact_query()[source]¶

test_node_subquery_fact_field()[source]¶

test_not_equals_operator()[source]¶

test_not_match_operator()[source]¶

test_precedence_a()[source]¶

test_precedence_b()[source]¶

test_precedence_within_resource_parameter_queries_a()[source]¶

test_precedence_within_resource_parameter_queries_b()[source]¶

test_resource_queries_for_exported_resources()[source]¶

test_resource_queries_for_exported_resources_with_parameters()[source]¶

test_resource_queries_with_regexp_title_matching()[source]¶

test_resource_queries_with_tags()[source]¶

test_resource_queries_with_type_and_title()[source]¶

test_resource_queries_with_type_title_and_parameters()[source]¶

test_single_quoted_strings()[source]¶

test_single_string_expressions()[source]¶

test_structured_facts()[source]¶

test_structured_facts_with_array_component()[source]¶

test_structured_facts_with_match_operator()[source]¶

test_structured_facts_with_wildcard_operator()[source]¶

class test_lexer.TestLexer(methodName='runTest')[source]¶

Bases: unittest.case.TestCase

Test cases for pypuppetdbquery.lexer.Lexer.

setUp()[source]¶

test_all_tokens()[source]¶

test_empty_queries()[source]¶

test_invalid_input()[source]¶

class test_parser.TestParster(methodName='runTest')[source]¶

Bases: unittest.case.TestCase

Test cases for pypuppetdbquery.parser.Parser.

setUp()[source]¶

test_boolean_values()[source]¶

test_dates_in_queries()[source]¶

test_double_quoted_strings()[source]¶

test_empty_queries()[source]¶

test_float_values()[source]¶

test_invalid_input()[source]¶

test_invalid_input_eof()[source]¶

test_logical_operators()[source]¶

test_negate_expression()[source]¶

test_node_subqueries()[source]¶

test_node_subqueries_with_block_of_conditions()[source]¶

test_resource_queries_for_exported_resources()[source]¶

test_resource_queries_for_exported_resources_with_parameters()[source]¶

test_resource_queries_with_type_and_regexp_identifier()[source]¶

test_resource_queries_with_type_title_and_parameters()[source]¶

test_single_string_expressions()[source]¶

test_structured_facts_with_wildcard_operator()[source]¶

Examples¶

Basic query for nodes using pypuppetdb:

import pypuppetdb
import pypuppetdbquery

pdb = pypuppetdb.connect()

pdb_ast = pypuppetdbquery.parse(
    '(processorcount=4 or processorcount=8) and kernel=Linux')

for node in pdb.nodes(query=pdb_ast):
    print(node)

Obtain named facts from nodes matching a query (using pypuppetdb):

import pypuppetdb
import pypuppetdbquery

pdb = pypuppetdb.connect()

node_facts = pypuppetdbquery.query_facts(
    pdb,
    '(processorcount=4 or processorcount=8) and kernel=Linux',
    ['/^lsb/', 'architecture'])

for node in node_facts:
    facts = node_facts[node]
    print(node, facts)

Obtain selected structured fact values from nodes matching a query (using pypuppetdb):

import pypuppetdb
import pypuppetdbquery

pdb = pypuppetdb.connect()

node_facts = pypuppetdbquery.query_fact_contents(
    pdb,
    '(processorcount=4 or processorcount=8) and kernel=Linux',
    ['system_uptime.days', 'os.lsb.~"dist.*"'])

for node in node_facts:
    facts = node_facts[node]
    print(node, facts)

pypuppetdbquery documentation¶