lkmltools.updater package

Submodules

lkmltools.updater.abstract_sql_reader module

class lkmltools.updater.abstract_sql_reader.AbstractSqlReader(config)

Bases: lkmltools.updater.definition_provider.DefinitionProvider

A reader that explicitly reads from relational DB using SQL and is able to run pd.read_sql

abstract get_connection()
Returns

a database connection, one that is compatible with pd.read_sql

get_credentials()

extract credentials from config

Returns

hostname port (int): port database (str): database name username (str): username password (str): password

Return type

host (str)

get_definitions()

get the definitions from the master source

Returns

dataframe

Return type

df (pandas dataframe)

lkmltools.updater.bq_definitions_provider module

Authors:

Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.updater.bq_definitions_provider.BqDefinitionsProvider(config)

Bases: lkmltools.updater.definition_provider.DefinitionProvider

A definitions provider that provides definitions from a BigQuery table

get_definitions()

get the definitions from the master source

Returns

dataframe

Return type

df (pandas dataframe)

lkmltools.updater.csv_definitions_provider module

Authors:

Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.updater.csv_definitions_provider.CsvDefinitionsProvider(config)

Bases: lkmltools.updater.definition_provider.DefinitionProvider

A definitions provider that provides definitions from a CSV file

get_definitions()

get the definitions from the master source

Returns

dataframe

Return type

df (pandas dataframe)

lkmltools.updater.definition_provider module

abstract base / interface for a class that provides definitions (descriptiopns) for dimensions, dimension_groups, and measures from some master source

Authors:

Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.updater.definition_provider.DefinitionProvider(config)

Bases: abc.ABC

a class that provides definitions for dimensions, dimension_groups, and measures

abstract get_definitions()

get the definitions from the master source

Returns

dataframe

Return type

df (pandas dataframe)

lkmltools.updater.definitions_provider_factory module

definitions provider factory

Authors:

Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.updater.definitions_provider_factory.DefinitionsProviderFactory

Bases: object

factory to instantiate DefinitionsProvider

static instantiate(class_name, config)

instantiate instances of definitions provider, given name of class

Parameters

config (JSON) – configuration JSON

Returns

instance of a DefinitionsProvider

lkmltools.updater.file_modifier module

modify file with additions or substitutions, and making as few other changes as possible (no formatting, whitespace, encoding etc)

Authors:

Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.updater.file_modifier.FileModifier(filename)

Bases: object

class that modifies file with additions or substitutions, and doing so with making as few other changes as possible (no formatting, whitespace, encoding etc)

COMMENT = '# programmatically added by LookML modifier'
handle_description_addition(definition_type, name, description)

add in a new description

Parameters

  • definition_type (str) – ‘measure’ or ‘dimension’

  • name (str) – name of measure or dimension

  • description (str) – description to add

Returns

nothing. Side effect is to add lines to self.lines

handle_desription_substitution(num_lines, definition_type, name, description)
as description exists, we need to find the header, then look for description after it,

consume all the lines of the current description, and add the new description

Parameters

  • num_lines (int) – number of lines in the existing description

  • definition_type (str) – ‘measure’ or ‘dimension’

  • name (str) – name of measure or dimension

  • description (str) – description to add

Returns

Nothing. Side effect to save to self.lines

is_header(line, header_type, header_name)
looking for start of dimension or header, e.g.

“dimension: header_name {“

Parameters

  • line (str) – line from a file

  • header_type (str) – e.g. dimension

  • header_name (str) – e.g. header_name (in example above)

Returns

is this chunk a header?

Return type

bool

modify(num_lines, definition_type, name, description, has_key)

modify an entry

Parameters

  • num_lines (int) – number of lines to substitute

  • has_key (bool) – do we have a description key for the definition_type,

  • name (str) – name of dimension, dimension_group, or measure

  • description (str) – correct description

Returns

nothing. Side effect is to update self.lines with correct info

write(filename)

write modified LookML to filename

Parameters

filename (str) – filepath of file to write to

Returns

nothing. Side effect is to write data to file

lkmltools.updater.lookml_modifier module

understand additions or modifications to be made and then delegate to filemodifier to make requested changes

Authors:

Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.updater.lookml_modifier.LookMlModifier(config)

Bases: object

class that understands additions or modifications to be made and then delegates to filemodifier to make requested changes

find_description(lookml, header_type, header_name)

get the description, if any, from this measure or dimension

Parameters

  • lookml (LookML) – instance of LookML

  • header_type (str) – ‘measure’ or ‘dimension’

  • header_name (str) – name of measure or dimension

Returns

description, if any, from dimension, dimension_group, or measure boolean: whether it has one

Return type

description (str)

modify(infilepath, outfilepath)

modify the LookML

Notes

default behavior is to match on full path when matching LookML files with the definitions source. However, you can configure to match on LookML file basename by setting "use_basename": true in the config

Parameters

  • infilepath (str) – path to input LookML file

  • outfilepath (str) – path of updated LookML to wtite to

Returns

nothing. Writes out modified file contents to file

lkmltools.updater.mysql_reader module

Read definitions from MySQL

author(s): Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.updater.mysql_reader.MySQLReader(config)

Bases: lkmltools.updater.abstract_sql_reader.AbstractSqlReader

Runs MySQL queries into pandas dataframes

get_connection()

Create and return connection to a MySQL database

Returns

connection to MySQL DB

lkmltools.updater.postgres_reader module

Read definitions from PostgreSQL

author(s): Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.updater.postgres_reader.PostgreSQLReader(config)

Bases: lkmltools.updater.abstract_sql_reader.AbstractSqlReader

Runs PostgreSQL queries into pandas dataframes

get_connection()

Create and return connection to a MySQL database

Returns

connection to PostgreSQL DB

lkmltools.updater.sqlite_reader module

Read definitions from SQLite

author(s): Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.updater.sqlite_reader.SQLiteReader(config)

Bases: lkmltools.updater.abstract_sql_reader.AbstractSqlReader

Runs SQLite queries into pandas dataframes

get_connection()

Create and return connection to SQLite database file

Returns

connection to SQLite file

Module contents