lkmltools package

Subpackages

• lkmltools.grapher package
  • Submodules
  • lkmltools.grapher.graph_animator module
  • lkmltools.grapher.lookml_grapher module
  • Module contents
• lkmltools.linter package
  • Subpackages
    • lkmltools.linter.rules package
      • Subpackages
        • lkmltools.linter.rules.fieldrules package
          • Submodules
          • lkmltools.linter.rules.fieldrules.all_caps_rule module
          • lkmltools.linter.rules.fieldrules.count_name_rule module
          • lkmltools.linter.rules.fieldrules.description_rule module
          • lkmltools.linter.rules.fieldrules.drill_down_rule module
          • lkmltools.linter.rules.fieldrules.lexicon_rule module
          • lkmltools.linter.rules.fieldrules.yesno_name_rule module
          • Module contents
        • lkmltools.linter.rules.filerules package
          • Submodules
          • lkmltools.linter.rules.filerules.data_source_rule module
          • lkmltools.linter.rules.filerules.filename_viewname_match_rule module
          • lkmltools.linter.rules.filerules.one_view_per_file_rule module
          • Module contents
        • lkmltools.linter.rules.otherrules package
          • Submodules
          • lkmltools.linter.rules.otherrules.no_orphans_rule module
          • Module contents
      • Module contents
  • Submodules
  • lkmltools.linter.field_rule module
  • lkmltools.linter.lookml_linter module
  • lkmltools.linter.rule module
  • lkmltools.linter.rule_factory module
  • Module contents
• lkmltools.updater package
  • Submodules
  • lkmltools.updater.abstract_sql_reader module
  • lkmltools.updater.bq_definitions_provider module
  • lkmltools.updater.csv_definitions_provider module
  • lkmltools.updater.definition_provider module
  • lkmltools.updater.definitions_provider_factory module
  • lkmltools.updater.file_modifier module
  • lkmltools.updater.lookml_modifier module
  • lkmltools.updater.mysql_reader module
  • lkmltools.updater.postgres_reader module
  • lkmltools.updater.sqlite_reader module
  • Module contents

Submodules

lkmltools.abstract_rule module

class lkmltools.abstract_rule.AbstractRule(config_dict=None)

Bases: abc.ABC

Abstract rule that can take a JSON configuration object

config_for_key(key)

has_key(key)

does the config dictionary have a key ‘key’?

Returns

whether there is a configuration dictionary and key is present

Return type

response (bool)

lkmltools.bq_writer module

Authors:

Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.bq_writer.BqWriter

Bases: object

Write data to BigQuery via GCS.

FILENAME = 'tmp_upload.csv'

upload(dataframe, config, destination_table_key)

Parameters

• dataframe (pandas dataframe) – dataframe to upload

• config (JSON) – JSON configuration

• destination_table_key (str) – table to write to

Returns

nothing. upload data to GCS

upload_to_bq(dataframe, gcs_project_id, project_id, dataset, tablename, target_bucket_name, bucket_folder, write_disposition='WRITE_APPEND')

Write some dataframe to BigQuery via GCS storage.

Parameters

• dataframe (pandas dataframe) – data to be written

• gcs_project_id (str) – GCS project ID

• target_bucket_name (str) – GCS bucket_name

• bucket_folder (str) – GCS bucket_folder

• project_id (str) – BQ project ID

• dataset (str) – BQ dataset

• tablename (str) – BQ tablename

Returns

nothing but side effect is to write data to GCS and then to BigQuery

lkmltools.google_auth_helper module

Authors:

Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.google_auth_helper.GoogleAuthHelper

Bases: object

decode_service_account(encoded_json)

decode base64 representation back to JSON

Parameters

encoded_json (bytes) – output of encode_service_account, ie base64 encoded service account

Returns

service account JSON

encode_service_account(raw_json)

encode JSON to base64 representation

Parameters

raw_json (JSON) – raw Google service account JSON

Returns

Base64 encoded Google service account JSON

write_decoded_sa_json_to_file(encoded_json, filename='key.json')

write decoded service account JSON to file

Parameters

• encoded_json (bytes) – base64 encoded service account JSON

• filename (str) – filename to write to

Returns

nothing. Dumps service account to file

lkmltools.lookml module

parse a LookML file from LookML to JSON

Authors:

Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.lookml.LookML(infilepath)

Bases: object

explores()

get explores (if any) from the LookML

Returns

explores (list) if any, None otherwise

has_explores()

does this have one or more explores?

Returns

bool, whether this has explores

has_views()

does this have one or more views?

Returns

bool, whether this has views

views()

get views (if any) from the LookML

Returns

views (list) if any, None otherwise

lkmltools.lookml_field module

class lkmltools.lookml_field.LookMLField(json_data)

Bases: object

A view field such a dimension, dimension_group, or measure

has_key(k)

does this have a key k?

Returns

whether this has key k

Return type

bool

is_dimension()

is this a dimension?

Returns

whether this is a dimension

Return type

bool

is_dimension_group()

is this a dimension_group?

Returns

whether this is a dimension_group

Return type

bool

is_measure()

is this a measure?

Returns

whether this is a measure

Return type

bool

lkmltools.simple_bq_writer module

Authors:

Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.simple_bq_writer.SimpleBqWriter

Bases: object

Class that writes dataframe to BigQuery directly (i.e. not via GCS). This is simple to configure, but it is slower than via GCS and it probably mean that you don’t have any backups

upload(df, config, destination_table, if_exists='append')

upload a dataframe df to a biquery table directly

Parameters

• df (pandas dataframe) – dataframe

• destination_table (str) – name of table to write to

• if_exists (str) – what to do if table already exists?

Returns

nothing. uploads dasts to BigQuery

lkmltools.util module

utility methods

Authors:

Carl Anderson (carl.anderson@weightwatchers.com)

class lkmltools.util.FieldCategory

Bases: enum.Enum

type of LookML field

DESCRIPTION = 'description'

DIMENSION = 'dimension'

DIMENSION_GROUP = 'dimension_group'

MEASURE = 'measure'

Module contents