kaskada.materialization

Module Contents

Classes

Destination

Helper class that provides a standard way to create an ABC using

FileType

Generic enumeration.

ObjectStoreDestination

Helper class that provides a standard way to create an ABC using

PulsarDestination

Helper class that provides a standard way to create an ABC using

MaterializationView

Functions

create_materialization(name, expression, destination, ...)

delete_materialization(name[, client])

get_materialization(name[, client])

list_materializations([search, client])

to_with_views(views)

format_output_prefix_uri(arg)

Formats the given arg to the expected pattern.

Attributes

logger

logger[source]
class Destination[source]

Bases: abc.ABC

Helper class that provides a standard way to create an ABC using inheritance.

abstract to_request()[source]
Return type:

Dict[str, Any]

class FileType[source]

Bases: enum.Enum

Generic enumeration.

Derive from this class to define new enumerations.

FILE_TYPE_UNSPECIFIED = 0[source]
FILE_TYPE_PARQUET = 1[source]
FILE_TYPE_CSV = 2[source]
class ObjectStoreDestination(file_type, output_prefix_uri)[source]

Bases: Destination

Helper class that provides a standard way to create an ABC using inheritance.

Parameters:
to_request()[source]
Return type:

Dict[str, Any]

class PulsarDestination(tenant='public', namespace='default', topic_name=None, broker_service_url='pulsar://127.0.0.1:6650', admin_service_url='http://127.0.0.1:8080', auth_plugin=None, auth_params=None)[source]

Bases: Destination

Helper class that provides a standard way to create an ABC using inheritance.

Parameters:

  • tenant (str) –

  • namespace (str) –

  • topic_name (Optional[str]) –

  • broker_service_url (str) –

  • admin_service_url (str) –

  • auth_plugin (Optional[str]) –

  • auth_params (Optional[str]) –

to_request()[source]
Return type:

Dict[str, Any]

class MaterializationView(name, expression)[source]

Bases: object

Parameters:

  • name (str) –

  • expression (str) –

create_materialization(name, expression, destination, views, slice_filter=None, client=None)[source]
Parameters:
Return type:

kaskada.kaskada.v1alpha.materialization_service_pb2.CreateMaterializationResponse

delete_materialization(name, client=None)[source]
Parameters:
Return type:

kaskada.kaskada.v1alpha.materialization_service_pb2.DeleteMaterializationResponse

get_materialization(name, client=None)[source]
Parameters:
Return type:

kaskada.kaskada.v1alpha.materialization_service_pb2.GetMaterializationResponse

list_materializations(search=None, client=None)[source]
Parameters:
Return type:

kaskada.kaskada.v1alpha.materialization_service_pb2.ListMaterializationsResponse

to_with_views(views)[source]
Parameters:

views (List[MaterializationView]) –

Return type:

List[kaskada.kaskada.v1alpha.materialization_service_pb2.WithView]

format_output_prefix_uri(arg)[source]

Formats the given arg to the expected pattern. Accepts “file:///path” and “/path” formats.

Parameters:

arg (str) – the input path or uri

Returns:

file uri formatted as file:///path

Return type:

str