steamship.plugin package#

Subpackages#

Submodules#

steamship.plugin.embedder module#

class steamship.plugin.embedder.Embedder(client: Optional[Steamship] = None, config: Optional[Dict[str, Any]] = None, context: Optional[InvocationContext] = None)[source]#

Bases: PluginService[BlockAndTagPluginInput, EmbeddedItemsPluginOutput], ABC

abstract run(request: PluginRequest[BlockAndTagPluginInput]) → InvocableResponse[EmbeddedItemsPluginOutput][source]#

Runs the core operation implemented by this plugin: import, export, blockify, tag, etc.

This is the method that a Steamship Plugin implements to perform its main work.

run_endpoint(**kwargs) → InvocableResponse[EmbeddedItemsPluginOutput][source]#

Exposes the Embedder’s run operation to the Steamship Engine via the expected HTTP path POST /tag

steamship.plugin.file_importer module#

class steamship.plugin.file_importer.FileImporter(client: Optional[Steamship] = None, config: Optional[Dict[str, Any]] = None, context: Optional[InvocationContext] = None)[source]#

Bases: PluginService[FileImportPluginInput, RawDataPluginOutput], ABC

abstract run(request: PluginRequest[FileImportPluginInput]) → InvocableResponse[RawDataPluginOutput][source]#

Runs the core operation implemented by this plugin: import, export, blockify, tag, etc.

This is the method that a Steamship Plugin implements to perform its main work.

run_endpoint(**kwargs) → InvocableResponse[RawDataPluginOutput][source]#

Exposes the File Importer’s run operation to the Steamship Engine via the expected HTTP path POST /import

steamship.plugin.generator module#

class steamship.plugin.generator.Generator(client: Optional[Steamship] = None, config: Optional[Dict[str, Any]] = None, context: Optional[InvocationContext] = None)[source]#

Bases: PluginService[RawBlockAndTagPluginInput, RawBlockAndTagPluginOutput], ABC

abstract run(request: PluginRequest[RawBlockAndTagPluginInput]) → InvocableResponse[RawBlockAndTagPluginOutput][source]#

Runs the core operation implemented by this plugin: import, export, blockify, tag, etc.

This is the method that a Steamship Plugin implements to perform its main work.

run_endpoint(**kwargs) → InvocableResponse[RawBlockAndTagPluginOutput][source]#

Exposes the Tagger’s run operation to the Steamship Engine via the expected HTTP path POST /tag

class steamship.plugin.generator.TrainableGenerator(client: Optional[Steamship] = None, config: Optional[Dict[str, Any]] = None, context: Optional[InvocationContext] = None)[source]#

Bases: TrainablePluginService[RawBlockAndTagPluginInput, RawBlockAndTagPluginOutput], ABC

get_training_parameters_endpoint(**kwargs) → InvocableResponse[TrainingParameterPluginOutput][source]#

Exposes the Service’s get_training_parameters operation to the Steamship Engine via the expected HTTP path POST /getTrainingParameters

run_endpoint(**kwargs) → InvocableResponse[RawBlockAndTagPluginOutput][source]#

Exposes the Tagger’s run operation to the Steamship Engine via the expected HTTP path POST /generate

abstract run_with_model(request: PluginRequest[RawBlockAndTagPluginInput], model: TrainableModel) → InvocableResponse[RawBlockAndTagPluginOutput][source]#

Rather than implementing run(request), a TrainablePluginService implements run_with_model(request, model)

train_endpoint(**kwargs) → InvocableResponse[TrainPluginOutput][source]#

Exposes the Service’s train operation to the Steamship Engine via the expected HTTP path POST /train

steamship.plugin.request module#

class steamship.plugin.request.PluginRequest(*, data: Optional[T] = None, context: Optional[PluginRequestContext] = None, status: Optional[Task] = None, isStatusCheck: bool = False)[source]#

Bases: GenericModel, Generic[T]

class Config[source]#

Bases: object

alias_generator() → str#
allow_population_by_field_name = True#
context: Optional[PluginRequestContext]#
data: Optional[T]#
is_status_check: bool#
status: Optional[Task]#
class steamship.plugin.request.PluginRequestContext(*, pluginId: str = None, pluginHandle: str = None, pluginVersionId: str = None, pluginVersionHandle: str = None, pluginInstanceId: str = None, pluginInstanceHandle: str = None)[source]#

Bases: CamelModel

Contains the context in which

plugin_handle: str#
plugin_id: str#
plugin_instance_handle: str#
plugin_instance_id: str#
plugin_version_handle: str#
plugin_version_id: str#

steamship.plugin.tagger module#

class steamship.plugin.tagger.Tagger(client: Optional[Steamship] = None, config: Optional[Dict[str, Any]] = None, context: Optional[InvocationContext] = None)[source]#

Bases: PluginService[BlockAndTagPluginInput, BlockAndTagPluginOutput], ABC

abstract run(request: PluginRequest[BlockAndTagPluginInput]) → InvocableResponse[BlockAndTagPluginOutput][source]#

Runs the core operation implemented by this plugin: import, export, blockify, tag, etc.

This is the method that a Steamship Plugin implements to perform its main work.

run_endpoint(**kwargs) → InvocableResponse[BlockAndTagPluginOutput][source]#

Exposes the Tagger’s run operation to the Steamship Engine via the expected HTTP path POST /tag

class steamship.plugin.tagger.TrainableTagger(client: Optional[Steamship] = None, config: Optional[Dict[str, Any]] = None, context: Optional[InvocationContext] = None)[source]#

Bases: TrainablePluginService[BlockAndTagPluginInput, BlockAndTagPluginOutput], ABC

get_training_parameters_endpoint(**kwargs) → InvocableResponse[TrainingParameterPluginOutput][source]#

Exposes the Service’s get_training_parameters operation to the Steamship Engine via the expected HTTP path POST /getTrainingParameters

run_endpoint(**kwargs) → InvocableResponse[BlockAndTagPluginOutput][source]#

Exposes the Tagger’s run operation to the Steamship Engine via the expected HTTP path POST /tag

abstract run_with_model(request: PluginRequest[BlockAndTagPluginInput], model: TrainableModel) → InvocableResponse[BlockAndTagPluginOutput][source]#

Rather than implementing run(request), a TrainablePluginService implements run_with_model(request, model)

train_endpoint(**kwargs) → InvocableResponse[TrainPluginOutput][source]#

Exposes the Service’s train operation to the Steamship Engine via the expected HTTP path POST /train

steamship.plugin.trainable_model module#

class steamship.plugin.trainable_model.TrainableModel[source]#

Bases: ABC, Generic[ConfigType]

Base class for trainable models.

Trainable models are not plugins. They are a thin wrapper around the state of a model designed to be used with the Steamship plugin system.

# State Management

100% of a TrainableModel’s state management should save to & read from a folder on disk via the methods save_to_folder and load_from_folder.

# Remote Saving and Loading

TrainableModel instances automatically save to a user’s Workspace on Steamship via save_remote method. They can load themselves from a user’s workspace via the load_remote method.

When saving a model, the caller provides handle, such as “V1” or “epoch_23”. This allows that particular checkpoint to be re-loaded. By default, every save operation also saves the model to the “default” checkpoint, overwriting it if it already existed. When a user loads a model without specifying a checkpoint, the “default” checkpoint will be used.

# Data Scope

A TrainableModel’s data is saved & loaded with respect to

  1. The user’s active Workspace, and

  2. The provided Plugin Instance within that workspace.

The active workspace is read from the Steamship client context, and the plugin_instance_id is supplied as a method argument on the save_remote and load_remote methods.

This organization enables a user to have arbitrarily many trained model instances of the same type colocated within a Workspace.

# Training

A training job is fully parameterized by the TrainPluginInput object.

# Result Reporting

A training job’s results are reported via the TrainPluginOutput object. These results include a reference to the save_remote output, but they do not include the model parameters themselves. For example, after training, one could write:

>>> archive_path_in_steamship = model.save_remote(..)
>>> output = TrainPluginOutput(archive_path_in_steamship=archive_path_in_steamship,
    ...
    )

That output is the ultimate return object of the training process, but the Plugin that owns this model need not wait for synchronous completion to update the Steamship Engine with intermediate results. It can use the Response.post_update to proactively stream results back to the server.

# Third-party / External Models

This model class is a convenient wrapper for models running on third party systems (e.g. Google’s AutoML). In such a case:

  • The train method would begin the job on the 3rd party system.

  • The save_to_folder method would write the Job ID and any other useful data to the checkpoint path

  • The load_from_folder method would read this Job ID from disk and obtain an authenticated client with the third party system.

  • Any run method the implementer created would ferry back results fetched from the third-party system.

  • Any status reporting in TrainPluginOutput would ferry back status fetched from the third-party system.

config: ConfigType = None#
abstract load_from_folder(checkpoint_path: Path)[source]#

Load 100% of the state of this model to the provided path.

classmethod load_from_local_checkpoint(checkpoint: ModelCheckpoint, config: ConfigType)[source]#
classmethod load_remote(client: Client, plugin_instance_id: str, checkpoint_handle: Optional[str] = None, use_cache: bool = True, model_parent_directory: Optional[Path] = None, plugin_instance_config: Optional[ConfigType] = None)[source]#
receive_config(config: ConfigType)[source]#

Stores config from plugin instance, so it is accessible by model on load or train.

save_remote(client: Client, plugin_instance_id: str, checkpoint_handle: Optional[str] = None, model_parent_directory: Optional[Path] = None, set_as_default: bool = True) → str[source]#
abstract save_to_folder(checkpoint_path: Path)[source]#

Saves 100% of the state of this model to the provided path.

abstract train(input: PluginRequest[TrainPluginInput]) → InvocableResponse[TrainPluginOutput][source]#

Train or fine-tune the model, parameterized by the information in the TrainPluginInput object.

abstract train_status(input: PluginRequest[TrainPluginInput]) → InvocableResponse[TrainPluginOutput][source]#

Check on the status of an in-process training job, if it is running externally asynchronously.

Module contents#