Resolwe Storage Framework

Resolwe Storage Framework is storage management system for Resolwe Flow Design. Currenlty it supports storing data to local filesystem, Google Cloud Storage and Amazon Simple Storage Service.

Testing connectors

When testing connectors make sure there are credentials available (see sample settings bellow for details). These tests start with storage_credentials_test_ and are skipped by default when running tox. Since tox does not allow decrypting files on pull requests these tests would fail on all pull requests blocking merge.

Example settings

# Storage connector sample settings.
STORAGE_LOCAL_CONNECTOR = 'local'
STORAGE_CONNECTORS = {
    # Code assumes that connector named 'local' exists and points
    # to the location FLOW_EXECUTOR["DATA_DIR"].
    # If this is not true BAD THING WILL HAPPEN.
    "local": {
        "connector": "resolwe.storage.connectors.localconnector.LocalFilesystemConnector",
        "config": {
            "priority": 0,  # If ommited, default 100 is used
            "path": FLOW_EXECUTOR["DATA_DIR"],
            # Public URL from where data is served
            "public_url": "/local_data",
            # Delete from here after delay days from last access to this storage
            # location and when min_other_copies of data exist on other
            # locations.
            "delete": {
                "delay": 2,  # in days
                "min_other_copies": 2,
            },
            "copy": {
                # Override default settings for this data_slug.
                "data_slug": {
                    "00hrfastqgz-66": {
                        "delay": 0
                    }
                },
                "delay": 10,
            }
        }
    },
    "S3": {
        "connector": "resolwe.storage.connectors.s3connector.AwsS3Connector",
        "config": {
            "priority": 10,
            "bucket": "genialis-test-storage",
            "copy": {  # copy here from delay days from creation of filestorage object
                "delay": 5,  # in days
            },
            # Region name is needed to generate valid pre-signed urls.
            "region_name": "eu-central-1",
            "credentials": os.path.join(
                PROJECT_ROOT, "testing_credentials_s3.json"
            ),
        },
    },
    "GCS": {
        "connector": "resolwe.storage.connectors.googleconnector.GoogleConnector",
        "config": {
            "bucket": "genialis-test-storage",
            "credentials": os.path.join(
                PROJECT_ROOT, "testing_credentials_gcs.json"
            ),
            "copy": {
                "delay": 2,  # in days
            },
        },
    },
}