Skip to main content

Preservation DataStores and
Storlet Engine

Storage Research

PDS Architecture

Preservation DataStores (PDS) in the Cloud constitutes a cloud broker that interconnects between the OAIS functions and the multiple diverse clouds. At the top, PDS Cloud exposes an HTTP-based external interface that is based on OAIS and supports operations for ingest, access and preservation actions of AIPs. At the bottom, it utilizes various generic cloud storage from different providers.

The Preservation Engine transforms the logical OAIS functions and information objects into processes and physical storage objects. It provides a flexible data model for a multi-tenant, multi-cloud environment, with dynamically configurable data management capabilities. The Preservation Engine sometimes requires performing data-intensive computational tasks, such as validation, transformation, migration, fixity checks, de-identification and data analysis. When the AIP service, migration service and fixity service require performing such tasks, they use storlets - computational modules running in a sandbox close to the data. Offloading OAIS-based functionality to the storage decreases probability of data loss, simplifies the applications and supports automation of preservation processes.

The Storlet Engine provides the cloud storage with a capability to include storlets that run in a sandbox close to the data. It is plugged into a private cloud or object storage such as OpenStack Swift and provides a powerful extension mechanism that makes the storage flexible, customizable and extensible. It can be used with the Preservation Engine or just by itself for workloads that don't need preservation but require data-intensive compute.

Preservation DataStores System Overview