Validators and Full nodes running on the Sui network need to have the highest levels of reliability and uptime in order to deliver a high throughput scalable blockchain. A critical part of running a stateful application reliably is to ensure that hardware failovers can be done with relative ease. If the disk fails or another type of outage affects the machine running your Validator, there should be an easy way to migrate the Validator without having to reprocess all chain history. 

Ensuring seamless failovers are where Snapshots come in. State snapshots have two forms in the Sui network, Formal and Database. Formal snapshots contain the minimal state that makes up all of the Validator consensus information at the end of an epoch. Database snapshots are truly a full copy of a node’s database. 

Snapshots don’t have a lot of utility unless they are stored where they can be accessed easily and reliably. When starting to upload snapshots at the Sui network genesis, Amazon Web Services' (AWS) S3 was the perfect choice as a reliable backend for storage which could be shared with the early node operators on the Sui network. Mysten Labs started hosting public snapshots on S3 which could be used by any node operator to quickly sync a Full node or Validator and introduce it to the network. 

However, the simple act of hosting an S3 bucket which allowed for public downloads of state snapshots turned out to be a more painful user experience than we expected. The format of the state snapshot necessitates that you use the AWS command line interface (CLI) for downloading the many files that are contained in a single snapshot. If you don’t have a pre-existing set of AWS credentials to plug in when calling the AWS CLI, you need to use a lightly documented incantation of the AWS CLI: aws s3 cp --no-sign-request. 

In addition to the user issues, Sui’s snapshots were growing at an exponential rate. As an ultra-high throughput blockchain, the amount of data Sui generates is almost unprecedented. Each time a state snapshot was being downloaded it would take hours to pull the over 800 gigabytes of data from S3 onto a node operator's host.

For any readers familiar with the math behind AWS S3 pricing, the public good which was s3://mysten-mainnet-snapshots became expensive quickly. S3 charges you per gigabyte on data transferred out of S3. Because most operators are running nodes outside of AWS, this applies to the Mysten Lab's snapshot bucket. We were quickly running up five figure monthly charges to host this public resource.

Cloudflare recently announced R2, an S3 competitor, which is unique in its pricing model: zero egress costs. This is a perfect fit for hosting a dataset which is regularly fetched, and a huge winning feature over S3. 

Rather than do a full migration, we chose to add R2 as an alternative source to S3, and move S3 to a Requester Pays model. S3 has great performance and features like global transfer acceleration that we did not want to give up. The major part of this migration was not adjusting our tooling to write to R2 (R2 is S3 API-compatible), but modifying the Sui application to easily read from R2. 

Supporting permissionless downloads on R2

Asking users to use the AWS CLI against R2 was not an acceptable experience; we instead wanted users to be able to point our tooling at db-snapshot.mainnet.sui.io and read the files hosted there with zero authentication required (having a zero authentication option was important to us as we want to make it as easy as possible for anyone to run a Sui Full node). 

AWS S3 request signing is a crucial aspect of interacting securely with Amazon S3 resources. When a request is made to an S3 bucket, whether it's for uploading an object, downloading a file, listing objects, or any other operation, the request needs to be signed to ensure its authenticity and integrity. The process typically involves creating an authenticated http request by signing it with the user's access keys (access key ID, secret access key). 

For publicly accessible files or objects, it is technically possible to bypass signing of requests for reading (but not listing) resources for most cloud providers but is not supported in the Rust object store library we are using. As such, we decided to add this support in our codebase. We wanted to add zero authentication support but give users the option to choose between restoring snapshots from S3 with signed requests (because buckets have requester pays mode enabled) or from R2 without signing. To do this cleanly, we first declared abstractions for common object store operations in our codebase:

#[async_trait]
pub trait ObjectStoreGetExt: std::fmt::Display + Send + Sync + 'static {
   /// Return the bytes at given path in object store
   async fn get_bytes(&self, src: &Path) -> Result<Bytes>;
}

#[async_trait]
pub trait ObjectStoreListExt: Send + Sync + 'static {
   /// List the objects at the given path in object store
   async fn list_objects(
       &self,
       src: Option<&Path>,
   ) -> object_store::Result<BoxStream<'_, object_store::Result<ObjectMeta>>>;
}

#[async_trait]
pub trait ObjectStorePutExt: Send + Sync + 'static {
   /// Write the bytes at the given location in object store
   async fn put_bytes(&self, src: &Path, bytes: Bytes) -> Result<()>;
}

#[async_trait]
pub trait ObjectStoreDeleteExt: Send + Sync + 'static {
   /// Delete the object at the given location in object store
   async fn delete_object(&self, src: &Path) -> Result<()>;
}

For cleanly switching between signed and unsigned implementations, our usage functions needed to be modified a bit as well:

/// Read object at the given path from input store using either signed or 
/// unsigned store implementation
pub async fn get<S: ObjectStoreGetExt>(store: &S, src: &Path) -> Result<Bytes>;

/// Returns true if object exists at the given path
pub async fn exists<S: ObjectStoreGetExt>(store: &S, src: &Path) -> bool;

/// Write object at the given path. There is no unsigned put implmenetation
/// because writing an object requires permissioned user signing requests
pub async fn put<S: ObjectStorePutExt>(store: &S, src: &Path, bytes: Bytes) -> Result<()>;

We then implemented the signed (by falling back to the object store library we were already using) and unsigned implementations (leveraging individual cloud providers' REST APIs) of the above traits: 

/// Implementation for making signed requests using object store lib
#[async_trait]
impl ObjectStoreGetExt for Arc<DynObjectStore> {
   async fn get_bytes(&self, src: &Path) -> Result<Bytes> {
       self.get(src)
           .await?
           .bytes()
           .await
           .map_err(|e| anyhow!("Failed to get file: {} with error: {}", src, e.to_string()))
   }
}

/// Implementation for making unsigned requests to [Amazon 
/// S3](https://aws.amazon.com/s3/).
#[derive(Debug)]
pub struct AmazonS3 {
   /// Http client wrapper which makes unsigned requests for S3 resources
   client: Arc<S3Client>,
}

#[async_trait]
impl ObjectStoreGetExt for AmazonS3 {
   async fn get_bytes(&self, location: &Path) -> Result<Bytes> {
       let result = self.client.get(location).await?;
       let bytes = result.bytes().await?;
       Ok(bytes)
   }
}

/// Implementation for making unsigned requests to [Google Cloud 
/// Storage](https://cloud.google.com/storage/).
#[derive(Debug)]
pub struct GoogleCloudStorage {
   /// Http client wrapper which makes unsigned requests for gcs resources
   client: Arc<GoogleCloudStorageClient>,
}

#[async_trait]
impl ObjectStoreGetExt for GoogleCloudStorage {
   async fn get_bytes(&self, location: &Path) -> Result<Bytes> {
       let result = self.client.get(location).await?;
       let bytes = result.bytes().await?;
       Ok(bytes)
   }
}
pub struct ObjectStoreConfig {
  /// Which object store to use i.e. S3, GCS, etc
  #[serde(skip_serializing_if = "Option::is_none")]
  #[arg(value_enum)]
  pub object_store: Option<ObjectStoreType>,
  /// Name of the bucket to use for the object store. Must also set
  /// `--object-store` to a cloud object storage to have any effect.
  #[serde(skip_serializing_if = "Option::is_none")]
  #[arg(long)]
  pub bucket: Option<String>,
  #[serde(default)]
  #[arg(long, default_value_t = false)]
  pub no_sign_request: bool,
  ...
}

impl ObjectStoreConfig {
  pub fn make_signed(&self) -> Result<Arc<DynObjectStore>, anyhow::Error> {
    match &self.object_store {
      Some(ObjectStoreType::File) => self.new_local_fs(),
      Some(ObjectStoreType::S3) => self.new_s3(),
      Some(ObjectStoreType::GCS) => self.new_gcs(),
      _ => Err(anyhow!("At least one backed is needed")),
    }
  }
}

pub trait ObjectStoreConfigExt {
   fn make_unsigned(&self) -> Result<Arc<dyn ObjectStoreGetExt>>;
}

impl ObjectStoreConfigExt for ObjectStoreConfig {
   fn make_unsigned(&self) -> Result<Arc<dyn ObjectStoreGetExt>> {
       match self.object_store {
           Some(ObjectStoreType::S3) => {
               let bucket_endpoint = { };
               Ok(AmazonS3::new(&bucket_endpoint).map(Arc::new)?)
           }
           Some(ObjectStoreType::GCS) => {
               let bucket_endpoint = { };
               Ok(GoogleCloudStorage::new(&bucket_endpoint)).map(Arc::new)?)
           }
           _ => Err(anyhow!("At least one backend is needed")),
       }
   }
}

With all of the above in place, we could cleanly switch between signed and unsigned implementations based on the user provided configuration:

let store: Arc<dyn ObjectStoreGetExt> = if store_config.no_sign_request {
  store_config.make_unsigned()?
} else {
  store_config.make_signed().map(Arc::new)?
}; 

We were close to our objective of supporting zero authentication snapshot downloads but not there yet. One last challenge involved the lack of a means to list files in an R2 bucket without signing requests (it is possible to allow public, unsigned list access on S3). And we are required to list files in a RocksDB snapshot directory before downloading it. We fixed this problem by adding a MANIFEST file with all file paths during the snapshot creation process. This MANIFEST is now the source of truth for all files and their relative paths in a snapshot directory.

Final result

In the end, by allowing R2 as the default snapshot download option we were able to reduce the cost of serving these snapshots by ~70 to 80 percent, in addition to lowering the barriers for starting and failing over nodes within the Sui network.

Note: This content is for general educational and informational purposes only and should not be construed or relied upon as an endorsement or recommendation to buy, sell, or hold any asset, investment or financial product and does not constitute financial, legal, or tax advice.