Migrating Petabytes of data to OCI Obj Store via Oracle FastConnect.

Recently I transfered large amount of data from the Next Cloud CMS to Oracle Content Management and then migrated off and onto WebCenter Content. It's been a couple months now and I thought I'd pull together a quick blog post to share some of the options you can use on the Oracle Cloud Platform to transfer Petabytes of data from on-prem to the Oracle Cloud.

In this post we will cover the following:

1. Options for Transferring Content to Oracle Cloud.
2. Moving Data via Oracle FastConnect to OCI Obj Store.
3. Ingesting data to CM Box from OCI Object Store.
4. Verifying Data transfer

1. Options for transferring content to the Oracle Cloud

There are two options available that I know today to get large datasets into the Oracle Cloud - OCI.

- Oracle FastConnect
- DTS Service

With Fast Connect you transfer over the network and with DTS you ship a hard drive to Oracle - here is a quick comparison this is how long it would take to get the data into the cloud.

1) Oracle FastConnect

If you have the bandwidth you can enable VPN access and enable high speed transfers from 1 to 10 Gbps that will allow you to quickly transfer data to the Oracle Cloud.

2) Data Transfer Service

If you don't have the bandwidth and your data center accepts DTS - not all do especially in EMEA regions.

You have 2 options when requesting DTS:

2.1)  Data Import - Disk
Ship out your own hard drive to Oracles Transfer Office and they will import the data and send back your hard drive.

The process is fairly straightforward; here is  a high-level flow of what happens:

Check out the FAQ here DTS -Disk

2.2) or Data Import - Appliance
the better option if you are transferring a large amount of data is to use the Data Transfer Appliance offering which unless things have changed I believe is free.

However there is limited support on which data centers offer this service:

Data Transfer Appliance service is supported in us-phoenix-1, us-ashburn-1, eu-frankfurt-1, uk-london-1, ap-tokyo-1, ap-osaka-1, sa-saopaulo-1, us-luke-1, us-gov-ashburn-1, ca-toronto-1, Oracle Cloud Infrastructure regions. Note that data originating in EU can be shipped only to Frankfurt.

Check out the FAQ Here DTS-Appliance

2. Moving Data via Oracle FastConnect to OCI Obj Store.

Once you have FastConnect Configured you can use the OCI CLI. There are other options using RClone but for this post I'll cover Oracles OCI Command Line Utility.

Transfer files using the OCI CLI

1. Install & Configure the OCI CLI
Download and install the CLI (available for Windows, macOS, and Linux).

https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm

Configure the CLI with your tenancy information, user credentials, and key files by running:

oci setup config

This will prompt you for region, tenancy OCID, user OCID, fingerprint, and private key path.

2. Create an Object Storage Bucket
In the OCI Console: Object Storage → Buckets → Create Bucket.

Provide a Name and choose a Storage Tier (e.g., Standard).

Note the bucket’s name and the compartment.

3. Upload Files
Using the CLI, upload a file to the bucket:

oci os object put \ --bucket-name <your-bucket-name> \ --name <file-name-in-ObjectStorage> \ --file <local-file-path>

For larger files, the CLI automatically performs multipart uploads (in chunks). You can tweak options like --part-size and --parallel-uploads if needed.

To upload multiple files you can use this Bash Script, which will preserve the folder structure.

#!/bin/bash

BUCKET_NAME="your-bucket-name"
SOURCE_DIR="/path/to/local/folder"
PREFIX_PATH="optional/prefix/in/bucket"  # leave blank if not needed

# Recursively find all files and upload them
find "$SOURCE_DIR" -type f | while read -r filepath; do
  # Compute object name by stripping source dir and prepending prefix
  relative_path="${filepath#$SOURCE_DIR/}"
  object_name="$PREFIX_PATH/$relative_path"

  echo "Uploading $filepath → $object_name"

  oci os object put \
    --bucket-name "$BUCKET_NAME" \
    --file "$filepath" \
    --name "$object_name" \
    --force
done

4. List Objects
Verify uploads by listing objects:

os object list --bucket-name <your-bucket-name>

5. Download Files (if needed)
To retrieve objects from Object Storage:

os object get \ --bucket-name <your-bucket-name> \ --name <file-name-in-ObjectStorage> \ --file <local-download-path>

Looking to Manage and Secure your Cloud Content?

Try out Fishbowl Solutions new CM Box Digital Experience Platform designed for the Oracle Cloud, with an inbuilt CMS, DAM and AI integrations that allow you to secure and find your content faster than before. With Enterprise Search, SSO and ability to securely select and share out content with partners.

3. Ingesting data to CM Box from OCI Object Store.

You can 2 options to connect the data from OCI Object either run the CM Box CLI or (Coming soon!) directly within CM Box WebUI target the assets to hook into a repository.

3.1 CM Box CLI

This allows you to configure the Object store to connect to and map a CSV file for metadata mappings to link the content held within a CM Box repository.

3.2 CM Box Web UI for hybrid object storage

The team are working on a new integration that will allow you directly from the CM Box WebUI to connect and link or import data from any Cloud Object Store.

This then provides users the ability to map their own meta data and choose files to be imported or linked into individual repositories.

4. Verifying Data transfer

Here is an all in one approach that uploads the file and verifies the data by checking the file size and MD5 Checksum of each of the files after they are uploaded and generates a log for you to review any files that have failed to be transfered.

#!/bin/bash

# ----------- Configuration -----------
BUCKET_NAME="your-bucket-name"
SOURCE_DIR="/path/to/your/local/folder"
LOG_UPLOAD="upload.log"
LOG_VERIFY="verify.log"

# Ensure required commands exist
command -v oci >/dev/null 2>&1 || { echo >&2 "OCI CLI not found. Aborting."; exit 1; }
command -v md5sum >/dev/null 2>&1 || { echo >&2 "md5sum not found. Aborting."; exit 1; }
command -v base64 >/dev/null 2>&1 || { echo >&2 "base64 not found. Aborting."; exit 1; }
command -v xxd >/dev/null 2>&1 || { echo >&2 "xxd not found. Aborting."; exit 1; }

# Clean previous logs
> "$LOG_UPLOAD"
> "$LOG_VERIFY"

echo "Starting upload of files in $SOURCE_DIR to bucket $BUCKET_NAME..."
echo "Logging uploads to $LOG_UPLOAD and verifications to $LOG_VERIFY"

find "$SOURCE_DIR" -type f | while read -r FILE; do
    REL_PATH="${FILE#$SOURCE_DIR/}"  # preserve folder structure
    echo "Uploading: $REL_PATH" | tee -a "$LOG_UPLOAD"

    oci os object put \
        --bucket-name "$BUCKET_NAME" \
        --file "$FILE" \
        --name "$REL_PATH" \
        --force >> "$LOG_UPLOAD" 2>&1

    if [ $? -ne 0 ]; then
        echo "[UPLOAD FAIL] $REL_PATH" | tee -a "$LOG_UPLOAD"
        continue
    fi

    # Compute local MD5 in base64
    LOCAL_MD5_HEX=$(md5sum "$FILE" | awk '{print $1}')
    LOCAL_MD5_BASE64=$(echo "$LOCAL_MD5_HEX" | xxd -r -p | base64)

    # Get object metadata from OCI
    META=$(oci os object head \
        --bucket-name "$BUCKET_NAME" \
        --name "$REL_PATH" 2>/dev/null)

    REMOTE_MD5=$(echo "$META" | grep -i 'opc-content-md5' | awk '{print $2}')
    REMOTE_SIZE=$(echo "$META" | grep -i 'content-length' | awk '{print $2}')
    LOCAL_SIZE=$(stat -c%s "$FILE")

    if [[ "$LOCAL_MD5_BASE64" == "$REMOTE_MD5" && "$LOCAL_SIZE" == "$REMOTE_SIZE" ]]; then
        echo "[VERIFY OK] $REL_PATH" | tee -a "$LOG_VERIFY"
    else
        echo "[VERIFY FAIL] $REL_PATH - size or MD5 mismatch" | tee -a "$LOG_VERIFY"
    fi
done

echo "✅ Upload + verification complete."
echo "Check $LOG_UPLOAD and $LOG_VERIFY for full logs."

Conclusion and final thoughts..

In my opinion you should use Oracle Data Transfer Service (DTS) when you need to move large amounts of data (TBs or PBs) to Oracle Cloud but lack fast or reliable internet. It’s ideal for one-time bulk migrations where shipping physical storage devices is more practical.

Use Oracle FastConnect when you need a dedicated, high-speed, low-latency connection to Oracle Cloud for ongoing data transfers, hybrid applications, or real-time access. It’s best for consistent, secure, high-performance connectivity that bypasses the public internet.

If you need to securely manage large amounts of data from Oracle Object Storage checkout Fishbowl Solutions Enterprise CM Box Platform or Oracle WebCenter Content.