Amazon S3: how the object store that became the data foundation for AI actually works

This article explains how S3 differs from an ordinary disk, how it achieves its legendary durability, how it controls access and cost, and why it has become the default resting place for data in machine learning architectures — from training pipelines to native vector storage for RAG.

What Amazon S3 is

Amazon S3 is a scalable object storage^?object storage: a model that stores data as independent objects with metadata in a flat namespace, without a directory hierarchy service in the Amazon Web Services cloud. It is designed to offer very high durability and performance at practically unlimited scale. Data is reached not through a file system but through networked API calls, from anywhere on the internet.

Unlike a traditional file system, object storage does not expose disks, partitions or mounted directories. Every piece of data is stored as an independent object identified by a unique key and accessed through an API rather than filesystem calls.

It helps to distinguish S3 from two other families of cloud storage right away.

S3 is something different — a pure object store with no notion of a disk mounted to an operating system or a directory tree. This apparent simplicity is the source of its scalability, because the service does not inherit the path-length limits or nesting depth imposed by traditional file systems.

Object architecture: buckets and objects

At the heart of S3 lies a flat namespace. The folders we see in the AWS console are a cosmetic illusion generated from the prefix in an object's name — technically no directories exist. The entire architecture rests on two concepts: buckets and objects.

A bucket is the top-level container for data. Every file placed into S3 must be assigned to a specific bucket. A bucket name must be globally unique across all AWS accounts worldwide, much like an internet domain name. When creating one, you choose a geographic region, which minimises network latency and satisfies legal data residency requirements. The default limit is 100 buckets per account and can be raised by AWS support. The capacity of a single bucket is practically unlimited — it can hold any number of petabytes, as long as you pay for the volume generated.

An object is the equivalent of a file, but described by a richer set of properties. A single object ranges from zero to 5 terabytes in size. Larger files are uploaded using the multipart upload mechanism, which splits the transfer into fragments; symmetrically, reads can be accelerated by fetching an object in parallel across multiple byte ranges (a byte-range GET, i.e. multipart download). An object consists of four elements:

• Key — a unique identifier within the bucket, a UTF-8 string up to 1024 bytes long.
• Value — the raw bytes of data, whether video, a Parquet dataset or a photograph.
• Metadata — a set of name-value pairs describing the object, split into system-defined and user-defined categories.
• Version ID — an identifier assigned when versioning is enabled on the bucket.

An important nuance is object immutability. S3 does not allow appending data to an existing file. To change something, you must upload the object again in full, creating a new version.

"Eleven nines" of durability and strong consistency

The feature that cemented S3's reputation is durability^?durability: The probability that an object, once stored, will not be lost over time — as opposed to availability, which is whether the data is reachable at a given moment. of 99.999999999 percent, the so-called eleven nines. AWS expresses it as the annual probability of losing an object, approximately …. This means that, statistically, if you stored ten million objects for one year, you would expect to lose only a tiny fraction of a single object. Hence the popular but simplified comparison of losing one object every ten thousand years.

S3 achieves this resilience through two mechanisms:

• Redundancy — in standard classes, data is automatically replicated across at least three geographically separated Availability Zones within a region. The failure, or even physical destruction, of one zone does not compromise data integrity.
• Self-healing — the service continuously verifies object checksums in the background and, on detecting silent media corruption (bit rot), automatically restores the damaged fragment from a redundant copy without client involvement.

In December 2020, AWS introduced a second fundamental guarantee: strong read-after-write consistency. Previously S3 operated on an eventual consistency model, in which a freshly written object could remain invisible to other nodes for a fraction of a second. This caused real problems in distributed analytics systems such as Apache Spark and Hadoop, and forced external workarounds. Now every read immediately after a write returns the latest version of the object, with no additional performance overhead. This change significantly simplified the construction of Big Data pipelines.

Storage classes: matching cost to data temperature

S3 offers an extensive cost-optimisation model based on storage classes. Their logic mirrors the "temperature" of data — from hot data needed instantly to frozen archives touched once a year.

• S3 Standard is the class for hot data requiring millisecond access and no retrieval fees. It is the most expensive to store but free of read charges, making it the natural choice for real-time backends and content served through CDNs.
• S3 Intelligent-Tiering automatically moves objects between hot and cold tiers by observing access patterns. In exchange for a small monthly monitoring fee, it protects teams from unexpected retrieval costs. It is a sensible default for data with an unpredictable usage profile, typical of data lakes.
• S3 Standard-IA and S3 One Zone-IA are classes for data read infrequently but required instantly on demand. They offer lower storage prices at the cost of retrieval fees. The One Zone variant drops replication across three zones, lowering the price but exposing data to loss if a single zone fails — suitable for easily reproducible data.
• The S3 Glacier family covers archives. Glacier Instant Retrieval provides immediate access to rarely used archival data. Glacier Flexible Retrieval offers retrieval from a few minutes to several hours depending on the chosen mode. Glacier Deep Archive is the cheapest option, priced at around one dollar per terabyte per month, intended for multi-year retention aligned with regulatory requirements, with retrieval times of several hours.
• A separate category is S3 Express One Zone, a very low-latency class operating in containers called directory buckets. It confines data to a single Availability Zone to deliver maximum performance for demanding workloads such as machine learning training loops.

Security: access control, encryption and versioning

Security in S3 rests on the shared responsibility model. AWS is responsible for the security of the infrastructure and data centres, while configuring access to the data rests with the customer. History records many breaches caused precisely by misconfigured, publicly open buckets — which is why every bucket is private by default.

Access control

Access is governed by two main mechanisms:

• S3 Bucket Policies are resource policies written in JSON, attached directly to the bucket. They let you specify precisely who can perform which operations, with conditions such as source IP address.
• IAM Policies (Identity and Access Management) define permissions on the identity side — users, roles and services — specifying allowed REST API actions such as GetObject or PutObject. AWS recommends the principle of least privilege here and avoiding administrative permissions.

An additional shield is Block Public Access — a set of four flags that globally block accidental data exposure, overriding erroneous rules. Auditing is provided by S3 Server Access Logging and AWS CloudTrail, which record who accessed which resources and when.

A separate and very practical mechanism is presigned URLs — time-limited, signed links^?signed links: URLs that carry a cryptographic signature generated from AWS credentials. The signature proves authorization and expires after a set time, so access can be granted without sharing the credentials themselves. that grant bounded access to a single object (read or write) without exposing AWS credentials. This is the most common way a backend lets a browser download a file from a private bucket or upload one directly, bypassing the application server.

Encryption

Encryption works on two levels. In transit^?In transit: Data while it is being transmitted over the network — for example between the user’s browser and the S3 servers., data is protected by HTTPS^?HTTPS: The encrypted version of the HTTP protocol — traffic between the browser and the server is protected, so no one in between can read or alter the data. with a TLS^?TLS: The cryptographic protocol that HTTPS is built on; it encrypts the connection and verifies the server’s identity. layer. At rest^?At rest: Data written and stored on disks in the data centre, when it is not currently being transmitted., several server-side encryption (SSE) variants are available:

• SSE-S3 is the default, free encryption using AWS-managed keys and the AES-256 algorithm.
• SSE-KMS hands key management to the AWS Key Management Service, enabling separation of duties and detailed auditing — to read an encrypted object, a user needs a separate kms:Decrypt permission.
• SSE-C lets the client supply their own key, which AWS never stores.

An alternative is client-side encryption, performed before data is sent to the cloud.

Versioning

S3 guards data against loss and unauthorized modification through several mechanisms:

• Versioning — a key defence against human error and ransomware attacks. Once enabled, each write to the same key creates a new version while preserving previous ones. Deleting an object without specifying a version merely applies a delete marker, and the data remains recoverable.
• S3 Lifecycle Policies — automatically move objects to cheaper classes or delete them according to defined time-based rules, controlling archive costs.
• S3 Object Lock — a stronger guarantee. Under a WORM (write once, read many) model it makes it impossible to overwrite or delete an object for a defined period — even for the account administrator. It is one of the most effective defences against ransomware and a common regulatory-compliance requirement, including legally binding retention and legal holds.

Performance and the cost model

S3 scales performance linearly with the number of key prefixes. A single prefix handles at least 3500 write requests (PUT, COPY, POST) and 5500 read requests (GET) per second, and appropriate namespace partitioning lets you multiply this throughput. For distributing content to users worldwide, AWS recommends integration with the CloudFront edge network, which offloads the source bucket.

In practice a prefix is the leading portion of a key — for example logs/2026/, images/, videos/ or users/. S3 scales traffic to each of them independently, so spreading objects across many prefixes multiplies the bucket's total throughput.

The cost model works on a pay-as-you-go basis and comprises three main elements:

• Storage cost — charged per gigabyte per month, from about 0.023 dollars in the Standard class down to a fraction of a cent in Glacier Deep Archive.
• Requests and retrieval fees — in cheaper classes they can surprise you with frequent reads.
• Data transfer — inbound traffic is usually free, while outbound to the internet and between regions is charged.

The most common trap is placing frequently read data in an archive class and overpaying for retrieval — which is why Intelligent-Tiering is often the safest compromise.

Integrations and operational features

Event notifications

S3 can emit an event on every change to an object's state — its creation, deletion or restoration from an archive. Notifications can be routed to four services:

• AWS Lambda — serverless processing.
• Amazon EventBridge — rule-based filtering and fan-out.
• Amazon SNS — broadcast.
• Amazon SQS — queuing.

This is the foundation of event-driven architectures: uploading a file can automatically trigger video transcoding, document indexing or an ETL pipeline^?ETL pipeline: Extract–Transform–Load: a process that pulls data from a source, transforms it into a target format, and loads it into a destination (e.g. a data warehouse)., without polling the bucket.

Replication: CRR and SRR

The eleven nines of durability protect data within a single region, but not against the failure of an entire region, nor do they bring data closer to users. Replication answers this. Cross-Region Replication (CRR) automatically and asynchronously copies objects to a bucket in another region — for disaster recovery, geographic compliance and lower local latency. Same-Region Replication (SRR) replicates within the same region, for example across accounts or for log aggregation. Both features require versioning to be enabled.

Transfer Acceleration

S3 Transfer Acceleration speeds up long-distance uploads and downloads by routing traffic through the nearest CloudFront edge location and then over AWS's optimised backbone. It helps when users worldwide upload large files to a bucket located in a single region.

Static website hosting

S3 can also serve static websites — HTML, CSS, JavaScript and image files — directly from a bucket, with no server at all. Combined with CloudFront and a TLS certificate, it is a cheap and scalable way to host sites and SPA front-ends. It is, in fact, the feature many people associate with S3 first.

S3 as the data foundation for AI and machine learning

In the artificial intelligence ecosystem, S3 serves as the central data repository. This stems from three use cases that together make it the default choice of data science teams.

Data lake

The first is the data lake^?data lake: a centralised repository storing raw data in any format, queried only at read time (schema-on-read). Traditional relational warehouses cannot cope with raw, diverse data at exabyte scale:

• video for computer vision
• logs from autonomous vehicle sensors
• text for language models

S3 lets you store data in raw form, in a schema-on-read^?schema-on-read: An approach where raw data is stored without a predefined structure, and the schema (format and types) is applied only at read time — as opposed to schema-on-write, where structure is enforced when the data is written. approach, and query it directly with tools such as Amazon Athena or Apache Spark on EMR clusters.

The evolution of this idea is the Lakehouse architecture — Amazon SageMaker Lakehouse combined with the open Apache Iceberg table format lets you treat data in S3 like an ordered relational database without copying it.

Training data

The second use case is training data. Models learn from gigantic datasets pulled directly from S3:

• Reading — the SageMaker Training Service reads objects at rest, often in the columnar Parquet format^?columnar Parquet format: An open, columnar data storage format — it stores values by column instead of by row, which makes analytics faster and compresses data heavily..
• Writing — training results, including model artifacts and checkpoints, are written back to buckets.

Thanks to versioning and audit logs, S3 supports data lineage tracking in MLOps practices.

Feature store

The third is the feature store, namely Amazon SageMaker Feature Store. It has a two-track architecture:

• Online Store — a fast in-memory store serving real-time predictions with millisecond latency, for example in card fraud detection.
• Offline Store — built precisely on S3, holds historical feature values for training and retraining models, supporting the Apache Iceberg format and historical queries.

Separating these two layers eliminates the dangerous problem of training-serving skew^?training-serving skew: a mismatch between the data used for training and the data available at prediction time, degrading model quality.

S3 in the training pipeline: high-throughput data access

The three use cases above show where data rests. A separate challenge is the speed at which it reaches the graphics processors. During the training of a large model, expensive GPUs cannot wait for data — a starved input pipeline^?starved input pipeline: A situation where the graphics processors (GPUs) sit idle because the input pipeline cannot supply data fast enough — the most expensive resource in the infrastructure goes to waste. means wasting the most expensive resource in the entire infrastructure.

Because S3 exposes data solely through a REST API rather than as a mounted disk, AWS develops a set of tools that combine object-store throughput with the convenience of a local file system.

Mountpoint for Amazon S3

Many existing applications expect a local file system rather than a REST API, and Mountpoint bridges that gap. Mountpoint for Amazon S3 is a high-throughput open-source file client that mounts an S3 bucket as a local file system. An application performs ordinary file operations — open, read — and Mountpoint automatically translates them into S3 object API calls. The tool is available for production use with read-heavy workloads: data lakes, machine learning training, image rendering, autonomous-vehicle simulation and ETL processes.

• Capabilities: It reads files up to 50 terabytes in size and can list, read and create objects.
• Limitations: It deliberately forgoes full POSIX semantics, however — it does not modify existing files, delete directories, or support symbolic links or file locking. It runs on Linux only and does not cover the Glacier archive classes or the Intelligent-Tiering archive tiers.

For workloads requiring full POSIX, AWS points to Amazon FSx for Lustre with its ability to link S3 buckets.

The S3 Connector for PyTorch

The Amazon S3 Connector for PyTorch is a library that delivers high throughput to PyTorch training jobs reaching for data in S3. It brings several conveniences:

• Dataset primitives: ready implementations of PyTorch’s dataset primitives — both the map-style variant for random access and the iterable-style variant for streaming sequential reads.
• Automatic optimisation: the connector optimises the downloading of training data and the writing of checkpoints on its own, freeing the engineer from manually listing buckets and managing concurrent requests.
• Checkpoints to S3: an interface for saving and loading checkpoints directly to S3, bypassing local disk, with support for distributed checkpoints and integration with PyTorch Lightning.
• Load distribution: when many training processes write simultaneously, it applies prefix strategies that spread the load across multiple S3 partitions and prevent throttling errors (503 Slow Down).

S3 Express One Zone

The S3 Express One Zone class mentioned earlier completes this picture on the latency side. By confining data to a single Availability Zone, it lowers access latency by up to an order of magnitude relative to the standard classes, which matters in tight training loops reading millions of small objects. Both Mountpoint and the PyTorch connector can address this class's directory buckets, combining its low latency with the convenience of the tools described above.

S3 Vectors: native vector storage for RAG and semantic search

Traditional object stores are excellent at storing files but know nothing about semantic similarity. The most recent step toward artificial intelligence is S3 Vectors — the first cloud object store with native support for storing and querying vectors. It answers the explosion of applications built on vector embeddings^?vector embeddings: a numerical representation of the meaning of data — text, image, audio — as a vector, enabling search by semantic similarity, where data is retrieved not by keywords but by similarity of meaning.

S3 Vectors lets you store and query vectors without provisioning any infrastructure, through a dedicated set of APIs. The scale is enormous:

• Index capacity: up to 2 billion vectors.
• Vector-bucket capacity: up to 10,000 indexes and as many as 20 trillion vectors in total.
• Query response time: measured in hundreds of milliseconds.

The key promise, however, is economics. AWS states a reduction of up to 90 percent in the cost of uploading, storing and querying vectors relative to traditional, always-on vector databases.

Integration with RAG, Bedrock and OpenSearch

S3 Vectors reveals its real value in integrations:

• Amazon Bedrock Knowledge Bases: the native connection lowers the cost of retrieval-augmented generation^?retrieval-augmented generation: a technique that couples a language model with retrieval of relevant fragments from a knowledge base so that answers rest on facts rather than the model's memory alone (RAG), in which a language model reaches for corporate documents instead of hallucinating.
• Amazon OpenSearch Service: enables a tiered strategy — infrequently queried vectors rest cheaply in S3, while those with the highest performance requirements are promoted to OpenSearch.

This is a deliberate division of labour — S3 Vectors does not replace a fast in-memory vector database serving a high number of queries per second, but complements it with a cheap, durable layer for enormous, long-term datasets.

Typical uses include:

• semantic search across large unstructured datasets — images, video, audio and text;
• RAG applications;
• durable memory for AI agents, for which cheap storage lets them retain context from every interaction without being forced to forget.

Conclusion

The name "Simple Storage Service" is a misleading understatement. Behind the apparent simplicity of uploading and downloading objects lies more than a decade of engineering: a flat object architecture, eleven nines of durability, strong consistency, a tiered cost model, extensive access control and encryption. The very features that made S3 the foundation of backups and web services now make it the central resting place for artificial intelligence data. As long as models learn from data, and data must lie somewhere durably and cheaply, S3 will remain one of the most important — though least visible — elements of modern digital infrastructure.

Sources

• Amazon Web Services — Amazon S3 documentation (aws.amazon.com)
• Amazon Web Services — Amazon SageMaker Feature Store (aws.amazon.com)
• Amazon Web Services — S3 data consistency model (aws.amazon.com)
• Amazon Web Services — Amazon S3 Vectors (aws.amazon.com)
• Amazon Web Services — Mountpoint for Amazon S3 (docs.aws.amazon.com)
• Amazon Web Services — Amazon S3 Connector for PyTorch (github.com/awslabs)
• AWS whitepapers — Storage best practices for data lakes (awsstatic.com)