AWS event source mapping — what it is and how it works

What is event source mapping?

An event source mapping (ESM) is a Lambda resource that reads items from stream- and queue-based services and then invokes a function with batches of records. That is how the AWS Lambda documentation defines it. Inside the mapping, components called event pollers^?event pollers: Components inside the mapping that actively poll the source for new records and invoke the function. Lambda auto-scales them based on traffic by default. actively poll the source for new messages and invoke the function.

The simplest way to think about it: an ESM is a managed consumer operated by Lambda. Instead of writing code that loops and polls SQS, Kinesis, or Kafka for new records, you configure an ESM — and Lambda runs the polling infrastructure for you, gathers records into batches, and invokes your function.

The whole flow in one picture:

An important clarification: an ESM is not a separate AWS service. It is a Lambda-managed resource that connects a supported source (queue or stream) to a function. It holds no business logic of its own — its job is to reliably pull data, group it into batches, filter, scale, and handle basic errors.

How does it work?

The operating model is pull, not push — one of the most important distinctions in the whole topic. Lambda polls the source itself, gathers records, assembles a batch, and invokes the function synchronously, passing the batch in the Records field.

Batching behavior is controlled by two parameters: BatchSize (the maximum number of records in a batch) and MaximumBatchingWindowInSeconds (the maximum time to gather records, from 0 to 300 seconds). Lambda invokes the function when any of three conditions is met:

where … is the gathering time, … is the MaximumBatchingWindowInSeconds window, and … is the number of records gathered. The 6 MB limit cannot be changed.

The default batching window depends on the source. For Kinesis, DynamoDB, and SQS it is 0 seconds — Lambda invokes the function as soon as records are available. For MSK, self-managed Kafka, Amazon MQ, and DocumentDB the default window is 500 ms.

BatchSize is not just another parameter to memorize — it is a deliberate trade-off:

For Kinesis and DynamoDB streams, Lambda by default preserves record order within a shard^?shard: The unit of partitioning and parallelism in a Kinesis or DynamoDB stream; records within a single shard are processed in order. and typically processes one batch per shard at a time. You can raise per-shard concurrency with ParallelizationFactor (1 to 10) — then several batches from a single shard are processed in parallel, while Lambda still maintains ordering for records that share the same partition key^?partition key: The key that routes a record to a specific shard; records with the same key keep their relative order..

Queue vs stream — two different models

This distinction is the single most important mental model in the topic. Although an ESM looks like one mechanism, "SQS → Lambda" and "Kinesis → Lambda" behave fundamentally differently.

In SQS, a retrieved message is temporarily hidden by a visibility timeout. If the batch is processed successfully, the messages are deleted from the queue; if it fails, once the visibility timeout expires the messages become visible again and can be reprocessed. In Kinesis and DynamoDB Streams, Lambda does not delete records: the ESM tracks its read position (checkpoint^?checkpoint: The stored read position in a stream; the ESM advances the checkpoint as it processes, so it knows which record to resume from.) and advances through the stream, and records disappear only when the stream's retention period expires — regardless of whether they were processed.

What are its key components?

An event source mapping is made up of several configurable layers:

• Event pollers — the components that poll the source. By default Lambda auto-scales them based on traffic.
• Batching configuration — BatchSize and MaximumBatchingWindowInSeconds, plus the hard 6 MB payload limit.
• Starting position — relevant only for stream sources:
  • Streams (Kinesis, DynamoDB, Kafka/MSK) — determines where reading begins: from the oldest available records or only from new ones.
  • Queues (SQS, Amazon MQ) — have no such parameter; consumption always starts from the currently available messages.
• Filtering (FilterCriteria) — lets you discard uninteresting records before they reach the function.
• Error handling — the mechanisms depend on the source type:
  • Streams (Kinesis, DynamoDB) — configurable retry count and maximum record age, batch splitting on error (BisectBatchOnFunctionError), and a destination for discarded batches.
  • Queues (SQS) — a failed batch returns to the queue after the visibility timeout expires; once maxReceiveCount is exceeded, messages go to a dead-letter queue (DLQ) configured on the queue itself, not on the ESM.
  • Shared — ReportBatchItemFailures (partial batch responses) works for both streams and SQS, so only the records that actually failed are retried.
• Scaling — automatic or, for selected sources, provisioned mode with explicit poller limits.

The knobs you tune most often, in one place:

Filtering deserves attention. As the event filtering documentation describes, the FilterCriteria object holds a list of patterns using syntax identical to Amazon EventBridge rules. By default you can define up to five filters per mapping (up to ten with a quota increase), combined with OR logic. Available operators include prefix, suffix, exists, anything-but, numeric comparisons, and equals-ignore-case. Filtering works for DynamoDB, Kinesis, Amazon MQ, MSK, self-managed Kafka, and SQS — but not for DocumentDB. The architectural consequence matters: records that do not match the filter never reach the function, so filtering genuinely reduces the number (and cost) of Lambda invocations.

What can it be used for?

AWS Lambda currently supports event source mappings for seven event-source categories: Amazon DocumentDB, DynamoDB Streams, Kinesis Data Streams, Amazon MQ, Amazon MSK, self-managed Apache Kafka, and Amazon SQS.

In practice, that translates into a set of common use cases:

• Task queue processing — SQS as a buffer between systems, with Lambda as a worker consuming messages in batches.
• Stream processing — Kinesis or Kafka for near-real-time analytics, IoT telemetry, logs, and clickstreams.
• Reacting to database changes — DynamoDB Streams lets you invoke a function on every record change (for indexing, replication, notifications).
• Integration with messaging systems — Amazon MQ (ActiveMQ/RabbitMQ) for applications using classic message brokers.

The common denominator is asynchronous consumption from queues and streams — usually with batching and often at high throughput. That said, an ESM is also the right mechanism for low volume (say, a few SQS messages per minute).

Event source mapping vs push-based Lambda invocation

The most important comparison is between the pull model (ESM) and push-based invocation. A note on terminology: in the AWS console both SQS and S3 are added via "Add trigger," so the word "trigger" can mislead. What matters is the direction of flow:

Push services (S3, SNS, EventBridge) push events to Lambda themselves, and the trigger configuration is stored on the source-service side. An ESM works the other way around: it is a resource inside Lambda that polls the source itself.

Compared with hand-building your own consumer (say, an EC2 application polling Kinesis in a loop), an ESM removes the burden of polling, checkpointing, poller scaling, and basic error handling — at the cost of less control over low-level behavior.

Error handling and idempotency

By default, when the function returns an error, the ESM retries the entire batch. For streams, that means pausing the shard until success or record expiry — a single "poison" record can block an entire shard:

Without extra configuration, the error on C can cause A, B, D, and E to be reprocessed too — records that already succeeded. Enabling ReportBatchItemFailures lets the function return exactly which records failed (by sequence number), so Lambda retries only from those. In addition, BisectBatchOnFunctionError splits the batch in half on error, narrowing the retry scope.

Because delivery is "at least once," the same message may be processed twice — a "charge the customer $100" message runs twice and double-charges the account. The fix is an idempotent^?idempotent: A property of an operation for which running it multiple times with the same input yields the same result as running it once — a correctness requirement under at-least-once delivery. function that recognizes already-processed events:

eventId = 12345

if alreadyProcessed(12345):
    ignore
else:
    chargeCustomer()
    markAsProcessed(12345)

This is why AWS emphasizes idempotency so strongly — it is not a cautious footnote but a correctness requirement under an at-least-once model.

Management and infrastructure as code

An ESM is created and configured through the AWS console, CLI, SDKs, or infrastructure-as-code (CloudFormation, AWS SAM). Three API operations cover the mapping's whole lifecycle:

AWS is responsible for hosting the pollers and scaling — you maintain no polling infrastructure of your own.

Key limitations and challenges

• At-least-once delivery — duplicates are baked into the model; idempotency is required.
• Shard blocking on error — without ReportBatchItemFailures or BisectBatchOnFunctionError, one faulty record can block an entire shard.
• 6 MB payload limit — fixed and non-configurable.
• Filtering has limits — five patterns by default, no DocumentDB support, and the exists operator only works on leaf nodes of the JSON tree.
• Non-obvious concurrency metrics — AWS warns that, because of short gaps between invocations, Lambda may briefly report higher concurrency than the number of shards.
• Irreversible 500 ms window change — once edited, you cannot return to the default value without a new mapping.

Why does it matter?

Event source mapping looks like a configuration detail but in practice decides whether an event-driven architecture runs stably. This is where processing order, fault tolerance, cost (via concurrency), and latency are settled. A developer who treats an ESM as a "black box" sooner or later hits the classic problems: a shard blocked by a single faulty record, unexpected duplicates, or a function invoked too often with batches that are too small.

The importance of this mechanism grows alongside the popularity of event- and stream-driven architectures. More and more systems — from IoT telemetry, through real-time analytics, to data pipelines feeding AI models — rely on Kinesis and Kafka streams and SQS queues. In these scenarios, the ESM is the quiet glue connecting the data source to the processing logic.

The provisioned mode introduced by Amazon (for MSK, Kafka, and SQS) signals the direction: increasing emphasis on predictable, low-latency processing of large volumes. For a junior, the takeaway is simple: understanding the pull model, the queue-vs-stream contrast, and error handling in an ESM is an investment that pays off in every serious Lambda project.

If, after reading this, you can look at "SQS → Lambda" and say without hesitation, "SQS does not invoke Lambda — Lambda polls SQS through an ESM, batches the messages, and invokes the function synchronously, and on success the messages are deleted; delivery is at-least-once, so the function must be idempotent" — then you truly understand event source mapping.

Sources

• AWS Lambda Developer Guide — How Lambda processes records from stream and queue-based event sources — link
• AWS Lambda Developer Guide — Control which events Lambda sends to your function (event filtering) — link
• AWS Lambda Developer Guide — Configuring partial batch response with Kinesis Data Streams and Lambda — link
• AWS Lambda Developer Guide — Using Lambda to process records from Amazon Kinesis Data Streams — link