CCDAK Sample Questions Answers

Since Confluent 5.4.0, Avro 1.9.1 is used. Since default value was added to enum complex type , the schema resolution changed from:

(<1.9.1) if both are enums:** if the writer's symbol is not present in the reader's enum, then an error is signalled. **(>=1.9.1) if both are enums:

if the writer's symbol is not present in the reader's enum and the reader has a default value, then that value is used, otherwise an error is signalled.

Using Future.get() to wait for a reply from Kafka will limit throughput.

The REST Proxy requires to receive data over REST that is already base64 encoded, hence it is the responsibility of the producer

This will allow consumers to wait and receive more bytes in each fetch request.

The Schema Registry stores all the schemas in the _schemas Kafka topic

Some messages may require multiple retries. If there are more than 1 requests in flight, it may result in messages received out of order. Note an exception to this rule is if you enable the producer settingenable.idempotence=true which takes care of the out of ordering case on its own. Seehttps://issues.apache.org/jira/browse/KAFKA-5494

unclean.leader.election.enable=true allows non ISR replicas to become leader, ensuring availability but losing consistency as data loss will occur

Kafka transfers data with zero copy and no transformation. Any transformation (including compression) is the responsibility of clients.

Multiple topics can be passed as a list or regex pattern.

Although any Kafka Streams application is stateless as the state is stored in Kafka, it can take a while and lots of resources to recover the state from Kafka. In order to speed up recovery, it is advised to store the Kafka Streams state on a persistent volume, so that only the missing part of the state needs to be recovered.

SpecificRecord is created from generated record classes

doc represents optional description of message

Kafka logs are append-only and the data is immutable

Here, we must commit the offsets right after receiving a batch from a call to .poll()

Here, you shouldn't commit offsets 11 or 10 as it would indicate that the message #10 has been processed successfully.

Consumer offsets are stored in a Kafka topic __consumer_offsets, and the Schema Registry stored schemas in the _schemas topic.

majority consists of 3 zk nodes for 5 nodes zk cluster, so 2 can fail

Replicas are spread across available brokers, and each replica = one broker. RF 3 = 3 brokers

Only one of the replicas is elected as partition leader. And a broker can definitely hold many partitions from the same topic on its disk, try creating a topic with 12 partitions on one broker!

Producers can set acks=1 to get acknowledgement from partition leader only.

JDBC connector allows one task per table.

Clients with new schema will be able to read records saved with old schema.

The Confluent Schema Registry is your safeguard against incompatible schema changes and will be the component that ensures no breaking schema evolution will be possible. Kafka Brokers do not look at your payload and your payload schema, and therefore will not reject data

KafkaConsumer is not thread-safe, KafkaProducer is thread safe.

GlobalKTables have their datasets replicated on each Kafka Streams instance and therefore no repartitioning is required

Setting a group.id that's consistent across restarts will allow your consumers part of the same group to resume reading from where offsets were last committed for that group

If a group is not specified, the kafka-console-consumer generates a random consumer group.

We can only add partitions to an existing topic, and it must be done using the kafka-topics.sh command

You will need to set the schemas.enable parameters for the converter to false for plain text with no schema.

Each consumer is assigned a different partition of the topic to consume.

Here, we need to change the setting max.poll.interval.ms (default 300000) to its double in order to tell Kafka a consumer should be considered dead if the consumer only if it hasn't called the .poll() method in 10 minutes instead of 5.

Both of these are retriable errors, others non-retriable errors. See the full list of errors and their "retriable" status herehttps://kafka.apache.org/protocol#protocol_error_codes

each consumer can process only 50 MB/s, so we need at least 20 consumers consuming one partition so that 50 * 20 = 1000 MB target is achieved.

Increasing the number of partition causes new messages keys to get hashed differently, and breaks the guarantee "same keys goes to the same partition". Kafka logs are immutable and the previous messages are not re-shuffled

Log compaction retains at least the last known value for each record key for a single topic partition. All compacted log offsets remain valid, even if record at offset has been compacted away as a consumer will get the next highest offset.

On the producer side, after receiving base64 data, the REST Proxy will convert it into bytes and then send that bytes payload to Kafka. Therefore consumers reading directly from Kafka will receive binary data.

Kafka Connect Sink is used to export data from Kafka to external databases and Kafka Connect Source is used to import from external databases into Kafka.

A hopping window is defined by two propertiesthe window's size and its advance interval (aka "hop"), e.g., a hopping window with a size 5 minutes and an advance interval of 1 minute.

Kafka Connect Sink is used to export data from Kafka to external databases and Kafka Connect Source is used to import from external databases into Kafka.

Setting unclean.leader.election.enable to true means we allow out-of-sync replicas to become leaders, we will lose messages when this occurs, effectively losing credit card payments and making our customers very angry.

Kafka replication mechanism makes it resilient to the scenarios where the broker lose data on disk, but can recover from replicating from other brokers. This makes Kafka amazing!