消息队列kafka详解：如何实现死信队列

Apache Kafka ڴŶУ Uber Crowdstrike ʵͰо

ʶʹκοɿܵDZزٵġƪ̽** Apache Kafka ܹ****ʹŶʵִʵ**ЩѡԶʵ֡Kafka StreamsKafka ConnectSpring ܺͲߡʵоչʾ UberCrowdStrike ɣ̹Լ˹ģɿʵʱ

Apache Kafka ΪҵܹϲļмʹսԣҵҲ Kafka Ϊԭƽ̨ (iPaaS)

Apache Kafka еϢģʽ

ҿʼƪ֮ǰ֪ڡJMSϢк Apache KafkaIJϵһ֣

• JMS Ϣ Apache Kafka 10 Ƚϱ׼
• ƪ**C ͨApache Kafka еŶ (DQL)**д
• ʹ Apache Kafkaʵ**-ظģʽ**
• __ƳѡȷϢϵͳľJMS Apache Kafka
• Ƴ JMS Ϣ Apache KafkaɡǨƺ/滻

ʲôŶмģʽ Apache Kafka У

Ŷ (DLQ)Ϣϵͳƽ̨ڵһַʵ֣洢δɹϢϵͳDZתϢǽƶŶС

ҵ**ģʽ (EIP)**ΪģʽͨǿԽͬʡ

صƽ̨ Apache Kafka** Kafka нϢ DLQ **ҪԭͨϢʽϢЧ/ȱʧ磬Ԥֵ߷ַᷢӦóڸ̬ĻУⲻڡ쳣޷Ϣһ

ˣͨҪʹме֪ʶMessage Queue м JMS IBM MQTIBCO EMS RabbitMQֲʽύ־ KafkaĹʽͬԭϢе DLQ ϢϵͳЩԭһһӳ䵽 Kafka磬MQ ϵͳеϢÿϢ TTLʱ䣩ڡ

ˣ** Kafka нϢ DLQ ҪԭϢʽϢЧ/ȱʧ**

Apache Kafka Ŷе

Kafka еŶһ Kafka ⣬պʹ洢ڴ޷һܵдϢ˸ʹ´ϢϢЧϢĴֹͣ

Kafka Broker ܱܶ˵ṩ

Kafka ܹ֧ broker r еDLQأKafka ִ΢ͬԭϣʹáƹܵܶ˵㡱ԭΪʲô봫ͳϢȣKafka չ֮á˺ʹڿͻӦóС

ƽ̨ʵָɾơÿ΢ӦóͨԼѡļͨŷʽʹʵ߼

ڴͳмϢУṩ߼еĿչԺԽϲΪֻмŶӲʵּ߼

καԶʵ Kafka Ŷ

Kafka еŶжʹõĿܡһЩΪŶṩ˿伴õĹܡǣʹJavaGoC++Python καΪ Kafka ӦóдŶ߼Ҳס

ŶʵԴһ try-catch Ԥڻ쳣ûзϢκ쳣뽫Ϣ͵רõ DLQ Kafka ⡣

ʧԭӦӵ Kafka ϢıͷСӦļֵԱ㽫ʷ¼´͹Ϸ

ŶеĿ伴 Kafka ʵ

㲢ҪʵŶСͿѾṩǵ DLQ ʵ

ʹԼӦóͨԿƴڳִʱ޸롣ǣ** 3rd Ӧóļɲһܿ缯ϰĴ**ˣDLQ øҪijЩС

Kafka Connect Ŷ

Kafka Connect Kafka ļɿڿԴ Kafka СҪ Connect Ⱥе

Ĭ£ʹЧϢʹô JSON תȷ AVRO תʱKafka Connect ֹͣɾЧϢһѡ񡣺̴

Kafka Connect DLQ úܼ򵥡ֻ轫ѡ ' errors.tolerance' ' errors.deadletterqueue.topic.name' ֵΪȷֵ

¡ Kafka Connect Deep Dive C ŶСʾʹ DLQ ϸִʾ

Kafka Connect ڴ DLQ еĴϢֻ貿һʹ te DLQ 磬Ӧó Avro ϢҴϢ JSON ʽȻʹ JSON ϢתΪ AVRO ϢԳɹ´

ע⣬Kafka Connect ûԴŶ

Kafka Streams ӦóеĴ

**Kafka Streams Kafka **ʽ Apache FlinkStormBeam ƹߡǣ Kafka ԭġζڵչҿɿĻܹйĶ˵

ֱʹ JavaJVM ̬ϵͳ Kafka Ӧó鼸ʹ Kafka Streams Kafka ı׼ Java ͻΪʲô

• Kafka StreamsֻǡһΧƳ Java ߺ API İװԼõĸӹܡ
• ߶ֻǶ뵽 Java ӦóеĿ⣨JAR ļ
• ߶ǿԴ Kafka صһ - ûж֤ġ
• Ѿ伴õؽԹܡ״̬Ƕʽ洢ڡʽѯȵȣ

Kafka Streamsù֮һĬϵķл쳣޷л¼쳣𻵵ݡȷл߼δļ¼Ͷܵ´󡣸ùܲΪŶУ伴õؽͬ⡣

Spring Kafka Spring Cloud Stream Ĵ

Spring ܶ Apache Kafka кܺõ֧֡ṩģԱԼд롣Spring-Kafka Spring Cloud Stream Kafka ָ֧Ժʹѡʱ/ԡŶеȡ

Spring ܹܷdzḻеأһѧϰߡˣdzʺ½ĿѾ Spring Ŀ

кܶܰIJչʾ˲ͬʾѡŶеĹٷ Spring Cloud Stream ʾSpring ʹü򵥵ע͹߼ DLQ̷ֱһЩԱӰķһЩϲֻ˽ѡΪԼѡʵѡɡ

Apache Kafka ߵĿչʹ

ͻԻУʵ֤ŶеҪԭͨǴӵⲿ Web ݿʧʱ Kafka ޷з͸ᵼijЩӦó̱һܺõĽ

Apache Kafka****Apache 2.0 ĿԴĿṩһпͻ˶еIJ Apache Kafka ͻ˰װһؼԵĸ򵥵/ APIԼչķ IO

ÿͨ Kafka Consumer дϢζڲҪеķ Kafka Consumer жȡͨ Kafka ĸӳ١µ缫˲ԡⲿݷḻŶӡ

һؼڵ Kafka Ӧóд/ظ Web ݿлһη͵ Web Ҫ

Parallel Consumer ͻ˾ǿ߼õӳٺͶ̬ҲԷ͵ŶС

ʹŶеϢ

͵ŶкûɣϢҪٱأ

Ŷ¼д⴦ݴľѷʽζŴ¼ֿݱ䡣

ڴʹŶеĴԡDO DONT ̽ʵ;ѵ

мѡڴ洢ŶеϢ

• **´**DLQеһЩϢҪ´ǣȣҪ⡣Զű༭Ϣ˹߷شҪ·ͣģϢ
• ɾϢһãִܻϢǣɾ֮ǰҵӦüǡ磬DZӦóʹôϢǿӻ
• ߼һѡǷԻȡʵʱ⣬Ǵ DLQ еÿϢ磬һ򵥵 ksqlDB ӦóӦм㣬ÿСʱϢƽκȷ Kafka ӦóеĴļ⡣
• ֹͣٻֻϢֹͣҵ̡öԶģҲ˾ȻֹͣҲ׳ Kafka ӦóɡҪDLQ ;ⲿ
• ****ѡֻŶʲôȻʹijЩҲܺã Kafka ӦóΪסKafka бʱ䣬ڸʱ֮ɾϢֻΪȷķʽɡ DQL ǷΪ̫죩

Apache Kafka Ŷеʵ

Kafka ӦóʹŶндһЩʵ;ѵ

• ЧϢҵԶ˹
  • ʵͨû˴ DLQ Ϣ
  • ѡ 1ҪվǻܹŶ
  • ѡ 2Ӧ֪ͨ¼ŶϵͳݴǽҪӼ¼ϵͳ·/޸ݡ
  • û˹ĻԹ뿼ɺ DLQ ڵıҪԡ෴ЩϢҲڳʼ Kafka Ӧóбԡʡ˴縺ءʩʽ
• ʵDZŶӣ磬ͨʼ Slack
• ÿ Kafka ȼֹͣɾ´
• ԵĴϢ͵ DLQ - ӦóΡ
• ԭʼϢǴ洢 DLQ УжıͷϢʱ䡢ӦóƵȣʹ´͹ųøס
• **Ҫ Dead Letter Queue Kafka **ȡᡣǽд洢ڵ DLQ пܶԽһ´û塣

סDLQ б֤˳ֹʹκ͵ߴøˣKafka DQL ʺÿ

ʱ Kafka ʹŶУ

̽һ²ӦýЩ͵Ϣ Kafka ŶУ

• DLQ ڱѹڴϢķֵʹ DLQ нһ⡣Kafka ־Ĵ洢Զѹ԰ԼٶȻȡݵķʽȡݣô󣩡ܵĻԵչߡʹĴ洢ռDLQ Ҳ޼¡⣬Ƿʹ DLQ ޹ء
• ʧܵDLQʧܶϢ DQL ޼£ʹڶ֮󣩡ϢҲ޷ӵϵͳҪ⡣ϢԸҪ洢ڳУȡڱʱ䣩

ʹԤģʽע

ͬҪǣ̽ijЩŶеĿԡ

****Schema RegistryһȷԷֹڸгķ Kafka ǿִȷϢṹ

ģʽעģʽĿͻ˼顣Confluent Server һЩʵڴṩ˶ģʽ飬ԾܾδʹģʽעߵЧϢ

Kafka Ŷеİо

ǿUberCrowdStrike Santander Bank о Kafka ʩʵʲŶסЩǷdzӡÿĿҪôӡ

Uber - ɿٴŶ

ڷֲʽϵͳУDzɱġ󵽸⣬ϵжϣģеķ׼þŵʶʹϡ

Uber ӪΧٶȣϵͳݴʧʱЭUber Apache Kafka ڸּ˹ģʵһĿꡣ

ЩԣUber չŶչ Kafka ¼ܹеãͨʹ n**´Ŷʵֽɹ۲Ĵжʵʱ**òѡļʻԱ˺ƻ 200 пɿУΪעʻԱ۳ÿг̵ÿӢﱣѡ

Uber ʾήļֱ½ DLQ

йظϢĶ Uber dzϸļ£ʹ Apache Kafka ɿٴŶС

CrowdStrike - ¼Ĵ

CrowdStrike һλڵ¿˹ݰ˹͡簲ȫ˾ṩƹغͶ˵㰲ȫв鱨繥Ӧ

CrowdStrike Ļʩ** ÿʹ Apache Kafka ڸ¼**ҵġ Apache Kaka 簲ȫϵСУҽκιģʵʱ̬Ƹ֪в鱨

CrowdStrike ʵ ɹʵŶкʹ

• ȷϵͳд洢ϢʩʹԲͼšCrowdStrike ʹ S3 洢洢DZڵĴϢע⣬Kafka ķֲ洢伴õؽ⣬洢ӿڣ磬 Confluent Cloud е޴洢
• ʹԶùʹ޸һʧΪֶɴܷdz׳
• ¼ҵ̲ƸŶ׼ͼ¼ȷʹáйʦϤ֯ϢIJԡ

** CrowdStrike 簲ȫƽ̨УģʵʱݴҪҪҲڴһ繥ǹʵЧݵĶϢ** JavaScript ©ãˣͨŶʵʱ

ɣ̹ - Ժ DLQ ϵ 2.0

ɣ̹Ӧóдݵͬݴپ޴ս¼ܹǵĻܹһҿչļܹΪSantander Mailbox 2.0

Santander ĹزתƵ** Apache Kafka ṩֵ֧¼Դ**

µĻ첽¼ļܹеһؼսǴ** Santander ʹԺ DQL Kafka ⹹Ĵ**Щ⣺

鿴 Santander ļɺ Consdata Kafka ݽԲԺ Apache Kafka еĿɿ¼ݡеϸϢ

Apache Kafka пɿҿչĴ

ڹɿܵƽ̨Ҫڲͬ⡣ýŶеԶʵֻʹõĿܣ Kafka StreamsKafka ConnectSpring ܻ Kafka IJߡ

ŲCrowdStrike ɣ̹еİоǺʵ֡µӦóܹʱҪһʼͿǵһ㡣ʹ Apache Kafka ʵʱֻܹΪʱܳɹŶೡľѡ