Next-Gen Data Modeling, Integrity, and Governance with YODA

Streaming Audio: Apache Kafka® & Real-Time Data

Player FM - Internet Radio Done Right

32 subscribers

اضافه شده در six سال پیش

Indhold leveret af Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®. Alt podcastindhold inklusive episoder, grafik og podcastbeskrivelser uploades og leveres direkte af Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® eller deres podcastplatformspartner. Hvis du mener, at nogen bruger dit ophavsretligt beskyttede værk uden din tilladelse, kan du følge processen beskrevet her https://da.player.fm/legal.

Species Unite

1
Andrew Stein: Living with Lions 46:39

for 9 dage siden46:39

Afspil senere

Lister

Liked

46:39

“If we march into that village and we start trying to persecute people for using poison, something that's very illegal, nobody's going to talk to us. We're not going to find out where the poison came from. We're not going to be able to shut anything down. We should take the approach that people are using poison because they're desperate, because they see no other alternative.” – Andrew Stein Andrew Stein is a wildlife ecologist who spent the past 25 years studying human carnivore conflict from African wild dogs and lions in Kenya and Botswana to leopards and hyenas in Namibia. His work has long focused on finding ways for people and predators to coexist. He is the founder of CLAWS , an organization based in Botswana that's working at the intersection of cutting-edge wildlife research and community driven conservation. Since its start in 2014 and official launch as an NGO in 2020, CLAWS has been pioneering science-based, tech-forward strategies to reduce conflict between people and carnivores. By collaborating closely with local communities, especially traditional cattle herders, CLAWS supports both species conservation and rural livelihoods—making coexistence not just possible, but sustainable.…

for 2 år siden 55:55

MP3•Episode hjem

In this episode, Kris interviews Doron Porat, Director of Infrastructure at Yotpo, and Liran Yogev, Director of Engineering at ZipRecruiter (formerly at Yotpo), about their experiences and strategies in dealing with data modeling at scale.
Yotpo has a vast and active data lake, comprising thousands of datasets that are processed by different engines, primarily Apache Spark™. They wanted to provide users with self-service tools for generating and utilizing data with maximum flexibility, but encountered difficulties, including poor standardization, low data reusability, limited data lineage, and unreliable datasets.
The team realized that Yotpo's modeling layer, which defines the structure and relationships of the data, needed to be separated from the execution layer, which defines and processes operations on the data.
This separation would give programmers better visibility into data pipelines across all execution engines, storage methods, and formats, as well as more governance control for exploration and automation.
To address these issues, they developed YODA, an internal tool that combines excellent developer experience, DBT, Databricks, Airflow, Looker and more, with a strong CI/CD and orchestration layer.
Yotpo is a B2B, SaaS e-commerce marketing platform that provides businesses with the necessary tools for accurate customer analytics, remarketing, support messaging, and more.
ZipRecruiter is a job site that utilizes AI matching to help businesses find the right candidates for their open roles.
EPISODE LINKS

Kapitler

1. Intro (00:00:00)

2. What is Yotpo? (00:02:29)

3. Building an ETL framework based on Spark (00:05:25)

4. What is Apache Spark? (00:10:18)

5. Decoupling the data model (00:15:40)

6. Using data mesh principles (00:18:51)

7. How to address different data personas (00:22:24)

8. What is the "shift left" movement? (00:26:35)

9. How can organizations change the way they treat their data? (00:28:47)

10. Use-cases for tooling and documenting data sets (00:31:01)

11. Schema vs. schema-less (00:32:07)

12. What is YODA? (00:40:07)

13. Takeaways from the conversation with Doron and Liran (00:48:35)

14. It's a wrap! (00:52:45)

265 episoder

#Tech #Tech News #News #Confluent #Event Stream Processing #Data #Event Driven Architecture #Open Source #Data In Motion #Kafka Cloud Native #Data Mesh #Data Pipeline #Serverless Kafka #Podcasting Education #Confluent, original creators of Apache Kafka® #original creators of Apache Kafka® #Apache Kafka® #Cloud IT #Real Time

Next-Gen Data Modeling, Integrity, and Governance with YODA

Streaming Audio: Apache Kafka® & Real-Time Data

32 subscribers

published for 2 år siden

Del

MP3•Episode hjem

Kapitler

1. Intro (00:00:00)

2. What is Yotpo? (00:02:29)

3. Building an ETL framework based on Spark (00:05:25)

4. What is Apache Spark? (00:10:18)

5. Decoupling the data model (00:15:40)

6. Using data mesh principles (00:18:51)

7. How to address different data personas (00:22:24)

8. What is the "shift left" movement? (00:26:35)

9. How can organizations change the way they treat their data? (00:28:47)

10. Use-cases for tooling and documenting data sets (00:31:01)

11. Schema vs. schema-less (00:32:07)

12. What is YODA? (00:40:07)

13. Takeaways from the conversation with Doron and Liran (00:48:35)

14. It's a wrap! (00:52:45)

265 episoder

همه قسمت ها

for 4 years siden35:18

Afspil senere

Lister

Liked

35:18

A developer community brings people with shared interests and purpose together. The fundamental elements of a community are to gather, learn, support, and create opportunities for collaboration. A developer community is also an effective and efficient instrument for exploring and solving problems together. The power of a community is its endless advantages, from knowledge sharing to support, interesting discussions, and much more. Tim Berglund invites Ale Murray (Global Community Manager, Confluent) and Robin Moffatt (Staff Developer Advocate, Confluent) on the show to discuss the art of Q&A in a global community, share tips for building a vibrant developer community, and highlight the five strategic pillars for running a successful global community: Meetups Conferences MVP program (e.g., Confluent Community Catalysts) Community hackathons Digital platforms Digital platforms, such as a community Slack and forum, often consist of members who are well versed on topics of interest. As a leader in the Apache Kafka® and Confluent communities, Robin expresses the importance of being respectful when asking questions and providing details to the problem at hand. A well-formulated and focused question will more likely lead to a helpful answer. Oftentimes, the cognitive process of composing the question actually helps iron out the problem and draw out a solution. This process is also known as the rubber duck debugging theory . In a global community with diverse cultures and languages, being kind and having empathy is crucial. The tone and meaning of words can sometimes get lost in translation. Using emojis can help transcend language barriers by adding another layer of tone to plain text. Ale and Robin also discuss the pros and cons of a community forum vs. a Slack group. Tune in to find out more tips and best practices on building and engaging a developer community. EPISODE LINKS Use PODCAST100 to get an additional $100 of free Confluent Cloud usage ( details ) How to Ask Good Questions Why We Launched a Forum Growing the Event Streaming Community During COVID-19 ft. Ale Murray Meetup Hub Announcing the Confluent Community Forum Watch the video version of this podcast Join the Confluent Community Learn more with Kafka tutorials, resources, and guides at Confluent Developer Live demo: Intro to Event-Driven Microservices with Confluent…

Velkommen til Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.