Artwork

Indhold leveret af The Data Flowcast. Alt podcastindhold inklusive episoder, grafik og podcastbeskrivelser uploades og leveres direkte af The Data Flowcast eller deres podcastplatformspartner. Hvis du mener, at nogen bruger dit ophavsretligt beskyttede værk uden din tilladelse, kan du følge processen beskrevet her https://da.player.fm/legal.
Player FM - Podcast-app
Gå offline med appen Player FM !

Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade

27:47
 
Del
 

Manage episode 453327749 series 2053958
Indhold leveret af The Data Flowcast. Alt podcastindhold inklusive episoder, grafik og podcastbeskrivelser uploades og leveres direkte af The Data Flowcast eller deres podcastplatformspartner. Hvis du mener, at nogen bruger dit ophavsretligt beskyttede værk uden din tilladelse, kan du følge processen beskrevet her https://da.player.fm/legal.

Scaling deployments for a billion users demands innovation, precision and resilience. In this episode, we dive into how LinkedIn optimizes its continuous deployment process using Apache Airflow. Rahul Gade, Staff Software Engineer at LinkedIn, shares his insights on building scalable systems and democratizing deployments for over 10,000 engineers.

Rahul discusses the challenges of managing large-scale deployments across 6,000 services and how his team leverages Airflow to enhance efficiency, reliability and user accessibility.

Key Takeaways:

(01:36) LinkedIn minimizes human involvement in production to reduce errors.

(02:00) Airflow powers LinkedIn’s Continuous Deployment platform.

(05:43) Continuous deployment adoption grew from 8% to a targeted 80%.

(11:25) Kubernetes ensures scalability and flexibility for deployments.

(12:04) A custom UI offers real-time deployment transparency.

(16:23) No-code YAML workflows simplify deployment tasks.

(17:18) Canaries and metrics ensure safe deployments across fabrics.

(20:45) A gateway service ensures redundancy across Airflow clusters.

(24:22) Abstractions let engineers focus on development, not logistics.

(25:20) Multi-language support in Airflow 3.0 simplifies adoption.

Resources Mentioned:

Rahul Gade -

https://www.linkedin.com/in/rahul-gade-68666818/

LinkedIn -

https://www.linkedin.com/company/linkedin/

Apache Airflow -

https://airflow.apache.org/

Kubernetes -

https://kubernetes.io/

Open Policy Agent (OPA) -

https://www.openpolicyagent.org/

Backstage -

https://backstage.io/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

  continue reading

49 episoder

Artwork
iconDel
 
Manage episode 453327749 series 2053958
Indhold leveret af The Data Flowcast. Alt podcastindhold inklusive episoder, grafik og podcastbeskrivelser uploades og leveres direkte af The Data Flowcast eller deres podcastplatformspartner. Hvis du mener, at nogen bruger dit ophavsretligt beskyttede værk uden din tilladelse, kan du følge processen beskrevet her https://da.player.fm/legal.

Scaling deployments for a billion users demands innovation, precision and resilience. In this episode, we dive into how LinkedIn optimizes its continuous deployment process using Apache Airflow. Rahul Gade, Staff Software Engineer at LinkedIn, shares his insights on building scalable systems and democratizing deployments for over 10,000 engineers.

Rahul discusses the challenges of managing large-scale deployments across 6,000 services and how his team leverages Airflow to enhance efficiency, reliability and user accessibility.

Key Takeaways:

(01:36) LinkedIn minimizes human involvement in production to reduce errors.

(02:00) Airflow powers LinkedIn’s Continuous Deployment platform.

(05:43) Continuous deployment adoption grew from 8% to a targeted 80%.

(11:25) Kubernetes ensures scalability and flexibility for deployments.

(12:04) A custom UI offers real-time deployment transparency.

(16:23) No-code YAML workflows simplify deployment tasks.

(17:18) Canaries and metrics ensure safe deployments across fabrics.

(20:45) A gateway service ensures redundancy across Airflow clusters.

(24:22) Abstractions let engineers focus on development, not logistics.

(25:20) Multi-language support in Airflow 3.0 simplifies adoption.

Resources Mentioned:

Rahul Gade -

https://www.linkedin.com/in/rahul-gade-68666818/

LinkedIn -

https://www.linkedin.com/company/linkedin/

Apache Airflow -

https://airflow.apache.org/

Kubernetes -

https://kubernetes.io/

Open Policy Agent (OPA) -

https://www.openpolicyagent.org/

Backstage -

https://backstage.io/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

  continue reading

49 episoder

Alle episoder

×
 
Loading …

Velkommen til Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

 

Hurtig referencevejledning

Lyt til dette show, mens du udforsker
Afspil