Co- Authors: Aditya Hegde and Saumi Bandyopadhyay 2022 was a year driven by change for the Talent Acquisition industry, with nearly 50k company mergers and acquisitions completed worldwide. As of November 2023, roughly 150K+ recruiters switched jobs in the previous 12 months as shown in Figure 1. These changes – whether at an organization level or a user level – result in ownership transfers of hiring entities. Figure 1: Talent pool report for recruiters - LinkedIn Talent Insights During mergers and acquisitions, the source company’s user licenses and data are transferred to the […]

Improving Recruiting Efficiency with a Hybrid Bulk Data Processing Framework

Streaming SQL in Data Mesh

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers. This robust framework empowers near real-time data processing for critical services and platforms, ranging from machine learning and notifications to anti-abuse AI modeling. With over 950 million members, ensuring that our platform is running smoothly is critical to connecting members to opportunities worldwide. In this case study, […]

Revolutionizing Real-Time Streaming Processing: 4 Trillion Events Daily at LinkedIn

For the last several years, internal infrastructure at LinkedIn has been built around a self-service model, enabling developers to onboard themselves with minimal support. We have various user experiences that let application teams provision their own resources and infrastructure, generally by filling out forms or using command-line tools. For example, developers can provision Kafka topics, Espresso tables, Venice stores and more via Nuage, our internal cloud-like infra management platform. These self-service integrations are typically owned by the teams that build and support the […]

CoFeed

Posts about #stream-processing from the top innovative companies, all in one place.