Luke Kalish

–Fast and easy migration to HPE’s data fabric saves money and improves performance, functionality, and security–

As is true for many companies that started their big data strategy on Hadoop and have been using it for years, Bidtellect was unexpectedly presented with a high renewal bill for support from their vendor. The bill motivated Bidtellect to migrate to a new vendor on which to run their cloud data management and analytics.

Bidtellect recognized their migration would not be easy. Real-time data management and analytics at petabyte scale is a mission critical function at Bidtellect; therefore, a migration must be carefully managed. Despite this enormous pressure, Bidtellect achieved complete migration to HPE Ezmeral in just five weeks. This case study examines the successful transition and benefits of moving to HPE Ezmeral to replace legacy Hadoop with a modern data fabric.

The data needs 

Bidtellect is a demand-side platform (DSP) within the online advertising ecosystem. The company runs online bid requests and auctions for digital advertising. Bidtellect works with clients to deliver ads and coordinates with publishers that need ads for their sites by determining the right ads that meet the specifications of the request. Each campaign and request are highly targeted (based on size of the advertisement, location of the reader, or mobile versus desktop). These requests are handled through an open auction approach, in which Bidtellect competes with other DSPs to win the ad space. Bidtellect has cultivated a large database of possible ads with its advertising clients, allowing the company to respond immediately to requests.

Bidtellect operates in a highly competitive landscape in which the management of data throughout its lifecycle is critical. Every time a customer makes a request, there’s a complex supply and demand interplay before a match is made. Bidtellect collects and captures all this data so that it can document delivery, analyze performance, and better serve its customers in the future.

Each time an ad is delivered, Bidtellect needs to collect tracking data to know which publishers need to be paid and deduct funds from the relevant customer campaigns. These auctions occur hundreds of thousands of times per second on Bidtellect’s platform, generating a tremendous amount of data. Both winning and losing bids need to be analyzed constantly to ensure future bids are as competitive as possible.

Data analysis and matching campaigns based on past performance of particular ad placements and win rates are also continuously collected. This data collection occurs in real time across multiple data centers, which is fed back into the company’s main cluster. Obviously, the company’s need for real-time big data management and analytics is immense — at one point, Bidtellect was running 41 Hadoop nodes with two petabytes of data globally.

The issue

Given the spiraling upward costs presented by the Hadoop vendor, Bidtellect recognized it was time to migrate to a new data platform; however, the migration had to happen quickly, as Bidtellect’s business would suffer due to any outages or delays. Bidtellect decided to migrate into the HPE Ezmeral ecosystem because it offered both MapR (now HPE Ezmeral Data Fabric) as a big data repository and Hadoop platform and Apache Spark as a data supply chain and data science environment. While the move was initially motivated by price increases and capability issues with the Hadoop vendor, other motivating factors included functionality, data access and efficiency issues, and security concerns.

The solution

Bidtellect advises its customers to take a targeted approach to advertising and make the greatest use of their media spend. The company’s systems avoid wasteful practices such as undisciplined firehosing of ads all over the internet. Like a high-performance race car on cruise control, Bidtellect’s optimization engine monitors and controls customer budgets, assuring an even spend throughout the lifetime of the campaign. To achieve this goal, it needs instant feedback.

The migration to HPE Ezmeral has been tremendously successful in achieving real-time feedback. Running Spark Streaming on YARN as its resource manager allows for tracking of each campaign’s spend in real time; and Spark for batch processing is critical for generating hourly, daily, and monthly aggregated reports.

The data now lives in the MapR (HPE Ezmeral Data Fabric) file system, which means it can be used in batch processing to sum up data on impressions and clicks and other critical campaign metrics, which are then used as the backbone for customer campaign performance reports. Additionally, MapR Streams serves as the data transportation backbone of all generated event data from multiple datacenters to a central cluster, where Spark processes the event data to create ML models. Bidtellect also relies on Aerospike for real-time frequency capping from streaming data, which helps to ensure users don’t see the same ad multiple times.

By moving to HPE Ezmeral, Bidtellect solved many of the data management problems it experienced with their legacy Hadoop vendor and achieved four key advantages:

1. More efficient data storage:

Bidtellect has a total of 120 different jobs operating off their stored data at any moment. Unlike when using the Hadoop vendor, Bidtellect is now able to deprioritize low value data to ensure saving space on their clusters.

2. Higher performance:

Bidtellect can better manage the speed and frequency of replications and can adjust on a volume-by-volume basis, meaning it can performance-tune data usage.

“We have transparency and visibility into data storage and usage, which we didn’t have before,” explains Luke Kalish, SVP of Engineering at Bidtellect. “Our clusters can hold more of our important datasets as we now have the ability to control the replication factor on a per-volume basis. Larger and less important data sets can have fewer replicas, freeing room for our most important data with higher replication factors.

3. Improved security:

Bidtellect now has complete control over permissions and user access — security it didn’t have with the Hadoop vendor. 

4. An easy-to-access file system architecture:

HPE Ezmeral’s file system architecture provides Bidtellect with NFS access to its Hadoop-based file systems. This means files in a Hadoop cluster can be accessed in read/write fashion as if they were normal files attached to a Linux machine — they don’t have to be moved to allow access, which is an inefficient and cumbersome practice.

The move to HPE Ezmeral has greatly aided Bidtellect in its data management needs. It also occurred at the fast speed the company required. Bidtellect lowered data costs, while simultaneously improving functionality. The company is also considering expanding its use of Spark to get even more out of the platform.

To learn more about HPE Ezmeral, visit https://www.hpe.com/software.

About the authors:

Shane Margraves brings more than 25 years of experience in global technology sales to his role at HPE, as vice president of worldwide sales for HPE Ezmeral.  Shane joined HPE through the acquisition of BlueData in 2018.  He came to BlueData from Dell EMC, where he served as vice president of global OEM storage sales and chief operating officer for global storage, converged, and data protection sales. At Dell EMC, he created the go-to-market strategy for a multi-billion-dollar business across multiple business units, dozens of products, and thousands of sellers. Before joining Dell EMC, he was senior director of sales at Hitachi Data Systems.

Scott Russmann is a Practice Director within HPE Ezmeral Professional Services. Scott has a broad technical background and over 15 years’ experience leading software engineering organizations for startups and large enterprises. Prior to joining HPE, Scott was the Director of Software Architecture at SecureSky, and Vice President of Software Development at NTT Security prior to that. Before joining NTT Security, Scott led the Central Region of Professional Services at MapR which was acquired by HPE in 2019. 

Luke Kalish has been developing tech in the digital advertising space for 10+ years and 20+ years developing software. As SVP of Engineering, Luke is responsible for overseeing the Quality, UI/UX, and Platform Engineering teams.  He is also the 3rd longest-tenured employee at Bidtellect and has worn many titles such as Lead Developer, Director of Technology, and Infrastructure. Prior to Bidtellect, Luke has worked for such companies as CNET, Mediawhiz (now Matomy) in addition to his own company.

Robert Komjathy is Director of Data Services at Bidtellect overseeing all data operations. In his nearly decade-long tenure at Bidtellect, he has proven himself a fiercely dedicated developer with attention to customer perspective and problem solving. Through his various roles, he has been integral to Bidtellect’s third-party integrations, product features and enhancements, operations, and bidding optimization engine. Prior to Bidtellect he worked for ShopLocal and earned an M.S. from University of Washington and his B.B.A. from Western Michigan University.

About Bidtellect:

Bidtellect is a performance-driven DSP specializing in context-first optimization, cookieless solutions, and native programmatic. We pride ourselves in premium supply quality and optimizing down to the placement level, while offering deep campaign insights across a wide range of KPIs for our brand, agency, and trading desk partners. Bidtellect consistently outperforms competitors in nearly every head-to-head performance test thanks to industry-leading brand safety technology, premium supply quality, superior context capabilities, and proprietary bid factoring and optimization technology: delivering ads that work.