Diray Media is a leading DRTV Direct Response Advertising Agency focusing on Integrated Direct Response (IDR), combining TV and Digital to create an integrated, customized campaign that maximizes ROI for its customers.
Diray enables its customers to get the power of TV, online, mobile, and social media working together to tell more stories, activate more responses, and drive more sales. Their proprietary software platforms, MediaSync and MediaInsights, ensure all media channels are working in unison to achieve optimum results. This technology integrates media buys and syncs with additional analytics and reporting to provide the best data allowing them to hyper target specific audiences to ensure their customer’s ad dollar is used effectively to maximize online and retail sales.
Challenges
Diray Media needed the ability to ingest, analyze, and report on large amounts of both new and updated data from numerous sources. Then they needed the ability to process the data into a Redshift Data Warehouse in a timely and efficient manner. The intraday ingestion of data contains not only new records sets, but also updates to previous records. Although Redshift is a perfect solution for large summarization queries over records spanning multiple years, updating the intraday record set in Redshift is neither efficient nor best practice.
Columnar Database Management Systems like Redshift are not optimal for multi-intraday data loads since many of those records, in Diray’s case, were updates. As updates occur, the system would become unreliable at the current resource node allocation level. While it was possible to add additional nodes to aid loading during those processing windows, the additional resources became underutilized when outside of those update windows.
In order to achieve an optimal architectural configuration centered on best practice, a better approach was needed for the loading and processing of data into Redshift. Additional architectural objectives include meeting the time window allocated for intraday data processing, mitigating and controlling costs, and enabling Diray to have the capability of reporting on the latest data.
Solution
The solution was designed to accommodate daily data feeds consisting of a sizable amount of new records, in addition to updates to existing records. We established an OLTP on Redshift to handle intraday activity. The core analytical Data Warehouse changed to Snowflake, and was loaded from the Redshift during off peak hours. All supplemental tables, including control tables, were moved to the RDS, eliminating superfluous data in Redshift which left only the data warehouse and legacy reporting tables in the cluster.
