Caylent Services
Data Modernization & Analytics
From implementing data lakes and migrating off commercial databases to optimizing data flows between systems, turn your data into insights with AWS cloud native data services.
Print fulfillment leader doubles access speed with AWS data lakehouse
Data Modernization & Analytics
2x
Increase in data access speed
4over is an industry leader in print order fulfillment for print resellers and industry professionals. Specializing in 4-color standardized products, 4over offers a suite of business identity products ranging from business cards to packaging and large format capabilities. Since its inception in 2000, 4over has experienced triple digit growth annually. In only a few short years, 4over has grown into a market leader in the Trade Print industry US-wide.
As a result of the company's immense growth, 4over has expanded from a single building in 2001 to seven fully operational facilities in California, Arizona, Texas, Illinois, Ohio, New Jersey and Florida. 4over’s production facilities range from 42,000 square feet to 105,000 square feet and are equipped with the state of the art presses and finishing equipment.
Challenge
4over wanted to help their business teams procure insights around customer behavior, product demand, and aligning production and supply with the needs of the market. They had a legacy data warehouse hosted in PostgreSQL with only a subset of their data. In order to create foundations for data analytics, 4over was looking to implement a data lakehouse on AWS in an agile manner via multiple phases. The first phase was focused on an MVP lakehouse foundation, enabling immediate and future expansion. Caylent worked with 4over to deploy an MVP data lakehouse that replaces the current PostgreSQL data warehouse. This involves:
- Migrating their existing data warehouse to new data lakehouse (“historical data”)
- Enabling incremental delta loads from source systems (“ongoing data”)
- Scalable data ingestion pipelines for future growth
- Data storage structure that balances between cost and value
"I always felt that Caylent was working on what a good architecture should be instead of getting the engagement completed as soon as possible. Caylent's engineers ensured our team had the training necessary to operate and innovate with our data infrastructure."
Jon Hirschi
Sr. Director, Software Engineering & DevOps
Solution
Caylent successfully migrated data from PostgreSQL to AWS using DMS and AWS Direct Connect for seamless connectivity between the legacy corporate data center and AWS. After full load to get historical data, Caylent’s team began the CDC (Change Data Capture).
4over’s data is stored in the “Raw” layer in Amazon S3. Glue Crawlers ensure data is cataloged in the Glue Data Catalog. For data transformation to the subsequent layers (Trusted and Analytics), Glue uses delta tables. Ad-hoc queries are performed using Amazon Athena on S3, while Amazon Redshift serves as the data warehouse. AWS Step Functions orchestrate the pipeline, and selective data replication is utilized for a subset of views directly from the source to meet data recency requirements. The CI/CD solution is managed with Terraform using TeamCity pipelines.
Results
As a result of implementing the data migration solution, 100 tables and views from the legacy PostgreSQL data warehouse were successfully migrated to AWS. This migration enabled the 4over to decommission their legacy data warehouse infrastructure. By transitioning to a cloud-native data lakehouse architecture, 4over gained the capability to consolidate enterprise data from many different sources in one place, doubling the speed at which they can access data and enabling downstream users to extract maximum value.
Caylent’s solution enabled incremental delta loads from the source systems, paving the way for future scalability and accommodating the addition of new data sources as 4over’s evolve.
