As enterprises pursue digital transformation, nearly all are seeking to become more data-driven. This often means combining information from a wide range of sources, applications and formats so that it can be analysed to derive useful business insights.
For many businesses, data integration begins with a simple data exchange. For example, an organisation may want to sync information from its inventory platform into its payment solution, or it may want to build a sales & marketing dashboard.
For this to work, organisations have to find ways to break down their data silos so that they can combine and analyse data in new ways. Sometimes they are working for legacy systems which are no longer supported, and pretty difficult to integrate. At the same time they need to their data secure and meeting their compliance needs as usual.
A growing number of enterprises are drafting data integration strategies. Here is our take on the key elements of a data integration strategy:
Who makes up the data integration team?
Will you have IT specialists with knowledge of programming who tackle all data integration tasks? Or do you want to enable knowledge workers from the business side to use data integration tools on their own? Your answer will have a major impact on the type of data integration solutions you choose to purchase. Many large enterprises have a chief data officer (CDO) and data governance team that establishes policy and procedures, but they deploy tools that enable self-service for analysts and other business users involved in day-to-day creation of reports and analytics.
What data might need integration? Many companies have already created an inventory of their data for security or compliance purposes. Revisiting this list can help you see how many different applications and data formats exist in your environment. If you have only a few different data silos, the most cost-effective strategy for you may be to choose a basic data exchange or ETL tool that can handle your specific needs. But if you have a lot of different silos to integrate — or you anticipate that you may want to integrate a lot of different types of data in the future — it might be better to go with a more full-featured data integration platform that can handle many different types of data.
Why are you integrating your data?
Many experts would recommend actually starting with this question. You need to understand the business reasons for your data integration in order to create a suitable strategy. Understanding the business case can also help you determine whether a particular solution will result in a more positive ROI.
When will you do your data integration?
If you are creating a data warehouse, the data integration will likely happen in advance of any analytics. If you are creating a data lake based on Hadoop or similar technologies that store data in its raw, unaltered form, some data integration will take place right before running analytics workloads. Many companies have both a data warehouse and a data lake. The architecture you choose will affect your data integration procedures and the type of technology you need.
Where will your data integration take place?
In the cloud, on premises or both? Will a cloud-based solution meet your needs or do you need an on-premise solution? Or will you take a hybrid approach with some integration occurring in the cloud?
How will you do your data integration?
This last question is the most complicated as it needs to include all data governance, data quality and data security considerations. Data governance is about making sure that the right people have the right access to the right data at the right time. The concept of data quality can be understood as the degree to which data is accurate, consistent, timely, is relevant for the current use, and aligns with business rules. Data security is associated with data governance, but is primarily concerned with preventing unauthorised access to data assets.
Now you should have an idea of the capabilities you need in a data integration tool. Many types of data integration tools are available, including master data management, data governance, data cleansing, data catalogues, data modelling and other tools that include some data integration capabilities.
Some of the most common data solutions we work with at Integrella include:
- APIs provide a programmatic way for one application to share data with another. These application-specific tools are highly technical and designed for use by developers.
- Data integration platforms incorporate a wide range of different capabilities, such as ETL, ELT, data governance, data quality, data security, etc. These tools can integrate data from a wide variety of different sources and are suitable for use by business users.
- Integration Platform as a Service (iPaaS) offerings are cloud-based tools for data integration. They offer ease of use and can integrate data from cloud-based sources, including software as a service (SaaS).
- Data migration services move data from one place to another and may offer some limited capabilities for data transformations as well. Most of the major cloud vendors offer migration services for moving data to the cloud.
Over 10 years in the integration business, we have developed a structured way of helping our clients make these decisions, and help them to define, implement and support their integration projects.
Get in touch to find out how we can help.