Leading AI Experts Advice on Data Preparation for AI Deployment

AI is becoming a significant part of day-to-day operations for companies around the world. In the latest McKinsey Global Survey on AI, 65 percent of respondents report that their organizations are regularly using gen AI, nearly double the percentage from previous survey just ten months ago. Companies across various industries—healthcare, finance, retail, and more—are using AI to solve problems and enhance user experiences. But AI’s effectiveness hinges on the quality of the data it learns from. Data is like the fuel that powers AI. If this fuel is messy, incomplete, or biased, the AI won’t function properly, just as a car won't run well on bad fuel.

Crude Data Hidden in Document Reserves

Research by McKinsey & Company revealed that 65% of companies use document management systems as part of their AI and data analytics strategy, emphasizing the role of documents and structured data in deploying AI solutions effectively.

While data can come from a variety of sources — social media, email, IoT devices, CRM systems, and more — it's important to recognize the role of documents in AI deployment. According to a research by Varonis, a surprising 30% of mission-critical data is still locked within documents, which are often overlooked in favor of more readily accessible digital content.

Moreover, organizations are expressing interest in unlocking potential insights from old legacy documents, some of which are as old as 50 years. These historical documents are rich in data that can inform future trends and enhance AI models. Contrary to the misconception that only recent data is valuable, data from years past can significantly contribute to comprehensive analysis and insightful AI outputs. AI thrives on vast and diverse datasets, making the inclusion of legacy documents a powerful strategy.

The Importance of Data Preparation in AI Deployment

In a “Challenges in AI and Machine Learning Projects" survey conducted by Dimensional Research, 96% of AI and machine learning (ML) projects encounter problems related to data quality, data labeling, and data preparation. This shows the importance of structured data in overcoming these challenges.

Foundation of AI Deployment:

Effective AI deployment starts with a strong foundation built on three key priorities:

Change Management: AI adoption often brings significant changes in workflows, roles, and responsibilities. Organizations need to prepare their teams through comprehensive change management strategies. This includes educating employees about AI capabilities, fostering a culture of continuous learning, and addressing concerns about job displacement or changes in daily tasks.

System Interoperability: AI systems often need to interact with various other software and platforms within an organization. Ensuring interoperability between different systems is crucial for seamless data flow and integration. This requires robust APIs, standardized data exchange formats, and the adoption of industry-specific interoperability standards.

Structured Data & Standardized Formats: The format in which data is stored and processed can significantly impact AI's ability to analyze and learn from it. Standardizing data formats across different departments and systems ensures that AI models can easily access and interpret data without needing extensive preprocessing. This standardization is essential for maintaining data quality and consistency.

Challenges of Data Preparation

Data preparation involves multiple steps, including collecting, cleaning, and organizing data emanating from transactional databases, CRM systems, IoT devices, social media platforms, and documents. To get this data to the point where it can be consumed by AI tools, organizations must design processes that include removing duplicate records, normalizing data entries to ensure consistency, and handling missing values through imputation techniques.

Here are some common challenges organizations face:

Incomplete Data: Missing information can lead to incorrect or unreliable AI predictions, much like trying to solve a puzzle without all the pieces.

Inconsistent Data: Different sources often use different formats or standards, which can confuse AI systems. Standardization and consistency are critical for accurate learning.

Noisy Data: Errors and irrelevant information (noise) can make it difficult for AI to detect useful patterns, similar to listening to a song on a radio with lots of static.

80% of the time spent on AI projects is devoted to data preparation tasks, such as cleaning, organizing, and labeling, demonstrating the critical role of managing structured data for successful AI deployment. - Forbes

Current Industry Landscape for AI adoption

Despite these challenges, many companies are heavily investing in AI, recognizing its potential to transform industries. Reports indicate that AI adoption is rapidly growing in sectors like healthcare, finance, and retail. However, data quality and preparation issues are common reasons many AI projects fail to meet their objectives.

For example, a survey by McKinsey found that over 80% of executives identified data issues as a primary cause of failure in their AI initiatives. Similarly, Gartner found that 87% of organizations consider data quality and integration issues a significant barrier to AI adoption.

Key Insights from Leading AI Experts

Insight 1: Importance of High-Quality Data

DJ Patil, a former U.S. Chief Data Scientist, influential in promoting data science and AI in government and industry, emphasizes the role of structured data in making AI systems robust and reliable. He has advocated for better practices in data management, including structuring data from documents to improve the efficiency and effectiveness of AI, especially in public sector applications.

For organizations, this means that documents and content must be clean, accurate, and consistently formatted. High-quality data ensures that AI systems can learn and operate effectively, producing reliable outputs.

Data structuring is a not a one-time project, however. Organizations must create scalable data management strategies that allow continuous updating and refining of data to keep AI systems accurate and up-to-date.

Insight 2: Data Cleaning, Organization and Interoperability

Angela Shen-Hsieh, an expert in AI and digital transformation, has frequently emphasized the critical importance of preparing and priming data effectively for successful AI implementation in organizations.

Shen-Hsieh emphasizes that for AI to be effective, data must be properly contextualized and structured. This involves not only cleaning and organizing the data but also understanding the specific needs of the AI applications being deployed. For document and content management, this might include tagging, categorization, and metadata enrichment, making the data more accessible and meaningful for AI algorithms.

According to Angela, priming data also involves ensuring that it can be easily integrated and interoperable across various systems within an organization. This is crucial because AI systems often require access to data from multiple sources. Consistent formats and standardization practices can help avoid silos and facilitate seamless data flow, enhancing AI's ability to draw insights.

Insight 3: Continuous Adaptation and Improvement

Ethan Mollick, a well-known professor at the Wharton School of the University of Pennsylvania, focuses on innovation, entrepreneurship, and the impact of AI on business and education. Mollick is a proponent of the iterative process in innovation, including the development of AI systems. This perspective suggests that data priming should be a continuous effort, with organizations regularly updating and refining their data to keep pace with changing business environments and new AI capabilities. Ongoing data management and governance ensure that AI systems remain relevant and effective over time.

Practical Strategies for Effective Data Preparation

To ensure AI systems perform accurately and ethically, organizations must adopt practical strategies for data preparation.

1. Define the Problem and Identify Data Needs

Start by clearly defining the business problem that the AI system aims to solve. Identify the specific data types that are relevant to this problem, whether structured (like transaction records) or unstructured (like text documents or images). Best practices include engaging with stakeholders to understand data requirements and setting measurable objectives for the AI project.

2. Collect Data from Reliable and Diverse Sources

Gather data from a variety of sources to ensure the AI system can generalize across different scenarios. In industries like manufacturing, this might include sensor data from machinery; in pharmaceuticals, clinical trial data and patient records; and in energy, geospatial data and regulatory documents. Diversifying data sources helps improve the robustness and reliability of AI models.

3. Clean and Preprocess Data

Data cleaning involves removing duplicate records, correcting errors, normalizing data formats, and handling missing values. For document data, this might mean using Optical Character Recognition (OCR) technology to digitize paper documents and ensuring that text data is free from formatting issues that could interfere with analysis. Normalization ensures consistency in data entry formats, while deduplication removes redundant information that can skew AI training.

4. Annotate and Label Data Correctly

Accurate labeling is a ‘must’ for training AI, especially in supervised learning contexts. In practice, this could involve using AI-assisted annotation tools to expedite the labeling process while ensuring accuracy. Best practices include setting clear guidelines for human annotators, using double-blind annotation where feasible, and employing automated tools to verify labeling accuracy.

5. Regularly Update and Monitor Data

AI systems need to be continuously fed with up-to-date information to remain relevant. Organizations should implement regular data refresh cycles and monitor AI outputs for signs of model drift—where the AI's performance degrades over time due to changes in the underlying data. Setting up automated alerts for anomalies in AI predictions can help identify and correct drift early.

Tools and Technologies for Data Preparation

A survey by Forrester found that 60% of enterprises are investing in AI to automate document processing tasks, with a primary focus on transforming unstructured data into structured formats for better analytics and decision-making.

Automated data preparation tools are increasingly being adopted to handle unstructured data. For example, by 2026, Gartner predicts that 70% of data preparation for AI projects will be done using self-service tools that automate data cleaning and transformation.

• Structured Data Software: Tools like Adlib ensure all data in documents is structured and machine-readable, which is crucial for effective AI training.

• Data Cleaning Software: Tools like Trifacta and Talend automate data cleaning, reducing manual errors and improving efficiency.

• AI Annotation Platforms: Platforms like Labelbox streamline data labeling, making it easier to manage and track the annotation process.

• Data Governance Tools: Tools like Alation help manage data quality, ensure compliance with regulations, and maintain data integrity.

• Automated Data Preparation Tools: AI-driven platforms like DataRobot automate various aspects of data preparation, including bias detection and feature engineering.

Expert Opinions on Future Trends

An IDC report highlighted that 56% of organizations view document data extraction and management as critical capabilities for AI and ML initiatives, particularly in sectors like healthcare, finance, and public administration.

In the financial sector, AI-driven document automation can reduce processing time by 80%, leading to faster decision-making and improved customer service.

AI-enabled document processing is projected to grow significantly, with the intelligent document processing (IDP) market expected to reach $6.8 billion by 2027, reflecting the increasing importance of transforming unstructured data into structured formats.

Real-Time Data Integration: AI systems will increasingly rely on real-time data feeds, requiring continuous data cleaning and monitoring. This trend is driven by the need for immediate decision-making in dynamic environments, such as finance (for fraud detection) or healthcare (for patient monitoring). Real-time integration means implementing streaming data platforms and robust monitoring systems to ensure data quality is maintained continuously.

Automation in Data Preparation: Automation will play a critical role in reducing the manual effort involved in data cleaning, labeling, and bias detection. Tools like Adlib can automate the transformation of unstructured documents into structured formats, which can then be used for machine learning. The use of AI to manage AI data preparation not only improves efficiency but also helps minimize human error and bias.

Governance and Compliance: As AI adoption grows, there will be increasing emphasis on data governance and compliance, particularly concerning document transformation and management. Organizations will need to implement strict data governance frameworks that cover data lineage, access controls, and auditability. Compliance with regulations like GDPR will be essential to avoid legal risks and maintain customer trust.

Conclusion

The success of AI depends heavily on well-prepared data. By following best practices—such as collecting diverse, high-quality data, moving towards automated data operations, and continuously monitoring AI systems—organizations can achieve accurate, reliable, and fair AI outcomes.

AI is quickly becoming a part of our future, but its success depends on the actions we take today. Focusing on high-quality data preparation, ethical guidelines, and leveraging automation for document processing and data transformation will be key to developing AI systems that are intelligent, responsible and aligned with human values.