Today, organizations face tough challenges in managing data. Businesses now sift through thousands of documents in different formats—PDFs, spreadsheets, images, and multimedia. Bogdan Raduta at FlowX.ai highlights that pulling these diverse sources together into something meaningful is critical.
Each data type comes with its own set of rules and structures. Without a way to integrate them, organizations end up with data silos. This leads to frustrating situations where users bounce between applications, copying and pasting info just to glean insights for better decision-making. Traditional data processing methods struggle in this landscape. Raduta points out that while conventional ETL (extract, transform, load) systems work well with structured data, they stumble when faced with the messiness and unpredictability of real-world data.
Even modern integration platforms often can’t grasp the nuances of natural language, making it hard to process diverse content effectively. Jesse Anderson from the Big Data Institute sees a gap in understanding what data science roles really entail. Many mistakenly believe that data scientists handle all engineering tasks, but Anderson argues otherwise. He says if you want to hear about data limitations, go to the ‘no team’ in data warehousing, and you’ll get told, “no, it can’t be done.” This mindset is damaging. When teams can’t move forward with data projects, progress stalls.
Anderson also notes that the term “data engineer” can mean different things. One definition refers to someone skilled in SQL, able to query data from various sources. Another refers to a software engineer with expertise in developing data systems. The latter can create complex architectures rather than relying on simpler systems that often use low-code or no-code solutions. According to him, coding ability is crucial for data engineers, especially as the complexity of data requirements increases.
However, building an effective data engineering team isn’t easy. It requires serious shifts within an organization. Anderson emphasizes the need to persuade executives to fund these teams, to ensure competitive pay, and to help stakeholders see how valuable a skilled data engineering group can be.
Justin Pront from TetraScience shares an example from the pharmaceutical industry. When a major pharma company attempted to analyze a year’s worth of bioprocessing data using AI, they hit a familiar barrier: the data was “accessible” but virtually unusable. Instruments generated readings in proprietary formats, leading to fragmented systems. This made answering basic questions, like experimental conditions, a tedious process.
Pront believes that scientific data really puts enterprise data systems to the test. Scientific data comes from various sensitive instruments and includes unstructured notes, which complicate the analysis. He identifies three principles vital for any organization looking to boost its data engineering efforts: adopting data-centric architectures, preserving context during data transformation, and ensuring unified access patterns for current and future analysis.
He explains that scientists typically view files as their main data containers, but this compartmentalizes information and loses vital context—making it tough to perform aggregate or exploratory analyses. He stresses that modern data engineering should prioritize the relationships and metadata that give data its value.
Ensuring data integrity is critical, particularly in industries like healthcare and finance. One minor error in scientific data can lead to big misinterpretations. That’s why reliable data collection and repeatable processing are essential.
Pront sees a struggle between providing immediate access to data and ensuring long-term utility. Scientists often use basic tools like spreadsheets to analyze data, which only adds to existing silos. By contrast, cloud-based datasets can help bridge this gap, allowing for quick analyses while maintaining data readiness for advanced applications and regulatory needs. He suggests that emerging data lakehouses, built on open formats like Delta and Iceberg, can facilitate unified governance and flexible access.
Returning to the challenge of managing diverse data types, Raduta notes that traditional ETL methods fall short. A promising development is the rise of large language models (LLMs). He believes these models offer a new approach by understanding context and extracting meaning from unstructured content. Instead of relying solely on deterministic ETL transformations, he argues that LLMs can transform documents into searchable data.
Raduta envisions an intelligent ingestion layer that accepts various input sources and comprehends the content of each. This marks a shift from one-size-fits-all solutions to more adaptable architectures.
Pront recommends that IT leaders treat data engineering as an evolving field. As Anderson points out, developing these capabilities requires a blend of programming and data science skills. IT leaders need to make a strong case to their boards and HR teams that attracting top-notch data engineers often comes with a price.
