Combating shadow IT, data and AI – how to increase control and deliver secure access to your data

The combination of the internet and a more digitally literate workforce has made technology central to how everyone does their jobs. Thanks to technologies such as the cloud, access to software has become democratized, cheaper, and more flexible, while data can be easily stored locally on laptops, and public generative AI chatbots can be used just by asking simple questions. This is driving the rise of shadow technology – unauthorized applications, data and AI used by employees without sign-off from the IT department. 

While these trends may deliver some benefits to individuals and departments, shadow IT, shadow data and shadow AI brings challenges around compliance, risk, and cost. How can Chief Information Officers (CIOs) and Chief Data Officers (CDOs) overcome the causes of shadow technology and what is the role of the data product marketplace in delivering greater control and compliance?

Employees and departments want to use technology to do their jobs. And they want this technology to meet their specific needs and requirements. Often, they feel that existing IT solutions are simply not fit for purpose, aren’t available, or are too complex to learn or use. Data cannot easily be found or is difficult to understand. In-house AI capabilities lag what is available on the internet. This all leads to frustration with existing corporate IT infrastructure and processes. 

Given that many cloud-based apps are now either free or low-cost, employees or managers prefer to buy or subscribe to them personally or via a company credit card, rather than clearing it with IT. Additionally, the line between applications used inside and outside work is blurring, leading to business conversations taking place on personal apps such as WhatsApp or data being stored in private Google Drives.

The combination of dissatisfaction with existing systems and the ease of access to unauthorized alternatives is driving an explosion in shadow technology. A 2023 Gartner survey found that 88% of organizations have seen Shadow IT usage, with employees deploying an average of 1,200 unauthorized apps per company. 

Shadow technology now covers four key areas: 

Shadow IT:

These are applications, devices or systems used without approval from IT. For example, this could be using tools such as Dropbox without company approval, signing up for SaaS tools online or using personal laptops/messaging apps for work. IT has no visibility or control over these unapproved tools and they are therefore not tracked or subject to company security or compliance policies and processes.

Shadow data:

This is data that exists within an organization but is not known, tracked, or governed by official data management systems and policies. It could be data generated by shadow IT systems or simply data such as spreadsheets copied from central systems and stored locally or in personal cloud-based solutions. Again, this brings risks as IT and data teams have no record of this data, leading to potential compliance issues and inconsistency as out-of-date information is used by employees and AI.

Shadow AI:

At first glance shadow AI may look like a subset of shadow IT, with employees using public generative AI tools, such as ChatGPT or Google Gemini as part of their roles. Showing the prevalence of shadow AI, a Gartner survey revealed that 69% of organizations suspect or have evidence that employees are using prohibited public generative AI tools. Unlike traditional shadow IT, shadow AI introduces new risks around data exposure and model behavior, with confidential data or intellectual property freely shared with public models, potentially in breach of regulations such as the GDPR.

• Shadow IT = using unapproved tools
• Shadow AI = using tools that learn from, process, or generate data

Dark data:

While dark data is a less well-known term, it is an equally important challenge for businesses. It is the data that an organization collects and stores but then does not use or analyze. This could be historical transaction data, emails or log files, for example. As it is not being used, it means it doesn’t deliver value but adds to costs as it is stored in corporate systems.

The combined threat of shadow technology

Often, organizations see a combination of all four of these issues:

•   Shadow IT = unknown tools
•   Shadow data = unknown data
•   Dark data = unused data
•   Shadow AI = uncontrolled intelligence acting on data

A growing issue is that Shadow AI often creates or amplifies the other three. For example:

•   Employee uses unapproved tools (shadow IT)
•   Inputs/outputs from these tools is stored outside corporate systems (shadow data)
•   This generated content is not tracked or reused (dark data)
•   Data is fed into public generative AI chatbots (shadow AI)

Shadow technology, including dark data, brings a range of risks and costs to the business that may not be clearly understood by the employees using them.

Security risks

Sensitive data, including personally identifiable information, may be stored outside corporate systems and safeguards. Data and apps are not properly governed, or subject to security policies. In terms of shadow AI, proprietary business information can be entered into public chatbots, inadvertently leaking it to the world and adding to the training data used for models, removing competitive advantage. Data on a personal device or cloud service can be hacked, stolen and subject to ransomware demands.

Compliance issues

Storing data locally or using unauthorized systems brings potential compliance issues. It may violate regulations such as the GDPR around how data is handled or where it is sent. Messages in private WhatsApp conversations can be difficult to access or may be deleted without a record being kept. If shadow AI is used, there’s no audit trail of how decisions are reached, breaching AI governance expectations and emerging regulations such as the EU AI Act. Organizations risk fines, reputational damage and lawsuits from compliance issues caused by shadow technology.

Additional costs

While they may seem like low-cost options to the employee, shadow technology brings costs to the business. Firstly, many purchases are charged back through corporate expenses, so are not free. Multiple people may subscribe to the same services, making them less cost-effective. They may duplicate what is already available, and existing systems remain unused, reducing their ROI. Cloud and processing costs will skyrocket – hitting IT and data budgets without delivering any benefits.

Inconsistent and incorrect results

Shadow data downloaded to individual laptops quickly goes out of date and is not governed or refreshed. Data therefore becomes inconsistent, with multiple versions of the truth emerging and no clear owner of the information. Without being verified, shadow AI can hallucinate, delivering plausible, yet incorrect results. This all leads to poor decision-making, hurting business performance.

Impact on value

CDOs are focused on turning their data into value. Invisible dark data is not analyzed or shared, hampering this aim. This removes a potential source of value for organizations, both for internal consumption and external sharing and monetization. Shadow data also impacts value as it is either not shared across the organization or becomes out of date and untrustworthy.

There are a range of approaches to managing and eliminating shadow technology. It typically requires a combination of enforcement technology, clear policies, education and listening to user needs. Essentially, if organizations can provide seamless access to usable solutions and the right data within their IT infrastructure, then most employees will simply use these rather than looking elsewhere.
 

Detecting shadow technology

To identify shadow IT and shadow AI organizations can use tools such as network monitoring, identity system logs and expense audits to spot unauthorized applications. For shadow data, discovery and data catalogs provide an inventory of all an organization’s data, even if stored on local machines. When it comes to dark data, businesses should deploy storage analysis to show which data has not been accessed recently and data lineage to track the data journey.

Reducing shadow technology

Simply banning shadow IT without providing alternatives will not work long-term. Tech teams should therefore provide approved alternatives or on-board and certify popular systems if they meet corporate security policies. New tools need to be introduced quickly, as often it is frustration with long deployment timelines that cause employees to look elsewhere. Creating and enforcing data classification and storage policies helps reduce shadow data, while data lifecycle policies and analytics identify which dark data can be archived or deleted, and which has high-value.

Educating employees

Most employees do not see the risks that their actions bring to the company. Education and training is therefore critical to show the issues shadow technology causes, particularly around security. Regular audits should be backed up by openness to suggestions for new apps to be added to the corporate IT infrastructure.

Governance frameworks

Adopting specific governance frameworks, such as DAMA-DMBOK, help provide the right processes and roles around data and IT. This enables clear visibility across the IT and data estate, and outlines the policies and actions required to identify and manage potential shadow technology.

Organizations have to meet the needs of employees when it comes to technology. This is particularly true around increasing access to trusted data. This enables both humans and AI to make better decisions, improve performance, innovate and increase efficiency. Traditional methods of data management and sharing, such as data catalogs, are not designed to be usable by non-experts, limiting their value. Instead, the risk is that employees will either not use data at all or will store it on the laptops, with data quickly becoming out of date and inconsistent.

Data product marketplaces solve this issue, connecting everyone in the business with trustworthy, understandable data that they can quickly and securely access and use. Data product marketplaces are a centralized, intuitive, self-service space for data, providing secure, audited access to certified data products and other data assets.

Data product marketplaces help overcome shadow technology challenges through:

• An intuitive, easy to use self-service interface that is based on the familiar experience offered by e-commerce marketplaces and including features such as ratings, AI-based search and similar data recommendations
• Fast deployment, meaning they are up and running within weeks, meaning that users don’t have to wait to access them
• A comprehensive, trusted single source of truth for data across the organization, avoiding the need to store data locally and turning dark data into value
• Understandable by all users (human and AI) through metadata, clear product descriptions and data availability in a range of formats, including visualizations
• Scalable and able to cope with the highest workloads and volumes of data, guaranteeing performance for all users
• Controlled and secure through built-in granular access management and approval workflows that prevent unauthorized access to data while encouraging sharing and consumption
• Increased collaboration, bringing together data owners, IT/data teams and users in a single space, enabling the creation and improvement of data products that meet business needs
• Strong governance policies that provide a catalog of all an organization’s data, monitor and track access and lineage and ensure there is a full audit trail for compliance

The widespread use of shadow technology leads to a lack of visibility and control across the data lifecycle. This adds to risk and costs, and most importantly prevents the organization’s data being shared and consumed at scale. Implementing the right safeguards, coupled with an intuitive self-service data product marketplace moves data and technology out of the shadows and drives tangible business impact.