The scene in the movie 3 Idiots, where Rancho asks the students and the head of the engineering college to define a non-existent term, feels strikingly relevant today amid the frenzy surrounding AI. In that scene, every student rushes to search for the made-up term in their textbooks without questioning the merit or genuine nature of the term itself. The frantic page-flipping and the desperate attempts to come up with an answer about something that doesn’t exist remind me of what’s happening among many CXOs today in their race to adopt Gen AI.

Every CXO is rushing to declare they have moved beyond the pilot stage and into monetization, claiming they have evolved from explorer to executioner, armed with a ready-made platform to deliver business impact. We have seen such madness and chaos before with every new wave of technology: Automation, AIOps, Cloud, VDI, and many more. So, is everything truly “green” in AI?

Recently, news circulated across the industry that Deloitte had refunded a sum to the Australian Government due to the errors in their output, apparently caused by AI. While some found this surprising, it wasn’t the first time something like this had happened. 

• In June 2024, McDonald’s stopped the use of its AI-powered drive-thru ordering system. 
• Replit, a coding assistant tool, accidentally deleted its actual database and created thousands of fictional users. 
• Air Canada was even compelled to compensate a passenger for the lies churned out by its chatbot. 

These are just a few instances of AI failures among the many that are happening around us. The fault doesn’t lie with the AI technology itself, but with the industry’s forecasters, analysts, and vendors.

We have heard these predictions many times before. When cloud computing emerged, industry analysts claimed that traditional IT infra services would vanish. Yet, even after 15 years, the cloud hasn’t replaced them. Around 2018-19, similar forecasts suggested that automation would eliminate half of all engineering jobs. Look where we are today.

Now, the same narrative is resurfacing. This time, it’s GenAI that’s expected to replace up to 50% of the IT workforce. Most vendors, particularly IT service providers, tend to echo these analyst predictions. But, can AI truly reduce the workforce significantly?

I remember the very first meeting I had with Professor Nandan Sudarsanam, IIT Madras, at his office. He clearly said that AI cannot guarantee perfect results. Rule-based programs will guarantee conformance to the rule, but AI models are benchmarked on predictive accuracy. At worst, they are just better than random chance; at best, we can get near-perfect predictions, so there is a wide range. This depends upon the quality of the data, availability of meaningful features and noise in the system, volume of data, complexity of the environment we seek to model, and data drift.

This means that if we deploy AI in a call center, up to 10% of customers may not get the right responses. Similarly, a code converter may not be able to convert all the legacy codes into a modern language, and a report generator may give unasked-for reports one out of 10 times. Are we ready to tolerate these margins of error?

Every CFO is asking for returns in the form of hard cash for the investment made on AI. The true quantitative returns can come from reducing the human workforce only, as AI cannot be expected to replace anything else, neither the hardware nor the software. If we have to replace humans with AI, the error percentage of the AI models has to be significantly low.

As of today, activities like writing, creating images, graphic design, and text analysis, are relatively more error-tolerant, with acceptable error rates of up to 50%. However, these activities alone cannot replace enough human effort to provide significant returns.

Some tasks like predicting an outage or a security threat in IT systems, forecasting demand for a specific item in the market, or recommending a loan to a customer, can tolerate a higher error margin (1% to 10%) because the consequences of an error are relatively minor. However, even here, the savings in the human workforce alone are often insufficient to generate a significant ROI.

The real savings will come from deploying AI in what I call ‘real-time business activities,‘ such as call centers, IT help desks, invoice, payment, claim, or order processing, fraud detection, and similar areas, where actions are executed and results get delivered in real time. These activities employ large human workforces across industries, however, AI must achieve extremely high accuracy in these tasks, typically producing fewer than 1% errors. 

But, how can we do that?

Well, there are many techniques to improve the accuracy (I mean reducing the errors), and one of them is Human-in-the-Loop (HITL). It means bringing humans to validate every outcome of AI models every single time. The moment we bring in HITL, the quality of the delivery improves, but not the savings. The CIO, CTO, or CISO will be happy with these qualitative improvements, but not the CFO who expects real cash.

In my last blog, I explored how AI is transforming data documentation and metadata management, turning catalogs into intelligent interfaces rather than static repositories. In case you missed it, read it here.

But here’s the next big question: if AI can now generate SQL queries, optimize pipelines, and even explain dashboards in plain English, which modern data platform is leading the way?

That’s what this installment tackles. I will compare how Databricks, Snowflake, and Microsoft Fabric are embedding AI and LLMs directly into their platforms to enhance data engineering and analytics.

Platform-Level AI

AI in data platforms is no longer an “add-on.” It’s being built natively into workflows, shaping how teams code, discover data, and generate insights. Key areas of infusion include:

• Code generation (SQL, Python, ETL)
• Data discovery and lineage automation
• Pipeline optimization
• Model development and deployment
• Natural language interfaces for analytics

Let’s explore how each platform is building these capabilities.

In the previous blog, I explored how AI enhances data quality and validation, from detecting outliers with ML models to imputing missing values and identifying schema drifts. I even explained how ML helps us move beyond rigid, rule-based checks toward adaptive, intelligent validation strategies. In case you missed it, read it here

In this blog, the focus shifts to one of the most persistent challenges in data engineering: documentation. Maintaining accurate, up-to-date metadata, lineage, and descriptions has long been a tedious, error-prone task. With AI, documentation can evolve into a living, automated asset, reducing manual overhead, improving transparency, and embedding intelligence directly into the data lifecycle.

Why is today’s documentation a bottleneck?

Data ecosystems grow fast. Pipelines evolve. Schemas change. Teams rotate. Amidst this pace, metadata and documentation often lag behind or become outdated altogether. Manual documentation is time consuming, prone to human error, and often deprioritized during delivery sprints. Robust documentation is, therefore, essential for:

• Data discoverability
• Better governance and compliance
• Impact analysis and debugging
• Seamless onboarding of resources and cross-team collaboration

AI-driven solutions for managing metadata

• Lineage graph generation from code

AI models can parse and understand code in SQL, Spark, Python, and orchestration logic (like Airflow, ADF) to generate precise data lineage maps. These models, including LLMs and code parsers, extract the semantic meaning and logical flow of data, eliminating the need for manual mapping. 

This ensures lineage is always in sync with the evolving codebase, accurately showing the source-to-target flow, transformations applied, and dependencies between jobs and tables. The automated approach provides a dynamic and reliable view of your data landscape.

• Entity and relationship detection

AI can infer logical entities (e.g., customer, product, order) and relationships between tables using metadata, field names, sample data and code context. Techniques like clustering based on naming patterns, LLM-driven context analysis, and statistical co-occurrence patterns can be used to detect the same. This allows intelligent grouping of assets in catalogs and supports knowledge graph generation.

• NLP-based column description generation

AI models can now automate the creation of data documentation. They analyze a column’s field name, data type, and usage context to generate clear, readable descriptions. LLMs trained on data dictionaries and contextual clues like the table name or pipeline logic can infer the column’s purpose and provide concise summaries.

       This automation offers several key benefits: 

• Accelerates documentation
• Supports data cataloging tools (e.g., Unity Catalog, Alation, Acryl Data)
• Improves user understanding and self-service analytics

Integration of AI-driven documentation into the data platform

An AI-driven documentation engine typically integrates with catalogs, wikis, and governance platforms. Below is a depiction of how it works:

Implementation considerations

• Accuracy vs. explainability: LLM outputs must be reviewed initially for correctness.
• Version control: Documentation should evolve with pipeline/code changes.
• Human-in-the-loop: Offer editing, approval, and override mechanisms.
• Security & PII handling: Sensitive fields need tagging and masking where applicable.

Closing thoughts

AI is reshaping metadata management from a chore to a strategic enabler. By automating lineage, descriptions, and structure inference, data teams can maintain clarity and control even as their environments scale in complexity. What once required weeks of manual work can now be continuously generated and updated.

Well-documented pipelines are no longer just a “nice to have”. They are foundational to trust, compliance, and operational agility. AI makes that foundation more scalable and sustainable than ever before.

In the previous blog, I explored how AI-powered data pipelines go beyond static workflows, enabling adaptive logic, real-time anomaly detection, and intelligent error handling. I also introduced architectural patterns that embed AI at the core of pipeline design, allowing systems to respond dynamically to evolving data conditions. In case you missed it, read it here

In this blog, the focus shifts to a vital pillar of the data lifecycle: data quality. AI is transforming validation from rigid rules to adaptive, learning-driven methods that scale with complexity. In autonomous pipelines, ensuring trust isn’t optional; it’s a strategic necessity.

Why do traditional approaches fall short?

Traditional data validation methods often fall short in modern environments:

• Manual rules: Static checks (nulls, ranges, regex) require constant maintenance as data evolves.
• Hardcoded logic: Rigid rules tied to fixed schemas break with change.
• Delayed detection: Errors surface after ingestion; too late to avoid downstream impact.
• No context awareness: Rules lack sensitivity to data source, time, or business meaning.
• Limited scalability: These methods struggle with high data volume, velocity, and schema drift.

In a nutshell, rule-based checks catch known issues but struggle to scale or handle the unexpected AI-driven data quality goes further i.e. proactively detecting anomalies, adapting to change, and safeguarding trust at scale.

AI techniques for data quality

• Outlier detection using machine learning – Machine learning models trained on historical data can learn normal patterns and flag deviations that suggest quality issues—often missed by static rule-based checks.

Key benefit → These models adapt to data patterns, enabling intelligent anomaly detection without relying on hardcoded rules.

• Missing value imputation

AI can fill missing values more intelligently than static defaults by learning from historical patterns and correlations across fields, improving data completeness without sacrificing accuracy.

    Popular techniques and usage

Key benefit → Minimizes manual cleanup and supports more reliable downstream analytics.

• Schema drift detection

AI models can detect subtle shifts in schema, data types, or value distributions that static rules often miss.By learning historical patterns, they flag changes early, preventing pipeline breaks and data quality issues.

                         Detection & correction workflow

Key benefit → Proactively detects structural shifts that could silently impact data quality, enabling faster diagnosis and resolution.

• ML-based validation rules

AI learns from past data to create smart validation rules on its own; no coding needed. Unlike fixed checks, these adapt over time, catching issues that traditional methods often miss.

Examples

Key benefit → Minimizes reliance on hardcoded logic, and allows generalization of learned rules across datasets with similar behavior, saving time and improving accuracy.

How it works in practice

• Profiling engine: Learns normal patterns from historical data to establish a behavioral baseline.
• Validation engine: Scores incoming data in real-time, flagging anomalies and unexpected changes.
• Feedback loop: Learns from human input to reduce false positives and improve accuracy over time.
• Rule generator: Suggests intelligent rules based on observed patterns for analyst approval.
• Remediation layer: Recommends or automates corrective actions to keep pipelines clean and flowing.

Implementation considerations

• Metadata integration: AI needs rich metadata context to make accurate decisions.
• Explainability: Models must provide clear reasoning to ensure transparency and trust.
• Performance & latency: Optimize models to avoid slowing down your data pipelines.
• Human-in-the-loop: Allow human review for edge cases where domain expertise is critical.

Closing Thoughts

AI is redefining data quality by moving beyond rigid validations to intelligent, adaptive frameworks that evolve with your data. By embedding learning driven mechanisms into pipelines, organizations can detect issues earlier, respond faster, and maintain trust at scale. As data complexity grows, AI won’t just support quality, it will become its foundation.

Identity and Access Management (IAM) products rely on one critical but often underappreciated capability: connectors. These are the bridges that link IAM systems to directories, applications, databases, SaaS platforms, and more. Whether provisioning user accounts, syncing attributes, or enforcing access policies, connectors are foundational to any IAM deployment.

Traditionally, connectors are:

• Hardcoded or scripted using a connector framework.
• Developed manually by product teams or integration consultants.
• Tedious to maintain as APIs, schemas, and protocols evolve.

But what if we reimagined this model? What if connectors became intelligent, autonomous, and adaptive? 

Traditional IAM connector frameworks: Where we are today

Most mature IAM products ship with:

• A connector framework that allows developers to create plugins or adapters.
• A catalogue of out-of-the-box (OOTB) connectors (e.g., for Active Directory, Workday, Salesforce).
• Professional services or customer teams who build custom connectors using SDKs, scripting, or tools like IBM TDI (Tivoli Directory Integrator).

This model, while proven, has limitations:

• Time-consuming connector development cycles.
• High reliance on professional services (PS) for customization.
• Breakage due to changing target system APIs or schema.
• Inflexibility when integrating modern, cloud-native apps.

Welcome to the era of Agentic AI–driven connector frameworks

Agentic AI can shift connector frameworks from static tools to dynamic, semi-autonomous systems. Here is how IAM products and enterprises can benefit:

1. Self-authoring AI connectors

• IAM products ship with a foundational connector-agent framework.
• AI agents observe target systems’ APIs, authentication methods, and data schemas.
• AI agents automatically generate connector logic, validate integration flows, and propose attribute mappings.
  

Example: An AI agent detects OpenAPI/Swagger spec from a SaaS app and builds a provisioning connector that supports SCIM. The agent auto-generates field mappings and tests connectivity end-to-end.

2. Adaptive connectors (self-healing)

• Agentic connectors detect and respond to API changes or schema drift in target apps.
• Instead of failing silently, they alert admins or auto-correct using pre-learned patterns.

Example: Out-of-the-box connectors often require upgrades due to API changes. An AI agent can identify the delta and patch the connector behavior dynamically.

3. AI-powered low-code/no-code builders

• Product teams provide a visual builder for integration workflows.
• AI agents act as copilots, suggesting best practices, policy templates, and data mappings.
• Connector building becomes accessible to non-developers (e.g., IT admins, analysts).

Outcome: Faster integrations, reduced complexity, and fewer PS hours needed.

Benefits for product teams

• Differentiation: Stand out with AI-powered extensibility.
• Speed: Reduce connector time-to-market from months to days.
• Sustainability: Self-healing connectors reduce support burden.
• Scalability: Let AI agents handle the “long tail” of niche integrations.

Benefits for enterprises

• Rapid onboarding of new apps without waiting for roadmap commitments.
• Reduced PS dependency, cutting costs and time.
• Higher resilience: Connectors don’t break with minor backend changes.

A glimpse into the future

Imagine an IAM product where connectors behave like intelligent agents:

• Continuously learning from logs, user patterns, and API telemetry.
• Adapting to changes in real-time.
• Collaborating with other agents (e.g., policy agents, identity proofing agents).

This is not science fiction; it is a natural progression from static software to agentic systems.

Final Thoughts

Connector frameworks are long overdue for reinvention. As Agentic AI matures, IAM vendors have a unique opportunity to:
Shift from code-heavy to AI-assisted development.

• Empower product and enterprise teams alike.
• Redefine integration as an intelligent, adaptive capability, and not a hardcoded plugin.

Let us move beyond the connector catalog; towards a world where IAM connectors build and maintain themselves.

At Neurealm, we recognized a common challenge faced by both our team and our clients: integrating AI meaningfully into the software development lifecycle (SDLC), even with the rise of AI coding assistants like GitHub Copilot X, Cursor, and AWS Q Developer—tools that already feature integrated AI agents and “vibe coding.” To address this, we evolved our approach by adopting developer-friendly workflows that require attention to formal design, structured architecture, or implementation detail—designed to streamline AI infusion across SDLC stages and drive meaningful productivity gains.

Sounds familiar? You might be thinking of the latest buzzword—’spec-coding,’ the more mature and predictable version of ‘vibe coding’—or perhaps the wave of AI-powered IDEs making headlines. The answer is: something close, but purpose-built to meet real enterprise demands. At Neurealm, we quietly introduced this approach within our own software engineering teams before these new tools/concepts emerged lately, bringing much-needed rigor and structure to enterprise-grade software development.

Rethinking Software Development in the Age of AI

Software development has come a long way—from traditional hand-coded programming to the rise of low-code and no-code platforms. While these visual-based approaches offer rapid development with minimal coding effort, they often fall short when it comes to scalability, flexibility, governance, and integration—especially in complex enterprise environments. In parallel, AI-assisted programming has gone through its own evolution. It began with rule-based and heuristic systems, progressed to machine learning models, and more recently—since 2020—has been shaped by large language models (LLMs). This has given rise to trends like ‘vibe coding,’ which is effective for rapidly building MVPs and prototypes but presents real concerns around code quality, security, and traceability. Now, we’re seeing the emergence of ‘spec-based coding’—a more disciplined and predictable evolution of AI-led development. While vibe coding may eventually mature into a structured and trustworthy part of the SDLC through better tools and integration, today’s reality is that many organizations remain cautious. Unstructured, undocumented workflows simply don’t align with the rigor that enterprise-grade engineering demands. Spec coding takes the best of AI-assisted development and adds the structure and discipline that enterprises need. It’s a step up from ‘vibe coding,’ which can feel fast and flexible but often lacks consistency and control. That said, even spec-based AI Coding Assistants can be hard for teams to adopt. Many AI development tools don’t follow clear, predefined workflows—something enterprise developers are used to and rely on to keep projects on track.

Evolving AI Tools, Lagging Strategy

Since 2023, AI coding tools have rapidly evolved—from interactive assistants with integrated AI agents to increasingly autonomous agents capable of handling tasks independently. These tools now support both ‘vibe coding’ and ‘spec-based’ programming, offering functionality that spans code autocompletion, generation, testing, code review, understanding, and debugging. The roadmap ahead points toward even more advanced capabilities, such as full-stack application generation from specifications, autonomous pull requests, automatic identification and fixing of issues in code, integration with CI/CD pipelines, and self-debugging or refactoring of entire codebases. Yet, in the rush to adopt these tools—often driven by FOMO or the pressure to follow trends—many organizations skip a critical step: taking a structured, transformation-led approach. While AI coding assistants promise productivity gains, realizing real ROI requires more than just tool adoption. Leaders often struggle to measure the actual productivity impact or to communicate value effectively to executive stakeholders. What’s often overlooked in this AI gold rush is the need for careful planning, change management, and the foundational setup required to ensure a successful, enterprise-wide rollout.

Our Take at Neurealm

One question we hear consistently from customers and prospects is: Is the promise of AI in development real, or just another myth? More and more, we’re being brought into thought leadership conversations where the central theme is whether there’s a practical, predefined workflow that enterprise teams can follow—using the AI coding assistants they’ve already adopted—without needing to embark on a long, complex transformation journey. Leaders are eager for quick wins and tangible value realization. However, with the ever-growing ecosystem of software development tools and methodologies, measuring productivity in a consistent, deterministic way—and connecting that to ROI—has become increasingly difficult. For many IT leaders, this lack of clear metrics and repeatable frameworks adds another layer of complexity to what should be a strategic and impactful shift. At Neurealm, we believe that the current perception gap—myth versus reality—and the demand for predefined workflows will gradually diminish as AI literacy in software development grows and the technology continues to mature. However, for now and in the near future, AI coding tools and humans are best viewed as pair programmers—supporting and enhancing human developers, not replacing them as independent peers. The human-in-the-loop remains essential to ensure quality, context, and engineering discipline.

From Markdown to Mastery: Automating the SDLC with AI

At Neurealm, we take a structured, specification-first approach to AI-assisted development. We guide AI coding assistants and their integrated agents using thoughtfully crafted Markdown files that span across key SDLC phases—Requirements & Planning, Design & Architecture, Development & Coding, Testing & QA, and Deployment & DevOps. These specifications are written with human judgment, peer review, and sound engineering discipline, ensuring clarity and intent throughout the lifecycle. Our development environments are powered by integrated AI agents in AI coding assistants within the IDE, supported by our MCP servers. These servers provide tools that interact with the terminal, file system, Git, Diagrams as Code, UI/UX Mockup Design, and Azure DevOps to semi-automate a variety of high-effort tasks. This includes requirements engineering and planning (Epics, Features, User Stories with Acceptance Criteria), generating technical design documents (Call Flows, Data Flows, API definitions, Mermaid, C4 Model Diagram, and IaC diagrams), creating test plans supporting QA (creating test cases, automated test execution results), source code documentation, and even conducting code reviews through tools like SonarQube Cloud.

Conclusion: Bringing Discipline to AI-Powered Development

AI coding tools have made impressive strides, offering everything from code generation to testing and debugging. But in the rush to adopt them, many organizations overlook a critical factor: the need for structured, well-defined workflows. Without clear processes and measurable outcomes, even the most advanced tools can fall short—especially in enterprise environments that demand consistency, quality, and traceability. At Neurealm, we’ve taken a different approach. By combining AI-assisted development with a specification-first framework, we ensure that human judgment, engineering rigor, and automation work together across the SDLC. This model not only supports faster development but also provides the structure enterprises need to scale AI adoption effectively. As the technology matures, success will depend not just on what tools you use, but on how intentionally you use them.

From Passwords to Passkeys

World Password Day 2025, now celebrated as World Passkey Day by the FIDO Alliance (Fast IDentity Online), marked a pivotal moment. This wasn’t just a symbolic shift; with over 100 organizations pledging support in under a month, it signaled a decisive industry pivot. The message is clear: the era of passwords, with their inherent vulnerabilities, is ending. Many security professionals and users alike constantly ponder, “is my password secure?”. The evolving threat landscape unequivocally suggests they are not. Weak, reused, and easily phishable credentials remain the primary attack vector for data breaches. For any security-conscious enterprise or IAM leader seeking how to prevent data breaches effectively, passwordless authentication isn’t just a future aspiration, it’s an immediate imperative. These transformative passwordless solutions are fundamental to modern cybersecurity.

Strategic Imperatives for IAM Product Leaders

For IAM product leaders, this shift necessitates moving beyond static workflows toward intelligent, continuously evolving platforms. The powerful combination of passwordless authentication and agentic AI signifies a fundamental transition from reactive security measures to proactive identity orchestration.

Fortifying Enterprise Zero Trust Architecture

For enterprises, this convergence forms the bedrock for a robust Zero Trust security model. Every login is continuously validated, device-aware, and inherently phishing-resistant, all without impeding workforce productivity. The future of authentication is not just passwordless; it is intelligent, adaptive, and secure by design. This architectural approach is central to how to prevent data breaches at a foundational level.

Unpacking FIDO2 Authentication

Central to this evolution is FIDO2, a modern authentication standard developed by the FIDO Alliance. To understand its power, one might ask, “How does passwordless authentication work?” FIDO2 integrates WebAuthn (a web-based API for creating and using strong, attested, scoped, public key-based credentials) and CTAP2 (Client to Authenticator Protocol 2, which enables communication between a client and an external authenticator) to facilitate secure, password-free logins. This is achieved by leveraging factors such as biometrics (something you are), physical devices or tokens (something you have), or behavioral patterns (something you do). While we are familiar with what multi-factor authentication is, FIDO2 takes this concept further by eliminating the vulnerable password component entirely, relying on public-key cryptography where user credentials never leave the device, making phishing attacks virtually impossible.

Addressing Security and Simplicity with Passwordless Authentication

Passwordless authentication directly addresses several critical challenges faced by organizations today, enhancing both security and user experience. It significantly reduces the risk of credential theft, phishing, and brute-force attacks, which are common with traditional passwords. Simultaneously, it simplifies the login process for users, eliminating the need to remember complex passwords and reducing password-related friction. This leads to fewer lockouts and improved productivity, making security both stronger and simpler to manage. These innovative passwordless solutions are key to fostering a frictionless yet secure environment.

Diverse Implementations of Passwordless Authentication

Passwordless solutions are not monolithic; their implementation varies across enterprise environments, offering versatile Passwordless Authentication Solutions.

• Windows Hello, for instance, enables seamless biometric sign-ins on Windows 10 and 11, with tight integration into Microsoft Entra ID. 
• MacOS and iOS now support Platform SSO and WebAuthn through Touch ID and Face ID, offering native biometric authentication. 
• Shared environments such as hospitals or call centers, roaming authenticators, biometric smartcards, or QR-based logins provide a balance between robust security and operational convenience.

Tangible Benefits for CISOs

For Chief Information Security Officers (CISOs), the adoption of passwordless authentication translates into actionable and measurable benefits. 

• It leads to a significantly reduced attack surface by eliminating the weakest link in the security chain. 
• Organizations also experience lower helpdesk costs, as 30–50% of IT support tickets are typically password-related. 
• Furthermore, embracing passwordless authentication contributes to improved compliance with Zero Trust principles, reinforcing a security posture where every access request is rigorously validated. This strategic shift is fundamental to modern strategies on how to prevent data breaches by hardening every access point.

The Future: Agentic AI in Authentication

The future of authentication extends beyond current passwordless implementations. Enter Agentic AI—a transformative layer where AI agents dynamically adapt authentication in real-time. These agents go beyond merely enforcing access; they understand context, learn user behavior patterns, and respond autonomously to evolving risks. Imagine a system that not only identifies sign-in anomalies but also dynamically adjusts the authentication path without human intervention, escalating security measures based on real-time threat intelligence.

In the previous article, I explored how Large Language Models (LLMs) influence data engineering. From generating SQL to automating metadata, LLMs offer great promise but also pose challenges around hallucinations, governance, and validation. I concluded that LLMs serve best as intelligent copilots, enhancing rather than replacing human oversight.

In this post, I shift my focus from individual AI capabilities to end-to-end smart data pipelines – exploring how AI can fundamentally alter the architecture, behavior, and design of the modern data pipeline.

Rethinking the Pipeline: From Static to Smart

Traditional data pipelines are deterministic and pre-defined, operating on scheduled triggers and fixed logic. Smart data pipelines, on the other hand, are dynamic systems infused with AI to sense, react, and adapt based on data quality, volume, schema drift, or anomalies.

Key characteristics of smart pipelines:

• Context-aware
• Feedback-driven
• Self-optimizing
• Observability-integrated

Core Capabilities of AI-Augmented Pipelines

• Real-Time Anomaly Detection

AI models monitor data flow and metrics to detect:

• Volume spikes or drops
• Schema drift
• Outliers or unexpected distributions

Design Consideration: Integrate ML models into streaming layers (e.g., Apache Spark Structured Streaming or Azure Stream Analytics) to raise alerts or automatically pause pipelines.

• Adaptive Transformation Logic

Instead of static transformation code, AI can suggest or apply transformations based on input schema or data profile.

Example:

• If a new column appears, suggest mapping rules or imputation logic
• Adjust joins or filters based on upstream schema changes

Design Consideration: Incorporate schema profilers and LLMs into ETL tools to dynamically generate transformation options.

• Intelligent Retries and Error Handling

Smart pipelines can auto-diagnose failures and retry using modified logic or fallback options.

Use Cases:

• Retry a failed ingestion job with alternate file format parsers
• Skip corrupted records and notify downstream users

Design Consideration: Leverage metadata stores and historical error logs to guide retry policies or escalate only on failure patterns.

• Lineage Awareness and Root Cause Tracing

AI models can analyze the flow of data across jobs and detect the impact of changes upstream.

Benefits:

• Automatically trace errors to source systems
• Suggest which downstream assets are affected
• Improve time-to-resolution in production

Design Consideration: Integrate with catalog tools and metadata APIs to enable LLMs to reason over DAGs and lineage graphs.

Architectural Patterns for AI-Augmented Pipelines

• Event-Driven AI Enhancers

• • Microservices or agents listen to data events and invoke AI models (e.g., schema drift detection).
  • Outputs inform pipeline decisions (e.g., reroute, transform, notify).

• Embedded ML Models in Orchestration

• • Orchestration platforms like Airflow or Azure Data Factory run embedded ML/LLM tasks before triggering core jobs.
  • Enables pre-checks or adaptive branching.

• Feedback Loop Integration

• • Model outcomes and human interventions are logged.
  • Reinforcement or fine-tuning improves future automation accuracy.

• Metadata-Centric Execution

• • Pipelines read metadata (e.g., data quality scores, PII tags) and dynamically adjust logic or flow.

Closing Thoughts

Smart data pipelines represent a significant leap from traditional ETL and orchestration models. By embedding AI into the heart of the pipeline, organizations can achieve real-time responsiveness, self-healing capabilities, and deeper observability.

However, such designs require a cultural and architectural shift—treating AI not as an external layer but as a first-class citizen within your data ecosystem. It also demands deeper integration with metadata systems, robust observability infrastructure, and continuous model retraining pipelines.

As data volumes and complexities grow, the need for pipelines that can sense, learn, and adapt in real time becomes not just beneficial—but essential.

A strong rapport between a product manager (PM) and a UX strategist is crucial for a product’s success, especially during the design phase. This collaborative relationship helps with a shared understanding, streamlines decision-making, and ultimately leads to a more user-centric and effective product design cycle. At Neurealm, we have successfully implemented this model for our customers by anchoring the complete software design and development lifecycle using this skillset combination. As a result, we have gained valuable insights into the advantages of such a model for our product-based customers. Here are our key takeaways:

1. Shared vision and aligned goals

The PM defines the “what” and “why” – the product vision, business objectives, market needs, and key performance indicators (KPIs). The UX strategist defines the “how” – how users will interact with the product to achieve their goals – focusing on user needs, pain points, and usability.

Outcome: Effective communication and understanding of perspectives. This leads to a truly shared vision where business goals are intrinsically linked with user needs. The PM and UX strategist can co-create a product strategy that is both viable and desirable, ensuring the design phase begins with a unified purpose. Without this, the design might cater to business goals but alienate users, or vice versa.

For a telecom customer, the Neurealm team proactively managed potential issues by sharing product roadmaps early and collaborating on a plan of action with all teams involved. This allowed the customer to anticipate and mitigate future pitfalls in product development and new product feature releases.

2. Efficient and effective communication

Good rapport encourages an environment of open and honest communication. PM and UX strategist can challenge each other respectfully, brainstorm ideas freely, and provide constructive feedback without fear of damaging the relationship. Clear communication minimizes misinterpretations of requirements, user research findings, and design decisions. It also saves time and resources that would otherwise be spent on rework.

Outcome: Effective communication leads to faster decision making. When there’s trust and understanding, decisions can be made more quickly and confidently. The PM trusts the UX strategist’s expertise in user experience, and the UX strategist understands the PM’s business constraints and opportunities.

In case of another customer, regular workshops and design reviews with UX strategist and PM working in sync helped in eradicating communication blockers. Understanding the business and market background on product decisions made tasks clear. With PM and UX strategist communicating effectively, the conversion of ideas from ‘what’ is to be achieved into ‘how’ it can be achieved in a given quarter became easy with very little rework. This reduced the risks and challenges in the implementation phase of the product.

3. Holistic understanding of user needs and business value

The PM brings market insights, competitive analysis, and an understanding of the business model. The UX strategist brings deep user research (interviews, surveys, usability testing), persona development, and journey mapping.

Outcome: Strategic business opportunities getting discussed and vetted. UX strategists can then explore these through a user lens. Conversely, the UX strategist’s user insights reveal unmet needs or pain points that the PM can leverage for new product features or improvements, leading to a product that is both commercially successful and genuinely useful.

For our healthcare and telecom customers, a holistic understanding of the product was critical to their success. The comprehensive view, treating the product as a whole rather than a collection of team-specific responsibilities, ensured the published product roadmap made complete sense.

4. Proactive problem solving and risk mitigation

With good rapport, both parties are more likely to proactively identify potential design flaws, usability issues, or business-user misalignment early in the design phase. They can then work together to brainstorm solutions, weigh trade-offs, and pivot if necessary.

Outcome: This prevents costly rework later in the development cycle. The UX strategist can more effectively advocate for the user’s needs when they have the PM’s trust and support, ensuring user-centricity remains a priority throughout the design process.

When serving our customers, the UX strategist and PM also worked together to anchor the complete SDLC. Detailed workshopping with Development, QA and customer success teams helped work towards a low risk agile sprints.

5. Seamless integration of user experience into product backlog

A strong PM-UX strategist relationship allows for better prioritization of UX-related tasks and features within the product roadmap and backlog. The PM understands the value of UX investments, and the UX strategist can articulate the business impact of addressing specific user needs. They share ownership of the user experience, ensuring that UX considerations are not an afterthought but are woven into the very fabric of the product.

For our product customers, daily tasks of the PM entailed creating detailed documents of the backlog, creating tickets and talking to all stakeholders to get the expectations right. After documentation, the PM and UX strategist had regular meetings and discussions to finalize the feature design. UX strategist and PM together had a working meeting with the development team to check on the feasibility and forecast future pitfalls. Timelines were also discussed during these meets. After the feedback and changes, the finalised design was then shared and the other teams like QA and customer teams were given a walkthrough to close the loop.

6. Stronger business outcomes & ROI

1. Reduced development waste: Identifying and addressing usability issues and validating user needs early (a core tenet of UX strategy supported by PM alignment) significantly reduces costly rework later in the development cycle.
2. Increased conversion rates & revenue: A well-designed user experience, born from PM-UX strategist collaboration, can directly lead to higher conversion rates (e.g., sign-ups, purchases) and customer lifetime value.
3. Improved market differentiation: Products that offer superior user experiences often stand out in crowded markets. The PM-UX strategist partnership can drive innovation w.r.t. how users interact with the product, creating a competitive advantage.

7. Enhanced creativity and innovation

The PM and UX strategist bring different but complementary skills and diverse perspectives. When they collaborate effectively, this diversity can spark more innovative solutions. Good rapport encourages “what if” discussions, where they can explore unconventional ideas and push the boundaries of the design, leading to truly differentiating features.

8. Empowered and motivated teams:

When user needs are clearly articulated (UX) and linked to business objectives (PM), the entire team has clarity and a stronger sense of purpose. A collaborative PM-UX strategist relationship minimizes ambiguity and conflicting directives, leading to reduced friction and a more harmonious and productive work environment.

While partnering with customers from different domains, we have seen that when PM and UX strategists anchor the design cycles, the other team members become more involved for a shared goal. There is more passion for the product vision. Working passionately from ideas to execution throughout the product development lifecycle towards this shared vision for the product success is a direct result of the collaborative leadership of UX strategist and PM.

Closing thoughts

A strong partnership between a UX strategist and a Product Manager is crucial for product success. This collaboration ensures the right product is built, and it’s built well, directly boosting adoption and retention rates.

Essentially, a robust rapport between these roles transforms the design phase from simple handoffs into a dynamic partnership. This synergy helps anticipate setbacks, manage scope creep, achieve design goals, and ensures teams deliver their best work. The result is a product that’s not only technically feasible and commercially viable, but also highly desired and intuitive for users, leading to a superior product experience and greater market success.