Operations | Monitoring | ITSM | DevOps | Cloud

Over the past two months, VirtualMetric DataStream delivered a substantial update cycle focused on resilience, productivity, and platform extensibility. This release strengthens the core architecture, makes pipeline development and troubleshooting significantly easier, and expands integration coverage across schemas, SIEMs, and cloud platforms. Let’s take a closer look.

Modern security operations face a growing architectural challenge: collect telemetry from everywhere, process it in real time, and route it to multiple platforms while maintaining data sovereignty, avoiding agent sprawl, and keeping costs under control. Single-model collection strategies force security teams to make compromises. Agent-only models create operational overhead and maintenance risk. Agentless-only approaches simplify operations but limit depth and flexibility.

Security data collection has become a first-class architectural concern for modern SOCs. Once collection is treated as a dedicated layer, separate from analytics and detection, the next question becomes practical: how should telemetry be collected in a way that aligns with this architecture? In the previous article, we examined why this shift occurred. Here, we focus on how different collection models (agent-based, agentless, and hybrid) fit into modern security data collection architectures.

Security teams are increasingly turning to Amazon Security Lake to consolidate security telemetry across cloud, network, and on-prem environments. Security Lake provides a unified, OCSF-based data repository that powers analytics, threat hunting, and machine learning across AWS services and third-party tools. But to take advantage of Security Lake’s capabilities, organizations must deliver clean, normalized, OCSF-compliant data, and this is where challenges arise.

Security operations teams depend on telemetry from everywhere: endpoints, identity systems, cloud platforms, networks, applications, and security tools. This data underpins detection, investigation, compliance, and response.

Microsoft Sentinel adoption often introduces unexpected complexity. While the platform delivers powerful SIEM and XDR capabilities, organizations frequently struggle with manual DCR configuration, inconsistent data quality, rising ingestion costs, and security risks associated with credential-based integrations. VirtualMetric DataStream is now available in the Microsoft Sentinel Content Hub, reducing the effort required to deploy normalized and cost-optimized data ingestion.

Elasticsearch has long been the backbone of security analytics for organizations that need fast search, flexible dashboards, and scalable visibility across massive datasets. It powers everything from threat hunting to compliance reporting and real-time investigation. But anyone who has operated Elasticsearch at scale also knows a quiet truth: Elasticsearch is only as strong as the data you feed it. And getting clean, consistent, usable telemetry into Elastic is often the hardest part.

This month’s DataStream update brings meaningful improvements across pipeline management, MSSP workflows, and endpoint visibility. We’ve focused on giving security teams more control over how data moves through their environment, expanding coverage for both Windows and Linux, and strengthening governance for multi-tenant deployments. Let’s walk through what’s new.

As security data volumes grow, so do the costs of processing and storing them. Microsoft Sentinel and other SIEM platforms charge based on data ingestion, which makes every decision about normalization rules critical and every duplicate log a direct expense. Enterprise-scale security data pipelines face a persistent problem: data duplication across normalization tiers. As logs move through multiple transformation stages, it’s often impossible to know in advance which version will succeed.