Skip to main content
Leidos
Search
  • Global
  • Employees
  • News
  • Investors
  • Contact
  • Company
    • Our Business

      We embrace and solve some of the world's toughest challenges.

      • Civil
      • Defense
      • Health
      • Intelligence
      • Dynetics
      • Leidos Innovations Center (LInC)
      IT analysts in discussion over monitors

      Leidos completes acquisition of 1901 Group, expanding digital modernization capabilities.

      Read Press Release

    • Mission, Vision and Values

      Everything we do is built on a commitment to do the right thing for our customers, our people, and our community.

      • Mission
      • Vision
      • Values
      man sitting on bleachers holding soccer ball

      Let’s come together and talk about the changes that are needed

      The Opioid Epidemic: A Call to Action

    • Leadership

      Meet the executive team responsible for leading our forward-thinking employees toward a shared success.

      • Roger Krone
      • James Reagan
      • Paul Engola
      • Roy Stevens
      • Steve Hull
      • Jerry Howe
      • Michele Brown
      • Jim Carlini
      • James Cantor
      • Kamal Dua
      • Jim Moos
      • Gerry Fasano
      • Vicki Schmanske
      • Liz Porter
      • David King
      • Board of Directors
      Roger Krone

      Meeting the challenges of an uncertain world

      An interview with CEO Roger Krone

    • History

      Leidos has a long history of innovative problem-solving and customer service, dating all the way back to 1969 when Dr. J. Robert Beyster founded his "crazy little company".

      • Explore our History
      • Dr. J. Robert Beyster
      • A Kaleidoscope of Innovation: The Story of Leidos
      Dr. Beyster

      Almost 50 years ago, visionary scientist J. Robert Beyster, Ph.D. founded the company that evolved into Leidos.

      Read More

    • Inclusion & Diversity

      Our commitment to inclusion and diversity is reflected in the way we engage our people, our customers, and our external partnerships through our innovative programs, sponsorships, and engagement.

      • CEO Letter
      • Our Commitment to Inclusion
      • An Inclusive Workforce
      • Operation MVP
      35th BEYA conference logo

      The 2021 class of awardees at the Black Engineer of the Year Awards (BEYA) is the largest in Leidos history with 37 employees to be honored.

      Breaking Records at 35th Annual BEYA Conference

    • Responsibility & Sustainability

      We're equally committed to using our time and resources to support people, enrich communities, and protect the environment.

      • CEO Message
      • Executive Summary
      • Community
      • Environment
      • People
      • Inclusion & Diversity
      • Political Activities
      • Suppliers and Small Business Relationships
      • About our Reporting
      • GRI™ Standards
      Leidos CEO Roger Krone

      Our response to COVID-19: A message to our valued customers, partners and colleagues

      Read Message

    • Ethics & Compliance

      We have a strong history of performance that rests on our foundation of integrity.

      • File a Report
      • History of Ethics
      • Ethics Program Structure
      • Code of Conduct
      World's Most Ethical Companies 2020 logo

      Leidos named one of the World’s Most Ethical Companies for third consecutive year

      Read More

    • Rankings & Awards

      Our innovative approach and the services and solutions we deliver frequently earn Leidos recognition from our industry and the media.

      • Enterprise Awards
      • Defense Awards
      • Civil Awards
      • Health Awards
      three award icons

      A frequent honoree on the corporate awards circuit, Leidos is proud of its many industry-specific and corporate-level achievements.

      Read More

    • Partners

      The Leidos Alliance Partner Network emphasizes connections through partnership and collaboration that drive innovation, advance technology and build efficiency.

      • Corporate Strategic Partners
      • Technology Integration Partners
      • Emerging Technology Partners
      D.C. United player Wayne Rooney with soccer ball mid-game with Leidos banner in background

      Leidos partnership with MLS club D.C. United takes the field again.

      Why we're in Black-and-Red's corner

    • Subsidiaries

      Our wholly-owned and majority-owned subsidiaries attract and retain top talent motivated to deliver results for clients.

      • 1901 Group
      • Dynetics, Inc.
      • Leidos Biomedical Research, Inc.
      • Leidos Digital Solutions
      • BEONTRA
      • QTC
      • Leidos Engineering, LLC
      • Systems Made Simple
      • Varec, Inc.
      IT analysts in conversation over monitors

      Leidos completes acquisition of 1901 Group, expanding digital modernization capabilities.

      Read Press Release

    • Contract Vehicles

      Whether you’re a potential government customer or a prospective supplier, through this listing you can find our technical and professional services under pre-negotiated terms and conditions.

      • Governmentwide MACs & IDIQs
      • GSA Schedules
      • Agency Specific IDIQs
      • International
    • Global

      With more than 400 locations in 30 countries, we continue to expand our presence and strengthen our international relationships.

      • United Kingdom & Europe
      • Australia
      • Israel
      • Middle East
      • Canada
      Graphic showing buildings connected by technology

      Meeting UK border management security needs, now and long into the future.

      More On Border Management

    • Trust

      Cybersecurity and data privacy is central to what we do; protecting data, systems, and infrastructure that are critical to our employees, customers, communities, and stakeholders.

      • Commitment to Privacy & Cybersecurity
      • Cybersecurity
      • Data Privacy
      • Privacy Statement
      Citizens walking with data flowing around them

      Today, tomorrow, and every day is Data Privacy Day

      Read the Article

  • Markets
    • Aviation

      Bringing efficiency and modernization to the global aviation market.

      • Air Traffic Management
      • ANSP Professional Services
      • Security Detection & Automation
      • Intelligent Approach
      • Scenario Planning & Forecasting
      • Passenger Flow Measurement
      • FODD & Fuels Management
      • Flight Management & Briefing
      Leidos Smartlane in use at an airport

      Safeguarding travel and trade at a global scale, Leidos completes acquisition of L3Harris Technologies’ Security Detection & Automation businesses.

      Learn About Our Portfolio

    • Defense

      Science for a safer world

      • Airborne
      • Autonomy & Autonomous Solutions
      • Command & Control
      • Cyber
      • Digital Modernization
      • Operations & Logistics
      • Training
      IT analysts in discussion over monitors

      Leidos completes acquisition of 1901 Group, expanding digital modernization capabilities.

      Read Press Release

    • Energy

      Trusted expertise for energy projects and programs

      • Site Management & Operations
      • Power Delivery
      • Integrated Energy Management
      • Project Finance & Development
      • Digital Utilities
      • Manufacturing Systems
      Seattle skyline

      Seattle City Light has been turning to Leidos for their utility industry experience and engineering talent.

      Read The Case Study

    • Government

      Collaborating with government agencies to help citizens thrive

      • Cybersecurity
      • Digital Modernization
      • Health IT
      • Forms Processing
      • Operations & Logistics
      • Mission Software Systems
      IT analysts in discussion over monitors

      Leidos completes acquisition of 1901 Group, expanding digital modernization capabilities.

      Read Press Release

    • Healthcare

      Building a healthy connection between technology and people

      • Hospital Systems
      • Federal Health
      • Military Health
      • Veteran Health
      Healthcare professional using tablet

      Taking a strategic approach to experience true digital transformation in healthcare

      Read White Paper

    • Homeland

      Integrating, applying, and advancing IT to tackle the ever-advancing challenges of protecting our borders.

      • Biometrics
      • Mission Cyber and Network Defense
      • Integrated Systems
      • Digital Modernization
      • Data Analytics
      • Intelligence & Linguistic Services
      • CBRN Defense
      • Human Capital Services
      • Complex Logistics
      Leidos Smartlane in use at an airport

      Safeguarding travel and trade at a global scale, Leidos completes acquisition of L3Harris Technologies’ Security Detection & Automation businesses.

      Learn About Our Portfolio

    • Intelligence

      Rapidly transforming vast and varied data into actionable intelligence

      • Intelligence Community Engineering
      • Collection, Analysis & Reporting
      • Operational Support & Training
      Analysts discussing geospatial data

      Delivering commercial innovations to mission realities for the NGA in St. Louis

      Learn more about digital modernization

    • Science

      Groundbreaking scientific research and environmental management best practices that make headways in the defense, civilian and health markets

      • Life Sciences
      • Autonomous Systems
      • Airborne & Ground ISR
      • Command and Control
      • Electronic Warfare
      • Environmental Science
      • Modeling, Simulation & Training
      • Sensors
      • Specialty Sciences
      Illustration representing health-focused scientific research

      NCI's Frederick National Laboratory has launched three initiatives focused on SARS-CoV-2.

      Read the Article

    • Space

      IT, engineering, and science expertise to execute on the most critical of exploration missions

      • NASA
      • Defense
      • Commercial
      Astronaut planting U.S. flag on the lunar surface

      Take an interactive tour of the Artemis mission and experience our human landing system in augmented reality.

      Learn More About Lunar, our Augmented Reality Experience

    • Surface Transportation

      Engineering and research in support of next generation transportation

      • Connected & Automated Vehicles
      • Human Factors Research
      • Analysis, Modeling & Simulation
      • Transportation Safety
      • Operations
      • Asset Management
      • Accelerated & e-Construction
      • Freight
      • Research Facility Management
      Semi-truck driving on highway

      End-to-end shipment visibility with TransVM

      View Fact Sheet

  • Capabilities
    • Cyber Operations

      We use intelligent automation and AI/ML-driven analytics, combined with detection and mitigation, to protect and defend the nation's largest and most attacked mission-critical networks.

      • Security Operations Center
      • Insider Threat
      • Cyber Analytics
      • Information Assurance
      • Risk Management Framework
      • Accredited Testing & Evaluation
      • EXCITE
      Security on a mission text overlaid on blue graphic

      Outpacing adversaries through the application of IT, engineering, and science.

      More on Securing What’s Important

    • Digital Modernization

      We deliver resilient IT solutions and managed services that leverage trusted AI and full-spectrum cyber to decrease vulnerabilities in complex IT operations.

      • User Engagement
      • Digital Workplace
      • Mobility
      • Cloud
      • Application Modernization
      • Data Center Modernization
      • SecDevOps
      • Network Modernization
      IT analysts in discussion over monitors

      Leidos completes acquisition of 1901 Group, expanding digital modernization capabilities.

      Read Press Release

    • Integrated Systems

      Delivering high-performing hardware and software systems to solve challenges in an array of specializations.

      • Autonomy & Autonomous Solutions
      • Biometrics
      • Command & Control
      • Security Detection & Automation
      • Sensors
      • SE Core
      • Tactical Data Links
      • Advanced Computing
      • Computational Fluid Dynamics
      • TRAPS
      Astronaut planting U.S. flag on the lunar surface

      Take an interactive tour of the Artemis mission and experience our human landing system in augmented reality.

      Learn More About Lunar, our Augmented Reality Experience

    • Mission Operations

      We leverage our in-depth domain knowledge and mission understanding to provide cost-effective tools and enhanced processes that are secure, resilient, and support our customer's critical missions.

      • Antarctic Support Contract
      • ISS Cargo Mission Contract
      • Logistic Commodities & Services Transformation Programme
      • Strategic National Stockpile
      • Defense Logistics Support
      Ice sheet in Antarctica

      Each year, Leidos moves more than six million pounds of cargo and coordinates 100 flight missions to and from Antarctica.

      About Our Role

    • Mission Software Systems

      Through disciplined processes, common tools, reusable frameworks, automation, collaboration, and domain expertise, our mission software systems are designed to deliver secure, mission quality software.

      • LEAF
      • OneSAF
      • Command & Control
      • Data Analytics
      • Healthcare Analytics
      Software on a mission text on blue background

      Mission quality software. Silicon Valley speed.

      More on Building Better Software

    • Enabling Technologies

      Our enabling technologies are the backbone of our core capabilities, supporting our customer's important work from the front lines.

      • Secure, Rapid Software
      • Trusted AI/ML
      • Full-Spectrum Cyber
      • Rapid Prototyping & Manufacturing
      Military radar simulation with green display, showing a glowing grid with coordinates and positioning numbers

      How we are deploying AI to solve the world’s toughest problems.

      Artificial Intelligence in Action

    • Technology Certifications

      Technology certifications with world-class providers.

      • AWS
      • Cisco
      • Microsoft
      • Scaled Agile
      • ServiceNow
      group of coworkers looking at computer monitor

      Helping our partners save money and improve performance.

      How the Cloud Saved Some Coin

  • Insights
    • Insights

      Expertise, industry leadership, unique perspectives, and more - directly from our employees and stakeholders.

      • Latest Insights
      • On A Mission
      • MindSET Podcast
      • Press Releases
      • Heroes of Leidos
      • Our Experts
      • Careers & Culture
      • Community
      • Articles
      • Case Studies
      • Infographics
      • Q&As
      Astronaut planting U.S. flag on the lunar surface

      Take an interactive tour of the Artemis mission and experience our human landing system in augmented reality.

      Learn More About Lunar, our Augmented Reality Experience

  • Careers
    • Latest Opportunities

      Keep up to date on the latest career opportunities at Leidos.

      • Military Veteran Program
      • External Referral Program
      • Intern & New Graduate Jobs
      • Careers with Leidos in the United Kingdom
      • Careers with Leidos in Australia
      • Opportunities for Leidos employees
      five coworkers sitting around a laptop

      Your most important work is still ahead. Explore opportunities at Leidos.

      View our Latest Opportunities

    • Life at Leidos

      A career at Leidos offers meaningful and engaging work, a collaborative culture, support for your career goals, while nurturing a healthy work-life balance.

      • Our Values
      • Our Culture
      • An Inclusive Workforce
      • Build a Career
      five coworkers sitting around a laptop

      Your most important work is still ahead. Explore opportunities at Leidos.

      View our Latest Opportunities

    • Pay & Benefits

      Pay and benefits are fundamental to any career decision, which is why our compensation packages reflect the importance of the work we do for our customers.

      • Compensation
      • Health & Wellness
      • Income Protection
      • Leave
      • Retirement
      • Family Benefits
      five coworkers sitting around a laptop

      Your most important work is still ahead. Explore opportunities at Leidos.

      View our Latest Opportunities

    • Frequently Asked Questions

      Find answers to the most commonly asked questions related to securing a role at Leidos.

      • Searching for a Role
      • Applying for a Job
      • Determining Application Status
      • System Access
      • Job Eligibility Requirements
      • Receiving Job Notifications
      • Our Talent Community
      • Career Insights
      five coworkers sitting around a laptop

      Your most important work is still ahead. Explore opportunities at Leidos.

      View our Latest Opportunities

  • Search Search
  • Company
  • Markets
  • Capabilities
  • Insights
  • Careers
Back to top

Breadcrumb

  1. Home
  2. Insights

Physical or virtualized data marts – what’s right for your organization?

Engineer at a server rack

Data marts and data warehouses have served the business users requirements for a long time. But in today’s agile environment, organizations cannot wait for weeks or even months for the data to be available if the companies follow a Software Development Life Cycle (SDLC) to create data marts.

Why were physical Data Marts created?

An organization typically created a physical data mart to meet one or more of the following requirements:

  • Blending of data: Data coming from varied sources which may be a mix of relational and non-relational data. Most of the Business Intelligence (BI) tools allow blending of data, but the performance of the tools suffers especially if the volume of data is large, and user experience for drill downs and aggregations is also not as per expectations.
  • Data transformations and aggregations: Transformations to raw data and aggregations to a higher granular level are main use cases for using Extract Transform and Load (ETL) tools to create data marts. Although basic transformations and aggregations can be done in BI tools but aggregating data on fly could be costly, and if the transformations are complex, we will have to use ETL tools.
  • Single source of truth: For financial companies and other companies where the data accuracy/consistency and reliability are of utmost importance, having a single source of truth is essential from a reporting perspective, and data marts can serve as a container for all financial and audit reporting.
  • Easy and direct integration to BI or visualization tools: Data marts have been generally created in relational databases which provide a direct (open database connectivity /Java database connectivity (ODBC/JDBC) source to the BI tools and don’t have integration challenges.

Are physical Data Marts the only solution?

Organizations may no longer need to build extensive and expensive physical data marts. With wide variety of distributed computing and virtualization tools on-premise and in the cloud, we can now deliver data to the business in an agile manner. Data Virtualization fulfills all the above features of physical data marts like blending of data, Data transformations and aggregations, single source of truth and integration to BI tools but also provide other advantages of:

  • Agility: Saves time in creation and delivery of data structures.  
  • Cost advantages: Saves cost in creation, upgrades and maintenance.
  • Extensibility: The existing marts or lakes can be quickly extended with new sources or one-time data sources.
  • Real Time and Streaming Analytics: For real-time analytics, data virtualization can provide near real-time capabilities. 
  • Performance: As data virtualization tools store data in server memory, it provides enhanced query performance for BI /Analytics tools.

Data Virtualization and Big Data Lake Architecture

Virtual data marts are providing a lot of value in following patterns of big data architecture:

  • Big Data Analytics: The newer data stores like NoSQL, document stores, and graph databases do not typically provide a standard JDBC/ODBC-based SQL interface which make them difficult to use for BI and reporting tools and for data analysts who are familiar with SQL queries. The Data Virtualization Platform provides an abstraction and federation layer that hides the complexities of the Big Data stores and makes it easy to integrate data from these stores with other data within the enterprise.
  • Hybrid Virtual Data Warehouses or Marts: More and more companies are moving data to Data lakes, but there is a still a substantial amount of data in traditional relation databases and appliances. The DV Platform can sit on top of both data stores and federates queries across both so that the hybrid data warehouse appears as a single data store to users. As shown below in the diagram we can leverage Virtualization to combine data from operational databases, Enterprise Data warehouse, flat files, and big data systems and provide a common Analysis layer.
Data lake illustration
A hybrid logical data lake illustration

Leveraging Data Virtualization Tools

Data Virtualization (DV) tools enable independently designed and deployed data structures to be leveraged together as a single source, in real time, and with limited (or often no) data movement. According to noted data virtualization expert Rick van der Lans, author of Data Virtualization for Business Intelligence Systems, “data virtualization is the technology that offers data consumers a unified, abstracted, and encapsulated view for querying and manipulating data stored in a heterogeneous set of data stores.”

There are various types of virtualization tools and technologies which can be categorized broadly as:

  • Traditional Virtualization Tools: Traditional virtualization tools like Composite / Denodo can provide a common business (canonical) layer from multiple sources of data. The virtualization layer provides pass-thru capabilities to the BI tools using direct ODBC / JDBC connectivity. But high-volume data may present challenges based on cluster sizing and server RAM limitations. Simple transformations can be done in these tools. This approach works fine with small to medium data sets with simple transformations.
  • Big Data Architecture Based Virtualization Tools: With distributed computing, new frameworks and tools are now in the market, which provide big data based (mostly Spark based) processing engines. These engines are constraint to a single server CPU/memory for processing a single job, as they can distribute a single job to multiple servers. Installations of these tools can utilize existing Spark or Hadoop cluster. These tools can fulfill high-performance requirements with large volumes without creating physical data structures. The data can be kept virtualized or written back to databases or file systems for historical and audit reporting purposes. Presto / Spark can be used for providing virtualized / real-time processing, and some of these frameworks are being used by companies like Airbnb, Netflix, and Facebook. A new generation of visualization tools like druid can be used for real-time visualizations. A new open source framework called Apache Arrow has been developed, and tools like Dremio look very promising for virtualization and big data processing. Big data based MOLAP tools like Kyvos, Atscale, or Kylin can also be used for a virtual layer but need prep work to get data in cubes.
  • Data Preparation Tools: Data prep tools are a new set of lightweight, agile ETL tools gaining popularity for creating virtualized data marts and logical data lakes. These tools can create a physical or virtual data mart or Data Lake, but development and execution are more agile than the traditional ETL tools. Alteryx and Paxata are some of the prominent tools.

Physical vs. Virtual – What’s right for your organization?

Companies looking for more agile options for providing blended high-volume data along with simple to complex transformations and persistence options should look at the above new frameworks and tools for BI / ETL and data preparation. Virtualization also becomes an important tool for any real-time reporting and analytics requirements.

Some of the key features of the data virtualization we should be considering during the selection process are:

  • Allow all types of data sources to be accessed, including Hadoop files, SQL databases, flat files, spreadsheets, packaged applications, social media networks, and messaging devices.
  • Complex data preparation, data cleaning and data manipulation functions should be built into the tool. Reusable user-defined functions (UDF’s) should be available. 
  • Detailed lineage and impact analysis capabilities that show from which data sources results are derived and which transformations have been applied to the data.
  • A searchable data catalog that contains definitions, descriptions, tags, and classifications of all the available data. This feature is indispensable for more investigative and experimental users, such as data scientists.
  • Advanced query optimization techniques for federated queries across a homogeneous set of data sources. Joins combining data stored in Hadoop with data stored in a SQL database should be executed as efficiently as possible.
  • Advanced query pushdown and parallel processing capabilities to handle complex queries on real big data. It’s important that query processing is executed as close to where data is stored as possible, regardless of the data storage technology used.
  • Centralized data security features for specifying on a granular level, i.e., which user is allowed to do what, with which data.
  • Having different access interfaces like SQL, REST/JSON, XQuery, MDX, SOAP/XML, or JMS.
  • Distributed processing and memory architecture for high volume processing and high volume in-memory storage.
  • Managed services or auto-scaling server architecture based on load and concurrency.

It could be challenging to look at the plethora of tools in the market today, but starting with some quick proof of concepts (POC’s) is always the best option to understand their capabilities and what best fits in your organization. To discover how Leidos can help your organization find the best solution, contact us today to speak to one of our experts.

Author
Rakesh Gupta
Rakesh Gupta Solutions Architect, Business COTS Solutions

Rakesh Gupta is the Solutions Architect Business COTS Solutions for Enterprise & Cyber Solutions within Leidos’ Defense and Intelligence group. In his role, Rakesh designs business intelligence, data strategy, and big data solutions for in premise and on cloud for large-scale programs across the Leidos portfolio including major customers such as IGC, and the Air Force. He has 25 years of experience with business intelligence, analytics, and data solutions and has worked with fortune 500 companies in various verticals including finance, manufacturing, and retail implementing these solutions.

Share
  • Share on Facebook
  • Share on Twitter
  • Share on LinkedIn
  • Share via Mail
  • Print Insight

Posted

November 5, 2018
Author
Rakesh Gupta

Tags

Data Science and Engineering
Digital Modernization
Integrated Systems
Mission Operations
Article
Defense
Government
Health
Homeland
Intelligence
Science
Surface Transportation

Sign up for our newsletters

Stay up to date on our expertise, industry leadership, unique perspectives, and more by subscribing to our newsletters.

  • Company
    • Our Business
    • Mission, Vision, and Values
    • Leadership
    • History
    • Inclusion & Diversity
    • Responsibility & Sustainability
    • Ethics & Compliance
    • Rankings & Awards
    • Partners
    • Subsidiaries
    • Contract Vehicles
    • Global
    • Trust
  • Markets
    • Aviation
    • Defense
    • Energy
    • Government
    • Healthcare
    • Homeland
    • Intelligence
    • Science
    • Space
    • Surface Transportation
  • Capabilities
    • Cyber Operations
    • Digital Modernization
    • Integrated Systems
    • Mission Operations
    • Mission Software Systems
    • Enabling Technologies
    • Technology Certifications
  • Insights
    • News
  • Careers
    • Life at Leidos
    • Pay & Benefits
    • Alumni
    • FAQs

Want to know more?

Contact us about product information and pricing, customer feedback, stockholder services, or just to voice a concern.

Get started

  • © 2021 Leidos

  • Trust
  • Privacy Statement
  • Ad Choices
  • Do Not Sell My Personal Information
  • © 2021 Leidos
  • Australian Privacy Statement
  • Australian Whistleblower Policy
  • Ad Choices
  • Do Not Sell My Personal Information

Utility Footer Navigation

  • Products
  • Customers
  • Suppliers
  • Investors
  • Employees
  • Accessibility
  • Twitter
  • Facebook
  • Instagram
  • Linkedin
  • YouTube