Blah, blah.

Ingestion

  1. Master Service
    1. Gets a list of Tasks to execute
    2. Activates all Agents that need to execute Tasks
  2. Control given to Agent
    1. Loads data from Source System(s) and extract it to Flat Files
    2. Updates information about source system Attributes (for relational data sets)
    3. Uploads extracted files to Azure Blob storage
    4. Informs Master Service of finished tasks
  3. Control given to Master Service
    1. Creates External Tables for finished tasks
    2. Creates Materialized Tables for all External Tables
    3. Updates Metadata columns of Materialized Tables
    4. Compares Materialized Tables with their corresponding Destination Tables and creates Change Table
    5. Apply changes to Destination Table using Load Pattern
    6. Save Change Table to Blob

Integration

  1. Master Service
    1. Gets a list of Tasks to execute
    2. Executes T-SQL statements (queries, stored procedures) on Destination database (Data Warehouse, for example)

Last edited Oct 31, 2016 at 10:42 PM by knyazs, version 4