Entity Wizard overview

The Entity Wizard in the NCC portal streamlines the process of creating and configuring Landing Zone entities, Bronze entities, and Silver entities for your data platform solution.

What is an entity?
An entity is a logical grouping of metadata that enables data extraction from a single connection or data source. In this context, entities include Landing Zone, Bronze, and Silver entities.

Entity Wizard workflow

The wizard guides you through four main steps:

1. Landing Zone
2. Bronze
3. Silver
4. Review and create

---

Step 1: Configure a Landing Zone entity

A Landing Zone entity is the initial stage for ingesting and storing raw data in your data platform. It provides a secure, scalable staging area for large volumes of unprocessed data, preserving the original format for downstream processing.

Landing Zone entities validate incoming data and resource types, ensuring data integrity before moving to the Bronze, Silver, or Gold layers. For example, if your source system contains 10 tables, create 10 Landing Zone entities—one for each table.

IMPORTANT
If a Landing Zone entity includes columns that require encryption, anonymization, or pseudonymization, the data is always stored as a Parquet file, regardless of the original format.

Add a Landing Zone entity

To configure a Landing Zone entity, provide the following details:

Source configuration

• Pipeline: Obsolete, can be ignored.
• Data source: Select the appropriate data source.
• Source schema: Enter the schema name where the data resides.
• Source name: Enter the table or file name to extract.
• Incremental: Choose whether to extract data incrementally.
• If enabled, specify:
  • Incremental column: The column used to filter and extract only new or updated records since the last extraction. If no previous extraction exists, a full load is performed.
  • Incremental partition: Optionally, specify a partition column to improve extraction performance.

IMPORTANT
Only incremental extraction using a date or datetime column is currently supported.
Learn more about incremental extraction.

• Has encrypted columns: Indicate if the data contains sensitive columns.

• If enabled, specify:

  • Encrypted columns: Enter the column name(s) and select the Encrypt option.
    Learn how to add encryption.

• Entity value: Use Entity value to add key-value pairs for extraction. These are often optional, but required for certain scenarios, such as:

• When extracting via a Notebook, provide CustomParametersJSON and NotebookName.

NOTE
To see the complete list of entity values applicable for the landing zone entities click here.

Target configuration

• Lakehouse: Select the lakehouse where your data will be stored.
• File path: Automatically generated path for storing your data.
• File name: Automatically generated name for the extracted data file(s).
• File type: Select the original data format. For tables, select Parquet.

IMPORTANT
If a Landing Zone entity includes columns that require encryption, anonymization, or pseudonymization, the data is always stored as a Parquet file, regardless of the original format.
Enabling this option may increase processing time.

---

Incremental loading

Incremental loading extracts only new or updated records since the last pipeline run, improving efficiency.

• Requirement: An incremental column (for example, DateModified) in your source.

NOTE
Deletions are not detected with incremental loads. To capture deletions, schedule periodic full loads.
For more information, see Loadgroups.

IMPORTANT
Only incremental extraction using a date or datetime column is currently supported.

---

Encryption

Enable encryption for specific columns in your database to protect sensitive data. Encryption safeguards confidential information against unauthorized access and is recommended for handling sensitive fields such as personal identifiers, financial data, or authentication credentials.
Encryption is supported for all sources with many integrated connection types.

Enable encryption

To encrypt your data, follow these steps:

1. Follow these steps when using encryption for the first time

2. Initialize Data Privacy Encryption in Fabric.

3. Enable Encryption

4. Select the Encrypted option for each column to encrypt.

5. Specify Columns

6. Enter the exact (case-sensitive) column names to encrypt.

IMPORTANT
When encryption is applied, the column is replaced by another column named {columnname}_encrypted.
These columns will always be of datatype String.
For more details, see Data Privacy.

1. Configure Key Name
2. In your entity configuration, add:
   • KeyName: Name of your encryption key

Example:

1. Foreign keys

   • When encrypting your data you can no longer use that column as a foreign key. In this case you want to hash your data. To not only encrypt your data but also hash, you can specify these columns in the entity value. Add foreignkeys to the entity values and specify the columns with the , as the seperator. !IMPORTANT: Know that when hashing your data, the encryption is less secure.
     When the hashing proces is done, there will be another collumn named {columnname}_hashed

   Example of the entity values:

NOTE
To enable encryption for your custom notebook, click here for the steps.

---

Step 2: Configure a Bronze Layer entity

The Bronze Layer allows you to create multiple entities from a single Landing Zone entity. This is useful for denormalized sources or nested data structures. For more information, see Bronze Entity Values.

Example:
A JSON file with invoices may contain an array of invoice lines. You can split this into "Invoice Header" and "Invoice Line" Bronze entities.

NOTE
To add multiple Bronze Layer entities based on the same Landing Zone entity, follow the Entity Wizard for one Bronze and Silver entity, then add additional Bronze and Silver entities through the Bronze Entity page and Silver entity page.

Add a Bronze Layer entity

To configure a Bronze Layer entity, provide the following details:

Source configuration

• Pipeline: Obsolete, can be ignored.
• File path: Automatically filled when creating a Landing Zone entity and cannot be altered in this stage.
• File name: Automatically filled when creating a Landing Zone entity and cannot be altered in this stage.
• Entity value: Similar to the Landing Zone entity, entity values differ per type of source. For example, for SQL sources the entity values are by default empty, but for an Excel file multiple Entity values are required to parse the data correctly.

NOTE
To see the complete list of entity values applicable for the Bronze entities click here.

• Column mappings: Use column mappings to identify columns to parse, rename if needed, and cast to the desired data type. Column mappings consist of three fields: SourceColumn, TargetColumn, and Data type.
• SourceColumn: Name of the column in the source
• TargetColumn: Name of the column parsed in Fabric
• Data type: Data type of the column

IMPORTANT
For nested columns, you can select the inner column if needed.

Example data:

{
  "Name": "John",
  "Age": 32,
  "Description": {"Hobby": "Travel"}
}

NCC Column mappings configuration:

SourceColumn	TargetColumn	Data type
Name	Name	String
Age	Age	Integer
Description.Hobby	Description_Hobby	String

Target configuration

• Lakehouse: Select the lakehouse where your data will be stored.
• Schema: Name of the schema where the Bronze table will be stored in the lakehouse.
• Name: Name of the table where the data will be stored.
• Primary keys: Add the primary keys of the table. A primary key is a column (or multiple columns) that uniquely identifies each row in a table.

NOTE
When column mapping is applied, use the TargetColumn instead of the SourceColumn.

---

Step 3: Configure a Silver Layer entity

The Silver Layer stores data in SCD Type 2 format, enabling historical tracking of changes.

Add a Silver Layer entity

To configure a Silver Layer entity, provide the following details:

Source configuration

• Pipeline: Obsolete, can be ignored.
• Schema: Automatically filled when creating a Bronze entity and cannot be altered in this stage.
• Name: Automatically filled when creating a Bronze entity and cannot be altered in this stage.
• Entity value: Use Entity value to add key-value pairs for extraction. These are often optional, but required for certain scenarios. To see the complete list of entity values applicable for the Silver entities click here.

Target configuration

• Lakehouse: Select the lakehouse where your data will be stored.
• Schema: Name of the schema where the Silver table will be stored in the lakehouse.
• Name: Name of the table where the data will be stored.
• Columns to exclude: Exclude specific columns from being loaded from Bronze to Silver.
• Columns to exclude from history: Exclude columns from historical tracking (for example, frequently changing comments or system-generated timestamps).

---

Step 4: Review and create entities

After configuring your entities, review all settings in the summary view. Once confirmed, create the entities in NCC.

---

Entity values reference

Landing Zone

Name	Description	Used for	Required	Default
timeout_total_in_seconds	Timeout notebook of entity in seconds	Notebook Sources	No	43200
timeout_per_cell_in_seconds	Timeout notebook cell of entity in seconds	Notebook Sources	No	1800
CustomParametersJSON	Parameter containing JSON styled key-value pairs	Notebook Sources	No
NotebookName	Name of the Notebook used to extract the data	Notebook Sources	Yes, when source is notebook
Database	Name of Database connected to	OneLake Sources	Yes, when source is OneLake
SourceLakehouseGuid	GUID of Lakehouse to connect to	OneLake Sources	Yes, when source is OneLake
SourceWorkspaceGuid	GUID of workspace where the Lakehouse is located	OneLake Sources	Yes, when source is OneLake
KeyName	The key name corresponding to the secret containing the key	Sensitive Sources	Yes, when encryption is enabled
PipelineName	Name of the pipeline in your ADF resource to run	ADF Sources	Yes, when source is ADF	PL_CUST_XXXX
CustomParametersJSON	Parameter containing JSON styled key-value pairs	ADF Sources	No	{"key1": "xxxx", "key2": "xxxx"}
relative_url	Relative URL to start the API call	REST	Yes, when source is REST
pagination_absolute_url	Pagination URL used for pagination	REST - ANONYMOUS_PAGINATION	No

IMPORTANT
When you enable Encryption for sensitive sources, make sure you have configured the variable data_privacy_key_vault on Tenant, Domain, or Environmental level.

Bronze Zone

All data source types

Name	Description	Used for	Required	Default
timeout_total_in_seconds	Timeout notebook of entity in seconds	All Sources	No	43200
timeout_per_cell_in_seconds	Timeout notebook cell of entity in seconds	All Sources	No	1800
valid_dq_deduplication_modes	Way to deduplicate data if primary keys are not sufficient	All Sources	No	none, other valid values: row, key

TIP
For more information about the Data Quality options and examples, go to Data-Quality.

CSV data source types

Name	Description	Used for	Required	Default
CompressionType	Compression type of the CSV	CSV Source	Yes, if CSV	none, valid: gzip, bzip2, lz4, snappy, deflate
ColumnDelimiter	The column delimiter	CSV Source	Yes, if CSV	;
RowDelimiter	The row delimiter	CSV Source	Yes, if CSV	\r\n
EscapeCharacter	The escape character	CSV Source	Yes, if CSV	\
Encoding	The encoding of the data	CSV Source	Yes, if CSV	UTF-8
FirstRowIsHeader	Determines if the first row is the header	CSV Source	Yes, if CSV	1 (0 = False, 1 = True)

TIP
If FirstRowIsHeader is False then the columns will be named _c0 ... _c99

JSON data source types

Name	Description	Used for	Required	Default
Collection	The name of the collection to extract data from	JSON Source	No
DateFormat	Used to change the default ISO-8601 format	JSON Source	No
multiline	Determines if the JSON contains multiline data	JSON Source	No	false

TIP
For more information about the Entity values or examples, go to Collection and DateFormat.

Excel data source types

Name	Description	Used for	Required	Default
SheetName	Sheet name where the data is located	Excel Source	Yes, if Excel
FirstRowIsHeader	The row where the header is located	Excel Source	Yes, if Excel	1
ColumnRange	The range of columns to extract	Excel Source	Yes, if Excel	A:E
RowsRange	Rows to extract	Excel Source	Yes, if Excel	ALL
NaValues	Replace string-formatted nulls with Null	Excel Source	No	NONE
Thousands	The character that separates the thousands	Excel Source	No
Decimal	Decimal delimiter	Excel Source	No	,
Comment	Rows containing comments to be excluded	Excel Source	No
SkipRows	Rows to be skipped	Excel Source	No

Silver Zone

Name	Description	Used for	Required	Default
timeout_total_in_seconds	Timeout notebook of entity in seconds	All Sources	No	43200
timeout_per_cell_in_seconds	Timeout notebook cell of entity in seconds	All Sources	No	1800
DQ_CleansingOptions	Parameter containing JSON styled key-value pairs used for data quality options	All Sources	No

TIP
For more information about the DQ_CleansingOptions or examples, go to DQ_CleansingOptions.