Hi all, I am pleased to announce that you can now download the SQL Server VNext CTP 1.0 to try out the SSIS Scale out feature and Dynamics AX/CRM Online OData Connector.

https://msdn.microsoft.com/en-us/library/mt788653.aspx

I would like to hear from you directly if you have any feedback on the feature, especially about the Scale Out as this is one of our key enterprise investment for SSIS VNext. Please email me directly at jwong@microsoft.com with your feedbacks. Enjoy!

• Support Scale Out of SSIS
  The Scale Out feature makes it much easier to run SSIS on multiple machines.
  After installing the Scale Out Master and Workers, the package can be distributed to execute on different Workers automatically. If the execution is terminated unexpectedly, the execution is retried automatically. Also, all the executions and Workers can be centrally managed using the Master.
  For more information, see Integration Services Scale Out

Note: In future CTP, we will work on the manageability and performance improvement for the scale out feature.

• Support for Microsoft Dynamics Online Resources
  The OData Source and OData Connection Manager now support connecting to the OData feeds of Microsoft Dynamics AX Online and Microsoft Dynamics CRM Online. (additional Authentication Type to support Dynamics AX/CRM on OData Connector Manager)

thanks

Jimmy

Hi All,

We are pleased to announce that an updated version of SQL Server Integration Services Feature Pack for Azure is now available for download. This release mainly has following improvements:

1. Support for Azure Data Lake Store
2. Support for Azure SQL Data Warehouse

Here are the download links for the supported versions:

SSIS 2012: https://www.microsoft.com/en-us/download/details.aspx?id=47367

SSIS 2014: https://www.microsoft.com/en-us/download/details.aspx?id=47366

SSIS 2016: https://www.microsoft.com/en-us/download/details.aspx?id=49492

Azure Data Lake Store Components

1.In order to support Azure Data Lake Store (ADLS), SSIS add below two components:

• Azure Data Lake Store Source:
  • User can use ADLS Source component to read data from ADLS.
  • Support Text and Avro file format.
• Azure Data Lake Store Destination:
  • User can use ADLS Destination component to write data into ADLS.
  • Support Text, Avro and Orc file format.
  • In order to use Orc format, user need to install JRE

2. ADLS components support two authentication options:

• Azure AD User Identity
  • If the Azure Data Lake Store AAD user or the AAD tenant administrator didn’t consent “SQL Server Integration Service(Azure Data Lake)” to access their Azure Data Lake Store data before, then either AAD user or AAD tenant administrator need consent SSIS application to access Azure Data Lake Store data. For more information about this consent experience, see Integrating applications with Azure Active Directory.
  • Multi-factor authentication and Microsoft account is NOT supported. Consider to use “Azure AD Service Identity” option if your user account need multi-factor authentication or your user account is a Microsoft account.
• Azure AD Service Identity
  • In order to use this option, you need to create an AAD application and service principal that can access Azure Data Lake resources. Then you need assign this AAD application corresponding permissions to access your Azure Data Lake resources. For more information about this authentication option, see Use portal to create Active Directory application and service principal that can access resources.

3. The ADLS source editor dialog is as below:

For more information about how to use Azure Data Lake Store components, see Azure Data Lake Store Components.

Azure SQL Data Warehouse

There are multiple approaches to load local data to Azure SQL Data Warehouse (Azure SQL DW) in SSIS. The blog post Azure SQL Data Warehouse Loading Patterns and Strategies gives a fine description and comparison of different approaches. A key point made in the post is that the recommended and most efficient approach that fully exploits the massively parallel processing power of Azure SQL DW is by using PolyBase. That is, first load data to Azure Blob Storage, and then to Azure SQL DW from there using PolyBase. The second step is done by executing a T-SQL sequence on Azure SQL DW.

While conceptually straightforward, it’s not an easy job to implement this approach in SSIS before this release. You have to use an Azure Blob Upload Task, followed by an Execute SQL Task, and possibly followed by yet another task to clean-up the temporary files uploaded to Azure Blob Storage. You also have to put together the complicated T-SQL sequence yourself.

To address this issue, this new release introduces a new control flow task Azure SQL DW Upload Task to provide a one-stop solution to Azure SQL DW data uploading. It automates the complicated process with an integrated, easy-to-manage interface.

On the General page, you configure basic properties about source data, Azure Blob Storage, and Azure SQL DW. Either a new table name or an existing one is specified for the TableName property, making a create or insert scenario.

The Mappings page appears differently for create and insert scenarios. In a create scenario, configure which source columns are mapped and their corresponding names in the to-be-created destination table. In an insert scenario, configure the mapping relationships between source and destination columns.

On the Columns page, configure data type properties for each source column.

The T-SQL page shows the T-SQL sequence for loading data from Azure Blob Storage to Azure SQL DW using PolyBase. It will be automatically generated from configurations made on the other pages. Still, nothing is preventing you from manually editing the T-SQL to meet your particular needs by clicking the Edit button.

For more information about how to use Azure SQL DW Upload Task, see Azure SQL DW Upload Task.

Hi all,

In the SQL PASS Summit of 2016, I presented a session about “Running SSIS on Azure VM – Do more with less money” and it was well received based on the session feedback. Therefore, I am extracting some key content from my SQL PASS presentation and sharing it on this blog so that more SSIS users can benefit from it.

Why running SSIS on Azure VM?

• Pay per use with support of BYOL
• Elasticity with cost options (1 Core 2GB 1TB -> 32 Core 512GB 64TB)
• Extremely Secure at physical/infrastructure/SQL level
• Reliable storage with sync local copy and async remote copy
• Fast storage that uses Solid-State Drives (SSD)
• High Availability SLA 99.95%, 99.99% with always-on
• Easy deployment with simply configuration
• Easy way to “Lift and Shift” your SSIS execution to the cloud
• Save IT costs by taking advantage of Automated Backup and Patching
• Lab / office being restructured or demolished

How to create new Azure VM to run SSIS?

There are 3 options you can consider to run SSIS on Azure VM as IaaS

Option 1: Create SQL VM with per-minute licensing

• Flexible option for non-EA customers or workload running for short time.
• Supports SQL 2016 (Enterprise, Standard, Dev) SQL 2014 SP1(Enterprise, Standard), SQL 2012 SP3 (Enterprise, Standard)

Option 2: Create SQL VM with an existing license (BYOL)

• For Enterprise Agreement (EA) customers
• Images don’t charge SQL Server license, just the cost for running Windows VM
• One free passive secondary replica license per Availability Group deployment
• Supports SQL 2016 (Enterprise, Standard) SQL 2014 SP1(Enterprise, Standard), SQL 2012 SP3 (Enterprise, Standard)

Option 3: Manual installation on Azure VM

• No automated SQL Server configuration experience (e.g. Automated Backup, Patching)
• No AlwaysOn Availability Group Portal template (for HA deployment on BYOL images)

How to migrate your SSIS packages / catalog to Azure VM?

There are 3 options you can consider to migrate your SSIS packages / catalog to Azure VM

Option 1: Copy over the package files via copy and paste and run deployment again. Use copy and paste functionality on remote desktop, or use tools like CloudBerry

• Simplest if you have only few SSIS projects to migrate (recommended)

Option 2: Use the Microsoft-provided power shell script to do a data level migration

• Best If you have large amount of projects to migrate

Option 3: Leverage Always-On deployment on-premises, use the Add Azure Replica Wizard to create replica in Azure, then add SSISDB to an AlwaysOn group using the Create AlwaysOn group” Wizard and launch “Enable SSIS to support alwaysOn Wizard” . Let it failover and have the new packages run against the catalog on the VM database instance

• Most complicated approach but this allows you to minimize your downtime

How to access on-premises data sources from Azure VM?

There are 3 options you can consider to access on-premises data sources

Option 1: VNET

• Free of charge up to 50 Virtual Networks across all regions for each subscription
• Public IP and Reserved IP Addresses used on services inside a Virtual Network is charged.
• Network appliances such as VPN gateway and Application gateway that are run inside a Virtual Network are also charged
• COST LEVEL: Medium

Option 2: Express Route

• Fast, reliable and private connection between Azure and on your premises
• Suitable for scenarios like periodic data migration, replication for business continuity, disaster recovery, and other high-availability strategies
• Extend your datacenter securely with more compute and storage
• COST LEVEL: High

Option 3: Your own company VPN

• COST LEVEL: Varies

Tips and Tricks

Script to install SSIS components on VM

What will the script do

• Get environment information.
• Make temp directory in script location.
• Download tools and connectors and install
• Remove temp directory and files. Log files will be reserved

Where to find

• On SQL VM Desktop
• https://gallery.technet.microsoft.com/scriptcenter/OOB-silent-installation-bc9339b6

How to use it

• Install.ps1
• install.ps1 -feature <toolName1>, <toolName2>…

What it installs

Script to migrate SSIS projects to VM

What will the script do

• Export catalog folders and projects to file system (.ispac).
• Import folders and .ispac files to catalog (catalog is recreated).

Where to find

• https://gallery.technet.microsoft.com/scriptcenter/ImportExport-projects-bca5f29f

How to use it

• CatalogExport.ps1 (default exporting to C:\SSIS)
• CatalogImport.ps1 (default importing from C:\SSIS. Catalog is recreated with a default secret written in the script.)

Scripts to schedule tasks to start / stop VMs

Create scheduled tasks to start a single VM or a set of VMs

• Start-AzureVMsOnSchedule.ps1 –ServiceName “MyServiceName” –VMName “testmachine1” –TaskName “Start Test Machine 1” -At 8AM
• Start-AzureVMsOnSchedule.ps1 –ServiceName “MyServiceName” –VMName “test*” –TaskName “Start All Test Machines” -At 8:15AM
• Download from https://gallery.technet.microsoft.com/scriptcenter/Start-Windows-Azure-b6c179b6

Create scheduled tasks to stop a single VM or a set of VMs

• Stop-AzureVMsOnSchedule.ps1 -ServiceName “MyServiceName” -VMName “testmachine1” -TaskName “Stop Test Machine 1” -At 5PM
• Stop-AzureVMsOnSchedule.ps1 -ServiceName “MyServiceName” -VMName “test*” -TaskName “Stop All Test Machines” -At 5PM
• Download from https://gallery.technet.microsoft.com/scriptcenter/Stop-Windows-Azure-Virtual-821896a8

Customer Examples for running SSIS on Azure VM (IaaS)

Example #1 – Cost Driven

• UK Customer in medical industry
• 1 Azure VM for SQL Server and 1 Azure VM for SSIS
• The SQL Server VM runs 24/7, the SSIS VM runs as on-demand
• Cost to run the SQL VM 24/7 and SSIS VM on-demand is about 4K pounds per year, 12K  pounds for 3 years.
• Cost to set up a physical lab for 1 beefy SQL Server machine and 1 SSIS machine with a 3 years deprecation lifespan is about 15K pounds
• 3K pounds saving for using SQL/SSIS on Azure VM
• Tips: use power script to start/shut down the SSIS VM for on-demand use in order to avoid charges when VM is idle.

Example #2 – Lift and Shift

• Canadian partner in e-Commerce business
• All databases reside on Azure – need to run SSIS on Azure too!
• 1 Azure VM running 24/7 for both SQL Server and SSIS per client
• Use Azure blob storage to store SSIS package files (.DTSX) and use PowerShell Script to trigger file copying and SSIS package executions
• Running with DS 11 SQL VM with P20(1TB) disk
• Tips: Everything on Azure, no worry about on-prem data access

Example #3 – Hosted Solution

• A Puerto Rico consulting firm provide healthcare/medicare BI solution for either on-premises or as a hosted deployment in Auzre.
• Their BI solution helps calculate the insurance quality, different KPIs and Measure for their client. Some calculation can takes up to 72 hrs to do
• 1 dedicated Azure VM for SSIS running 24/7 for all clients
• Dev / test environment all on Azure VM as needed (A6 or A7)
• Cheaper to run in Azure and more flexible to setup new server
• Tips: Considering to combine SQL Server and SSIS on same machine

Previously, SSIS package execution is limited to a single machine. However, the new feature Scale Out, available in SQL Server vNext CTP1 has broken the limit.

Scale Out provides parallel package execution by distributing the executions to multiple machines so that the overall execution time can be significantly reduced. The user is able to submit a job of running multiple packages and these packages will be run in parallel on different machines. In the past, when package execution processes compete for resources such as CPU, memory etc., user will have to manually set up multiple machines with multiple SSIS catalogs and they have to write scripts to trigger execution themselves. With the Scale Out feature that is available in SSIS vNext CTP 1.0, all of these burdens are removed. You can scale out your package executions easily both on premises or on Azure VM as IaaS.

Execute packages in Scale Out

To execute packages in Scale Out from SQL Server Management Studio, right-click on the SSISDB node or the project/package to execute and select “Execute in Scale Out”. The menu item is enabled only if the server is enabled as Scale Out Master during catalog creation. See MSDN doc, Run packages in Scale Out.

In the “Execute Package In Scale Out” dialog, multiple packages can be selected to execute. Each individual package can be configured just like executing on local machine.

Furthermore, a new configuration, called “Retry count” is available for a package executing in Scale Out. It specifies the retry count if a package execution fails.

Set up a hybrid SSIS Scale Out

In cases that both on cloud vms and on premises machines are demanded to run SSIS packages, there’s no need to manage two catalogs, one for on cloud and one for on premises. Instead, a hybrid SSIS Scale Out can be built to manage all the resources.

Example: How to set up a hybrid Scale Out

The sample Scale Out is composed of a Scale Out Master on Azure VM, a Scale Out Worker on Azure VM and a Scale Out Worker on premises. (If Scale Out Master is on premises, it requires some VNet configurations to support connection from Scale Out Workers on Azure VM. To keep it simple, we place Scale Out Master on Azure VM here.) The MSDN doc Walkthrough: Set Up Integration Services Scale Out introduces the detailed instructions to install Scale Out Master and Scale Out Worker.

1. Allocate two Azure VMs and set public and private IP to static
2. Install Scale Out Master on Azure VM

During the installation of Scale Out Master, choose “Use an existing SSL certificate” instead of “Create a new SSL certificate”.

The reason is that the default created certificate does not contain public IP in the CNs, which will lead to the unsuccessful connection from on premises Scale Out Worker later. The certificate can be created and installed through the following command:

MakeCert.exe -n CN={public IP address};CN={private IP address} SSISScaleOutMaster.cer -r -ss Root -sr LocalMachine

(In future releases, the user will be able to set CNs if “Create a new SSL certificate” is selected.)

3. Add Inbound security rule for Scale Out Master VM. Open the port set during Scale Out Master installation.

4. Install Scale Out Worker on Azure VM and on premises and install Worker certificates to the VM of Scale Out Master
Remember to specify public IP in Master endpoint for on premises Worker and private IP for on cloud VM.

Done.

After setting up a hybrid SSIS Scale Out, you can enjoy manage on cloud and on premises resources with one SSIS catalog.

SSIS Scale Out supports choosing the machines to execute packages in the “Machine Selection” page of “Execute Package In Scale Out” dialog. As a result, you can easily choose to run the packages on cloud or on premises. To ensure the package is executed, at least one of the workers you selected must be enabled.

The actual machine that ran the package can be verified on the package execution report. To open the report, right-click on the package node and select “Reports”, “All executions”. Then, click “Overview” of the execution you care about. The status of the execution can also be learnt from this report.

PowerShell is a powerful task automation tool from Microsoft. With Azure PowerShell cmdlets, you can write PowerShell scripts to automate various Azure-related tasks. It would be quite nice if we can run PowerShell scripts within SSIS. Although SSIS does not provide something like Execute PowerShell Script Task out of the box, it does have an Execute Process Task which can be used to run PowerShell scripts just as easily.

Specifically, the PowerShell.exe command-line comes with a -File parameter:

Runs the specified script in the local scope (“dot-sourced”), so that the functions and variables that the script creates are available in the current session. Enter the script file path and any parameters. File must be the last parameter in the command, because all characters typed after the File parameter name are interpreted as the script file path followed by the script parameters and their values

That is, command line like

PowerShell.exe -File D:\script.ps1

effectively launches the PowerShell process to execute the script file D:\script.ps1. This command line operation amounts to the following Execute Process Task configuration:

As simple as that.

Please note that you may need to enable PowerShell script execution on the machine in order for this to work, which can be done using steps similar to the following:

1. Run gpedit.msc to open the Group Policy Editor;
2. On the left tree view, navigate to Local Computer Policy\Computer Configuration\Administrative Templates\Windows Components\Windows PowerShell;
3. On the right pane, double click on Turn On Script Execution;
4. On the editor that pops up, choose Enabled with an allow all scripts execution policy option.

As an illustration of managing Azure resources using Azure PowerShell cmdlets, the following script template can be used to delete files on Azure Storage:

$context = New-AzureStorageContext -StorageAccountName … -StorageAccountKey …
Remove-AzureStorageBlob -Container … -Blob … -Context $context

Note that wildcard characters like * can be used in the -Blob parameter to match and delete multiple files at once. For details, please refer to the Remove-AzureStorageBlob cmdlet.

Dear Customers,

I’m happy to announce that we have released a new version of SSIS Azure Feature Pack (AFP) highlighting on HDInsight support upgrade. The download links are as follows:

• AFP for SQL Server 2012
• AFP for SQL Server 2014
• AFP for SQL Server 2016
• AFP for SQL Server vNext

Since the introduction of HDInsight support in AFP, there have been profound changes made to the HDInsight service on Azure. This new release aims to support these changes.

1. Change from classic deployment model to Azure Resource Manager (ARM) model. This is an on-going effort for all Azure resources, not just limited to HDInsight. It changes the way Azure resources are managed. More details can be found here. Concerning our case, this affects the creation/deletion of HDInsight clusters which are done by the Azure HDInsight Create Cluster Task/Azure HDInsight Delete Cluster Task, respectively. In previous versions, the classic certificate-based Azure Subscription Connection Manager is used by the tasks for resource-management authentication. In this new release, a new Azure Resource Manager Connection Manager is introduced for this purpose, and the two control flow tasks are updated to use this new connection manager in place of the original one. Currently, only service principal authentication is supported by the new connection manager.
2. Change from Windows-based clusters to Linux-based. In the beginning, only Windows-based clusters are supported. Nowadays, however, Windows-based clusters are being deprecated, and Linux-based ones are taking the place. As explained by the HDInsight team: “For continued investment on the open source big data technologies, future releases of HDInsight will be available only on Linux OS. There will not be any future release of HDInsight on Windows OS. The last release of HDInsight on Windows was HDI 3.3. The support for HDI 3.3 expired on 06/27/2016 and it will be deprecated on 07/31/2017.” Following this trend, since this release, the Azure HDInsight Create Cluster Task creates Linux-based clusters instead of Windows-based ones as done in previous versions. Compared to Windows-based clusters, Linux-based require two extra properties in Azure HDInsight Create Cluster Task, namely SshUserName and SshPassword which are used to remote-connect to the clusters via SSH.
3. Introduce the new Azure HDInsight Connection Manager. This new connection manager is used by Azure HDInsight Hive Task/Azure HDInsight Pig Task to identify the target HDInsight cluster to run the script and supply authentication information.
4. Extra properties in Azure HDInsight Hive/Pig Task to fetch script execution outcome and error logs. The extra properties include an Azure Storage Connection Manager and a blob container name which are used to specify the default storage account and container associated with the cluster, respectively.

Regards,
Lingxi

Dear all,

I am very pleased to announce that the Linux version SSIS is available in CTP 2.1.

Since March 2016 Microsoft announced that SQL Server plan to support Linux platform, SSIS team has been working on any possibility to also support SSIS on Linux OS. Now you can download SQL Server Integration Services CTP 2.1 and try it out on your Linux machines or VM by command lines.

In this version, you can:

• Run SSIS packages on Linux OS, monitor and track execution status and result;
• Extract data from and to most common used sources and destinations on Linux platform;
• Do common used transformation on Linux platform;

SSIS on Linux is one of key investments in 2017. So please email me directly (lle@microsoft.com) if you have any feedbacks on it.

Installing the SSIS Package on Ubuntu

To install the mssql-server-is Package on Ubuntu, follow these steps:

1. Import the public repository GPG keys:

         $ curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add

2. Register the Microsoft SQL Server Ubuntu repository:

        $ curl https://packages.microsoft.com/config/ubuntu/16.04/mssql-server.list | sudo tee /etc/apt/sources.list.d/mssql-server.list

3. Run the following commands to install SQL Server Integration Services:

        $ sudo apt-get update

       $ sudo apt-get install -y mssql-server-is

4. After installation, please run ssis-conf:

       $ sudo /opt/ssis/bin/ssis-conf

5. Once the configuration is done, set path:

       $ export PATH=/opt/ssis/bin:$PATH

6. (Optional) if you user is not in ssis group, add current user to ssis group:

        $ sudo gpasswd -a “current user” ssis

7. Copy your SSIS package to your Linux machine and run:

       $ dtexec /F “your package” /DE “protection password”

To see the Attunity connectors in the SSIS Toolbox, you always have to install the version of the connectors that targets the same version of SQL Server as the version of SQL Server Data Tools (SSDT) installed on your computer. (You may also have earlier versions of the connectors installed.) This requirement is independent of the version of SQL Server that you want to target in your SSIS projects and packages.+

For example, if you’ve installed the latest version of SSDT, you have version 17 of SSDT with a build number that starts with 14. This version of SSDT adds support for SQL Server 2017. To see and use the Attunity connectors in SSIS package development – even if you want to target an earlier version of SQL Server – you also have to install the latest version of the Attunity connectors, version 5.0. This version of the connectors also adds support for SQL Server 2017.+

Check the installed version of SSDT in Visual Studio from Help | About Microsoft Visual Studio, or in Programs and Features in the Control Panel. Then install the corresponding version of the Attunity connectors from the following table.+

As you may already known, SSIS Scale Out is available in SQL Server vNext CTP1. it provides the capability to parallel package execution by distributing the execution to multiple machines. In addition, you can get better performance of execution log writing from SSIS Scale Out.

For the user who is used to using verbose logging level to run SSIS package, the performance of package execution will be downgraded heavily when verbose logging level is enabled, in this case the SSIS Scale Out is an excellent choice for you even you don’t need to run multiple packages parallelly.

I did a performance test to compare the performance of Scale Out execution and non-Scale Out execution. I designed a package with an empty data flow task in a for loop container as below. The for loop container is to run the empty data flow task for 5000 times.

Executing this package in non-Scale Out mode with none execution logging level it will cost 54 seconds to complete the package execution without writing any execution log. After enabling the verbose logging level, comparing taking more than 130 seconds to complete the package execution in non-Scale Out mode, it only takes around 80 seconds if executing in Scale Out. The performance is much improved.

You can get more detail about how to execute package in Scale Out from here.

Dear all,

We just announced SSIS on Linux CTP2.1 refresh, in this refresh,  besides many other improvements, we add a new feature which are queried by many customers during CTP2.1 – ODBC support.

When the package is designed to use the ODBC connection, it will communicate with the ODBC driver manager and drivers installed on Linux system, the driver manager we have tested is unixODBC, and the drivers we have tested are MySQL and SQL Server ODBC driver, in theory, we support any unicode ODBC driver as long as the driver follows the ODBC specification, ANSI ODBC driver is not supported.

With this feature, it extends the data source support SSIS on Linux, allow user to have similar experience on ODBC connections as windows, user can do the similar ODBC configurations as windows like ODBC logs, DSN and so on. Package with ODBC connections can be seamlessly migrated from windows to Linux without any changes.

To use the Linux driver, during package design, the connection string can either contains DSN or a normal connection string.

• To use DSN, the DSN need to be configured on Linux Driver manager, refer to unixODBC document on how to configure DSN on Linux system.
• To use a normal connection string, it needs contain Linux driver name, for example, this is a sample SQL Server connection string “server=server name or IP;uid=SQL Server Login name;pwd= SQL Server Login password;Driver=linux driver name like {ODBC Driver 13 for SQL Server};database=database name;”, and note the windows authentication is not supported for SSIS on Linux.

For Linux ODBC driver manager installation, you can refer to unixODBC site or SQL Server site.

For SQL Server Linux ODBC driver installation, refer to this link.

For MySQL Linux ODBC driver installation, refer to this link.

You can use command isql shipped with unixODBC driver manager to validate the driver manager and driver installed, refer to this link for more detail on how to use isql command.

One note: when using ODBC destination component, setting TransactionSize to a proper value will help to improve the performance.

Before we start, let us consider below questions:

• As SSDT is not supported on Linux, we have to design SSIS packages on Windows. But do we need to make any changes on these packages before we run them on Linux?
• If we would like to use ODBC to connect any databases like Oracle, MySQL on Linux, is the ODBC configuration the same as Windows?

Ok. In order to answer these questions, let's take a very simple example: move data from MySQL to SQL Server. Both of MySQL and SQL Server are running on different Linux machines. How does it happen? We should follow these steps one by one:

Step1: Design your SSIS package on SSDT.

So far, SSDT only supported Windows platform. All package design and validation happen on Windows machine.

Our example is simple, we only need one OLEDB Destination for SQL Server and ODBC Source for MySQL. Of course, you can add any tasks or transformations between them.

Step2: Configure ODBC connection.

In configuration of ODBC connection, we have two options:

• Configure a DSN and use it in the connection string;

• Or we can write a normal connection string;

The prerequisite of either option is that we need to install the same version MySQL driver for Windows platform. The reason why we should do this is that we design package on Windows by SSDT and SSDT only knows database driver on the same machine.

Step3: Create DSN on Linux if you use DSN in ODBC connection string.

Don't forget to install the same version MySQL driver on Linux. Finally we will run SSIS package on Linux.

By the way, we should install 32 bit driver on Windows, but 64 bit on Linux. That's historic problem of SSDT. We won't discuss it here.

If you use DSN in ODBC connection string, please create the same DSN on Linux.

Step4: Copy your SSIS package to Linux and run.

After above configuration, we can copy the SSIS package to Linux machine by some tools like WinSCP.

In the end, we run SSIS package by Dtexec.

Conclusion

Although we only can develop SSIS packages on Windows and run it on Linux, package with ODBC connections can be seamlessly migrated from windows to Linux without any changes. What you should do is to install ODBC driver on both Windows and Linux with the same version.

Dear all,

I am very pleased to announce that SSIS on Linux supports RedHat in SQL Server 2017 RC1. Besides Ubuntu, you can install SSIS on RedHat and execute your packages.

If you have any feedbacks on it, please email me directly (lle@microsoft.com).

Install SSIS on RedHat

To install the mssql-server-is Package on RedHat, follow these steps:

1. Enter superuser mode.

         $ sudo su

2. Download the Microsoft SQL Server Red Hat repository configuration file.

         $ curl https://packages.microsoft.com/config/rhel/7/mssql-server.repo > /etc/yum.repos.d/mssql-server.repo

3. Exit superuser mode.

         $ exit

4. Run the following commands to install SQL Server Integration Services.

         $ sudo yum install -y mssql-server-is

5. After installation, please run ssis-conf.

         $ sudo /opt/ssis/bin/ssis-conf setup

6. Once the configuration is done, set path.

         $ export PATH=/opt/ssis/bin:$PATH

7. Copy your SSIS package to your Linux machine and run.

         $ dtexec /F "your package" /DE "protection password"

Hi all,

I'm pleased to announce that the first preview version of SSIS designer for Visual Studio 2017 is now available! This is a new milestone of SSIS designer. VS2017 is quite different with previous versions, which requires us to spend much more time and effort to fit our designer into the new VS. Now you can get the installer from this page. For a complete list of changes and known issues, please see the changelog.

Thank you all for your patience! Please let us know if you meet any issues.

Dear SSIS Users,

I'm happy to announce that we have released a new version of SSIS Azure Feature Pack, strengthening the connectivity for Azure Data Lake Store (ADLS).

Below is the list of improvements for this release:

1. Azure Data Lake Store File System Task: This task is designed to provide users the capability to perform file system operations on ADLS. As of now, download/upload operations are supported. More operations will be added in the future. Different from existing ADLS Data Flow Source/Destination, this task can:
   • copy files as opaque data instead of tabular data;
   • copy a batch of files instead of a single file;
   • support the File Expiry feature.
2. Foreach ADLS File Enumerator: This enumerator enables users to enumerate files in an ADLS directory with filters.

You can download this new version of Azure Feature Pack from the following links:

• Azure Feature Pack for SQL Server 2012
• Azure Feature Pack for SQL Server 2014
• Azure Feature Pack for SQL Server 2016
• Azure Feature Pack for SQL Server 2017

Cheers,
Lingxi

As we continue our march towards General Availability (GA) of SQL Server Integration Services (SSIS) in Azure Data Factory (ADF), we are excited to announce the release of three interrelated features for Public Preview: Enterprise Edition, Custom Setup, and 3^rd Party Extensibility for Azure-SSIS Integration Runtime (IR).

Enterprise Edition

The Enterprise Edition of Azure-SSIS IR lets you use the following advanced/premium features:

• Change Data Capture (CDC) components
• Oracle/Teradata/SAP BW connectors
• SQL Server Analysis Services (SSAS)/Azure Analysis Services (AAS) connectors and transformations
• Fuzzy Grouping/Lookup transformations
• Term Extraction/Lookup transformations

You can provision/configure your Azure-SSIS IR Enterprise Edition via PowerShell script or ADF app:

For more info, see https://docs.microsoft.com/en-us/azure/data-factory/how-to-configure-azure-ssis-ir-enterprise-edition.

The Oracle/Teradata/SAP BW connectors require you to install additional components on your Azure-SSIS IR Enterprise Edition and this can be done through the next newly released feature.

Custom Setup

The custom setup interface for Azure-SSIS IR lets you alter the default operating configuration/environment, e.g. to start additional Windows services, and or install additional components, e.g. APIs, assemblies, drivers, extensions, etc. on each node of your Azure-SSIS IR.  In general, it provides an interface to add your own setup steps during the provisioning/reconfiguration of your Azure-SSIS IR.

You can configure your custom setup by preparing a script + its associated files and uploading them into a blob container in your Azure Storage account.  You can then provide a Shared Access Signature (SAS) Uniform Resource Identifier (URI) of your container when you provision/reconfigure your Azure-SSIS IR via PowerShell script or ADF app:

Each node of your Azure-SSIS IR will download the script + its associated files from your container and execute your custom setup with elevated privileges.  When your custom setup is finished, each node will also upload the standard output of execution and other logs into your container.

For more info, see https://docs.microsoft.com/en-us/azure/data-factory/how-to-configure-azure-ssis-ir-custom-setup.

Using custom setup, you can install free/unlicensed components, e.g. those that are custom-made, Open Source, etc. and paid/licensed components, e.g. those that are provided by ISVs who have adapted their licensing model for Azure-SSIS IR through the next newly released feature.

3rd Party Extensibility

The flexibility/scalability of Azure-SSIS IR makes the typical licensing methods used by ISVs for on-premises installations of their components inadequate.  Consequently, we have created new Windows environment + SSIS system variables for ISVs to bind and validate their licenses.

For more info, see https://docs.microsoft.com/en-us/azure/data-factory/how-to-develop-azure-ssis-ir-licensed-components.

Here are some of our ISV partners who have adapted their components/extensions for Azure-SSIS IR:

• Theobald Software:
  • https://theobald-software.com/blog/2018/02/02/making-xtract-is-available-in-azure-ssis/
• Pragmatic Works:
  • https://pragmaticworks.com/Products/Task-Factory/Feature/Azure-Data-Factory-Edition-SSIS
  • http://blog.pragmaticworks.com/install-activate-task-factory-on-an-azure-data-factory-ssis-integration-runtime
• COZYROC:
  • https://www.cozyroc.com/blog/ssis-18-now-support-adfv2-ir
• AecorSoft:
  • http://www.aecorsoft.com/blog/2018/4/26/use-azure-data-factory-to-bring-sap-data-to-azure
• KingswaySoft:
  • http://www.kingswaysoft.com/blog/2018/04/30/Running-SSIS-Integration-Toolkit-on-the-Cloud 
• ZappySys:
  • https://zappysys.com/blog/run-ssis-azure-data-factory-deploy-monitor-ssis-cloud/
• More coming soon, so watch this space!

If you have any questions with regards to building components/extensions for Azure-SSIS IR, please contact ExtendAzureSSIS@microsoft.com.

As we continue our march towards General Availability (GA) of SQL Server Integration Services (SSIS) in Azure Data Factory (ADF), we are excited to announce the release of Execute SSIS Package activity as the most direct and code-free method to invoke SSIS package executions within ADF pipelines.

In the past, our customers resorted to using Stored Procedure activity to do the same, but they were required to create a linked service to connect to Azure SQL Database (DB) server hosting their catalog of SSIS projects (SSISDB) and T-SQL script to invoke stored procedures in SSISDB that create, configure, and start SSIS package executions (see https://docs.microsoft.com/en-us/azure/data-factory/how-to-invoke-ssis-package-stored-procedure-activity).

Now they can simply drag-n-drop the first-class Execute SSIS Package activity into their ADF pipeline, select Azure-SSIS Integration Runtime (IR) to run their package, and specify the path to their deployed package in SSIDB:

For more info, see https://docs.microsoft.com/en-us/azure/data-factory/how-to-invoke-ssis-package-ssis-activity.

Execute SSIS Package activity lets you do the following:

• Schedule SSIS package executions on Azure-SSIS IR by triggering runs of ADF pipelines that contain Execute SSIS Package activities. As you may have known already, SQL Server on premises and Azure SQL Managed Instance (MI) have SQL Server Agent and MI Agent, respectively, as a first-class SSIS job scheduler.  Azure SQL DB, on the other hand, does not have any.  Consequently, if you use Azure SQL DB to host SSISDB, you can use ADF as a first-class SSIS job scheduler.

• Modernize your ETL (Extract – Transform – Load) workflows by chaining/grouping Execute SSIS Package activities with other activities within ADF pipelines. For example, you can create Web activities with webhooks to Azure Automation PowerShell (PSH) runbook and chain Execute SSIS Package activity between them to start and stop your Azure-SSIS IR on demand/just in time before and after your package execution (see https://docs.microsoft.com/en-us/azure/data-factory/how-to-schedule-azure-ssis-integration-runtime).

• Complement your ELT (Extract – Load – Transform) workflows by splicing/injecting built-in SSIS connectors, tasks, and data transformations within ADF pipelines. If there is no built-in SSIS component that is suitable for your purpose, you can always use custom/Open Source/3^rd party components for it (see https://docs.microsoft.com/en-us/azure/data-factory/how-to-configure-azure-ssis-ir-custom-setup).  In short, you can mix-n-match various SSIS and ADF components to get the best of both ETL and ELT worlds, completing/enhancing your data integrations along the way.

If you use SSIS project/package parameters in your packages, you can assign values to them when running Execute SSIS Package activities within ADF pipelines in the following manners:

• Create SSIS execution environments in SSISDB using SQL Server Management Studio (SSMS) with variables and their values that correspond to your project/package parameters. Configure your SSIS packages using SSMS to associate those environment variables to your project/package parameters.  Switch between those environments when running Execute SSIS Package activities by specifying different paths to those environments in SSIDB on Settings tab of ADF User Interface (UI)/app (see the above screenshot), essentially assigning different values of environment variables to your project/package parameters.

• Assign values/expressions/functions that can refer to ADF system variables to your project/package parameters in JSON format on Advanced tab of ADF UI/app. For example, you can assign ADF pipeline parameters to your SSIS project/package parameters:

In the near future, we will provide more direct and code-free methods on ADF UI/app to assign values to SSIS project/package parameters, connection managers, and property overrides just like using SSMS.

Speaking of SSMS, once you run your pipelines containing Execute SSIS Package activities, you can monitor the pipeline runs on ADF UI/app and obtain the relevant SSISDB operation/execution IDs from the output of your activity runs to check more comprehensive execution logs/error messages on SSMS:

One more thing, we have also released a new scheduling feature on SSMS that will fast-track the introduction of ADF capabilities to SSIS users.  This feature lets you schedule SSIS package executions on Azure-SSIS IR by automatically generating ADF pipelines, Execute SSIS Package activities within those pipelines, and ADF triggers that schedule the pipeline runs, all through a familiar UI just like using SQL Server Agent on premises:

The auto-generated ADF pipelines/activities/triggers can then be monitored/edited/extended on ADF UI/app.

For more info, see https://docs.microsoft.com/en-us/sql/integration-services/lift-shift/ssis-azure-schedule-packages-ssms?view=sql-server-2017.

We hope you will find these new features useful to modernize and extend your ETL/ELT workflows, so please do not hesitate to contact us if you have any feedbacks/questions/issues and we will follow up ASAP.  Thank you as always for your support.

Dear Customers,

We are happy to announce that we have released a new version of SSIS Azure Feature Pack with a brand-new Azure Data Lake Analytics Task and enhanced Azure SQL DW Upload Task.

1. Azure Data Lake Analytics Task: Azure Data Lake Analytics is an on-demand analytics job service that simplifies big data. Instead of deploying, configuring, and tuning hardware, you write queries directly to transform your data and extract valuable insights. The analytics service can handle jobs of any scale instantly by setting the dial for how much power you need. You only pay for your job when it is running, making it cost-effective. Through Azure Data Lake Analytics Task in SSIS Azure Feature Pack, you can now orchestrate Azure Data Lake Analytics U-SQL jobs as part of SSIS workflow to process big data on cloud.
2. Azure SQL DW Upload Task: While the original task only supported copying from local file system, using Azure Blob Storage for staging, the new one now supports copying directly from Azure Blob Storage.

You can download this new version of Azure Feature Pack from the following links:

• Azure Feature Pack for SQL Server 2012
• Azure Feature Pack for SQL Server 2014
• Azure Feature Pack for SQL Server 2016
• Azure Feature Pack for SQL Server 2017