How to Create flat file dynamically from SQL Server Table or View in SSIS Package - SSIS Tutorial

Scenario: Download Script

We have a requirement to create an SSIS Package that should create an output file dynamically from a Table or View.
The columns of View or Table can change anytime but we don't want to open package and remap column/columns. In short our SSIS Package should not care about the number of columns returned by Table or view.

Solution:

As the definition of Table or View can change anytime, It will be good idea to use Scirpt Task and handle the situation. If we use builtin source and destination Transformation, they will fail in case name or number of columns will change in source table or view.


Let's processed with step by step approach

Step 1:
Create the following variables as shown below, the values for the variables can be configured by using SSIS configuration,
This will give us freedom to change the name of file, directory path, delimiter ("," or "" etc.) and file extension (txt or csv etc.) without making modification to SSIS Package.

VarDelimiter :  Provide the delimiter that you want for your flat file
VarDirectoryPath :  Provide the directory path where you would like to create flat file dynamically
VarFileExtension : Provide the extension of file you like to create
VarFileName : Provide the name of file that you would like to create dynamically from SQL Server Table
VarLogFolder: Provide the path for Log folder in case error occur in Script Task
VarObject :  Create Object type variable to save records from SQL Table or View.



Create variables in SSIS Package to create flat file dynamically from SQL Server Table or View


Step 2:
Create OLE DB Connection manager by going to Connection Manager pane. You have to provide SQL Server Instance Name and then choose the database. After creating, I have renamed to DB_Conn_TechBrothersIT as shown below.

Create OLE DB Connection manager in SSIS Package to generate flat file dynamically


Step 3:

After creating variable, Drag Execute SQL Task in control flow pane and configure as show below. Please notice that we are using Select * here. that means get all the columns data from table or view.



Get data from SQL Server Table or View and save to Object type variable in SSIS Package to create flat file dynamically


Map the full result set to Object type variable in Execute SQL Task in SSIS Package as shown below.


Save the query results in Object type variable by using Execute SQL Task in SSIS Package


Step 4:

Bring Script Task to Control Flow Pane and then open by double clicking. We will be using C# as scripting language in our demo, you can use visual basic if you like.
Go to ReadOnlyVariable and then add the user variables those we have created above in Step 1.



Add variables to Script Task in SSIS Package to create flat file dynamically in SSIS Package from SQL Server Table or View


Step 5:

Click on Edit Script button, paste following code and save the the changes and Finally OK button.


Under #region Namespaces, I have added below code

using System.IO;
using System.Data.OleDb;


Under public void Main() { 
I have added below code.
            string datetime = DateTime.Now.ToString("yyyyMMddHHmmss");
            try {

                //Declare Variables
           string FileDelimiter = Dts.Variables["User::VarDelimiter"].Value.ToString();
           string DirectoryPath = Dts.Variables["User::VarDirectoryPath"].Value.ToString();
           string FileName = Dts.Variables["User::VarFileName"].Value.ToString();
           string FileExtension=Dts.Variables["User::VarFileExtension"].Value.ToString();
                

                OleDbDataAdapter A = new OleDbDataAdapter();
                System.Data.DataTable dt = new System.Data.DataTable();
                //Get Data From Object Type variable
                A.Fill(dt, Dts.Variables["User::VarObject"].Value);
                
                //Construct File Full Path by using Varaible values
              string FileFullPath = DirectoryPath + FileName +"_"+datetime+ FileExtension;

          int i = 0;
          
           StreamWriter sw = null;
           sw = new StreamWriter(FileFullPath, false);
           
                // Write the Header Row to File
           int ColumnCount = dt.Columns.Count;
           for (int ic = 0; ic < ColumnCount; ic++)
           {
               sw.Write(dt.Columns[ic]);
               if (ic < ColumnCount - 1)
               {
                   sw.Write(FileDelimiter);
               }
           }
           sw.Write(sw.NewLine);

           // Write All Rows to the File
           foreach (DataRow dr in dt.Rows)
           {
               for (int ir = 0; ir < ColumnCount; ir++)
               {
                   if (!Convert.IsDBNull(dr[ir]))
                   {
                       sw.Write(dr[ir].ToString());
                   }
                   if (ir < ColumnCount - 1)
                   {
                       sw.Write(FileDelimiter);
                   }
               }
               sw.Write(sw.NewLine);
           }
           sw.Close();
                Dts.TaskResult = (int)ScriptResults.Success;
        }
             catch (Exception exception)
            {
                // Create Log File for Errors
                using (StreamWriter sw = 
File.CreateText(Dts.Variables["User::VarLogFolder"].Value.ToString()
                    + "\\" + "ErrorLog_" + datetime + ".log"))
                {
                    sw.WriteLine(exception.ToString());
                    Dts.TaskResult = (int)ScriptResults.Failure;
                }

            }



Step 6:
Save the script and then Run your SSIS Package, it should create flat file in destination folder with date-time. You can change the type of extension , delimiter and can test by adding or dropping/adding columns to source table or view if all works fine.

I executed with by provide extension csv, delimiter as , and it created below flat file from my table.
How to create flat file dynamically from SQL Server Table or View in SSIS Package


Check out our other posts/ Videos on related topics