Sunday, October 18, 2015

Redshift: Upload Data to Redshift from SQL Server using Python Script

Download Executable Attachment

Sometimes we need to upload data to Redshift from SQL Server. But there is no straightforward way to do that. And we have to do the following task manually to import the data to redshift.
  1. Export data from SQL Server to a text file.
  2. Compressed the file to GZIP.
  3. Upload the file to S3.
  4. Execute copy command in redshift.

Wednesday, September 9, 2015

SSIS: Perform upsert (Update/Insert) using SSIS Package

It is a common scenario, when we need a script, package or something else to perform insert and update in a table simultaneously. Sometimes the task seems easy and sometimes it takes so much time to perfectly synchronize the databases. In this blog post we will try to create a SSIS package that performs the synchronization task by performing upsert (Insert/Update) using SSIS lookup transformation.

Saturday, August 8, 2015

Traditional (Row-Oriented) Database Vs Columnar Database

What is the real difference of columnar database over traditional (row-oriented) database? 

Columnar database is a database system that stores data in a column-wise fashion while in traditional database data has stored in row-wise fashion. Let’s see an example so that we can visualize how data will be stored for these two approaches. Below image shows our example record set.

Tuesday, July 14, 2015

Redshift: How to alter column datatype in redshift

There is no direct way to change the column datatype in redshift. If we want to change the column name we can use redshift alter statement with rename keyword like,

alter table BDPlayers rename column category to grade;

Thursday, July 2, 2015

Redshift: Store and Querying JSON object in Redshift

Problem: How to store JSON objects in redshift and also apply filter, group by, order by etc on that object?

Solution: Suppose we have a sparse table, where we need to add many columns and it could be exceed the maximum column limit supported by redshift. In that situation we can store the data for a row in key:value pairs in a single JSON string. JSON uses UTF-8 encoded text string so we can store JSON string in redshift text or varchar type column. For demonstration purpose we have created a table and insert some dummy JSON object. Below is the script.

Thursday, June 18, 2015

Redshift: Some Interesting behaviour of Redshift Query

I have found some interesting behaviour of redshift query. It is interesting in a sense that the query runs perfectly in redshift but not in other database (SQL Server or PostgreSQL). Let’s see the behaviour. First one describes in this blog post by brother Khorshed Alam.

Sunday, June 7, 2015

SSIS: Excel Import 255 character Truncation issue and Solution

Few days ago we have created a SSIS package to load excel data in our SQL Server database. When we run the package for a sample excel file, it runs perfectly. But when we run the package for original excel file it fails.

Sunday, May 31, 2015

How to Create, Deploy and Configure SSIS Package: Part 1

Data loading from various source is a common task for data centric application. SSIS package is a very handy and easy to learn tools to load data from various source like excel, flat file or other database. In this SQL Server Integration Service(SSIS) series we try to learn, how to create, deploy, configure and monitor SSIS package. In first article of the series we will create a simple SSIS package which extract data from excel file then load the data to a SQL server database.

Initial Setup

Before we do anything else, we need to create necessary table to which we want to load data from the excel file. Lets create the.

       Id int IDENTITY(1,1) PRIMARY KEY,
       Name nvarchar(500) NULL,
       FirstColumn nvarchar(500) NULL,
       SecondColumn nvarchar(500) NULL,
       GroupOrder int NULL