extract transform load

Posted by mitch on Stack Overflow See other posts from Stack Overflow or by mitch
Published on 2011-06-26T23:43:01Z Indexed on 2011/06/27 0:22 UTC
Read the original article Hit count: 185

Filed under:

etl

|

database-migration

Wikipedia defines a 'typical' ETL cycle as :

Cycle initiation

Build reference data

Extract (from sources)

Validate

Transform (clean, apply business rules, check for data integrity, create aggregates or disaggregates)

Stage (load into staging tables, if used)

Audit reports (for example, on compliance with business rules. Also, in case of failure, helps to diagnose/repair)

Publish (to target tables)

Archive

Clean up

..What is meant by 'Build reference data'?

© Stack Overflow or respective owner

Related posts about etl

SQL SERVER – Introduction to Adaptive ETL Tool – How adaptive is your ETL?

as seen on SQL Authority - Search for 'SQL Authority'
I am often reminded by the fact that BI/data warehousing infrastructure is very brittle and not very adaptive to change. There are lots of basic use cases where data needs to be frequently loaded into SQL Server or another database. What I have found is that as long as the sources and targets stay… >>> More
SQL SERVER – 4 Tips for ETL Software IDE Developers

as seen on SQL Authority - Search for 'SQL Authority'
In a previous blog, I introduced the notion of Semantic Types. To an end-user, a seamlessly integrated semantic typing engine significantly increases the ease of use of an ETL IDE (integrated development environment, or developer studio). This led me to think about other ease-of-use issues I have… >>> More
PostgreSQL to Data-Warehouse: Best approach for near-real-time ETL / extraction of data

as seen on Stack Overflow - Search for 'Stack Overflow'
Background: I have a PostgreSQL (v8.3) database that is heavily optimized for OLTP. I need to extract data from it on a semi real-time basis (some-one is bound to ask what semi real-time means and the answer is as frequently as I reasonably can but I will be pragmatic, as a benchmark lets say we… >>> More
ETL Operation - Return Primary Key

as seen on Stack Overflow - Search for 'Stack Overflow'
I am using Talend to populate a data warehouse. My job is writing customer data to a dimension table and transaction data to the fact table. The surrogate key (p_key) on the fact table is auto-incrementing. When I insert a new customer, I need my fact table to reflect the id of the related customer… >>> More
Kimball University: Three ETL Compromises to Avoid

as seen on SQL Server Central - Search for 'SQL Server Central'
Why neglecting slowly changing dimensions, failing to capture metadata and overlooking scope creep can be the undoing of a dimensional data warehousing initiative. >>> More

Related posts about database-migration

Create a set column with database migration in rails

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello. I need to add a new column to my users table in the database. I want the type of the column to be set. The column represents the users gender. There should be two options to the set. One form Male "m" and the other for Female "f". But i havent found any doccumentation for adding a column with… >>> More
Database migration pattern for Java?

as seen on Stack Overflow - Search for 'Stack Overflow'
Im working on some database migration code in Java. Im also using a factory pattern so I can use different kinds of databases. And each kind of database im using implements a common interface. What I would like to do is have a migration check that is internal to the class and runs some database schema… >>> More
question about MySQL database migration

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there: If I have a MySQL database with several tables on a live server, now I would like to migrate this database to another server. Of course, the migration I mean here involves some database tables, for example: add some new columns to several tables, add some new tables etc.. Now, the… >>> More
Cakephp database migration error

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello All, I am using Ubuntu + cakephp 1.3. I am trying the database migration with the help of cakeDC migration plugin. I configured the plugin as per the instructions. But when i goes to the terminal, goes to the path of the application application_path/app/ dir then fire a command 'cake migration… >>> More
Can I generate a ruby on rails database migration file from a MySQL sql file

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I have a sql script file that upon import creates a table in a MySQL database and fills it with 2800 record. These are all the postal codes for the country Belgium. Now I would like to make a Ruby on Rails database migration file from this. Any idea how I can do this? Thanks, Michael >>> More