Question: Introduction In this project, you will do work similar to an ETL engine using Linux tools and utilities. All of the following should be automated

 Introduction In this project, you will do work similar to an

ETL engine using Linux tools and utilities. All of the following should

Introduction In this project, you will do work similar to an ETL engine using Linux tools and utilities. All of the following should be automated with one driving bash script that runs each step or multiple steps Use a variety of different Linux utilities/filters to accomplish the tasks (tr, awk, sed, bash, sort, cut etc). You may only use standard Linux utilities, bash scripts that act as filters, and c programs that act as filters. Do not use programming languages we have not covered in class (python, perl, ruby, etc) and try not to use the same program to solve each problem. However, do use the program that makes the most sense for your problem. For example, awk is a good utility for reports and a good candidate for several other steps as well Each step should be written using a separate script or command. Do not write one big progranm to perform all the steps. The goal is to have separate programs that could be reused later for other ETL problems The script should trap reasonable errors and print useful error messages to help the user resolve The script should print messages to standard output to indicate which steps of the ETL process If no parameters are passed to your script, it should print a usage statement that makes it easy problems have completed for the user to figure out how to run the script. IfI can't figure out how to run your script, I can't grade it! The bash script should accept parameters for the following Remote file transfer parameters o remote-server: server name or ip address o remove-userid: userid to login to the remote machine. (assume you are using ssh keys so that a password is not required) remote-file: full path to the remote file on the remote server local file transfer location (optional, may assume current working directory) o o MariaDB parameters o mysql-user-id o mysql-database Keep in mind that for grading, this script will be tested with a different file from the test file. Try to think about what type of errors you could run into and handle them in your script Introduction In this project, you will do work similar to an ETL engine using Linux tools and utilities. All of the following should be automated with one driving bash script that runs each step or multiple steps Use a variety of different Linux utilities/filters to accomplish the tasks (tr, awk, sed, bash, sort, cut etc). You may only use standard Linux utilities, bash scripts that act as filters, and c programs that act as filters. Do not use programming languages we have not covered in class (python, perl, ruby, etc) and try not to use the same program to solve each problem. However, do use the program that makes the most sense for your problem. For example, awk is a good utility for reports and a good candidate for several other steps as well Each step should be written using a separate script or command. Do not write one big progranm to perform all the steps. The goal is to have separate programs that could be reused later for other ETL problems The script should trap reasonable errors and print useful error messages to help the user resolve The script should print messages to standard output to indicate which steps of the ETL process If no parameters are passed to your script, it should print a usage statement that makes it easy problems have completed for the user to figure out how to run the script. IfI can't figure out how to run your script, I can't grade it! The bash script should accept parameters for the following Remote file transfer parameters o remote-server: server name or ip address o remove-userid: userid to login to the remote machine. (assume you are using ssh keys so that a password is not required) remote-file: full path to the remote file on the remote server local file transfer location (optional, may assume current working directory) o o MariaDB parameters o mysql-user-id o mysql-database Keep in mind that for grading, this script will be tested with a different file from the test file. Try to think about what type of errors you could run into and handle them in your script

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Databases Questions!