Data Warehousing and Big Data – Inventory Management Data Warehouse
This individual coursework for CS7079 Data Warehousing and Big Data requires students to design, implement and test a data warehouse based on a business case scenario and then export and migrate data to a Big Data platform for further processing. The assessment focuses on the inventory management business process of ABC Consumer Electronics Outlet Ltd, a multi-channel consumer electronics retailer operating from six stores around London and conducting online business across the UK and Europe. The company manages more than 10,000 products across approximately ten categories and more than 200 brands. The company already uses cloud-based and business applications including Vend, Linnworks and Xerox, which generate large volumes of transactional data. However, these datasets are stored separately by the individual applications, making it difficult for managers to produce integrated reports and perform analysis to support business decision-making. A business analyst has therefore recommended the development of a data warehouse capable of meeting the company's reporting and analytical requirements. For this coursework, the solution is focused specifically on the inventory management business process. The inventory management case involves three main business activities: sending purchase orders to suppliers when stock reaches minimum levels, receiving purchase orders and storing stock in appropriate locations, and controlling and maintaining stock levels including adding new products and adjusting existing stock. The required analysis includes daily stock levels for the previous month, weekly identification of products at minimum stock levels, analysis of stock levels by brand, product type and supplier, daily and weekly sent and received stock orders, and analysis of received stock orders by supplier and month. Students must analyse and design a dimensional data model that represents these business activities. This includes defining the grain of three central fact tables, identifying appropriate dimensions, defining dimension attributes and fact measures, and producing simple star schemas showing the relationships between fact tables and dimensions. The design must be justified according to the available data sources and the reporting and analysis requirements. The implementation stage requires students to create the relational database using Microsoft SQL Server Management Studio. Students must create the database, dimension tables and fact tables, establish appropriate primary-key and foreign-key constraints, and demonstrate implementation and testing through SQL commands and their results. Test data must then be populated into the data warehouse. The Big Data component requires migration of test data from the data warehouse to an Apache Hadoop platform using the Hortonworks Data Platform. Students must export the data warehouse data to an external data file, migrate the file into Apache HDFS, create a suitable data structure for loading the data into Hive, and demonstrate Apache Pig for manipulating the loaded data. Implementation and testing of the Big Data storage environment must also be demonstrated through commands and results. The coursework concludes with a personal reflective section in which students discuss what they have learned throughout the overall coursework and the challenges encountered during the process. The final submission is expected to be a well-written, structured and well-presented report combining data warehouse design, database implementation, testing, Big Data migration and reflective learning.
Read Model Answer →