Data Warehouse Creation: Test Results
From ICISWiki
| # | script version | MySQL version | Operating System | Machine Specs | Name of tester | Date and time started | Date and time ended | Result/ Status |
Result Details Stored Procedure Name |
Result Details
Duration | Database | Count Germplasmselect count(distinct gid) from germplsm; |
Count study datasets select distinct(count e.represno) from effect e, factor f where e.factorid = f.factorid; | Count wh_columnsselect count(*) from wh_columns; | count wh_dataselect count(*) from wh_data; | Count wh_datasetsselect count(*) from wh_datasets; | Indices for warehouse tables | |
| 1 | 06262009b (create_icis_dms_warehouse.sql) | 5.1.26 | Windows | Intel(R) Core2 Duo T7250 2.00 GHz
2 GB RAM | Mylah | June 28, 2009, 15:00 | June 28, 2009 15:00 | Syntax errors | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | | |
| 2 | 06262009b (create_icis_dms_warehouse.sql) | 5.0.22 | Windows | Intel(R) Core2 Duo T7250 2.00 GHz
2 GB RAM | Mylah | June 28, 2009, 15:23 | June 28, 2009, (to retrieve from log) | 2/3 tables reported done
ERROR 1054 (42S22): Unknown column 'c.datacol' in 'where clause' | (activate_icis_dms_warehousing())
|
|
IRIS iris_myisam from backup made Sept 28, 2008 | 2,433,855 | 1,910 | 35,584
| 17,729,276
| 1,944
No log about completion of data Population
| Warehouse table indices defined BEFORE data population | |
| 3 | 06262009B (create_icis_dms_warehouse.sql) | 5.0.32 | Linux |
Castor Server see bottom of page for specs | Jeff |
June 28, 2009, 11:00 | June 29, 2009 |
1/3 tables done ERROR yet unknown/ session terminated | (activate_icis_dms_warehousing())
|
| IWIS
iwis3_myisam | | | 68, 356 | above 50 million records | | Warehouse table indices defined BEFORE data population | |
| 4 | 06302009B (create_icis_dms_warehouse.sql) | 5.0.32 | Linux |
Castor Server see bottom of page for specs | Jeff | June 29, 2009,
20:00 | June 30, 2009 | 1/3 tables done
ERROR: Disk Space Full for writing | (activate_icis_dms_warehousing())
|
|
IWIS iwis3_myisam | | | 68, 356 | 52,841,248 | | Warehouse table indices defined BEFORE data population | |
| 5 | 06302009B (create_icis_dms_warehouse.sql) | 5.0.32 | Linux | Castor Server
see bottom of page for specs | Jeff | July 01, 2009, 13:30 |
July 02, 2009, 4:40 | All tables generated | (activate_icis_dms_warehousing())
|
TOTAL RUNTIME: 15 hours and 10 mins | IWIS iwis3_myisam | 56,663,997 | 2,021 | 68,356 | 53, 883, 238 | 2,021 | Warehouse table indices defined BEFORE data population | |
| 6 | 06262009b (create_icis_dms_warehouse.sql) | 5.0.32 | Linux | Castor Server
(see bottom of page for specs) | Ching | July 02, 2009, 16:45 | |
Process stopped after get_wh_data(). get_wh_data_all_row_numbers() CALLed manually.
| (activate_icis_dms_warehousing())
|
Duration unknown | IWIS iwis3_myisam | 56,663,997 | 2,021 | 68,356 | 53, 883, 238 | 2,021 | Warehouse table indices defined BEFORE data population | |
| 7 | 06262009c (create_icis_dms_warehouse.sql) | 5.0.22 | Windows | Intel(R) Core2 Duo T7250 2.00 GHz
2 GB RAM | Mylah |
July 3, 2009, 16:10 | | No error but last stored procedure execution time was not logged | (activate_icis_dms_warehousing())
|
| IRIS
iris_myisam from backup made Sept 28, 2008 | 2,433,855 | 1,910 | 35,584 | 17,729,276 | 1,944 | Warehouse table indices defined BEFORE data population | |
| 8 | 06262009B (create_icis_dms_warehouse.sql) | 5.0.32 | Linux | Castor (see bottom of page for specs) | Ching | July 14, 2009 20:12 | End time not recorded (to retrieve from log) | All tables generated. | (activate_icis_dms_warehousing())
|
|
IMIS imis_myisam | 159,516 | 10,800 | 97,212 | 3,282,648 | 10,800 | Warehouse table indices defined BEFORE data population | |
| 9 | create_cropfinder_warehouse.sql | 5.0.32 | Linux | Castor (see bottom of page for specs) | Ching | July 22, 2009 11:15 | July 22, 2009 21:18 | |
(activate_cropfinder_warehousing())
| Total runtime: approx 9 hours |
IWIS3 iwis3_myisam | | | | | | | |
| 10 | create_icis_dms_warehouse.sql | 5.0.32 | Linux | Castor (see bottom of page for specs) | Franjel | July 27, 2009 13:10 | | | activate_icis_dms_warehousing('iwis3_public_07072009'); | Completed in 14 hours 25 min 34.73 sec |
IWIS3 iwis3_public_07072009 | 5,250,467 | 2,080 | 72,137 | 57,405,518 | 2,080 | Warehouse table indices defined BEFORE data population | |
| 12 | count_datapoints.sql | 5.0.32 | Linux | Castor (see bottom of page for specs) | Ching | | | | create_wh_countdatapoints() | Completed in 2 mins, 51.25 secs (601 rows in new table wh_countdatapoints) |
IRIS iris_myisam | | | | | | | |
| 13 | count_datapoints.sql | 5.0.32 | Linux | Castor (see bottom of page for specs) | Ching | | | | create_wh_countdatapoints() | Completed in 11 mins, 40.04 secs (437 rows in new table wh_countdatapoints) |
IWIS3 iwis_myisam | | | | | | | |
| 14 | create_icis_dms_warehouse.sql | 5.0.32 | Linux |
CRIL4 (see bottom of page for specs) | Luisa | July 15, 2009 | July 16, 2009 | Created wh_data, wh_columns, wh_datasets and wh_countdatapoints | activate_icis_dms_warehousing() |
Completed in 1 day 9 hours 58 mins 54.46 sec (397 records in wh_countdatapoints) |
IWIS3 central iwis3p_myisam_20090918 | 5,068,850 | 596 | 72,137 | 57,405,518 | 2,080 | Warehouse tables indices defined AFTER data population | |
| 15 | create_cropfinder_warehouse.sql | 5.0.32 | Linux | CRIL4
(see bottom of page for specs)
| Luisa | July 16, 2009 | July 18, 2009 |
Able to process the create_fieldsetup() and it took 1 min and plus seconds. The create_factors() process took 2 days plus and didn't continue. | activate_cropfinder_warehousing() | Not completed | IWIS3 public | 5,068,850 | 596 | | | | Warehouse tables indices defined AFTER data population | |
| 16 | create_cropfinder_warehouse.sql | 5.0.32 | Linux | CRIL4
(see bottom of page for specs)
| Luisa | Oct 5, 2009 | Oct 9, 2009 | Able to process the create_fieldsetup() and it took 1 min and plus seconds.
The create_factors() process took 2 days 15 hours 12 mins 33.17 secs. The create_variates() has an error "unknown column w.valtype in field list". The w.valtype should be in wh_countdatapoints. The create_icis_dms_warehouse.sql that creates the wh_countdatapoints is the previous version before the cropfinder warehouse is updated. | activate_cropfinder_warehousing() | Not completed |
IWIS3 public
IWIS3 central |
5,068,850
5,258,801 |
596
600 |
72,137
72,152 |
57,405,518
58,806,065 |
2,080
2,086 | Warehouse tables indices defined AFTER data population | |
| 17 | create_icis_dms_warehouse.sql | 5.0.32 | Linux | CRIL4
(see bottom of page for specs)
| Ching | Oct 20, 2009 | Oct 20, 2009 | All tables (wh_columns, wh_data, wh_dataset, wh_countdatapoints) generated. | activate_icis_dms_warehousing() | Completed in 7 hours 41 mins 54.28 secs |
IRIS iris_myisam_20091006
IRIS central |
2,826,697 |
1944 | 35,616 | 20,409,082 | 1944 | ||
| 18 | wh_create_germplasm_details.sql | 5.0.32 | Linux | CRIL4
(see bottom of page for specs)
| Ching | Dec 16, 2009 | Dec 16, 2009 | create_wh_germplasm_details() | Completed in 2 hours, 59 mins and 9.92 secs |
IRIS iris_myisam_20091106
IRIS central |
|
|
| |
Machine Specs
Castor server
Server name and IP: IRRI Castor server (202.123.56.214)
Server OS: Linux Debian
Hardware specs: 8 processors; Intel(R) Xeon(R) CPU; E5405 @ 2.00GHz
24 GB Memory
Memory Allocation (MySQL):
CRIL4 Server (database preparation server)
Server name and IP: IRRI CRIL4 server (172.29.4.169)
Server OS: Linux Debian
Hardware specs: Pentium D 2.8Ghz, 2GB Memory, 500GB HD

