Data Warehouse Creation: Test Results

From ICISWiki

Jump to: navigation, search


Creation of: Data Warehouses required by Zeus-ICIS
#
script version
MySQL version
Operating System
Machine Specs Name of tester
Date and time started
Date and time ended
Result/ Status

Result Details

Stored Procedure Name

Result Details


Duration

Database
Count Germplasm
select count(distinct gid) 
from germplsm;

Count study datasets

select distinct(count e.represno) 
from effect e, factor f 
where e.factorid = f.factorid;
Count wh_columns
select count(*) 
from wh_columns;
count wh_data
select count(*)
from wh_data;
Count wh_datasets
select count(*)
from wh_datasets;
Indices for warehouse tables

1
06262009b (create_icis_dms_warehouse.sql)
5.1.26
Windows
Intel(R) Core2 Duo T7250 2.00 GHz

2 GB RAM

Mylah
June 28, 2009, 15:00
June 28, 2009 15:00
Syntax errors
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a


2
06262009b (create_icis_dms_warehouse.sql)
5.0.22
Windows
Intel(R) Core2 Duo T7250 2.00 GHz

2 GB RAM

Mylah
June 28, 2009, 15:23

June 28, 2009, (to retrieve from log)
2/3 tables reported done


reported :

ERROR 1054 (42S22): Unknown column 'c.datacol' in 'where clause'

(activate_icis_dms_warehousing())
  • get_wh_columns


  • assign_column_order



  • get_wh_data




  • get_wh_data_all_row_numbers


  • 4 min 16.27 sec
  • 6 min 14.52 sec
  • 3 hours 30 min 27.28 sec
  • 5 hours 26 min 21.49 sec

IRIS

iris_myisam from backup made Sept 28, 2008

2,433,855
1,910
35,584


TODO: check if correct values were stored

17,729,276


TODO: check if correct values were stored



1,944

No log about completion of data Population


TODO: verify

Warehouse table indices defined BEFORE data population

3
06262009B (create_icis_dms_warehouse.sql)
5.0.32
Linux

Castor Server

see bottom of page for specs

Jeff

June 28, 2009,

11:00

June 29, 2009

1/3 tables done

ERROR yet unknown/ session terminated

(activate_icis_dms_warehousing())
  • get_wh_columns


  • assign_column_order


  • 10 min
  • 10 min
IWIS

iwis3_myisam



68, 356
above 50 million records

Warehouse table indices defined BEFORE data population

4
06302009B (create_icis_dms_warehouse.sql)
5.0.32
Linux

Castor Server

see bottom of page for specs

Jeff
June 29, 2009,

20:00

June 30, 2009
1/3 tables done

ERROR: Disk Space Full for writing

(activate_icis_dms_warehousing())
  • get_wh_columns


  • assign_column_order
  • 9 min
  • 10 min

IWIS

iwis3_myisam



68, 356
52,841,248

Warehouse table indices defined BEFORE data population

5
06302009B (create_icis_dms_warehouse.sql)
5.0.32
Linux
Castor Server

see bottom of page for specs

Jeff
July 01, 2009, 13:30

July 02, 2009,

4:40

All tables generated (activate_icis_dms_warehousing())
  • get_wh_columns


  • assign_column_order


  • get_wh_data


  • get_wh_data_all_row_numbers


  • set_wh_data_dataval_with_valdesc

TOTAL RUNTIME:

15 hours and 10 mins

IWIS iwis3_myisam
56,663,997
2,021
68,356
53, 883, 238
2,021
Warehouse table indices defined BEFORE data population

6
06262009b (create_icis_dms_warehouse.sql)
5.0.32
Linux
Castor Server

(see bottom of page for specs)

Ching
July 02, 2009, 16:45

Process stopped after get_wh_data().

get_wh_data_all_row_numbers() CALLed manually.


(activate_icis_dms_warehousing())
  • 10 minutes: get_wh_columns(), assign_column_order()


  • get_wh_data


  • get_wh_data_all_row_numbers


  • set_wh_data_dataval_with_valdesc

Duration unknown

IWIS iwis3_myisam
56,663,997
2,021
68,356
53, 883, 238
2,021
Warehouse table indices defined BEFORE data population

7
06262009c (create_icis_dms_warehouse.sql)
5.0.22
Windows
Intel(R) Core2 Duo T7250 2.00 GHz

2 GB RAM

Mylah

July 3, 2009,

16:10


No error but last stored procedure execution time was not logged
(activate_icis_dms_warehousing())
  • get_wh_columns



  • assign_column_order



  • get_wh_data




  • get_wh_data_all_row_numbers



  • set_wh_data_dataval_with_valdesc



  • 4 min 32.80 sec
  • 6 min 28.66 sec
  • 3 hours 22 min 59.41 sec
  • 5 hours 26 min 57.09 sec
  • no log
IRIS

iris_myisam from backup made Sept 28, 2008

2,433,855
1,910
35,584
17,729,276
1,944
Warehouse table indices defined BEFORE data population

8
06262009B (create_icis_dms_warehouse.sql)
5.0.32
Linux
Castor (see bottom of page for specs)
Ching
July 14, 2009 20:12
End time not recorded (to retrieve from log)
All tables generated.
(activate_icis_dms_warehousing())
  • get_wh_columns



  • assign_column_order



  • get_wh_data



  • get_wh_data_all_row_numbers



  • set_wh_data_dataval_with_valdesc




  • 1 min, 12.44 secs


  • 5 min, 32.25 sec

IMIS

imis_myisam

159,516
10,800
97,212
3,282,648
10,800
Warehouse table indices defined BEFORE data population

9
create_cropfinder_warehouse.sql
5.0.32
Linux
Castor (see bottom of page for specs)
Ching
July 22, 2009 11:15
July 22, 2009 21:18

(activate_cropfinder_warehousing())

  • create_fieldsetup()
  • create_factors()
Total runtime: approx 9 hours

IWIS3

iwis3_myisam








10
create_icis_dms_warehouse.sql
5.0.32
Linux
Castor (see bottom of page for specs)
Franjel
July 27, 2009 13:10


activate_icis_dms_warehousing('iwis3_public_07072009');
Completed in 14 hours 25 min 34.73 sec

IWIS3

iwis3_public_07072009

5,250,467
2,080
72,137
57,405,518
2,080
Warehouse table indices defined BEFORE data population

12
count_datapoints.sql
5.0.32
Linux
Castor (see bottom of page for specs)
Ching



create_wh_countdatapoints() Completed in 2 mins, 51.25 secs (601 rows in new table wh_countdatapoints)

IRIS

iris_myisam








13
count_datapoints.sql
5.0.32
Linux
Castor (see bottom of page for specs)
Ching



create_wh_countdatapoints() Completed in 11 mins, 40.04 secs (437 rows in new table wh_countdatapoints)

IWIS3

iwis_myisam








14
create_icis_dms_warehouse.sql
5.0.32
Linux

CRIL4

(see bottom of page for specs)

Luisa
July 15, 2009
July 16, 2009
Created wh_data, wh_columns, wh_datasets and wh_countdatapoints
activate_icis_dms_warehousing()

Completed in 1 day 9 hours 58 mins 54.46 sec (397 records in wh_countdatapoints)

IWIS3 central

iwis3p_myisam_20090918

5,068,850
596
72,137
57,405,518
2,080
Warehouse tables indices defined AFTER data population

15
create_cropfinder_warehouse.sql
5.0.32
Linux
CRIL4

(see bottom of page for specs)


Luisa
July 16, 2009
July 18, 2009

Able to process the create_fieldsetup() and it took 1 min and plus seconds.

The create_factors() process took 2 days plus and didn't continue. 

activate_cropfinder_warehousing()
Not completed
IWIS3 public
5,068,850
596



Warehouse tables indices defined AFTER data population

16
create_cropfinder_warehouse.sql
5.0.32
Linux
CRIL4

(see bottom of page for specs)


Luisa
Oct 5, 2009
Oct 9, 2009
Able to process the create_fieldsetup() and it took 1 min and plus seconds.

The create_factors() process took 2 days 15 hours 12 mins 33.17 secs.

The create_variates() has an error "unknown column w.valtype in field list". The w.valtype should be in wh_countdatapoints. The create_icis_dms_warehouse.sql that creates the wh_countdatapoints is the previous version before the cropfinder warehouse is updated.

activate_cropfinder_warehousing()
Not completed

IWIS3 public


IWIS3 central

5,068,850


5,258,801

596


600

72,137


72,152

57,405,518


58,806,065

2,080


2,086

Warehouse tables indices defined AFTER data population

17
create_icis_dms_warehouse.sql
5.0.32
Linux
CRIL4

(see bottom of page for specs)


Ching
Oct 20, 2009
Oct 20, 2009
All tables (wh_columns, wh_data, wh_dataset, wh_countdatapoints) generated. activate_icis_dms_warehousing() Completed in 7 hours 41 mins 54.28 secs

IRIS iris_myisam_20091006


IRIS central

2,826,697

1944

35,616 20,409,082 1944
18
wh_create_germplasm_details.sql
5.0.32
Linux
CRIL4

(see bottom of page for specs)


Ching
Dec 16, 2009
Dec 16, 2009
create_wh_germplasm_details() Completed in 2 hours, 59 mins and 9.92 secs

IRIS iris_myisam_20091106


IRIS central







Machine Specs

Castor server

Server name and IP: IRRI Castor server (202.123.56.214)
Server OS: Linux Debian
Hardware specs: 8 processors; Intel(R) Xeon(R) CPU; E5405 @ 2.00GHz

24 GB Memory

Memory Allocation (MySQL):

CRIL4 Server (database preparation server)

Server name and IP: IRRI CRIL4 server (172.29.4.169)

Server OS: Linux Debian

Hardware specs: Pentium D 2.8Ghz, 2GB Memory, 500GB HD

Personal tools