Since the project has been rescheduled, it was necessary to reshape our team.
This is necessary because our work load has not changed, it was just the project dead line that has been postponed.
The result of such reshape is still not clear, but the team will be downsized from 4 to 3 people. This does not exclude a temporary downsize to 2 people for up to one month within a couple of months or so. Only time will tell.
Monday, February 15, 2010
Day 314 - Waiting for Resolutions
We have solved most of the issues reported, mostly were wrong mapping specifications, but some implementations bugs were also found.
Almost all issues are set for the functional team to solve and relate with rule specification.
Things are at a stable speed, but a bit slow from my point of view.
Almost all issues are set for the functional team to solve and relate with rule specification.
Things are at a stable speed, but a bit slow from my point of view.
Monday, February 1, 2010
Day 307 - Good Migration Results
The prioritization had some real effects.
Some things that were stalled for weeks were finally resolved and this weekend migration was the best we ever had. Nevertheless, there's still some very old issues, 3 months old, that need to be fixed.
GIS has also been cleaned up, which means the loading was performed without any trash on the system, and the test team can now perform the tests without calling us to check if there's a real problem or is it just some dirty data false problem.
Things look promising from now on. Let's see if it keeps this way.
Some things that were stalled for weeks were finally resolved and this weekend migration was the best we ever had. Nevertheless, there's still some very old issues, 3 months old, that need to be fixed.
GIS has also been cleaned up, which means the loading was performed without any trash on the system, and the test team can now perform the tests without calling us to check if there's a real problem or is it just some dirty data false problem.
Things look promising from now on. Let's see if it keeps this way.
Wednesday, January 27, 2010
Day 301 - Priorities
Finally management has shaken things up and is enforcing the teams to close issues, specially issues that are open for too long.
This is part of a bigger strategy, finalizing the mapping and loading of some business areas. There have been three areas selected, clients, simulations and credit insurance, I'm involved in two of them and I'm happy to see that some of these things are now moving into cruse speed.
This is part of a bigger strategy, finalizing the mapping and loading of some business areas. There have been three areas selected, clients, simulations and credit insurance, I'm involved in two of them and I'm happy to see that some of these things are now moving into cruse speed.
Friday, January 22, 2010
297 - Data Loading Files Performance Problem
The migration scope has increased.
New source data has come into play and thus new mapping specifications have been made.
This has resulted in a really big data loading performance problem.
Our data transformation procedure is still fast, but the creation of the GIS loader files from the transformed data is starting to give us some headaches.
The GIS data loader has a flat file structure that is very verbose, the loading of each single value of each record is done through a 300 character text file line.
This means that a single database table row is transformed into, something like, the same number of text lines as the number of columns the row has. Plus the file and records header and footer structure.
As an example, house insurances data transformation is performed in around 4 hours, sequential time, and it generates about 184 millions of records. This is all performed in the transformation server.
These records are then exported into the GIS data loader file format from the Windows directly into the AS/400. This procedure is now taking much time, over 6 hours, sequentially, in the best case scenario.
This is obviously too much time, so we are exploring several hypotheses, ranging from creating a parallel file write process; writing the files locally, with and without compression, and transfer them via FTP to AS/400; cluster indexes with full coverage maintained in a different disk; splitting the database schemes across several disks.
Some of these techniques can, and will, be combined.
We have to tune the process from our side since there is not much the GIS or the AS/400 can do when it comes to massive data load tuning.
We are facing a lot of hard work in these next days.
New source data has come into play and thus new mapping specifications have been made.
This has resulted in a really big data loading performance problem.
Our data transformation procedure is still fast, but the creation of the GIS loader files from the transformed data is starting to give us some headaches.
The GIS data loader has a flat file structure that is very verbose, the loading of each single value of each record is done through a 300 character text file line.
This means that a single database table row is transformed into, something like, the same number of text lines as the number of columns the row has. Plus the file and records header and footer structure.
As an example, house insurances data transformation is performed in around 4 hours, sequential time, and it generates about 184 millions of records. This is all performed in the transformation server.
These records are then exported into the GIS data loader file format from the Windows directly into the AS/400. This procedure is now taking much time, over 6 hours, sequentially, in the best case scenario.
This is obviously too much time, so we are exploring several hypotheses, ranging from creating a parallel file write process; writing the files locally, with and without compression, and transfer them via FTP to AS/400; cluster indexes with full coverage maintained in a different disk; splitting the database schemes across several disks.
Some of these techniques can, and will, be combined.
We have to tune the process from our side since there is not much the GIS or the AS/400 can do when it comes to massive data load tuning.
We are facing a lot of hard work in these next days.
Day 293 - Loading Status Queries
It seems the loading procedure feedback queries cannot be optimized any further by us.
We don't even have full comprehension about the data model, since there is no documentation and we had to learn by direct observation and trail and error.
We request help to the GIS team so that they help us in validating the queries and tuning the process.
We don't even have full comprehension about the data model, since there is no documentation and we had to learn by direct observation and trail and error.
We request help to the GIS team so that they help us in validating the queries and tuning the process.
Wednesday, January 13, 2010
Day 287 - Project Due Date Postponed
The project due date has been postponed almost two months. The D-Day date is now June 1st.
This happened because sub-projects related with the target system, such as testing, were delaying.
This is actually the first time I'm on a data migration project were, once set, the official D-Day had been postponed.
As far as I know, the data migration itself was one of the few projects that was on schedule.
The data migration will have to comply with this date and this probably mean that for one month we will have one extra resource. This is still unclear and thus the resize of the team is still under analysis.
This happened because sub-projects related with the target system, such as testing, were delaying.
This is actually the first time I'm on a data migration project were, once set, the official D-Day had been postponed.
As far as I know, the data migration itself was one of the few projects that was on schedule.
The data migration will have to comply with this date and this probably mean that for one month we will have one extra resource. This is still unclear and thus the resize of the team is still under analysis.
Subscribe to:
Posts (Atom)
Posts
